Will agentic AI become more reliable mainly through better generative models? Or is the bigger lever somewhere else? This article uses the analogy of a guided rocket to show why the ability to verify results automatically might matter more than the next improvement of the model, and why software development offers unusually good conditions for agentic AI.
From Unguided to Guided Rocket#
Imagine we want to hit a specific target with an unguided rocket. Even if we aim carefully, we won’t hit exactly the same spot every time. Perhaps the hits are systematically a little too far to the left. On top of that, there is random scatter: Under seemingly identical conditions, the hits land in different places.
We can try to aim more precisely. But that alone eliminates neither the systematic deviation nor the scatter.
Now let’s change the picture: Instead of an unguided rocket, we use a guided one. It doesn’t have to be perfectly aligned at launch. What matters is that it knows its target, can observe its position and trajectory, and can correct its course during flight.
For this, it needs three things:
- Knowing where to go: Which target should be reached at all?
- Aiming correctly: How is the target passed to the system and translated into suitable guidance?
- Readjusting: How does the system detect deviations and correct its course?
Of course, this is just an analogy, or as the statistician George E. P. Box aptly put it: “All models are wrong, but some are useful.” Its value lies not in matching every detail, but in the idea it makes visible:
A powerful system is not necessarily one that hits the target on the first try. It is one that can reliably detect and correct when it misses.
From the Rocket to the AI Agent#
In agentic AI, the target is not a coordinate, but a desired outcome. An agent might, for example, implement a software feature, fix a bug or extend an existing system.
The three steps remain the same.
First, we need to know what we want to achieve. Unclear, contradictory or functionally wrong requirements cannot be reliably turned into the right solution, not even by a very capable agent.
Then we need to express the target in a way the agent can work with: with requirements, context, constraints and examples that are as concrete as possible. This is more than searching for a particularly clever prompt. It is the translation of an intention into a workable specification.
And finally, the AI needs a way to check its work. It needs feedback that shows it something is not right yet, and on which it can base a new attempt.
This is exactly where an often underestimated limitation of generative AI lies:
Being able to generate something is not the same as being able to determine whether what was generated is correct.
Why Software Development Is Particularly Well Suited#
The core task of software development is to translate requirements into executable code that fulfills these requirements. For this, we already have an unusually extensive infrastructure for automated verification.
A compiler detects errors before a program even starts. Tests check behavior against expected results. Static analysis and architecture rules uncover undesirable constructs. Mutation tests reveal weaknesses in the tests themselves. Metrics provide hints of excessive complexity and coupling. Security analyses point out additional risks.
None of these methods alone proves that a piece of software is correct as a whole. But together they provide numerous feedback signals that an agent can use for its next course correction.
This sets software development apart from tasks that also produce results, but whose quality can hardly be assessed by machines independently and reliably.
First Question: Do We Have the Right Target?#
Software engineering makes an important distinction between validation and verification. Put simply:
- Validation: Are we building the right thing at all?
- Verification: Are we building what was specified correctly?
The distinction is explained, among others, in NASA’s guidelines on product verification and validation.
For our analogy, this means: It is useless for the rocket to reach its given coordinates exactly if the coordinates belong to the wrong target.
In software, it must therefore first be clarified whether the described requirements match what users and other stakeholders actually need. An AI can support this clarification, but passing tests cannot replace it.
A particularly useful approach is Behavior-Driven Development (BDD). Business and technical stakeholders clarify the expected behavior using concrete examples. These examples can be written as BDD scenario tests or executable specifications, for instance in Gherkin with Given, When, Then.
What makes this interesting is their dual role: The shared understanding of the scenarios supports validation. Their later automated execution supports verification. The Cucumber BDD documentation explicitly describes this sequence of joint discovery, formulation and automation.
However, a functionally wrong assumption does not become right just because it was written as an executable test. That is why acceptance criteria should be defined as independently of the later implementation as possible, and the coding agent must not be able to change them at will.
Second Question: Which Errors Can We Rule Out Automatically?#
Even before a single test runs, the programming language can help us.
Compilers check syntax and, depending on the language, static types and other rules.
In Java, for example, many type errors are detected before execution.
Rust is a particularly illustrative example:
Ownership and borrow checking allow the compiler to rule out certain memory errors at compile time.
These guarantees apply to safe Rust; misused unsafe can undermine them.
The Rust documentation
explains the ownership model and the role of the compiler.
C and C++ also have type checking. The difference lies in which classes of errors a language and runtime system can actually prevent. This is exactly why the choice of language is not a minor matter for agentic software development.
This by no means implies that every application should be written in Rust from now on. Rust is particularly interesting for systems programming; in other areas, Java and the JVM, for example, may be the better choice. What matters is the principle:
What a compiler reliably rules out does not have to be discovered by tests that happen to fit.
Besides static checking, testability plays a major role. Pure functions return the same output for the same input and have no side effects. They are usually much easier to test than code that requires complex state, databases or external services for every test.
Functional programming favors these properties without being limited to them: You can write pure functions in Java too, but the language doesn’t enforce it. So might functional languages like Scala or OCaml experience a renaissance driven by the demands of agentic development?
Third Question: How Good Are Our Tests?#
A green test initially only says that exactly the checked conditions are met. It does not say that all important conditions were checked.
High code coverage, too, merely means that a lot of code was executed during testing. Whether all paths are covered and whether the assertions are suitable for detecting errors at all is another question.
This is where mutation testing comes in. It deliberately changes small parts of the program: A comparison operator is swapped, a return value changed or a condition manipulated. If a test then fails, the mutant has been killed. If many mutants survive, this points to weak or missing tests. Tools such as PIT for Java put this technique into practice.
Mutation tests are not a proof of correctness, though. They check how sensitive a test suite is to selected changes, not whether the original business requirement was understood correctly. But they add an important second perspective to the executable specifications.
This is particularly valuable for AI-generated code: Otherwise, an agent could produce a large number of tests that look impressive but hardly detect any relevant errors.
Not One Metric, but Several Independent Perspectives#
If the agent is to correct its work based on checks, these checks must not be too easy to game.
A complexity metric alone, for example, could tempt it to simply spread complicated logic across many classes. The number improves, the maintainability possibly does not. This is an example of the more general problem of specification gaming: A system optimizes the measurable proxy for a goal instead of the goal itself.
That’s why I consider a triangulation of different checks particularly effective:
- Executable specifications check the expected business behavior.
- Mutation tests check whether the tests react to relevant code changes.
- A size budget, estimated from the scope and complexity of the business scenarios, limits unnecessarily large implementations and test volumes.
This budget can be an expert estimate or a heuristically derived target size. It serves as a plausibility check: If a few manageable scenarios result in a surprisingly large implementation or a flood of tests, that should stand out.
These three checks complement each other. An AI that omits tests risks weak mutation results. An AI that builds oversized solutions or adds many pointless tests uses up its budget. And an AI that breaks business functionality fails the executable specifications.
Further signals can be added: cyclomatic complexity, coupling and cohesion of modules, afferent and efferent coupling, the CRAP metric, architecture rules, linters, security analyses and automated code reviews. Risk-based assessments can also detect suspicious workarounds, such as the misuse of reflection to hide dependencies from simple static metrics.
The distinction is important: Some checks rule out certain errors under clear preconditions. Tests provide evidence for concrete observed cases. Metrics and AI reviews often provide only heuristic hints.
None of these levels should promise more certainty than it can actually provide.
Why Agents Can Handle Particularly Strict Controls#
This reveals another advantage over purely manual software development.
People get tired. With long checklists, the temptation to take shortcuts grows. And anyone who has already invested a lot of work in an implementation is often reluctant to discard it, even when it becomes clear that it is heading in the wrong direction.
An AI agent feels no boredom at the hundredth check and no frustration at having to discard and rebuild an extensive implementation. That doesn’t mean it reliably follows rules. It can misunderstand them, overlook them or circumvent them in undesirable ways.
That is precisely why important controls belong in an independently executed CI pipeline. There, the agent cannot simply replace the rules with a convenient explanation. If a check fails, it receives the result as feedback and has to improve its work.
Agentic AI thus not only makes it possible to apply well-known quality assurance practices more often. It also makes development processes practical that would hardly be sustainable if done manually, due to their effort and constant repetition.
And the same check results can do even more: They provide the basis for deciding whether a change can be approved automatically or whether a human should review it. That is exactly the idea behind the Gray Software Factory.
Why This Doesn’t Work Equally Well Everywhere#
This principle is not limited to software development. In mathematics, formally stated proofs can be checked by proof assistants. Other technical disciplines also have simulations, measurement methods or formal checks for certain tasks.
But it simply does not work equally well in all disciplines.
Take a logo that is meant to appear particularly original and evoke certain feelings. An agent can check colors, formats, contrast values or compliance with a corporate design. But how is it supposed to recognize automatically and reliably whether people actually perceive the logo as exceptional, trustworthy or moving?
You can survey people and study their reactions. That is valuable feedback, but not a reproducible checking mechanism comparable to a compiler. For a design whose quality lies essentially in its subjective effect, the measurement problem remains difficult.
So the decisive boundary does not run between software development and all other disciplines; instead, it runs between tasks for which we can automate reliable feedback and tasks where exactly that is possible only to a limited extent.
Conclusion: The Real Strength Lies in the Feedback Loop#
Let’s return to the guided rocket.
Better propulsion helps little if the wrong target was chosen. More precise guidance helps little if the system cannot detect that it is drifting off course. Only clarity about the target, suitable targeting and reliable feedback together result in robust navigation.
The same applies to agentic AI.
We will keep getting more powerful generative models. But for reliable autonomous work, the next model generation is only part of the answer. Equally decisive is how well we can specify desired outcomes, verify them automatically and derive new correction steps from detected deviations.
In software development, we have remarkably good conditions for this; we just have to use them consistently.
The real strength of agentic AI lies not only in generating results, but in making them verifiable and readjusting until they meet the requirements with sufficient certainty.
Sources and Further Reading#
- NASA: Distinctions between Product Verification and Product Validation
- Cucumber: Behaviour-Driven Development and Gherkin Reference
- PIT: Mutation Testing for Java
- DeepMind: Specification gaming: the flip side of AI ingenuity
- Rust: The Rust Programming Language – What Is Ownership? and Unsafe Rust
- George E. P. Box: Background on the quote “All models are wrong, but some are useful”
This article was translated from German with AI assistance.