↓ Skip to main content

Not Unit Tests, but Our Units Are the Problem

·15 mins
Michael Hönnig
Author
Michael Hönnig
Until autumn 2027 only available for consulting engagements with a small number of hours – remote, in German and English.

Should we give up unit tests because coding agents only produce huge numbers of fine-grained tests anyway? Or do we define the unit wrongly? This article shows why high coverage and mutation scores can be deceptive with agentic AI and what module depth has to do with refactoring safety.

Same metrics, different refactoring freedomTwo test suites with the same metrics, 100 percent coverage and 100 percent mutation score each. On the left, one test pins down every single building block: restructuring is blocked. On the right, just as many tests sit only on the outer frame, that is, on the contract; the building blocks inside are already cut differently: restructuring is free. Coverage and mutation score do not say where the tests attach.Coverage 100%Mutation Score 100%Coverage 100%Mutation Score 100%=✗ Refactoring blockedevery test pins down one building block✓ Refactoring freethe tests hold only the contractSame Metrics. Different Refactoring Freedom.Coverage and mutation score do not say where the tests attach.
Two test suites with the same metrics: one nails down every building block, the other holds only the contract

“Forbid Your Coding Agent to Write Unit Tests!”
#

Since coding agents develop ever larger parts of our software, I keep reading a provocative demand: We should forbid the agents to write unit tests. They only produce huge numbers of fine-grained tests anyway, the argument goes, which break again with the next restructuring.

I understand this criticism. And yet the consequence makes me uneasy.

If we abolish unit tests, we lose a fast feedback loop. We lose the ability to check complex business rules within seconds. And we may give up mutation testing – one of the most interesting tools for finding out whether our tests detect errors at all.

But what if the critics see a real problem and merely draw the wrong conclusion?

My thesis: Unit tests are not the problem. We define the unit too small.

The Perfect Test Suite That Makes Refactoring Harder
#

Let us imagine the implementation of a REST endpoint in a backend whose business functionality is spread over twenty classes. For every class, a coding agent generates its own tests. Every method is checked carefully. In the end we have a magnificent 100 percent code coverage and a mutation score of almost 100 percent.

Looks excellent.

Now we want to restructure the business logic: merge classes, move responsibilities, introduce new internal components and remove others. The observable behavior is supposed to stay exactly the same. That is precisely what refactoring means: changing the internal structure of code without changing its observable behavior.

Nevertheless, numerous tests fail – not because the behavior has changed, but because the tests lock in constructors, internal methods, call sequences or other implementation details.

That is one side. The other: Together with the implementation code that disappears, its tests disappear as well, although the functionality still exists, only structured differently. So precisely during the restructuring we lose the tests that were supposed to show us that the behavior has been preserved.

We have very successfully tested the current structure. But how well have we secured the permanently promised behavior?

Code coverage says which code was executed during the tests. A mutation score says which share of the generated, considered mutants was detected by the tests. Both are useful pieces of information. Neither of the two metrics says on its own at which level of abstraction the tests secure the behavior.

And exactly this information is decisive for refactoring.

A Unit Does Not Have to Be a Class
#

Equating a unit with a method or a class is convenient, but by no means compelling and in most cases even rather wrong. Martin Fowler, too, writes in Unit Test that he often treats several closely related classes as a single unit.

A unit can be a domain component: a state machine, a pricing engine, a rule set or an entire backend module. Internally it may have many classes and methods. What matters is that it offers a manageable, stable behavioral contract to the outside.

Let us take the same twenty classes as before. Instead of testing each one individually, we check the domain component through its public API. The internal classes are not replaced by test doubles, that is, not by stand-ins such as stubs, fakes or mocks; only external infrastructure is replaced by suitable test doubles where necessary. This way a sensible unit is tested as a whole and can also be restructured as a whole.

What exactly is a test double?

A test double is any object that takes the place of a production object in a test, just like a stunt double in a film. The term comes from Gerard Meszaros (xUnit Test Patterns); Martin Fowler summarizes the kinds in Test Double:

  • Dummy: is only passed around but never used, for example to fill a parameter list.
  • Fake: a working but simplified implementation that is not suitable for production, for example an in-memory database.
  • Stub: provides canned answers to the calls made during the test.
  • Spy: a stub that additionally records how it was called.
  • Mock: is pre-programmed with expectations of the calls it is supposed to receive, and verifies them.

In everyday language, “mock” is often used for all five kinds; this article says “test double” when the kind does not matter.

Now we can redistribute the twenty classes internally without having to change the tests – as long as the public behavioral contract is preserved.

That is the refactoring freedom I expect from a test suite.

Two test boundaries for the same twenty classesOn the left, each of the twenty classes is a unit of its own with its own tests; the tests lock in the structure, and a restructuring breaks or removes tests. On the right, the same twenty classes lie without tests of their own behind the public API of a domain component; the tests attach only to this API, infrastructure is replaced by test doubles, and the inside remains free to restructure.Unit = Classevery class has its own testsTests lock in the structurerestructuring breaks or removes testsUnit = Domain Componenttests only through the public APIpublic APItest doubles for infrastructureTests protect the behaviorfree to restructure inside
The same twenty classes, two test boundaries: per class or at the public API of the domain component

We can only restructure safely what lies below the interface through which our tests check.

One important restriction: Not every test of a small function is bad. An isolated algorithm with a stable contract can be an excellent unit. It becomes problematic when tests permanently declare accidental implementation boundaries to be contracts.

TDD: Development Steps Are Not Automatically Permanent Test Boundaries
#

Test-driven development does not automatically protect us from this either.

The London school is often associated with outside-in and a heavier use of mocks, the Chicago or Detroit school with classic TDD, often practiced inside-out, and a preference for state-based tests. These attributions are simplifications; no school necessarily prescribes tiny units, even though unfortunately many fellow developers understand and practice it that way. Martin Fowler describes both schools in Mocks Aren’t Stubs under the names “mockist” and “classicist”.

Both approaches can produce very small tests during development. That is not necessarily wrong: A test can help to think through a single design step.

But not every test that was useful while an implementation came into being has to remain part of the regression test suite permanently.

A human can decide later which tests should be merged, replaced or deleted. A coding agent does not necessarily do that. Without explicit guidelines, a test suite easily grows along the class structure of the moment. The agents thus possibly adopt a pattern that is already widespread in human-written codebases – and scale it.

The answer is not to abolish unit tests or TDD. The answer is to deliberately distinguish development tests from tests for explicit contract boundaries.

The Missing Dimension: Tested Module Depth
#

Two test suites can have the same coverage value and the same mutation score and still offer very different refactoring freedom.

What is missing is the depth of the tested component: How much coherent internal implementation lies behind the interface through which the behavior is checked?

The terms “deep” and “shallow module” come from John Ousterhout’s A Philosophy of Software Design (chapter “Modules Should Be Deep”): A deep module hides a lot of functionality behind a simple interface, a shallow one offers little functionality behind an interface that is elaborate by comparison.

With a deep component, a test protects the outer contract while many internal design decisions remain free. With a shallow component, hardly anything behind the tested boundary is replaceable.

Deep and shallow module under a testOn the left a deep module: a narrow interface above a lot of implementation. The test attaches to the interface and protects the contract; below it, many design decisions remain free to restructure. On the right a shallow module: a wide interface above little implementation. The test pins down almost everything; hardly anything behind the boundary is replaceable.Deep Moduletestinterfacea lot ofimplementationfree torestructuredepthTest protects the contractmany design decisions remain freeShallow Moduletestinterfacelittle implementationdepthTest pins down almost everythinghardly anything behind it is replaceable
Deep and shallow module according to Ousterhout: the depth behind the tested interface determines how much remains free to restructure

Hence my pointed thesis:

Coverage and mutation score are incomplete as a measure of refactoring safety without the tested module depth.

This is not yet a finished, scientifically validated metric. For now it is a requirement for a better assessment of test suites: We should not only ask how much is tested and how effectively the tests detect mutants, but also which internal structure they allow to be changed without being touched.

It is not enough simply to count the number of classes behind an API. An artificially narrow execute(Command) interface with a hundred independent operations is not good encapsulation. And a huge module does not become maintainable merely because its API is small. Sensible module depth also involves cohesion, coupling, module breadth (the effective breadth of the API) and module size.

Cohesion decides whether what lies behind the interface really belongs together and can be tested as a whole. Coupling decides how much has to be replaced from the outside before such a test runs at all: The more a module depends on others, the more test doubles it needs and the less the test says about the real interplay.

How these properties could be measured deserves an article of its own.

Mutation Testing at the Right Level
#

Mutation testing is particularly interesting for this consideration.

Suppose a domain component A consists of the internal components AA, AB and AC. If we kill mutants in AA, AB and AC exclusively through their respective internal unit tests, the high overall score says little about whether we can restructure A as a whole.

For the refactoring safety of A we should therefore determine a separate mutation score:

  • What is mutated is the relevant production code within A, including its internal components.
  • Only tests that address A through its defined public test boundary count toward the score.
  • Internal tests may help during development, but they do not raise this score.
Which tests count toward the mutation score of component AComponent A contains the internal components AA, AB and AC, in which mutants are generated. Tests that address A through its public test boundary count toward the mutation score of A. The internal tests of AA, AB and AC bypass the test boundary, access the subcomponents directly and do not count toward A's score.tests through A's test boundarycount toward A's scorepublic test boundary of Acomponent AAAwith mutantsABwith mutantsACwith mutantsinternal tests of AA, AB and ACdo not count toward A's score
Module-related mutation score: only tests through A's test boundary count

This gives rise to the idea of a module-related, potentially hierarchical mutation score. A module can be well secured at its own level although the tests of its subcomponents look different – and vice versa.

This score does not prove complete refactoring safety either: Mutation operators cover only certain kinds of errors; some mutants are equivalent or not observable through the chosen API. But the metric answers a far more fitting question than a global score across all tests.

What Would This Look Like in a Spring Boot Backend?
#

In a typical backend, a domain test boundary can be, for example, the REST API of a module. But it does not have to be: A self-contained state machine or a domain rule set can form an equally sensible test boundary.

For fast module-related mutation tests, I would execute the domain implementation completely and selectively replace slow external dependencies – such as database access, messaging or external HTTP services. In Spring Boot, MockMvc can be a suitable starting point. A pure @WebMvcTest that also replaces the domain services with test doubles would be too shallow for this.

A domain module in a Spring Boot backend as a unitA domain module, for example Order, as a vertical slice: The module test calls the REST API of the module through MockMvc; this API serves as the public test boundary. RestController, Service as well as Entities, Rule Set and State Machine are executed for real in the test. Repository, messaging adapter and HTTP client at the edge of the module are replaced by test doubles; the real infrastructure behind them, that is, database, broker and external services, is used only in adapter integration tests and scenario tests.Module test with MockMvcREST API of the module = public test boundarydomain module, e.g. Order@RestController@ServiceEntitiesRule SetState Machine@RepositorydatabaseMessaging AdapterbrokerHTTP Clientexternal servicesreal in testdoublesreal infrastructure only in adapter integration and scenario tests
A domain module as a unit: the test attaches to the REST API, the domain logic runs for real, only the adapters at the edge are test doubles

Of course, test doubles carry risks of their own: They can be configured incorrectly and, especially as mocks, lock in internal call details themselves. That is why we need further tests with a different focus:

  • BDD or scenario tests check the domain behavior as fully integrated as possible, for example with Gherkin and real infrastructure.
  • Adapter integration tests specifically check repositories against a real database or messaging adapters against a real broker, for example with Testcontainers.

These tests complement the fast module mutation tests. They do not all have to feed into the same mutation score.

And there are limits: Modules can be coupled indirectly through shared database tables or external systems. Module-related mutation tests therefore prove neither complete module independence nor the correctness of all integrations. Additional tests may be necessary for that. This is important, but not the topic of this article.

Test Boundaries Belong to the Architecture
#

If we want coding agents to implement this strategy reliably, we must not leave the choice of the test level to them anew every time.

Module boundary, contract boundary and test boundary should coincide; in many codebases they do not. Modules have no clearly promised contract, and tests attach to places that were never meant as a contract.

The architecture should explicitly define which domain interfaces count as permanent test boundaries, which internal components are not to be tested directly and which infrastructure adapters may be replaced in fast tests.

In Java or Kotlin, annotations and architecture checks could help with this; package boundaries, ArchUnit, Spring Modulith or – where appropriate – the Java Platform Module System are options as well. What matters is not the specific technology, but that the rules become automatically verifiable.

An agent may then well write fine-grained tests as a temporary development aid. Before a pull request is completed, however, it must be ensured that the permanent test suite secures the behavior at the defined test boundaries. Superfluous tests that merely conserve internal structure can be dropped.

Conclusion: Not Less Safety, but the Right Safety
#

The demand to forbid coding agents to write unit tests falls short for me. It reacts to a real problem: Tests can be tied so closely to the implementation that they make every major restructuring harder.

But the solution cannot be to give up fast, precise feedback loops – least of all with agentic AI.

Agentic AI takes the coding off our hands. But precisely because of that we finally have the time to deal with software engineering: with module boundaries, contracts and test strategies. That makes the difference between vibe coding, as Andrej Karpathy called it, and vibe engineering, as Simon Willison proposes.

We have to take the unit seriously anew: as a deliberately delimited, cohesive component with a stable behavioral contract.

Then coverage and mutation testing get a context in which they also become more meaningful for refactoring safety. Not every tiny test that came into being during TDD has to stay. What matters is whether the permanent tests protect the behavior while the implementation remains changeable.

Have I always done what is described here correctly myself? Certainly not. Something about this topic has made me uneasy for a long time. Now I have thought it through thoroughly and am hopefully on the right track. But I still have to put it into practice in my own repositories – at least where the criticality justifies the effort.

The next question is therefore: What makes a module deep in the first place – and how do we measure that without being fooled by a small API or a large amount of code?

That will be the subject of a later follow-up article on module breadth, module depth, cohesion, coupling and module size.

Sources and Further Reading
#

This article was translated from German with AI assistance.