The Test That Survives Its Own Implementation

You ask your agent to cover a new function. It comes back with eight tests, all green. You move on.

Sometime later you're staring at the diff and a nagging question surfaces: what would actually break if this function returned the wrong thing?

The answer, more often than you'd like, is nothing. The tests pass because they were written to match the implementation, not to constrain it.

The mirror-writing failure mode

An agent writing tests under time pressure does the natural thing: it reads the implementation, then writes assertions that mirror what the code already does. If the function says return a + b, the test says assert add(2, 3) == 5. That's not a test. That's a transcription.

The test cannot fail, because it was derived from the same logic it's supposed to police. It's consistent with the code, not a check on the code. This is the same gap I keep hitting: an agent can tell you two pieces of code look consistent, but it can't tell you either of them is correct.

The ten-second sanity check

You don't need a mutation-testing framework to catch this. Before you trust the new test file, do the cheapest version of mutation testing by hand:

1. Open the implementation file.
2. Change one line — swap a + to a -, flip a condition, return a hardcoded wrong value.
3. Run the new tests.

If the tests stay green, they weren't testing the thing you thought they were. They were testing the implementation's ability to match itself.

I do this on every agent-authored test file now. It takes ten seconds. It has, more than once, caught a test suite that was 80% cosmetic.

The tautology tell

There's a faster signal than running the mutation. Read the test names. If they read like descriptions of the implementation rather than specifications of behavior, they're mirror tests:

• test_addsTwoNumbers — implementation description, weak.
• test_returnsSumWhenBothPositive — behavior specification, stronger.
• test_handlesEdgeCase_whenInputIsZero — behavior specification, and the only one that would catch a real bug.

Mirror tests describe what the code does. Good tests describe what the code must do, independent of how it does it. The first kind is safe on first run. The second kind is safe in a year, when someone refactors the function and quietly breaks the contract.

Why agents do this

It's not malice. It's that the agent is optimizing for the signal in front of it: green tests. The fastest path to green is to transcribe the implementation. The slow path — and the valuable one — is to reason about what the caller actually depends on, then write tests that would fail if that dependency were violated.

The fix is to change what you ask for. Don't say "add tests for this function." Say "name the three ways a caller could be wrong about this function, and write a test for each." That reframes the task from transcribing to specifying, and the tests that come back are the kind that survive their own implementation being rewritten.

The same gap in API land

This failure mode isn't limited to unit tests. I see it constantly in API work: a contract test that hits the endpoint and asserts the response matches the current schema. Green, every time. But the test was generated from the same implementation it's checking, so it can never tell you the contract drifted from what the client actually expects.

The fix is the same shape — run the check against an external source of truth, not against the implementation's own output. That's the whole reason I wound up building Powerduck the way I did: local-first, re-run the spec against the live endpoint, stop trusting the markdown. The principle carries over. A test that was written by the same hand that wrote the code is, at best, a proof of internal consistency. It is not a proof of correctness.

The habit

Before you merge an agent-authored test file, ask one question: if I deleted the implementation tomorrow, which of these tests would still pass?

The answer tells you exactly how much protection you actually bought.