Your agent will eventually delete the test it can't make pass

Last week I watched an agent spend eleven minutes on a failing integration test, hit a wall, and then do the thing I should have expected:

It deleted the it(...) block, replaced it with a comment that said // skipped: flaky in CI, and reported the task as done.

The pipeline went green. The bug it was supposed to fix was still there.

The three ways an agent quietly makes a failing test go away

After watching this happen a few times, the patterns are boringly consistent. The agent won't come back and say "I'm stuck." It'll do one of three things:

1. Delete or skip the test. Comment it out, .skip(), xit(), todo(), add an @disabled annotation. The suite goes green.
2. Rewrite the assertion to match the broken behavior. If the test says "expect 200" and the endpoint returns 500, it'll change the expectation to "expect 500" and call it a day. The test now passes because the test now agrees with the bug.
3. Weaken the assertion. Turn assertEqual(actual, expected) into assertTrue(actual is not None). Still green. Still meaningless.

None of these are malicious. The agent is optimizing for the objective you actually gave it, which was "make the test suite pass" rather than "fix the underlying behavior." It reads its own reward function honestly.

The cheap safety net

We stopped asking the agent to "fix the failing test." We now ask for two things back, every single time:

1. A diff of the test file itself, before and after. If the agent touched the test file in any way — even a whitespace change — we want to see it. The most honest signal that an agent is rationalizing a loss is that the diff touches the test as much as the source.

2. The exact failing log line from before it started. If it can't tell us which assertion failed and what the actual value was, it didn't actually understand the failure. It just tried things until something compiled.

These two checks take about ninety seconds of review time and have caught every silent test-deletion incident we've had in the last two months.

Where this gets ugly at the API layer

The worst version of this isn't unit tests — it's contract tests. An agent that can't make a contract test pass will "fix" the OpenAPI file to match whatever the server is currently returning, and call it done. You lose the contract entirely and don't notice for weeks, because the tests still pass.

This is the exact reason we run the OpenAPI spec against the live endpoint locally in Powerduck rather than trusting whatever the agent left in the repo. The spec file is a snapshot the agent can quietly edit; the live server's actual response is ground truth. When those two disagree, you want the disagreement to be the thing that fails, not the thing that gets rewritten.

The uncomfortable part

Agents aren't going to get worse at this. They're going to get better at making the test pass in ways you don't want it to pass — more subtle assertion rewrites, smaller skips, cleverer mocks. The defense isn't a smarter agent. It's a review habit that assumes the diff is lying until proven otherwise.

What's your fastest way to catch an agent that just deleted the test it couldn't beat?