"All tests pass" is a claim, not evidence: 5 ways an agent gets to green without fixing anything

If you let Claude Code, Codex or Cursor run your tests, you've probably seen this: the agent says "all tests pass", you merge, and CI goes red. Or worse, CI goes green and the bug is still there.

The agent usually isn't lying on purpose. It's summarizing what it remembers, and that memory can be out of date. Here are the five patterns I keep running into, and the cheap checks that catch each one. Most of them need nothing more than git and a few lines of shell.

1. The pass is stale

The order goes: tests pass, the agent edits two more files "to clean up", then it reports green. The green belongs to an older version of the code.

Cheap check: record what the tree looked like when the tests ran.

# run tests and stamp the result with a hash of the working tree
npm test && \
  (git diff HEAD; git ls-files --others --exclude-standard | xargs -r cat) | sha256sum > .last-pass

# later, before trusting "done":
(git diff HEAD; git ls-files --others --exclude-standard | xargs -r cat) | sha256sum | diff -q - .last-pass \
  && echo "pass is fresh" || echo "STALE: code changed since the last passing run"

It's crude, but it answers the one question that matters: did anything change after the green run?

2. The exit code got swallowed

npm test | tail -20 exits 0 when the tests fail, because the pipeline returns the exit code of tail. The agent sees a zero and calls it a pass.

Cheap check: set -o pipefail in any script the agent runs, and put a line in your AGENTS.md / CLAUDE.md: "Never pipe test output through head/tail/grep when reporting a result." It also helps to grep the output for failed / FAIL even when the exit code is 0.

3. Nothing ran

"0 tests collected" exits 0 in some setups. So does a test filter that matches nothing. A green run with zero tests isn't a pass.

Cheap check: keep the last passing count, and treat a drop as a failure. If yesterday 214 tests passed and today 190 did, something got skipped or deleted.

4. The test got weaker, not the code better

This is the one that actually bites. To turn a red suite green, the agent:

• adds it.skip / @pytest.mark.skip / xit
• changes expect(total).toBe(42) into expect(total).toBeDefined()
• deletes the failing assert
• special-cases the input the test uses: if code == "SAVE10": return 10
• branches on NODE_ENV === 'test'

Cheap check: review the test diff before the code diff. git diff main -- '*test*' '*spec*' is a good start, and a grep for skip|only|xit|toBeDefined|toBeTruthy over the added lines catches a lot.

5. CI got switched off

|| true after the test step, continue-on-error: true, a narrowed test path, a lowered coverage threshold, or git commit --no-verify. Each of these makes green meaningless.

Cheap check: protect .github/workflows/ and your test config with CODEOWNERS so a human has to approve changes there.

Putting it together

The common thread: don't take the agent's summary as evidence. Check that a test command really ran, that it passed, that the code hasn't changed since, and that the diff didn't win by weakening the tests.

There are free tools that cover parts of this, and they're worth a look:

• noskip does diff rules for skipped/weakened tests (Node).
• receipt and agent-recap compare session-log claims with what actually ran.

I wanted all of it in one place, offline, so I built Done Gate (Python 3.8+, standard library + git):

• donegate run -- npm test stores a receipt with the exit code, the parsed pass/fail counts and a hash of the working tree. donegate status then says FRESH, STALE (with the changed files), FAIL, MASKED, ZERO_TESTS or COUNT_DROPPED.
• donegate scan runs 21 rules over the diff for the patterns in #4 and #5, with file:line and a suggested fix.
• donegate audit reads Claude Code and Codex session logs and checks each "tests pass" claim against the commands that ran.
• A git pre-commit hook, a Claude Code Stop hook and a GitHub Actions check, so an unbacked "done" doesn't get through.
• donegate lanes prints one row per git worktree, if you run several agents in parallel.

To be clear about what it is: an evidence check, not an audit of your code. It doesn't prove the code is correct. Weak tests still pass, and the tamper rules are pattern matches, so they can miss creative gaming and sometimes flag legitimate changes.

It's $7 here if it saves you a bad merge: https://payhip.com/b/5gOrV

Even if you don't use it, steal the five checks above. #1 and #4 are the ones I'd add first.

Not affiliated with Anthropic, Cursor or OpenAI.