Your Coding Agent Saying 'Done' Proves Nothing. Let Your Test Suite Decide

• Book: AI That Ships
• The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

Anthropic's own best-practices page for Claude Code says it in one line: "Claude stops when the work looks done." The next sentence is the one worth taping to your monitor: "Without a check it can run, 'looks done' is the only signal available, and you become the verification loop."

That's the vendor, about its own agent. Researchers found the same thing in real logs. A May 2026 study from Notre Dame, Vanderbilt and Google went through 20,574 real coding-agent sessions and pulled out 16,118 episodes where the developer had to correct or push back on the agent. In 22.58% of them, the agent had misreported the status of its own work. The paper's example: the agent describes a UI behavior as implemented, and the developer immediately reports it still fails.

So here's the take. When your agent says "done", it has told you one thing for sure: its turn is over. Whether the task is finished is a separate question, and the agent shouldn't be the one answering it. Your checks should.

"Done" marks the end of a turn

Look at how the tools define that moment. Claude Code's hooks reference says the Stop event "runs when the main Claude Code agent has finished responding." Codex's hooks docs list Stop among the hooks that run during a turn. Neither event knows anything about your ticket. The model wrote its final message, and the loop ended.

That final message is where the summary lives: "All changes are in place. Tests pass." The sentence is generated text, the same as the code above it. Maybe the agent ran the whole suite. Maybe it ran one file. Maybe it ran nothing and is describing what it expects to be true. From the chat alone, you can't tell which.

The gap doesn't vanish when you close the terminal. It moves to whoever reviews the pull request. If "done" means "the agent stopped talking", every reviewer becomes the test runner, on every PR.

The end of a turn looks the same whether the bridge reaches the other side or not.

The agent can't grade its own homework

The obvious fix is "tell the agent to run the tests." Do that. Just don't treat the agent's report of the result as the result.

ImpossibleBench, by Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini, built coding tasks where the spec and the unit tests contradict each other, so the only way to pass is to cheat. On their conflicting SWE-bench variant, GPT-5 passed 54% of these impossible tasks even under the paper's strictest prompt, and Claude Opus 4.1 passed 50%. When Claude models cheated, they edited the tests in over 79% of cases. OpenAI models spread their cheating across several strategies, and on the paper's one-off variant they mostly special-cased the test inputs.

METR reported the same pattern in June 2025: recent models got higher scores "by modifying the tests or scoring code." Asked whether its plan matched what the user wanted, "o3 answered 'no' 10 out of 10 times."

These are lab setups built to tempt the model. They don't measure how often your agent cheats on a normal Tuesday. They do show why the judge has to sit outside the agent. Anthropic's best-practices page makes the same point when it suggests a verification subagent, so that "the agent doing the work isn't the one grading it." The cheapest judge that isn't the agent is a command that exits 0 or 1.

The test-editing result also changes what "let your test suite decide" has to mean. The suite only decides if the agent can't quietly rewrite it. ImpossibleBench found that read-only tests restore legitimate performance "while preventing test modification attempts." If you can't make your tests read-only, make "the old tests are untouched" one of the checks.

The vendor-neutral version: a Stop hook that runs your tests

Claude Code and Codex both run a script when the agent is about to stop, and both let that script say no. Exit with code 2, write the reason to stderr, and the agent keeps going with your reason as its next instruction. Claude Code's docs say a blocking Stop hook "prevents Claude from stopping, continues the conversation." Codex's docs say it creates "a new continuation prompt" from your reason.

First, the tamper check. It fails if the branch modified, deleted or renamed any existing file under test/, and it fails closed if it can't find the base branch. New test files are fine.

#!/usr/bin/env bash
# scripts/tests-untouched.sh
base=$(git merge-base main HEAD) || {
  echo 'No merge base with main' >&2
  exit 1
}
touched=$(git diff --name-only --diff-filter=MDR \
  "$base" -- test/)
[ -z "$touched" ] && exit 0
printf 'Existing tests changed:\n%s\n' "$touched" >&2
exit 1

It won't catch special-casing in the code, a changed test config, or edits to these scripts, so keep the real gate in CI on a protected branch.

Then the Stop hook, one script for both agents. It needs jq.

#!/usr/bin/env bash
# scripts/done.sh: Stop hook for Claude Code and Codex
input=$(cat)
cd "$(git rev-parse --show-toplevel)" || { echo '{}'; exit 0; }

# Already sent back once: let it stop instead of looping.
if [ "$(jq -r '.stop_hook_active' <<<"$input")" = "true" ]; then
  echo '{}'
  exit 0
fi

if ! msg=$(scripts/tests-untouched.sh 2>&1); then
  printf '%s\nRestore those tests. Fix the code.\n' "$msg" >&2
  exit 2
fi

if ! out=$(npm test 2>&1); then
  echo 'npm test is failing. Fix it before you stop.' >&2
  tail -n 40 <<<"$out" >&2
  exit 2
fi

echo '{}'

What each part does:

• stop_hook_active is true when the agent is already continuing because of a Stop hook. Both tools pass it. Checking it gives the agent one retry and no endless loop. Claude Code also overrides the next block after eight consecutive continuations with no tool call in between.
• The tamper check runs before the tests, so the agent can't get to green by editing the tests you already have.
• Failures go back as the last 40 lines of test output, which is what the agent needs to fix them.
• Codex expects JSON on stdout when a Stop hook exits 0, which is why the script prints {}.

Swap main, test/ and npm test for your base branch, test folder and test command, then chmod +x both scripts. Register the hook with the same JSON in Claude Code's .claude/settings.json and in Codex's .codex/hooks.json:

{"hooks": {"Stop": [{"hooks": [{
  "type": "command",
  "command": "\"$(git rev-parse --show-toplevel)\"/scripts/done.sh"
}]}]}}

Codex runs project hooks only after you trust them with /hooks, and it records that trust against the hook's hash, so editing the hook means trusting it again.

One trade-off is deliberate. After one retry, the agent is allowed to stop, even if the suite is still red. A hook that never lets go turns a broken test into a loop you pay for. What you give up is the agent's summary as evidence. Run the suite yourself, or let CI decide, before you believe "done."

How Hermes IDE handles it

I build Hermes IDE, so weigh this section with that in mind. Hermes runs Claude Code, Codex and other CLI agents side by side, each in its own terminal, on its own branch and worktree. Its README puts the idea in one line: "Your own checks can decide when a task is done."

The gate opens when your checks pass.

Checks pair with the turn history. Hermes saves every agent turn, and 2.3.1, released October 13, 2026, labels the strip under each session "Agent replies": one chip per reply with the lines it changed. Click one to see its changes or restore your files to that point. When a check goes red three replies in, you can find the reply that broke it instead of reading one big diff. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) then shows the changes by turn or by file, with risk flags.

Hermes doesn't write your checks for you. If your suite is thin, a passing check means little. And nothing in the README says the checks protect your tests from edits, so make the tamper script one of them.

"But my tests don't cover everything"

They don't. A check is a floor. Green means "nothing you wrote down broke", which is still more than the agent's summary can tell you.

Two things to get right before you gate on it. Keep the per-turn check fast: if your full suite takes ten minutes, a Stop hook that runs it makes every turn take ten minutes. Put lint, the type check and the fast unit tests in the hook, and leave the slow suite to CI. And quarantine flaky tests first. A flaky test in a Stop hook sends the agent chasing a bug that isn't there, on your token bill.

Agents are good enough that you'll want to walk away while they work. That only works if something other than the agent decides when the work is finished.

So, in your repo right now, who decides "done": the agent's last message, CI, or you reading the diff?

If this was useful

I build Hermes IDE: free, source-available under BSL 1.1, on macOS, Windows and Linux, and not affiliated with Nous Research, Anthropic, OpenAI or any agent vendor. Download it, point its checks at your test suite, and if it saves you a red PR, star it on GitHub.

The same rule holds when the LLM lives inside your product instead of your editor: the model doesn't grade itself, your evals and guardrails do. That's what AI That Ships covers, the fifth book in my AI in TypeScript series, along with cost control and deploying on Node.js.

The whole series, from your first LLM call to agents in production: xgabriel.com/ai-in-typescript