Why I stopped trusting "fixed" from my coding agent

Agent edits a file, test goes green, agent says "fixed". But that green test only tells you one thing: the command passes now. It doesn't tell you it failed before. And it doesn't tell you it would fail again without the change.

So I built FixProof, a skill file plus a small Python CLI (stdlib only, 3.9+) that runs those checks itself. Here's what it looks like on the slugify example from the repo.

The bug

def slugify(text):
    # BUG: spaces are removed instead of turned into dashes
    return text.strip().lower().replace(" ", "")

The test expects slugify("Hello World ") to be hello-world. Note the double space.

Step 1: record the failure first

$ python3 scripts/verify.py init --repro "python3 -m unittest test_slugify" --suite "python3 -m unittest test_other"
FixProof baseline: reproduction outcome = fail (exit 1)
Bug reproduced. Apply the fix, then run: verify.py check

Step 2: a fix that looks done

return text.strip().lower().replace(" ", "-")

Feels right. But the double space turns into hello--world:

$ python3 scripts/verify.py check
FixProof 0.3.0  status: FAILED  (exit 1)
  - fix does not solve the bug: reproduction still fails (3/3 runs)

An agent that skims the output would call this done. The CLI doesn't.

Step 3: the real fix

return "-".join(text.strip().lower().split())

FixProof 0.3.0  status: VERIFIED  (exit 0)
  - baseline failed, fix passes all reruns, revert test fails without the fix, no regressions
  revert test: fails_without_fix

The revert test is the part I care most about. It temporarily removes the fix, runs the repro again, checks it fails, and puts the file back. If the test still passes without your change, the test isn't actually testing the bug, and you get UNVERIFIED instead of VERIFIED.

The five results

• VERIFIED: failed before, passes after on every rerun, fails again when reverted, nothing else broke
• FAILED: repro still fails, or something that passed before now fails
• PARTIALLY VERIFIED: repro passes but something's missing (no suite, no revert test)
• UNVERIFIED: bug never reproduced, test passes even without the fix, flaky reruns
• BLOCKED: missing tool, timeout, not a git repo

What it can't do

Being honest here:

• the agent picks the repro command. If that command doesn't really hit the bug, VERIFIED just means the wrong command was verified
• an agent could delete .fixproof/ and make a new log. If it matters, rerun check yourself
• needs git
• if the fix lives inside the test file, it can't be reverted cleanly
• macOS isn't tested yet

Try it

git clone https://github.com/Livecristen/fixproof.git ~/.claude/skills/fixproof

(or ~/.codex/skills/ / .cursor/skills/)

It's v0.3.0, MIT: https://github.com/Livecristen/fixproof

Curious whether the status rules match how you work with agents. Feedback welcome.