Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

No proof, no bug: making an AI code reviewer prove each defect with a failing test

When a coding agent finishes a change, the usual "review" is the same model reading its own output. It tends to agree with itself. During Meta's AI developer hackathon I tried a stricter rule for the reviewer: it may on

When a coding agent finishes a change, the usual "review" is the same model reading its own output. It tends to agree with itself.

During Meta's AI developer hackathon I tried a stricter rule for the reviewer: it may only report a defect if it ships a test that fails against the code in a sandbox. No failing test, no bug report.

The benchmark

I built FinCode-30: 30 small finance tasks (rounding, business-day calendars, payment splits). The hard part was not writing code from scratch, a cheap model does that fine. The hard part is code that already exists, passes its tests and is still wrong. So every task has:

  • public tests and hidden tests
  • three buggy variants: plausible one-line bugs that pass every public test and fail at least one hidden test
  • the clean reference, which no reviewer is allowed to break

On top of that there are 6 packages of three modules each, with one bug hidden in one module.

Results with an open-weights model

Held-out split (20 tasks, 60 buggy variants), prompts frozen before the run, the same open-weights model in every role:

Review mode Bugs fixed Correct code damaged
Free-text review 41/60 (68%) 0/20
Self-check (the model writes its own tests) 54/60 (90%) 0/20
Proof-gated review 57/60 (95%) 0/20

On the three-module packages: 17/18 for proof-gated review against 12/18 for free-text review. Cost was about 6 to 9 cents per target.

Where it did not help

With Meta's strongest hosted model as the reviewer, every condition fixed almost everything, and plain self-check was as good as proof-gated review and far cheaper. So the rule mostly lifts weaker models. What it adds for strong models is an audit trail: every reported defect comes with a test that fails, and each run keeps code, claims, tests and results, hashed with SHA-256 so it can be re-checked later.

One surprise: the reviewer's proven claims sometimes also failed on my own reference solutions. I read each of them against the spec; none contradicted it, and about half were real defects my references shared (for example dates with a trailing newline).

Limits

One model family, 30 tasks, finance only. I don't know yet how much of this carries over to larger codebases or messier domains.

Code, benchmark and full result tables: https://github.com/matrixagentsocial/matrix-globe

If you run agent code review in CI, I'd like to hear where this breaks.

This article was written with the help of AI (Claude) and checked against the published result tables before posting.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.