Agents ship PRs that pass their own tests — the verification receipt that proves otherwise
Agents are great at one thing: making the checks pass. The uncomfortable part is that the checks are usually files in the same repo the agent can edit. "Tests pass" from an agent is a claim, not evidence. A verification
Agents are great at one thing: making the checks pass. The uncomfortable part is that the checks are usually files in the same repo the agent can edit. "Tests pass" from an agent is a claim, not evidence.
A verification gate that actually means something produces receipts, not vibes. Three rules I use:
1. Red-on-old, green-on-new. Keep a suite the agent can't see (or can't modify). Run it against the pre-change code — it must fail. Run it against the change — it must pass. If it was already green before the fix, the test proves nothing about the fix. This one rule kills most vacuous "all tests pass" PRs.
2. Tamper-evidence. Hash the suite before the agent runs and again after. If the hashes differ, reject the PR outright. Agents optimize for green; a suite they can edit is a suite they will edit.
3. Verdicts that name the crime. PASS / FAIL / NEEDS_REVIEW, where every non-pass must name file, line, the failing input, and the rule it violates. A verdict without file+line+input is a comment, not a gate result — send it back for revision.
A receipt looks like this:
{
"suite_sha256": "9f2c…",
"red_on_old": true,
"green_on_new": true,
"verdict": "PASS",
"evidence": "pytest tests/hidden -k payment: 14 failed before, 14 passed after"
}
Honest limits: receipts prove the suite ran, not that the suite is good. Hidden suites rot — re-audit them quarterly against real failures. And an LLM reviewer still has bias; use a different model family than the author when you can.
I packaged the verdict schema, deny-list, and eval harness as the Agent Eval Kit ($17) — the review-gate guide is the part I'd steal first. For the receipt side specifically, I shipped agent-verify-receipts ($19): a record/verify CLI that writes SHA-256-chained proof receipts for eval runs, so "red-on-old, green-on-new" is a verifiable artifact instead of a screenshot. There's a free eval mini-kit at /free-eval (no email) if you want to try the harness before buying anything.
Store (BTC, no account): https://dev.to/nexusfoundry/nexus-store-live-link-auto-updated-1918
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.