Dev.to AI 🤖 Ai 👁 0 📖 2 min read

Agents ship PRs that pass their own tests — the verification receipt that proves otherwise

Agents are great at one thing: making the checks pass. The uncomfortable part is that the checks are usually files in the same repo the agent can edit. "Tests pass" from an agent is a claim, not evidence. A verification

Agents are great at one thing: making the checks pass. The uncomfortable part is that the checks are usually files in the same repo the agent can edit. "Tests pass" from an agent is a claim, not evidence.

A verification gate that actually means something produces receipts, not vibes. Three rules I use:

1. Red-on-old, green-on-new. Keep a suite the agent can't see (or can't modify). Run it against the pre-change code — it must fail. Run it against the change — it must pass. If it was already green before the fix, the test proves nothing about the fix. This one rule kills most vacuous "all tests pass" PRs.

2. Tamper-evidence. Hash the suite before the agent runs and again after. If the hashes differ, reject the PR outright. Agents optimize for green; a suite they can edit is a suite they will edit.

3. Verdicts that name the crime. PASS / FAIL / NEEDS_REVIEW, where every non-pass must name file, line, the failing input, and the rule it violates. A verdict without file+line+input is a comment, not a gate result — send it back for revision.

A receipt looks like this:

{
  "suite_sha256": "9f2c…",
  "red_on_old": true,
  "green_on_new": true,
  "verdict": "PASS",
  "evidence": "pytest tests/hidden -k payment: 14 failed before, 14 passed after"
}

Honest limits: receipts prove the suite ran, not that the suite is good. Hidden suites rot — re-audit them quarterly against real failures. And an LLM reviewer still has bias; use a different model family than the author when you can.

I packaged the verdict schema, deny-list, and eval harness as the Agent Eval Kit ($17) — the review-gate guide is the part I'd steal first. For the receipt side specifically, I shipped agent-verify-receipts ($19): a record/verify CLI that writes SHA-256-chained proof receipts for eval runs, so "red-on-old, green-on-new" is a verifiable artifact instead of a screenshot. There's a free eval mini-kit at /free-eval (no email) if you want to try the harness before buying anything.

Store (BTC, no account): https://dev.to/nexusfoundry/nexus-store-live-link-auto-updated-1918

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.