Prove It Works: Building an Eval That Can Catch a Broken Benchmark

A benchmark with flawed tests measures the tests. So I built two suites — one clean, one with 12 flaws planted in the tests — and watched a do-nothing agent start scoring points while my reference agent lost 8.3 points to the tests themselves.

This is the full experiment: calibrating the graders first, pre-registering the audit bar, catching 11 of 12 flaws — and being honest about the one that got away.

The setup

Local, deterministic, no API keys — a lab for auditing your eval before your eval audits your agent. The setup is a clean 60-task suite (arithmetic, extraction and classification; gold answers cross-checked by an independent second implementation) and an identical twin with 12 flaws planted in the tests.

What the flaws did

• Null agent (outputs nothing): 0/60 on the clean suite, 2/60 on the flawed suite
• Echo agent: 0/60 on the clean suite, 4/60 on the flawed suite
• Reference agent: 60/60 on the clean suite, 55/60 on the flawed suite — 8.3 points lost to the tests, not the tasks

Calibrate the graders first

Three graders were validated on 150 labelled items before any audit run: exact-match 80.0%, normalized 100%, rubric 99.3%. The point is stated plainly in the lab: an uncalibrated grader is just another flawed test.

The audit

The audit bar was registered before the run: at least 10 of the 12 flaws caught, at most 3 false flags. The audit caught 11 of 12 planted flaws with zero false flags. The one miss, flaw-09, had its gold and its source document disagreeing (300 vs 315) from the same table — shared-source flaws are the hard class, and that is exactly what to test for next.

If you build or trust agent benchmarks, audit the proof itself, not just the score.

Repo: https://github.com/ranadhir19/agent-eval-lab