SpecGuard: When AI Coding Agents Write Tests That Approve Bugs

Ask a coding agent to write checks for a requirement and it will often copy what the existing code does instead. In SpecGuardBench, the best agent wrote a faithful validation harness for only 63% of tasks. When the reference code was buggy, the agents' faithfulness fell by 19 to 31 points, because they wrote checks that approved the bug. Here is a free, in-browser way to test any model for this.

The problem: tests that agree with the code

Here is a one-line requirement:

slugify(title) MUST return a string that contains only lowercase letters a–z, digits 0–9 and single hyphens, with no leading or trailing hyphen.

You give a coding agent this requirement, the current implementation and the existing test, and ask it for a validation harness: a set of checks that tells you whether an implementation meets the requirement.

The easy way out is to run the current implementation, record its outputs and assert that any other implementation returns exactly the same thing. That harness passes on the current code and looks thorough. It is also wrong in two ways:

1. It approves whatever the reference does. If the reference has a bug, the harness treats the bug as correct behaviour.
2. It rejects valid code. The requirement says nothing about accented letters. One correct implementation turns "Café au lait" into caf-au-lait, and another turns it into cafe-au-lait. Both meet the requirement. A harness that copies the reference fails the second one.

This matters more as agents write more of the code. The harness is what decides whether the agent's next change is accepted. If it checks the old code instead of the requirement, it blocks valid changes and lets real bugs through.

What the research found

SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications? (arXiv:2610.16870) tested this directly. The benchmark has 291 Python tasks covering eight kinds of verification logic. For each task there is a correct implementation, a buggy one that breaks the requirement but still passes the existing test, and valid alternatives that behave differently wherever the requirement says nothing.

A harness counts as faithful only if it rejects the bug and accepts every correct version.

Then the authors set a trap. They showed the agents the buggy implementation as the reference. If an agent were really working from the requirement, the reference wouldn't matter. It mattered a lot:

Faithfulness fell by 19 to 31 points. The agents trusted the code they were shown more than the requirement they were given.

The solution: SpecGuard, a benchmark that runs in your browser

SpecGuard is a small, free re-creation of SpecGuardBench that runs entirely in a browser tab. It ships as a tab in ZeeSpec, an open-source spec builder for AI coding agents, next to the EPIC Score tab that grades the spec itself.

It has 16 JavaScript tasks, two for each of the paper's eight types of verification logic:

Each task works like the paper's. The harness author sees the requirement, how to call the code, a reference implementation and the existing test. Kept hidden from the author are a correct implementation, a buggy one that passes the existing test, and a valid alternative.

You pick who writes the harnesses:

• A local model. It runs on your own GPU through WebGPU and is downloaded once. Nothing is sent to a server.
• Three baselines that need no GPU, which show how the scoring works.

Every harness runs in a separate background worker with no network access and a time limit of a few seconds, so model-written code can't do anything harmful or hang the page.

What the baselines show

The hand-written oracle: 100%

A harness that checks only what the requirement says is faithful on all 16 tasks. This confirms that every task can be solved: each one has a correct answer, and the scoring recognises it.

"Mirror the reference": 0%

This baseline does the easy thing described above: it asserts that the code behaves exactly like the reference. It catches every bug, which looks good. But it also rejects every valid alternative, so not one of its harnesses is faithful.

The breakdown by type shows the same pattern in all eight categories:

Open any task to see why. Here is the slugify task. The copied harness correctly rejects the bug ("Hello!" becomes hello-, with a trailing hyphen). It also rejects the valid alternative, because cafe-au-lait isn't what the reference returned, even though the requirement never mentions accents.

The harness is editable. You can fix it, click Re-run with my edits and watch the false alarm disappear.

The mirror trap: the bug becomes the rule

Switch "Reference shown to the author" to Buggy implementation (mirror trap) and run the same baseline. Now it catches 0% of the bugs, because the bug is what it copied, and it still rejects every correct implementation. This is the paper's trap experiment in its most extreme form: a harness that copies the code can never be better than the code.

Replaying the existing test: 0%

The last baseline shows why "the tests pass" proves nothing here. By design, every buggy implementation passes the existing test, so re-running it catches nothing.

Put your model on the leaderboard

Choose Local model, pick a model and click Run benchmark. Each run adds a row to a leaderboard kept in your browser, shown next to the paper's results for comparison. Then run it again with the mirror trap switched on. The gap between the two runs shows how much your model relies on the reference code instead of the requirement.

Copy share text gives you a one-line summary of your result to post.

The page also explains how scoring works and repeats the paper's trap results, so anyone you share it with can check the method:

Why this is useful

• It tests the right skill. Most coding benchmarks ask whether an agent can write code that passes tests. This one asks whether it can write tests that judge code correctly, which is the skill that keeps agent-written code honest.
• It makes a quiet failure visible. A harness copied from the code looks thorough and passes on day one. The cost shows up later, as blocked valid changes and approved bugs. The mirror trap shows that cost in seconds.
• It gives you a better instruction for your agents. If the requirement doesn't say it, don't assert it. Check error messages, ordering and extra fields only when the spec asks for them.
• It's private and free. Everything runs in your browser. Models run on your own GPU, and harnesses run in a sandbox with no network access.
• It pairs with spec scoring. EPIC Score, in the next tab, checks whether a spec is clear enough to build from. SpecGuard checks whether a model can turn a requirement into checks without copying the code. Vague specs leave room for copying, and precise ones make copying easy to spot.

Limits

SpecGuard is a small re-creation, not the benchmark itself. It has 16 hand-built JavaScript tasks against the paper's 291 Python ones, and its harness authors are single local models rather than agents that can run tools. Its scores are not directly comparable with the paper's numbers, and the leaderboard labels them that way. The task set is checked automatically: every hidden implementation is confirmed to behave as intended, and the hand-written oracle scores 100%.

Disclaimer: this article is my own interpretation of the SpecGuardBench paper. SpecGuard is an independent, simplified re-creation, not the authors' benchmark or code. The tasks, baselines and scoring are my reading of their method, and any errors are mine. For the authors' exact method and results, see the paper.

Try it: vishalmysore.github.io/zeespec/app/#specguard · Source: github.com/vishalmysore/zeespec · Paper: SpecGuardBench (arXiv:2610.16870) · Companion: How Bad Is Your Spec? Scoring Specs for AI Coding Agents