Your Agent Verifier Was Validated on the Raw Trace. Production Shows It a Summary.
Your Agent Verifier Was Validated on the Raw Trace. Production Shows It a Summary. By Igor Eduardo · Austin, TX · https://igoreduardo.com A new workshop paper, Do Verifier Judgments Survive Abstraction? Measuring Veri
Your Agent Verifier Was Validated on the Raw Trace. Production Shows It a Summary.
By Igor Eduardo · Austin, TX · https://igoreduardo.com
A new workshop paper, Do Verifier Judgments Survive Abstraction? Measuring Verification Transport for LLM Agent Evaluation (arXiv 2611.18471, posted 2026-11-13), tests an assumption buried in almost every agent eval pipeline: that a verifier which judges an agent run correctly will keep judging it correctly when it sees the same run in a different form.
That assumption matters because of how agent systems are built now. A verifier (an LLM that reads an agent's trajectory and decides whether it did the right thing) is usually validated on a benchmark's clean, step-by-step trace. In production, the same execution reaches the verifier through layers:
- Composed tools. Five low-level calls get wrapped in one "workflow" action, and the verifier only sees the wrapper and its result.
- Delegation. A sub-agent does part of the work, and the verifier sees "delegated to worker → worker returned X", not what the worker did.
- Compacted history. Long runs get summarized to fit context, so earlier steps are replaced by a compaction marker.
None of these change what the agent actually did. They change what the verifier can see. The paper calls the property it measures Verification Transport: does a correct verdict survive when only the view of the execution changes?
What they report
The authors froze real trajectories from four public agent-evaluation benchmarks (process correctness, safety, pairwise preference and interactive agent behavior; 250, 758, 441 and 754 eligible instances), and built three matched views of each one: native, abstracted, and restored. Task, trajectory, outcome, ground truth, criterion and verifier are held fixed. Only the evidence exposed to the verifier changes. They ran seven verifier models across several families under four verification protocols: direct judgment, evidence-grounded judgment, an agentic verifier that can request provenance, and a deliberative second-pass audit.
Counting only judgments that were correct on the native view:
- About a third break under abstraction. The transport failure rate is 31.0% under macro abstraction (composed tools), 30.9% under delegation, and 20.3% under history compaction. Across the four benchmarks, macro and delegation land at 25.5–32.9%, compaction at 16.2–24.7%.
- It is widespread but not uniform. The effect shows up across verifier models, with magnitude varying substantially by model.
- "More careful" protocols fail differently, not less. The direct single-pass verifier has the lowest failure rate (16.4% / 17.3% / 14.1% for macro / delegation / compaction). The deliberative audit has the highest (49.3% / 50.2% / 30.7%). But the failure type differs: under macro abstraction the direct verifier gives 15.8% wrong verdicts and 0.7% no verdict, while the evidence-grounded protocol gives 10.6% wrong and 24.0% no verdict, and the deliberative one 9.4% wrong and 39.9% no verdict. The authors are explicit that a lower failure rate does not necessarily mean a more reliable verifier, and that cross-protocol comparisons need care because each protocol's native-correct subset differs.
- Most failures are about missing evidence, and are reversible. Restoring the original execution provenance recovers 85.5% of transport failures overall, while degrading only 5.2% of judgments that had survived abstraction. Recovery is 87.9% (macro), 86.9% (delegation) and 79.8% (compaction).
- Fidelity of what you restore matters. In an ablation, simply asking again on the same abstracted view recovers least; restoring ordered action and tool identities without their contents recovers more; restoring the exact provenance recovers most. For one model on one benchmark, the recovery rate climbs from 22.5% to 42.8% to 80.0% across those three, and the same ordering holds in all four model–benchmark pairs they ablated. Hiding more of the evidence (roughly 25% → 75% of a block) also raises failure, for example 24.3% → 31.4% for one model on one benchmark.
Scope, in their words: three controlled abstraction boundaries, frozen trajectories rather than abstractions that arise during live execution, restoration that assumes the original provenance is available, and a metric that measures preservation of initially correct verdicts, not overall verifier competence. Results may differ in other domains and verifier designs. Those are their numbers, not mine. I have not re-run the experiments, and I am deliberately not naming the verifier models or ranking them.
The stake
Teams are about to treat "our LLM judge agrees with labels on the benchmark" as "our LLM judge can grade our agent".
Those are different claims. The first was measured on a trace format your production stack does not produce. The second depends on what your harness actually forwards: the wrapper or the calls inside it, the sub-agent's return value or its work, the summary or the steps it replaced. If a third of correct verdicts can flip or vanish when only the view changes, a native-view accuracy number tells you surprisingly little about the verifier you deployed.
And it compounds. Verifier verdicts don't just sit in dashboards. They pick prompts, filter training data, gate releases and trigger safety stops. A verifier that silently loses evidence at a delegation boundary will reward agents that delegate the risky part.
Opinion (one sentence)
An agent verifier's accuracy is a property of the verifier and the view it was given, so any reported verifier accuracy should name the trajectory representation it was measured on, and a verifier should be re-validated on every view your production harness actually produces.
That is a preference with a limiter, not a benchmark result. If your agent is a short, flat, single-agent loop with no tool composition, no sub-agents and no summarization, native-view validation may be enough. The moment you add any of those three layers, you have changed the input distribution of your grader.
Abstention is not a pass either
The detail I would steal first is the split between wrong verdict and no verdict.
A verifier that answers "insufficient evidence" more often under abstraction looks safer than one that confidently guesses. Sometimes it is. But in a pipeline, a missing verdict has to go somewhere: it gets defaulted to pass, defaulted to fail, dropped from the denominator, or sent to a human queue nobody has staffed. Each of those choices changes your reported success rate without anything about the agent changing.
So "my evidence-grounded verifier has fewer wrong verdicts" is only half a claim. The other half is "and here is what happens to the runs it refused to grade". The paper's numbers make this concrete: the protocols with the fewest wrong verdicts under abstraction were also the ones that stopped answering most often.
What a view-aware verifier report contains (menu, not recipe)
| Column | Question it answers | Failure if skipped |
|---|---|---|
| Native-view accuracy | Does the verifier agree with labels on the clean trace? | No baseline at all |
| Per-view accuracy | Does it still agree on composed, delegated and compacted views of the same runs? | Benchmark accuracy credited to a production grader |
| Wrong vs. no-verdict split | When it fails, does it guess wrong or refuse? | Abstention silently counted as a pass or dropped |
| Abstention routing | Where do refused runs go, and who reviews them? | A hidden default decides your success rate |
| Provenance recovery | Does forwarding the inner steps fix the flips? | You can't tell missing evidence from a weak verifier |
| Harness diff | Which layer (wrapper, sub-agent, summarizer) hides what? | Regressions blamed on the agent, caused by the view |
None of that needs a new model or a private harness. It needs matched views of runs you already have, and the discipline to report the verifier's accuracy per view instead of once.
Where this connects to habits I already keep public
My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit here is the one I keep coming back to: an evidence path is part of the system under test. In retrieval, an answer can only be audited if the passages that produced it travel with it. For agents, this paper shows the same thing one layer up. A verdict can only be trusted if the execution evidence that justifies it reached the verifier. If your harness drops it, the verifier is grading a story about the run, not the run.
Steal this check
Pick one agent whose runs are graded by an LLM verifier and add four things before your next release:
- Freeze a slice of real runs with labels. Even a small set you trust.
- Render each run the way production does. Wrapped tools, delegated steps, compacted history, whatever your stack actually forwards, alongside the full trace.
- Grade both views with the same verifier and criterion. Report flips, and split them into wrong verdicts and refusals.
- Re-grade the flips with the inner steps restored. If most recover, your problem is the harness, not the judge.
Then ask whether your verifier's headline accuracy still describes the verifier you are running.
What I am not claiming
- I am not reporting this paper's results as my own, and I have not reproduced them.
- I am not ranking verifier models, agent frameworks or vendors.
- I am not saying LLM verifiers are useless. The paper's own result is that most failures recover when provenance is restored, which is a fixable engineering problem.
- I am not publishing verifier prompts, abstention rules or acceptance cutoffs. Those depend on what a wrong pass and a wrong block cost you.
The point
A verifier doesn't grade a run. It grades the view of the run you handed it. Validate it on every view your harness produces, count refusals separately from wrong answers, and forward the evidence the verdict depends on.
Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.