Did the attack really work?
We checked an AI-agent security benchmark's verdicts against what the agents actually did. Here is what we found, and the open tool we built so anyone can check findings like these. AI agents now read our email, book ou
We checked an AI-agent security benchmark's verdicts against what the agents actually did. Here is what we found, and the open tool we built so anyone can check findings like these.
AI agents now read our email, book our travel and move our money. Before one of them goes near anything real, someone usually runs it through a security benchmark: a set of planted attacks, and an automated check that decides whether each attack worked. Those verdicts end up as numbers on leaderboards, and the numbers decide which models look safe.
At TRUSCOR, we wanted to know something simple: when the check says an attack worked, did it? And when it says the agent was safe, was it?
So we checked.
What we did
We took AgentDojo, a widely used benchmark for AI-agent security, and compared its verdicts with the tool calls the agents actually made. We did this on the 19,381 runs its authors published, and on new runs of 24 current models from eight developers. We published the plan for every study before running it.
Real attacks scored as safe
On the published runs, the benchmark and our audit agree on 97.7% of the runs that did not crash. The check does most of its job.
But in 433 runs, the agent carried out the attacker's decisive action, such as paying the attacker, and the benchmark scored it as "no attack". That is 11.9% of the runs our audit counts as attacks. On new runs under the current version of the benchmark, it missed 55 of 744.
Warnings scored as attacks
What we didn't expect: the error also runs the other way.
For one goal, the benchmark decides with a check on the text of the agent's reply. A problem with that check had been reported publicly before; we measured what it does on current models. When a Claude 5.x model spotted the planted instruction and warned the user about it, the warning itself was scored as a successful attack. Three AI models from three developers read the disputed replies blind, and they read all 219 Claude 5.x replies the benchmark scored as attacked as warnings. Claude 4.5 models, which say nothing about the planted text, are never scored as attacked.
A model that warns you can look less safe than one that stays quiet.
How much this matters depends on the mix of goals. Our sample runs that goal more often than the benchmark does. On it, the four Claude 5.x models were scored as attacked in 10.9% to 12.8% of cases; weighted to the benchmark's own mix, they score 1.8% to 2.1%.

Benchmark rate (AgentDojo's check) and audit attack rate (the attacker's decisive call executed) for 24 models. Preprint v1.2.
Does a recorded attack still work?
A single run of an attack is one draw. We re-ran 240 recorded attacks ten times each. 224 came back in at least half of their runs, and 201 could be shown to do so at 95% confidence.
What we took from it: one run is not proof of how often an attack works.
Upgrades stopped most known attacks, but not all: 3 of 40 Grok 4.1 Fast attacks still worked on Grok 4.7.
We also tested rules, declared in advance, for which setup changes can stop an attack. They narrowly missed their pre-set test on the three models the protocol planned (a gap of 12.8 points, 95% interval 9.2 to 16.3, against a criterion of 10). They met it over six models in an extension we added after seeing interim results (14.7, 11.9 to 17.4). Rules measured on other models carried over to a held-out model in 1 of 9 tests. We report all of it.
What we built: .sova
To make findings like these checkable, we built .sova: an open-source tool and record format. A .sova file seals one attack claim with the recipe to re-run it, the evidence, the rules for a fair comparison, and its limits. Change one byte and it shows, offline.
pip install sova-oss
sova verify finding.sova # check offline that nothing changed
sova audit runs/ # check benchmark verdicts against the tool calls
sova rerun finding.sova # re-run a finding on a model you choose
Opening a record runs nothing: sova rerun calls no model unless you pass --authorize and your own key. Today it re-runs AgentDojo findings. Re-running one of our findings on GPT-4o and GPT-5.5, three runs each, cost about USD 0.05.

Real output of sova verify and sova audit on the published GPT-4o bundle.
Limits
- All of this comes from one benchmark, one attack template and simulated apps. Nobody's real money moved.
- The blind readers were AI models.
- The paper's sole author, our founder Gautam R. Patil, wrote the audit rules, the protocol, the runner and the analysis. Every run is public so that nobody has to take that on trust.
- TRUSCOR develops SOVA, so we have a stake in it.
We shared every disputed case with AgentDojo's maintainers on their GitHub (issue #235) before posting anything. We're grateful to them for the benchmark and for publishing their runs; this work wasn't possible without them.
What's next
This is the start. Three more papers are in the works: on failures that hide until ordinary conditions line up, on replaying an agent's record to find where it went wrong, and on settling disagreements between security scanners by running the attack.
Read it, check it
- Preprint: https://doi.org/10.5281/zenodo.24290457 (cite version 1.2: doi:10.5281/zenodo.24321025)
- Tool: https://github.com/TRUSCOR/SOVA-OSS ยท
pip install sova-oss - Data: https://huggingface.co/datasets/Gautam-R-Patil/agentdojo-verdict-audit
- Rebuild the scoreboard in your browser: https://www.kaggle.com/code/gautamrpatil/rebuild-the-agentdojo-audit-scoreboard
- Code and every run: https://github.com/TRUSCOR/research
If you run agent evaluations and want a finding checked, re-run or audited, reply to this post or write to us at truscor.org.
Gautam R. Patil, founder, TRUSCOR
Originally published at gautamrpatil.substack.com.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.