I tested my security scanner on 480 real vulnerabilities. It pointed at 5% of them.

I maintain vetcode, an open-source security check for AI coding agents. Its main job is malware: it looks at a package before Claude Code installs it, and at a repository right after it is cloned. On that job it has numbers I trust. On the same 1,000 malicious and 1,000 clean npm packages it catches 947 with 26 false alarms, where GuardDog catches 903 with 43.

It also had a few dozen rules for web bugs: SQL built from strings, HTML written without escaping, shell commands built from variables. I had never measured them against real bugs. This post is about what happened when I did, and what I changed.

The measurement

The GitHub Advisory Database has reviewed advisories with links to the commit that fixed each bug. I took 480 of them, 60 for each of eight classes: cross-site scripting, SQL injection, command injection, SSRF, path traversal, open redirect, code injection and unsafe deserialization. They come from 262 repositories in 8 ecosystems, published between June 2025 and November 2026.

I kept only advisories with exactly one fix commit that changes existing code in a handful of files. That way the lines the commit changed mark where the bug was. For each case I fetched only those files, at the commit before the fix and at the fix, and ran the scanner on both. A case counts when the scanner reports a finding of the right kind within 3 lines of a line the fix changed.

Before looking at anything, I split the cases in two by a hash of the advisory id. 226 went into a tune half that I'm allowed to look at and adjust rules on. 254 went into a check half that is only ever counted. The script refuses to show me what it found on a check case. I added that rule because every earlier number I measured on a set I had tuned on looked better than the first look at the next set.

The result

The scanner had a finding of the right kind where the fix was in 23 of 480 cases. That's 5%.

On the same day, the same rules reported 1,532 web findings on 72 well-known open-source applications that I presume are fine.

So the web rules were noisy on clean code and almost never stood where a real bug was. Both have one cause. Most rules matched a dangerous call with the request on the same line, like exec(req.query.cmd). Real bugs are rarely written like that. The value arrives through a parameter or a model field, and the call is a few functions away. Or the bug is a check that exists but is incomplete: an SSRF filter that misses one address form, a path check that a symlink walks around. Some are plain, like eval(field_type) in Python with the value coming from a tool definition, and still got no finding.

What I changed

I stopped calling those matches findings. The scanner now reports two separate things:

• Findings, when a rule has a measured record, or sees request data reach the dangerous call.
• Places to read: a dangerous call whose argument is not fixed text, like a program started with a variable, text run as code, HTML written without escaping, or SQL built from parts. They have no severity and don't fail a build. They take one line per kind in the report, with the question a reviewer should ask of each.

I didn't pick by hand which rules became places. A script reads both measurements, how often a rule stood at a real bug on the tune half and how often it fires on the 72 clean apps, and writes the list. A rule reports places when it fires at least 10 times on the apps and more than 20 times for every real bug it hit. Twelve rules qualified. The Python deserialization rule (6 real bugs, 8 hits on clean apps) stayed a finding.

I also added a small catalogue of dangerous calls in eight languages. Five of its entries fired often on clean apps and never at a real bug (PHP include with a variable: 174 times on clean code, 0 at a bug), so the same script drops them.

The numbers after

The honest number is the last row. 17% is what a set I designed against looks like; 6% is what an untouched set says. It still points at only about one real injection bug in nine overall. It has no places for SSRF, path traversal or redirects, because every outgoing request and every opened file would be one. And 33 of the 55 cases are still pointed at after the fix, because the dangerous call is still there and what made it safe lives somewhere else.

So the change didn't make the scanner good at web bugs. It made it honest about them: 60% fewer findings for an agent to read on clean code, twice as many real bugs with something pointing at them, and a report that no longer pretends a pattern match is a vulnerability.

Something I measured and didn't build

The other way agents get attacked is through what they read: a web page or a tool's answer that carries instructions. Before building a guard for that, I ran my patterns for poisoned CLAUDE.md files on two public datasets. They caught the one fixed "ignore all previous instructions" template every time. They caught 0 of 1,054 plain injected instructions in InjecAgent and 2 of 263 in deepset's prompt-injection set. They also flagged 0.7% of clean documents. A guard like that would look like protection without being one, so I didn't ship it. Telling an injected instruction from a user's own needs a model that sees both.

What I took from it

1. Measure against real bugs, not your own examples. My unit tests all passed. The rules matched exactly the shapes I wrote them for.
2. Freeze a half before you look. The difference between my tune half (17%) and check half (6%) is the size of the lie I would have told without it.
3. Let measurements decide what a rule is, instead of arguing case by case. I had spent days patching individual rules for each new set of apps, and each fix helped one set and not the next.
4. Publish the bad numbers too. They're in the benchmark README next to the good ones, with the scripts to reproduce them.

vetcode is free, MIT licensed and has no dependencies: vetcode.openyard.cloud. If you run it, false alarms and misses are the reports I want most.

I wrote this post with help from an AI assistant (Claude). All measurements come from scripts in the repository, and I checked every number against them.