Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 6 min read

Our AI reviewer caught 10 out of 10 known vulnerabilities. Here is why I told people to ignore that number.

Every AI code review tool has the same demo. Someone opens a PR with a planted bug, the bot comments on line 42, everyone nods. It is theatre. The only question that matters is whether the thing works on your code, on a

Every AI code review tool has the same demo. Someone opens a PR with a planted bug, the bot comments on line 42, everyone nods. It is theatre. The only question that matters is whether the thing works on your code, on a Tuesday, when nothing was planted.

So when we built our own reviewer, we did the obvious thing: we took ten commits that fixed real CVEs in widely used projects, reverted each one so the working tree contained the vulnerable code again, and asked the tool to review the diff with no hints. Then we published the result β€” including a section explaining why the result is close to meaningless.

We got 10/10. And that is the least interesting sentence in this post.

The method, because a number without a method is marketing

For each case:

  1. Take a commit that fixed a real vulnerability in a widely used project (Django, axios, lodash).
  2. git revert --no-commit it, so the diff looks like somebody just introduced the problem.
  3. Strip documentation and test changes from the patch. This matters more than it sounds β€” the release notes name the CVE, and a reviewer that reads them is reading the answer key, not the code.
  4. Run the reviewer on HEAD~1 with the security agent only. No repository context, no PR description, no hints.
  5. A case counts as caught when a finding names the vulnerable file and describes the right class of bug.

Ten cases, one pass each, security agent alone, a small and cheap model, 2026-11-20.

Two of the ten had to be thrown out and re-run. In the first attempt the patch still contained the release notes, and the reviewer quoted the CVE title back at us, verbatim. That is not review, that is OCR with confidence. It was the first hint that our benchmark was measuring something other than what we thought.

Why a perfect score is a warning sign

A reverted fix is suspicious by construction. Reverting a security fix removes a length check, a validation call, a sanitizer invocation. The diff says "I deleted the guard clause" in plain sight. Any reviewer that looks at the hunk will notice, because the shape of a security fix is itself a tell. Real bugs are not introduced by deleting guards. They are introduced by accident, at 11pm, inside a 400-line feature that was mostly about something else β€” and that is a fundamentally harder detection problem.

It measures recall and never precision. Nothing in these ten runs says how often the reviewer comments on clean code. Most PRs are fine. A tool that finds every planted bug and also comments on a third of all normal PRs is not half-good, it is unusable, because every false positive costs a human the minutes it takes to read it, decide it is wrong, and move on.

Ten cases is an anecdote with a table. Add two cases and the headline moves twenty points. There is no confidence interval in a sample that size, and no honest way to present one.

Severity labels are unreliable. One case β€” password hash disclosure to view-only admin users β€” was rated low by the model. Django shipped it as a security release. The finding text was correct; the colour was not. If your workflow is "only look at critical and high", a mislabelled severity is functionally a miss even though the tool technically saw the bug.

All four of these are written on the public benchmark page, above the fold, next to the 10/10. Specifically: "A perfect score here is a sign that the benchmark is weak, not that the tool is perfect."

What happened on real pull requests

Then we ran the same reviewer over ordinary PRs, the kind where nothing is wrong. It produced a confident, plausible-sounding comment on roughly a third of them.

Some were real. Most were pattern matching on style, or a fabricated failure mode that the surrounding code already handled three lines down. Each one cost a human a few minutes of "wait, is this actually broken?" β€” which is exactly the review time that is supposed to be the thing we are saving.

That reframes the design problem. The question is not how do we find more bugs. It is when should the reviewer stay quiet.

Three things changed as a result:

  • Anchor every finding to a line and a failure path. If the model cannot name the input that triggers the problem and the consequence that follows, it does not get to speak. This cut findings by more than half and made the remaining ones readable.
  • Treat silence as a correct answer. "No comment" is the right output for most diffs, and it needs to be an explicit, cheap, first-class outcome rather than an accident of the prompt.
  • Demote severity to a hint. The text of the finding is the signal. The label is a sorting convenience that is wrong often enough that no workflow should gate on it.

Recall went down. Review time went down more. That trade is the entire product.

Why publish the weakness next to the strength

Because the failure mode of this category is not bad models. It is a demo culture where every tool shows a screenshot of a planted bug being found, and nobody shows the false positive rate. If we publish only the 10/10 we become another screenshot.

There is also a selfish reason. A benchmark you are willing to describe, with the method written down and the harness shipped as a shell script, is one other people can attack. Ours is a shell script that fetches a fix commit, reverts it, strips docs and tests, runs the CLI, and diffs against the files the fix touched. It is not pretty and it is not peer reviewed. That is precisely why it is written down: so the next version can be better, and so someone can tell us where it is wrong.

The next version of the page needs twenty ordinary pull requests and a false positive count. Until it has them, treat 10/10 as a floor on capability, not a rate of quality.

If you are evaluating a reviewer

A short list, whether you use ours or anyone else's:

  • Ask for the false positive rate on clean PRs, not the number of bugs found. Almost nobody has this number, and the ones who do will tell you an unflattering one.
  • Ask whether the eval strips the answer key (release notes, tests, commit messages). If it doesn't, you are measuring the model's ability to read.
  • Ignore severity labels in the demo and read the finding text. Ask yourself whether you would have shipped the code after reading it.
  • Ask how many cases. If the answer is under thirty, it is a story, not a benchmark.

We published ours at 10/10 with four paragraphs explaining why that does not count for much. If you want to break it, the harness and the raw runs are in the repo, and I would genuinely like the counterexample.

RevHive is the tool this post is about: a GitHub App, or a CLI you run inside your own network with your own model key. The benchmark page and the harness are public, including the four limitations:

If you have a better way to measure a reviewer's noise ratio, I want to read it β€” that is the next thing we are building.

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.