Same reviewer, three models: is the noise sampling or model choice?
The multi-pass experiment ended with a confession: nineteen commits on one model cannot tell you whether the ±21-point cross-pass spread we measured is a property of that model, of the provider, or of the task. The next
The multi-pass experiment ended with a confession: nineteen commits on one model cannot tell you whether the ±21-point cross-pass spread we measured is a property of that model, of the provider, or of the task. The next cell we could afford was the 10-CVE suite — the same ten revert-diffs from the original benchmark — run on three models with everything else held constant.
The design, including the flag we had to build
Three models, same pipeline (two agents, same prompts, same reconciliation code), one pass per cell:
- `agnes-3.0-flash` — the model behind every number we have published so far
- `agnes-2.5-pro` — same provider, a reasoning model
- `sensenova-ds` (DeepSeek V4.1 via a free gateway) — a different provider entirely
One design problem surfaced immediately. Our CLI silently rotates to a smaller model when the primary hits a rate limit — the right behavior for production uptime, and exactly wrong for a benchmark, where a substituted model's answer pollutes the cell it lands in. (It polluted our *first* benchmark that way, before we knew to look.) So we built `--fallback none`: the named model's answer, or a recorded failure. Never a substitution.
Results: every model missed a different case
Hit = the report names the core vulnerability the revert reintroduced (judged on finding title, description, and anchored file path; the three disagreement cases were verified manually). Raw JSON for all thirty runs is in the repository under `benchmark-runs/2026-11-21-3model/`.
| case | agnes-3.0-flash | agnes-2.5-pro | sensenova-ds |
|---|---|---|---|
| c01 SSRF / local file read | hit | hit | hit |
| c02 password hash disclosure | hit | hit | hit |
| c03 empty vs missing headers | hit | hit | hit |
| c04 unescaped log messages | hit | hit | hit |
| c05 unbounded multipart memory | hit | hit | hit |
| c06 forged formset actions | hit | hit | hit |
| c07 translation parser memory | hit | hit | miss |
| c08 unvalidated URL rendering | hit | hit | hit |
| c09 axios SSRF | hit | miss | hit |
| c10 template command injection | hit | hit | hit |
| total | 10/10 | 9/10 | 9/10 |
The headline is not that one model won. It is that **the two misses do not overlap**. `agnes-2.5-pro` returned risk 0 and zero findings on c09 — silent on a reintroduced SSRF. `sensenova-ds` missed c07 entirely. No single model is strictly better; an ensemble voting 2-of-3 would have caught all ten. We are not claiming that as a feature — it is a hypothesis the next cell has to earn.
The case that flipped
The original benchmark reported agnes-3.0-flash at 8/10, missing c02 (password hash disclosure). Today, the same model on the same revert returns a CRITICAL anchored at `django/contrib/admin/utils.py:435` describing exactly that vulnerability.
Attribution honesty: we cannot credit a single change. Since that first run the security prompt gained the evidence-cap rubric, findings gained real file anchoring, and severity reconciliation moved from prose to code. Anchoring is post-processing, so it did not make the model *see* the bug — but prompt changes and plain sampling noise are both live candidates, and one run cannot separate them. What we can say: the published 8/10 is no longer the current number, and the published number will be re-baselined multi-pass before anyone treats either as a rate.
The fail-closed gate met real data
On `sensenova-ds`, the free tier's TPM limit kept killing the LogicAgent — in 6 of 10 runs. Under the old pipeline, those runs would have shipped a verdict computed from half the checks. Instead, every one of them carries a withheld risk score: findings listed, verdict refused, reason in the summary. Six withheld reports are the clearest demonstration we have that the gate does its job on real provider failures — a partial review cannot masquerade as a clean one.
Practical read: the free tier is workable for *evaluation* (the SecurityAgent completed all ten cases and found 9/10 vulnerabilities), but production-shaped reviews need a provider that finishes what it starts. That is why our BYOK table still puts paid DeepSeek in the recommended slot.
Which noise is bigger?
Risk-score spread, median (raw JSON behind every number):
- Across three models, same case, same pass: **median 30** (n=3 cases where all three models scored; the withheld runs are excluded, not zeroed)
- Across passes, same model, same case: **median 21** (n=19 commits, the multi-pass cell)
- Across two models of the same provider, same case: **median 14** (n=9)
Reading it: crossing providers moves the verdict about as much as re-rolling the same model. Swapping models inside one provider moves it less than either. Neither number is a substitute for the other — **model choice does not buy you stability, and multi-pass does not buy you a model that sees c09**. If your CI gates on a single-run score from a single model, you are stacking both noise sources and calling the result a verdict.
One correction in our own tooling
Our first automated pass of the hit matrix scored one case wrong: the matching script compared keywords against finding titles and descriptions but not anchored file paths, so a correct critical on `django/utils/translation/trans_real.py` read as a miss. We caught it while verifying the disagreement cases, fixed the script, and re-ran the matrix before publishing anything. We mention it because it is the same failure mode we criticize in vendor benchmarks — an eval script that quietly mislabels results — and it happened to us on a Saturday afternoon with the publish button in reach.
Honest limits
- One pass per cell. The multi-pass post showed single-run numbers carry ±21 points of sampling noise; these hit rates are one sample, not a rate. The 10/10 will get a multi-pass re-baseline before we print it on the benchmark page.
- Hit judgment is ours, on named findings, verified manually only for the disagreement cases.
- CVE-diff shapes: these are reverts of real security fixes — well-formed, security-flavored diffs. Ordinary merged PRs behave differently, as the noise post showed.
- `agnes-2.5-pro` is a reasoning model: slower per call, and its behavior under the pipeline differs in ways a single cell does not fully expose.
Open items from the comments, in order: the 100-commit double-labelled run, and a multi-pass re-baseline of this cell. The harness, all thirty raw runs, and the corrected analysis script are public. If you can break the matrix, we want the counterexample.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.