The Most Precise Plugin in My Benchmark Scored 10.8%

Six ESLint security plugins, one labelled corpus of 69 vulnerable fixtures,
same harness, same day — [email protected],
@microsoft/[email protected], ESLint 10.8.1, Node 24.

I wrote the fixtures. That sentence has to come before the table, because
it changes what every row below is worth.

TP, FP and FN are
the four boxes everything below is derived from.

Three rows have a piece of their own, linked above — one tool read closely
rather than ranked. This is seventeen plugins
compressed into one table, and compression is the subject.

Read the column the ranking hides

F1 blends precision and recall, which is how a table like this stops being
informative. Split them and a different tool leads.

eslint-plugin-no-unsanitized sits second from the bottom at 10.8% F1. Its
precision is 80% — 4 true positives against 1 false positive, the highest
of any community plugin here. It detects four things. It is right about
substantially all of them.

That is not a consolation prize. A tool with 80% precision and a narrow scope
is one you leave switched on. A tool that finds more and is wrong more often is
one somebody disables in month three — and a rule nobody runs has a recall of
zero regardless of what the table says.

Why my row says 100%, and why you should discount it

A 100% row against fixtures written by the people who wrote the rules is the
expected outcome of grading your own homework. It is not evidence of
superiority and I will not present it as any.

What it is: a regression gate we happen to publish. It says our rules still
detect what we built them to detect. On the day it stops saying 100%, something
broke.

Three limits follow:

• The fixtures encode our threat model. A plugin scoped to a different surface — browser DOM, Angular, Electron — scores low here for reasons that say nothing about its quality on that surface.
• F1 on a self-authored corpus measures agreement with our idea of the problem, not correctness in the world.
• Recall against 69 of my fixtures is not recall against the field.

@microsoft/eslint-plugin-sdl is the clean example: 15.6% here, and it is a
well-made tool aimed at frontend and Microsoft-stack hardening. Pointing a
Node-backend corpus at it measures the mismatch, not the plugin.

The check I ran before publishing any of it

I did not take the F1 column on trust — including my own row.

Every value was recomputed from its own TP/FP/FN with 2·P·R/(P+R). All six
matched the published figure to within a tenth of a point. Then the check that
actually matters: TP + FN = 69 on every row. If those totals disagreed, the
rows would be describing six differently-scoped runs and the table would not be
a comparison at all.

A minute's work, and the difference between a table you can defend and one you
copied. A number without its measurement is a
memory,
and a rank throws away the distribution behind
it — a leaderboard is
the least recoverable form a measurement can take.

What the table is for

Not for picking a plugin. For two narrower things.

A scope map. The recall column tells you what each tool was built to see.
Four of 69 is not failure; it is a statement of scope, useful when you are
deciding what to run alongside what.

A regression gate. Which is what it is to me, and the only claim here I
would defend without qualification.

If you want the version of this you can trust without knowing me, run it
yourself. The corpus and the harness are in
the plugins repo under
benchmarks/, and the suite that produced this table is one command:

npx tsx benchmarks/suites/ilb-juliet/run.ts

Pin the versions above or your numbers will not match mine — eslint-plugin-security's
recall moves between releases, which is exactly why an unpinned comparison
table is worth nothing. Then add fixtures for the threats you care about. The
result will disagree with mine, and yours will be the one worth having.

Whose fixtures is your security tooling being graded on — and have you ever read them?