We re-ran our MCP scanner comparison on 14,933 npm and PyPI tool lists, in gVisor, with two more scanners

A week ago we ran three MCP security scanners on a tool-poisoning benchmark and 986 real servers. Three things in that post were weak. The real servers were all remote HTTP servers, while most local MCP tools are npm and PyPI packages. We compared only two other scanners. And it all ran on a workstation.

So we ran it again: five scanners instead of three, 14,933 package tool lists next to the 986 servers, and every scanner, ours included, in a gVisor container with no network. Here is what moved, and what didn't.

What didn't move

Our scanner, WARDEN, went from ruleset v10 to v12 in the week between. v11 decodes escapes, HTML entities, percent encoding and base64 before the rules run; v12 stops excusing a quoted attack phrase when the sentence asks the model to carry it out. On every set we have, v12 gives exactly the verdicts v10 gave, server for server: 171 of 218 held-out MCPTox tools blocked, 3 of 986 real servers.

The other two scanners didn't move either. mcp-audit 0.18.2 and mcp-shield 1.0.4, rebuilt from their registries into a fresh image, reproduce last week's numbers exactly: 25 and 41 of the 218. That matters more than it sounds: it means the harness is not what produces the numbers.

We also had to check one thing about ourselves. The 171 only counts if no rule written since v10 saw the held-out half of MCPTox. We compared every line added to our rules, tests and classifier since then with the held-out tools, eight words at a time: no match. The same check on the v10 change finds 77 matches from the half v10 was allowed to read, so it does catch copying when there is some.

Two more scanners, and what the others really are

Glama lists seven tools next to WARDEN. We read each one before deciding whether it belonged in a comparison of tool-definition scanners.

Three of the seven send your server to someone else's machine for the verdict. That is a fine product, but it is not something you can run on a replayed server with the network off, so it is not in a table next to tools you can. Snyk Agent Scan stays out for the same reason, and last week its free tier refused every request.

One more note on mcp-shield: it has an optional model mode, --claude-api-key, and we did not run it. The model is hard-coded as claude-3-7-sonnet-20250219, which Anthropic retired in February 2026, so every call now fails silently and the scan carries on. And even with a working model, the mode only adds an "AI Risk Level" line to a tool its patterns already flagged: it changes neither a verdict nor the exit code. The numbers below are mcp-shield's numbers.

On the benchmark, the two newcomers sit with the other keyword scanners:

The marker row is the same lesson as last week. truecopy goes from 27 to 141 and mcpsentry from 8 to 136 the moment the attack says <IMPORTANT>. mcp-shield is the one scanner the marker doesn't help: 41 either way. A test set full of markers measures the markers.

14,933 real tool lists

Our transparency log, HISTOR, installs npm and PyPI MCP servers in a gVisor sandbox and records what each one answers to tools/list. We took every package tool list it had already observed, 14,933 of them, through its public API, and replayed them to all five scanners.

These are unlabelled. Nobody knows which are hostile, so they measure the second number: how many honest servers a scanner blocks.

• What the mistakes look like. mcp-audit calls full-width Chinese punctuation, a "，" or a "（", or a single Greek or Cyrillic letter a homoglyph: 51 of its 58 sample blocks. mcp-shield blocks on a word: "token", "API key", "credentials", or "..", which it reads as path traversal. mcpsentry's base64 pattern matches any 40 letters and slashes, which JSON Schema paths and lists like PNG/JPG/HEIC/TIFF produce. truecopy mostly trips on .env in environment tooling.
• WARDEN's 23 in the sample. None deserves a block outright. Four are arguable: three tools that take a raw private key as an argument, so the key would pass through the model, and one that uploads any local file to a public image host. The rest are our rules misreading ordinary documentation, such as "you MUST first authenticate using aws_sso_login" between a server's own tools.
• Not every scanner can read every server. mcp-shield's MCP client rejects 26 of the tool lists outright, because an inputSchema is not of type object, and then checks nothing on them.

Two packages worth reading

Most blocks on real servers are mistakes. Two tool lists, though, deserve a human look, and our rules block neither:

• A behaviour hijack. One PyPI package wraps its description in a fake <critical_requirement … directly_user_requested="true"> block, says "What was prior defined as first is now second", and orders the model to call its tool before any reply to any input. It takes no data. It claims an authority the user never gave and reorders the model's instructions. WARDEN reports it as advisory. mcpsentry blocks it, on "you must immediately"; mcp-shield blocks it too, on a keyword, not on the hijack.
• A hidden data flow. Another sends a file's content, a directory listing and the working directory from the agent to the server, each with "Do NOT display … to the user". For a local document index that may be tidiness. It is also exactly what you would write to move files quietly. No scanner blocks it, and WARDEN reports nothing.

Both were flagged at high by WARDEN's optional classifier, which brings us to the part rules can't do.

The classifier, again

scan can send each tool definition to a model you choose and ask whether its text directs the model. It is off by default and advisory unless you pass --classifier-blocks. Its prompt changed since last week: it now asks for a concrete violation of authority instead of flagging any imperative, tuned on a synthetic stress set, not on MCPTox.

Last week's prompt flagged 147 of the 218 at high; this one flags all of them and none of the clean servers. On real servers most of its high flags are workflow orders like "call this tool first". It still misses what the rules catch: an injection in a tool's name, a private key asked for as a parameter, rm -rf ~. Every classifier run in this comparison together cost about $1.70.

How it ran, and one thing we broke

Last week the scanners ran on a workstation. This time:

• Fixed inputs. A manifest with the SHA-256 of every one of the 16,830 input files, written before the runs.
• Fixed tools. One image for the four other scanners, one for WARDEN, both on the same base. Every installed file matches the package published on npm or PyPI, except the install record pip writes itself. WARDEN was installed from an npm pack of the measured commit, byte for byte the 0.16.0 on npm.
• Isolation. gVisor, no network, a read-only root, 2 GB of memory, one bind mount.
• Two hosts, one image. The packages were split by a hash of each package's id across two of our servers that run the same image ID. 100 packages ran on both: the five scanners gave identical verdicts and finding codes on both, 100 of 100.

The servers we used are not idle. On one of them a web service hung for about forty minutes while the run was in its heaviest phase, starting eight Node processes at a time under gVisor, and again later. We paused, restarted the service, capped the run at 2 CPUs with the lowest scheduling weight, and reran the 1,193 scans that had timed out instead of answering: 383 on that server, and 810 from the other one, where mcp-audit's fixed 10-second connect window was too short next to our transparency log's own sandbox (800 mcp-audit, 10 mcp-shield). The timeouts were never counted as verdicts.

Our interest, and the limits

• We are not neutral. We publish WARDEN under MIT and wrote the attacks and every judgment. That is why the sets, the harness, the raw verdicts, the manifest and all judgments are public.
• One benchmark is one benchmark. MCPTox comes from three templates. A held-out half by server is not a held-out attack style.
• The packages are a snapshot. HISTOR's crawl was running while we read it.
• One version of each scanner. mcp-audit has since reached 0.21.0; we kept 0.18.2 to compare with last week. Each block line is our reading of that scanner's own severity scale.

Try it

npx -y @aimarket/[email protected] scan

scan reads the MCP configs of Claude Code, Claude Desktop, Cursor, VS Code and Windsurf, connects to every server they start, and vets the tool definitions before a model sees them. It has no dependencies, needs no account, and sends no tool text anywhere unless you turn on the classifier.

Links:

• Full study, in English, Русский, Español, Français and 中文.
• Harness, Dockerfiles, manifest and results: scripts/scanner-comparison.
• Every judgment, with a one-line reason: judgments-2026-10-16.json.

If you maintain one of these scanners and think we read your severity scale wrong, open an issue on alexar76/warden. We would rather fix the comparison than defend it.