Seventeen days after the letters: npm and PyPI malware feeds, measured again
In October we measured how often package-malware feeds are right on npm and PyPI: 242,072 records, what registries did with them, how many of the flags left up were wrong, and what four open scanners catch. Each feed got
In October we measured how often package-malware feeds are right on npm and PyPI: 242,072 records,
what registries did with them, how many of the flags left up were wrong, and what four open
scanners catch. Each feed got its numbers a week before publication. The method promised a second
measurement two to three weeks after those letters, on one date fixed in advance, so we could see
what changes with time and what changes after someone is told. That date was 13 November.
This post is about the second look: what moved, what did not, and two mistakes the repeat caught in
our own first round.
Our interest, as before. One of the feeds listed nine of our own PyPI packages as malware in
October; that is why we started. The full write-up (section 10 is the repeat) and the method are public, and the data are on Zenodo: 10.5281/zenodo.24415130.
One date, decided before looking
The repeat had to avoid the easiest bias: picking the day after seeing something interesting. On
10 November the date was written into the method and the letters file, before any feed data was
read: day 17 after the letters. Every part ran that day, in gVisor sandboxes as an unprivileged
user, from snapshots pinned by commit.

What ran on 13 November. Everything in the boxes was fixed in the method between 25 October and
10 November. Select the image for the full-size SVG.
The day did not go as planned in one way. Halfway through, our main study host's provider started
throttling its CPU (steal time 62β75%), and sandbox installs began to hit their 120-second limit
for reasons that had nothing to do with the packages. Those attempts are kept and marked void; the
rest of the run moved to a second host with byte-identical scanner images. The results were sealed
at 05:14 UTC the next morning, and the archive has the same SHA-256 on three servers.
The flags registries ignored stay ignored
We re-checked the same 7,483 sampled flags against npm and PyPI. Since 27 October, registries acted
on 34 more rows. None went back up. No feed-and-ecosystem stratum moved significantly once you
correct for testing 32 of them (the smallest Holm-adjusted p was 0.12, Amazon Inspector's npm
"research" records, +9 rows).
Vulnetix's records that registries had not acted on in October are still mostly up: npm 55 β 56 of
400 acted on, PyPI 28 β 30. Whatever a registry decides about a flag, it mostly decides in the first
weeks.
New flags behave differently. Of the npm packages OpenSSF flagged between the two snapshots, 55.8%
were already removed or replaced by 13 November; GitHub's advisories reached 58.2% on npm and 77.3%
on PyPI. Vulnetix's 28 new records in that window: none. Many of those flags were days old at the
check (486 of GitHub's 958 npm flags were under a week), so these shares are lower bounds.
The packages we read ourselves in October tell the same story from the other side. Of the ones we
read as malicious, npm acted on 3 of 45 in the meantime and PyPI on 0 of 12; of those we read as
benign, 1 of 57 on npm and 1 of 84 on PyPI were removed (we read both again; the labels stand).
Eleven of those twelve PyPI packages come from Amazon Inspector records. Two of the eleven turned
benign when we re-read a slice of our labels blind (below). We read every file of the other nine
again on 15 November, without running anything; all nine were still served, and we reported them to
PyPI's security team that day, six as clear cases and three for its judgement. We do not name them
here, because Amazon Inspector's PyPI records were rated as one frame, and naming the malicious ones
would name the benign ones by elimination. The counts above stop at 13 November. On 19 November all
nine versions were still served, none was yanked, and PyPI had not answered.
What the feeds changed in their own records
Withdrawals tell you whether a feed corrects itself. We dated every change against our letters:
| Feed | Withdrawn before our letters | On the day | After |
|---|---|---|---|
| OpenSSF (all tools) | 0 | 1 | 5 |
| Vulnetix npm | 2,001 | 0 | 8 |
| Vulnetix PyPI | 4,761 | 0 | 44 |
| GitHub advisories | no earlier copy | 1 | 4 |
Vulnetix's large withdrawal is dated before any letter went out: it came from its own review of the
first week of its database, and the records carry a modification date of 25 October (Vulnetix
points out that this is when the record changed, not a separate withdrawal timestamp). After the
letters it withdrew 52 more records and reclassified 6 between malware and vulnerability, with 8
more reclassified on the letter day. A change after a letter can be the letter's effect; a study
like this one cannot tell.
A placeholder that looks like a package
The first mistake the repeat found was in our registry check. npm has started answering a flagged
name with a staged-publishing stub, version 0.0.0-stage, a package.json and a README that says it
is a "Temporary package placeholder", while a new version waits for review. Our checker knew npm's
older holder, 0.0.1-security, but not the stub, so a name that served only the stub counted as
still live. For the new npm flags this moved 854 rows from "live" to "placeholder".
We found it by reading: 54 of 107 packages in a supplementary sample turned out to hold no code at
all. The fix and the rerun were done the same day, the as-run files are kept beside the corrected
ones, and the method's change log says what changed and why. If you check registries yourself, the
logic fits in a few lines:
PLACEHOLDER = re.compile(r"^(0\.0\.1-security|0\.0\.0-stage)(\.\d+)?$")
def status(name, versions, probe):
"""'live' if a non-placeholder version still has a tarball, else 'placeholder' or 'gone'."""
real = {v: m for v, m in versions.items() if not PLACEHOLDER.match(v)}
if not versions:
return "gone"
if not real:
return "placeholder"
for v, m in sorted(real.items(), reverse=True):
try:
if probe(m["dist"]["tarball"]):
return "live"
except urllib.error.HTTPError:
continue # npm can list old versions whose tarball is gone
return "placeholder" if len(real) < len(versions) else "gone"
That is an excerpt; the runnable script (npm_served.py, Python 3.9+, standard library only) adds
the registry fetch and an offline self-test:
python3 npm_served.py --self-test
python3 npm_served.py lodash left-pad
self-test: 5 cases pass
lodash live
left-pad live
Checked on 14 November 2026 with Python 3.9.6.
The second mistake: three "benign" packages that were npm's holder
Our October report rated a random sample of the flags registries had left up for a month. One row,
npm records that credit no tool, said 3 of 7 were benign. Re-reading the packages removed since,
we found those three were never packages: they had been bundled at npm's own holder, which npm has
served for those names since March 2016. Two models and an analyst rated a README that says
"Security holding package".
A rule from October should have taken them out of the frame, but their old versions still answered,
so the check of that day called them live. The row is now 4 of 4 malicious. It is corrected in all
five language versions of the report, dated.
Our own labels, read again blind
The method also asked for a blind re-read of a further 15% of October's labels, without seeing the
first verdict. Of 24 labels, 21 agree. The three that changed are all Amazon Inspector packages we
had called malicious and now read as benign. One was our miss on re-checking; two are judgement
calls: hidden code whose only job is to sign requests, and an MCP server whose remote-code tool runs
a URL its own caller supplies.
With the re-read labels, Amazon Inspector's PyPI records would be 34 of 44 benign (77.3%) instead of
32 (72.7%), and its npm records 8 of 18 (44.4%) instead of 7 (38.9%). The published October numbers
stand as measured; the report gives both.
When the file we read is newer than the flag
October's rating had left out tools with fewer than ten rated packages, so this time we drew a
supplementary sample from them, flags older than 14 days that were still served, rated and read the
same way:
| Stratum | Malicious | Spam | Benign | Benign share (95% CI) |
|---|---|---|---|---|
| GitHub advisories, npm | 6 | 3 | 39 | 81.3% (68.1β89.8%) |
| No tool credited, npm | 8 | 26 | 16 | 32.0% (20.8β45.8%) |
| kam193 (Bad packages), PyPI | 3 | 0 | 4 | 4 of 7, too few |
kam193's maintainer asked a fair question about those four: were we reading the code the records
were about? A record can cover every version of a name, and a name can come back. So for every rated
package we compared the upload time of the file we read with the first publication of its record.
Of the 39 GitHub npm packages we read as benign, 19 were uploaded more than 30 days after their
record. Without them, the row is 20 of 29 benign, 69.0% (50.8β82.7%), not 81.3%. Two of kam193's
four are versions served again with other code, more than 30 days after the record.

Benign share with and without the rows whose file was uploaded more than 30 days after its record.
Readings of 13 November, upload times compared on 14 November.
The labels stand: what someone checking a name against the feed is warned about today is the code we
read. For these rows, though, the false alarm comes from a name or version that was served again,
not from a package someone misjudged. The unattributed npm row has none of this, and Vulnetix none.
Scanners: same rules, a new week of uploads
We ran the same four scanners, same versions, on a second random week of new npm and PyPI uploads
(released 24β31 October and scanned on 13 November; the first week was released 10β17 October). False alarms
per 1,000 packages, first week β second:
| Scanner | npm | PyPI |
|---|---|---|
| GuardDog | 235 β 169 | 318 β 306 |
| malcontent (HIGH or CRITICAL) | 135 β 106 | 239 β 192 |
| malcontent, CRITICAL only | 15 β 5 | 25 β 10 |
| OSSGadget (any finding) | 828 β 803 | 990 β 980 |
Recall on the 390 known-malware samples reproduced the published figures to the package. One thing
we had not seen in October: 20 of the first week's 200 npm packages, and 24 of the second week's,
are npm's staged-publishing stubs. No scanner flags them, so they quietly lower npm false-alarm
rates; without them the rates are 11β15% higher.
We also added two scanners that were not in October's run. Atomdrift Scan, the successor to
malcontent recommended by its author, run offline with its LLM option off, found 48% of npm and 58%
of PyPI malware samples and raised almost no false alarms: 0 of 396 popular packages, 2 of 397 and 2
of 396 recent uploads. Its training data includes DataDog's dataset, which is where our malware
samples come from, so its recall deserves caution; the false alarms are the more telling number.
vulnetix malscan, which Vulnetix suggested we run, caught 31.5% of npm and 30.5% of PyPI samples,
with 50β79 false alarms per 1,000 on npm and 220β249 on PyPI, mostly on source archives. Vulnetix
cautions that a bare run on one package is not how the tool works in production; these numbers are
for that bare run.
Is DataDog's malware all malware?
Every scanner in this study is measured against DataDog's malware dataset. malcontent's author asked
in October whether that dataset holds up under our codebook. We took 100 of its samples at random,
had two models rate them blind, and read every one they did not both call malicious, plus a quarter
of the rest: 79 hostile, 6 "shells" whose only route to harm is a dependency or URL we cannot check
from the package, 13 benign, 2 inconclusive. None of the shells did anything in a two-minute sandbox
install with a sinkhole for every network call.

Recall three ways on the same 100 samples. Measured on 13 November 2026; timeouts count as misses.
Every scanner looks better once the benign samples are taken out of the denominator: GuardDog goes
from 77% to 90%, OSSGadget from 85% to 91%. A recall number is only as good as the labels under it.
Indicators that match ordinary code
Vulnetix also publishes a free STIX feed of domains and URLs. 71 of its domain indicators are in the
top 10,000 of the Tranco list on the run date. Read one by one, 57 are ordinary services (YouTube,
AWS Cognito, ChatGPT, Coinbase and OWASP among them) and 14 are legitimate services that can be
abused, such as file hosting or a chat webhook API.

Matching the feed against package text. The benign sets hit as often as the malware set.
Matched against package text, the feed finds indicators in benign packages as often as in malware:
41% of popular PyPI packages against 27% of PyPI malware. Used that way, it cannot tell them apart.
Two hundred packages nobody flagged
To bound how many "false alarms" in the recent-upload sets are not false, we read 200 of those
packages at random, flagged or not. None was malicious and none was spam. With 50 per set and
ecosystem, that puts the malicious share of each stratum at no more than about 7%.
That reading also has limits worth saying plainly: some of these packages are 35β100 MB. Each was
read through a structured digest (manifest, every install-time hook and the files it runs, every
strong codebook signal in every file, obfuscation markers), and every flagged file was opened in
full. One package hid its request-signing code in an encoded pickle; we disassembled it without
running it.
What did not work
The fresh-malware set, meant to measure scanners on malware newer than their versions, is empty:
DataDog's dataset had no sample dated after 24 October on the run date. That question stays open.
We also asked npm's and PyPI's security teams how feed reports reach them. npm's security address
refused the letter (it takes no outside mail), and PyPI had not answered by 19 November.
A second scorer re-scored the transparency section's 300 records: agreement was strong on evidence,
reason and method (ΞΊ 0.89β0.95) and weaker on linked evidence (0.62), almost all of it one question,
whether a GitHub advisory link counts as evidence. Two scores changed. Both readings are in the
report.
What the feeds and scanners said
Every feed and scanner in these numbers got them, with the method, on 14 November, a week before
this post. By 19 November two had answered, kam193 and Atomdrift; the others had not.
Kamil MaΕkowski, who runs kam193's Bad packages feed, went through the feed's four records we read as
benign. Two list versions uploaded after the malicious ones, one through a bug in the record and one
by listing too many versions; both will be corrected. In the third, the malicious code was in
earlier versions of the same package and was later removed, so the record stays. The fourth holds
no malicious code itself but came from the campaign's uploader and was never put to use before the
campaign was taken down. The feed's three still-served packages that we read as malicious had, by
this answer, been reported to PyPI a few days earlier. Our labels are those of 13 November; the feed's corrections do
not change them, but they explain two of the four.
Thomas StrΓΆmberg, malcontent's author and behind Atomdrift Scan, found the results correct. Two
points narrow our caveats: DataDog's samples are a small part of the roughly 300 million files,
benign and hostile, that Atomdrift Scan was trained against; and malcontent and GuardDog were trained
on DataDog's set too, probably OSSGadget as well, so the recall caution applies to every scanner
here, not only to Atomdrift. Its rules were tuned with the local LLM mode on, which our run had off;
Atomdrift Scan 3.0 is meant to work either way. Its low false-alarm rate is a setting: the default
aims at 25 false positives per 100 million files, and -L0 to -L250 move it. Our numbers stay
those of v2.13.0 at its defaults, as the method fixed them.
What happens next
Answers that arrive after this post, and anything PyPI does about the nine packages, go into the
report's section 10 with their dates. The sealed run, the
readings and the scripts are in the study's repository, and the data archive, version 6, is on Zenodo
(10.5281/zenodo.24415130) with two scripts that recompute every number from its files.
This study and post were prepared with AI assistance (Claude, an Anthropic model): the runs,
readings and drafting were done with it, under the owner's direction.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.