Fixing one bug in my security tool would have hidden every alert in my test project

I maintain depquiet. It sits on top of pip-audit, uv audit and npm audit and tries to cut down the noise. The one rule I care about is that it should never hide something real.

This week I found out how close I came to breaking it with what looked like a simple bug fix.

The setup

depquiet had an EPSS filter. EPSS is a score from FIRST.org that estimates how likely a CVE is to get exploited in the next 30 days. My defaults were:

• under 10%: filter it out
• under 30%: move it to "review"

I picked those numbers because they felt sensible. Most CVEs never get exploited, so why bug people about them?

Then I noticed the filter basically never ran on Python projects. EPSS only works with CVE ids, and pip-audit gives you ids like PYSEC-2023-74. uv audit gives you GHSA-.... The CVE is there, but as an alias, and depquiet wasn't looking at aliases. No CVE, no score, no filtering.

Easy fix, right? Look up the alias.

But that would turn the filter on for every Python user at once, so I stopped and thought about what it would actually throw away. I didn't like the answer, so I shipped the alias fix and also stopped EPSS from filtering by default. More on why below.

Then I ran it on real data

After the release I made a tiny project with three old pins:

requests==2.25.0
urllib3==1.26.4
jinja2==2.11.2

pip-audit found 22 vulnerabilities. With the alias fix, all 22 got an EPSS score. The highest was 4%. The lowest was 0.2%.

So if I had only fixed the alias and kept my 10% default, the filter would have thrown away all 22. Every single one. That list includes requests leaking your .netrc credentials and urllib3 forwarding Proxy-Authorization headers. The 30% rule would have pushed all of them down to "review" too.

I knew it would filter a lot. I didn't expect it to filter everything. The tool would have told me the project was fine.

What I got wrong

I was treating EPSS like it says something about my app. It doesn't. It tells you how likely a CVE is to be exploited somewhere on the internet. A 0.5% score means nobody is spraying that bug at scale right now. It doesn't tell you if your code passes credentials through a proxy you don't control.

Using that number to hide alerts means a global statistic gets to decide what you see about your own project. That's not what I want this tool to do.

What I changed

In v0.42:

• EPSS doesn't filter anything by default anymore. It just sorts. Known exploited (CISA KEV) go first, then highest EPSS, then severity.
• If you still want the filter, --epss-threshold 0.05 works. You just have to turn it on yourself.
• Lookups use the CVE alias now, so Python and npm findings actually get scores.
• Tiny scores show as 0.4% instead of 0%. Seeing "0%" next to a credential leak felt wrong. Going through the whole release afterwards also turned up two other things I'm not happy about. depquiet init was still writing the old 10% threshold into new config files. And if you passed a --kev-file that didn't exist, depquiet crashed with exit code 1, which is the same code it uses for "found vulnerabilities". So a CI job that tolerates findings would just pass, without ever checking KEV. Both are fixed in 0.42.1.

If you build tools like this

Run your filter on real data and look at what it throws away, not just what it keeps. My numbers looked fine on paper. On an actual project they would have hidden everything.

pip install depquiet, or use the GitHub Action samuelrojas-dev/depquiet@v0. If you think I'm wrong about EPSS, I'd like to hear why: github.com/samuelrojas-dev/depquiet