An advisory published that afternoon blocked our hotfix and every rollback we could build

Our storefront is a Node application, and for two years its pipeline has run npm audit --audit-level=high on every build. npm audit sends the dependency tree from package-lock.json to the registry, which checks it against the GitHub Advisory Database and answers with every known vulnerability in it. With --audit-level=high the command exits non zero if any of them is high or critical, and the build stops.

On a Friday in October a release went out with a bug in promotion codes. A code could be applied twice to a basket that also held a gift card. Finance noticed at half past four, and a two line fix was in a pull request by ten to five. Its build failed in the audit step.

The fix touched no dependency. That afternoon an advisory had been published for a small library four levels below our bundler, a regular expression that could be made to run for a very long time on crafted input. We only use it while building, never in the code customers run, and no patched version existed yet. The commit that had passed at noon would have failed at five.

An administrator could bypass our branch protection, and it took forty minutes to find one. The merge then started the deploy workflow, which builds again and audits again, and it failed in the same step. We looked at rolling back instead and found that our rollback rebuilds the previous release tag, so it would have failed too. The promotion bug stayed live for two hours and ten minutes, and about nine hundred orders took the discount twice. On Monday morning every open pull request in the repository was red.

The audit was right about the library. It was in the wrong place. A check that reads a database which changes every hour does not describe the commit. It describes the commit and the time it ran, and we had put it in the path that every urgent change has to take.

The audit now runs in two places. On pull requests it runs with --omit=dev, against production dependencies only, and only when package-lock.json has changed, so it judges the change in front of it. A nightly job audits main in full and opens an issue for each new finding, with an owner and a date. Images are built once, and deploys and rollbacks ship an image that already exists instead of building again, so neither of them asks anything that can change overnight. When we need a patched version of a transitive dependency before its parent releases one, we pin it with overrides in package.json, and the nightly job reports any override older than thirty days.

A high severity finding is still urgent for us. It now goes to the people who own it, not to whoever happens to be shipping the next fix.

The commit was the same at noon and at five. The answer changed because the question was being put to somebody else's database.

– Sergey Shinder