A Poisoned npm Package Infected Our Production Server, Here's How We Found and Removed It

How a routine deploy infected our CI server, how we hunted it down, and what actually fixed it.

The setup
Our on-prem Ubuntu server runs ~20 GitHub Actions self-hosted runners and a dozen Next.js apps under pm2. One shared box, many repos — remember that detail, it matters later.

Day 0: "Is this suspicious?"
While running a routine security audit, one check jumped out — two strange processes:

node -e global.i='7-v1729';global.e='NPM';global.r=require;...
Heavily obfuscated JavaScript, running as our deploy user, with established HTTPS connections to an unknown IP: 181.214.149.148:443.

Deobfuscated, the loader did one thing: fetch JSON from a remote server and eval() whatever came back. A classic remote-code stager — the attacker could execute anything, anytime, on our production box.

The forensics trail
What we ruled out, step by step:

• ❌ SSH brute-force — auth logs clean, all logins from our LAN
• ❌ Rogue SSH keys, extra root accounts, cron/systemd persistence — clean
• ❌ Malicious install scripts — we reinstalled with ignore-scripts=true… and it came back anyway
• ❌ App source code & workflow file — clean
• ❌ NODE_OPTIONS/env injection, npmrc tampering, pm2 modules — clean

What we confirmed:

• ✅ The loaders spawned during npm install + build of one specific app, delivered through a CI deploy
• ✅ The parent chain: pm2 → npm start → next-server → malware. Every app boot respawned it
• ✅ global.e='NPM' — the malware itself tags its delivery vector: a poisoned npm package

The scariest moment: after a full wipe (rm -rf node_modules .next + fresh install), the malware respawned within a minute — spawned mid-install, then orphaned itself to PPID 1 so it survived pm2 stop.

The resolution
We never got the exact package name — and that's a realistic detail worth sharing. Here's what the evidence showed: a dependency version that was live-poisoned on the npm registry on deploy day, cached locally, and pulled/patched upstream within 48 hours (this is how real supply-chain waves work — npm yanks compromised versions fast). Once we cleaned the npm cache and reinstalled fresh, the exact same sequence came back clean — verified end-to-end with kernel-level auditd logging armed.

What actually fixed it

1. Contain: iptables DROP on the C2 IP (made persistent!), stop the infected app, kill the orphaned loaders
2. Freeze the vector: stop ALL runners — deploy freeze until clean
3. Wipe properly: rm -rf node_modules .next plus npm cache clean --force — the poisoned tarballs live in ~/.npm/_cacache and reinfect you from cache
4. Rotate everything: GitHub PATs, runner tokens, org secrets, every .env, SSH passwords. The malware had 24h of access — cleanup does not un-steal secrets
5. Verify with a real deploy, then bring runners back one at a time
6. Schedule a full host rebuild — after a remote-code stager, "no evidence of a second stage" ≠ "no second stage"

Lessons learned

• ignore-scripts=true is not enough. It blocks install hooks but NOT require-time payloads that run when your app boots.
• Self-hosted runners are a blast radius multiplier. One poisoned dependency in ONE repo = full compromise of every project on the box. Isolate runners.
• npm's cache can reinfect you. Wiping node_modules without npm cache clean is not a clean install.
• Orphaned processes survive pm2 restarts. Check for PPID 1 processes you don't recognize.
• Block C2 IPs at the firewall AND persist the rule — an in-memory iptables rule dies on reboot.
• Rotate credentials immediately. This is the step everyone delays and the one that matters most.
• A ufw status that says inactive on a production server is a finding in itself. 🙃

This was my first incident of this kind — if you run self-hosted runners on a shared box, I hope you never need this post. But if you do: contain, freeze, wipe (cache too!), rotate, verify, rebuild.