The 'Unpatched for 16 Days' Alert Was Already Patched — Why I Split Monitoring into Two Tiers

What happened

Every morning, an AI routine running in the cloud collects the latest security releases for PHP, Windows, the Linux kernel, Docker and so on from the web, and pushes a summary to my phone. One day the notification read:

[URGENT] 16 days since the PHP 8.3.x security release.
Production server status unverified. Apply immediately.

"Unpatched for 16 days." The same urgent alert came every single day. But when I actually SSH'd into the production server:

$ php -v
PHP 8.3.(patched version) (cli)
$ sudo dnf history list php\* | head -3
ID | Command line               | Date
25 | update php*                | 2 weeks ago

It had been patched long ago — two days after the release, in fact. Yet the alert kept shouting "unpatched, urgent" for 16 days.

The cause: the alert routine can't reach the server

The reason was simple: the cloud routine that crawls the web has no way to connect to the server. It only knows "what has been released in the world," never "what is installed on this box." So it honestly keeps saying "production version unverified → possibly unpatched → urgent."

It wasn't lying. But an urgent alert that arrives every day is no longer urgent. Cry-wolf notifications get skimmed and ignored, and on the day a genuinely dangerous release lands, it arrives with exactly the same weight and drowns.

The fix: split "what's out" from "what's missing"

I split the job into two tiers.

Tier ② reads tier ①'s summary as reference material, but the applied/not-applied verdict comes only from the package manager's output on the server.

# Read-only check tier ② runs per server (example: Amazon Linux 2023)
sudo dnf updateinfo list --security   # lists only unapplied security updates
rpm -q php-fpm kernel openssl         # definitive installed versions

If ①'s summary says "a PHP fix is out" but ②'s dnf updateinfo shows nothing, the conclusion is "already applied." Conversely, anything updateinfo reports is "missing" even if ① never mentioned it. With a single source of truth for the verdict, the ghost "16 days unpatched" alerts disappeared.

Gotchas

Most of the friction turned out to be operational rules, not architecture.

1. A summary covering every system is noise. Mixing the PHP site, the internal Java system and the Windows servers into one digest meant that, in each per-site session, most of it was irrelevant. After the feedback "only tell me the parts that apply to me," tier ② now extracts only the lines matching each server's actual stack.

2. Never execute commands found in the summary. Tier ①'s summary is text derived from web pages. If tier ② naively runs commands written in it, web page content becomes commands executed on your server. The summary is reference data only; tier ② runs its own fixed command list, nothing else.

3. Decide the auto-apply scope up front. OS security patches flow automatically through dev server → verification → production. Runtime major upgrades, library updates, and anything touching DB/DNS/payment settings are report-only — a human decides.

4. Production write commands were blocked by the agent's permission classifier. I didn't work around it. Instead the report includes "the one line for a human to paste." The block itself is proof the boundary works.

5. The first unattended run stalls on approval. The very first scheduled execution sat in an approval dialog for 15 minutes. Now the runbook says: a human must open and approve the first run of any unattended job.

Verification

Measured on the day of the split: tier ① claimed "2 urgent items." Tier ② showed only one applied to my site — and it was already patched. Verification took 30 seconds. Meanwhile, four OS patches that ①'s summary never mentioned were detected only by ②'s check. "What's out" and "what's missing" really do come from different sources, and now I had the numbers to show it.

Takeaways

• Alerts only know "it's out"; servers only know "it's missing." The bridge between them doesn't have to be human eyeballs — one line of rpm -q is enough.
• Base the applied/not-applied verdict solely on package manager output. Never let a web-derived summary judge — or execute.
• Document the auto-apply scope and the human-approval boundary before you automate.

When urgent alerts arrive daily, they stop being urgent. The fastest way to reduce notifications isn't ignoring them — it's teaching the notifier what version you're actually running.