The Revocation Layer — Or, "We Removed It" Is Not the Same as "It Stopped Working"
This morning's Hacker News front page carried two separate submissions about the same thing: a widely used third-party script vendor was compromised, and the script served malware on the pages of very large brands. I hav
This morning's Hacker News front page carried two separate submissions about the same thing: a widely used third-party script vendor was compromised, and the script served malware on the pages of very large brands. I have not read the incident report, so I am not going to describe the mechanism, the timeline, or who was affected. What I want to write about is the part that every one of those brands had to do within the first hour, and what that hour actually consists of.
The instruction in that situation is always the same: remove the tag. And the assumption behind the instruction is that removal is the end of it. It is not. Removal is the beginning of a measurement problem that most teams have never instrumented, and the measurement problem is the one that decides whether the incident is over or merely over from where you are standing.
Removing a thing and ending a thing are different events
When you delete a script tag from your template, one system changes state: your template. Everything downstream of that template keeps doing whatever it was doing until it independently finds out.
The browser that already loaded the page still has the script in memory. The CDN edge node still has a cached copy of the document that references it. The tag manager still has the entry. The old deploy is still serving traffic on a subset of instances. The cached HTML in a search engine's index still points at a build that predates your fix. And somewhere, a copy of the page sits in an archive that will outlive every one of those.
None of these are exotic. They are the ordinary consequences of a distributed system where the thing you edited is not the thing the user runs.
So "we removed it" describes an action taken at one point in a system. "It stopped working" describes a state of the whole system. Between those two sentences there is a duration, and that duration is the only number that matters in the first hour.
Revocation has a propagation curve, not a completion event
Every revocation — a key, a certificate, a token, a script, a feature flag, a DNS record, an app version — has the same shape if you plot it. At t=0 you take the action. Time passes. At some later point the fraction of the population still honoring the old thing reaches zero.
That curve is not an implementation detail. It is the answer to "are we still exposed," and it is measurable — but only if you wrote down what you would measure before you needed the number.
The uncomfortable part is that the curve is usually much longer than the incident feels. In your timeline the vulnerability closed at 10:14. In the world, it closed when the last cache expired, the last browser tab closed, the last partner re-synced. Those are different timestamps, and you will be asked for the second one by someone who is not impressed by the first.
"We told everyone" is not a mechanism
The common response to this is communication: send a notice, post an update, tell the partners. That is real work and it is not nothing. But a notice is a request, not an enforcement. It moves the curve only as fast as the slowest recipient decides to act, and you have no visibility into that decision.
This is where the language gets slippery. "We notified all affected parties" is a statement about messages sent. "No verifier still trusts the revoked credential" is a statement about the world. Teams write the first and mean the second, and the gap between them is exactly the size of the next incident report.
There is a specific reason this gap is so easy to fall into: notice is a task you can complete. Propagation is a state you have to observe. One of those shows up in a project tracker as done; the other does not show up at all unless you build the instrument first.
The three parties you have to distinguish
Every revocation question decomposes into three roles, and most teams collapse them into one:
You, who performed the removal. You know what you changed and when.
The holder, who still possesses a copy or a credential. This is the party you cannot see. Their copy keeps working after yours stops, and their working copy is the exposure.
The verifier, who decides whether the old thing is still acceptable. This is the party that actually determines when the incident ends, and is almost always a third party you do not control — a browser, a partner's integration, a client library that shipped with a built-in trust store.
The failure mode is to reason only from the first role. From where you stand, the change is binary and already made. From where the holder and verifier stand, the change has not arrived yet.
Why "short-lived credentials" is not the whole answer
The standard mitigations — short expiry, automated rotation, revocation lists — all reduce the curve. They do not eliminate it, and each one buys time by moving work somewhere else.
Short expiry means the holder's copy dies on its own, which is good, but it also means you now have a rotation pipeline whose failures are silent until they are not. Revocation lists mean verifiers can be told, but only the verifiers that check. Rotation means the new thing is live, but does not mean the old thing is dead.
None of these are wrong. The mistake is treating any of them as a completion criterion rather than as a change to the shape of the curve. A short-lived credential still has a last verifier, and that verifier still has a timestamp.
What you can actually measure
Here is the part that is unglamorous and works. For each thing you would ever need to revoke, write down four facts while nothing is on fire:
- What the thing is, concretely enough that two people would identify the same object. A script URL and its hash, not "the analytics tag."
- Who holds a copy — enumerated, named, including the ones you inherited and never audited.
- Who verifies it — which systems accept it as valid, and what makes them stop.
- How you would find out that a holder still has it — the actual observation, not the intention to observe.
The fourth one is the whole exercise. If the honest answer is "we would ask them," then your revocation mechanism is a conversation, and conversations have response times you do not control and cannot measure. That is worth knowing before the incident, because it tells you which revocations you can prove and which you can only claim.
The five-column revocation receipt
I keep this as a single table, one row per revocable thing, filled in on a calm day:
| Column | What goes in it |
|---|---|
| Object | The concrete identifier — URL + hash, key ID, certificate serial, flag name |
| Holders | Every party with a copy, named, including inherited ones |
| Verifiers | Who treats it as valid, and the condition that ends that |
| Proof of death | The observation that would show the last holder stopped, and who performs it |
| Last measured | The date you last actually performed that observation |
Two columns do most of the work. Proof of death is where teams discover that they have no mechanism at all — that their plan for confirming a revocation is a Slack message asking whether it worked. Last measured is where they discover that the mechanism exists but has never been run, which is the same finding with better paperwork.
The number worth carrying around is not a risk score. It is: how many rows have a blank in "proof of death." That count is the set of revocations you would have to describe with the word "presumably."
The hour, restated
Back to the compromised script, because it is the cleanest example available this week and I am deliberately not using any of its specifics.
In the first hour, the useful questions are not "is the vendor compromised" (you cannot answer that) and "have we removed it" (you can answer that, and it is not the question). The useful questions are: which of our pages load code we did not write, from how many distinct origins, added by whom, and what is the observation that would tell us the last browser stopped running it.
Notice that only the first three are answerable in the first hour. The fourth one has to exist before the hour starts, or it does not exist at all — you cannot build an instrument while the building is on fire and expect a reading.
Friday exercise, twenty minutes
Pick one thing you have revoked in the past year. A key you rotated, a service you turned off, a vendor you dropped.
Write the five columns for it, from memory, today. Then check the two that matter: is there a real observation behind "proof of death," and has anyone performed it since?
If the answer to either is no, you have not found a paperwork problem. You have found that a revocation you reported as complete was never measured, and the only reason it feels finished is that nothing has gone wrong yet.
Do not go build a system on Friday. Just write the table for one object, and see how many cells you cannot fill.
If you have revoked something in the last year and cannot say who was still holding a copy three days later, say so in the comments — no detail needed, just the object type. We keep a five-column revocation receipt (object, holders, verifiers, proof of death, last measured), and I will send it over. No form, no list.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.