The Replay Layer — Or, "We Caught Up" Is Not the Same as "It Only Happened Once"

This morning the FAA issued a national ground stop, and by afternoon the story had moved from the sky to the software: airlines rebooking in bulk, a rail booking system struggling under the load of people trying to change plans, status pages that could say a category was degraded but not whether your specific trip was. Every one of those systems has a queue, and every one of those queues will replay.

That is the part of an outage nobody puts in the postmortem. We measure how long the system was down and how long it took to come back. We rarely measure what happened on the way back — specifically, how many things happened twice.

Recovery is a replay, not a resume

The word "resume" is misleading. It suggests a system that paused and then continued from the exact point it stopped, like a video. Almost nothing works that way. Real systems buffer: messages queue, webhooks retry, cron jobs miss their window and fire late, clients resend requests that timed out, operators run the backfill script they wrote at 3am.

So recovery is not resumption. It is a replay of a compressed, out-of-order, partially duplicated stream of events — and the system on the other side was designed assuming each event arrives once.

The sentence that gets said at the end of this process is "we're caught up." It is almost always true and almost always about the wrong thing. Queue lag returning to zero is a statement about depth. It says nothing about the effects that were applied while the queue drained.

A queue metric cannot see a double charge

Consider a payment system during a two-hour outage. Requests back up. On recovery, the queue drains in eleven minutes. Lag graph returns to baseline. Everyone goes home.

Now consider what "drained" means. If the consumer crashed after charging the card but before acknowledging the message, the message comes back. The card is charged twice. The customer sees two identical charges, disputes one, and the finance team spends a week reconciling. None of that appears in the lag graph, because the queue did its job: it delivered. It delivered the same event twice, which is exactly what at-least-once delivery promises.

This asymmetry is the whole problem. Dropping work is visible: a customer does not get the email, a job does not run, someone complains. Duplicating work is silent: two charges look like two purchases, two emails look like a campaign, two provisioned accounts look like a customer with two seats. Systems are tuned to avoid the visible failure, which means they are tuned toward the invisible one.

The timeout is the bug

Most duplicates are born at a single moment: a request that succeeded, whose response was lost.

The client sends a charge request. The server charges the card. The response times out on the way back. The client now has a choice: treat the timeout as failure and retry, or treat it as unknown and check. Code written under deadline pressure treats timeout as failure, because that is the branch that keeps the queue moving. So the client retries, and the server charges again.

The honest reading of a timeout is not "it failed." It is "I do not know." And the only safe default for "I do not know" is: assume it succeeded, then verify — query the state, look for the effect, reconcile by intent rather than by response.

Almost every duplicate I have seen in production traces back to someone collapsing "unknown" into "failed" because the failure branch was easier to write.

Where the second delivery hides

The usual suspects, in roughly the order they cause damage:

• Payment and billing calls. Retried on timeout, replayed from a dead-letter queue, or re-run by a backfill. Money is the effect that is hardest to reverse quietly.
• Notification sends. An outage produces a burst of queued emails and pushes. Replay turns "we told them once" into "we told them four times," which trains recipients to ignore the channel — the alert-fatigue problem, arriving from the other direction.
• Provisioning and entitlements. Two seats, two API keys, two subscriptions. Silent until the invoice or the access review.
• Webhook handlers. Receive the same event three times because the sender retries; the handler was written as if it would only ever see it once.
• Backfill scripts. Written during the incident, run twice because the first run's output was not recorded. The second run is not idempotent because nobody thought of it as a replay.

None of these are exotic. They are the ordinary write paths of ordinary systems, and the only reason they do not duplicate more often is that outages are rare enough to keep the sample small.

"Idempotent" is a claim with a scope

Most teams will say their handlers are idempotent. Ask three follow-up questions and the claim usually shrinks.

First: what is the key derived from? If it is derived from the request — a random UUID generated per attempt — then every retry is a new request and nothing is deduplicated. The key has to be derived from the intent: this customer, this amount, this invoice, this window.

Second: where does the key live, and for how long? An in-memory set survives until the process restarts, which is precisely when replays happen. A dedupe window shorter than the retry window is decoration.

Third: is the key written in the same transaction as the effect? If the guard is recorded before the charge and the charge fails, you have lost a legitimate event. If it is recorded after and the process dies in between, you have a duplicate. The guard and the effect have to commit together, or the guard is a guess.

A useful one-line test: for each write path, write down the key, its storage, its lifetime, and whether it commits atomically with the effect. Paths where you cannot fill all four are not idempotent — they are idempotent-looking.

Why this is an evidence problem, not just an engineering one

Replay damage does not stay in engineering. It lands directly on the records that compliance and customer trust depend on.

If a customer was charged twice, the correction is not a code fix, it is a restitution: who was charged, how much, when it was returned, and how you know. If a disclosure email went out four times, the disclosure receipt — which claims each recipient was told once, at a time — is now false in a way that is hard to explain to a regulator. If a deletion request was replayed and the second run deleted records that the first run's receipt says were retained for a legal hold, the receipt proves the opposite of what happened.

Every ledger on our own shelf — disclosure, correction, retraction, restitution, deletion — assumes one event produced one row. Replay is the failure mode that breaks that assumption at the root, and it does so silently, because the second row is indistinguishable from a legitimate second event.

The artifact: a replay receipt

The useful artifact is small, and it is per write path, not per incident:

Four columns. Most teams can fill the first and stall on the second and fourth, because the second requires knowing what the handler actually does on re-entry and the fourth requires having looked.

The number that comes out of this is not a percentage. It is a count: how many of your write paths have a guard whose lifetime is longer than your longest retry window, and whose write commits atomically with the effect. That count is the honest measure of whether "we caught up" means anything.

The Friday exercise

Take the last incident where you drained a backlog — a queue, a webhook retry storm, a backfill. Pick the three write paths that touch money or outbound messages. For each one, answer one question in writing:

What did the second delivery do?

If the answer is "nothing, it is idempotent," write down the key, where it is stored, how long it lives, and whether it commits with the effect. If you cannot fill those four, the honest answer is "we do not know," and that is the finding — not because something broke, but because you cannot currently tell whether it did.

If you have drained a queue after an outage and you are not sure whether something happened twice, say so in the comments. We keep a four-column replay worksheet — what can be delivered twice, what the second delivery does, where the guard lives, what you observed — and I will send it over. No form, no email list.