Autopsy #01: Anatomy of a Double Payment in Production
Incident Report On Friday at 14:10 UTC, a two-line PR replaced a native <form> submit with a custom onClick handler on the checkout button to support Apple Pay telemetry. QA verified the happy path on desktop Chrome ov
Incident Report
On Friday at 14:10 UTC, a two-line PR replaced a native <form> submit with a custom onClick handler on the checkout button to support Apple Pay telemetry. QA verified the happy path on desktop Chrome over office fiber and approved the release.
Within 40 minutes of the deploy, finance alerts flagged duplicate settlements on mobile Safari and throttled Android devices.
-
T+00:00: Deploy
[email protected]reaches 100% of edge PoPs. -
T+08:00: Support receives the first ticket reporting two \$99.00 charges for a single
cart_id. -
T+19:00: Payment gateway logs show 0.42% of
POST /v1/chargesrequests arriving in pairs spaced 80 ms to 3,400 ms apart, both returning200 OKwith distinctcharge_idvalues. -
T+42:00: Refund webhooks lag 48 hours behind support volume; checkout traffic is rolled back to
4.18.1.
The Red Herring
The on-call engineer's first Slack reflex was to add disabled={isSubmitting} on the <button> element and guard the handler with if (isSubmitting) return;. The patch passed every local Playwright test on localhost with zero duplicate requests.
In production, the patch failed to reduce the 0.42% duplicate rate for two reasons:
-
Async render lag:
setIsSubmitting(true)schedules a React render for the next frame;button.disabledstaysfalsewhen a secondclickenters the call stack 12 ms later. -
Premature timeout reset: A 3-second client timeout in
catchresetisSubmittingtofalseon slow 3G connections while the firstPOSTsocket remained open in flight.
Causal Chain
The duplicate charge requires five consecutive layer failures. A synchronous DOM event outruns asynchronous React state, uncoalesced callbacks fire two network requests, boolean flags allow re-entry after a timeout, uncancelled sockets finish in the background, and a stateless server commits both payloads.
Failure Simulator
Layer Dissection
01 → 02: Synchronous DOM Dispatch Outruns Async React Commit
In DOM Event Propagation (Capture, Target, Bubble), Blink dispatches click events synchronously from the native input queue into the JavaScript call stack. When onClick invokes setIsSubmitting(true), React enqueues a state update for the next render pass while button.disabled remains false during the current frame.
A second tap arriving 12 ms later hits the same active DOM node. Because Debounce, Throttle & rAF Coalescing is absent on the leading edge, both closures read isSubmitting === false and enter submitPayment().
02 → 03: Uncoalesced Handlers Feed Re-Entrant Boolean State
Even when a leading-edge lock blocks sub-second double-clicks, cellular handoffs introduce a 3.2-second TLS or gateway stall. Under Finite State Machines for UI, a SUBMITTING state cannot transition back to IDLE on a client-side timer while the HTTP socket remains open.
Boolean flags (isSubmitting, hasTimedOut) violate that invariant. A catch block executes setIsSubmitting(false) at t = 3,000 ms, re-enabling the button while Request #1 is still traversing the carrier network.
03 → 04: Re-Enabled UI Leaves Orphaned Sockets on the Wire
When the user taps "Pay" a second time, Request #2 enters the network stack alongside Request #1. Without AbortController and Cooperative Cancellation, the client never signals intent to drop Request #1, and an aborted fetch() only closes the local socket after the server may have parsed the TCP payload.
-
Client view: Marks Request #1 failed at
t = 3,000 msand dispatches Request #2. -
Server view: Receives Request #1 at
t = 3,150 msand Request #2 att = 3,420 ms.
04 → 05: Stateless Retries Commit Twice Without Idempotency Keys
Both HTTP requests reach the payment endpoint as independent mutations. As documented in Outbox Queues, Idempotency Keys & Causal Ordering, HTTP POST is non-idempotent by specification; only an explicit Idempotency-Key header scoped to the checkout intent lets the gateway deduplicate Request #2 and replay Request #1's receipt.
Architectural Fix & Invariants
type PayState = 'idle' | 'submitting' | 'settled' | 'failed';
export function createPaymentSubmitter(intentUuid: string) {
let state: PayState = 'idle';
const controller = new AbortController();
return async function submit(payload: unknown): Promise<Response | null> {
if (state === 'submitting' || state === 'settled') return null;
state = 'submitting';
try {
const res = await fetch('/v1/charges', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'Idempotency-Key': intentUuid },
body: JSON.stringify(payload),
signal: controller.signal,
});
state = res.ok ? 'settled' : 'failed';
return res;
} catch (err) { state = 'failed'; throw err; }
};
}
Architectural Invariants:
-
Synchronous transition guard: Mutate
idle → submittingsynchronously before anyawaitor React render commit to block same-frame double-clicks. -
Stable intent key: Generate
intentUuidonce per checkout form mount and reuse it across all retries so the gateway deduplicates re-submissions. -
Terminal settled state: Treat
settledas final for a givenintentUuid; starting another payment requires a fresh mount and UUID. -
Cooperative teardown: Call
controller.abort()on unmount so stale responses never mutate unmounted UI state.
Golden Rule: Never guard a financial mutation with React's asynchronous render state; lock synchronously in memory and deduplicate idempotently on the server.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.