Dev.to WebDev 🛠 Dev 👁 0 📖 3 min read

Autopsy #01: Anatomy of a Double Payment in Production

Incident Report On Friday at 14:10 UTC, a two-line PR replaced a native <form> submit with a custom onClick handler on the checkout button to support Apple Pay telemetry. QA verified the happy path on desktop Chrome ov

Autopsy #01: Anatomy of a Double Payment in Production

Incident Report

On Friday at 14:10 UTC, a two-line PR replaced a native <form> submit with a custom onClick handler on the checkout button to support Apple Pay telemetry. QA verified the happy path on desktop Chrome over office fiber and approved the release.

Within 40 minutes of the deploy, finance alerts flagged duplicate settlements on mobile Safari and throttled Android devices.

  • T+00:00: Deploy [email protected] reaches 100% of edge PoPs.
  • T+08:00: Support receives the first ticket reporting two \$99.00 charges for a single cart_id.
  • T+19:00: Payment gateway logs show 0.42% of POST /v1/charges requests arriving in pairs spaced 80 ms to 3,400 ms apart, both returning 200 OK with distinct charge_id values.
  • T+42:00: Refund webhooks lag 48 hours behind support volume; checkout traffic is rolled back to 4.18.1.

The Red Herring

The on-call engineer's first Slack reflex was to add disabled={isSubmitting} on the <button> element and guard the handler with if (isSubmitting) return;. The patch passed every local Playwright test on localhost with zero duplicate requests.

In production, the patch failed to reduce the 0.42% duplicate rate for two reasons:

  • Async render lag: setIsSubmitting(true) schedules a React render for the next frame; button.disabled stays false when a second click enters the call stack 12 ms later.
  • Premature timeout reset: A 3-second client timeout in catch reset isSubmitting to false on slow 3G connections while the first POST socket remained open in flight.

Causal Chain

The duplicate charge requires five consecutive layer failures. A synchronous DOM event outruns asynchronous React state, uncoalesced callbacks fire two network requests, boolean flags allow re-entry after a timeout, uncancelled sockets finish in the background, and a stateless server commits both payloads.

Failure Simulator

Interactive Failure Simulator — Click to step through the live model

Layer Dissection

01 → 02: Synchronous DOM Dispatch Outruns Async React Commit

In DOM Event Propagation (Capture, Target, Bubble), Blink dispatches click events synchronously from the native input queue into the JavaScript call stack. When onClick invokes setIsSubmitting(true), React enqueues a state update for the next render pass while button.disabled remains false during the current frame.

A second tap arriving 12 ms later hits the same active DOM node. Because Debounce, Throttle & rAF Coalescing is absent on the leading edge, both closures read isSubmitting === false and enter submitPayment().

02 → 03: Uncoalesced Handlers Feed Re-Entrant Boolean State

Even when a leading-edge lock blocks sub-second double-clicks, cellular handoffs introduce a 3.2-second TLS or gateway stall. Under Finite State Machines for UI, a SUBMITTING state cannot transition back to IDLE on a client-side timer while the HTTP socket remains open.

Boolean flags (isSubmitting, hasTimedOut) violate that invariant. A catch block executes setIsSubmitting(false) at t = 3,000 ms, re-enabling the button while Request #1 is still traversing the carrier network.

03 → 04: Re-Enabled UI Leaves Orphaned Sockets on the Wire

When the user taps "Pay" a second time, Request #2 enters the network stack alongside Request #1. Without AbortController and Cooperative Cancellation, the client never signals intent to drop Request #1, and an aborted fetch() only closes the local socket after the server may have parsed the TCP payload.

  • Client view: Marks Request #1 failed at t = 3,000 ms and dispatches Request #2.
  • Server view: Receives Request #1 at t = 3,150 ms and Request #2 at t = 3,420 ms.

04 → 05: Stateless Retries Commit Twice Without Idempotency Keys

Both HTTP requests reach the payment endpoint as independent mutations. As documented in Outbox Queues, Idempotency Keys & Causal Ordering, HTTP POST is non-idempotent by specification; only an explicit Idempotency-Key header scoped to the checkout intent lets the gateway deduplicate Request #2 and replay Request #1's receipt.

Architectural Fix & Invariants

type PayState = 'idle' | 'submitting' | 'settled' | 'failed';
export function createPaymentSubmitter(intentUuid: string) {
  let state: PayState = 'idle';
  const controller = new AbortController();
  return async function submit(payload: unknown): Promise<Response | null> {
    if (state === 'submitting' || state === 'settled') return null;
    state = 'submitting';
    try {
      const res = await fetch('/v1/charges', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', 'Idempotency-Key': intentUuid },
        body: JSON.stringify(payload),
        signal: controller.signal,
      });
      state = res.ok ? 'settled' : 'failed';
      return res;
    } catch (err) { state = 'failed'; throw err; }
  };
}

Architectural Invariants:

  • Synchronous transition guard: Mutate idle → submitting synchronously before any await or React render commit to block same-frame double-clicks.
  • Stable intent key: Generate intentUuid once per checkout form mount and reuse it across all retries so the gateway deduplicates re-submissions.
  • Terminal settled state: Treat settled as final for a given intentUuid; starting another payment requires a fresh mount and UUID.
  • Cooperative teardown: Call controller.abort() on unmount so stale responses never mutate unmounted UI state.

Golden Rule: Never guard a financial mutation with React's asynchronous render state; lock synchronously in memory and deduplicate idempotently on the server.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.