The Payment Succeeded. The Response Didn't. Now What?

An engineering deep dive into API retries, idempotency keys, PostgreSQL concurrency, and the failure that happens between two systems.

You order something online, enter your card details, and click Pay.

Five seconds later, the page gives up. No receipt. No confirmation. Just a timeout and a button that says Try again.

There are two possibilities. The payment never happened, or it happened and the confirmation didn't get back to you. From where you're sitting, both look exactly the same.

That small uncertainty has an uncomfortable consequence: clicking Try again could be the correct way to finish your purchase, or the beginning of a second charge.

I started looking into idempotency because I wanted to understand how real backend systems deal with this. At first, the solution sounded almost trivial: give every payment a unique ID and remember it. That description is technically useful, but it hides the hard questions. What if two requests arrive together? What if the server crashes after taking the money? What if the database says one thing and the payment provider says another?

The interesting part of idempotency is not the UUID. It's the engineering around everything that can fail after the UUID has been created.

This is a research-based explanation of those problems, with simplified SQL and system diagrams. It is not a benchmark or a claim that I ran these examples against a production payment service.

A timeout is not a failed payment

Consider a simplified checkout flow. The browser sends a request to an API. The API records the operation in its database. The database commits. The API returns a success response.

Now remove only the last step. The response gets lost between the API and the browser.

Figure 1. The database has committed, but the browser still sees a timeout.

This is the part that's easy to miss: a network error reports what the caller observed, not what the receiver committed.

The client can observe a broken connection. It cannot infer that the backend rolled back its work. Even an explicit client-side timeout doesn't necessarily stop the work still running on the server.

A few possible failure points make the distinction clearer:

If the API's side effect is read a profile, that uncertainty is usually manageable. If the side effect is charge a card, it becomes a correctness problem.

That is why reliable retries need a stronger contract than “send the request again if fetch throws.”

What idempotency actually means

In ordinary language, an idempotent operation has the same intended effect whether it is applied once or repeatedly.

Setting an account's status to active is different from adding 100 credits:

// Repeating this keeps the same intended final state.
account.status = "active";

// Repeating this changes the balance every time.
account.credits += 100;

Idempotency is about the effect, not the exact HTTP response or number of internal instructions executed. A repeated request can still create an access log, return a different status, or read from a database. What matters is whether it repeats the business action that should occur once.

HTTP already distinguishes methods by these semantics. Under RFC 9110, section 9.2.2, GET, PUT, and DELETE are idempotent by their defined intent. POST isn't inherently idempotent. An application can, however, provide an idempotent contract for a particular POST operation.

A second DELETE /users/42 might return 404 rather than the first request's 204. That doesn't violate the idea: the resource remains deleted. The requested effect is unchanged.

For payment APIs, we need an explicit way to say that two HTTP requests belong to one intended payment.

The key is an identifier for intent

Take this request:

POST /payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: 7e615d8a-5208-4b8c-a0dc-3e3c8770961b

{
  "orderId": "order_401",
  "amountMinor": 5000,
  "currency": "USD"
}

Here, amountMinor means the smallest unit of the currency, so 5000 represents $50.00.

The client generates the key when it starts this one intended operation. If the request times out, it sends a retry with the same key. If the customer intentionally starts another purchase, the client uses a new key.

Figure 2. A client can send the same payment request more than once without creating a new intended operation.

It would be tempting to remove the key and just compare request bodies. But that creates another bug.

Suppose a customer deliberately buys the same coffee twice. Both requests contain the same product, quantity, and price. Those are two legitimate operations. A payload comparison alone could mistake the second purchase for a retry of the first.

Figure 3. Same parameters are not the same thing as the same intent.

This distinction appears in Amazon's article on making retries safe with idempotent APIs. Amazon's approach emphasizes a caller-provided request identifier rather than assuming identical parameters always mean identical intent.

In a real API, the key also needs a clear scope. Typically, the same raw key should be interpreted within a customer or tenant's identity, not across every account in the platform. A key isn't an authentication token. The server must still authenticate the caller and authorize the operation.

What about the same key with different parameters?

Imagine the first attempt uses $50, while the retry uses the same key but changes the amount to $500.

That is no longer an innocent duplicate. Replaying the old response silently would conceal a mismatch, while executing the new request could repeat or change the business effect.

One approach is to store a request fingerprint alongside the key, derived from canonical operation fields such as order ID, amount, currency, and account. The backend checks the fingerprint on a repeated request. A mismatch returns a validation or conflict error, as specified by the API.

The fingerprint is not a substitute for the idempotency key. The fingerprint answers “Do these parameters match?” The key answers “Which intended operation is this?”

There is also a subtle implementation detail: canonicalization matters. Hashing arbitrary raw JSON isn't enough if equivalent requests can use different field ordering, formatting, or defaults. The contract should define what is considered the same request.

The race condition hidden in two harmless-looking lines

Here is the first implementation many of us would imagine:

const existing = await db.findByKey(key);

if (!existing) {
  const payment = await createPayment();
  await db.saveKey(key, payment);
}

Read it slowly. There is a gap between checking the key and creating the payment.

Now deploy two API instances behind a load balancer. The same payment request gets retried almost immediately, while the original request is still in flight.

1. Server A checks the key and finds nothing.
2. Server B checks the same key and also finds nothing.
3. Server A creates the payment.
4. Server B creates the payment.
5. Both eventually try to save the same key.

Figure 4. A non-atomic check can let both server instances trigger the protected side effect.

A uniqueness error at step five doesn't erase what happened at steps three and four. If those earlier steps called an external provider, the system may already have issued two charges.

This is a classic check-then-act race. Both instances made individually reasonable decisions using information that was stale by the time they acted.

Putting a mutex in one Node.js process doesn't solve it when other instances are running on different machines. Checking Redis first doesn't magically solve a side effect committed somewhere else. We need the authority that owns the invariant to decide who can claim the operation.

For database-local effects, PostgreSQL can provide that authority.

Let PostgreSQL coordinate the operation

Start with a table that owns the idempotency identity:

CREATE TABLE idempotency_requests (
    account_id BIGINT NOT NULL,
    idem_key TEXT NOT NULL,
    request_hash TEXT NOT NULL,
    payment_id BIGINT,
    response_code INTEGER,
    response_body JSONB,
    PRIMARY KEY (account_id, idem_key)
);

The composite primary key enforces that one account can have at most one committed record for a given key. This is a database rule, not an if statement in the application.

A transaction can try to claim a new key with:

INSERT INTO idempotency_requests
  (account_id, idem_key, request_hash)
VALUES
  ($1, $2, $3)
ON CONFLICT (account_id, idem_key)
DO NOTHING
RETURNING idem_key;

The RETURNING result tells the application whether this transaction inserted the row. PostgreSQL's unique index coordinates competing insert attempts. If two transactions try the same key, one may wait for the other to commit or roll back before it can decide what to do.

But this insert by itself does not make a payment operation safe. We also have to put the protected database-side effect and its saved result into the same transaction.

Here's the logical flow:

BEGIN

  Try to insert the idempotency key.

  If this transaction claimed the key:
      Create the database payment record.
      Store its response under the same key.
      COMMIT.

  Otherwise:
      Read the previously committed operation.
      Compare request parameters.
      Return the saved result or reject the mismatch.

END

Figure 5. One transaction coordinates the key claim, local effect, and saved result.

Why is the shared transaction necessary?

Consider two tempting designs:

Save the key first, commit, then create the payment. If the server crashes in between, the key exists but the payment doesn't. A retry might mistake the operation for a completed one.

Create the payment first, commit, then save the key. If the server crashes in between, the payment exists but the key doesn't. A retry might create the payment again.

The transaction lets the database treat the key, payment record, and saved result as one all-or-nothing unit. Either they commit together or their uncommitted changes roll back.

This is the part of ACID that matters most in this example: atomicity. Isolation and uniqueness also matter because multiple transactions can compete for the same operation.

A PostgreSQL detail that surprised me

At its default READ COMMITTED isolation level, PostgreSQL gives each statement its own suitable snapshot of committed data. An INSERT ... ON CONFLICT DO NOTHING can decline to insert because of a transaction whose row was not visible in the original statement snapshot. A subsequent SELECT can then see the newly committed record.

That means the duplicate path should be designed around actual PostgreSQL concurrency semantics. You shouldn't assume one earlier SELECT proved that the key was absent, or that SELECT FOR UPDATE can lock a row that has not been inserted.

A unique constraint is a precise safeguard for claiming a previously absent identity. More aggressive isolation levels can be useful for other invariants, but turning on SERIALIZABLE doesn't replace the need to define your business rules, and applications must be ready to retry transactions aborted with serialization failures.

PostgreSQL documents these behaviors in its sections on transaction isolation and INSERT ... ON CONFLICT.

The order ID provides another layer

Suppose a buggy client creates two different idempotency keys for the same checkout order. The key-based deduplication won't recognize those as the same operation.

An application that permits only one payment record per order can enforce a separate unique constraint on (account_id, order_id).

This guards a different invariant. The idempotency key represents the request attempt group. The order ID represents the business resource. It is often worth protecting both.

The database is not the payment provider

So far, everything has happened inside PostgreSQL. No real card was charged. That makes the transaction boundary relatively clear.

Real checkout systems don't stop at a database insert. They call an external payment processor, and that processor owns its own transactions and state.

Now consider a different failure:

1. Your backend saves a local operation as PROCESSING.
2. It calls the payment provider with a stable operation identifier.
3. The provider charges the customer successfully.
4. Your backend crashes before it saves the provider's response.

What does your database know? Only that the local operation was in progress. What does the provider know? The charge succeeded.

You cannot ROLLBACK that charge with a PostgreSQL transaction. The provider is a separate system with a separate commit boundary.

This leads to the design principle I find most important in the entire topic: never confuse a local database's atomicity with global atomicity across services.

A more realistic architecture gives each operation a durable identity and explicit lifecycle:

These names are a proposed model, not a universal API standard. The distinction between FAILED and UNKNOWN matters far more than the exact labels.

The backend should reuse a provider-supported idempotency identity across retries, keep the provider's payment ID when available, and have a way to query or reconcile an uncertain outcome. If the provider does not offer reliable deduplication or lookup, the system cannot confidently promise that retries will never create a second external effect.

Also, wrapping a long HTTP request to a provider inside an open SQL transaction does not solve this. It may increase lock contention and exhaust database connections while still being unable to roll back the provider's work.

This is why payment reliability is often a state-machine and recovery problem, not just a SQL problem.

A saved 500 response is not necessarily an invitation to retry differently

Stripe's documentation on idempotent requests contains an interesting detail: after an endpoint begins execution, Stripe saves the status code and response body for a given key and may return that same result for subsequent requests, including 500 responses.

Stripe also compares parameters for reused keys, documents how concurrent conflicts may be treated, and allows old keys to be pruned after they are at least 24 hours old. Reusing a key after pruning can start a new request.

These are Stripe's documented behaviors, not guarantees shared by all APIs. In particular, seeing a repeated 500 does not prove that the underlying business operation was rolled back. The appropriate recovery depends on the payment operation's state and the provider's specific API contract.

The lifetime of a key can be a correctness decision. If you delete the only deduplication record too early, an old request that arrives late can look like a new operation.

Amazon discusses an even stranger case: a client retries a resource-creation request after that resource has already been deleted. Its engineering guidance favors preserving the original request's meaning rather than unexpectedly creating a new resource. That's a useful reminder that retries are about the original intent, not blindly redoing the current instruction from scratch.

Idempotency and retry policy are separate mechanisms

So far we've focused on making a repeat safe. That doesn't mean a system should retry every error as quickly as it can.

Suppose an upstream service is overloaded. A thousand callers time out and immediately send three more requests each. The retries may make the overload worse, causing yet more timeouts.

A retry policy needs its own rules:

• Which failures are transient enough to retry?
• How long should the client wait before declaring a timeout?
• How many retries fit within the overall deadline?
• Should the server's Retry-After instruction be respected?
• How do backoff and jitter keep many clients from retrying together?
• At which layer should retrying happen, so the client, gateway, and service don't each multiply the attempts?

Exponential backoff increases delays between attempts. Jitter introduces randomness so the attempts spread out rather than landing in synchronized bursts. Neither is a substitute for idempotency when the request changes state.

Amazon's Builders' Library discussion of timeouts, retries, and jitter explains the overload problem well.

A simple way to remember the split is this: idempotency controls repeated effects; retry policy controls repeated attempts.

What does “exactly once” actually refer to?

I've come to distrust the phrase exactly-once processing unless it comes with a clearly defined boundary.

A single payment operation can involve several different events:

1. An HTTP request arrives at the API.
2. The handler runs.
3. A PostgreSQL transaction commits.
4. An external payment provider processes a charge.
5. A webhook is delivered.
6. An order service updates the order status.

A request may arrive more than once, and a webhook may be delivered more than once. Still, each consumer can be designed so that repeated delivery doesn't create another intended business effect.

That is the more precise goal: allow repeat attempts while coordinating the effects that must happen at most once.

But at most once and eventually completed aren't the same guarantee. If a request is lost and the system never retries or reconciles it, the effect may happen zero times. Reliability needs both duplicate protection and a plan for making progress after failure.

The same lesson shows up in event-driven architecture. The transactional outbox pattern can atomically store a database change and an event to publish later. But the relay may publish the event more than once after a crash, so consumers still need to handle duplicate deliveries.

In other words, protecting one boundary does not automatically protect every downstream boundary.

The questions I'd ask in a design review

After reading these designs, I think the most useful way to assess an idempotent API is to walk through the failures before choosing the implementation.

What counts as one operation? Two identical purchases may be legitimate. Two retries of one purchase are not two operations.

Who creates the operation ID? If the client creates a new ID on every retry, the server can't recognize the retry group.

Where is the atomic claim? Two processes should not both be able to decide that they own a key and perform the protected effect.

Which systems can commit independently? A PostgreSQL transaction cannot undo a payment provider's charge, message broker publish, or outgoing email.

What happens when the result is unknown? “Processing,” “failed,” and “not sure” must not be collapsed into one vague error.

How long is the identity retained? A delayed retry after expiration might restart an operation unless another business invariant prevents it.

What would the failure test look like? The meaningful test cases include two requests arriving together, a crash before commit, a crash after commit but before response, a provider timeout, reused keys with changed parameters, and a very late retry.

You can ask all of those questions before writing a line of application code. They reveal more about the design than a happy-path demo does.

What I took away

I began this topic thinking about a familiar API concern: the user presses a button twice, so the backend should avoid duplicate records.

The deeper problem is uncertainty between independently failing components. HTTP doesn't tell the client whether the server committed. A local mutex doesn't protect several servers. A database constraint can't undo an external charge. And a retry policy can turn a small outage into a larger one if it floods an already struggling dependency.

The solution isn't one magic header. It's a sequence of contracts: a stable operation identity, a safe way to claim that identity, transaction boundaries that match the actual side effects, and a recovery path for cases where the outcome isn't known.

If I had to reduce the entire subject to one sentence, it would be this:

A retry repeats a request. A reliable system makes sure that repeating the request doesn't accidentally repeat the business decision.

That principle applies to payments, but also to order processing, background jobs, cloud resource provisioning, and almost any system where a timeout can hide a successful operation.

References and further reading

• RFC 9110, §9.2.2: Idempotent Methods
• Stripe API reference: Idempotent requests
• Amazon Builders' Library: Making retries safe with idempotent APIs
• PostgreSQL: INSERT and ON CONFLICT
• PostgreSQL: Transaction Isolation
• Amazon Builders' Library: Timeouts, retries, and backoff with jitter
• Transactional Outbox pattern