Idempotency keys: how to make a retried POST do its job once

The client did nothing wrong, which is the part that took me a while to accept. It sent a request, got no answer, and sent it again, which is exactly what every HTTP client is told to do. The problem is that "no answer" covers two situations that look identical from the client's side: either the request never arrived, or it arrived, did its work, and the response got lost on the way back. The client can't tell which. So it retries, which is correct, and the server does the work twice, which is not.

Timeouts and flaky networks have always caused this. What's newer is AI agents calling your API, and in my experience they retry harder than any human-written client, because a model that sees an error will usually just try again with the same arguments until something works.

What idempotent means

An operation is idempotent if doing it once and doing it five times leave the system in the same state and give the client the same answer. "Set the user's email to X" is idempotent. "Add 5 to the balance" is not.

HTTP sorts its methods this way already. GET, HEAD, PUT and DELETE are defined as idempotent. Deleting something already gone returns 404 rather than 204, so the response changes between calls, but the state doesn't, and the state is what the definition is about.

POST has no such guarantee, and can't really, because POST means "process this" and the server decides what processing means. Create an order, send an email, charge a card. The awkward thing is that the operations that actually matter in an API, the ones involving money or messages, are nearly all POSTs, so the protocol gives you nothing where you need it most.

The Idempotency-Key header

The fix has been in payment APIs for years and is now an IETF draft, "The Idempotency-Key HTTP Header Field," from the HTTPAPI working group. It's still a draft and hasn't become an RFC, but the design is simple enough that it's the convention regardless.

The client generates a unique key, usually a UUID, for each logical operation. "Charge this customer $5 for this order" gets one key, and every retry of that operation sends the same key in an Idempotency-Key header. The server uses it to recognise a request it has already seen and return the original answer instead of doing the work again.

The word that matters there is "logical." The key identifies what the user is trying to accomplish, not the individual HTTP request. I've seen clients generate a fresh key inside the retry loop, so every attempt carries a different one, and at that point you've implemented the header without any of the behaviour.

What the server has to do

Each unsafe request with a key lands in one of four states, and the implementation is basically these four branches. GET and HEAD skip the whole thing; they're already idempotent and storing their responses would just be a cache with worse semantics.

First arrival: claim the key, run the handler, store the response status and body. The claim has to be atomic, because two copies can arrive within the same millisecond on two instances, and if the claim is a read followed by a write, both read "nothing here" and both run.

Retry after the first finished: return the stored response, run nothing. I add an Idempotent-Replayed: true header so I can tell real responses from replays in logs.

Retry while the first is still running: there's nothing to replay yet. Return 409 Conflict with Retry-After and let the client ask again in a second. This feels like it should be rare and it isn't, for reasons I'll get to in testing.

Same key, different body: something is wrong on the client. Hash the body on the first request, store the hash, compare on every retry. A mismatch gets 422 Unprocessable Content, which is what the draft recommends. Hash a canonical form of the body, with object keys sorted, or a client that re-serialises {"a":1,"b":2} as {"b":2,"a":1} on retry gets a 422 it didn't earn.

Keys expire. 24 hours is the usual window, long enough for any realistic retry schedule and short enough that the store doesn't grow forever.

Implementation with Express and Redis

The middleware depends on a small interface over Redis rather than the ioredis client directly, so tests can swap in a fake. It also makes it obvious the whole thing needs three commands.

import { createHash } from "node:crypto";
import type { Request, Response, NextFunction } from "express";

declare global {
  namespace Express {
    interface Request {
      userId?: string;         // set by auth middleware
      idempotencyKey?: string; // set here, for handlers to forward downstream
    }
  }
}

// The subset of ioredis we use. Lets tests pass a fake.
export interface KeyStore {
  set(key: string, value: string, ...args: (string | number)[]): Promise<"OK" | null>;
  get(key: string): Promise<string | null>;
  del(key: string): Promise<number>;
}

type IdempotencyRecord =
  | { state: "running"; hash: string }
  | { state: "done"; hash: string; status: number; body: unknown };

const SAFE_METHODS = new Set(["GET", "HEAD", "OPTIONS"]);
const RUNNING_TTL_SEC = 30;          // lock held while the handler runs; exceed it and a retry runs again
const DONE_TTL_SEC = 60 * 60 * 24;   // replay window, 24h

// Sort object keys so {"a":1,"b":2} and {"b":2,"a":1} hash the same.
export function canonicalJson(value: unknown): string {
  if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`;
  if (value && typeof value === "object") {
    const entries = Object.keys(value).sort()
      .map((k) => `${JSON.stringify(k)}:${canonicalJson((value as Record<string, unknown>)[k])}`);
    return `{${entries.join(",")}}`;
  }
  return JSON.stringify(value ?? null);
}

export function hashBody(body: unknown): string {
  return createHash("sha256").update(canonicalJson(body)).digest("hex");
}

The claim is one Redis command. SET ... NX only writes if the key doesn't exist and returns null otherwise, so the first request wins and every other copy sees null. That one flag does all the concurrency work here, which I find slightly unnerving but is also why it's reliable.

export function idempotency(store: KeyStore) {
  return async function middleware(req: Request, res: Response, next: NextFunction) {
    const key = req.header("Idempotency-Key");
    if (!key || SAFE_METHODS.has(req.method)) return next();
    if (!req.userId) return res.status(401).end();

    req.idempotencyKey = key;
    const storeKey = `idem:${req.userId}:${key}`;
    const hash = hashBody(req.body);

    // Atomic claim. Only one concurrent request wins.
    const claimed = await store.set(
      storeKey, JSON.stringify({ state: "running", hash }), "NX", "EX", RUNNING_TTL_SEC,
    );

If the claim failed, read what's there and pick one of the three replies.

    if (claimed === null) {
      const existing = JSON.parse((await store.get(storeKey)) ?? "null") as IdempotencyRecord | null;
      if (!existing) return res.status(409).set("Retry-After", "1").end(); // lock expired between SET and GET
      if (existing.hash !== hash) {
        return res.status(422).json({ error: "Idempotency-Key reused with a different request body" });
      }
      if (existing.state === "running") {
        return res.status(409).set("Retry-After", "1").json({ error: "Request with this key is still in progress" });
      }
      return res.status(existing.status).set("Idempotent-Replayed", "true").json(existing.body);
    }

If the claim succeeded, wrap res.json so the response gets stored when the handler sends it. Anything that ends the response without storing a record, which covers thrown errors, 5xx, and handlers that use res.send, releases the claim so the client can retry. A 4xx is stored and replayed on purpose: the same body will fail validation the same way, so there's no reason to run it again to find out.

    // Store the response when the handler sends it. 5xx is not stored, so the client may retry it.
    let stored = false;
    const originalJson = res.json.bind(res);
    res.json = (body: unknown) => {
      if (res.statusCode < 500) {
        stored = true;
        const record: IdempotencyRecord = { state: "done", hash, status: res.statusCode, body };
        store.set(storeKey, JSON.stringify(record), "EX", DONE_TTL_SEC)
          .catch((err) => console.error("idempotency: failed to store response", storeKey, err));
      }
      return originalJson(body);
    };

    // next() returns void in Express, so try/catch here never sees a handler error.
    // Release the claim whenever the response ends without a stored record: thrown error, 5xx, res.send().
    res.on("close", () => {
      if (!stored) store.del(storeKey).catch((err) => console.error("idempotency: failed to release claim", storeKey, err));
    });

    next();
  };
}

Wire it up after body parsing and auth. The handler validates its input before anything reaches Stripe, and it forwards a key on the Stripe call. Your key protects your server from running the handler twice; Stripe's protects the charge if your handler runs once but your call to Stripe times out and gets retried. Stripe scopes idempotency keys to your Stripe account, not to your users, so forward the caller-scoped key — otherwise two users who both send k1 would collide at Stripe.

import express from "express";
import Redis from "ioredis";
import Stripe from "stripe";
import { z } from "zod";

const app = express();
app.use(express.json());
app.use(authenticate);                  // sets req.userId
app.use(idempotency(new Redis(process.env.REDIS_URL)));

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);
const ChargeBody = z.object({ amount: z.number().int().positive(), currency: z.enum(["usd", "gbp"]) });

app.post("/charges", async (req, res) => {
  const parsed = ChargeBody.safeParse(req.body);
  if (!parsed.success) return res.status(400).json({ error: parsed.error.flatten() });
  const intent = await stripe.paymentIntents.create(
    { amount: parsed.data.amount, currency: parsed.data.currency },
    { idempotencyKey: `${req.userId}:${req.idempotencyKey}` },
  );
  res.status(201).json({ id: intent.id, status: intent.status });
});

The client side

The server half only works if the client reuses the key, so here is the client half: a fetch wrapper with exponential backoff, jitter so a thousand clients that timed out together don't retry together, and respect for Retry-After.

export async function postWithRetry(url: string, body: unknown, opts: { attempts?: number } = {}) {
  const key = crypto.randomUUID(); // one key per logical operation, reused on every retry
  const attempts = opts.attempts ?? 5;

  for (let attempt = 0; ; attempt++) {
    let res: Response | undefined;
    try {
      res = await fetch(url, {
        method: "POST",
        headers: { "Content-Type": "application/json", "Idempotency-Key": key },
        body: JSON.stringify(body),
        signal: AbortSignal.timeout(10_000),
      });
      if (res.ok || (res.status < 500 && res.status !== 409 && res.status !== 429)) return res;
    } catch (err) {
      if (attempt + 1 >= attempts) throw err; // network error or timeout
    }
    if (attempt + 1 >= attempts) return res!;

    const retryAfterSec = Number(res?.headers.get("Retry-After"));
    const backoffMs = Math.min(30_000, 500 * 2 ** attempt);
    const jitterMs = Math.random() * backoffMs;
    const waitMs = retryAfterSec > 0 ? retryAfterSec * 1000 : backoffMs + jitterMs;
    await new Promise((r) => setTimeout(r, waitMs));
  }
}

The key is generated once, outside the loop, and that line placement is the entire point of the function. If you're exposing this to an AI agent through a tool definition, generate the key inside your tool wrapper rather than asking the model to supply it. Models are not reliable about reusing the same UUID across attempts.

Edge cases

Server crashes mid-request. The claim is written as running, then the process dies before the handler finishes. Nothing deletes the key, so every retry gets 409 until it expires. That's why the running claim gets 30 seconds and only the finished record gets 24 hours: thirty seconds is a stale lock the client can wait out, twenty-four hours is an outage for that operation. The TTL cuts the other way too. If a handler legitimately takes longer than 30 seconds, the lock expires while it's still running, a retry claims the key fresh and runs the handler again, and you're back to the duplicate you started with. Set the running TTL comfortably above your slowest handler, and if that number is uncomfortable, the handler should probably be a queued job.

Multiple instances. An in-memory Map works in development and fails the moment you run two copies of the service, because the retry lands on the instance that never saw the first request. The store has to be shared: Redis, or a database table with a unique constraint on the key and INSERT ... ON CONFLICT DO NOTHING as the claim. For a lot of services the table is honestly the better choice.

Scoping. The store key is idem:${userId}:${key}, and that prefix is not decoration. Without it, a client who guessed or leaked another client's key could receive the other client's stored response, order details and all. Scope by whatever identifies the caller: user ID, API key ID, tenant.

Testing it

The test that matters is the concurrent one. Fire the same request five times at once and assert the side effect happened exactly once. This runs against a fresh Express app on an ephemeral port, with a fake auth middleware and the in-memory KeyStore fake in place of Redis.

import assert from "node:assert/strict";
import type { AddressInfo } from "node:net";

// fresh app for the test, with a fake auth middleware:
// app.use((req, _res, next) => { req.userId = "u1"; next(); });

let charges = 0;
app.post("/charges", async (req, res) => {
  await new Promise((r) => setTimeout(r, 20)); // pretend to talk to Stripe
  charges++;
  res.status(201).json({ chargeId: "ch_1", amount: req.body.amount });
});

const srv = app.listen(0, "127.0.0.1");
await new Promise((r) => srv.once("listening", r));
const base = `http://127.0.0.1:${(srv.address() as AddressInfo).port}`;
const send = (key: string, body: unknown) =>
  fetch(`${base}/charges`, {
    method: "POST",
    headers: { "content-type": "application/json", "Idempotency-Key": key },
    body: JSON.stringify(body),
  });

const results = await Promise.all([1, 2, 3, 4, 5].map(() => send("k1", { amount: 500 })));
assert.equal(charges, 1);
assert.deepEqual(results.map((r) => r.status).sort(), [201, 409, 409, 409, 409]);

When I ran this, the part that surprised me was the shape of that second assertion, and it took me a minute to work out why I'd expected something else. I had assumed the four losing requests would get the cached 201, because that's what "idempotent" sounds like it promises. They got 409 every time, because all five arrived during the 20 milliseconds the first was still running and there was no response to replay yet. The replay only shows up on the sixth request, sent after the first finished. That's correct, and it's what the 409 state is for, but it means a client has to treat 409 as "wait and try again" rather than as an error, or you've traded a duplicate problem for a "my request failed for no reason" problem.

The other thing testing caught, which I'm less proud of. My first version wrapped await next() in a try/catch and deleted the claim in the catch. The unit test passed, because it called the handler directly and the rejection came back through next. Against a real Express app, a route that throws produced a 500 from Express's error handler, the catch never ran, and the retry got 409 for the next 30 seconds. next() returns nothing in Express. In Express 5 a rejected async handler goes to the error handler, not to whoever called next; in Express 4 it's an unhandled rejection unless you add a wrapper like express-async-errors, and the request hangs until the client gives up. Either way the close listener releases the claim. It exists because of that run, and it's a good argument for testing middleware through the framework rather than around it.

Checklist

• Client generates one UUID per logical operation and reuses it on every retry.
• Server claims the key atomically (SET NX or a unique constraint) before running the handler.
• Body is hashed in canonical form and compared; a mismatch gets 422.
• In-progress gets 409 with Retry-After; finished gets the stored response.
• Thrown errors and 5xx release the claim; 2xx and 4xx are stored.
• Running claim has a short TTL, longer than your slowest handler; stored response has a long one.
• Keys are scoped per caller, and forwarded to downstream APIs that support them.
• Store is shared across instances.
• One test sends N concurrent copies through the real framework and asserts one side effect.

When you don't need this

If the operation is already idempotent, skip all of this. A PUT that replaces a record, a DELETE, or an upsert keyed on something the client controls gives the same state on the tenth retry as the first. You can often get idempotency from the data model too: if an order carries a client-supplied order number with a unique constraint, the second INSERT fails and you return the existing order, and that's the whole implementation. The header-and-Redis approach is for the POSTs where there's no natural key and the side effect lives outside your database, which in practice means charges, emails, and anything that calls someone else's API and can't be taken back.