Idempotent Retries for LLM Calls: Stop Paying Twice for One Request

Disclosure: this post is written by the team building OROO Blue, an AI API gateway. The techniques below are vendor-neutral.

LLM requests are slow compared to most API calls, which makes them unusually likely to hit timeouts. And a client-side timeout hides an awkward truth: the model may have finished the request; you just never got the response. Retry naively and you pay for the tokens again, and any side effects in your app (saving a message, sending a notification, deducting credits) can run twice.

Here's a practical pattern for making those retries safe.

1. Key the action, not the attempt

Generate an idempotency key when the user action happens (clicking Send, submitting a form), not when each HTTP request is made. Reuse it for every retry of that action.

# generic pseudocode, not tied to any SDK
key = new_unique_id()            # once per user action
for attempt in 1..MAX_ATTEMPTS:
    res = call_llm(req, idempotency_key=key, timeout=T)
    if res.ok: return res
    if not retryable(res.error): raise res.error
    sleep(backoff_with_jitter(attempt))
raise GaveUp

If your provider or gateway supports idempotency keys, pass the key along. Either way, use it in your own system too.

2. Dedupe side effects on your side

Store the key alongside the result. Before writing a reply, sending a message or charging a user, check whether that key was already processed. A unique constraint on the key column is a simple, reliable guard against races.

3. Only retry what can succeed

• Retry: timeouts, connection resets, rate-limit responses, temporary 5xx errors. Honour any retry-after hint.
• Don't retry: malformed requests, auth failures, validation errors. Fail fast.

4. Backoff with jitter, and a cap

Exponential backoff (1s, 2s, 4s…) plus random jitter stops a crowd of clients from retrying in lockstep. Cap both attempts (3–5 is typical) and total wait time.

5. Handle the double-click

Disable the button while a request is in flight. If the user resubmits anyway, reuse the existing key.

6. Log around the key

Log key, attempt number, error class, latency and token usage. When a bill looks off, you can replay the exact story.

7. Put a ceiling on spend

Retry bugs multiply cost quietly. Per-key rate and token limits plus usage alerts turn a runaway loop into a minor incident.

Where we fit

In OROO Blue, idempotency keys prevent double charges on retries, and per-key model, token and rate limits help cap runaway loops. It provides one API and one key for OpenAI, Google and Anthropic models, with prepaid INR billing for Indian teams: https://orooblue.com

Original post: https://orookids.blogspot.com/2026/11/retry-without-paying-twice-idempotency.html

How do you handle retries for streaming responses that drop midway: retry with the same key, or start fresh?