AI API Timeouts: When to Retry and How to Avoid Duplicate Requests

Your AI API request times out. You click Retry, and the second request succeeds. Later, the logs show two upstream attempts for what looked like one action.

That is possible without either side behaving mysteriously. A timeout tells you that one component stopped waiting. It does not, by itself, tell you whether the component on the other side accepted, completed, or cancelled the work.

The useful question is not simply “Should I increase the timeout?” It is “What do I know about the first attempt, and what could happen if I send another one?”

First, identify which timeout actually happened

An AI request can pass through a browser, an application, a reverse proxy, a gateway, and an upstream service. Each can enforce a different limit.

A client exception and an HTTP error response are different observations. If your HTTP client raises a read timeout, you may not have an HTTP status code at all. If you receive 504 Gateway Timeout, a server acting as a gateway or proxy did not receive a timely upstream response. Neither observation proves that no upstream work occurred.

In HTTPX, the timeout categories are:

• Pool: waiting to acquire a connection from the client's connection pool.
• Connect: waiting to establish a socket connection.
• Write: waiting to send a chunk of request data.
• Read: waiting to receive a chunk of response data.

These categories help locate the wait. They are not a universal taxonomy for every SDK, so check the library and version you actually use.

A confirmed pool timeout before dispatch is different from a read timeout after the request body was sent. In the latter case, the server may already be processing the request. Even when diagnosing a pre-dispatch failure, check whether an SDK or another layer already made an earlier attempt.

A read timeout is not an overall deadline

For HTTPX, a read timeout limits the wait for a chunk of data. It does not mean the entire streamed answer must finish within that duration.

A stream that keeps receiving data can run longer than the configured read timeout. A heartbeat may keep the connection active while the user still has no useful answer. Conversely, a legitimate pause can exceed a short read timeout before the first text appears.

Use separate limits for separate purposes:

1. Connection and pool limits prevent indefinite waits for transport resources.
2. An inactivity limit detects a stalled read or write.
3. An application deadline bounds the whole logical operation, including queues, attempts, and backoff.

For illustration, suppose an operation has a 50-second overall budget and its first attempt consumes 30 seconds. A retry has less than 20 seconds left after allowing for backoff and cleanup. Starting a fresh 50-second timer would silently expand the user's original wait.

Use a monotonic clock for elapsed durations inside one process. Propagate the remaining budget through components you control using a defined contract; do not assume that an arbitrary deadline header will be honored by another service.

Increasing a browser timeout will not override a shorter proxy limit. Disabling all timeouts can leave abandoned operations occupying connections and workers.

Treat the first attempt's outcome as a separate fact

After a timeout, record what is known rather than converting every case into “failed, safe to retry.”

A practical distinction is:

• Not dispatched: reliable local evidence shows this attempt did not leave the client.
• Explicitly rejected: the service returned a rejection; use that endpoint's documented semantics to decide whether a retry is appropriate.
• Outcome unknown: the request may have been accepted, but no authoritative final result was received.
• Partial output: some answer arrived, but the protocol did not report successful completion.
• Completed: an authoritative result confirms success.

These are application concepts, not proposed HTTP status codes. They are useful because a lost response and a rejected request need different recovery paths.

If an API provides a documented job-status or result-retrieval endpoint, use it to reconcile a known operation before creating another one. A request ID may help correlate logs, but it is not automatically a retrievable job ID.

POST retries need endpoint-specific guarantees

Many AI generation endpoints use POST. Repeating the same JSON does not make a POST request idempotent.

HTTP semantics define idempotency in terms of the intended server effect of repeated requests. They do not promise identical response text. Setting a deterministic-looking model parameter is not a substitute for a request-deduplication contract.

RFC 9110 says a client should not automatically retry a non-idempotent request unless it knows the semantics are idempotent or can detect that the original request was never applied.

Some APIs provide idempotency keys. Use them only according to the specific endpoint's documentation. Before relying on one, establish:

• Which operation and endpoint support the key?
• What account or tenant scope does it use, and how long is it retained?
• What happens when two requests with the same key arrive concurrently?
• What happens if the same key is reused with different parameters?
• Does a retry return the original result, an in-progress response, or a conflict?

Adding an arbitrary header does not create these guarantees. A gateway forwarding the header also does not prove that the upstream implements deduplication.

Separate UI deduplication from upstream deduplication

For an application you control, assign a stable operation ID before sending the first attempt. Disable repeated clicks while that operation is active, and use a durable record to track its state.

For example, an operation record might include an authenticated owner, operation ID, request fingerprint, attempt references, state, and result reference. Keep credentials and unnecessary prompt content out of diagnostic logs.

Claiming an operation must be atomic. Otherwise, two workers can both observe that it is missing and both start work. A request fingerprint should detect accidental reuse of an operation ID with changed parameters; it should not globally merge two intentional requests just because their prompts match.

This can prevent duplicate submissions inside your application. It cannot guarantee exactly-once execution across an upstream boundary. If a worker crashes after the upstream accepts the request but before the local result is saved, the local state is still uncertain unless the upstream contract provides a way to reconcile it.

Design an explicit recovery path for that uncertainty. For some products, the right action is to ask the user to start a new generation after explaining that the previous result could not be confirmed. For others, an asynchronous job with durable status is a better fit, if the API supports that workflow.

Cancellation and a broken stream need their own handling

Closing a tab or aborting a local HTTP call does not prove that upstream generation stopped. Cancellation has to propagate through every relevant layer, and the upstream must support the applicable cancellation behavior. Check the API's documented usage and charging rules separately.

When the user cancels, stop scheduling retries and release local resources. Record cancellation independently from a timeout, and distinguish a cancellation request from a confirmed cancelled operation.

A broken stream also needs an honest UI state. An initial HTTP 200 is not proof that the entire generation completed successfully; a stream can fail later.

Preserve partial output with an “interrupted” indication. If the user requests another generation, show it as a new attempt. Do not silently append a fresh answer to the old partial answer as if the original stream resumed. Resume only when the endpoint explicitly supports it and defines the required cursor or event identifier.

For agents that call tools, deduplicate side effects at the tool or business-action boundary too. Preventing a duplicate model request does not undo a tool action that already happened.

Choose one owner for the retry budget

Retries can multiply across layers. If the application, SDK, and gateway each allow three total attempts, one user action could lead to as many as 27 upstream attempts in a fully nested failure path.

That is an illustration, not a claim about any particular provider. The point is to inspect the complete path rather than count only the loop in your own code.

Choose a retry owner and a shared total-attempt budget. For retries allowed by the endpoint's semantics, use bounded backoff with jitter, honor documented server guidance, and check the remaining operation deadline before dispatching again.

Backoff controls timing. It does not make an ambiguous POST safe to replay.

During an incident, a useful sanitized record includes the operation ID, attempt number, timeout category, elapsed time, whether dispatch occurred, whether any content arrived, and any safe-to-share upstream request ID. Keep successful, interrupted, cancelled, and unknown outcomes distinct.

For integrations through YW Code API (ywcode), or any other AI API gateway, check the timeout and retry behavior of both the client and the forwarding path. Keep the logical operation separate from its physical HTTP attempts.

Before retrying, answer three questions: could the first attempt already have run, is this endpoint safe to replay, and is there enough of the original deadline left? If any answer is unknown, make that uncertainty visible instead of hiding it behind another request.

Related reading: Measure TTFT, queue time, and streaming latency and handle AI API 429 errors.

References: HTTPX timeout documentation, MDN: 504 Gateway Timeout, and RFC 9110, section 9.2.2.

AI disclosure: This article was written by an AI agent. Examples are illustrative and are not results from a live provider test.