Dev.to WebDev 🛠 Dev 👁 0 📖 4 min read

From 44 seconds to 6: how I handle 429 rate limits without slowing down

I had a tool that usually answered in a few seconds, and sometimes took 44. For a batch job that's annoying. For a tool called by AI agents, which typically wait around 30 seconds for an answer, it's broken: the agent ge

I had a tool that usually answered in a few seconds, and sometimes took 44. For a batch job that's annoying. For a tool called by AI agents, which typically wait around 30 seconds for an answer, it's broken: the agent gets "still running" and the user gets nothing.

The upstream service was rate-limiting me with HTTP 429 Too Many Requests. My first instinct was "add better backoff". That was the wrong instinct. Here's what was actually happening, and what fixed it.

Where the 44 seconds went

I pulled the run log and lined up timestamps. The time broke down roughly like this:

  • ~18 s: opening the first connection through a proxy session stalled. Nothing failed; it just hung.
  • ~12 s: after a 429, opening a replacement session was slow again.
  • ~7.5 s: at the very end, the process waited up to 5 seconds for background listeners before exiting.

The actual rate-limit waits were a small part of it. Most of the time was spent waiting on things that were never going to succeed quickly.

The sneaky one was the first. I had a 15-second timeout configured, but it was a read timeout. It limited each read, not the whole request. A connection that trickled bytes slowly, or stalled during setup, could blow straight past it.

Fix 1: Total-time budgets, not just read timeouts

Most HTTP clients let you set separate timeouts for connect, read, write and pool. Set all of them, and also wrap the whole operation in an overall deadline:

import asyncio
import httpx

TIMEOUT = httpx.Timeout(connect=5.0, read=7.0, write=5.0, pool=5.0)

async def fetch(client: httpx.AsyncClient, url: str) -> httpx.Response:
    # Hard cap on the whole request, including slow trickles.
    return await asyncio.wait_for(client.get(url, timeout=TIMEOUT), timeout=8.0)

I also capped each query (which needs several requests) at 16 seconds. If it can't finish, the tool returns a clear error item instead of hanging. A fast, honest failure is better than a slow success the caller never sees.

Fix 2: On 429, switch, don't wait

Classic advice for 429 is exponential backoff: wait 1 s, 2 s, 4 s... That's right when you are the problem and the server needs you to slow down globally. It's wrong when the limit is tied to an identity (an IP or session) and you have another identity available.

My tool runs through rotating proxy sessions, so the fix was:

  • Keep a spare session already open.
  • On a 429, a stall or a network error, switch to the spare immediately and start opening a new spare in the background.
  • Only back off (briefly) if all sessions are limited.
async def get_with_failover(pool, url):
    for attempt in range(3):
        session = await pool.current()
        try:
            resp = await fetch(session, url)
            if resp.status_code != 429:
                return resp
        except (httpx.TransportError, asyncio.TimeoutError):
            pass
        pool.rotate()            # promote the warm spare
        pool.warm_spare()        # start opening the next one, don't await
    await asyncio.sleep(0.5)     # everything limited: short pause, then last try
    return await fetch(await pool.current(), url)

The important part is that the replacement is already warm. Switching to a session you still have to open just moves the wait somewhere else.

Fix 3: Race the start

Opening the first session was the slowest, least predictable step. So I open two at startup and use whichever is ready first; the other becomes the spare.

async def first_ready(*coros):
    tasks = [asyncio.create_task(c) for c in coros]
    done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    winner = done.pop().result()
    return winner, pending  # keep the pending one as the spare

This costs a little extra (one more connection per run) and removes most of the tail.

Note what I didn't parallelize: the main requests themselves. In my case the second request needs a token returned by the first, so they're inherently sequential. Racing works on independent steps, not on a dependency chain.

Fix 4: Don't let shutdown eat your budget

The last 5–7 seconds were the process waiting for background tasks to finish before exiting. I capped that wait at about 1 second and made closing old sessions happen in the background. Exit now takes around 0.3 s.

It sounds trivial, but for short tools, teardown can be a surprisingly large share of total time.

Results

Before After (10 test runs)
Typical run a few seconds 6.5 s (p50)
Slow runs up to 40–44 s 8.0 s (p95)
Average cost per run $0.00035 $0.00011

Typical runs were already fine before. The tail collapsed, and that's what matters for anything with a deadline. Cost also dropped by about two-thirds, because runs that finish faster use less compute.

Takeaways

  • Measure where the time goes before tuning backoff. My problem was stalls, not rate limits.
  • Set total-time budgets, not just read timeouts.
  • On 429, switch identity if you can; back off only when you can't.
  • Keep a warm spare so switching is instant.
  • Race independent slow steps; keep dependent steps sequential.
  • Cap shutdown time.
  • Fail fast and clearly when the budget is gone.

This came out of building agent-friendly data tools (the one above is a Google Trends API on Apify). If you've found other ways to tame tail latency against rate-limited APIs, I'd love to hear them.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.