Per-customer rate limits and 429s in Express and FastAPI

Most rate limiting tutorials limit by IP address. That's the right tool for a login form, and the wrong one for an API that customers pay for. A customer's traffic can come from fifty servers behind fifty addresses, or from one office NAT shared with someone else's script. What you want to cap is the customer: "Acme gets 5 requests per second, however many keys or machines they use."

This post builds that in Express and in FastAPI with nothing but the framework, then covers the 429 response and the parts that tend to go wrong: ordering, retries and running more than one process.

A token bucket per customer

A token bucket is two numbers per customer: how many tokens are in the bucket, and when you last looked. The bucket holds at most capacity tokens and refills at refillPerSecond. Each request takes one token; an empty bucket means 429. A customer who's been quiet can send a burst of capacity requests, then continues at the refill rate.

You don't need a timer. Refill lazily when a request arrives:

// rate-limit.js
// One token bucket per customer. Refilled lazily when a request arrives, so no timers.
export function tokenBucket({ capacity, refillPerSecond, now = () => performance.now() / 1000 }) {
  const buckets = new Map(); // customerId -> { tokens, updatedAt }

  return function take(customerId) {
    const time = now();
    const bucket = buckets.get(customerId) ?? { tokens: capacity, updatedAt: time };
    bucket.tokens = Math.min(capacity, bucket.tokens + (time - bucket.updatedAt) * refillPerSecond);
    bucket.updatedAt = time;
    buckets.set(customerId, bucket);

    if (bucket.tokens < 1) {
      // Round up: a client that retries a moment too early just gets another 429
      return { allowed: false, retryAfter: Math.ceil((1 - bucket.tokens) / refillPerSecond) };
    }
    bucket.tokens -= 1;
    return { allowed: true, remaining: Math.floor(bucket.tokens) };
  };
}

Two details matter here. The clock is performance.now(), which is monotonic, so an NTP adjustment can't refill or drain every bucket at once. And retryAfter is rounded up, because telling a client to come back in 0 seconds produces a retry that fails again.

Express: authenticate first, then limit

// app.js
import express from "express";
import { tokenBucket } from "./rate-limit.js";

// Stand-in for your key table. Acme has two keys but one bucket.
const KEYS = new Map([
  ["key_acme_1", "acme"],
  ["key_acme_2", "acme"],
  ["key_globex", "globex"],
]);
const CAPACITY = 5;
const take = tokenBucket({ capacity: CAPACITY, refillPerSecond: 1 });

function authenticate(req, res, next) {
  const key = req.get("authorization")?.replace(/^Bearer\s+/i, "");
  const customerId = key && KEYS.get(key);
  if (!customerId) return res.status(401).json({ error: "invalid_api_key" });
  req.customerId = customerId;
  next();
}

function rateLimit(req, res, next) {
  const result = take(req.customerId);
  res.set("RateLimit-Policy", `"default";q=${CAPACITY};w=${CAPACITY}`);
  if (result.allowed) return next();

  res.set("Retry-After", String(result.retryAfter));
  res.set("RateLimit", `"default";r=0;t=${result.retryAfter}`);
  res.status(429).json({ error: "rate_limited", retry_after: result.retryAfter });
}

const app = express();
app.get("/health", (req, res) => res.send("ok")); // before the limiter: never limited
app.use("/v1", authenticate, rateLimit);
app.get("/v1/things", (req, res) => res.json({ customer: req.customerId }));

app.listen(3000);

Send six requests in a row, alternating Acme's two keys, and the sixth gets:

HTTP/1.1 429 Too Many Requests
RateLimit-Policy: "default";q=5;w=5
Retry-After: 1
RateLimit: "default";r=0;t=1

{"error":"rate_limited","retry_after":1}

Globex is unaffected, because it has its own bucket.

The order of authenticate and rateLimit is the whole point. Authentication runs first for two reasons:

• The bucket key has to be the customer, not the API key string. Key it on the key and a customer with five keys gets five times the limit. You only know the customer after authentication.
• Unknown keys should never reach the limiter. If they did, anyone could fill your Map with junk keys. Junk keys get a 401 instead. (If you're worried about key guessing, add a per-IP limit on failed authentications at the edge.)

Also note that Express middleware only applies to routes registered after it. /health above is open on purpose; a router mounted in some earlier file would be open by accident.

FastAPI: dependencies, and why they should be async

In FastAPI the natural place for this is a dependency chain: one dependency resolves the customer, the next one spends a token.

# main.py
import math
import time

from fastapi import Depends, FastAPI, Header, HTTPException

# Stand-in for your key table. Acme has two keys but one bucket.
KEYS = {"key_acme_1": "acme", "key_acme_2": "acme", "key_globex": "globex"}
CAPACITY = 5
REFILL_PER_SECOND = 1.0
buckets: dict[str, tuple[float, float]] = {}  # customer_id -> (tokens, updated_at)


def take(customer_id: str) -> tuple[bool, int]:
    now = time.monotonic()
    tokens, updated_at = buckets.get(customer_id, (CAPACITY, now))
    tokens = min(CAPACITY, tokens + (now - updated_at) * REFILL_PER_SECOND)
    if tokens < 1:
        buckets[customer_id] = (tokens, now)
        return False, math.ceil((1 - tokens) / REFILL_PER_SECOND)
    buckets[customer_id] = (tokens - 1, now)
    return True, 0


async def current_customer(authorization: str = Header(default="")) -> str:
    scheme, _, key = authorization.partition(" ")
    customer_id = KEYS.get(key) if scheme.lower() == "bearer" else None
    if customer_id is None:
        raise HTTPException(401, detail="invalid_api_key")
    return customer_id


# async def on purpose: a plain def dependency runs in a thread pool,
# and two threads could both spend the last token.
async def rate_limited(customer_id: str = Depends(current_customer)) -> str:
    allowed, retry_after = take(customer_id)
    if not allowed:
        raise HTTPException(
            429,
            detail="rate_limited",
            headers={
                "Retry-After": str(retry_after),
                "RateLimit-Policy": f'"default";q={CAPACITY};w={math.ceil(CAPACITY / REFILL_PER_SECOND)}',
                "RateLimit": f'"default";r=0;t={retry_after}',
            },
        )
    return customer_id


app = FastAPI()


@app.get("/v1/things")
async def things(customer_id: str = Depends(rate_limited)):
    return {"customer": customer_id}

Run it with uvicorn main:app and the sixth quick request gets a 429 with the same three headers, and a body of {"detail":"rate_limited"}. FastAPI wraps HTTPException details in detail; add an exception handler if you want the same body shape as the Express version.

Two FastAPI-specific notes:

• async def is doing real work here. FastAPI runs plain def dependencies in a thread pool. The read-modify-write in take() isn't atomic, so two threads can read the same bucket and both spend its last token. An async def dependency runs on the event loop, and take() has no await in it, so nothing else can run in the middle. If you need a sync dependency, put a threading.Lock around the bucket update.
• A dependency can't limit what it doesn't run on. Routes that don't declare Depends(rate_limited) aren't limited. If you'd rather cover everything under a prefix, put the dependency on the APIRouter (APIRouter(dependencies=[Depends(rate_limited)])). If you move the check into a middleware instead, remember that middlewares run before dependencies, so the customer has to be resolved in a middleware too.

What the 429 should carry

• Retry-After in seconds. It tells the client how many seconds to wait before retrying. Seconds don't depend on the client's clock being right, which an HTTP date does.
• RateLimit-Policy and RateLimit. These come from an IETF draft (draft-ietf-httpapi-ratelimit-headers). q is the bucket size and w its window in seconds; r is what's left and t is when to try again. It's still a draft and the field names have changed over the revisions, so treat them as a bonus and Retry-After as the one clients should rely on.
• A machine-readable error code in the body, so clients can tell a rate limit apart from other 429s.

That last point matters more than it looks. If your plans have monthly quotas, "you've used this month's allowance" is often also a 429, and its Retry-After can be days away. A client that sleeps on every Retry-After will park a job for a week; one that ignores it and retries with backoff will keep hammering an answer that won't change. A reasonable client rule:

const wait = Number(res.headers.get("retry-after") ?? "1");
if (wait > 60) throw new Error(`Not retrying: ${body.error}, retry after ${wait}s`);
await new Promise((resolve) => setTimeout(resolve, wait * 1000));

Put the same rule in your API docs, and give the two cases different error codes so people can tell them apart in their logs.

More than one process

Everything above lives in one process's memory. Put four instances behind a load balancer and a customer whose requests spread evenly gets about four times the limit. You have three choices:

1. Accept it. A per-process cap can be enough to curb runaway scripts, as long as you can live with a customer reaching 4× the configured limit. It won't enforce a per-customer limit you've promised in a contract.
2. Divide the limit by the number of instances, rounded up and never below 1. That keeps the total close to what you set without any shared state, but you need a reliable instance count, and a client whose keep-alive connection pins it to one instance gets only that instance's share.
3. Use a shared store such as Redis, with the bucket update in a Lua script so it's atomic. That's exact, at the cost of a network round trip per request and a decision about what happens when Redis is down.

Decide which one you're doing before autoscaling decides for you.

Test it with a fake clock

tokenBucket() takes now as a parameter so tests don't have to sleep. Pass a function that returns a number you control, then check the edges: exactly capacity requests pass, the next one fails with retryAfter: 1, half a token isn't enough, and a long pause refills the bucket to capacity and no further.

I work on UsageRail, which does the per-customer part of this as middleware for Express (and FastAPI, Hono, Fastify, Django and Next.js), with the limits set per plan in a dashboard instead of in code: https://usagerail.com/docs/express