Our moderation model answers in 0.3 s or in 19 s, nothing in between. Hedged requests took us from 15/30 to 26/30

We run a small AI help chat for websites, and every message in it passes a safety check before anything else happens. A visitor's question is classified by Llama Guard 3 (the 8B model, on Cloudflare Workers AI) before it reaches retrieval and the chat model, and the model's answer is classified again, together with the question, before it is shown and stored. Only an explicit "unsafe" verdict refuses the turn.

The chat streams its answers, so the check sits on the critical path. We gave it a budget of 3 seconds. Longer than that and the visitor is staring at a silent chat window wondering if it broke.

Then we measured the latency, on 2026-10-19, with 30 calls. It was bimodal in a way we hadn't seen from an inference API before: about half the calls came back in 0.1 to 0.3 seconds, and the rest took anywhere from 3 to 19 seconds. Almost nothing in between. A warm path and a cold path, presumably, though Workers AI doesn't say.

With that distribution, a single call against a 3-second timeout answered 15 times out of 30. A coin flip, which for a safety check is embarrassing: half of all turns went through unchecked purely because the classifier drew the slow path.

The fix is from a 2013 paper

Raising the timeout doesn't help. The slow path runs up to 19 seconds, so a timeout long enough to catch it makes the chat unusable, and any timeout short enough for the chat keeps losing the slow half.

What does help is noticing that the two paths are independent draws. If one call lands on the slow path with probability ~0.5, two calls both land on it with probability ~0.25. This is the "hedged request" from Dean and Barroso's The Tail at Scale: when a reply is slow, send the same request again and take whichever answer arrives first.

Ours starts a copy of the same classification every second while no call has answered, inside the same 3-second budget:

export const GUARD_TIMEOUT_MS = 3_000;
export const GUARD_HEDGE_MS = 1_000;

const call = () => ai.run(GUARD_MODEL, { messages, max_tokens: 20, temperature: 0 });
const copies = Array.from({ length: Math.ceil(GUARD_TIMEOUT_MS / GUARD_HEDGE_MS) - 1 }, (_, i) =>
  new Promise((resolve, reject) => {
    timers.push(setTimeout(() => call().then(resolve, reject), GUARD_HEDGE_MS * (i + 1)));
  }));
const out = await Promise.race([
  call(), ...copies,
  new Promise((_, reject) => {
    timers.push(setTimeout(() => reject(new Error(`timeout ${GUARD_TIMEOUT_MS} ms`)), GUARD_TIMEOUT_MS));
  }),
]);

So at most three calls race: the original, a copy at 1 s, a copy at 2 s. When the original draws the fast path (half the time), it answers in 0.3 s and no copy ever starts, because the timers get cleared in a finally. The copies only cost money on the turns that were about to fail anyway.

Same 30-call measurement, same 3-second budget, with hedging: 26 answers out of 30.

Two details that matter more than the race

Don't hedge errors. The first version of this raced everything, and that's wrong. A hedge is a bet that the same request will be fast the second time. That's true for a slow call, and false for a quota error or an outage, where the retry burns money to fail identically. So an error from any call ends the whole check at once; only silence gets hedged. In the code above that falls out of Promise.race naturally, since a rejection wins the race the same way a result does.

Decide what a missing verdict means before you ship. Hedging got us from 15 to 26 out of 30, not to 30. The remaining failures time out, and something has to happen with that turn. We fail open: the failure is logged and the turn continues unchecked, because for a help chat on a shop's website, a chat that goes mute whenever a classifier flakes is worse than one unchecked message. A moderation pipeline for user-generated content that stays up forever would reasonably make the opposite choice and fail closed. Either way, pick it deliberately; the default you don't choose is still a choice.

The measured numbers, for easy comparison, all on the same day with 30 calls each: fast path 0.1–0.3 s for about half the calls, slow path 3–19 s for the rest; one call inside 3 s: 15 of 30; hedged copies each second inside 3 s: 26 of 30.

This runs in production in Formgong's help chat, where the same check also writes the category of anything flagged into the message log. If you've dealt with bimodal inference latency on Workers AI or elsewhere, we'd like to hear what distribution you saw and what you did about it, and if you try the chat and something about it annoys you, tell us in the comments or at [email protected].