Five Myths About Retrying an LLM Call, and the Fault-Injection Drill That Tests Them

Your retry loop is maybe eight lines. Why does it double your bill and duplicate side effects?

I keep seeing the same pattern in reviews. The retry logic looks correct, and everyone stops reading there.

That is the problem. Retries are where cost, correctness, and latency quietly disagree.

Below are five claims I hear often, the evidence that breaks them, and a small drill you can run in an afternoon.

Label: the code here is a template, not a benchmark. I have not run it against your stack. Run the drill yourself and trust your own numbers, not mine.

Myth 1: "A retry is basically free"

A retry is not a repeat. It is a second request with the same input.

You pay to send the prompt again, on every attempt. Long system prompts make this expensive fast.

Corrected model: retry cost scales with input_tokens × attempts × concurrent_callers. Measure attempts, not errors.

Ask yourself: does your dashboard count attempts, or only final outcomes? Most count outcomes.

Myth 2: "HTTP 200 means the model finished"

A 200 means the transport succeeded. It says nothing about the answer.

Truncated completions, empty content, and stream cuts all arrive as 200. Your retry loop never fires.

Corrected model: treat finish_reason, content length, and schema validity as part of the success check. A response is not a result.

How many of your retries are actually triggered by business-level incompleteness? Usually zero.

Myth 3: "Exponential backoff with jitter is enough"

Backoff controls timing. It does not control volume.

If every caller retries four times, a partial outage becomes a full one. Jitter spreads the pain, not the load.

Corrected model: add a retry budget per request and per minute. Cap total attempts across the fleet.

Rules that survive contact with production:

1. Retry 429 and 5xx only. Never retry 400 or 401.
2. Honor Retry-After when present. It beats your formula.
3. Stop after a fixed attempt count. Then fail over or fail closed.
4. Emit one metric per attempt, tagged by error class.

Myth 4: "Retrying is idempotent because the call is stateless"

The model call is stateless. Your handler is not.

If the tool call wrote a ticket, charged a card, or sent mail, a retry repeats the side effect.

Corrected model: retries are only safe behind an idempotency key or a commit barrier. The barrier is a local check: did the side effect already happen?

Rhetorical but useful: what would your retry loop do to a payment webhook twice in a row?

Myth 5: "You can resume a stream where it stopped"

You cannot resume mid-token. The connection is gone, and so is the partial state.

What you can do is checkpoint at message boundaries. Re-request with the completed prefix as context.

Corrected model: streams are not resumable transports. They are progress signals. Persist what you received, then decide.

The drill: inject faults, count attempts

This is the part that changes minds. Ten minutes of setup, one table of evidence.

The stub server uses only the standard library. No dependencies, no keys.

# stub_server.py - deterministic fault injection, Python 3.9+
import json, threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

FAULTS = ["ok", "truncate", "500", "429", "slow"]
state = {"n": 0}
lock = threading.Lock()

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        with lock:
            fault = FAULTS[state["n"] % len(FAULTS)]
            state["n"] += 1
        if fault == "500":
            self.send_response(500); self.end_headers()
            self.wfile.write(b'{"error":"boom"}'); return
        if fault == "429":
            self.send_response(429)
            self.send_header("Retry-After", "1"); self.end_headers(); return
        body = {"choices": [{
            "message": {"role": "assistant", "content": "ok"},
            "finish_reason": "length" if fault == "truncate" else "stop",
        }]}
        raw = json.dumps(body).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)
    def log_message(self, *args):
        pass

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8787), H).serve_forever()

Now the client. It records every attempt in a ledger.

# drill.py - counts attempts and re-sent input per call
import json, random, time, urllib.error, urllib.request

URL = "http://127.0.0.1:8787/v1/chat"
PROMPT = "summarise this ticket in one line"  # swap for a real payload

def classify(code, body):
    if code == 429:
        return "throttle"
    if code >= 500 or code == 0:
        return "transient"
    if code == 200 and body["choices"][0]["finish_reason"] == "length":
        return "truncated"
    return "ok"

def call_once():
    req = urllib.request.Request(
        URL,
        data=json.dumps({"prompt": PROMPT}).encode(),
        headers={"Content-Type": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=5) as r:
            return r.status, json.loads(r.read())
    except urllib.error.HTTPError as e:
        return e.code, {}
    except Exception:
        return 0, {}

def drill(cases=10, budget=4):
    ledger = []
    for _ in range(cases):
        input_tokens = len(PROMPT.split())  # crude proxy, use real usage
        for attempt in range(1, budget + 1):
            code, body = call_once()
            cls = classify(code, body)
            ledger.append({"attempt": attempt, "code": code,
                           "class": cls, "input_resent": input_tokens})
            if cls in ("ok", "truncated"):
                break
            delay = min(2 ** attempt * 0.1, 1.0)
            time.sleep(delay * random.uniform(0.5, 1.5))
    return ledger

if __name__ == "__main__":
    rows = drill()
    print(json.dumps(rows, indent=2))
    print("attempts:", len(rows))
    print("input tokens re-sent:", sum(r["input_resent"] for r in rows))

Run it:

python stub_server.py &
python drill.py | tail -n 2

Then read the ledger. Do you see attempts your old loop would have skipped? Do you see truncated responses marked as success?

The decision table I actually use

Where MonkeyCode fits in this drill

A fault drill needs two things: a stub server and a second route to fail over to.

MonkeyCode's free model access gives me that second route without a card. The operator-supplied free server option can host the stub, so the drill does not touch production networking.

I would not run the whole harness blindly there. Check the current quotas and limits yourself first.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it as the failover target you test against, not as the reason the drill exists. The drill works with any second endpoint.

Limitations, and who should skip this

• The stub returns fixed shapes. It tests your control flow, not model quality.
• Token counts in the script are a proxy. Swap in real usage numbers.
• Retry budgets need tuning per route. One global number hides the problem.

Skip this if you have a single caller and no side effects. You have bigger wins elsewhere.

Skip it too if your platform already enforces retry budgets and idempotency keys. Audit those settings instead.

What changes after the drill

You stop treating retries as error handling. You start treating them as load, cost, and duplicate-work decisions.

That reframing is the whole payoff. The eight-line loop stays short, but now it has a reason for every line.

What would your ledger show tomorrow?