Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 11 min read

Your Agent Retried the Tool Call and Charged the Card Twice: A Zero-Cost Resilience Layer in Python

Last month one of my automations sent the same customer email three times in four minutes. The tool call had timed out, the retry layer did exactly what it was told, and the customer got a copy for each attempt. Nothing

Last month one of my automations sent the same customer email three times in four minutes. The tool call had timed out, the retry layer did exactly what it was told, and the customer got a copy for each attempt. Nothing in the logs looked like an error β€” every attempt was logged as an honest timeout, and every attempt had actually done its job.

That is the part of agent reliability nobody warns you about. Retries are the first thing you add and the last thing you reason about. They fix the failure you can see (a dropped request) and create the one you cannot (a side effect that happened twice, in a system that has no idea it happened once).

This post is a small resilience layer for tool calls in pure standard-library Python: exponential backoff with jitter, a circuit breaker, an idempotency key that survives retries, and a dead-letter queue that parks work instead of dropping it. Every number below comes from a measured run on a virtual clock, so you can reproduce all of it on a laptop with no API key and no network.

Two bugs that "just add retries" creates

The lost response. The request reached the service, the side effect committed, and the response was lost on the way back. Your code sees a timeout. The truth is a success plus a mystery. Retrying is now the most dangerous thing you can do, because the operation already happened β€” a second charge, a second email, a second PR comment.

The herd. A dependency starts failing for everyone at once, and every worker waits the same 2 seconds before trying again. They come back in lockstep, hammer the service at exactly the moment it is weakest, and turn a short wobble into a long outage. Fixed backoff is a synchronisation mechanism you did not mean to build.

Both are addressable with patterns that are older than LLMs β€” but only if the retry layer is designed as one unit with the side effect, not bolted on around it.

The four moving parts

  1. RetryPolicy β€” exponential delay, capped, with jitter, so retries spread out instead of arriving together.
  2. CircuitBreaker β€” closed β†’ open β†’ half-open. When the dependency is down, fail fast instead of queueing.
  3. IdempotencyStore β€” one key per logical operation, reused by every attempt, so a retry can never repeat the effect.
  4. DeadLetterQueue β€” a JSONL parking lot for operations that exhausted their attempts, with a replay() to drain it once the dependency is healthy again.

A ToolExecutor wires all four together and emits one structured JSON event per attempt, so a retry storm is visible in a log rather than inferred from a billing surprise.

1. Backoff that does not synchronise

class Jitter(str, Enum):
    NONE = "none"
    FULL = "full"      # uniform(0, cap)      - best at spreading a herd
    EQUAL = "equal"    # cap/2 + uniform(0, cap/2)

@dataclass
class RetryPolicy:
    max_attempts: int = 4
    base_delay: float = 0.2
    factor: float = 2.0
    max_delay: float = 8.0
    jitter: Jitter = Jitter.FULL
    seed: Optional[int] = None

    def raw_delay(self, attempt: int) -> float:
        return min(self.max_delay, self.base_delay * (self.factor ** (attempt - 1)))

    def delay(self, attempt: int) -> float:
        cap = self.raw_delay(attempt)
        if self.jitter is Jitter.NONE:
            return cap
        if self.jitter is Jitter.FULL:
            return self._rng.uniform(0, cap)
        return cap / 2 + self._rng.uniform(0, cap / 2)

The cap matters more than the curve. Without max_delay, attempt 6 at base=0.2, factor=2 waits 6.4s and attempt 8 waits 25.6s β€” inside a request a human is waiting on. I keep max_delay at 3–8s for interactive tools and let the dead-letter queue absorb the rest.

seed exists for the same reason the clock is injectable: a test that cannot reproduce the delay distribution is a test you will delete the second it flakes.

Here is what jitter buys, measured on 50 workers that all fail their first attempt and retry once with a 400ms base delay:

jitter distinct retry times retries in the busiest 25ms buckets used
none 1 50 1
equal 50 10 8
full 50 6 16

Without jitter, all 50 arrive in the same 25ms window. With full jitter the peak is 6 and the load is spread over 16 windows. That is the difference between a dependency recovering and a dependency being retried to death.

Full jitter also wins on the tail without changing the success rate at all. Five hundred operations against a downstream failing 35% of the time per attempt, 120ms latency, up to 4 attempts:

policy success failed ops downstream calls p50 p95 max
no retry 63.2% 184 500 120ms 120ms 120ms
retry, no jitter 98.2% 9 771 120ms 960ms 1,880ms
retry, equal jitter 98.2% 9 771 120ms 937ms 1,729ms
retry, full jitter 98.2% 9 771 120ms 840ms 1,577ms

Retries take success from 63% to 98% at a cost of 54% more downstream calls. Jitter does not touch success or call volume β€” it only shortens the tail (p95 βˆ’12%, max βˆ’16%), which is exactly what you want from it: better worst case for the same budget.

2. A breaker, so an outage does not become a backlog

A retry policy alone is a machine for making a bad situation worse: every queued operation keeps calling a service that is already refusing connections, and each one holds a worker for the full backoff window.

def allow(self) -> bool:
    if self.state is BreakerState.CLOSED:
        return True
    if self.state is BreakerState.OPEN:
        if self.clock.monotonic() - self.opened_at >= self.cooldown:
            self._set_state(BreakerState.HALF_OPEN)
            self.half_open_calls = 0
        else:
            self.rejected += 1
            return False
    if self.half_open_calls >= self.half_open_max_calls:
        self.rejected += 1
        return False
    self.half_open_calls += 1
    return True

Three details that are easy to get wrong:

  • Count consecutive failures, and reset on any success. A single stray 500 during a healthy hour should never contribute to opening the breaker.
  • Half-open needs a probe budget. After cooldown, let a small fixed number of calls through (I use 2). A failed probe reopens immediately; successes close it. Without a cap, "half-open" becomes "fully open with extra steps" the moment traffic is high.
  • One breaker per tool, not per process. My charge_card tool and my search tool have nothing in common, and a search outage must not stop payments.

3. One key per operation, not per attempt

This is the part that fixes the double email, and it is a design decision about identity, not about retrying.

def make_idempotency_key(tool: str, payload: Any) -> str:
    blob = json.dumps({"tool": tool, "payload": payload}, sort_keys=True, default=str)
    return hashlib.sha256(blob.encode()).hexdigest()[:32]

The key is derived from the tool name and the payload, so the same logical operation gets the same key on attempt 1 and attempt 4. The executor reserves it before the first call, hands the same key to every attempt, and only marks it complete on success. The downstream β€” or a local store, if you own both sides β€” records the key atomically with the effect. A retry arriving after a committed-but-lost response is then recognised as a duplicate and replays the stored answer instead of repeating the write.

Measured on 300 charge operations where 30% of responses are lost after the effect commits:

key strategy effects written extra effects duplicate attempts blocked
one key per operation 300 0 125
new key per attempt 425 +125 (+41.7%) 0

Same code, same retry count, same 298 successes β€” the only difference is whether the key identifies the operation or the attempt. A uuid4() generated inside the retry loop is the single most expensive line in a retry layer, because it turns every retry into a new piece of work.

Two corollaries I learned the hard way:

  • The key has to travel with the request. If the downstream does not accept an idempotency key and you do not control it, then retrying a non-idempotent verb is not safe at any backoff setting. Retry only reads, or check state afterwards.
  • Never put the attempt number, timestamp, or a random nonce in the payload before hashing. It looks like harmless metadata; it silently makes the key per-attempt and reintroduces the bug.

4. Park it, do not drop it

An operation that exhausted its attempts is not garbage. It is work a person asked for, and the correct outcome is "delayed", not "lost".

def _dead_letter(self, key, payload, attempts, reason):
    self.dlq.add({"tool": self.tool, "key": key, "payload": payload,
                  "attempts": attempts, "reason": reason})

def replay(executor: ToolExecutor, entries: list[dict]) -> int:
    recovered = 0
    for entry in entries:
        res = executor.call(payload=entry["payload"], key=entry["key"])
        if res.ok:
            recovered += 1
    return recovered

Replay passes the original key back in, which is what makes replay safe: if the operation actually landed before it was parked, the sink recognises the key and no second effect occurs. A dead-letter queue that replays with fresh keys is a duplicate generator with a nice name.

The executor loop

def call(self, payload=None, key=None, new_key_per_attempt=False) -> CallResult:
    self.calls += 1
    base_key = key or make_idempotency_key(self.tool, payload)

    cached = self.store.get(base_key)
    if cached is not None:                       # same operation, already done
        self.dedup_hits += 1
        return CallResult(key=base_key, ok=True, result=cached, deduplicated=True)

    self.store.claim(base_key)
    last_error = None
    for attempt in range(1, self.policy.max_attempts + 1):
        if attempt > 1:
            self.clock.sleep(self.policy.delay(attempt - 1))
        if not self.breaker.allow():             # fail fast, park the work
            self.fast_failures += 1
            self._dead_letter(base_key, payload, attempt, "circuit_open")
            return CallResult(key=base_key, ok=False, fast_failed=True,
                              error="circuit_open", attempts=attempt)
        try:
            result = self.fn(key=base_key, attempt=attempt)
        except TerminalError as exc:             # 400s, auth, bad input: no retry
            self.breaker.record_failure()
            self._dead_letter(base_key, payload, attempt, f"terminal:{exc}")
            return CallResult(key=base_key, ok=False, error=f"terminal:{exc}")
        except RetryableError as exc:            # 429, 5xx, timeouts: retry
            last_error = str(exc)
            self.breaker.record_failure()
            continue
        else:
            self.breaker.record_success()
            self.store.complete(base_key, result)
            return CallResult(key=base_key, ok=True, result=result, attempts=attempt)
    self._dead_letter(base_key, payload, self.policy.max_attempts, f"exhausted:{last_error}")
    return CallResult(key=base_key, ok=False, error=f"exhausted:{last_error}")

The split between RetryableError and TerminalError is the highest-leverage line in the file. Retrying a 400 six times costs seconds and returns six identical 400s. Classify at the adapter β€” the one place that sees the HTTP status β€” not at the call site, where every tool would have to remember.

Measured end to end: a 30-second outage

Two hundred operations, one issued every 250ms, against a dependency that is dead for the first 30 seconds. Same policy in both runs, up to 4 attempts, full jitter, 100ms per call.

configuration succeeded live fast-failed parked downstream calls mean wait/op max wait/op
retry only 173 0 27 333 250ms 1,620ms
with breaker 72 127 128 229 60ms 1,460ms

The breaker run is not "better" at succeeding during the outage β€” it gives up locally and parks the work. Its value is the wait time: 60ms average instead of 250ms, and 31% fewer calls into a service that is refusing every one of them. Every parked operation was replayed afterwards and 200/200 were accounted for (72 live + 128 recovered), with the breaker walking open at 1.6s β†’ half_open at 16.8s β†’ open again at 16.9s β†’ half_open at 32.0s β†’ closed at 32.4s as the dependency came back at 30s.

That transition log is the second thing the breaker buys you: a timeline of when the dependency was down, produced without a monitoring stack.

Testable because time is injectable

Nothing above is testable if the tests actually sleep. Every component takes a clock, and the fake one advances virtual time:

class FakeClock(Clock):
    def __init__(self):
        self.now = 0.0
        self.sleeps = []
    def monotonic(self) -> float:
        return self.now
    def sleep(self, seconds: float) -> None:
        if seconds > 0:
            self.sleeps.append(seconds)
            self.now += seconds

The suite is 23 unittest cases and runs in 4 milliseconds: backoff curve and its cap, jitter bounds under 50 draws, a breaker that opens, half-opens, rejects the third probe, and reopens on a failed probe, a key that ignores payload ordering, an expired attempt that lands exactly one effect, an open breaker that never touches the downstream, and a full outage where every parked operation comes back through replay. All four experiments in this post use the same virtual clock, so the numbers above are reproducible with python3 experiments.py.

What I would do differently

  • Give the whole call a deadline, not just an attempt budget. Four attempts with growing backoff can outlive the caller. Stop retrying when the remaining budget is smaller than the next delay.
  • Make the dead-letter queue replay idempotent by construction β€” original keys, one at a time, with a cap. A bulk replay after an outage is a herd of its own.
  • Log the outcome, not the error. One JSON line per attempt with tool, key, attempt, latency_ms, breaker, gave_up turns "why was this slow on Tuesday" into a jq query.
  • Do not retry the model. If the agent's plan was wrong, a second call with the same prompt produces the same wrong plan more expensively. Retry the tool, then re-plan.
  • Treat a retried tool call as a new transaction only if you can name its identity. No key, no retry.

The pattern is not new β€” it is the same one payment APIs have used for a decade. What changed is that now a language model decides when to call the tool, dozens of times per task, without knowing whether the last call landed. The layer that knows is the one you wrote around it.

Run it yourself

resilience.py          # policy, breaker, idempotency, DLQ, executor (~310 code lines)
experiments.py         # the four measured experiments (virtual clock, no network)
tests/test_resilience.py  # 23 unittest cases, ~4ms
reports/measurements.json # every number in this post
python3 -m unittest discover -s tests
python3 experiments.py

Standard library only. No key, no account, no bill.

Further Resources

The full resilience module, the experiment harness, the test suite, and the JSONL event schema I use in production are collected in my workflow pack: The Complete 2026 AI Side-Hustle Blueprint: Zero-Cost Prompts & Workflows. Everything in this post runs without it.

Note: this post contains one link to my own workflow pack. The code above is free, standalone, and does not require it.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.