We asked dev.to to break our AI agent design. The community rebuilt it

An agent is sending 100 emails. The pod dies after the 40th.

Replay the run, and 40 customers get the email twice. Don't, and 60 never get it.

Our runtime already refuses the first option: a run that called a tool is never replayed without a human. But that leaves the 60 waiting. So a few weeks ago I published a proposal for idempotency keys at the tool level, and asked readers to tear it apart.

They did, in dozens of comments, from people who run payment flows, Azure Service Bus pipelines, paid search tools and posting APIs. One measured our central claim and split it in two. One found a bug in our code. One wrote a full RFC on our repo.

The ideas came from the thread. The decisions were ours to make, and not every suggestion made it. This is the design they produced: v2.

New here? The series so far: why our agents refuse to replay a run that already acted, why hashing the arguments breaks with LLMs, and what our agents' memory costs in tokens.

TL;DR

• The comments exposed two mistakes: a canonicalisation bug, and the fact that generated text has nothing stable to hash.
• Five decisions came out of it: a business key, a ledger with six states, uniqueness enforced by the database, an epoch only a human or a rule can bump, and a frozen approved plan.
• Four real cases shaped them: a one-shot token, a lagging audit log, two parallel tool calls, and a bug in our own dataset import.
• Some of it runs in production today. Most of it is open, as issues with four good first tasks.

1. The two mistakes the comments exposed

Mistake one: we blamed the LLM for a serialisation bug. One reader measured it. Five representations of the same tool call (same recipient, subject, amount and note), hashed with sha256 over json.dumps:

0.1 + 0.2 serialises as 0.30000000000000004. NFD and NFC are different bytes. A trailing newline is a different subject. That's a known bug class with a known fix, and it has nothing to do with LLMs.

Mistake two, the real one: generated text has nothing stable to hash. Four drafts of one intent gave four distinct hashes, correctly, because they are four different strings. No canonicalisation fixes that.

So the rule from the first post gets sharper. Generated content stays out of the key by construction, not by convention: the function that builds the key has no parameter for it.

CANON_VERSION = "v1"

def effect_key(event_id: str, tool: str, target: str,
               amount_minor: int | None = None, scope: str = "") -> str:
    # No parameter for the subject or the body: generated text can't get in.
    if amount_minor is not None and not isinstance(amount_minor, int):
        raise TypeError("amounts are integer minor units, never floats")
    parts = {"event": event_id, "tool": tool, "target": normalize_target(target),
             "amount": amount_minor, "scope": scope}
    blob = json.dumps(parts, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
    digest = hashlib.sha256(unicodedata.normalize("NFC", blob).encode()).hexdigest()
    return f"{CANON_VERSION}:{digest}"

Two property tests come with it: N representations of one call give one key, and N drafts of one intent give one key while all N bodies differ. And the v1: prefix answers a trap the same reader pointed out: if a dependency ever changes how it serialises floats or normalises text, every in-flight key changes silently and replays stop deduping. Versioning the canonicaliser turns that into a new key space instead of a silent failure.

2. The five decisions

Decision 1: the key is the business shape of the effect

The key is the event (or, for a scheduled agent, the occurrence the scheduler planned), the tool, the resolved target, the amount in integer minor units, and a scope. Three rules hold it together:

• Resolved, not typed. If the model writes "invoice 7" from context and gets it wrong on replay, that's generated content again. The tool looks the record up and keys on what it found.
• The effect, not the decision. If run one emailed Anna and run two decides to skip her, the effect for Anna is still done.
• Time from the trigger, never the clock. One reader proposed a time bucket. A wall-clock bucket breaks on the replay itself (crash at 23:59, replay at 00:01, new key, two emails), so we anchored it to the trigger instead.

Decision 2: a ledger with six states

"Done or not done" didn't survive real providers.

The guarantee: unknown is never treated as "didn't happen". A provider 500 leaves the key unknown, because the call may have landed. A target we can't find on the provider side is unknown, not revoked. And the row stores the full raw provider response, written before anything parses it.

Decision 3: the database decides, not the prompt

The pending row is committed in its own transaction before the call, against a unique index on the key. Two calls with the same key can't both send: the loser of the insert skips. It's the same trick our Microsoft Graph webhooks already use to reject re-deliveries.

CREATE TABLE effect_ledger (
  key           text    NOT NULL,  -- 'v1:' || sha256(canonical shape of the effect)
  epoch         integer NOT NULL DEFAULT 0,
  state         text    NOT NULL,  -- pending | done | unknown | failed | stranded | revoked
  run_id        text    NOT NULL,
  raw_response  bytea,             -- written before anything parses it
  receipt       text,
  superseded_by text,              -- the user or policy that bumped the epoch
  PRIMARY KEY (key, epoch)
);

Decision 4: an epoch the model can't touch

Sometimes the first email really landed and someone wants a second, better one. Same key, so the ledger can only skip. The answer, designed by a reader: an epoch in the key, bumped only when a done effect is deliberately superseded.

The guarantee: the bump never comes from the model. The conformance test fits in one line: could any token the model produced change whether the bump happens? If yes, the bump is invalid. Outside means a human action, or a deterministic policy registered before execution. A failed attempt never needs a bump: it already freed the key.

Decision 5: replay against a frozen, approved plan

Keys handle overlap between runs, not drops. So when a human approves a plan, it's frozen as plan rows, one (target, key) per recipient, before the first effect. And the approved text is frozen with it, so the remaining 60 people get the email a human approved, not a rewrite.

The guarantee changes the metric itself: not "did the run finish", but "did every approved target get exactly one effect". Our email-campaign skill already stops for explicit approval before sending, so the approved list is the natural set to freeze.

3. The cases that changed the design

A token you can never get back. A posting API returns 201 with a one-shot verification challenge in the body, to answer within five minutes, and never exposes it again. One truncated log line, and the object is created but can never complete. That case gave us the stranded state and the raw response stored before parsing.

An audit log that lags. Checking the provider at startup works better than any write-time ordering, but some providers record an effect minutes late. A miss right after a crash isn't proof, so every pending key gets a settle window.

Two tool calls in one turn. A model can return several tool calls at once, and a runtime may run them in parallel. Our skill tells the model to send one call per recipient and not to parallelize, but that's a prompt, not a guarantee. That case is why decision 3 lives in the database.

A bug in our own code. The v1 post showed our BI dataset import as an example of a content-derived key, including its scar: a soft-deleted row keeps the deterministic id, so the import falls back to a random one. A reader noticed that after a deletion, two replays make two datasets. It's still open, and it's task 4 in section 5.

Two more threads shaped the edges: a paid search tool that bills again on replay (key on the records, never the query; a record missing from a re-pull is unseen, not gone), and an Azure Service Bus pipeline that confirmed retention should follow the business event, not a clock.

4. What's implemented, and what isn't

Running in production today:

• A failed agent run that called any tool is refused for replay with a 409 listing those tools, unless a human forces it. An unrecorded tool history counts as unknown and is refused too.
• A run stuck in running is reconciled to error after six hours, with a single conditional write.
• Webhook re-deliveries are rejected by a unique index.
• Several data tools are read-only by construction: Oracle in a read-only transaction, MongoDB refusing write stages, Redshift validating its SQL before running it.
• Retries are bounded, a provider quota pauses the whole queue, and a run is capped at 25 model calls.
• A workflow that fails on a node can be resumed from that node, with everything before it replayed from an encrypted snapshot instead of re-executed. The same mechanism lets a workflow pause on a human approval and resume when someone answers.

Since the thread:

• A reader's regression test showed that our quota pause is runtime-wide: one 429 holds the whole queue, for every tenant and every key. They turned it into #509, with test scenarios, code references and acceptance criteria, linked to #466 for the replay part. A PR scoping the pause to the provider quota is in review (#513).
• On the next post of the series, a reader showed how an empty reply could slip through a quality check as "grounded". Two days later, Stefan Pantelich fixed it in our bench (#514). He has merged seven PRs into the repo since this series started.

Not built yet, and open: everything in section 2. Each piece is an issue:

• #466 RFC: tool-level idempotency keys and an effects ledger
• #473 Intent epoch, written by Alex Chen
• #476 Raw provider response and the stranded state, from ANP2 Network's case
• #477 Plan rows and the frozen approved payload, from Max Kaya and Stéfan
• #478 Write-once tools_executed, raised by Cenk KURTOĞLU
• #479 Table ownership moved to a migration role
• #480 Resume an interrupted workflow from its last completed node, raised by Richard Smith

5. Start here: four first tasks

You don't need to read the whole spec to help. These four stand on their own:

1. The canonicaliser and its two property tests. A pure function and two tests, with the v1: prefix. No database, no agent. Comment on #466 to take it.
2. The ledger table, with its unique index and the epoch column. The foundation everything else builds on. #466 and #473.
3. The write-once trigger on tools_executed. Small, testable, and it turns a convention into something the database enforces. #478.
4. The dataset import fallback. When a soft-deleted dataset still holds the deterministic id, the import falls back to a random one (src/apowerb/tools_store/portfolio/bi_datasets.py). Derive it from the deletion stamp instead, so two replays after a deletion make one dataset. Comment on #466 to take it.

Comment on the issue before you start, so we agree on the shape. The tech team reviews every PR, and every contributor is credited by name, the same way everyone in this post is.

The people who made it better

None of them had to. Most of us had never met. They brought scars from other stacks, measured instead of arguing, and sometimes disagreed, which made the design stronger.

Thanks to @qisuancloud, @maxbuildsdev, @anp2network, @skillselion, @kartik-nvjk, @manojkagitha, @alexcodebytes, @ascendra_ventures_92c21f2, Kevin Hall, Stéfan, and on the previous post, Cenk KURTOĞLU and Richard Smith.

• Source code: github.com/apowerb/apowerb
• Run it locally (Docker Compose) or on Kubernetes (Helm): github.com/apowerb/apowerb-hosting
• Documentation: docs.apowerb.com
• Try it without installing anything: apowerb.com

What's the failure case we still haven't covered? The last two comment sections wrote the best parts of this design. Let's see what this one writes.