Dev.to AI 🤖 Ai 👁 0 📖 26 min read

When an AI Agent Resumes, How Do You Prevent a Tool from Running Twice?

When an AI Agent Resumes, How Do You Prevent a Tool from Running Twice? An agent can successfully send a refund request, lose the response to a later model call, and then restart from an earlier checkpoint. If recovery

When an AI Agent Resumes, How Do You Prevent a Tool from Running Twice?

An agent can successfully send a refund request, lose the response to a later model call, and then restart from an earlier checkpoint. If recovery simply replays the tool call, the customer may receive a second refund even though the first call worked. The hard part is not “making the model remember.” It is preserving a durable identity for one business intent across retries, recording what the executor knows, and asking the external system to deduplicate the repeated request.

The practical answer is a small replay protocol: assign each intended side effect a stable operation ID; persist the intent before dispatch; pass that ID as the provider’s idempotency key when supported; persist the provider result before giving control back to the model; and resume by consulting the ledger rather than rediscovering completed work. If the provider cannot deduplicate and a timeout leaves the outcome unknown, do not guess. Reconcile against the provider or ask a person before retrying.

This article is for engineers building tool-using agents, workflow runtimes, and task orchestrators. It is not a claim that an agent framework can guarantee exactly-once execution across arbitrary networks. That guarantee is impossible to infer from a local timeout alone. The goal is narrower and useful: make duplicate effects difficult, make ambiguous effects visible, and make recovery a deliberate state transition instead of an accidental second action.

The decision in one minute

Use a stable operation ID for the business action, not just a transient model-call ID. Persist a record before dispatch, and persist a result after the provider confirms it. Replaying a completed operation should return the recorded result to the agent without calling the tool again. Replaying an operation whose result is unknown should enter reconciliation, not blind execution. For providers that support idempotency keys, send the same key and identical parameters on every retry of the same intent. For actions without provider-side deduplication, build a read-before-retry reconciliation path or require human resolution for irreversible effects.

There are two distinct identities. A model tool-call ID identifies a particular message-level invocation. A business operation ID identifies the intended action, such as “issue the refund for invoice 17, adjustment 2.” If a resumed model generates a new tool-call ID for the same intent, deduplication based only on the old call ID can fail. If the key is based only on a customer or invoice, two legitimate separate refunds may collapse into one. The key must represent exactly one intended effect.

What the sources establish—and what they do not

The primary operational evidence is not a prevalence statistic. It is a concrete class of failure reported by users. In an open Agno issue, the reporter describes a successful tool call followed by a later model error; a retry of the entire run then repeats the already-completed tool. This is an issue report and reproduction, not a representative survey of frameworks or production incidents. It is useful because it exposes the recovery question that a happy-path demo does not answer.

Research points to a similar boundary problem. The preprint “When Tool Calls Succeed but Workflows Fail: Anomalies at the Agent-Tool Boundary” studies failures where the agent runtime and independently supplied tools do not share atomic transaction semantics. Its important caution is that a tool call’s success/failure signal may be too coarse to establish whether an external effect occurred, whether it can be compensated, or whether repeating it is safe. This is a research result about the studied systems and failure model, not a universal measurement of every agent stack.

The vendor documentation shows that idempotency is a system contract, not a prompt instruction. Stripe’s idempotent-request documentation describes client-supplied keys, stored results, parameter comparison, and retention behavior. The details matter: Stripe notes that keys may be pruned after at least 24 hours, and reuse after pruning can initiate a new request. A team must therefore align its own retry window with the provider’s retention window; “we used a key once” is not an eternal guarantee.

For organizational risk work, NIST’s AI Risk Management Framework resource links to a Playbook that groups suggested actions under Govern, Map, Measure, and Manage, while explicitly saying that its suggestions are voluntary and are not a checklist to follow wholesale. That is a useful management lens for assigning ownership, documenting the tool boundary, measuring unresolved outcomes, and managing escalation. It does not prescribe an idempotency implementation or prove a particular agent architecture safe; those decisions still need provider-specific evidence and tests.

Limits and retry behavior are also stack-specific. Microsoft’s function-tool documentation describes maximum iterations, function calls, and duration as controls for automatic invocation, and notes that limits are best effort and checked after a requested batch. These limits prevent unbounded work but do not, on their own, stop a completed side effect from being repeated during recovery. A different control is needed for replay safety.

First classify the retry

“Retry” often refers to at least four different events. A transient provider error before dispatch may be safe to retry. A connection timeout after dispatch is ambiguous: the provider may have committed the action even though the caller saw no response. A model failure after a successful tool response is not a tool failure at all, yet a whole-run retry can replay the tool. A workflow resume from a checkpoint may re-execute the last incomplete node, including its side effects. Treating these cases as one generic exception path is the root of many duplicate-action bugs.

Before deciding whether to retry, record where the failure occurred. Was a request constructed? Was it dispatched? Did the provider return a response? Was that response durably stored? Did the model receive it? Was a later stage the actual failure? The answer should come from runtime events or a ledger, not from the wording of an exception. “The model call failed” says nothing about whether the preceding payment, email, or database write succeeded.

The safe default is conservative. If a read-only query failed before returning data, another identical query is usually acceptable. If a non-idempotent action could have crossed the network boundary, the default state is unknown, not failed. That distinction changes the operator’s next step: failed may be retried under policy; unknown must be reconciled or escalated.

The four-state action ledger

A useful minimal ledger has four states: prepared, in_flight, succeeded, and unknown. prepared means the runtime accepted a specific business intent and assigned its stable operation ID. in_flight means dispatch may have begun. succeeded means a provider result was durably captured. unknown means the executor cannot prove either success or failure. A separate failed state is appropriate only when the system knows the effect did not occur—for example, local validation rejected the request before dispatch or the provider returned a definitive non-committing response.

Each record should contain the run ID, operation ID, tool name and version, normalized parameter hash, provider, status, attempt count, creation and update times, provider request ID, result reference, and a redacted error classification. Keep secrets and unnecessary personal data out of this ledger. The record is an operational index, not a second copy of the whole customer database. Store enough to detect a changed request and find the authoritative result.

The parameter hash is a guardrail. If a retry presents the same operation ID with materially different parameters, do not silently reuse the prior result or issue a new effect. Return a conflict and require a new operation ID after an explicit decision. This catches accidental key reuse, unstable model arguments, and bugs where a replay changes amount, recipient, or destination.

A replay protocol that preserves intent

The runtime should perform a deterministic sequence around every side-effecting tool. First, validate the requested action and determine whether it is permitted. Second, map the call to a stable business operation ID. Third, atomically insert or load the ledger row. Fourth, if the row is already succeeded, return the stored result without dispatch. Fifth, if the row is unknown or in_flight, run reconciliation or stop for review. Sixth, only a new prepared operation may be claimed for dispatch. Finally, persist the provider outcome before returning it to the model.

This is not an atomic transaction with the provider. The local store and remote API are separate systems. There is always a crash window between the remote side effect and the local success write. A provider idempotency key closes much of that gap by letting the receiver recognize a repeated request. If the receiver does not support deduplication, a reconciliation query can sometimes identify the effect after the crash. If neither is available, the system cannot honestly promise that an automatic retry is safe.

The action identity should be stable across a recovery of the same user intent, but different for a new intent. A reasonable derivation can include the durable workflow ID, a business object key, an explicit action sequence or approved step ID, and a versioned action kind. Avoid hashing the entire prompt: wording changes should not turn the same approved action into a new one, and identical prompts can represent separate legitimate actions. Generate the ID at the workflow boundary and persist it; do not ask the language model to invent uniqueness.

flowchart TD
  A[Validated business intent] --> B[Create and persist operation ID]
  B --> C{Ledger lookup}
  C -->|Succeeded| D[Reuse recorded result]
  C -->|Unknown or in flight| E[Reconcile provider status]
  E -->|Confirmed committed| F[Persist result and resume]
  E -->|Confirmed not committed| G[Retry with same key if safe]
  E -->|Still ambiguous| H[Stop and escalate]
  C -->|New prepared action| I[Claim atomically]
  I --> J[Dispatch with provider idempotency key]
  J --> K{Outcome}
  K -->|Confirmed| L[Persist success before model resumes]
  K -->|Ambiguous timeout| M[Mark unknown]
  L --> D
  M --> E

Ordering and the crash window

Persisting prepared before dispatch is necessary because it proves the runtime had accepted the intent. Persisting in_flight before the network call gives recovery a conservative signal that dispatch may have happened. Persisting succeeded only after the remote result is captured makes the result reusable. If the process crashes after the provider commits but before the local success write, recovery sees in_flight; it must not conclude “nothing happened.” It should query the provider using its request ID, idempotency key, or business reference.

Some systems may write the response body to a durable inbox before updating the ledger. That can improve recovery if the process dies between those writes. The important invariant is that a confirmed result is stored before the agent is allowed to progress into another model turn that might fail and trigger replay. If the result is only in memory, a later exception can discard the only evidence of success.

For database-only tools, the state change and ledger update may be committed in one database transaction. That is stronger because the two writes share a transaction boundary. For a payment processor, email provider, or SaaS API, the local database and remote effect do not share one transaction. Do not describe a local transaction as end-to-end exactly-once semantics; it only protects the local part.

Keys: too broad, too narrow, or unstable

An idempotency key that is too broad suppresses legitimate work. A key such as customer-42 would make every later action for that customer appear to be the same action. A key that is too narrow changes for every attempt and therefore fails to deduplicate. A random key generated during each retry has the same problem. The key must be created once per business intent and survive process restarts, queue redelivery, and model regeneration.

Include a namespace and version in the key so one operation type cannot collide with another: for example, workflow-81:invoice-17:refund-2:v1. Use an opaque stable identifier if exposing business identifiers to the provider would be inappropriate. Persist the mapping from that identifier to the business intent locally. The key does not need to reveal the user’s natural-language request.

Keep request parameters stable for a given key. Stripe’s documentation explicitly describes parameter comparison for reuse of a key. Other providers have their own semantics, retention periods, and error behavior. Read the relevant documentation rather than assuming every “idempotency key” works identically. If a retry follows a material request change, mint a new operation after human or policy approval; do not mutate the meaning of an existing key.

An executable local demonstration

The following Python example simulates a provider that commits a refund but loses the first response. The SQLite provider table and caller ledger make the behavior visible across a process restart. The same key returns the original provider result, while a different key produces a distinct refund. This is a local teaching example, not a drop-in payment integration; a real provider’s documented key scope and retention rules govern production behavior.

import hashlib
import json
import sqlite3

DB = "agent-replay-demo.sqlite3"

def connect():
    db = sqlite3.connect(DB)
    db.execute("PRAGMA journal_mode=WAL")
    return db

def provider_refund(key, invoice, cents, lose_first_response=False):
    """Fake remote provider with durable idempotency semantics."""
    with connect() as db:
        row = db.execute(
            "SELECT response FROM provider_keys WHERE idem_key = ?", (key,)
        ).fetchone()
        if row:
            saved = json.loads(row[0])
            if saved["invoice"] != invoice or saved["cents"] != cents:
                raise ValueError("same idempotency key, different parameters")
            return saved
        response = {
            "provider_refund_id": "rf_" + hashlib.sha256(key.encode()).hexdigest()[:12],
            "invoice": invoice,
            "cents": cents,
        }
        db.execute("CREATE TABLE IF NOT EXISTS provider_keys "
                   "(idem_key TEXT PRIMARY KEY, response TEXT NOT NULL)")
        db.execute("INSERT INTO provider_keys VALUES (?, ?)",
                   (key, json.dumps(response)))
        if lose_first_response:
            raise TimeoutError("simulated lost response after provider commit")
        return response

def execute_once(operation_id, invoice, cents, lose_response=False):
    params = json.dumps({"invoice": invoice, "cents": cents}, sort_keys=True)
    digest = hashlib.sha256(params.encode()).hexdigest()
    with connect() as db:
        db.execute("CREATE TABLE IF NOT EXISTS action_ledger "
                   "(operation_id TEXT PRIMARY KEY, params_hash TEXT, "
                   "status TEXT, result TEXT)")
        row = db.execute(
            "SELECT params_hash, status, result FROM action_ledger "
            "WHERE operation_id = ?", (operation_id,)
        ).fetchone()
        if row and row[0] != digest:
            raise ValueError("operation ID reused with changed parameters")
        if row and row[1] == "succeeded":
            return json.loads(row[2])
        if not row:
            db.execute("INSERT INTO action_ledger VALUES (?, ?, 'prepared', NULL)",
                       (operation_id, digest))
        db.execute("UPDATE action_ledger SET status='in_flight' "
                   "WHERE operation_id=?", (operation_id,))
    try:
        result = provider_refund(operation_id, invoice, cents, lose_response)
    except TimeoutError:
        # Do not mark failed: the remote effect may have committed.
        raise
    with connect() as db:
        db.execute("UPDATE action_ledger SET status='succeeded', result=? "
                   "WHERE operation_id=?", (json.dumps(result), operation_id))
    return result

if __name__ == "__main__":
    key = "workflow-81:invoice-17:refund-2:v1"
    try:
        execute_once(key, "invoice-17", 2500, lose_response=True)
    except TimeoutError as error:
        print("first attempt is ambiguous:", error)
    recovered = execute_once(key, "invoice-17", 2500)
    replayed = execute_once(key, "invoice-17", 2500)
    print("recovered:", recovered)
    print("same result on replay:", recovered == replayed)

Run it in an empty directory with python demo.py; it creates one SQLite file. The provider simulation stores its idempotency result before raising the timeout, intentionally modeling “the receiver committed, the response was lost.” On the next run the same key returns that original result. The caller-side ledger in this small sample is incomplete for a production system: after the timeout its row remains in_flight, so a mature implementation would explicitly query the provider or repeat the request with the same key, then record the resolved result. The fake provider makes the repeated request safe; a real one must document that guarantee.

Why a plain “done” flag is not enough

A boolean done=true written before dispatch can lose an action: the process may crash after writing the flag but before sending the request. A done=true written after dispatch can still duplicate an action: the provider may commit, then the caller may crash before writing the flag. The state machine needs an explicit ambiguous phase, and the remote system needs a way to resolve it. The ledger cannot turn an uncertain network outcome into certainty by relabeling it.

Likewise, storing a tool result only in the conversation transcript is not always durable enough. The transcript may be discarded, compacted, or absent from the checkpoint restored by the orchestrator. Tool messages can be useful evidence, but the runtime should tie the outcome to an operation ID in an authoritative execution ledger. If history says a payment result exists and the ledger says no such operation was recorded, that inconsistency itself should stop automatic replay.

There is also a concurrency race. Two workers may resume the same run simultaneously. Both can read prepared and dispatch before either writes in_flight. Use a database uniqueness constraint and atomic claim, such as an update from prepared to in_flight conditioned on the old state, or a lease with fencing token. A process-local Python set does not protect against multiple workers, restarts, containers, or a second region.

Reconciliation when the outcome is unknown

Reconciliation is a read path that asks the system of record whether the intended action occurred. Query by the provider idempotency key, provider request ID, external transaction reference, or a stable business key. Confirm not just that an object exists, but that the relevant attributes match: recipient, amount, currency, invoice, destination, and status. If the provider returns a matching committed result, record succeeded and resume from that result. If it proves no request was accepted, transition to a retryable failed or back to prepared according to policy.

If the provider offers no lookup by key and no idempotent retry, the outcome can remain genuinely indeterminate. A person may need to inspect a provider dashboard or contact support. The user-facing message should say “we cannot yet confirm whether this action completed,” not “it failed.” This wording matters operationally: an agent that apologizes for a “failed” refund may immediately try again and create the duplicate it was supposed to prevent.

Reconciliation itself can be delayed or eventually consistent. A “not found” result immediately after a timeout might only mean the provider has not indexed the action yet. Define a polling window, a backoff schedule, and a terminal escalation condition. Do not convert a temporary absence into proof of non-execution unless the provider documents that interpretation. Preserve the original operation ID through the entire waiting period.

Decision table for the next move

Observed condition Ledger action Automatic next step Human involvement
Local validation failed before dispatch failed Correct input, then create a new approved operation if meaning changes Only for policy exceptions
Provider returned a documented non-committing error failed or retryable Retry with same operation identity if provider guidance permits If retry policy exhausted
Provider committed and response was durably stored succeeded Reuse stored result; continue workflow No
Request timed out after dispatch; provider supports idempotency unknown then resolve Retry with identical parameters and the same provider key If response remains ambiguous beyond policy window
Request timed out; provider has a queryable reference unknown Reconcile by reference before another action If lookup conflicts or cannot resolve
Request timed out; no key and no lookup unknown Stop; do not blindly replay Yes, before any repeat
Same key arrives with changed parameters conflict Reject and require a new operation ID and approval Usually yes for consequential changes

This table is a decision aid, not a guarantee that a provider error code always reveals commit status. Provider-specific documentation takes precedence. When semantics are unclear, treat the state as ambiguous.

How to resume the agent without replaying completed work

Separate planning from execution. The model may propose a tool call, but the deterministic executor decides whether that operation is new, already complete, currently claimed, or unresolved. On resume, feed successful recorded outputs back into the model context or reconstruct the next workflow step from durable state. Do not ask the model to infer whether an earlier email was sent from conversational prose alone.

For a multi-action workflow, each external side effect needs its own operation record. If step one sends a customer email and step two updates a CRM, a crash after the email but before the CRM should not replay the email. Resume at the first unresolved operation, not at the start of the whole agent run. A checkpoint at the end of the full model turn is too coarse if several tools can execute inside that turn.

The executor should distinguish fresh calls from resumed calls. A fresh call can be eligible for approval and dispatch. A resumed call must first consult the operation ledger. A repeated provider call ID may be a useful signal, but model and provider IDs can be scoped to a turn or response and may not survive all checkpoint arrangements. Treat them as supporting identifiers, not the only durable business key.

A compact event model

One useful event sequence is intent.accepted, operation.prepared, operation.claimed, provider.requested, provider.confirmed, result.persisted, and agent.resumed. Each event should carry the same operation ID and a correlation ID for the run. Store only necessary, redacted parameters or a hash, and keep the provider result reference. The sequence lets an operator answer: did we accept the action, did we attempt it, what did the provider say, and did the agent see that result?

Avoid logging a credential, full payment card data, or unrestricted personal content in an event record. Use the provider’s opaque request ID and a controlled pointer to the result where possible. Access to the ledger should be restricted because it can reveal business activity even when it contains no secret. Define retention consistent with the business record and privacy policy; deduplication should not become an excuse to retain every prompt forever.

Monitoring should alert on old in_flight rows, repeated operation IDs with mismatched parameter hashes, high replay counts, and unknown actions beyond their resolution window. Track these by tool class and provider, not just a single aggregate. A rise in ambiguous outcomes after a provider change is more actionable than a generic “agent failed” rate.

Where a generic retry policy still helps—and where it fails

Retries are not inherently unsafe. A model request that fails before a tool is executed can often be retried. A read-only search can often be repeated. A write guarded by a provider-supported idempotency key can often be retried with the same key and parameters. A database update designed as “set status to paid” may be idempotent with respect to the final value, although its notifications, audit triggers, or downstream events may not be. Classify the entire effect chain, not just the HTTP verb or database statement.

The LangGraph recursion-limit guidance is a useful counterpoint to indiscriminately lowering limits: some valid graphs have long paths or intentional cycles, and hitting a recursion limit can indicate that the configured bound is too small as well as that a loop is unintended. A limit is a circuit breaker, not a diagnosis. Similarly, disabling all retries can reduce duplicate risk but create unnecessary manual work for safe reads and recoverable provider errors.

A relevant 2026 talk, “AI Agents Are Just Distributed Systems Now”, makes the same boundary point from an operator’s perspective: a timeout means the outcome is unknown, while request IDs, idempotency, status lookup, limits, tracing, and recovery help manage it. That is useful framing, not API documentation. The actual guarantee must come from the provider contract. The talk’s broader coverage of permissions and compensation also highlights a limit of this article’s focus: a safe retry identity does not replace scoped authority or a plan for partially completed multi-service workflows.

The recommendation is therefore risk-based. Apply stronger idempotency and human gates to payments, messages, orders, account changes, and irreversible external actions. Keep normal retry behavior for low-impact reads with bounded attempts. For mixed workflows, do not let a single retry setting silently wrap both categories. Separate model-request retries, tool transport retries, and whole-workflow retries, and let each layer consult the side-effect ledger.

Failure modes and what they look like

The retry gets a fresh key. The provider sees a new action and accepts it again. Log key derivation inputs in redacted form and test that restart, queue redelivery, and model regeneration preserve the ID.

The key is reused for a changed request. A wrong recipient or amount is hidden behind a previous result or rejected by the provider. Compare canonicalized parameter hashes and fail closed on mismatch.

The ledger is updated after the model continues. A later model error loses the only result and whole-run retry invokes the tool again. Persist the result before returning control to the model.

The ledger says “failed” after a timeout. The network outcome is ambiguous, but the runtime labels it retryable and repeats an irreversible effect. Use unknown after possible dispatch and reconcile first.

Two workers claim the same row. A read-then-write check races. Use a unique constraint, conditional update, lease, or fencing token and test concurrent resumes.

Provider retention expires. The same key is retried after the provider has forgotten it. Store provider retention assumptions and stop automatic retries before that boundary; reconcile or ask for review.

Only the final node is checkpointed. Earlier side effects in the same node are replayed on recovery. Record each side-effecting operation independently or make the whole node safe to replay.

The audit trail leaks sensitive data. Operators over-log prompts, arguments, or results. Store minimal identifiers, hashes, statuses, and controlled references; redact before indexing or alerting.

A reproducible acceptance test

Build a fake provider with a durable idempotency-key table and an action counter. For each test, start with an empty database. Have the provider commit an action, then intentionally drop the response. Kill or restart the caller. Resume the same workflow with the same operation ID. Assert that the provider has one business effect and the agent receives the original result. Then repeat with a changed parameter under the same key and assert that the request is rejected. Finally, run two workers concurrently against the same prepared action and assert that only one obtains the dispatch claim.

Add a second provider mode without idempotency or lookup support. Simulate a timeout after dispatch and verify that the workflow enters unknown, does not dispatch a second time, and requests reconciliation. This test is important because an implementation that only passes the fake idempotent provider test may still blindly repeat actions for providers with weaker semantics.

The method here is a proposed deterministic test protocol, not a measured benchmark. It has no production sample, no measured failure rate, and no claim about how often duplicate actions occur. Its value is that it turns a vague reliability promise into observable assertions: one action, same result on replay, changed request rejected, concurrent claim serialized, and unresolved outcome held for review.

Rollout sequence for a real system

Start with an inventory of tools and classify them as read-only, idempotent writes, compensable writes, or irreversible writes. “Compensable” does not mean harmless: a cancellation or reversal may leave fees, notifications, audit entries, or customer confusion. For each provider, document idempotency scope, retention, parameter matching, lookup options, and error meanings. If documentation is unclear, ask the vendor or run a controlled sandbox test before enabling automatic recovery.

Next, add an operation ID and ledger for one high-risk tool. Put a uniqueness constraint in the database and record every transition. Keep the operation ID outside the model’s generated prose. Make the executor return a previously stored result on a successful replay and refuse a changed parameter hash. Add explicit unknown handling and an operator queue before broadening rollout.

Run fault injection: crash before dispatch, crash after provider commit, drop the response, fail the next model call, expire the worker lease, and start two recovery workers. Capture the expected state transition and provider effect count for each case. Only then enable automatic retries for the provider modes that pass. Expand to other tools based on their own semantics rather than copying the same policy everywhere.

Choosing between three architectures

Architecture Best fit Benefit Main limitation
Provider idempotency key plus local ledger External API offers stable deduplication Strong retry behavior with simple reconciliation Key scope and retention are provider-specific
Durable workflow history with per-tool outcomes Long-running multi-step agent jobs Resume can skip completed steps and reuse results Still needs receiver keys for network ambiguity
Human approval and reconciliation queue Irreversible effects without safe provider support Avoids guessing when outcome is unknown Slower and requires an operational owner

These patterns can be combined. A durable workflow engine preserves step history; a provider key guards the remote boundary; a human queue handles the cases neither can resolve. No one layer replaces the others.

The operational contract to publish internally

Write a short contract for each side-effecting tool: what counts as one business intent; who may authorize it; how the operation ID is created; which fields are immutable under that ID; what the provider’s deduplication semantics are; what states can be retried automatically; how an unknown outcome is reconciled; when a person must decide; and what evidence is retained. Keep the contract next to the tool implementation and test it in CI.

The contract should include a “do not retry” condition that is machine-readable. For example, unknown && irreversible && no_provider_lookup should stop the workflow and emit an escalation event. Avoid burying this decision in a long system prompt. The model can explain the situation to a user, but the executor must enforce the transition.

Consulting guidance can help frame the organizational importance of resilience, but it does not replace a provider’s API contract or an engineering test. Deloitte’s analysis of an enterprise AI control plane describes orchestration as coordinating models, agents, workloads, and decisions across an organization. Use that as management framing for ownership and observability, not as evidence that a particular retry design guarantees correctness. The mechanism must be validated against the actual API and runtime.

Limits of the proposed pattern

The ledger itself can be unavailable, corrupted, or restored from a stale backup. Idempotency keys can expire, providers can have bugs, and “success” may not mean the downstream business process is complete. A provider can accept a payment request but later reverse it. A message API can accept a send request while downstream delivery remains uncertain. The action state should describe exactly which boundary has been confirmed.

The pattern also cannot create transactional guarantees where none exist. If two independent providers must both commit, the workflow may need a saga with compensating actions and explicit partial-completion states. A compensation is another side effect and requires its own identity and replay protection. If the action is legally or financially consequential, model uncertainty and approval requirements with the domain owner; do not let a generic agent framework determine policy.

Finally, this protocol protects repeated execution of an identified intent. It does not prove that the original intent was correct, that the user authorized it, or that the model selected the right recipient. Authorization, validation, content review, and business policy remain separate gates. Idempotency prevents a duplicate; it does not make a bad first action good.

Research method and evidence limits

This article was researched on 24 October 2026. The source review covered primary API and framework documentation, one recent research preprint, one public framework issue report, the NIST AI RMF Playbook, one consulting architecture analysis, and a technical talk with a timestamped transcript summary. The issue report was treated as a concrete example of a question developers are asking, not as proof of how common the failure is. The preprint’s measurements and taxonomy belong to its authors’ stated dataset and method; they should not be generalized into an industry-wide incident rate. The NIST Playbook is voluntary risk-management guidance, not a technical retry standard. The talk and consulting analysis are contextual sources, not primary evidence of an API guarantee. The worked code example is a proposed reproducible local test, not a production benchmark: it was not run against a live payment provider, and it does not establish an empirical success percentage.

The recommendation is an engineering inference from the boundary conditions documented above: if the caller cannot know whether a remote side effect committed, a local timeout cannot safely label that effect failed. A receiver-side idempotency contract, authoritative reconciliation, or a human decision is needed to resolve the ambiguity. Teams should validate the claim against their selected provider and runtime before enabling automatic replay.

Checklist before enabling automatic resume

  • [ ] The tool is classified by effect and reversibility.
  • [ ] Each business action receives one stable operation ID before dispatch.
  • [ ] The ID survives process restart, queue redelivery, and model regeneration.
  • [ ] Parameter changes under an existing ID are rejected.
  • [ ] The operation ledger uses durable storage and a concurrency-safe claim.
  • [ ] Results are persisted before the agent continues to another model turn.
  • [ ] Provider idempotency scope, retention, and matching rules are documented.
  • [ ] unknown is a first-class state after an ambiguous network outcome.
  • [ ] Reconciliation checks the system of record before repeating uncertain work.
  • [ ] Providers without safe retry support route consequential ambiguity to a person.
  • [ ] Fault-injection tests cover crash windows, dropped responses, and concurrent resumes.
  • [ ] Logs are redacted and retain only the minimum evidence needed for recovery.

Bottom line

An agent should not decide from a timeout whether a side effect happened. The runtime should preserve the action’s identity, record what it knows, and use receiver-side idempotency or reconciliation to resolve uncertainty. Persisting each tool outcome before another model turn prevents a later language-model failure from erasing execution evidence. When the external outcome cannot be proven, stop and escalate rather than retrying on hope.

The smallest useful implementation is not a giant orchestration platform. It is a stable operation ID, a durable four-state ledger, an atomic claim, a documented provider key, and a test that commits the external action while dropping the response. If that test proves one effect and a reusable result on resume, the system has a concrete recovery contract. If it cannot pass, automatic replay is not yet safe for that tool.

For a related deployment perspective on limiting an agent’s role, protecting connected systems, and arranging human recovery, see this commercial-agent guardrails guide. The operational decision remains local to each action: retry only when the system can demonstrate that repetition is safe; otherwise reconcile or ask.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.