Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 25 min read

Your AI Agent Retried the Tool Call. The First One Had Already Succeeded.

The payment went through. The response disappeared. The agent tried again. Your customer was charged twice. Imagine this. You ask your AI agent to pay an outstanding invoice. The agent calls your payment tool. The p

The payment went through. The response disappeared. The agent tried again. Your customer was charged twice.

Imagine this.

You ask your AI agent to pay an outstanding invoice.

The agent calls your payment tool.

The payment provider processes the transaction successfully. The money is charged. The transaction is committed.

But something goes wrong.

The response never reaches your application.

Your tool times out.

The agent sees an error:

Tool execution failed: Request timed out.

So it tries again.

The second request reaches the payment provider.

Another payment is created.

Two successful charges. One user request.

Your agent wasn't trying to do anything malicious. Your payment API wasn't necessarily broken. Your database didn't necessarily lose any data.

Your system simply couldn't distinguish between an operation that failed and an operation that succeeded without returning a response.

This is one of the most dangerous failure modes in AI agent infrastructure.

And it becomes particularly interesting when you realize that the agent might not be the only component retrying the operation.

Your HTTP client can retry it.

Your tool framework can retry it.

Your queue can redeliver it.

Your worker can restart it.

And the model itself can decide to call the tool again.

Every component thinks it is helping.

Together, they can create duplicate side effects.

Let's look at why this happens and how to design an execution system that can survive it.

1. First, understand what actually happened

Before discussing architecture, let's reconstruct the incident.

Assume an invoice is worth $480.

The user asks:

Pay invoice INV-2048 for $480.

The agent generates a tool call:

{
  "tool": "charge_invoice",
  "arguments": {
    "invoice_id": "INV-2048",
    "amount": 480,
    "currency": "USD"
  }
}

Your backend forwards the request to the payment provider.

Here is the actual sequence of events.

TIME        AGENT / APPLICATION          PAYMENT PROVIDER
----------------------------------------------------------------

T+00ms      Agent requests payment
            |
T+20ms      Tool calls payment API  ---->
                                          Receives request

T+150ms                                  Creates charge
                                         $480 charged
                                         Transaction committed

T+180ms     Waiting for response  <----- Response gets lost

T+5s        Request times out

T+5.1s      Agent receives error
            "Tool execution failed"

T+5.3s      Agent retries payment  ---->
                                          Receives new request

T+5.5s                                  Creates another charge
                                         $480 charged
                                         Transaction committed

T+5.7s      Agent receives success <----

----------------------------------------------------------------
                                     TOTAL CHARGED: $960

Illustrative failure timeline: the first transaction succeeds, but the caller never receives confirmation.

Notice what the agent actually knows.

It knows that it did not receive a response.

It does not know that the payment failed.

These are two completely different statements.

A timeout tells you what the caller observed. It does not tell you what the server committed.

This distinction is fundamental to distributed systems.

A request can have several outcomes:

Situation What actually happened What the caller knows
Request never reached the server Nothing happened Usually a connection failure
Server rejected the request Nothing was committed Received an error
Server processed the request Operation succeeded Received success
Server processed the request, response was lost Operation succeeded Timeout
Server is still processing Outcome is pending Timeout or pending response

The fourth situation is the dangerous one.

From the caller's perspective, the outcome is unknown.

From the provider's perspective, the transaction is complete.

Your execution system needs to handle both realities.

2. Why AI agents make retries more complicated

Retries are not new.

We've been building distributed systems with network timeouts, message redelivery, worker crashes, and transient service failures for decades.

So why does this deserve special attention in AI agent systems?

Because an agent introduces another decision-making component into the retry loop.

Consider a traditional application:

Frontend
   |
   v
Backend API
   |
   v
Payment Provider

A frontend might retry a request after a timeout.

But in an agent application, the execution path can look like this:

                USER REQUEST
                     |
                     v
                 AI AGENT
                     |
                     v
                TOOL REQUEST
                     |
                     v
               TOOL EXECUTOR
                     |
                     v
                HTTP CLIENT
                     |
                     v
              PAYMENT PROVIDER

Now introduce independent retry mechanisms.

Component Why it might retry
Model It receives an error and decides to try again
Agent framework It has an automatic tool retry policy
Tool executor It retries transient execution errors
HTTP client It retries connection or service errors
Queue The message wasn't acknowledged
Worker The process restarted before completing the job
Payment provider integration The SDK retries a request

The problem is not that retries exist.

The problem is that each layer may have a different understanding of what constitutes the same operation.

A tool-call ID is not a business operation ID

This distinction is particularly important.

Imagine the agent produces:

First invocation:

tool_call_id = call_A123

It times out.

The model then generates another invocation:

Second invocation:

tool_call_id = call_B456

From the model framework's perspective, these are two different tool calls.

But from the user's perspective, both requests represent one intention:

Pay invoice INV-2048 for $480.

A new tool-call ID must not automatically mean a new payment operation.

This is where many agent implementations make a subtle mistake.

They use the tool invocation identifier as the identifier of the business operation.

Those identifiers have different responsibilities.

Identifier What it identifies
conversation_id The conversation
task_id The agent's task
tool_call_id One tool invocation
operation_id One logical business operation
attempt_id One execution attempt
provider_reference The downstream provider's transaction

The operation ID must remain stable across retries, even when the tool-call ID changes.

This is the foundation of safe retries for AI agents.

3. Idempotency: the engineering concept that prevents duplicate operations

Let's introduce the term that solves much of this problem.

An operation is idempotent when applying it multiple times has the same intended effect as applying it once.

For example:

PUT /users/123

With the same representation and intended state, repeating the request should not create another user.

But consider:

POST /payments

Repeating this request can create another payment.

HTTP semantics are explicit about this distinction.

RFC 9110, Section 9.2.2 defines idempotent methods and explains why clients need to be careful automatically retrying non-idempotent requests.

The important detail is that idempotency applies to the intended effect of the operation, not simply whether the HTTP request succeeds.

What does an idempotent payment operation look like?

Suppose the application generates a unique operation ID:

operation_id = op_9f71c3

It sends this identifier with the payment request.

First attempt:

POST /payments
Idempotency-Key: op_9f71c3

The provider processes the payment.

The response is lost.

The application retries:

POST /payments
Idempotency-Key: op_9f71c3

The provider recognizes the same logical request and applies its documented idempotency behavior instead of creating another payment.

The application can then recover the original result.

The critical property is:

Same logical operation
        |
        v
Same idempotency key
        |
        v
Same intended side effect

A new operation should receive a new key.

Important: don't generate an idempotency key from the payload alone

This is a surprisingly common design mistake.

Imagine two legitimate purchases:

{
  "product_id": "item-100",
  "quantity": 1,
  "amount": 50
}

The customer might intentionally purchase the same item twice.

A hash of the request payload would be identical.

But these are two different business intentions.

Amazon explains this exact problem in its engineering article, Making retries safe with idempotent APIs.

Its recommendation is to use a unique caller-provided request identifier to express intent, rather than guessing that identical parameters must represent duplicate requests.

A better design separates two things:

Idempotency key: Identifies the logical operation.

Request fingerprint: Verifies that the same key has not been reused with different parameters.

For example:

operation_id:
op_9f71c3

request_fingerprint:
SHA256(
    tenant_id +
    invoice_id +
    amount +
    currency +
    operation_type
)

On a retry:

  • Same operation ID, same fingerprint: treat it as the same operation.
  • Same operation ID, different fingerprint: reject it.
  • Different operation ID, same fingerprint: potentially a new intentional operation.

This is a very important distinction.

4. The architecture: put an execution coordinator between the agent and the tool

I would not let the agent directly trigger a sensitive external operation.

Instead, introduce a durable execution coordinator.

Its responsibility is simple:

Remember what the application intended to execute, whether it has already executed, and what result was produced.

The architecture becomes:

                  USER
                    |
                    v
                AI AGENT
                    |
                    v
             PROPOSED TOOL CALL
                    |
                    v
          VALIDATION + AUTHORIZATION
                    |
                    v
           OPERATION COORDINATOR
                    |
              +-----+-----+
              |           |
              v           v
        OPERATION      IDEMPOTENCY
          LEDGER         CHECK
              |           |
              +-----+-----+
                    |
                    v
             TOOL EXECUTOR
                    |
                    v
           DOWNSTREAM PROVIDER
                    |
                    v
            EXECUTION RESULT
                    |
                    v
           PERSIST RESULT STATE
                    |
                    v
              AGENT RESPONSE

Let's understand the responsibilities.

Component Responsibility
Agent Proposes an operation
Validation Checks arguments and business constraints
Authorization Determines whether the user can perform the operation
Operation coordinator Identifies the logical operation
Operation ledger Stores durable execution state
Tool executor Performs the authorized operation
Provider Processes the external side effect
Reconciliation Resolves uncertain outcomes

The coordinator is not just another retry wrapper.

It is the place where the system maintains the identity and lifecycle of the operation.

A crucial design decision

The application should create the operation identity before executing the side effect.

For a sensitive action, this can happen when a user approves the proposed payment.

For example:

{
  "operation_id": "op_9f71c3",
  "task_id": "task_2026_114",
  "operation": "invoice.payment",
  "resource": "INV-2048",
  "amount": 480,
  "currency": "USD",
  "status": "PENDING"
}

The model can propose an operation.

The trusted application creates and controls its operation ID.

The model should not be responsible for generating a fresh idempotency key on every retry.

5. Build a durable operation ledger

A retry-safe execution system needs persistent state.

An in-memory dictionary is insufficient.

Consider this implementation:

completed_operations = {}

async def execute_payment(operation_id, amount):

    if operation_id in completed_operations:
        return completed_operations[operation_id]

    result = await payment_provider.charge(amount)

    completed_operations[operation_id] = result

    return result

It looks reasonable until the process crashes.

Or two workers receive the same operation simultaneously.

Or the provider succeeds but the application crashes before storing the result.

In all three situations, the application can execute the payment again.

The operation state must live in durable storage.

A PostgreSQL schema

Here's a simplified schema:

CREATE TABLE tool_operations (
    tenant_id UUID NOT NULL,
    operation_id TEXT NOT NULL,

    task_id UUID NOT NULL,
    agent_id TEXT NOT NULL,

    operation_type TEXT NOT NULL,
    request_hash TEXT NOT NULL,

    status TEXT NOT NULL,

    provider_reference TEXT,
    response_json JSONB,

    attempt_count INTEGER NOT NULL DEFAULT 0,

    lease_owner TEXT,
    lease_expires_at TIMESTAMPTZ,

    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    PRIMARY KEY (tenant_id, operation_id)
);

The operation_id is an application-generated string identity (for example op_9f71c3), not a database-minted UUID β€” the whole point is that your trusted application owns it and reuses it across retries. The composite primary key ensures that an operation ID is unique within the tenant's operation namespace.

A production implementation should also constrain valid status values, index its recovery queries, and define a retention policy.

The execution state machine

Do not model execution as simply:

SUCCESS / FAILED

That is insufficient for distributed operations.

A provider can process an operation while the application is still waiting for confirmation.

Use more meaningful states:

             PENDING
                |
                v
            IN_PROGRESS
                |
          +-----+------+
          |            |
          v            v
       SUCCEEDED     UNKNOWN
                       |
                       v
                  RECONCILING
                       |
                +------+------+
                |             |
                v             v
             SUCCEEDED     RETRYABLE
                              |
                              v
                         IN_PROGRESS

Other terminal state:

FAILED_FINAL

What do these states mean?

Status Meaning
PENDING Operation recorded, not yet executed
IN_PROGRESS A worker currently owns the execution
SUCCEEDED Operation completed and its result is persisted
UNKNOWN The downstream outcome is uncertain
RECONCILING The system is checking the actual downstream result
RETRYABLE A failure was established as safe to retry
FAILED_FINAL The operation was definitively rejected or failed

Why is UNKNOWN so important?

Because a timeout must not automatically transition an operation to FAILED_FINAL.

That would turn an unknown outcome into a false statement.

And a false statement is exactly how duplicate side effects happen.

6. Implementing a retry-safe execution flow in Python

Let's put the concepts together.

The following is an illustrative Python service design. It uses an operation ledger, a stable operation ID, a request fingerprint, and a provider-side idempotency key.

The database and provider adapters are interfaces; their atomicity and idempotency contracts must be implemented by the underlying services.

First, define the operation:

from dataclasses import dataclass
from decimal import Decimal
from uuid import UUID


@dataclass(frozen=True)
class PaymentIntent:
    operation_id: str
    tenant_id: UUID
    invoice_id: str
    amount: Decimal
    currency: str


@dataclass(frozen=True)
class PaymentResult:
    provider_reference: str
    status: str
    amount: Decimal
    currency: str

Notice that operation_id is part of the trusted application-created intent.

It should not be regenerated just because the model made another tool call.

Step 1: Calculate a canonical request fingerprint

import hashlib
import json


def fingerprint(intent: PaymentIntent) -> str:

    payload = {
        "tenant_id": str(intent.tenant_id),
        "invoice_id": intent.invoice_id,
        "amount": str(intent.amount),
        "currency": intent.currency.upper(),
        "operation_type": "invoice.payment",
    }

    canonical = json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
    )

    return hashlib.sha256(
        canonical.encode("utf-8")
    ).hexdigest()

The fingerprint protects against reusing the same operation ID with different payment parameters.

Do not include volatile fields such as the current timestamp, retry count, or tool-call ID.

Those values change between attempts even when the business operation does not.

Step 2: Claim the operation atomically

Conceptually:

record = await ledger.claim(
    tenant_id=intent.tenant_id,
    operation_id=intent.operation_id,
    request_hash=fingerprint(intent),
)

The claim() method must be implemented using database-level concurrency control.

For example:

  • A unique constraint on the operation identity.
  • A transaction around insertion or state inspection.
  • Row-level locking or an atomic conditional update.
  • A lease for the worker that owns the execution.
  • A request-hash comparison for existing records.

A second worker must not independently claim the same operation and submit another payment.

Importantly, do not hold a database transaction open while waiting for an external HTTP request. Use a short transaction to claim the work, perform the network operation outside it, and persist the result in another transaction.

Step 3: Return an existing result when available

if record.status == "SUCCEEDED":
    return record.response

The operation has already completed.

There is no reason to call the provider again.

For in-progress or uncertain operations:

if record.status in {"IN_PROGRESS", "UNKNOWN"}:
    return await reconcile_or_resume(record)

The reconciliation path must check the provider's state or safely resume using the same downstream idempotency key.

It must not blindly create another payment.

Step 4: Execute with a stable provider key

provider_key = (
    f"invoice-payment:{intent.operation_id}"
)

result = await payment_provider.create_charge(
    invoice_id=intent.invoice_id,
    amount=intent.amount,
    currency=intent.currency,
    idempotency_key=provider_key,
)

The downstream provider's idempotency contract is important here.

It must recognize retries using the same key and apply its documented duplicate-request behavior.

The provider response should then be persisted:

await ledger.mark_succeeded(
    operation_id=intent.operation_id,
    provider_reference=result.id,
    response=result.to_dict(),
)

Step 5: Handle timeouts as unknown outcomes

This is the part that many implementations get wrong.

try:
    result = await payment_provider.create_charge(
        invoice_id=intent.invoice_id,
        amount=intent.amount,
        currency=intent.currency,
        idempotency_key=provider_key,
    )

except TimeoutError:

    await ledger.mark_unknown(
        operation_id=intent.operation_id,
    )

    return {
        "operation_id": str(intent.operation_id),
        "status": "UNKNOWN",
        "message": (
            "The provider response was not received. "
            "The operation is being reconciled."
        ),
    }

The important thing is what this code does not do.

It does not generate a new operation ID.

It does not generate a new provider idempotency key.

It does not tell the model that the payment definitely failed.

It preserves the uncertainty.

A retry is safe only when the system preserves the identity of the original operation and knows how to handle its previous outcome.

7. The most dangerous gap: your database cannot control another company's database

There is a subtle problem with the previous implementation.

Your PostgreSQL database and an external payment provider are two independent systems.

They do not share a transaction.

Consider this sequence:

1. Insert operation record.
2. Set status = IN_PROGRESS.
3. Send request to payment provider.
4. Provider creates charge.
5. Provider returns success.
6. Your application crashes.
7. Result is never written to PostgreSQL.

After restart, your database still says:

status = IN_PROGRESS

But the payment has already happened.

This is a distributed transaction problem.

You cannot solve it by putting another try/except around the HTTP request.

Three ways to handle it

Option A: Downstream idempotency support

The provider accepts a stable idempotency key.

On retry, submit the identical request using the same key, subject to the provider's documented retention and request-matching rules.

This is usually the simplest approach.

Option B: Query by a stable business reference

If the provider supports transaction lookup, associate the external operation with a stable merchant reference.

When the response is lost:

  1. Query the provider using that reference.
  2. Check whether the transaction exists.
  3. Retrieve its actual status.
  4. Update your local operation ledger.
  5. Only retry when the provider contract establishes that doing so is safe.

Option C: Keep the operation in an unknown state

What happens when the provider doesn't support idempotency and cannot reliably identify the earlier request?

Do not pretend that you can guarantee exactly-once execution.

Keep the operation in UNKNOWN, prevent an automatic duplicate side effect, and route it to reconciliation or human investigation.

This is much safer than charging someone twice because a timeout looked like a failure.

An important distributed-systems rule

The local idempotency ledger protects your application from repeating its own work.

It cannot independently guarantee that a remote system executed a side effect exactly once.

You need a downstream idempotency contract or a reliable reconciliation mechanism.

8. Why an idempotency key alone is not enough

Let's assume you have implemented idempotency keys.

Are you done?

Not yet.

There are several failure modes worth handling explicitly.

Failure 1: The agent uses a new key on retry

First attempt:
key = payment-001

Second attempt:
key = payment-002

The provider sees two different operations.

Two payments can be created.

Fix: Generate the operation identity in trusted application state and reuse it throughout the workflow.

Failure 2: Same key, different parameters

First request:

{
  "invoice_id": "INV-2048",
  "amount": 480
}

Retry:

{
  "invoice_id": "INV-2048",
  "amount": 840
}

Same key. Different intent.

The system should reject this request.

AWS discusses parameter mismatch handling in its idempotent API design guidance. The IETF's current Idempotency-Key Internet-Draft also defines request fingerprints as a mechanism for detecting this situation.

Idempotency-Key HTTP Header Field β€” draft-08.

Fix: Bind each operation ID to a canonical request fingerprint.

Failure 3: Two workers execute simultaneously

Imagine two workers receive the same queue event.

Both check the operation ledger.

Both see that the operation is not completed.

Both submit the payment.

A normal application-level if statement cannot prevent this race.

Fix: Use a database uniqueness constraint, atomic claims, and worker leases.

For long-running operations, leases should have expiration and ownership checks. A stale worker must not be able to overwrite the result of a newer worker.

Failure 4: The idempotency record expires too soon

Suppose your system retains the operation record for 24 hours.

An agent workflow gets paused for two days.

It resumes and submits the same operation.

The provider no longer recognizes the old idempotency key.

A duplicate may be created.

Fix: Retain the durable business operation record for the required business and replay horizon. Align it with the downstream provider's idempotency retention policy.

Do not assume that provider keys remain valid forever.

Failure 5: The provider succeeds, but your result persistence fails

Provider: SUCCESS
Your database: WRITE FAILED
Agent: NO RESPONSE

This is not a normal failure.

It is an uncertain result.

Fix: Preserve the operation identity and reconcile using the provider key or transaction reference.

The system should never convert an ambiguous external result into an automatic fresh operation.

9. What about queues, workers, and background jobs?

The same problem appears when AI agents use asynchronous execution.

For example:

AI Agent
    |
    v
Tool Executor
    |
    v
Message Queue
    |
    v
Worker
    |
    v
Payment API

Suppose the worker successfully processes the message but crashes before acknowledging it.

The queue may deliver that message again.

This behavior is not unusual.

Amazon SQS Standard queues provide at-least-once delivery. AWS explicitly recommends designing consumers to be idempotent because a message can be delivered more than once.

Amazon SQS β€” At-least-once delivery.

AWS Lambda also documents that event-source mappings can process events more than once and recommends idempotent function code.

Lambda best practices.

The important point:

A message being delivered twice must not mean the business operation happens twice.

Introduce an operation consumer

A queue message should carry the stable operation identity:

{
  "event_id": "evt_901",
  "operation_id": "op_9f71c3",
  "operation_type": "invoice.payment",
  "tenant_id": "tenant_42"
}

The worker should load the operation record before executing anything.

async def handle_message(message):

    operation = await ledger.get(
        tenant_id=message.tenant_id,
        operation_id=message.operation_id,
    )

    if operation.status == "SUCCEEDED":
        return

    await execute_operation(operation)

This example omits the actual concurrency claim deliberately. Production workers must atomically claim or resume the operation, not just read a status and then execute.

Don't confuse an exactly-once queue feature with exactly-once business execution

Some messaging systems offer exactly-once delivery or processing features under defined conditions.

For example, Google Cloud Pub/Sub documents exactly-once delivery for supported pull subscriptions within a region.

But this does not automatically provide exactly-once execution across an external payment provider, your database, and your AI agent.

Google Cloud Pub/Sub β€” Exactly-once delivery.

A queue can manage message acknowledgment.

It cannot automatically make every downstream side effect part of the same atomic transaction.

10. Use the transactional outbox when execution crosses systems

There is another common failure in event-driven systems.

Your application commits a database transaction and then tries to publish an event.

await db.save_payment(payment)

await queue.publish({
    "operation_id": operation_id
})

What happens if the database commit succeeds but the process crashes before publishing?

The payment exists.

The event was never published.

A downstream worker never learns that it needs to continue the workflow.

Reversing the order creates the opposite problem: the event can be published even though the database transaction later fails.

This is known as the dual-write problem.

The transactional outbox pattern addresses it.

The design

Write the business state and an outbox event in the same database transaction.

BEGIN;

INSERT INTO tool_operations (
    tenant_id,
    operation_id,
    operation_type,
    request_hash,
    status
)
VALUES (
    :tenant_id,
    :operation_id,
    'invoice.payment',
    :request_hash,
    'PENDING'
);

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload
)
VALUES (
    :event_id,
    :operation_id,
    'PAYMENT_REQUESTED',
    :payload
);

COMMIT;

A separate dispatcher reads the outbox and publishes events.

                 DATABASE
                     |
             +-------+-------+
             |               |
             v               v
       Operation State   Outbox Event
             |               |
             +-------+-------+
                     |
                     v
               Dispatcher
                     |
                     v
                 MESSAGE
                   QUEUE
                     |
                     v
                  WORKER

Now the application cannot commit the operation record without also recording the event that needs to be dispatched.

There is still an important detail:

The dispatcher can publish an event and crash before marking it as published.

It may publish the same event again.

So the consumer must also be idempotent.

The outbox solves the dual-write problem. It does not magically eliminate duplicate message delivery.

AWS explains this distinction in its Transactional Outbox Pattern guidance.

11. Not every operation should use the same retry strategy

Retrying a failed read operation is very different from retrying a payment.

We need to classify tool operations.

Operation Retry strategy
Read a customer record Usually safe to retry, subject to authorization and consistency requirements
Search documents Usually safe to retry
Generate an internal draft Retryable if duplicate drafts are acceptable or deduplicated
Create a support ticket Use a stable operation ID
Send an email Deduplicate using a durable outbound message identity
Export a report Use a stable export job identity
Charge a payment Provider idempotency plus reconciliation
Issue a refund Stable operation identity and provider-level controls
Deploy software Release/deployment identity and environment-level state checks
Delete customer data Durable state tracking and reconciliation

Retry only errors that are safe to retry

A useful classification:

Potentially retryable

  • Temporary connection failures.
  • Rate limits.
  • Explicit transient server failures.
  • Worker interruptions.

Not automatically retryable

  • Invalid arguments.
  • Authentication failures.
  • Authorization denials.
  • Business-rule rejections.
  • Payment declines.
  • Unknown external outcomes.

An important nuance: even HTTP 500 does not necessarily mean that no side effect occurred. The service may have committed the operation and failed while producing the response.

That is why idempotency and outcome reconciliation matter more than simply checking the HTTP status code.

Use exponential backoff and jitter

Don't make every worker retry immediately.

A simple strategy:

import random


def retry_delay(attempt: int) -> float:

    base_delay = 1.0
    max_delay = 30.0

    exponential = min(
        max_delay,
        base_delay * (2 ** attempt),
    )

    return random.uniform(0, exponential)

This is full jitter: each retry waits for a random duration between zero and the calculated exponential ceiling.

Also impose:

  • A maximum number of attempts.
  • A total execution deadline.
  • A retry budget.
  • Per-provider concurrency limits.
  • Respect for server-provided retry instructions when applicable.

Retries are not free.

When a downstream service is overloaded, aggressive retries can increase its load and make recovery harder. Google's SRE book covers exactly this dynamic β€” how well-intentioned client retries turn a partial outage into a full one β€” in Addressing Cascading Failures, and AWS discusses the mitigations in Timeouts, retries, and backoff with jitter.

12. Give the agent an honest execution result

There is a small but important interface-design issue.

What does the tool return after a timeout?

Many systems return something like:

{
  "success": false,
  "error": "Payment request timed out"
}

This response is misleading.

Did the provider reject the payment?

Did the network fail before the request was sent?

Did the payment succeed but the response get lost?

The tool doesn't know.

The model receives a false failure signal and may generate another payment request.

A better response is:

{
  "operation_id": "op_9f71c3",
  "status": "UNKNOWN",
  "message": "The payment outcome is being verified.",
  "retry_allowed": false
}

For an operation that is still processing:

{
  "operation_id": "op_9f71c3",
  "status": "IN_PROGRESS",
  "message": "The original operation is still being processed.",
  "retry_allowed": false
}

For a completed operation:

{
  "operation_id": "op_9f71c3",
  "status": "SUCCEEDED",
  "provider_reference": "ch_123456",
  "amount": 480,
  "currency": "USD"
}

The agent should receive the outcome of the original operation, not an invitation to create another one.

Keep the operation identity outside the model's control

I would design the tool interface around this principle:

async def charge_invoice(
    invoice_id: str,
    amount: Decimal,
    currency: str,
    execution_context: TrustedExecutionContext,
):
    ...

The trusted context is injected by the application.

It contains the original operation ID, tenant identity, user identity, authorization context, and task metadata.

The model cannot replace that identity simply by generating a different tool-call ID.

If the model changes the invoice or amount, the application must validate whether the new request represents the same approved operation. A materially different action requires a new authorization and a new operation identity.

13. How to test this properly: break the response on purpose

A normal happy-path test will never expose this problem.

You need fault injection.

I would build the following tests before allowing an agent to perform externally visible operations.

Test Injected failure Expected result
1 Provider succeeds, response is dropped One operation; outcome reconciled
2 Application crashes after provider success Retry returns original result
3 Two workers receive the same operation Only one effective side effect
4 Agent generates a new tool-call ID Same business operation ID is reused
5 Same key, different amount Request rejected
6 Queue redelivers the same message No duplicate business effect
7 Idempotency record expires Old operation is not silently treated as new
8 Provider is unavailable Controlled retry or pending state
9 Provider has no idempotency support Ambiguous outcome enters reconciliation
10 Agent requests a genuinely new payment New operation ID permits a new intentional action

The test that matters most

I would start with this exact scenario:

GIVEN:

A payment operation has a stable operation ID.

WHEN:

The provider successfully charges the customer.

AND:

The HTTP response is dropped before reaching the application.

AND:

The agent invokes the same logical operation again
with a different tool_call_id.

THEN:

The system must reuse the original operation ID.

AND:

The provider must receive the same idempotency key
if its API supports this contract.

AND:

The system must return the original payment result
after reconciliation.

ASSERT:

Exactly one payment was created.

This is a much stronger test than asserting that the agent received a successful response.

It verifies the thing that matters: the actual side effect.

Measure more than the HTTP status

Track:

Metric Why it matters
tool_operation_attempts_total Shows how often execution is retried
tool_operation_duplicates_total Measures duplicate invocations
idempotency_replays_total Counts requests served from existing operation state
operation_unknown_total Tracks unresolved external outcomes
operation_reconciliation_duration Shows how long uncertainty persists
operation_duplicate_effects_total Detects actual duplicate side effects

Every attempt should be traceable to the same operation ID.

Log the tool-call ID separately so you can see when multiple model invocations refer to one business operation.

Do not log payment credentials, authentication tokens, or sensitive customer data just to make debugging easier.

14. What about workflows involving several tools?

The problem becomes even more complicated when an agent executes multiple operations.

Imagine an agent handling an order:

1. Create invoice
2. Charge customer
3. Update inventory
4. Send confirmation email
5. Update CRM

What happens if step 4 fails after steps 1–3 have succeeded?

Should the agent restart the entire workflow?

Absolutely not.

That would risk creating another invoice, charging the customer again, and reducing inventory twice.

The correct approach is to treat the workflow as a series of durable, independently identifiable steps.

                  ORDER WORKFLOW
                        |
                        v
                  CREATE INVOICE
                        |
                        v
                  CHARGE CUSTOMER
                        |
                        v
                 UPDATE INVENTORY
                        |
                        v
                 SEND CONFIRMATION
                        |
                        v
                    UPDATE CRM

Each step should have:

  • A stable operation identity.
  • Its own durable state.
  • Explicit retry rules.
  • A recorded result.
  • A way to reconcile uncertain outcomes.

For longer workflows, use an orchestration or saga pattern.

A saga coordinates independent operations and defines how to recover when a later step fails.

But remember: a compensation is not a magical rollback.

Refunding a successful payment is a new business operation. Sending a correction email does not erase an email already received. Reversing inventory may require another controlled transaction.

This is why a durable workflow is more reliable than asking the model to repeat the entire plan after an error.

Temporal's documentation makes a particularly useful distinction: its workflow can record a single completed activity result, but an activity itself may execute multiple times during retries and may partially complete more than once.

Temporal β€” Activities.

That is an important distributed-systems lesson for agent developers.

15. A practical production checklist

Before giving an AI agent access to tools that create external side effects, I would verify these requirements.

Operation identity

  • [ ] Every business operation gets a stable server-generated ID.
  • [ ] Tool-call IDs and operation IDs are separate.
  • [ ] Retries reuse the original operation ID.
  • [ ] New intentional operations receive new IDs.

Execution state

  • [ ] Operation state is stored durably.
  • [ ] Database uniqueness prevents duplicate claims.
  • [ ] Concurrent workers are coordinated atomically.
  • [ ] Worker leases and stale-worker recovery are defined.
  • [ ] Unknown outcomes are not marked as failed.

External services

  • [ ] Stable provider idempotency keys are used where supported.
  • [ ] Same-key requests are checked against the same parameters.
  • [ ] Provider key expiry is understood.
  • [ ] Provider lookup and reconciliation are available where necessary.
  • [ ] Services without deduplication have a safe ambiguous-outcome policy.

Agent orchestration

  • [ ] Agent retries cannot create fresh business operation identities.
  • [ ] Tool responses distinguish failure, pending, and unknown.
  • [ ] Model retries cannot bypass operation state.
  • [ ] Sensitive operations have appropriate authorization and approval controls.

Testing and monitoring

  • [ ] Response-loss scenarios are tested.
  • [ ] Worker-crash scenarios are tested.
  • [ ] Concurrent duplicate invocations are tested.
  • [ ] Duplicate external side effects are monitored.
  • [ ] Unresolved operations have an escalation path.

If you cannot answer these questions, your agent may be safe during a successful demonstration but unreliable during real production failures.

16. The real lesson: retries are not the enemy

We should not disable retries.

Networks fail. Workers crash. APIs become temporarily unavailable. Message queues redeliver events.

Retries are an essential reliability mechanism.

But a retry must represent the continuation of an existing intention, not the creation of a new intention.

That's the distinction AI agent infrastructure needs to get right.

Your agent can call a tool twice.

Your worker can process a message twice.

Your HTTP client can resend a request.

Your provider can receive the same request multiple times.

None of these events should automatically create another business operation.

The execution layer must know what the user originally asked for, what has already happened, and what remains uncertain.

And it must preserve that knowledge even when the model, worker, application, or network restarts.

A model's retry is not permission to repeat a side effect.

The next time you see an AI agent report a timeout, don't immediately ask why the request failed.

Ask a more important question:

Did the operation fail, or did we simply fail to find out that it succeeded?

Because in production, those two situations can cost your customer very different amounts of money.

Further reading

These are the engineering references behind the architecture discussed in this article.

  1. RFC 9110 β€” HTTP Semantics, Section 9.2.2: Idempotent Methods
  2. AWS Builders' Library β€” Making retries safe with idempotent APIs
  3. IETF β€” Idempotency-Key HTTP Header Field, draft-08 (October 22, 2026)
  4. AWS Lambda Powertools β€” Idempotency for Python
  5. Amazon SQS β€” At-least-once delivery
  6. AWS Prescriptive Guidance β€” Transactional Outbox Pattern
  7. Google Cloud Pub/Sub β€” Exactly-once delivery
  8. Google SRE β€” Addressing Cascading Failures
  9. Temporal β€” Activities

One question for the engineering community:

Have you ever seen an API operation succeed while the caller received a timeoutβ€”and what did your system do when the client retried?

I'm particularly interested in how teams handle the UNKNOWN state when an external side effect may already have happened.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.