Your AI Agent Retried the Tool Call. The First One Had Already Succeeded.
The payment went through. The response disappeared. The agent tried again. Your customer was charged twice. Imagine this. You ask your AI agent to pay an outstanding invoice. The agent calls your payment tool. The p
The payment went through. The response disappeared. The agent tried again. Your customer was charged twice.
Imagine this.
You ask your AI agent to pay an outstanding invoice.
The agent calls your payment tool.
The payment provider processes the transaction successfully. The money is charged. The transaction is committed.
But something goes wrong.
The response never reaches your application.
Your tool times out.
The agent sees an error:
Tool execution failed: Request timed out.
So it tries again.
The second request reaches the payment provider.
Another payment is created.
Two successful charges. One user request.
Your agent wasn't trying to do anything malicious. Your payment API wasn't necessarily broken. Your database didn't necessarily lose any data.
Your system simply couldn't distinguish between an operation that failed and an operation that succeeded without returning a response.
This is one of the most dangerous failure modes in AI agent infrastructure.
And it becomes particularly interesting when you realize that the agent might not be the only component retrying the operation.
Your HTTP client can retry it.
Your tool framework can retry it.
Your queue can redeliver it.
Your worker can restart it.
And the model itself can decide to call the tool again.
Every component thinks it is helping.
Together, they can create duplicate side effects.
Let's look at why this happens and how to design an execution system that can survive it.
1. First, understand what actually happened
Before discussing architecture, let's reconstruct the incident.
Assume an invoice is worth $480.
The user asks:
Pay invoice INV-2048 for $480.
The agent generates a tool call:
{
"tool": "charge_invoice",
"arguments": {
"invoice_id": "INV-2048",
"amount": 480,
"currency": "USD"
}
}
Your backend forwards the request to the payment provider.
Here is the actual sequence of events.
TIME AGENT / APPLICATION PAYMENT PROVIDER
----------------------------------------------------------------
T+00ms Agent requests payment
|
T+20ms Tool calls payment API ---->
Receives request
T+150ms Creates charge
$480 charged
Transaction committed
T+180ms Waiting for response <----- Response gets lost
T+5s Request times out
T+5.1s Agent receives error
"Tool execution failed"
T+5.3s Agent retries payment ---->
Receives new request
T+5.5s Creates another charge
$480 charged
Transaction committed
T+5.7s Agent receives success <----
----------------------------------------------------------------
TOTAL CHARGED: $960
Illustrative failure timeline: the first transaction succeeds, but the caller never receives confirmation.
Notice what the agent actually knows.
It knows that it did not receive a response.
It does not know that the payment failed.
These are two completely different statements.
A timeout tells you what the caller observed. It does not tell you what the server committed.
This distinction is fundamental to distributed systems.
A request can have several outcomes:
| Situation | What actually happened | What the caller knows |
|---|---|---|
| Request never reached the server | Nothing happened | Usually a connection failure |
| Server rejected the request | Nothing was committed | Received an error |
| Server processed the request | Operation succeeded | Received success |
| Server processed the request, response was lost | Operation succeeded | Timeout |
| Server is still processing | Outcome is pending | Timeout or pending response |
The fourth situation is the dangerous one.
From the caller's perspective, the outcome is unknown.
From the provider's perspective, the transaction is complete.
Your execution system needs to handle both realities.
2. Why AI agents make retries more complicated
Retries are not new.
We've been building distributed systems with network timeouts, message redelivery, worker crashes, and transient service failures for decades.
So why does this deserve special attention in AI agent systems?
Because an agent introduces another decision-making component into the retry loop.
Consider a traditional application:
Frontend
|
v
Backend API
|
v
Payment Provider
A frontend might retry a request after a timeout.
But in an agent application, the execution path can look like this:
USER REQUEST
|
v
AI AGENT
|
v
TOOL REQUEST
|
v
TOOL EXECUTOR
|
v
HTTP CLIENT
|
v
PAYMENT PROVIDER
Now introduce independent retry mechanisms.
| Component | Why it might retry |
|---|---|
| Model | It receives an error and decides to try again |
| Agent framework | It has an automatic tool retry policy |
| Tool executor | It retries transient execution errors |
| HTTP client | It retries connection or service errors |
| Queue | The message wasn't acknowledged |
| Worker | The process restarted before completing the job |
| Payment provider integration | The SDK retries a request |
The problem is not that retries exist.
The problem is that each layer may have a different understanding of what constitutes the same operation.
A tool-call ID is not a business operation ID
This distinction is particularly important.
Imagine the agent produces:
First invocation:
tool_call_id = call_A123
It times out.
The model then generates another invocation:
Second invocation:
tool_call_id = call_B456
From the model framework's perspective, these are two different tool calls.
But from the user's perspective, both requests represent one intention:
Pay invoice INV-2048 for $480.
A new tool-call ID must not automatically mean a new payment operation.
This is where many agent implementations make a subtle mistake.
They use the tool invocation identifier as the identifier of the business operation.
Those identifiers have different responsibilities.
| Identifier | What it identifies |
|---|---|
conversation_id |
The conversation |
task_id |
The agent's task |
tool_call_id |
One tool invocation |
operation_id |
One logical business operation |
attempt_id |
One execution attempt |
provider_reference |
The downstream provider's transaction |
The operation ID must remain stable across retries, even when the tool-call ID changes.
This is the foundation of safe retries for AI agents.
3. Idempotency: the engineering concept that prevents duplicate operations
Let's introduce the term that solves much of this problem.
An operation is idempotent when applying it multiple times has the same intended effect as applying it once.
For example:
PUT /users/123
With the same representation and intended state, repeating the request should not create another user.
But consider:
POST /payments
Repeating this request can create another payment.
HTTP semantics are explicit about this distinction.
RFC 9110, Section 9.2.2 defines idempotent methods and explains why clients need to be careful automatically retrying non-idempotent requests.
The important detail is that idempotency applies to the intended effect of the operation, not simply whether the HTTP request succeeds.
What does an idempotent payment operation look like?
Suppose the application generates a unique operation ID:
operation_id = op_9f71c3
It sends this identifier with the payment request.
First attempt:
POST /payments
Idempotency-Key: op_9f71c3
The provider processes the payment.
The response is lost.
The application retries:
POST /payments
Idempotency-Key: op_9f71c3
The provider recognizes the same logical request and applies its documented idempotency behavior instead of creating another payment.
The application can then recover the original result.
The critical property is:
Same logical operation
|
v
Same idempotency key
|
v
Same intended side effect
A new operation should receive a new key.
Important: don't generate an idempotency key from the payload alone
This is a surprisingly common design mistake.
Imagine two legitimate purchases:
{
"product_id": "item-100",
"quantity": 1,
"amount": 50
}
The customer might intentionally purchase the same item twice.
A hash of the request payload would be identical.
But these are two different business intentions.
Amazon explains this exact problem in its engineering article, Making retries safe with idempotent APIs.
Its recommendation is to use a unique caller-provided request identifier to express intent, rather than guessing that identical parameters must represent duplicate requests.
A better design separates two things:
Idempotency key: Identifies the logical operation.
Request fingerprint: Verifies that the same key has not been reused with different parameters.
For example:
operation_id:
op_9f71c3
request_fingerprint:
SHA256(
tenant_id +
invoice_id +
amount +
currency +
operation_type
)
On a retry:
- Same operation ID, same fingerprint: treat it as the same operation.
- Same operation ID, different fingerprint: reject it.
- Different operation ID, same fingerprint: potentially a new intentional operation.
This is a very important distinction.
4. The architecture: put an execution coordinator between the agent and the tool
I would not let the agent directly trigger a sensitive external operation.
Instead, introduce a durable execution coordinator.
Its responsibility is simple:
Remember what the application intended to execute, whether it has already executed, and what result was produced.
The architecture becomes:
USER
|
v
AI AGENT
|
v
PROPOSED TOOL CALL
|
v
VALIDATION + AUTHORIZATION
|
v
OPERATION COORDINATOR
|
+-----+-----+
| |
v v
OPERATION IDEMPOTENCY
LEDGER CHECK
| |
+-----+-----+
|
v
TOOL EXECUTOR
|
v
DOWNSTREAM PROVIDER
|
v
EXECUTION RESULT
|
v
PERSIST RESULT STATE
|
v
AGENT RESPONSE
Let's understand the responsibilities.
| Component | Responsibility |
|---|---|
| Agent | Proposes an operation |
| Validation | Checks arguments and business constraints |
| Authorization | Determines whether the user can perform the operation |
| Operation coordinator | Identifies the logical operation |
| Operation ledger | Stores durable execution state |
| Tool executor | Performs the authorized operation |
| Provider | Processes the external side effect |
| Reconciliation | Resolves uncertain outcomes |
The coordinator is not just another retry wrapper.
It is the place where the system maintains the identity and lifecycle of the operation.
A crucial design decision
The application should create the operation identity before executing the side effect.
For a sensitive action, this can happen when a user approves the proposed payment.
For example:
{
"operation_id": "op_9f71c3",
"task_id": "task_2026_114",
"operation": "invoice.payment",
"resource": "INV-2048",
"amount": 480,
"currency": "USD",
"status": "PENDING"
}
The model can propose an operation.
The trusted application creates and controls its operation ID.
The model should not be responsible for generating a fresh idempotency key on every retry.
5. Build a durable operation ledger
A retry-safe execution system needs persistent state.
An in-memory dictionary is insufficient.
Consider this implementation:
completed_operations = {}
async def execute_payment(operation_id, amount):
if operation_id in completed_operations:
return completed_operations[operation_id]
result = await payment_provider.charge(amount)
completed_operations[operation_id] = result
return result
It looks reasonable until the process crashes.
Or two workers receive the same operation simultaneously.
Or the provider succeeds but the application crashes before storing the result.
In all three situations, the application can execute the payment again.
The operation state must live in durable storage.
A PostgreSQL schema
Here's a simplified schema:
CREATE TABLE tool_operations (
tenant_id UUID NOT NULL,
operation_id TEXT NOT NULL,
task_id UUID NOT NULL,
agent_id TEXT NOT NULL,
operation_type TEXT NOT NULL,
request_hash TEXT NOT NULL,
status TEXT NOT NULL,
provider_reference TEXT,
response_json JSONB,
attempt_count INTEGER NOT NULL DEFAULT 0,
lease_owner TEXT,
lease_expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (tenant_id, operation_id)
);
The operation_id is an application-generated string identity (for example op_9f71c3), not a database-minted UUID β the whole point is that your trusted application owns it and reuses it across retries. The composite primary key ensures that an operation ID is unique within the tenant's operation namespace.
A production implementation should also constrain valid status values, index its recovery queries, and define a retention policy.
The execution state machine
Do not model execution as simply:
SUCCESS / FAILED
That is insufficient for distributed operations.
A provider can process an operation while the application is still waiting for confirmation.
Use more meaningful states:
PENDING
|
v
IN_PROGRESS
|
+-----+------+
| |
v v
SUCCEEDED UNKNOWN
|
v
RECONCILING
|
+------+------+
| |
v v
SUCCEEDED RETRYABLE
|
v
IN_PROGRESS
Other terminal state:
FAILED_FINAL
What do these states mean?
| Status | Meaning |
|---|---|
PENDING |
Operation recorded, not yet executed |
IN_PROGRESS |
A worker currently owns the execution |
SUCCEEDED |
Operation completed and its result is persisted |
UNKNOWN |
The downstream outcome is uncertain |
RECONCILING |
The system is checking the actual downstream result |
RETRYABLE |
A failure was established as safe to retry |
FAILED_FINAL |
The operation was definitively rejected or failed |
Why is UNKNOWN so important?
Because a timeout must not automatically transition an operation to FAILED_FINAL.
That would turn an unknown outcome into a false statement.
And a false statement is exactly how duplicate side effects happen.
6. Implementing a retry-safe execution flow in Python
Let's put the concepts together.
The following is an illustrative Python service design. It uses an operation ledger, a stable operation ID, a request fingerprint, and a provider-side idempotency key.
The database and provider adapters are interfaces; their atomicity and idempotency contracts must be implemented by the underlying services.
First, define the operation:
from dataclasses import dataclass
from decimal import Decimal
from uuid import UUID
@dataclass(frozen=True)
class PaymentIntent:
operation_id: str
tenant_id: UUID
invoice_id: str
amount: Decimal
currency: str
@dataclass(frozen=True)
class PaymentResult:
provider_reference: str
status: str
amount: Decimal
currency: str
Notice that operation_id is part of the trusted application-created intent.
It should not be regenerated just because the model made another tool call.
Step 1: Calculate a canonical request fingerprint
import hashlib
import json
def fingerprint(intent: PaymentIntent) -> str:
payload = {
"tenant_id": str(intent.tenant_id),
"invoice_id": intent.invoice_id,
"amount": str(intent.amount),
"currency": intent.currency.upper(),
"operation_type": "invoice.payment",
}
canonical = json.dumps(
payload,
sort_keys=True,
separators=(",", ":"),
)
return hashlib.sha256(
canonical.encode("utf-8")
).hexdigest()
The fingerprint protects against reusing the same operation ID with different payment parameters.
Do not include volatile fields such as the current timestamp, retry count, or tool-call ID.
Those values change between attempts even when the business operation does not.
Step 2: Claim the operation atomically
Conceptually:
record = await ledger.claim(
tenant_id=intent.tenant_id,
operation_id=intent.operation_id,
request_hash=fingerprint(intent),
)
The claim() method must be implemented using database-level concurrency control.
For example:
- A unique constraint on the operation identity.
- A transaction around insertion or state inspection.
- Row-level locking or an atomic conditional update.
- A lease for the worker that owns the execution.
- A request-hash comparison for existing records.
A second worker must not independently claim the same operation and submit another payment.
Importantly, do not hold a database transaction open while waiting for an external HTTP request. Use a short transaction to claim the work, perform the network operation outside it, and persist the result in another transaction.
Step 3: Return an existing result when available
if record.status == "SUCCEEDED":
return record.response
The operation has already completed.
There is no reason to call the provider again.
For in-progress or uncertain operations:
if record.status in {"IN_PROGRESS", "UNKNOWN"}:
return await reconcile_or_resume(record)
The reconciliation path must check the provider's state or safely resume using the same downstream idempotency key.
It must not blindly create another payment.
Step 4: Execute with a stable provider key
provider_key = (
f"invoice-payment:{intent.operation_id}"
)
result = await payment_provider.create_charge(
invoice_id=intent.invoice_id,
amount=intent.amount,
currency=intent.currency,
idempotency_key=provider_key,
)
The downstream provider's idempotency contract is important here.
It must recognize retries using the same key and apply its documented duplicate-request behavior.
The provider response should then be persisted:
await ledger.mark_succeeded(
operation_id=intent.operation_id,
provider_reference=result.id,
response=result.to_dict(),
)
Step 5: Handle timeouts as unknown outcomes
This is the part that many implementations get wrong.
try:
result = await payment_provider.create_charge(
invoice_id=intent.invoice_id,
amount=intent.amount,
currency=intent.currency,
idempotency_key=provider_key,
)
except TimeoutError:
await ledger.mark_unknown(
operation_id=intent.operation_id,
)
return {
"operation_id": str(intent.operation_id),
"status": "UNKNOWN",
"message": (
"The provider response was not received. "
"The operation is being reconciled."
),
}
The important thing is what this code does not do.
It does not generate a new operation ID.
It does not generate a new provider idempotency key.
It does not tell the model that the payment definitely failed.
It preserves the uncertainty.
A retry is safe only when the system preserves the identity of the original operation and knows how to handle its previous outcome.
7. The most dangerous gap: your database cannot control another company's database
There is a subtle problem with the previous implementation.
Your PostgreSQL database and an external payment provider are two independent systems.
They do not share a transaction.
Consider this sequence:
1. Insert operation record.
2. Set status = IN_PROGRESS.
3. Send request to payment provider.
4. Provider creates charge.
5. Provider returns success.
6. Your application crashes.
7. Result is never written to PostgreSQL.
After restart, your database still says:
status = IN_PROGRESS
But the payment has already happened.
This is a distributed transaction problem.
You cannot solve it by putting another try/except around the HTTP request.
Three ways to handle it
Option A: Downstream idempotency support
The provider accepts a stable idempotency key.
On retry, submit the identical request using the same key, subject to the provider's documented retention and request-matching rules.
This is usually the simplest approach.
Option B: Query by a stable business reference
If the provider supports transaction lookup, associate the external operation with a stable merchant reference.
When the response is lost:
- Query the provider using that reference.
- Check whether the transaction exists.
- Retrieve its actual status.
- Update your local operation ledger.
- Only retry when the provider contract establishes that doing so is safe.
Option C: Keep the operation in an unknown state
What happens when the provider doesn't support idempotency and cannot reliably identify the earlier request?
Do not pretend that you can guarantee exactly-once execution.
Keep the operation in UNKNOWN, prevent an automatic duplicate side effect, and route it to reconciliation or human investigation.
This is much safer than charging someone twice because a timeout looked like a failure.
An important distributed-systems rule
The local idempotency ledger protects your application from repeating its own work.
It cannot independently guarantee that a remote system executed a side effect exactly once.
You need a downstream idempotency contract or a reliable reconciliation mechanism.
8. Why an idempotency key alone is not enough
Let's assume you have implemented idempotency keys.
Are you done?
Not yet.
There are several failure modes worth handling explicitly.
Failure 1: The agent uses a new key on retry
First attempt:
key = payment-001
Second attempt:
key = payment-002
The provider sees two different operations.
Two payments can be created.
Fix: Generate the operation identity in trusted application state and reuse it throughout the workflow.
Failure 2: Same key, different parameters
First request:
{
"invoice_id": "INV-2048",
"amount": 480
}
Retry:
{
"invoice_id": "INV-2048",
"amount": 840
}
Same key. Different intent.
The system should reject this request.
AWS discusses parameter mismatch handling in its idempotent API design guidance. The IETF's current Idempotency-Key Internet-Draft also defines request fingerprints as a mechanism for detecting this situation.
Idempotency-Key HTTP Header Field β draft-08.
Fix: Bind each operation ID to a canonical request fingerprint.
Failure 3: Two workers execute simultaneously
Imagine two workers receive the same queue event.
Both check the operation ledger.
Both see that the operation is not completed.
Both submit the payment.
A normal application-level if statement cannot prevent this race.
Fix: Use a database uniqueness constraint, atomic claims, and worker leases.
For long-running operations, leases should have expiration and ownership checks. A stale worker must not be able to overwrite the result of a newer worker.
Failure 4: The idempotency record expires too soon
Suppose your system retains the operation record for 24 hours.
An agent workflow gets paused for two days.
It resumes and submits the same operation.
The provider no longer recognizes the old idempotency key.
A duplicate may be created.
Fix: Retain the durable business operation record for the required business and replay horizon. Align it with the downstream provider's idempotency retention policy.
Do not assume that provider keys remain valid forever.
Failure 5: The provider succeeds, but your result persistence fails
Provider: SUCCESS
Your database: WRITE FAILED
Agent: NO RESPONSE
This is not a normal failure.
It is an uncertain result.
Fix: Preserve the operation identity and reconcile using the provider key or transaction reference.
The system should never convert an ambiguous external result into an automatic fresh operation.
9. What about queues, workers, and background jobs?
The same problem appears when AI agents use asynchronous execution.
For example:
AI Agent
|
v
Tool Executor
|
v
Message Queue
|
v
Worker
|
v
Payment API
Suppose the worker successfully processes the message but crashes before acknowledging it.
The queue may deliver that message again.
This behavior is not unusual.
Amazon SQS Standard queues provide at-least-once delivery. AWS explicitly recommends designing consumers to be idempotent because a message can be delivered more than once.
Amazon SQS β At-least-once delivery.
AWS Lambda also documents that event-source mappings can process events more than once and recommends idempotent function code.
The important point:
A message being delivered twice must not mean the business operation happens twice.
Introduce an operation consumer
A queue message should carry the stable operation identity:
{
"event_id": "evt_901",
"operation_id": "op_9f71c3",
"operation_type": "invoice.payment",
"tenant_id": "tenant_42"
}
The worker should load the operation record before executing anything.
async def handle_message(message):
operation = await ledger.get(
tenant_id=message.tenant_id,
operation_id=message.operation_id,
)
if operation.status == "SUCCEEDED":
return
await execute_operation(operation)
This example omits the actual concurrency claim deliberately. Production workers must atomically claim or resume the operation, not just read a status and then execute.
Don't confuse an exactly-once queue feature with exactly-once business execution
Some messaging systems offer exactly-once delivery or processing features under defined conditions.
For example, Google Cloud Pub/Sub documents exactly-once delivery for supported pull subscriptions within a region.
But this does not automatically provide exactly-once execution across an external payment provider, your database, and your AI agent.
Google Cloud Pub/Sub β Exactly-once delivery.
A queue can manage message acknowledgment.
It cannot automatically make every downstream side effect part of the same atomic transaction.
10. Use the transactional outbox when execution crosses systems
There is another common failure in event-driven systems.
Your application commits a database transaction and then tries to publish an event.
await db.save_payment(payment)
await queue.publish({
"operation_id": operation_id
})
What happens if the database commit succeeds but the process crashes before publishing?
The payment exists.
The event was never published.
A downstream worker never learns that it needs to continue the workflow.
Reversing the order creates the opposite problem: the event can be published even though the database transaction later fails.
This is known as the dual-write problem.
The transactional outbox pattern addresses it.
The design
Write the business state and an outbox event in the same database transaction.
BEGIN;
INSERT INTO tool_operations (
tenant_id,
operation_id,
operation_type,
request_hash,
status
)
VALUES (
:tenant_id,
:operation_id,
'invoice.payment',
:request_hash,
'PENDING'
);
INSERT INTO outbox_events (
event_id,
aggregate_id,
event_type,
payload
)
VALUES (
:event_id,
:operation_id,
'PAYMENT_REQUESTED',
:payload
);
COMMIT;
A separate dispatcher reads the outbox and publishes events.
DATABASE
|
+-------+-------+
| |
v v
Operation State Outbox Event
| |
+-------+-------+
|
v
Dispatcher
|
v
MESSAGE
QUEUE
|
v
WORKER
Now the application cannot commit the operation record without also recording the event that needs to be dispatched.
There is still an important detail:
The dispatcher can publish an event and crash before marking it as published.
It may publish the same event again.
So the consumer must also be idempotent.
The outbox solves the dual-write problem. It does not magically eliminate duplicate message delivery.
AWS explains this distinction in its Transactional Outbox Pattern guidance.
11. Not every operation should use the same retry strategy
Retrying a failed read operation is very different from retrying a payment.
We need to classify tool operations.
| Operation | Retry strategy |
|---|---|
| Read a customer record | Usually safe to retry, subject to authorization and consistency requirements |
| Search documents | Usually safe to retry |
| Generate an internal draft | Retryable if duplicate drafts are acceptable or deduplicated |
| Create a support ticket | Use a stable operation ID |
| Send an email | Deduplicate using a durable outbound message identity |
| Export a report | Use a stable export job identity |
| Charge a payment | Provider idempotency plus reconciliation |
| Issue a refund | Stable operation identity and provider-level controls |
| Deploy software | Release/deployment identity and environment-level state checks |
| Delete customer data | Durable state tracking and reconciliation |
Retry only errors that are safe to retry
A useful classification:
Potentially retryable
- Temporary connection failures.
- Rate limits.
- Explicit transient server failures.
- Worker interruptions.
Not automatically retryable
- Invalid arguments.
- Authentication failures.
- Authorization denials.
- Business-rule rejections.
- Payment declines.
- Unknown external outcomes.
An important nuance: even HTTP 500 does not necessarily mean that no side effect occurred. The service may have committed the operation and failed while producing the response.
That is why idempotency and outcome reconciliation matter more than simply checking the HTTP status code.
Use exponential backoff and jitter
Don't make every worker retry immediately.
A simple strategy:
import random
def retry_delay(attempt: int) -> float:
base_delay = 1.0
max_delay = 30.0
exponential = min(
max_delay,
base_delay * (2 ** attempt),
)
return random.uniform(0, exponential)
This is full jitter: each retry waits for a random duration between zero and the calculated exponential ceiling.
Also impose:
- A maximum number of attempts.
- A total execution deadline.
- A retry budget.
- Per-provider concurrency limits.
- Respect for server-provided retry instructions when applicable.
Retries are not free.
When a downstream service is overloaded, aggressive retries can increase its load and make recovery harder. Google's SRE book covers exactly this dynamic β how well-intentioned client retries turn a partial outage into a full one β in Addressing Cascading Failures, and AWS discusses the mitigations in Timeouts, retries, and backoff with jitter.
12. Give the agent an honest execution result
There is a small but important interface-design issue.
What does the tool return after a timeout?
Many systems return something like:
{
"success": false,
"error": "Payment request timed out"
}
This response is misleading.
Did the provider reject the payment?
Did the network fail before the request was sent?
Did the payment succeed but the response get lost?
The tool doesn't know.
The model receives a false failure signal and may generate another payment request.
A better response is:
{
"operation_id": "op_9f71c3",
"status": "UNKNOWN",
"message": "The payment outcome is being verified.",
"retry_allowed": false
}
For an operation that is still processing:
{
"operation_id": "op_9f71c3",
"status": "IN_PROGRESS",
"message": "The original operation is still being processed.",
"retry_allowed": false
}
For a completed operation:
{
"operation_id": "op_9f71c3",
"status": "SUCCEEDED",
"provider_reference": "ch_123456",
"amount": 480,
"currency": "USD"
}
The agent should receive the outcome of the original operation, not an invitation to create another one.
Keep the operation identity outside the model's control
I would design the tool interface around this principle:
async def charge_invoice(
invoice_id: str,
amount: Decimal,
currency: str,
execution_context: TrustedExecutionContext,
):
...
The trusted context is injected by the application.
It contains the original operation ID, tenant identity, user identity, authorization context, and task metadata.
The model cannot replace that identity simply by generating a different tool-call ID.
If the model changes the invoice or amount, the application must validate whether the new request represents the same approved operation. A materially different action requires a new authorization and a new operation identity.
13. How to test this properly: break the response on purpose
A normal happy-path test will never expose this problem.
You need fault injection.
I would build the following tests before allowing an agent to perform externally visible operations.
| Test | Injected failure | Expected result |
|---|---|---|
| 1 | Provider succeeds, response is dropped | One operation; outcome reconciled |
| 2 | Application crashes after provider success | Retry returns original result |
| 3 | Two workers receive the same operation | Only one effective side effect |
| 4 | Agent generates a new tool-call ID | Same business operation ID is reused |
| 5 | Same key, different amount | Request rejected |
| 6 | Queue redelivers the same message | No duplicate business effect |
| 7 | Idempotency record expires | Old operation is not silently treated as new |
| 8 | Provider is unavailable | Controlled retry or pending state |
| 9 | Provider has no idempotency support | Ambiguous outcome enters reconciliation |
| 10 | Agent requests a genuinely new payment | New operation ID permits a new intentional action |
The test that matters most
I would start with this exact scenario:
GIVEN:
A payment operation has a stable operation ID.
WHEN:
The provider successfully charges the customer.
AND:
The HTTP response is dropped before reaching the application.
AND:
The agent invokes the same logical operation again
with a different tool_call_id.
THEN:
The system must reuse the original operation ID.
AND:
The provider must receive the same idempotency key
if its API supports this contract.
AND:
The system must return the original payment result
after reconciliation.
ASSERT:
Exactly one payment was created.
This is a much stronger test than asserting that the agent received a successful response.
It verifies the thing that matters: the actual side effect.
Measure more than the HTTP status
Track:
| Metric | Why it matters |
|---|---|
tool_operation_attempts_total |
Shows how often execution is retried |
tool_operation_duplicates_total |
Measures duplicate invocations |
idempotency_replays_total |
Counts requests served from existing operation state |
operation_unknown_total |
Tracks unresolved external outcomes |
operation_reconciliation_duration |
Shows how long uncertainty persists |
operation_duplicate_effects_total |
Detects actual duplicate side effects |
Every attempt should be traceable to the same operation ID.
Log the tool-call ID separately so you can see when multiple model invocations refer to one business operation.
Do not log payment credentials, authentication tokens, or sensitive customer data just to make debugging easier.
14. What about workflows involving several tools?
The problem becomes even more complicated when an agent executes multiple operations.
Imagine an agent handling an order:
1. Create invoice
2. Charge customer
3. Update inventory
4. Send confirmation email
5. Update CRM
What happens if step 4 fails after steps 1β3 have succeeded?
Should the agent restart the entire workflow?
Absolutely not.
That would risk creating another invoice, charging the customer again, and reducing inventory twice.
The correct approach is to treat the workflow as a series of durable, independently identifiable steps.
ORDER WORKFLOW
|
v
CREATE INVOICE
|
v
CHARGE CUSTOMER
|
v
UPDATE INVENTORY
|
v
SEND CONFIRMATION
|
v
UPDATE CRM
Each step should have:
- A stable operation identity.
- Its own durable state.
- Explicit retry rules.
- A recorded result.
- A way to reconcile uncertain outcomes.
For longer workflows, use an orchestration or saga pattern.
A saga coordinates independent operations and defines how to recover when a later step fails.
But remember: a compensation is not a magical rollback.
Refunding a successful payment is a new business operation. Sending a correction email does not erase an email already received. Reversing inventory may require another controlled transaction.
This is why a durable workflow is more reliable than asking the model to repeat the entire plan after an error.
Temporal's documentation makes a particularly useful distinction: its workflow can record a single completed activity result, but an activity itself may execute multiple times during retries and may partially complete more than once.
That is an important distributed-systems lesson for agent developers.
15. A practical production checklist
Before giving an AI agent access to tools that create external side effects, I would verify these requirements.
Operation identity
- [ ] Every business operation gets a stable server-generated ID.
- [ ] Tool-call IDs and operation IDs are separate.
- [ ] Retries reuse the original operation ID.
- [ ] New intentional operations receive new IDs.
Execution state
- [ ] Operation state is stored durably.
- [ ] Database uniqueness prevents duplicate claims.
- [ ] Concurrent workers are coordinated atomically.
- [ ] Worker leases and stale-worker recovery are defined.
- [ ] Unknown outcomes are not marked as failed.
External services
- [ ] Stable provider idempotency keys are used where supported.
- [ ] Same-key requests are checked against the same parameters.
- [ ] Provider key expiry is understood.
- [ ] Provider lookup and reconciliation are available where necessary.
- [ ] Services without deduplication have a safe ambiguous-outcome policy.
Agent orchestration
- [ ] Agent retries cannot create fresh business operation identities.
- [ ] Tool responses distinguish failure, pending, and unknown.
- [ ] Model retries cannot bypass operation state.
- [ ] Sensitive operations have appropriate authorization and approval controls.
Testing and monitoring
- [ ] Response-loss scenarios are tested.
- [ ] Worker-crash scenarios are tested.
- [ ] Concurrent duplicate invocations are tested.
- [ ] Duplicate external side effects are monitored.
- [ ] Unresolved operations have an escalation path.
If you cannot answer these questions, your agent may be safe during a successful demonstration but unreliable during real production failures.
16. The real lesson: retries are not the enemy
We should not disable retries.
Networks fail. Workers crash. APIs become temporarily unavailable. Message queues redeliver events.
Retries are an essential reliability mechanism.
But a retry must represent the continuation of an existing intention, not the creation of a new intention.
That's the distinction AI agent infrastructure needs to get right.
Your agent can call a tool twice.
Your worker can process a message twice.
Your HTTP client can resend a request.
Your provider can receive the same request multiple times.
None of these events should automatically create another business operation.
The execution layer must know what the user originally asked for, what has already happened, and what remains uncertain.
And it must preserve that knowledge even when the model, worker, application, or network restarts.
A model's retry is not permission to repeat a side effect.
The next time you see an AI agent report a timeout, don't immediately ask why the request failed.
Ask a more important question:
Did the operation fail, or did we simply fail to find out that it succeeded?
Because in production, those two situations can cost your customer very different amounts of money.
Further reading
These are the engineering references behind the architecture discussed in this article.
- RFC 9110 β HTTP Semantics, Section 9.2.2: Idempotent Methods
- AWS Builders' Library β Making retries safe with idempotent APIs
- IETF β Idempotency-Key HTTP Header Field, draft-08 (October 22, 2026)
- AWS Lambda Powertools β Idempotency for Python
- Amazon SQS β At-least-once delivery
- AWS Prescriptive Guidance β Transactional Outbox Pattern
- Google Cloud Pub/Sub β Exactly-once delivery
- Google SRE β Addressing Cascading Failures
- Temporal β Activities
One question for the engineering community:
Have you ever seen an API operation succeed while the caller received a timeoutβand what did your system do when the client retried?
I'm particularly interested in how teams handle the UNKNOWN state when an external side effect may already have happened.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.