AI Agent Run Leases: Stop Stale Workers From Repeating Side Effects

An agent run can look dead, get claimed by a replacement worker, and then wake up just long enough to send the email you thought the replacement was handling. A queue lease helps, but it is not the whole safety boundary.

The reliable pattern is a run lease plus a fencing token. The lease decides who may work right now. The fencing token lets the system that performs a side effect reject an older worker even if that worker still has credentials and a network connection. This guide shows how to use both for long-running AI agent jobs.

The failure a normal retry cannot fix

Imagine an agent that prepares and sends a contract summary:

1. Worker A claims run r_42 for 60 seconds.
2. A pauses during a model call or loses its connection to the database.
3. Its lease expires, so Worker B claims the run and receives a newer ownership token.
4. B finishes the review and sends the summary.
5. A comes back and continues from an in-memory checkpoint. It sends the summary again.

Nothing has to be malicious or badly coded for this to happen. A slow garbage-collection pause, a network partition, a container reschedule, or a long tool call is enough. Message systems deliberately make work visible again after a visibility timeout so it can recover. AWS explicitly notes that a consumer that runs past that timeout can cause another consumer to process the same message; at-least-once delivery is a normal constraint, not an edge case (AWS guidance).

For a read-only task, duplicate work wastes tokens. For a write, it can duplicate a ticket, overwrite a CRM field, or send a second customer message. Treat a run lease as an ownership protocol, not as a timer around a model call.

What a run lease should guarantee

A lease is a time-bounded claim on a specific run. Store the minimum state in a durable database or queue, never only in worker memory.

The core rule is simple: a worker may execute only while its lease is valid and its fence is the current fence for that run. Do not let a worker infer continued ownership because it started the task.

Choose a lease duration from your failure budget

Short leases recover quickly but need more renewals. Long leases reduce renewal traffic but leave work unavailable longer after a crash. Start with a duration slightly above the p99 time between durable checkpoints—not the whole run time.

For example, a 45-second lease with a 15-second heartbeat works when the agent saves after each tool call or model stage. An eight-minute uncheckpointed tool call does not become safe because the lease is eight minutes; it merely delays recovery.

The same trade-off appears in managed queues. Azure Service Bus returns a message for redelivery when its lock expires, and supports renewal for active processing (Microsoft documentation). Use that behavior as a delivery primitive, then implement application-level ownership around agent state.

Add a fence: the part most designs miss

A lease alone protects the scheduler. It does not protect a downstream API from a stale worker.

Each successful claim increments a fence number:

Worker A claims r_42 -> fence 17
Lease expires
Worker B claims r_42 -> fence 18
Worker A wakes up -> must be rejected

Pass run_id and fence to every internal side-effect boundary. That boundary stores the highest fence it has accepted for the run and rejects lower values. This is called fencing because each replacement owner raises a barrier behind it.

The critical comparison belongs where the effect happens, not only in the orchestrator. A stale worker can miss a failed heartbeat, keep an old HTTP client alive, or retry after a process restart. The email service, write proxy, or domain transaction must be able to say: “fence 17 is older than the accepted fence 18; no action.”

Claim atomically

Here is a compact PostgreSQL-style claim. The transaction makes availability, ownership, and the fence update one decision.

UPDATE agent_runs
SET state       = 'running',
    lease_owner = :worker_id,
    lease_until = now() + interval '45 seconds',
    fence       = fence + 1,
    updated_at  = now()
WHERE run_id = :run_id
  AND state IN ('queued', 'waiting', 'running')
  AND (lease_until IS NULL OR lease_until < now())
RETURNING run_id, fence, lease_until, checkpoint;

If no row returns, do not work. Another worker owns the run, the run is terminal, or it is not ready. Avoid a read-then-write claim; two workers can both observe an expired timestamp before either writes.

When a worker resumes from a checkpoint, record the checkpoint version and only move forward. Replaying a model call is usually acceptable. Repeating an external write is not. Split planning from committing so the side effect has its own durable operation record.

Renew conservatively and stop on lost ownership

Renewal must be conditional on the owner and fence you originally acquired. A timer that extends any lease on the row can revive an owner that has already been replaced.

type Lease = { runId: string; workerId: string; fence: number };

async function renewLease(db: Db, lease: Lease): Promise<boolean> {
  const result = await db.query(
    `UPDATE agent_runs
       SET lease_until = now() + interval '45 seconds', updated_at = now()
     WHERE run_id = $1
       AND lease_owner = $2
       AND fence = $3
       AND state = 'running'
       AND lease_until > now()
     RETURNING lease_until`,
    [lease.runId, lease.workerId, lease.fence]
  );
  return result.rowCount === 1;
}

If renewal returns false, immediately enter lost ownership mode:

• stop scheduling new model or tool calls;
• cancel work that supports cancellation;
• do not commit a pending result;
• write diagnostic evidence only if that write is itself safe; and
• discard local state after a bounded cleanup period.

Do not “try one last time.” Ownership has changed or can no longer be proven. A worker should be pessimistic here. This is also why an agent needs cancellation semantics separate from a UI button: stopping the visible stream does not automatically stop a background action.

Fence the side-effect boundary

Suppose agents send customer email through an internal delivery service. Make the service accept an idempotency key and a fence, then enforce both in one transaction.

INSERT INTO agent_effects (run_id, effect_key, fence, status, payload)
VALUES (:run_id, :effect_key, :fence, 'accepted', :payload)
ON CONFLICT (run_id, effect_key)
DO UPDATE SET
  fence = EXCLUDED.fence,
  payload = EXCLUDED.payload,
  status = 'accepted'
WHERE agent_effects.fence < EXCLUDED.fence
RETURNING fence, status;

This sketch needs a policy choice. For an irreversible send, it is often safer to accept only the first fence that reaches a sent state and reject all later attempts. For a replaceable draft, a newer fence may supersede an earlier accepted draft. Make that choice explicit per effect type.

An external vendor may not understand fences. Put a small internal outbox or action proxy in front of it. The proxy can store (run_id, effect_key, fence), enforce the rule, and use its own idempotency key with the vendor. Never assume the external call failed just because your request timed out; reconcile its result before retrying.

Design the agent workflow around durable boundaries

The safest agent loop has short, recoverable segments:

1. Claim the run and receive its fence.
2. Load a versioned checkpoint and validate the input again.
3. Run a bounded model or tool step.
4. Persist the output, tool evidence, and next checkpoint transactionally.
5. Renew before starting the next segment.
6. Before a side effect, re-check ownership and send the fence to the action boundary.
7. Persist the confirmed result, then mark the run terminal.

This gives a replacement worker something concrete to resume. It also prevents a long context transcript from becoming the only record of what happened. Keep the checkpoint purposeful: selected inputs, validated intermediate result, operation IDs, and the next permitted step. Do not persist hidden reasoning as a substitute for observable evidence.

Where leases fit with related controls

Leases are not a replacement for the rest of your reliability design.

Use them together. A dead-letter queue holds a task that needs a decision; a lease and fence make sure that decision is not undermined by an old worker continuing in the background.

Test the ugly cases before traffic finds them

Happy-path tests prove very little. Add deterministic failure tests to your agent runtime:

• Pause Worker A after it reads a checkpoint; let its lease expire; let B claim the run; resume A and verify the action proxy rejects A’s lower fence.
• Drop renewal responses while the database still processes them; ensure the worker stops if it cannot prove ownership.
• Make a model call last longer than the lease; ensure it cannot commit a tool action after losing the lease.
• Timeout after the external service may have accepted a request; verify reconciliation happens before any replay.
• Restart a worker between “effect accepted” and “effect confirmed”; confirm that a replacement uses the idempotency key instead of sending again.
• Cancel a run during a tool call; verify both the run state and the action boundary reject later work.

Record the run ID, fence, worker ID, lease expiry, checkpoint version, external operation ID, and final disposition in each trace. Those fields make a production incident answerable without guessing from logs.

A practical rollout path

You do not need to rebuild every agent at once. Start with the highest-risk write action: email, record update, deployment, payment, or a browser action that changes customer data. Add an action proxy with idempotency and fence checks. Then add run leases to the workers that invoke it. Finally, split long runs into checkpointed stages.

Measure lost-lease events, rejected stale-fence attempts, renewal failures, and time spent in recovery. A nonzero stale-fence rejection count is not automatically a bug; it can be proof that the boundary stopped a duplicate. Investigate trends, especially when a single model or tool stage routinely outlives its lease.

The useful mental model is blunt: leases are permissions with an expiry date. A worker that cannot prove it still holds permission must lose the ability to act. That one rule turns a queue timeout from a duplicate-side-effect incident into a recoverable handoff.

FAQ

What is an AI agent run lease?

An AI agent run lease is a time-limited ownership claim for one background run. Only the worker holding the unexpired lease may advance the run, renew it, or request a side effect.

Why is a fencing token needed if I already use a queue lock?

A queue lock protects delivery, but a worker can continue after its lock expires or after a network partition. A fencing token lets the downstream action boundary reject that stale worker’s older ownership generation.

How long should an agent lease be?

Set it slightly above the p99 interval between durable checkpoints, then renew well before expiry. Avoid using one long lease for a whole multi-minute run; split the work into checkpointed stages instead.

Should read-only agent tasks use fencing tokens?

Usually the cost of duplicate read-only work is lower, so an idempotency key and lease may be enough. Add fencing when stale output could overwrite newer state, drive a costly downstream job, or create an observable user-facing result.

What should happen when an agent loses its lease?

It should stop issuing new work, cancel interruptible calls, avoid committing local results, and allow the current owner to resume from durable state. Do not retry a side effect from the old worker.

Can a fence prevent every duplicate external action?

No. If an external provider does not enforce your fence and a request times out after it may have succeeded, you still need an idempotency key and reconciliation. The fence protects your own boundary; reconciliation resolves uncertain outcomes.