Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging

The billing job ran at 02:00 like always. The worker charged 1,800 customers, then got killed by a deploy one second before it could mark the batch as done. The scheduler saw an unfinished run and retried it. 1,800 customers were charged twice.

The scheduler did exactly what it promised: at-least-once delivery. The code assumed exactly-once.

TL;DR

• In a system with networks and crashes, you can pick at-most-once (may skip) or at-least-once (may repeat). True exactly-once delivery isn't on the menu.
• "Exactly-once" in product docs usually means exactly-once processing inside one system, under conditions.
• At-most-once loses work silently. At-least-once duplicates work loudly. Loud is easier to fix.
• Make the effect happen once: idempotency keys, unique constraints, and passing keys to downstream APIs.
• My opinion: if your job handler isn't safe to run twice, it has a bug that just hasn't happened yet.

Where the duplicates come from

The crash window is the core problem. Every job has a moment between "did the thing" and "recorded that I did the thing":

1. read job
2. charge card        <- side effect happens
3. mark job done      <- crash here = retry = second charge

Swap 2 and 3 and you get the opposite problem: mark done, crash, never charge. No ordering of these two steps fixes it, because they live in different systems (your DB and the payment provider).

Other sources:

• Timeouts that weren't failures: the HTTP call succeeded, the response was lost, the caller retried.
• Overlapping runs: a slow run is still going when the next tick starts.
• Failover: two scheduler nodes both think they're leader for a few seconds.
• Manual reruns: someone hits "run now" to be safe.

Why not at-most-once?

At-most-once is easy: mark done first, then act, never retry. Duplicates disappear. So do some invoices, emails and webhooks, and nothing tells you which ones. You trade a visible problem for an invisible one.

At-least-once with idempotent handlers gives you the reliability of retries without the duplicates. That's the combination worth building.

Pattern 1: idempotency key per effect

Derive a deterministic key from what you're doing, not when you're doing it:

key = f"invoice:{customer_id}:{billing_period}"   # same on every retry

Store it with a unique constraint before or alongside the effect:

CREATE TABLE effects (
  key        text PRIMARY KEY,
  status     text NOT NULL,          -- pending | done
  run_id     text NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);

def charge_once(customer_id, period, amount):
    key = f"invoice:{customer_id}:{period}"
    claimed = db.execute(
        "INSERT INTO effects(key, status, run_id) VALUES (%s, 'pending', %s) "
        "ON CONFLICT (key) DO NOTHING RETURNING key", (key, RUN_ID)).fetchone()
    if not claimed:
        row = db.execute("SELECT status FROM effects WHERE key=%s", (key,)).fetchone()
        if row.status == "done":
            return "skipped"
        # pending from a crashed run: verify with the provider before retrying
    payments.charge(customer_id, amount, idempotency_key=key)
    db.execute("UPDATE effects SET status='done', updated_at=now() WHERE key=%s", (key,))

The deeper version of this, including the pending state, is in Make scheduled job handlers idempotent: safe to run twice.

Pattern 2: pass the key downstream

The crash window between your DB and someone else's API only closes if the other side deduplicates too. Most payment and messaging APIs accept an idempotency key header. Use the same key you stored:

curl https://api.payments.example/charges \
  -H "Idempotency-Key: invoice:cus_42:2026-11" \
  -d amount=4900

On a retry, the provider returns the original result instead of charging again. If a downstream API has no idempotency support, that's the riskiest call in your job, and it deserves a "check before act" lookup.

Pattern 3: make the receiving side dedupe

If you send scheduled webhooks, include a stable event ID and tell receivers to dedupe on it. If you receive them, store processed IDs. Signatures prove who sent a request, not that it's the first copy. That's why How to verify HMAC-SHA256 signed webhooks pairs signature checks with a timestamp window, and you still want an ID check on top.

Pattern 4: separate "retry" from "give up"

At-least-once needs a limit. After N attempts, move the job somewhere a human will see it instead of retrying forever. That's what a dead-letter queue is for: Dead-letter queues for HTTP jobs.

What about "exactly-once" features?

Some systems offer exactly-once semantics within their own boundary, such as transactional reads and writes inside one log or one database. They're useful, but the guarantee ends where your side effect leaves that system: an email, a card charge, an HTTP call. At that boundary you're back to keys and dedupe.

A scheduler that delivers HTTP jobs, whether that's your own cron box or a service like delay.run, can promise retries and timing. Only your handler can promise the effect happens once. Webhook receivers in particular need to answer quickly so the sender doesn't retry while you're still working, which is the subject of the next post: Return 200 fast.

My take

"Exactly-once" is a property of the whole system, not a checkbox on a scheduler. Design every job assuming it will run twice, sometimes concurrently, and the scheduler's delivery semantics stop mattering much.

Over to you

Where do you keep idempotency keys: same database as the business data, Redis with a TTL, or the downstream API only? I'm curious how long people keep them before cleaning up.