Designing a delivery fallback queue when a gift-card supplier's API goes down

Digital gift-card storefronts live or die on one fragile promise: instant delivery. A customer pays, and a code should land in their inbox seconds later. Behind that promise usually sits a supplier API that provisions codes from inventory. When that API goes down — timeouts, 500s, or a wholesale outage — your checkout doesn't have to go down with it. A delivery fallback queue turns an outage into a delay.

Treat delivery as asynchronous from the start

The root of most outage pain is a synchronous purchase flow: the checkout handler calls the supplier API, waits, and returns either the code or an error. If the supplier hangs, your request threads pile up and the whole storefront slows down.

Instead, split purchase from delivery. At checkout, create an order with a "pending delivery" state and enqueue a delivery job. A worker picks the job up and attempts the supplier call. The customer gets a page that says: "Payment received — your code is on its way." That single separation is what makes every other resilience strategy possible.

Set a retry budget, not an infinite loop

Retries are where naive queues die. Hammering a broken supplier with unlimited retries turns a partial outage into a full one — and burns your API rate limits. Give every job a retry budget:

• Attempt limit: 5–8 tries per order is a sane default.
• Backoff: exponential with jitter — e.g. 30s, 2min, 8min, 30min. Jitter matters because dozens of failed jobs retrying in lockstep create a thundering herd when the supplier recovers.
• Circuit breaker: if 50%+ of attempts to a supplier fail over a rolling 10-minute window, stop attempting for a cooldown period (say 15 minutes) and leave jobs queued. This protects your queue from itself.

When the budget is exhausted, don't silently drop the order. Route it to a dead-letter queue that a human reviews. A code fulfilled manually is infinitely better than a chargeback.

Queue by supplier, not one global pipe

If you source codes from more than one supplier, a single FIFO queue lets one supplier's outage block everyone. Keep separate queues per supplier (and per product family where it matters). A failure in supplier A's queue shouldn't starve supplier B's jobs of worker capacity. Partitioning also makes the circuit breaker per-supplier rather than store-wide, so healthy suppliers keep delivering while the broken one rests.

Communicate status before customers ask

Nobody refreshes an empty inbox quietly. The fallback queue should drive proactive status updates:

1. Immediately: payment confirmed, code is being prepared.
2. After ~10 minutes of delay: still working on it, no action needed.
3. After the job lands in the dead-letter queue: contact support, with an automatic refund if manual fulfillment isn't possible.

Publish an honest delivery-time expectation at checkout too. Stores known for fast, reliable delivery — like Kadokey, built around instant gift-card delivery — set the bar for everyone else. If your baseline is "usually under a minute," your status pages and email copy should say exactly that, and flag when you're outside it.

Reconcile, then refund

Every fallback flow ends in one of two states: fulfilled or refunded. Schedule a reconciliation pass (every few minutes during an outage) that finds orders still undelivered past your SLA and either re-queues them or triggers an automatic refund. Refunds that arrive before the customer files a chargeback protect both your revenue and your payment processor standing.

A minimal architecture

You don't need exotic infrastructure: a durable queue (Postgres-backed, Redis Streams, or SQS), a worker pool with per-supplier partitions, a circuit breaker, a dead-letter queue, and status callbacks. The whole design fits in a focused weekend of work — and it's the difference between an outage that costs you a weekend and one that costs you customers.