Retry the Step, Not the Job

Quick scenario. A customer places an order. Your code texts them, then emails the receipt. Two API calls, one after the other. What could go wrong?

Quite a lot, as it turns out.

The bug

Here's the job, sketched in a few lines:

async function notifyCustomer(order) {
  await sendText(order);   // works
  await sendEmail(order);  // email service hiccups, throws
}

// somewhere else
await retry(() => notifyCustomer(order), { attempts: 3 });

The text goes out. The email fails. The retry wrapper does exactly what it was told and runs notifyCustomer again, from the top. Another text. Another failed email. One more round, and the email finally lands.

The customer now has three identical texts and a receipt, and is wondering if they got charged three times. You paid for every one of those texts.

Nothing crashed. The retry just didn't know the text was already sent.

The fix: retry the step, not the job

Treat the two calls as separate steps in one sequence. The text finishes and gets marked done. The email fails, and only the email is tried again.

A few things make that retry behave:

• Backoff. Wait a few seconds, then longer if it fails again, so a struggling service gets room to recover.
• A plan B. Decide in advance what happens if retries run out. Maybe a push notification tells the customer what they needed to know.

One text, one message that reaches them, no confusion.

Then traffic shows up

Same shop, Black Friday, hundreds of orders at once. Most APIs cap how many calls you can make. A bulk email service might allow 100 a minute, for example. Go over and calls get rejected, and rejected calls get retried, so the pile-up grows.

The calmer approach is a queue: extra calls wait their turn and go out at a pace the API accepts. Nobody gets dropped.

The boring stuff that bites later

Two more things show up eventually. The text service wants a phone number in one shape and the email service wants customer details in another, so someone has to reshape the data between calls. And when someone asks ""did order 4521 get its email?"", you want to open the run and see every step, how long it took, and how many retries it needed. Not grep logs and guess.

Where this leaves you

You can build all of this by hand, and plenty of teams do. It's just the same plumbing every time. Unmeshed has it built in: chaining, retries with backoff, fallbacks, rate-limit queuing, data transformation between calls, and a dashboard for latency, success rates and retry counts.

How do you handle retries across steps in your own code? Curious what's worked."