Should You Take Orders When the Payment Gateway Is Down?
Black Friday is close, which means a lot of teams are about to find out which parts of their checkout they actually depend on. A typical order touches more systems than people expect. Inventory has to be reserved. Payme
Black Friday is close, which means a lot of teams are about to find out which parts of their checkout they actually depend on.
A typical order touches more systems than people expect. Inventory has to be reserved. Payment has to be authorized. A shipping label gets created. The confirmation email goes out. Somewhere on the page, a recommendation service suggests what else the customer might want. Each of those is a separate service, often run by a separate team or a separate company.
Under normal traffic, all of them answer quickly and nobody thinks about the order they run in. Under peak traffic, one of them will slow down or fail. The useful question isn't whether that happens. It's whether your team has already decided what the order flow should do when it does, or whether that decision gets made at 2 a.m. by whoever is on call.
Not every step deserves the same treatment
The easiest way to see this is to go through the order steps one at a time and ask a blunt question about each: if this fails, should the customer's order fail too?
Recommendations are the easy case. If the recommendation service times out, show bestsellers or show nothing. Nobody has ever abandoned a purchase because the "you might also like" row was empty. This step should have a short timeout and permission to be skipped.
The confirmation email is almost as easy. The order is real whether or not the email went out. If the email provider is struggling, queue the message and send it later. A confirmation that arrives twenty minutes late is fine. An order that failed because of an email is not.
The shipping label can usually wait too. Labels are needed before the warehouse ships, not before the customer clicks "buy." If the carrier's API is overloaded, the order can be accepted and the label created once the API recovers.
Inventory is where things get serious. If you can't confirm stock, you risk selling the same last unit to five people. Most teams treat a failed inventory reservation as a reason to stop the order, or at least to show a clear "can't confirm availability" message.
And then there's payment.
The payment question
If the payment gateway is down or timing out, you have roughly two options.
You can fail the order. The customer sees an error and maybe tries again later. Your books stay clean, nobody gets goods they didn't pay for, and you lose some sales. On a day like Black Friday, a customer who hits an error often just buys from someone else.
Or you can accept the order and hold the payment for retry. The customer sees "order received," the order sits in a pending state, and the system keeps trying to authorize the payment once the gateway recovers. You keep the sale, but you take on new problems. Some of those retries will fail because the card is declined, and now you have to contact a customer who thinks they've already bought something. Inventory reserved for a held order has to be released if the payment never goes through. Fraud checks that normally happen at checkout now happen after the fact.
Neither option is free. The right answer depends on your margins, what you sell, how often your gateway actually has trouble, and how good your team is at cleaning up afterwards.
Writing the decision down
Whatever you choose, the worst version is the one where the choice lives only inside scattered try/catch blocks. At that point the behaviour during an outage is whatever the last person to touch the code happened to write.
It helps to describe the order flow somewhere that makes each step's failure rule explicit. Here's an illustrative example. It's pseudo-config, not any specific tool's syntax.
workflow: place_order
steps:
- name: reserve_inventory
call: inventory.reserve
on_failure: fail_order
- name: authorize_payment
call: payments.authorize
retry: { max_attempts: 3, backoff: exponential }
on_failure: hold_for_retry # or fail_order. This is the real decision.
- name: create_shipping_label
call: shipping.create_label
on_failure: retry_later
- name: confirmation_email
call: email.send_confirmation
on_failure: retry_later
- name: recommendations
call: recs.for_order
timeout: 300ms
on_failure: skip
The format matters less than the result. Anyone on the team can read this and know what happens when payments fail, without reading the payment service's code. If the business decides on Thursday night to switch from hold_for_retry to fail_order, that's one line, not a hunt through three services.
The same file also shows which steps can safely fall behind. Retries for labels and emails can be held in a queue and worked through after the rush, so the slow parts of the system don't drag down the parts customers are waiting on.
Before the traffic arrives
If you only do one thing this week, take your order flow and label every external call with one of three outcomes: fail the order, retry later, or skip. Then check that the code actually does what the label says. Gaps between the two tend to show up only under real load, which is the worst time to find them.
It's also worth running a small drill. Point one dependency at a stub that times out, place a test order, and watch what happens. If nobody can predict the outcome before you try it, that's the step to sort out first.
So, what would you do?
Back to the title. When the payment gateway is down during your busiest hour, do you fail the order or accept it and retry the payment later?
If your team has lived through this, I'd like to know which way you went, whether you'd choose the same way again, and what the cleanup looked like afterwards.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.