🎃 The Real Nightmare Isn't a Bug. It's a Bug Nobody Notices. 🎃

Imagine deploying a new feature.

All tests pass. CI is green. No errors in the logs. Everything looks fine.

Then a user reports that their payment was processed twice.

Here's a simplified example of what can go wrong:

async function processPayment(order) {
  try {
    return await paymentAPI.charge({
      amount: order.total,
      customerId: order.customerId
    });
  } catch (error) {
    if (error.code === "TIMEOUT") {
      return await paymentAPI.charge({
        amount: order.total,
        customerId: order.customerId
      });
    }

    throw error;
  }
}

Looks reasonable, but there's a problem.

The first request might have reached the payment service and completed successfully. Only the response got lost.

The retry sends another charge request.

Now imagine this happening with a real customer.

One way to prevent this is to use an idempotency key:

async function processPayment(order) {
  return await paymentAPI.charge({
    amount: order.total,
    customerId: order.customerId,
    idempotencyKey: `payment-${order.id}`
  });
}

Now the payment service can recognize repeated requests for the same operation and return the original result instead of charging the customer again.

Of course, this depends on the payment provider correctly supporting and implementing idempotency.

That's what makes production bugs interesting. Sometimes the code works exactly as written. The problem is that nobody considered what would happen under real-world conditions.

A green test suite is good news, but it doesn't mean you've tested every possible failure.

Sometimes the scariest bugs are the ones that never show up in your test results. 👻

Happy Halloween! 🎃