Idempotency Isn't Enough — What I Got Wrong About M-Pesa Reconciliation

A few weeks ago I wrote about handling M-Pesa STK Push timeouts and webhook verification. In the comments, Justin Wilson pushed back on the reconciliation snippet I'd shared:

"The key missing safeguard is idempotency. Store transaction keys and make retries safe so delayed callbacks cannot create duplicate orders."

He was right, and it turned out to be more right than either of us knew at the time. A few days after that post, one of my own production payments got stuck in exactly the way the article described. Chasing it down didn't just confirm Justin's point, it surfaced a second bug hiding behind the first one: a naive reconciliation job can be perfectly idempotent and still leave a paying customer with nothing.

This is the writeup of what actually happened, and the fix that came out of it.

The setup

A customer initiated an M-Pesa payment through STK Push. They got the prompt on their phone, entered their PIN, and — nothing. No confirmation in the app. I checked the payment record: PENDING. I checked the logs for anything payment-related: nothing. Not a single line.

That absence of logs was the first real problem. Before I could fix anything, I had to be able to see what was happening at all.

Finding the actual failure

Adding logging at the charge-initiation point immediately explained the "no logs" mystery, and revealed the shape of the bug:

Paystack M-Pesa charge response: http=200 status=true dataStatus=pay_offline
display=Please complete authorization process on your mobile phone

The charge had initiated correctly. pay_offline is Paystack's way of saying "waiting on the customer to finish on their phone" — this is the expected state right after the STK prompt goes out. The problem was downstream: the webhook that's supposed to confirm the final result never arrived, or never got processed. The payment sat in PENDING because nothing ever told it otherwise.

This is exactly the failure mode from my original post: the callback drops, the transaction is stuck, and only a background reconciliation job can recover it.

So I built one. Almost.

The fix that wasn't quite a fix

I added a scheduled job that sweeps PENDING payments, re-verifies each one directly against Paystack, and updates the status. Structurally, it looked a lot like the snippet from my first post:

async function reconcilePendingTransactions() {
  const pending = await db.payment.findMany({ where: { status: 'PENDING' } });

  for (const tx of pending) {
    const result = await verifyWithPaystack(tx.reference);

    if (result.status === 'success') {
      await db.payment.update({
        where: { id: tx.id },
        data: { status: 'SUCCESS' },
      });
    }
  }
}

I also added the thing Justin flagged: real safeguards against duplicate processing. Ticket issuance is guarded by an "already issued for this payment?" check. Stream access grants are an idempotent upsert keyed on user + resource. Running this job twice, or on three server replicas at once, can't create two tickets for the same payment or double-grant access. On that specific point, Justin's fix was already correct and already built in.

The job ran. It found the stuck payment. It verified it with Paystack: real success, real money moved. It flipped the row to SUCCESS. And the customer still didn't get their ticket.

The bug underneath the bug

Look again at that reconciliation snippet, and at the excerpt from my original post's version too. Both do the same two things, in the same order:

1. Confirm the payment succeeded with the provider.
2. Mark the row SUCCESS.

Fulfilment — actually granting whatever the customer paid for — happens somewhere else, assumed to follow from step 2. In my production code, it did follow, almost every time. This time, it threw an error partway through (a tenant-context bug in how the job ran, unrelated to payments specifically), and by the time it failed, the payment row already said SUCCESS.

That's the failure mode neither Justin's comment nor my original post's snippet accounts for: once a payment is marked SUCCESS, nothing ever looks at it again. The reconciliation job only sweeps PENDING rows. A payment that's SUCCESS-but-unfulfilled isn't stuck in the sense the job was built to detect — it's stuck in a way that's invisible to it. The customer paid. The system believes the transaction is closed. Nobody is delivered anything, and nothing is watching for that specific gap.

Idempotent retries protect you from doing fulfilment twice. They don't protect you from marking something settled before fulfilment happens even once.

The actual fix

The fix is to invert the order: fulfil first, and only mark the payment SUCCESS once fulfilment has actually succeeded.

async function reconcilePendingTransactions(
  onPaymentSettled: (payment: Payment) => Promise<void>,
) {
  const pending = await db.payment.findMany({ where: { status: 'PENDING' } });

  for (const tx of pending) {
    const result = await verifyWithPaystack(tx.reference);
    if (result.status !== 'success') continue;

    try {
      // Grant the ticket / stream access / whatever they paid for FIRST.
      // Must be idempotent — this can run more than once for the same
      // payment, exactly as Justin described.
      await onPaymentSettled(tx);

      // Only mark SUCCESS once fulfilment has actually happened.
      await db.payment.update({
        where: { id: tx.id },
        data: { status: 'SUCCESS' },
      });
    } catch (err) {
      // Fulfilment failed. Leave the row PENDING. The next run will pick
      // it back up and try again — this is what makes the retry safe,
      // and it's why fulfilment has to be idempotent in the first place.
      console.error(`Fulfilment failed for ${tx.reference}, will retry`, err);
    }
  }
}

One line moved — the fulfilment call went from implicit ("something else handles this after") to explicit and sequenced before the status update — and it closes a gap that idempotency alone doesn't touch. The two ideas work together: idempotency makes retrying safe, and fulfil-before-settle is what guarantees a retry actually happens when something goes wrong.

I traced through both my production system and a payments package I maintain separately, and found the identical bug in both: a reconciliation job that marked things settled before confirming delivery, with a // TODO: connect this to your fulfilment logic comment standing in for something that was never actually wired up. Comments don't retry. Only code does.

What this actually looked like once it was fixed

In production, this runs as a scheduled job across three server replicas behind a load balancer, coordinated with a simple distributed lock so only one replica runs the sweep at a time:

[replica-1] Scheduled reconcile: another replica holds the lock — skipping.
[replica-2] Scheduled reconcile: checked=1 synced=1
[replica-3] Scheduled reconcile: another replica holds the lock — skipping.

And when I deliberately broke fulfilment to test the failure path:

Fulfilment failed for OTV-xxxx — reverting to PENDING for retry

The row went back to PENDING, exactly as intended, and the next tick picked it up and retried cleanly once the underlying bug was fixed. That log line is the actual proof this works — not "the tests pass," but "watch it fail safely and recover."

The short version, for anyone who just wants the rule

If your reconciliation job (or your webhook handler, for that matter) can mark something SUCCESS and fulfilment can independently fail, you have this bug, whether or not your fulfilment logic is idempotent.

• Idempotent fulfilment protects you from double-delivering when a retry happens.
• Fulfil-before-settle is what guarantees a retry happens at all.

You need both. I had shipped code — and sold code — with only the first one, and it took a real stuck payment to notice the second one was missing.

Thanks to Justin for the original comment. It was right, and chasing down exactly how right it was is what turned into this post.

If you're integrating Paystack, Flutterwave, or M-Pesa STK Push in Node.js and want the fixed version of this pattern (fulfilment hook required, throws until you wire it in, so it can't ship silently broken) rather than building it from scratch: African Payment Gateways Engine. The free africa-payments-utils package covers the phone-normalization and webhook-signature pieces from my first post, MIT licensed, no strings attached.