Money paths that fail closed: what I changed on a legacy payments platform

I own architecture and security for a payments and mobility platform in East Africa. Three markets, five payment rails (three mobile-money providers, two card processors), a PHP monolith that predates me, and MongoDB as the system of record. Not the stack I would pick on a blank page, but a bank-issued card and real wallets run on it, so the job was to make the money paths boring before anything else.

This is what that looked like. No company names, and I have kept to what is actually shipped. The open items are at the end.

Rule one: claim before effect

Every money operation claims a unique, one-time key before it does anything. A second attempt with the same key is refused: no second charge, no second ledger entry. The primitive is small:

key = idem.key("outstanding_deduct", trans_id)   # deterministic, namespaced
if key == "":  legacy_path()                     # unsafe/empty input -> old behaviour, logged
try:   idem.claim(key)                           # insert into a dedicated collection,
                                                 # unique partial index on key
except DuplicateKey:                             # Mongo 11000 = someone already did this
       return ok_noop                            # ack, do nothing
deduct(trans_id)                                 # the effect comes strictly after the claim

It is wired across roughly 45 money operations. Different sites use whichever dedupe fits: an atomic status compare-and-set (Paid -> Syncing before a rider credit), a unique insert on a repay key for mobile-money callbacks, the provider event id for webhooks, and the processor's own idempotency header on live card top-ups.

The subtle decisions are not in the happy path:

• Duplicate = refuse. That is the fail-closed part and it is non-negotiable.
• Claim write itself fails (infra fault) = let the money through, log loudly. We chose not to block legitimate payments because the idempotency store had a bad minute. That is fail-open for one specific fault and it is documented as such.
• Crash after the claim, before the effect = the operation is un-retryable. It lands in a logged reconcile queue instead of a retry loop. The window is narrow; it is not zero.

The P0 we found while wiring this: a shared driver-debt deduction path had no idempotency at all on the non-primary rails. One guard now covers every rail.

Proof was cheap and worth doing: eight unit tests pin the key contract, a review confirmed claim-before-effect ordering at every site, and a live gate test showed the first insert winning and the duplicate returning 11000 with exactly one row left behind.

Callbacks: never trust the body

Mobile-money providers call you back. Rules we enforce on that path:

• Callbacks are accepted only from the provider's network (IP allow-list). The bank rail's signature check is built and waiting on production values.
• On confirmation we credit the stored push amount, never the amount in the callback body. Body-vs-stored mismatch is rejected and logged.
• A re-delivered success is a no-op that still returns an ACK, so the gateway stops retrying.
• Card webhooks are signature-verified; an invalid signature is rejected. Hosted-checkout returns are verified server-side with the provider before any credit, and a background reconciler sweeps the credits the return call missed.
• Repeated fraud declines increment a counter and switch card top-up off past a threshold.

The callback recorder

On 1 November an unmatched callback was lost. We could not say what the bank had sent. The fix went live the next morning: every payment callback from the bank partner leaves a redacted row, matched or not. Ids, result code, amount, receipt, last four digits of the phone, a matched_pending flag, the action taken and why.

Two design points. The recorder runs after the 403 gate, so it never records unauthenticated noise. And its write is isolated from the decision, so a recorder fault cannot change the reply to the bank. The failure conditions are explicit: no row, a row with a full phone number or any card number, a credit without a row, or a 500 on the callback path. An admin health view shows the services up or down, the event table, and Force/Hide actions with a 14-day purge; a concurrent Force gets a 409, and only root can grant the Force right.

What the money audit found

Four live defects, all shipped fixes:

1. A wallet double-count, fixed with an atomic compare-and-set.
2. An admin adjustment that could push a wallet negative.
3. A top-up race.
4. A double-send.

Debit, credit and ledger entry now commit in one multi-document transaction on the critical paths, on a three-node replica set with majority-acknowledged writes. Every top-up, payout and card charge also appends a de-duplicated, best-effort audit event that never rolls back the money; the audit log is hash-chained and strips PAN, card, CVV and OTP keys on the way in.

Releases that refuse to be hand-copied

The incident that justified the release gate: a payment cron job was dead for nine days because a load-time compatibility fault was swallowed silently. A separate hand-copy to the live host left a stale dependency map and more than a thousand stray files.

The live release path now:

1. Compile and load every module in a sandbox.
2. A separate sign-off flag for any change touching money code (MONEY_RELEASE=yes, set by a human).
3. Build the new version alongside the running one and switch in a single instant.
4. Automatic post-release health checks.
5. Automatic rollback if a check definitively fails; the last several releases stay on disk for an instant manual revert.

Hand-copied code is refused; only releases built from the shared repository are accepted. Access is SSH key-only, elevation is scoped to release actions, and every use is logged.

Keeping card data out of the building

Card entry happens in the processors' hosted sessions and hosted fields. The platform receives a session id or a token and keeps token id, last four, expiry and scheme. Payment pages have no card inputs, load one allow-listed third-party script, and relay results by postMessage to an allow-listed origin. Retired card-handling endpoints return 410.

We checked rather than asserted: a scan of all 218 collections found no CVV value and no card-number field; a scan of 79 log files (3.4 million lines, 15 days) found zero Luhn-valid numbers. The one exception, three 2017-era payout records with card-shaped values in a bank-account field, was masked to last four. Request bodies are not logged, phone numbers are redacted in callback paths, the bank OAuth token cache is sealed, and backups are encrypted with the key held offline.

Honest caveat: one issuer card-reveal flow passes clear PAN and CVV through the platform for a single request. It is gated by KYC, a 300-second single-use OTP token, ten reveals per hour, no-store, and an audit write that must succeed before the reveal is returned. Whether that stays or moves to an issuer-hosted view is an open decision with the bank and the assessor.

Seeing the quiet failures

Prometheus, Grafana with Alertmanager, Loki for logs, Tempo traces through an OpenTelemetry collector, exporters for host, database and web server. The piece I would carry to any platform is the silent-failure detector: it classifies response envelopes so that a money or auth failure returning a 200 still raises an alert. The legacy system had none of this.

Still open

• Move the money-critical surfaces (wallet, ledger, settlements, card transactions) to PostgreSQL for constraints and versioned migrations; Mongo keeps rides, location and telemetry.
• A balance-reconciliation job, and re-queuing a handful of stalled ledger fan-out rows.
• Enforce MFA for every admin (available, not yet enforced everywhere), central write-once logs with 12-month retention, file-integrity monitoring, a Luhn-checked log scrubber, an external pen test, and the assessor-led review.
• The idempotency collection grows without bound. It is tiny; a TTL is a future option, not a problem today.

A question for you

When the idempotency store itself is down, does your money path fail open or fail closed? We chose open for that one fault and closed for everything else. I would like to hear where others drew the line, and what it cost them.

Eric Gitangu, Nairobi. Site: https://developer.ericgitangu.com. LinkedIn: https://www.linkedin.com/in/ericgitangu. GitHub: https://github.com/ericgitangu.