How to Keep Billing State Correct When Webhook Events Repeat, Arrive Late, or Fail

When building apps in production, every billing provider you decide to use tells your app what happened to a customer, such as a payment that succeeded, a plan that changed, or a subscription that ended. Your app then turns each message into a change in its own database: a plan column, a monthly allowance, or a flag that lets someone in.

Those messages travel over a network that duplicates, delays and drops, so the same payment message can arrive three times, a cancellation can arrive after the upgrade that replaced it, and a server can stop half way through a message. Each case goes on to corrupt a customer's state in a different way, and none of them shows up when you test with one clean message.

Which brings me to a question a developer friend of mine asked me last week: how does an app keep a customer's plan and allowance correct when the events that change them can repeat, arrive out of order, or stop in the middle?

This post builds the answer as a small pipeline: an endpoint that stores each event before it answers, a queue in Postgres that holds the stored events, and a worker that applies them with effects that are safe to repeat.

To show this in practice, I built an example app called "Bottle", it's a reminder app with a free plan and a paid plan using Kinde under the hood to send billing events.

Kinde is a platform for authentication and billing, and here it is the billing provider whose plan and payment changes the app must follow. After building the app, I tested the pipeline against the events Kinde sent for my own account and I'll be breaking it all down in this article.

What goes wrong when a handler applies each message as it arrives

First, we will start with the handler most apps write, which reads the message, changes the database, and answers 200 (OK). That works on a quiet day with a clean network, but three things can potentially break it.

The same event arrives more than once. If the app applies a payment and the 200 answer never reaches the provider, the provider sends the payment again. The handler adds the monthly allowance a second time. And will do the same a third time if a third copy arrives.

An old event arrives after a newer one. A customer upgrades, so the old plan is cancelled. The cancellation happened first, but a retry delays it, and it arrives last. A handler that only trusts the order of arrival will set the customer back to Free.

The server stops in the middle. A plan change needs two writes: the plan, then the allowance that goes with it. If the process stops between them, the customer is on Pro with a Free balance. The provider retries, but by then the state is already split.

I wrote a handler that applies each message on arrival and ran nine deliveries of three events through it. The nine are the plan change three times, the payment five times, and one old cancellation. The worker stops once during the first plan change, and this is what the account looked like along the way:

• After delivery 2, the plan was Pro and the balance was still 10, because the process stopped before the allowance write.
• After delivery 7, the plan was Free, because the old cancellation arrived last and won.
• After delivery 9, the balance was 2,950. The correct balance is 500.

The account ended on Pro only because the plan change happened to arrive a third time. The handler had no rule that made it come out right, and the rest of this post builds those rules.

What the provider promises

A fix has to match what the provider actually does, so I read Kinde's webhook documentation before I designed anything. These are the parts that matter for correctness:

• The body of each request is a JWT (a signed token) with the content type application/jwt. You check its signature against the keys Kinde publishes at /.well-known/jwks on your Kinde domain.
• Every request carries a webhook-id header. Kinde describes it as unique to a webhook and an event, and the same on every attempt. The docs tell you to use it as the idempotency key.
• A 200 answer means delivered. Any other answer, or no answer, makes Kinde try again.
• The retry schedule is roughly: immediately, then after 5 seconds, 30 seconds, 2 minutes, 10 minutes, 1 hour and 4 hours, then every 12 hours until 36 hours have passed.
• An endpoint that fails for 72 hours is disabled, and the account owner gets an email.

That list explains the three failures. The promise is "we keep trying until we hear 200", so a copy arrives whenever your 200 gets lost, even if your app applied the event correctly. A retry that waits 10 minutes also arrives after every event sent in those 10 minutes, which is how order is lost. The docs do not promise an order for events on one endpoint, and the retry schedule shows why no one could. The pipeline assumes no order and expects copies.

The event itself carries what the handler needs. Here is a customer.plan_changed event from my account, with the fields I use and the rest left out:

{
  "type": "customer.plan_changed",
  "event_id": "event_01a1…",
  "timestamp": "2026-10-21T02:29:24.481368869+11:00",
  "event_timestamp": "2026-10-21T02:29:23.403633+11:00",
  "data": {
    "user_id": "kp_4940…",
    "customer_id": "customer_0198…",
    "customer_type": "user",
    "plan": { "key": "customer_pro_plan" }
  }
}

The plan is an object, and its key is at data.plan.key. The user and customer ids tell the handler which account to change. The two timestamps matter later, in the section on ordering.

The pipeline

The design follows from two facts. An event can arrive many times, so the app needs one stored copy and a counter. An event can also arrive at any time, so the app must apply it in a way that does not depend on when. Four rules do that work.

Rule 1: Store the event, then answer

The endpoint does three things and nothing else: it checks the signature, writes the event to a table, and answers 200. Applying the event happens after the answer.

If the signature is not valid, the endpoint answers 401 and stores nothing. If the database write fails, the endpoint answers with an error, and Kinde tries again later. The 200 therefore means one thing: the event is stored. It does not mean the event is applied. That difference is what lets Bottle survive a crash, because a stored event can run again and a lost event cannot.

The signature check is one call to the jose library, which fetches Kinde's keys once and caches them. After the check, the event goes into inbox_events:

INSERT INTO inbox_events (source, dedupe_key, type, payload, occurred_at, …)
VALUES ($1, $2, $3, $4::jsonb, $5, …)
ON CONFLICT (source, dedupe_key)
DO UPDATE SET seen_count = inbox_events.seen_count + 1
RETURNING id, (xmax = 0) AS inserted

Rule 2: One key for each event

The webhook-id header is the dedupe key, and the table has a unique constraint on it. The first copy inserts a row. Every later copy hits the conflict, adds one to seen_count, and does nothing else.

The last line of the query tells the caller which case happened. In Postgres, xmax is 0 for a row the statement just created, so (xmax = 0) is true only for a new event. Bottle also writes one row to a receipts table for every delivery. That gives the app a record of all nine copies even though only three events exist.

I considered keeping seen ids in memory or in a cache. Both lose the ids when the process restarts, and a restart is exactly when copies arrive. The key has to live in the same database as the event.

Rule 3: Apply each event in one transaction, with a lease

A worker claims stored events and applies them. Two workers can run at once, for example when the endpoint starts one after its answer and a timer starts another, and they must not take the same event, so the claim uses a clause Postgres provides for this:

UPDATE inbox_events
   SET status = 'processing', attempts = attempts + 1,
       locked_until = now() + interval '30 seconds'
 WHERE id IN (
   SELECT id FROM inbox_events
    WHERE (status IN ('received', 'failed') AND next_attempt_at <= now())
       OR (status = 'processing' AND locked_until < now())
    ORDER BY occurred_at, id
    FOR UPDATE SKIP LOCKED
    LIMIT 25)
RETURNING *

SKIP LOCKED makes a second worker pass over rows the first worker holds. The 30-second lease handles a worker that dies. When the lease ends, the row is claimed again.

The worker then runs the handler and marks the event applied inside one transaction. Either both happen or neither does. If the process stops half way, the database rolls back, and the event runs again from the start. That removes the third failure from the list, because the plan write and the allowance write can no longer be separated.

When a handler throws, the event goes back to failed and waits. The waits are 5 seconds, 30 seconds, 2 minutes, 10 minutes and 1 hour. After six attempts the event is marked dead, which keeps it visible without running it forever.

I chose Postgres over a message queue for one reason. The claim, the effect and the mark must share a transaction, and a separate queue cannot join one. Billing events are also low in volume, so one more service would add cost without adding safety.

Rule 4: Make every effect safe to repeat

The first three rules reduce how often a handler runs twice, but only this rule makes a second run harmless, and it has to protect two effects.

The allowance is a ledger row, and the row has a key made of the account and the month. Writing it again does nothing:

INSERT INTO ledger (account_id, period, delta, reason, entry_key)
VALUES ($1, $2, $3, $4, 'pro:' || $1 || ':' || $2)
ON CONFLICT (entry_key) DO NOTHING

Both the payment handler and the plan change handler try to write this same row. Whichever event arrives first grants the Pro allowance, and the other finds the row and skips it. That is why a payment that arrives before its plan change does no harm. The plan change grants the allowance when it comes.

The plan has its own protection, and it depends on time rather than arrival.

Arrival order is not the order things happened

Each stored event has an occurred_at time taken from the event, not from the moment the copy arrived. A plan change applies only if its time is newer than the plan change already stored:

UPDATE accounts SET plan = $2, plan_changed_at = $3
 WHERE id = $1 AND plan_changed_at < $3
RETURNING id

If the update changes no row, the handler records ignored: a newer plan change is already stored and stops.

The diagram shows the effect. The cancellation happened three minutes before the upgrade, but it arrives seventh. The database already holds a newer plan change, so the cancellation does nothing and the customer stays on Pro.

The event has two timestamps. In the two events I stored, event_timestamp was about one second earlier than timestamp. I read the first as the time of the change and the second as the time Kinde sent the message. Bottle uses event_timestamp when it is present and falls back to timestamp. The choice matters, because the guard compares times, and a time that means "when it was sent" would reorder events that were sent late.

Replay and ignored events

Two features come from the same idea: a stored event is a fact you can look at again.

An ignored event keeps its reason. When a handler chooses to do nothing, it writes why. The log shows "ignored: a newer plan change is already stored", not a silent skip. An event the handler cannot use shows up the same way.

A replay puts a stored event back in the queue. It sets the status to received and lets the worker run it again. Replay is safe because of Rule 4. A replayed payment finds its ledger row and writes nothing. A replayed plan change finds a newer change and is ignored. Replay is how you run stored events again after a handler changes or after an outage, and it needs no new code path.

In Bottle, each billing event on the account page has a Replay link. The route behind it only accepts events that belong to the signed-in account, because replay should not let one customer requeue another customer's events.

What happened with live events

The tests cover the rules, but I also wanted events sent by Kinde itself. In Kinde, I moved my own user from Pro to the Starter plan and back to Pro. Kinde sent the events to Bottle's live endpoint. To test a late delivery, I then replayed the older Starter change after the Pro change had applied.

The log on the account page shows the result in the screenshot above. The Pro change reads "plan set to pro, Pro allowance granted". The older Starter change reads "ignored: a newer plan change is already stored". The account shows the Pro plan and 500 bottles for the month.

The same nine deliveries from the first section run through the pipeline on Bottle's Engine Room page. The page uses the same intake code and handlers as the live endpoint, on a fresh test account each time.

The numbers are these:

• 9 copies arrived and 3 events were stored. The other 6 copies were counted and dropped.
• 1 plan change stopped in the middle. It stayed stored, and the next copy ran it to the end.
• The late cancellation was ignored, because a newer plan change was stored.
• The account ended on Pro with a balance of 500, and the allowance was granted once.

The handler that applies on arrival ended with 2,950. That is 2,450 more bottles than the plan allows, and nothing in the database records how they got there.

Limits

This design has edges.

• Order depends on the provider's clock. The guard compares event times from Kinde. Two plan changes with the same timestamp are treated as a tie, and the second one is ignored.
• The live log shows two event types. The plan change and the cancellation came from real Kinde events. The payment handlers and the overdue-invoice handler follow the documented event names and are covered by tests, but I did not watch Kinde send them.
• The token's timestamp has no age limit. Kinde's docs mention checking the timestamp for replay protection. Bottle does not reject old tokens. The dedupe table already refuses a second copy of any webhook-id, but it does not expire a token.
• A dead event needs a person. After six attempts, an event is marked dead and stays visible. Bottle does not page anyone. A production system needs an alert on that status.
• Postgres is the queue. It suits the low volume of billing events. A stream of millions of events a day is a job for a message queue.

The rule that holds it together

The stored event is the fact, and a delivery is one attempt to tell you about it. Once the app treats it that way, a copy becomes a counter, a late event becomes a comparison of times, and a crash becomes a retry. None of the three needs special code, because each one is the same case: a stored event that has not been applied yet.

To run it yourself, clone the repository, run npm test for the 23 tests, and run npm run dev for the app. The Engine Room page runs the nine deliveries in your browser.

After testing it out yourself, do drop a comment below telling me your findings, even if you test it with a completely different billing provider.

FAQ

Why answer 200 before the event is applied?
The provider stops retrying once it hears 200. Applying an event can be slow or can fail. If the app answers only after applying, every failure becomes a retry of an event that may be half applied. Storing first makes the answer fast and keeps the event safe.

Which key should I dedupe on?
Use the key the provider documents. For Kinde, that is the webhook-id header, which stays the same across attempts. Store it with a unique constraint in the same database as the event.

Can I use a queue like SQS instead of Postgres?
Yes, but the claim, the effect and the "applied" mark can no longer share one transaction. You then need the effects to be safe to repeat on their own, which Rule 4 requires anyway.

How do I test retries and reordering?
Write down a sequence of deliveries that includes copies, a late event and a crash, and run it against the handler. The Engine Room page does this, and the repository has the same sequence as a test.

Sources

• About webhooks, Kinde docs
• Manage customer activity webhooks, Kinde docs
• Bottle source code