Payment Webhooks Explained: How to Handle Failed, Duplicate, and Out-of-Order Events

A customer completes an online payment. The payment provider verifies the transaction, although the order is marked as unpaid in your application.

Or worse, the same payment notification arrives twice, triggering duplicate fulfillment.

These problems often originate from how an application handles payment webhooks.

Webhooks allow payment providers to notify applications when transactions change status. However, reliable webhook handling requires more than creating an HTTP endpoint and updating an order record.

Network failures, repeated deliveries, asynchronous processing, and unexpected event ordering can all cause inconsistencies.

In this guide, we'll explain how payment webhooks work, why delivery problems occur, and how developers can build reliable webhook handlers using signature verification, idempotency, durable storage, and background processing.

1. What Are Payment Webhooks?

A payment webhook is an HTTP request that gets sent by the payment provider to the application whenever something interesting happens.

Instead of repeatedly requesting payment status through an API, your backend receives notifications when the provider reports changes.

Common payment events include:

• Payment successfully completed.
• Payment authorization failed.
• Refund processed.
• Chargeback or dispute created.
• Subscription payment succeeded or failed.

For example, a webhook integration may follow this sequence:

1. A customer submits a payment.
2. The payment provider processes the transaction.
3. The provider generates an event.
4. The event is sent to your webhook using an HTTP POST method.
5. Your backend validates and records the event.
6. A worker updates the application's payment or order state.

The webhook informs your application that something happened. It should not automatically be treated as permission to perform every downstream action without verification.

2. Why Payment Webhooks Fail

Even when a payment succeeds, its webhook delivery may fail.

Network and Server Failures

Temporary network interruptions, unavailable servers, or DNS problems can prevent notifications from reaching your application.

Slow Webhook Handlers

If a webhook endpoint performs several database queries, sends emails, and calls external APIs before responding, it may exceed the provider's response deadline.

The provider may interpret the delivery as unsuccessful and retry.

Application Errors

Malformed requests, unexpected event types, and unhandled exceptions can prevent successful processing.

Temporary Database Failures

An event may reach your server while its database is unavailable.

If the application acknowledges the webhook before storing it reliably, the notification may be lost.

Key principle: Receiving a webhook is not the same as successfully processing its associated business operation.

3. Why Duplicate Webhook Events Happen

The payment provider might send you the event multiple times.

Suppose a provider sends a successful payment notification.

Your server updates the order but crashes before returning an HTTP response.

As the payment provider never receives any confirmation, it keeps trying to send the event.

Without duplicate protection, your application might:

• Create a second fulfillment request.
• Send duplicate confirmation emails.
• Record the same payment twice.
• Grant the same subscription entitlement repeatedly.

This is why payment webhook handlers need idempotency.

Idempotency means repeating an operation produces the same intended result without creating additional unintended effects.

Event-Level vs. Business-Level Idempotency

Event-level deduplication prevents your application from processing the same notification ID repeatedly.

Business-level idempotency prevents duplicate operations even when different event notifications refer to the same underlying payment.

Both matter.

For example, two distinct notifications could reference the same successful payment. Recording unique event IDs alone does not guarantee that an order will only be fulfilled once.

A stronger design uses a unique business identifier, such as an order ID or payment ID, to protect fulfillment operations.

4. Verify Webhook Signatures Before Processing

Never assume a request is trustworthy simply because it reaches your webhook URL.

An attacker could submit fabricated payment notifications to an unsecured endpoint.

Payment providers use different authentication mechanisms, including cryptographic signatures, to help applications verify webhook authenticity and integrity.

For signed webhooks, verification generally involves:

1. Receiving the original webhook payload.
2. Reading the provider's signature information.
3. Validating the signature using the appropriate secret or verification key.
4. Rejecting requests that fail verification.

The exact procedure is provider-specific.

For instance, Stripe's Node.js SDK provides constructEvent() for verifying supported signed webhook events.

For this, it needs the original request body as parsing JSON invalidates the signature check.

Example: Verifying Stripe Webhooks in Node.js

The following example uses Express, Stripe's SDK, and PostgreSQL.

Install the dependencies:

npm install express stripe pg dotenv

Create a PostgreSQL table for storing events:

CREATE TABLE webhook_inbox (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  received_at TIMESTAMPTZ DEFAULT NOW()
);

Now create the webhook endpoint:

require("dotenv").config();

const express = require("express");
const Stripe = require("stripe");
const { Pool } = require("pg");

const app = express();
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);

const db = new Pool({
  connectionString: process.env.DATABASE_URL
});

app.post(
  "/webhooks/stripe",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    let event;

    try {
      event = stripe.webhooks.constructEvent(
        req.body,
        req.headers["stripe-signature"],
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (error) {
      console.error("Invalid webhook signature");
      return res.sendStatus(400);
    }

    try {
      await db.query(
        `INSERT INTO webhook_inbox
         (event_id, event_type, payload)
         VALUES ($1, $2, $3)
         ON CONFLICT (event_id) DO NOTHING`,
        [event.id, event.type, event]
      );

      return res.sendStatus(200);
    } catch (error) {
      console.error("Webhook storage failed");
      return res.sendStatus(500);
    }
  }
);

// Register JSON parsing after the webhook route.
app.use(express.json());

app.listen(3000);

The endpoint performs three important operations:

• Validates the webhook signature.
• Stores each event using a unique event ID.
• Returns an acknowledgment only after the database write succeeds.

If the same event arrives again, PostgreSQL ignores the duplicate insertion.

If the database operation fails, the endpoint returns an error so the provider can retry according to its delivery policy.

Important: This is an illustrative receiver, not a complete production payment integration. It assumes a compatible Stripe snapshot-event payload and properly configured credentials, database access, and endpoint security. It also needs a background worker to process stored events.

5. Why You Should Process Webhooks Asynchronously

Webhook handlers should acknowledge valid events quickly.

A good design keeps event reception separate from business logic:

Payment Provider → Webhook Endpoint → Durable Inbox or Queue → Background Worker → Business Database

The webhook endpoint verifies and stores the notification.

A background worker processes it independently.

This design offers several advantages:

• Shorter webhook response times.
• Better handling of temporary service failures.
• Controlled processing during traffic spikes.
• Easier retries and error investigation.
• Reduced risk of losing accepted notifications.

In the PostgreSQL example, the webhook_inbox table acts as a durable inbox.

A worker can select pending events, process them, and update their status.

For concurrent workers, PostgreSQL's FOR UPDATE SKIP LOCKED can help prevent multiple workers from claiming the same event simultaneously.

However, event processing must also be transactionally safe.

A worker crash between fulfillment and marking an event complete can still cause repeated side effects.

Protect the Business Operation

For example, if fulfillment is represented in PostgreSQL, define a uniqueness constraint:

CREATE TABLE fulfilled_orders (
order_id TEXT PRIMARY KEY,
payment_id TEXT NOT NULL UNIQUE,
fulfilled_at TIMESTAMPTZ DEFAULT NOW()
);

A worker can insert the fulfillment record using ON CONFLICT DO NOTHING, making the database operation repeat-safe.

The worker should apply relevant order changes and mark the inbox event processed within a suitable database transaction.

External operations, such as sending emails or calling shipping services, need their own idempotency strategy. A transactional outbox or a fulfillment API supporting idempotency keys can help.

The goal is not merely to prevent duplicate database rows. It is to prevent duplicate real-world actions.

6. How to Handle Out-of-Order Webhook Events

Webhook events are not always delivered in the order they occurred.

Imagine this sequence:

1. A payment attempt fails.
2. The customer retries successfully.
3. The successful payment event reaches your application.
4. A delayed failure event arrives afterward.

If your application blindly applies each notification, the delayed failure could incorrectly change a paid order back to an unpaid state.

A robust webhook implementation should not assume arrival order equals business-event order.

Use Valid Payment State Transitions

Define the states your application supports.

For example:

pending → authorized → paid

Other possible transitions may include:

pending → failed

paid → partially_refunded

paid → refunded

These are illustrative application states, not a universal payment-provider state machine.

A delayed notification should not automatically override a newer confirmed payment state.

Retrieve Current Provider State When Necessary

When event ordering affects a business decision, retrieve the current payment or subscription state from the provider's API.

For instance, before changing an order from paid to failed, the application can verify the actual current status of the associated payment.

Always follow the provider's documented event semantics. Some events represent independent historical facts rather than updates that should overwrite a single status field.

7. Handle Retries Without Creating New Problems

Retries are essential because distributed systems experience temporary failures.

A well-designed webhook consumer should distinguish between transient errors and permanent failures.

Transient errors may include temporary database outages, timeouts, or unavailable dependencies.

These can often be retried safely.

Permanent errors may include invalid payloads, unsupported event formats, or data that cannot be processed without intervention.

Repeatedly retrying a permanently invalid event may create unnecessary load.

For background processing, use bounded retries with exponential backoff and jitter.

After repeated failures, move the event into a failed or dead-letter state so it can be investigated.

Importantly, returning HTTP 200 to a payment provider should mean that the event has been safely accepted, not necessarily that every downstream operation has finished.

If an event cannot be stored reliably, do not acknowledge it as successfully accepted.

8. Test These Payment Webhook Scenarios

Before deploying a payment webhook integration, developers should test more than successful payment transactions. A reliable implementation must handle unexpected events, repeated notifications, and temporary system failures without creating payment inconsistencies.

Valid Webhook Events: Verify that legitimate webhook requests pass signature verification, are stored successfully, and receive an appropriate acknowledgment.

Invalid Signatures: Submit a webhook request with an invalid signature and confirm that the application rejects it without processing the event.

Duplicate Event IDs: Send the same webhook event multiple times and verify that it is not processed repeatedly.

Multiple Events for One Payment: Test different webhook events referencing the same payment and confirm that they do not trigger duplicate order fulfillment.

Out-of-Order Events: Deliver an older notification after a newer one and verify that the application preserves the correct payment status.

Temporary Database Failures: Simulate a database outage and confirm that the webhook endpoint returns an error when it cannot reliably store the event.

Background Worker Failures: Simulate a worker crashing during processing and verify that the event can be retried without duplicating business operations.

Unsupported Event Types: Submit events that the application does not handle and confirm that they are safely ignored or recorded without unintended effects.

High Webhook Traffic: Simulate a sudden increase in incoming notifications and verify that the application can accept, queue, and process events without overwhelming its workers.

Also test recovery after prolonged downtime and ensure monitoring can identify events that remain unprocessed.

Use your payment provider's sandbox or testing environment rather than actual customer transactions.

9. Production-Ready Payment Webhook Checklist

Before deploying, confirm that your implementation:

Uses HTTPS and verifies webhook authenticity.

Preserves the raw request body when required for signature verification.

Stores accepted events durably.

Implements event-level deduplication.

Protects payment-related business operations against duplicates.

Handles notifications arriving out of order.

Processes slow operations outside the HTTP request.

Retries recoverable failures safely.

Logs event IDs, processing status, and relevant errors.

Protects signing secrets and sensitive payment information.

Provides a recovery process for failed or missed notifications.

Avoid logging complete sensitive payment data or exposing webhook secrets in application logs.

These controls help make the payment integration more reliable as transaction volume grows.

Final Thoughts

Payment webhooks are a critical connection between payment infrastructure and application business logic.

A successful integration must do more than receive notifications.

It needs to verify incoming events, preserve delivery information, prevent duplicate operations, handle unexpected ordering, and recover from failures.

The most important design principle is straightforward:

Treat webhooks as reliable inputs only after verification, and design every downstream payment operation to tolerate retries and unexpected delivery order.

That approach helps developers maintain accurate order states, prevent duplicate fulfillment, and build payment systems that remain dependable as businesses scale.

Technical References

The following official resources provide additional technical guidance on webhook implementation, security, and event handling.

• Stripe — Webhooks
• Stripe Node.js SDK — Webhook Signature Verification
• Adyen — Handle Webhook Events
• OWASP — Webhook Security Cheat Sheet

AI Disclosure: This article was generated with AI assistance. Code examples are illustrative and should be independently reviewed and tested before production use.