Designing Reliable Financial Webhooks: Retries, Signatures, and Event Ordering

Webhooks are a convenient way to notify applications when something changes. A payment processor can notify a merchant about a completed payment, a billing service can report an invoice update, or a financial platform can send an account-status event.

But a webhook is not the same as a guaranteed, exactly-once message.

Requests can time out. Receivers can be temporarily unavailable. Events can be delivered more than once or arrive out of order.

For financial systems, these cases need explicit handling rather than assumptions.

This tutorial walks through a practical webhook design covering authentication, duplicate delivery, retries, and event ordering.

1. Define the Delivery Contract

Start by deciding what the sender guarantees.

A typical webhook system may provide at-least-once delivery: an event can be sent again if the sender does not receive a successful acknowledgement.

This means a consumer must be prepared to receive the same logical event more than once.

Document the event contract:

• A unique event ID
• An event type
• A schema version
• An occurrence timestamp
• A resource identifier
• A delivery or retry policy

For example:

{
  "id": "evt_92ac",
  "type": "payment.completed",
  "version": 1,
  "occurred_at": "2026-10-28T10:15:00Z",
  "data": {
    "payment_id": "pay_741",
    "status": "completed"
  }
}

This is illustrative JSON, not a standard shared by every provider.

The event ID identifies the logical event. A delivery attempt should not receive a new logical event ID simply because the request is retried.

2. Verify the Signature Before Processing

A webhook endpoint exposed to the internet must not trust incoming JSON merely because it resembles a valid event.

A common approach is to sign the raw request body using an HMAC secret shared between the sender and receiver.

The sender computes a message authentication code; the receiver independently calculates the expected value and compares it with the supplied signature using a constant-time comparison.

A simplified Python example:

import hashlib
import hmac

def verify_signature(
    raw_body: bytes,
    supplied_signature: str,
    secret: bytes,
) -> bool:
    expected = hmac.new(
        secret,
        raw_body,
        hashlib.sha256,
    ).hexdigest()

    return hmac.compare_digest(
        expected,
        supplied_signature,
    )

This example assumes the sender uses a plain hexadecimal HMAC-SHA256 signature. Real providers may use a different header format, include timestamps, or use another signing scheme.

Always follow the provider's documented protocol.

Important safeguards include:

• Verify the exact raw bytes, not re-serialized JSON.
• Keep signing secrets outside source control.
• Rotate secrets using a controlled procedure.
• Reject invalid signatures before applying business changes.
• Apply timestamp and replay protections when supported by the protocol.

A valid signature proves that the message was signed by someone with the shared secret. It does not, by itself, prove that the event is new or that it arrived in the correct order.

3. Make Event Processing Idempotent

Suppose the sender delivers evt_92ac, but the receiver processes it successfully and then crashes before returning HTTP 200.

The sender may retry. If the consumer executes the same financial action again, it could create duplicate ledger entries or repeat an unintended operation.

Use a durable event record with a uniqueness constraint.

For example:

CREATE TABLE webhook_events (
    event_id TEXT PRIMARY KEY,
    event_type TEXT NOT NULL,
    received_at TIMESTAMP NOT NULL,
    processed_at TIMESTAMP NULL,
    status TEXT NOT NULL
);

The primary key prevents two records with the same event ID from being inserted.

However, the table alone does not make the entire workflow idempotent. The event claim, business-state update, and resulting database changes need an appropriate transaction boundary.

Conceptually:

1. Verify the signature.
2. Validate the event schema.
3. Begin a database transaction.
4. Claim the event ID using a unique constraint.
5. If it was already processed, return the appropriate acknowledgement.
6. Apply the business-state transition.
7. Mark the event as processed.
8. Commit the transaction.

If the business update fails, the transaction should not leave a misleading processed marker.

For external side effects—such as sending another webhook, calling a payment provider, or sending an email—use a durable job or transactional outbox pattern rather than assuming a database transaction can roll back a remote request.

4. Acknowledge Quickly and Process Reliably

Webhook senders often expect a response within a limited time.

If the receiver performs slow business logic before responding, a timeout may trigger unnecessary retries.

A common architecture is:

Incoming webhook
       |
       v
Verify signature
       |
       v
Validate envelope
       |
       v
Persist event durably
       |
       v
Return successful acknowledgement
       |
       v
Background worker
       |
       v
Apply business transition
       |
       v
Record result and metrics

The key condition is that the event must be durably stored before the receiver acknowledges it.

Otherwise, the endpoint could return success and lose the event if the process crashes before persistence.

This design separates delivery from processing. It also makes retries and operational recovery easier to reason about.

5. Design a Bounded Retry Policy

Retries help when failures are temporary, but unlimited rapid retries can overload an unhealthy service.

A practical retry policy may use exponential backoff with jitter.

For example:

import random

def retry_delay(
    attempt: int,
    base_seconds: float = 1.0,
    max_seconds: float = 300.0,
) -> float:
    ceiling = min(
        max_seconds,
        base_seconds * (2 ** attempt),
    )
    return random.uniform(0, ceiling)

This function calculates a delay for an attempt number. It does not implement a complete retry scheduler.

A production system also needs:

• A maximum retry window or attempt count
• Clear classification of retryable failures
• Durable retry state
• A dead-letter or manual-recovery path
• Monitoring for events that repeatedly fail

The correct policy depends on the sender's delivery contract. If the provider controls retries, the receiver should follow the provider's documented acknowledgement rules rather than inventing a conflicting retry mechanism.

6. Do Not Assume Events Arrive in Order

Consider two events for the same payment:

1. payment.pending
2. payment.completed

If the second event arrives first and the first arrives later, blindly applying each event could incorrectly move the payment back to a pending state.

Possible safeguards include:

Monotonic state transitions

Define which transitions are valid. A completed payment should not return to pending simply because an older event arrived late.

Resource versions

If the sender provides a monotonically increasing resource version, reject or ignore stale updates according to the documented contract.

Per-resource sequencing

Where strict ordering is required, partition work by resource ID or use a sequencing mechanism that preserves order for that resource.

Reconciliation

Periodically compare local state with an authoritative source to detect discrepancies.

Timestamps alone may not be sufficient to establish event order. Clock differences, delayed delivery, and equal timestamps can make them unreliable as the only ordering mechanism.

7. Test Failure Scenarios, Not Just the Happy Path

A webhook integration can pass its basic test and still fail under realistic delivery conditions.

Include tests for:

• Invalid signatures
• Missing required fields
• Duplicate event IDs
• Database failure during processing
• Worker crashes after claiming an event
• Delayed and out-of-order events
• Temporary downstream failures
• Retry exhaustion
• Secret rotation
• Unexpected event versions

For idempotency, test the same event repeatedly and verify that the final business state is equivalent to processing it once.

For ordering, generate sequences in different arrival orders and assert that the final state follows the documented state-transition rules.

Also test recovery after a process restart. Durable event handling should not depend on an in-memory queue surviving a crash.

8. Monitor the Entire Lifecycle

HTTP status codes alone do not tell you whether an event was eventually processed correctly.

Useful metrics include:

• Signature verification failures
• Events accepted and rejected
• Duplicate event deliveries
• Processing latency
• Retry counts
• Dead-letter queue size
• Oldest unprocessed event age
• Business-state reconciliation failures

Use correlation IDs to connect delivery logs, queue records, and processing traces.

Avoid logging signing secrets, authentication headers, or unnecessary personal and financial data. Observability should make failures diagnosable without exposing sensitive information.

Conclusion

Reliable financial webhooks require more than an endpoint that returns HTTP 200.

Verify signatures, persist events before acknowledging them, handle duplicate delivery with durable idempotency, define retry behaviour, and protect business state from out-of-order events.

The most important design principle is to make failure behaviour explicit. A well-designed integration should remain understandable when requests time out, workers crash, and the same event arrives multiple times.

Disclosure: This article was prepared with AI assistance. The author should validate the examples, adapt them to the actual provider protocol, and add their own engineering experience before publishing.