Building Resilient Webhook Handlers in Node.js Without Crashing Your Express Server

Most developers treat incoming webhooks like standard API endpoints: the request comes in, the server processes the data, updates the database, sends out notifications, and finally returns a response.

In development, this looks completely fine. In production, it eventually brings your server to a halt.

Here is why that architecture breaks down under real traffic, and how to fix it without overcomplicating your setup.

The Three Traps

1. Gateway Timeouts

Third-party platforms typically expect a response within 5 to 10 seconds. If your database query hangs, an email service lags, or your server is busy, the remote provider drops the connection and marks the delivery as failed.

2. The Retry Storm

When a provider thinks your endpoint failed, its system automatically retries. If your server was already struggling with one slow task, dozens of automatic retries will flood your endpoint simultaneously, causing a cascading freeze.

3. Duplicate Deliveries

Webhooks operate on "at-least-once" delivery. Network blips, provider resends, and server timeouts mean you will inevitably receive the exact same payload multiple times. If your system assumes every event is unique, you end up with duplicate orders, double charges, or spam notifications.

The Production Rule: Fast ACK, Asynchronous Work

The solution comes down to changing your execution order:

1. Verify first: Confirm the request payload matches the provider's signature so you aren't processing fake data.
2. Acknowledge immediately: Return a success status code right away. Do not wait for database writes, file uploads, or outbound emails.
3. Process in the background: Offload the actual heavy lifting to a background task or message queue.
4. Enforce idempotency: Always store the event ID in your database. Before running any action, check if that ID has already been recorded. If it has, discard the event immediately.

By separating acknowledgment from processing, your webhook endpoint stays fast, survives third-party timeouts, and handles traffic spikes without falling over.

How does your current backend handle incoming events—do you process them inline, or offload them straight away?