The Delivery ID Gap: Deduping Webhook Replays When the Provider Sends No ID

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

The comment that exposed the gap

The rate-limit thread on this series converged on a clean pipeline: verify the signature first, rate-limit second, dedup third. A reader (rushdeploy) then pushed it one layer further — store the provider's delivery ID and reject repeats, so replays fail even while both secrets still verify during a secret-rotation grace window.

He was right. And it immediately exposed a gap in my own setup: some of my providers don't send a delivery ID at all.

GitHub sends X-GitHub-Delivery. Stripe sends the event id. But a few smaller providers I integrate with send… nothing. No request ID, no event ID, just a signed payload. My dedup store had a "delivery_id" column and no plan for the rows that never get one.

The fallback hash (and why it's weaker)

The fallback is a deterministic hash of timestamp-normalized payload + sender key:

• Timestamp-normalized matters: some providers resend the same event with a new sent_at field. Normalize the fields you don't care about before hashing, or every retry looks like a new event.
• Sender key matters: the HMAC-verified identity (API key header, tenant id) must be part of the key. Without it, two senders firing identical payloads collide — and worse, during the grace window an attacker's replay and a legit sender's natural retry can hash to the same thing across senders.

Be honest about what this buys: it kills accidental double-processing, not malicious replay. Without a real delivery ID, a natural retry and a replayed capture are indistinguishable. The hash closes the "provider retried and I processed twice" hole, which in practice is the common case. It does nothing against an attacker holding the leaked secret, who mints fresh signatures with fresh hashes. That's the same caveat as the delivery-ID path — the dedup layer only narrows the replay window, it never replaces verification.

The observability upgrade: count the fallback, don't just log it

The naive move is to log "fallback triggered" and move on. The useful move is to increment a per-provider counter every time the hash fallback fires.

The number you actually want is the fallback rate per provider, not raw hits. Here's why: a provider that used to trigger the fallback 100 times a day and suddenly goes quiet didn't get quieter — they probably started sending real delivery IDs. That's your signal to tighten that path: switch the dedup key for that sender from the hash to the real ID, and now you have a strong replay guard where you used to have a weak one.

Raw hit counts can't tell you that story. A rate that drops to zero can. In n8n I keep this in static data or my metrics sink, keyed by sender — one increment per fallback, one dashboard panel per provider. Cheap, and it turns "weaker dedup" from a permanent condition into a migration list.

Treat the fallback window as weaker evidence

While the fallback is active for a sender, the dedup evidence is weaker — so I deliberately run a weaker dedup window for those senders:

1. Shorter grace TTL. The dedup store TTLs at grace-window length plus clock skew; for fallback-only senders I shrink the window itself. A shorter window means a smaller blast radius for the key I trust less.
2. Stricter per-sender rate bucket. Pair the weaker dedup with a tighter rate limit on that sender. It doesn't close the hole — nothing does without a real ID — but it bounds how much damage fits through it.

The principle: match the strength of your defenses to the strength of your evidence. Verified delivery ID → full grace window, normal buckets. Hash fallback → short window, strict buckets. No ID at all → the review queue, not the dedup store (a human decides, per the sweeper rules).

The shape of the whole thing in n8n

One Code node after verification, before the limiter:

// Dedup node: real delivery ID when you have one, hash fallback when you don't
const crypto = require('crypto');
const sender = $json.verified_sender;           // from the verify step
const deliveryId = $json.headers['x-delivery-id'] ?? $json.body.event_id ?? null;

let key, strong;
if (deliveryId) {
  key = `${sender}:${deliveryId}`;
  strong = true;
} else {
  // timestamp-normalized payload: drop provider-added volatile fields
  const { sent_at, received_at, _meta, ...stable } = $json.body;
  const digest = crypto.createHash('sha256')
    .update(JSON.stringify(stable) + ':' + sender).digest('hex');
  key = `${sender}:fb:${digest}`;
  strong = false;
  // observability: the fallback rate per provider is the metric
  const c = ($getWorkflowStaticData('global').fb ??= {});
  c[sender] = (c[sender] ?? 0) + 1;
}

const seen = $getWorkflowStaticData('global').dedup ??= {};
if (seen[key]) throw new Error(`REPLAY_GUARD: duplicate ${strong ? 'delivery' : 'hash'} key ${key}`);
seen[key] = Date.now();   // TTL'd by the sweeper: grace window (+ skew), shorter when !strong

Two callouts from the comment thread that survived into the code: the counter increments on every fallback trigger (not just logged), and the TTL behind the sweeper knows whether the key was strong. Strong keys get the full grace window; fallback keys get the short one.

The layering, restated

This series keeps arriving at the same shape, because it's the shape that works:

1. Verification answers forgery: is this request really from the sender?
2. Keyed rate limit answers hammering: even real senders get bounded.
3. Dedup answers replay: did I already see this exact event?

None substitutes for the others. And within dedup, the delivery ID is the strong key and the hash is the weak one — the system's defenses should know which it holds, count the weak ones, and shrink its trust in them accordingly. A dedup layer that can't tell strong from weak evidence is honest about nothing, and the honest part is the whole feature.

The checklist I run now

• [ ] Delivery ID extracted per provider; null is an expected value, handled, not an error
• [ ] Hash fallback: timestamp-normalized payload + verified sender key
• [ ] Every fallback trigger increments a per-provider counter (rate, not raw hits)
• [ ] Watch for a provider's fallback rate going quiet → switch that sender to the real ID
• [ ] Dedup TTL = grace window + clock skew; shorter window for fallback-only senders
• [ ] Stricter per-sender rate bucket while fallback is active
• [ ] No ID and no safe hash → review queue, not the dedup store

I'm building AI Automation Lab — production-ready n8n workflow templates. The webhook hardening patterns from this series ship in the paid templates; my open-source patterns live at github.com/zhp910318/n8n-ai-automation-lab.

Try the free starter: AI Hook Generator — free n8n workflow template, MIT licensed on GitHub.