The Workflow That Retried a Dead Provider 200 Times: Backoff, Jitter, and Circuit Breakers in n8n

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

Two hundred retries, zero recoveries

One morning a provider went down, and my workflow did exactly what I had told it to do: retry on failure. By the time the provider came back, the workflow had retried the same failed call two hundred times. Nothing had recovered. The retries had burned quota, filled the log with identical errors, and — because the retry loop held the execution open — delayed every job queued behind it.

The retry had not been a recovery strategy. It had been a way to turn one outage into two hundred identical failures, each one logged as if it were new information.

A retry is not a recovery plan

The mistake was treating "retry" as a single setting. In n8n, Retry On Fail is easy to switch on, and for a transient blip it works: one retry, a second later, and the call lands. But a retry policy is really three decisions, and the default answers all three badly: how long to wait between attempts, how many attempts to make, and when to stop trying altogether.

An immediate retry assumes the failure was a blip. A provider outage is not a blip. Retrying every few seconds against a dead provider does not help the provider recover, and it does not help you either — it just converts your waiting time into load on a system that is already down, and on your own queue, which fills with jobs that cannot proceed.

Backoff with a cap, and respect Retry-After

The first fix is backoff: each attempt waits longer than the last. Mine doubles the wait — 1 second, 2, 4, 8 — up to a cap of five minutes. The cap matters as much as the doubling. Uncapped exponential backoff turns a ten-minute outage into a job that retries once an hour and finishes a day late, long after anyone still wants the result.

Two details I keep explicit. First, if the provider returns a Retry-After header, that header wins over my backoff schedule. The provider is telling me when it will be ready; ignoring that to retry on my own schedule is just impatience with extra steps. Second, the backoff clock belongs to the job, not to the workflow run. A job that has already waited four minutes should not restart its backoff from one second because a new execution picked it up.

Jitter, or every job retries at once

Backoff alone has a trap that only shows up at scale: if fifty jobs fail at the same moment, they all back off on the same schedule, and they all retry at the same moment. The provider comes back, gets hit by fifty retries in the same second, and goes down again. This is the thundering herd, and it is how a recovery becomes a second outage.

The fix is jitter: add a random spread to each wait, so retries land across a window instead of on a tick. I use full jitter — the wait is a random value between zero and the backoff cap for that attempt — because it spreads the load widest. The cost is that some individual retries wait longer than the nominal schedule. That is a good trade: one job waiting an extra minute is cheap; fifty jobs retrying in the same second is expensive.

The circuit breaker: stop trying, loudly

Backoff and jitter make retries polite. They do not answer the question that matters most: when do you stop retrying at all? That is the circuit breaker's job.

After a set number of consecutive failures — mine is five for provider calls — the breaker opens, and the workflow stops calling that provider entirely for a fixed window. Jobs that arrive while the breaker is open do not retry; they go straight to the deferred queue with a clear reason: provider circuit open. When the window ends, the breaker goes half-open and lets exactly one probe call through. If the probe succeeds, the breaker closes and normal traffic resumes. If it fails, the breaker opens again for another window.

Three things make this work instead of just adding another layer of waiting. First, the breaker state lives outside any single execution, in the same Postgres table as my job queue, so a restart does not reset the failure count. Second, opening the breaker is an alert, not a log line — "provider X circuit open after 5 failures" pages me, because a provider that has failed five times in a row is a fact I need, not a detail to discover later. Third, the deferred jobs are not lost: they sit in the queue with their original payloads, and when the breaker closes they run in order, instead of having been retried two hundred times into a dead provider.

Not every failure deserves a retry

None of this applies to every error. A retry policy that treats all failures the same will retry the ones that can never succeed. My rule, carried over from earlier articles in this series: 429 and 5xx and timeouts are retryable, with backoff; 4xx validation errors and 401/403 auth failures are not — retrying a bad payload or a dead credential just burns attempts and hides the real problem. The breaker counts only retryable failures, so a burst of validation errors does not trip it and a dead credential does not get two hundred retries either. Each failure class gets the recovery it actually needs, not the one the retry setting defaults to.

The test is a dead provider, not a slow one

Testing this on a provider that is merely slow proves almost nothing, because a slow provider still answers eventually. The test that matters is a mock provider that fails every call for ten minutes, then recovers. Assert four things. First, the total attempt count stays inside the budget — five attempts, not two hundred. Second, the gaps between attempts grow and spread, not cluster on a tick. Third, after the fifth consecutive failure no further calls go out until the breaker window ends, and the deferred jobs are visible in the queue with their reason attached. Fourth, the single probe call after the window is the only call that goes out before the breaker closes, and the queued jobs run once each when it does. Count calls at the mock provider, not executions in n8n — the execution count tells you how many times you tried, and the provider count tells you how much damage you did.

The principle

A retry is a bet that the next attempt will succeed, and every bet costs quota, log space, and queue time. Backoff makes the bet cheaper, jitter stops fifty bets landing at once, and the circuit breaker knows when to stop betting altogether. The goal is not to retry harder. It is to retry only while retrying could plausibly work, and to say so, loudly, the moment it cannot.

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.