My crawler said 1 in 7 websites were down. They weren't, and retrying made it worse.

I'm building Fresh Leads, a tool that finds local businesses and grades their websites so agencies know who to pitch. One of the strongest signals it reports is simple:

"Their website is down."

If you sell web design, that's the best cold-call opener there is. It's also the worst thing to get wrong. If a customer calls a coffee shop and says "your site is down" while the owner is looking at it working fine, the customer loses the call and I lose the customer.

So when I finally checked those verdicts properly, the result hurt.

The measurement

I took 47 sites that one crawl had recorded as unreachable and opened each one from my laptop.

6 of them loaded instantly. That's 13% wrong. A second, larger run gave the same answer: 56 of 413, or 14%. Two of the "dead" sites belonged to well-known coffee chains, and both were serving pages perfectly well.

The cause wasn't hard to guess. The crawler runs on a serverless platform, so every request comes from a datacentre IP address. Cloudflare, Shopify and similar platforms block or challenge datacentre traffic as a matter of routine. To my crawler, a site that blocks it looks the same as a site that doesn't exist.

"Did not answer us" and "is down" are not the same sentence.

I tried three fixes. The first two failed, and the way they failed taught me more than the fix that worked.

Attempt 1: just retry the failures

The original pipeline already had a "quiet re-check" stage. Any site that failed during the main crawl (24 fetches running at once) was retried later, three at a time with a longer timeout. The reasoning sounded right: a big concurrent crawl causes throttling, so a gentle second try should get a truer answer.

That stage took up 22% of every search's runtime. Nothing recorded what it actually recovered, so I added two counters, recheck_tried and recheck_recovered, and let it run on real production searches.

36 sites retried across 6 production searches
 0 recovered

Not "low". Zero.

And looking back, it could never have been anything else. A block is perfectly consistent. If an IP gets blocked, it gets blocked again eight seconds later, and it'll keep getting blocked at any concurrency and with any timeout. Retries catch slow hosts. They can't catch refusals, and refusals were nearly all I had.

I deleted the stage. Searches got 22% faster, and the results didn't change at all.

Attempt 2: wait six hours and check again

Next idea: if retrying immediately can't tell a block from a dead site, maybe time can. A dead site stays dead. A flaky one recovers.

So I wrote confirmDown(). A first failure is recorded with a timestamp but never shown to the customer. Only a second failure at least six hours later, from a fresh connection, is allowed to say "down". A success at any point clears the record.

export function confirmDown(previous: Audit | null, next: Audit, now = Date.now()): Audit {
  // It answered: whatever happened before, it isn't down.
  if (next.reachable !== false) return { ...next, firstDownAt: null, downConfirmed: false };

  const firstDownAt = previous && !previous.reachable && previous.firstDownAt
    ? previous.firstDownAt
    : new Date(now).toISOString();

  const since = now - new Date(firstDownAt).getTime();
  // A clock that has gone backwards must not confirm anything.
  const confirmed = Number.isFinite(since) && since >= SIX_HOURS;
  return { ...next, firstDownAt, downConfirmed: confirmed };
}

It's clean, testable, and it feels rigorous. Two days later I checked it against reality: of 60 sites it had confirmed down, 13 loaded instantly from a normal connection.

unconfirmed failures   14% wrong
confirmed failures     22% wrong

Confirmation made it worse.

The reason is obvious once you see it. Waiting six hours filters out transient failures, like a server that was briefly overloaded. Those were the cases that would have recovered anyway. A datacentre block isn't transient. It fails the first check and the second one too. So the filter removed the harmless cases and left a pool that was concentrated with exactly the false verdicts it was supposed to exclude.

Time can't tell a block from a dead site. I needed a different signal.

Attempt 3: read the error I was throwing away

The signal had been there the whole time. My fetch wrapper looked like this:

try {
  return await fetch(url, { signal });
} catch {
  return null; // 🪦 every reason for failure, flattened into one
}

Node's fetch tells you why a request failed, in error.cause.code. And some of those reasons are proof, while others aren't:

The last row is exactly what being blocked from a datacentre looks like. It's indistinguishable from a slow or hostile server, so I can't call it "down".

So I kept the reason:

let lastFailure: string | null = null;

try {
  const res = await fetch(url, { signal });
  lastFailure = null;
  return res;
} catch (e) {
  // Node puts the useful part in `cause.code`: ENOTFOUND when DNS has nothing,
  // ECONNREFUSED when nothing is listening, a timeout or reset when something
  // is there but won't talk to us.
  const cause = (e as { cause?: { code?: string } })?.cause;
  lastFailure = cause?.code ?? (e as Error)?.name ?? "unknown";
  return null;
}

And I made reachable three-valued instead of a boolean:

export function failureIsFatal(code: string | null | undefined): boolean {
  return code === "ENOTFOUND" || code === "ECONNREFUSED";
}

// After both http:// and https:// have failed:
return {
  reachable: failureIsFatal(lastFailure) ? false : null,
  failure: lastFailure,
  // ...every other finding is null too: we didn't see the page
};

• true: we fetched it.
• false: we have proof it's down (no DNS record, or the port refused).
• null: we couldn't reach it, and we can't say why.

confirmDown() stayed, but only for the false case. A second look is cheap, and a wrong "down" is still the most expensive thing the product can say. What it no longer does is turn an inconclusive failure into a confident one just because it happened twice.

The null that wasn't "fine"

One trap is worth calling out, because I nearly fell into it.

My scoring code treats an audited site with no problems found as a good site. If "couldn't reach it" had been stored as "audited, no problems found", every blocked site would quietly have been graded healthy. The bug would just have flipped from "falsely down" to "falsely fine".

So null means not audited. The lead shows up as "we haven't checked their website yet", and its grade can go up later once we manage to look. It never drops now because of a claim we can't back up.

I also found that a deadline test was asserting the bug. When an audit ran out of its time budget, it returned reachable: false, which reported my timeout as a fact about their website. That test now expects null.

The cache still believed the old rule

Fixing the crawler didn't fix the data it had already written. When I counted the live table, there were 2,141 businesses whose sites showed as down, and 1,501 of them had no failure code at all. They'd been decided by a rule I'd already retired. Seventy per cent of the claim rested on evidence I'd stopped trusting.

I didn't run a backfill. The verdict is now earned on read: a row has to say it failed twice and that it failed with a fatal code. A row that can't answer the second question reads as "not checked", and the next crawl settles it under the current rule.

What I took away from it

1. Measure what your retry actually recovers. Mine cost 22% of every search and recovered nothing. Two integer counters would have told me that months earlier.
2. Retries and waiting only fix transient failures. If the failure is consistent, as a block is, repeating the check just confirms the wrong answer with more confidence.
3. Filtering can concentrate errors instead of removing them. If your filter removes the cases that were never the problem, what's left can be worse than what you started with.
4. Never catch { return null } at a boundary that makes a decision. The reason for the failure was the signal.
5. "Unknown" deserves its own value. A boolean forced me to pick between two lies. null let the code tell the truth: we didn't manage to look.

The fix ended up being a few lines. The lesson was that "we couldn't reach it" is a fact about you, not about them, and the code should never say otherwise.

Have you been burned by a check that was confidently wrong? Or a retry loop that turned out to do nothing? Tell me in the comments.