Your fake Chrome User-Agent is why that site returns 403

I spent twenty minutes convinced a site had gone offline.

It hadn't. I was the one who broke it — by putting a Chrome User-Agent on a curl command.

Here is the original measurement. Same machine, same connection, same few seconds, one URL:

UA="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"

curl -sS -o /dev/null -w "%{http_code}\n" -A "$UA" https://alistapart.com/about/contribute/
# 403

curl -sS -o /dev/null -w "%{http_code}\n" https://alistapart.com/about/contribute/
# 200

curl -sS -o /dev/null -w "%{http_code}\n" -A "" https://alistapart.com/about/contribute/
# 200

The fake browser is the one that gets blocked. The honest curl is the one that gets the page.

The 403 is not a firewall error, it's a challenge page

The body gives it away:

<title>Checking your browser...</title>
<script src="/__challenge/probes.js"></script>

That is a JS challenge. The site is not refusing to serve the document — it is refusing to serve it to a client that claims to be Chrome but cannot run the JavaScript it just handed you.

Which is the whole point. A request that says Chrome/126 and then presents no browser behaviour is a more suspicious artifact than a request that says curl. It is self-contradictory, and inconsistency is cheap to detect. Being honest costs you nothing here.

The inverse case is real too

An empty User-Agent fails on the same logic, from the other direction. Requesting https://dev.to/ with no UA at all:

HTTP/2 403
server: Varnish
content-length: 0

Zero-byte body, no challenge, just a Varnish-layer rejection. "Claims to be nothing" is also easy to classify.

So there are two failure directions, and they look identical if you only look at the status code:

The control experiment that should be in every scraper repo

One request tells you nothing about a site. It tells you about your request. So run three, and log which one produced which code:

test_url () {
  local u="$1"
  local ua="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
  printf '%-46s chrome=%s default=%s empty=%s\n' "$u" \
    "$(curl -sS -o /dev/null -w '%{http_code}' -m 25 -A "$ua" "$u")" \
    "$(curl -sS -o /dev/null -w '%{http_code}' -m 25 "$u")" \
    "$(curl -sS -o /dev/null -w '%{http_code}' -m 25 -A "" "$u")"
}

test_url https://alistapart.com/about/contribute/
test_url https://www.smashingmagazine.com/contact/
test_url https://www.infoq.com/
test_url https://blog.logrocket.com/
test_url https://dev.to/

Run just now, that prints:

alistapart.com/about/contribute/        chrome=403  default=200  empty=200
smashingmagazine.com/contact/           chrome=200  default=200  empty=200
infoq.com/                              chrome=405  default=405  empty=405
blog.logrocket.com/                     chrome=200  default=200  empty=200
dev.to/                                 chrome=200  default=200  empty=403

Read that table carefully, because it is doing more work than it looks:

• One URL out of five is UA-sensitive (ALA). If I had tested only that one, I would have concluded "UA spoofing is broken everywhere". If I had tested only the other four, I would have concluded "UA spoofing is fine". Both conclusions would have been wrong, and the fix would have been wrong in opposite ways.
• 405 is not a UA problem at all. InfoQ returns it identically to all three clients. That is a method/policy answer, and no header change will move it. Chasing it with UA rotation is pure waste.
• The direction of the effect is not consistent. Chrome-UA fails on ALA, empty-UA fails on dev.to. There is no "best UA" to hardcode. There is only "check, per site, per request shape".

Two things this is not

It is not a transport failure. Separately from all of the above, some hosts are simply unreachable from a given network. https://news.ycombinator.com/ returns 000 — curl's "no HTTP response at all", in this case a TLS-level refusal on my egress. All three UA variants returned 000. If you see the same failure across every UA, you are not debugging a header. Stop editing the header.

The distinction matters because the two failure modes have opposite fixes: a 000 is a network/egress problem, a 403 may be a client-fingerprint problem, and a 405 is neither.

It is not a licence to bypass anything. If a site hands you a challenge page, that is the site answering your request. A scraper that solves the challenge to get around a documented "no automated access" policy is a different activity from a scraper that notices it is sending a self-contradictory request. The three-UA control tells you whether your own client is the problem. It is a diagnostic, not a bypass — and the honest answer is sometimes "this site does not want a script here, go use their API or their form".

The rule I ended up with

A 403 is two different bugs wearing the same status code. Before you write "site blocks scrapers" in a README, re-run the request with the default UA and with an empty one. Record all three codes. Only the pattern is evidence.

Twice now, on this same codebase, the two-UA retest has overturned a conclusion I had already written down: once when a failed request got filed as "the endpoint is gone", and once when it got filed as "the submission form no longer accepts email". Both times the site was fine and my client was lying about itself.

The checker behind the table above is a few lines of shell — I run the three variants in a wrapper that prints one row per URL, because a status code without its control group is not a measurement. If you maintain scrapers or an uptime check, the cheapest upgrade available to you is probably the second and third request.

Written with AI assistance and verified by hand — all responses above were pulled live on 2026-10-15 from a single machine and connection.