Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.

114 sites, one robots.txt template, 31 banned user agents. I probed 7 of them and got three different answers.

The network I audit runs 114 sites behind the same edge. Every one of them serves a robots.txt. I had never read one end to end, because I assumed the file was boring. It is 2,756 characters at the median and half of it is a legal notice.

So I pulled all 114, parsed them, then did the thing robots.txt never gets: I tested whether the bans actually fire.

The file is half legal notice

All 114 returned 200. All 114 are text/plain. None of them accidentally served an HTML 404 page, which is where this kind of audit usually starts.

Here is the shape of a median file, by character count:

The comment block is not junk. It is a rights reservation under Article 4 of the EU Copyright in the Digital Single Market directive, explaining what search, ai-input, ai-train and use mean before the rules start. It is written for a lawyer, not for a crawler.

But the split matters for a different reason. When a file is 51% prose, nobody reads the other 49%. I know this because I wrote the deployment and I did not read it.

Thirty-one identical bans across 114 files

The union of banned user agents across all 114 files is 31. Each individual file contains exactly 31. Not 30, not 32 — the same 31 strings, in the same order, byte for byte, in all 114 files.

Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Diffbot,
Google-Extended, GPTBot, omgili, anthropic-ai, Claude-Web, cohere-ai,
MistralAI-Training, meta-externalagent, GoogleOther, Baiduspider,
PetalBot, AwarioRssBot, Google-CloudVertexBot, QualifiedBot, Cotoyogi,
ICC-Crawler, atlassian-bot, FishBot, BorderxBot, NavuBot,
SemrushBot-SWA, WARDBot, magpie-crawler, KimiBot, CitibotSiteCrawler

Each of those 31 gets the same two lines:

User-agent: Baiduspider
Disallow: /

That is a full-site ban. No path escaped, no rate limit, no partial allowance.

Read the list again. Roughly sixteen of those names are AI training or grounding crawlers, which is the stated intent. The rest are not. Baiduspider is Baidu's search crawler. PetalBot is Huawei's search crawler. GoogleOther is Google's miscellaneous fetcher, not Gemini training. atlassian-bot fetches link previews. SemrushBot-SWA is a site audit tool. ICC-Crawler is an academic index.

One list, two very different jobs, and the search crawlers are on the wrong side of it.

The wildcard group forbids nothing

Here is the part that surprised me more than the ban list. Every file has two User-agent: * groups — 113 of the 114 files, anyway. The first sits at the top of the managed block, right under the Content-Signal line. The second sits at the very bottom, after # END Cloudflare Managed content, where the site's own config gets appended.

Both of them contain exactly one directive:

User-agent: *
Allow: /

There is not a single Disallow in either wildcard group. Across all 114 files. So the effective policy for any crawler that is not one of the 31 named agents is: crawl everything, no restrictions, no crawl-delay.

The file is 2,756 characters long and imposes zero constraints on general crawling. It is a blocklist wearing the costume of a policy document.

Every file also carries the same content signal — search=yes,ai-train=no,use=reference — in all 114. So the machine-readable intent is consistent. The rules underneath it are not.

Two status codes, two different layers

Reading rules tells you what the operator intended. It does not tell you what the edge does. So I sent 14 user agents at 6 domains in the same minute, same URL, same headers apart from the UA string. The result was identical on all 6 domains.

Two control probes matter here. The random crawler UA — a name that appears in no list anywhere — gets a 200 with the full page. So this is not a blanket "block all bots" rule that catches unknown agents by heuristics. It is name matching against a list.

And the two responses come from two different places:

• 403 is the edge deciding. The body is a 25-byte sentence written by the WAF rule. The request never reaches the origin.
• 520 is the edge reporting that the origin did not answer usefully. error code: 520 is Cloudflare's own body for "web server returned an unknown error". The request passed the edge and the origin returned nothing usable.

That distinction is the whole article. Same declared policy, two enforcement mechanisms, and they behave differently in ways that matter to a crawler.

Here is the probe, if you want to run it against your own sites:

import ssl, urllib.request, urllib.error

UA = {
    "chrome":    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                 "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "googlebot": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "baidu":     "Mozilla/5.0 (compatible; Baiduspider/2.0; "
                 "+http://www.baidu.com/search/spider.html)",
    "gptbot":    "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; "
                 "GPTBot/1.2; +https://openai.com/gptbot",
    "unknown":   "Mozilla/5.0 (compatible; ZzqRandomCrawler/1.0; +https://example.invalid/bot)",
}

ctx = ssl.create_default_context()

def probe(url, ua):
    req = urllib.request.Request(url, headers={"User-Agent": ua, "Accept": "text/html,*/*"})
    try:
        with urllib.request.urlopen(req, timeout=20, context=ctx) as r:
            body = r.read()
            return r.status, len(body), body[:40].decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        body = e.read()
        return e.code, len(body), body[:40].decode("utf-8", "replace")

for name, ua in UA.items():
    status, size, head = probe("https://example.com/", ua)
    print(f"{name:10} {status} {size:>7}B  {head!r}")

The unknown row is the one people skip. Without it you cannot tell a name-based blocklist from a generic bot heuristic, and those two have completely different consequences when you add a new crawler to your stack.

The bans that never fired

Of the 31 declared bans, I tested 7 by name. Five fired. Two did not.

PetalBot and GoogleOther are both named with Disallow: / in all 114 files. Both got a 200 and a full page. The declaration exists; nothing enforces it. If you are relying on robots.txt alone to keep either of them out, you are not keeping them out.

Then the reverse. AhrefsBot and SemrushBot are not in the list. The list has SemrushBot-SWA, which is the site-audit variant, but not the main SemrushBot. Neither name appears anywhere in the 2,756 characters. Both got a 520.

So the blocklist and the enforcement layer disagree in both directions. Things that were banned were let in. Things that were never mentioned were shut out. Whatever is generating the file and whatever is enforcing it at runtime are not reading from the same source, and nobody has reconciled them since the template was applied.

This is the same failure mode as a forecast that declares 70% confidence and delivers 62% — the declaration is not the behaviour, and the gap only shows up when you score outcomes instead of reading claims. I have been running that measurement on a set of forecasting experiments, and the method is identical: state the claim, then test it, then compare. A calibration experiment that scores stated confidence against realised outcomes makes the point faster than any audit report: the gap is usually in the places nobody instrumented.

A 520 is the wrong way to say no

If the goal is "do not crawl me", 403 is the correct answer. It is a 4xx: a permanent, explicit refusal. Crawlers treat it as an instruction and stop.

520 is a 5xx. To every crawler on the internet, a 5xx means "the server is broken, try again later". Google's documentation is explicit that persistent 5xx responses cause crawl rate reduction and, eventually, removal from the index. Baidu behaves the same way, with less patience.

So the sites that meant to block an AI crawler are instead telling Baidu that they are down.

I ran the Baiduspider probe three times against five domains, four seconds apart, to rule out a transient origin failure. Fifteen requests, fifteen 520s. Then I tried Baiduspider-render, the rendering variant: 520 as well.

Then I tried the sitemap. https://<domain>/sitemap-index.xml with the Baiduspider UA returned 520 too. Not the sitemap contents — the same 16-byte error line.

Whatever is producing that 520 is upstream of the routing, so it does not distinguish a homepage from a sitemap from a stylesheet. From Baidu's perspective the entire host is unreachable, and there is no diagnostic it can act on, because 520 carries no information about what went wrong.

The sites that got a clean 403 are in better shape than the ones that got a 520, and both of them are worse off than the ones that got a 200 when they should not have.

The sitemap was in perfect shape the whole time

Here is the part that stings. Of the 114 sites, 113 declare a Sitemap: line in robots.txt. I fetched all 113 and checked the root element, not just the status code — a 200 that returns HTML is a much more common failure than a 404.

113 out of 113 returned real XML. 83 were <urlset>, 30 were <sitemapindex>, all served as text/xml. Correct content type, correct root element, parseable, reachable with a normal user agent.

So the discovery chain was intact. The sitemap was well-formed, declared in robots.txt, and served correctly. The only thing standing between that sitemap and a search engine was a 520 that fires before routing.

That is the worst shape a technical problem can take: every component passes its own unit test, and the pipeline still delivers nothing. When I want to know what a system actually does rather than what its components claim, I go to the primary record — the way the Form D filings behind a funding number tell you more than any press summary of it. The status code is the primary record here. The robots.txt is the press release.

Checking your own network

Three checks, in order of how often they find something:

1. Does the ban fire? Take every agent in your robots.txt and probe your homepage with that exact UA string. Compare against a Chrome control and an unknown-crawler control. Three outcomes, three different bugs: 200 means the ban is decorative, 403 means it works, 5xx means it works in the worst possible way.

2. Is anything blocked that you never declared? The unknown-crawler control catches blanket rules. Then probe your analytics tools, your SEO tools, your uptime monitor. AhrefsBot and SemrushBot were being refused on 114 sites that never named them.

3. Is your wildcard group actually restrictive? Print it. If it is one Allow: / with no Disallow, your file is a blocklist and nothing else — and every new crawler that appears next quarter will walk straight in until someone adds it by hand.

from urllib.robotparser import RobotFileParser
import urllib.request

txt = urllib.request.urlopen(
    urllib.request.Request("https://example.com/robots.txt",
                           headers={"User-Agent": "Mozilla/5.0"}), timeout=20
).read().decode("utf-8", "replace")

rp = RobotFileParser()
rp.parse(txt.splitlines())

for agent in ("Googlebot", "Bingbot", "Baiduspider", "PetalBot", "GPTBot"):
    print(f"{agent:12} allowed={rp.can_fetch(agent, 'https://example.com/')}")

RobotFileParser alone gets you the declaration side in about ten lines. It will tell you PetalBot is banned. It will not tell you PetalBot walked in anyway, which is why the probe matters more than the parse.

What I am changing

Two fixes, in this order.

First the 520. Any refusal that returns 5xx for a named crawler should be a 403 or nothing at all. A 5xx is a lie about your own uptime, and search engines bill you for it in crawl budget before they bill you in rankings.

Second the list. Search crawlers and training crawlers should not share an allow/deny list, because they have opposite economics: one brings you traffic, the other takes your content. Baiduspider, PetalBot and GoogleOther come off the list. They do not train models. Keeping them in the same block as GPTBot costs index coverage and buys nothing.

The template itself is the harder problem. A managed block that is identical across 114 files is a good default and a bad policy — it encodes one vendor's opinion about which agents matter, and it ships that opinion to every property without asking whether the property agrees. The content layer already works this way: a coin page on one of the sites carries the same header, the same analytics script and the same caveats as the other 492 entries in that sitemap, because the template decided for all of them. Every site in the network inherited 31 decisions nobody made.

I have 114 sites and 3,534 declared bans across them. Five of the seven I tested were real. The other two were furniture, and the one that mattered most was answered with a 5xx.

Half the file was a legal notice. I should probably have read the other half sooner.