I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them

Two thirds of the sites blocking GPTBot are also blocking the crawler that sends them traffic. I measured it.

Blocking GPTBot is a reasonable decision. OpenAI documents it as the crawler that gathers training data, you get nothing back for it, and plenty of publishers have decided that trade isn't worth making. Fine.

The problem is that GPTBot is not the only OpenAI token in your robots.txt, and the other two do completely different jobs:

• GPTBot — training corpus. Fetches content to train future models. Sends you nothing.
• OAI-SearchBot — builds the search index behind ChatGPT's browsing. This is what makes your page eligible to be cited.
• ChatGPT-User — the fetch that happens while a person waits for an answer. A citation here is a link a human can click.

Block the first and you've opted out of training. Block all three and you've opted out of being found, which is a different decision that a lot of people appear to be making by accident.

How I measured it

I wanted actual numbers rather than vibes, so:

• Frame: Tranco top-1M (the research-grade domain ranking), downloaded 2026-11-07.
• Sample: the 100 highest-ranked domains per ccTLD — .com, .de, .fr, .es, .it, .jp, .nl. Deterministic, so anyone can reproduce it.
• Fetch: robots.txt over HTTPS directly, www. then bare, redirects followed, 12s timeout.
• Verdict: for path /, using RFC 9309 group selection — the exact user-agent group if one exists, otherwise *, never both — with longest-match precedence and Allow winning ties.
• Read successfully: 528 of 700. The rest were unreachable (137) or served an HTML page where robots.txt should be (35).

The numbers (n=528)

The line that matters: of the 161 sites blocking GPTBot, 107 — 66.5% — also block a traffic-bringing crawler. Only 10.2% of the whole sample made the distinction and blocked training while keeping discovery open.

And 17.2% block an AI crawler that could cite them while carefully keeping Googlebot allowed. Those sites clearly care about being found. They just don't appear to know that User-agent: * plus a Disallow catches a lot more than Google's competitors.

It splits by market, and not how I expected

Germany is the interesting one. German sites block GPTBot at nearly the same rate as .com (30.9% vs 34.4%) but block traffic crawlers at less than half the rate (12.3% vs 29.7%). That is a market making the deliberate version of the decision. Japan barely blocks anything.

The Netherlands is the opposite — the highest blocking rate in the sample on both counts.

The honest limitation

These are high-ranking domains. They are not small business websites, and the figures should not be read as a picture of the small-business web. Large publishers have legal teams with opinions about training data; a plumber's site has whatever the theme shipped. A second sample of lower-ranked domains would be needed before claiming anything about the long tail — I haven't run it yet.

Why "which line decided it" is the whole problem

When I started checking individual sites I kept hitting the same thing: the owner had no idea their file blocked what it blocks. Not because they misread it, but because robots.txt precedence is genuinely unintuitive:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

A lot of people read that as "block everything, but let Google in." It is — but it also blocks every AI crawler, including the two that could send readers. There's no error, no warning, nothing in Search Console. The file is doing exactly what it says and not at all what was meant.

Worse, the groups don't combine. Under RFC 9309 a crawler obeys the group matching its own token, and falls back to * only if no such group exists. So:

User-agent: Google-Extended
Disallow: /

...does nothing whatsoever to Googlebot — different token, different group. And a named empty group beats a restrictive wildcard:

User-agent: *
Disallow: /

User-agent: GPTBot

That allows GPTBot everything. The group exists, it has no rules, so nothing is disallowed. People write this believing it's a block.

So I open-sourced the checker

Every number above came out of a matcher I needed anyway, so it's on GitHub under MIT:

npx github:orenhacham-lab/ai-crawler-access nytimes.com

Zero dependencies, no build step, one network request. For every crawler it prints the verdict and the exact line that produced it — which turned out to be the only part people actually find convincing, because "your file blocks ChatGPT" gets argued with and "line 14, Disallow: /, in the * group that applies to you because you have no OAI-SearchBot group" does not.

As a library:

import { checkRobots, summarize } from 'ai-crawler-access'

const result = await checkRobots('example.com')
console.log(summarize(result))   // training vs discovery, separated

The crawler dataset is separate and CC BY 4.0, as JSON and CSV. Every row's token spelling, purpose and robots.txt behaviour comes from the operator's own documentation, with the URL and the date it was read in the row — nothing inferred from log files. Tokens I've seen in the wild but can't find documentation for are listed separately as candidates, because "we saw this" and "the operator says this" are not the same claim and shouldn't sit in one table.

• Code: https://github.com/orenhacham-lab/ai-crawler-access
• Dataset: https://www.gotopseo.com/api/public/ai-crawlers (no key, no signup, CORS open)
• Hosted checker: https://www.gotopseo.com/en/ai-crawlers

Corrections to the dataset are the contribution I want most — particularly an operator doc page for a token currently sitting in candidates.

What I'd actually do with this

If you want out of training but still want to be cited, the file is explicit and boring:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Named groups for every token you have an opinion about. Don't rely on * to express a nuanced position — it can't, and the 66.5% figure above is what happens when people try.

Two caveats worth saying out loud. Google-Extended and Apple's equivalent are control tokens that crawl nothing by themselves; disallowing them changes how already-crawled content may be used, and both operators document that it does not remove you from search results. And a handful of agents have operators who document that robots.txt may be ignored — for those, your file states a request, not an outcome.