We Read the robots.txt of the Top 10,000 Sites. Half That Block AI Training Also Block AI Search

Blocking GPTBot became a common move in 2023 and 2024. A site adds two lines to robots.txt, and OpenAI stops using its pages to train models. Plenty of teams did it, and it's a reasonable call.

Some sites went further than GPTBot and blocked AI crawlers in bulk, including the ones that put a site into ChatGPT, Claude and Perplexity answers. We wanted to know how often that happens, so we read the robots.txt of the 10,000 most popular sites on the web.

What we found across 7,357 of the top 10,000 sites.

How we checked

We took the Tranco top 10,000 list on November 1, 2026, and fetched each site's robots.txt once, with an honest research user agent. 7,357 sites answered. Most of the rest are CDN and API domains that don't serve a website.

A site "blocks" a crawler when its robots.txt disallows the whole site for that user agent. If a site served an HTML page at /robots.txt, we counted it as having no file, because that's how crawlers treat it too.

Two limits up front. This covers robots.txt only, so firewall blocks aren't in these numbers, and the real share of blocked crawlers is higher. And these are the biggest sites on the web, not a typical small business.

The results

• 77.6% have a real robots.txt.
• 17.3% block at least one AI training crawler.
• 9.7% block at least one AI search or assistant crawler.
• 54.5% of the sites that block AI training also block AI search.

That last number is the one that matters. More than half of the sites that opted out of model training also opted out of being cited in AI answers. We doubt most of them meant to.

How often each crawler is shut out of the whole site.

Broken down by crawler:

Googlebot and Bingbot are almost never blocked, since nobody wants to vanish from search. The AI search crawlers are blocked three to five times as often as Googlebot, even though they do the same job for a different kind of search.

Training crawlers and search crawlers are different bots

Most AI companies run separate crawlers for separate jobs, and robots.txt treats each user agent on its own:

• Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent) collect pages to train future models. (Google-Extended isn't a separate bot. It's a token Google checks, and blocking it doesn't touch Search or AI Overviews.) Blocking them is a business decision, and it doesn't remove you from AI answers.
• Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so an assistant can answer a question and cite its sources. Block these and you can't be cited.
• User agents (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked the assistant to read it, for example after pasting your link into a chat.

So a robots.txt that says "no training, yes answers" looks like this:

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Stay in AI answers and search
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Each crawler follows only the group that names it, and falls back to User-agent: * when none does, so a blanket User-agent: * / Disallow: / will catch every AI crawler you didn't list by name. If your file has one, give the search crawlers their own groups, as in the example.

The block that doesn't show up in robots.txt

robots.txt is only half of it. Many sites sit behind a CDN or firewall with a "block AI bots" setting, and when it's on, the AI crawler gets a 403 before it ever reads robots.txt. Your file can say Allow: / while every AI crawler is turned away at the door.

That's why our numbers are a floor. It's also the check people skip, because you can't see it by reading your own robots.txt.

Check your own site in a few seconds

We built a free AI crawler checker for exactly this. Paste a URL and it reads your robots.txt for the AI search, assistant and training crawlers above, plus Googlebot, Bingbot, Applebot and DuckAssistBot, and tells you which ones are blocked.

It also fetches your homepage twice, once as a browser and once as an AI agent. If the browser gets through and the AI agent gets a 401, 403, 429 or 503, it flags the firewall, which is the block robots.txt can't show you. When a search crawler is blocked, it writes the lines to add to robots.txt and leaves your training blocks alone.

Every crawler, what it feeds, and whether your site lets it in (here, example.com).

If you'd rather start clean, our robots.txt generator lets you pick which crawlers to allow and builds the file. It runs entirely in your browser, and neither tool needs an account.

Choose per crawler, then copy the file.

Crawler access is only the first step to showing up in AI answers. Here's what the full AI visibility check looks at once the bots can get in:

The short version

1. Blocking GPTBot, ClaudeBot or Google-Extended keeps you out of training data, not out of AI answers.
2. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot keeps you out of AI answers.
3. More than half the top sites that block training also block search. Check that yours isn't one of them.
4. Look past robots.txt: a firewall setting can block every AI crawler while your file says Allow.