We tested 2,000 websites: 45% of GPTBot blockers also opt out of ChatGPT search answers

Originally published on the CrawlBeacon blog.

Short answer: In October 2026 we tested 2,000 popular websites with the same code behind CrawlBeacon's free AI Bot Readiness Checker. Of the 894 that returned a normal web page, 8.9% block at least one AI search bot in robots.txt, and 45% of sites that block OpenAI's training bot GPTBot also block OAI-SearchBot, the bot that decides whether a site can appear in ChatGPT search answers. Blocks in the firewall, where robots.txt doesn't show them, are rarer: 2.3% of sites.

Last updated: October 2026 · Sample: 2,000 domains from the Tranco list (894 analyzed) · Data source: CrawlBeacon AI Bot Access Study

Key findings

• 8.9% of popular websites block at least one AI search bot in robots.txt (80 of 894 sites tested in October 2026).
• 45% of websites that block GPTBot also block OAI-SearchBot (50 of 111). GPTBot collects AI training data; OAI-SearchBot decides whether a site can appear in ChatGPT search answers. Blocking both keeps a site out of ChatGPT search answers, not just training.
• 2.3% of websites block AI search bots in their firewall while robots.txt allows them (21 of 894), mostly mid-size publishers such as Newsweek, Engadget, Lifehacker, the Daily Mail and C-SPAN.
• 0.8% of websites charge AI crawlers with HTTP 402 "pay per crawl" responses, including X (Twitter), Rolling Stone, the Irish Mirror and The Dispatch.
• About 4% of websites block any request that pretends to be a bot, including IBM, Salesforce and WordPress.com, which makes many AI access tests misleading.

What is the difference between AI search bots and AI training bots?

AI search bots fetch pages so an AI assistant can show and cite them in answers; AI training bots collect content that may be used to train AI models. Each company runs separate bots for each job, and site owners can allow or block them independently.

OpenAI says each of its settings is independent: sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, while disallowing GPTBot signals that content should not be used for training. Google says Google-Extended does not affect inclusion in Google Search.

How many websites block AI search bots?

8.9% of the popular websites we tested block at least one AI search bot (OAI-SearchBot, Claude-SearchBot or PerplexityBot) in robots.txt. The share is similar across site sizes:

The percentage falls slightly as sites get smaller, but at these sample sizes the differences are within the margin of error, so they should not be read as a trend.

Large sites that block AI search bots in robots.txt include Amazon (across its regional stores), LinkedIn, Reddit, X, Pinterest, CNN, the BBC, The Guardian, IMDb and Wired. Many of these choices are deliberate.

Do websites that block GPTBot also block ChatGPT search?

Often, yes: 45% of the websites that block GPTBot in robots.txt also block OAI-SearchBot, which removes them from ChatGPT search answers as well as from AI training.

This closely matches an independent scan of the Tranco top 5,000, which found that 238 of 535 sites blocking GPTBot also blocked OAI-SearchBot (44%). Two different samples, nearly the same answer.

Some of these sites intend to block every AI bot. Others group AI bots together or copy a community block list that includes the search bots, and end up blocking AI search when they only meant to opt out of training.

How do you block AI training but stay visible in AI search?

Block the training bots and explicitly allow the AI search bots in robots.txt. This configuration opts out of OpenAI, Anthropic and Google model training while keeping a site eligible for ChatGPT, Claude and Perplexity search answers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Then check that your firewall or CDN isn't blocking the search bots, because robots.txt can't override a firewall rule. You can test both in one step with the AI Bot Readiness Checker.

Do firewalls and CDNs block AI bots?

Rarely, on their own: 2.3% of the websites we tested refused AI search bots in their firewall or CDN while robots.txt allowed them. robots.txt only says what a site asks crawlers to do; a firewall can refuse a bot before it reads robots.txt, so we also requested each homepage live, once as a normal browser and once as each of 8 AI bots (listed in Methodology).

Every site below refused the bots in two separate test runs:

• Newsweek and Engadget refuse OAI-SearchBot.
• Lifehacker shows all three AI search bots a challenge page.
• The Daily Mail (.com and .co.uk) refuses OAI-SearchBot, Claude-SearchBot and PerplexityBot, although its robots.txt only restricts parts of the site.
• C-SPAN refuses OAI-SearchBot and Claude-SearchBot.
• BioMed Central and Springer refuse PerplexityBot only.
• At least one site, a university research centre, served bots the block page of Anubis, an open-source tool built to stop AI scrapers.

Firewall-only blocks were most common among mid-size sites (ranks 1,001–10,000), where 12 of the 21 cases appeared.

Which websites charge AI crawlers for access?

0.8% of the websites we tested answered AI search bots with HTTP 402 Payment Required, the status code used by pay-per-crawl schemes such as Cloudflare's pay per crawl. They include X (Twitter), Rolling Stone, the Irish Mirror and The Dispatch.

A 402 is a different answer from a block: access is offered for a price rather than refused.

Why is testing AI bot access hard?

Because many firewalls block anyone pretending to be a known bot, a test that sends an AI bot's user agent from its own server can mistake bot verification for AI blocking. About 4% of the websites we tested behave this way.

Our first pass flagged IBM, Salesforce and WordPress.com as blocking AI search bots. Before publishing, we re-tested with a control: a request that claims to be Googlebot. Those sites refused the fake Googlebot too. Their firewalls verify bot identity by IP address, so our requests, which don't come from OpenAI's or Google's servers, were correctly refused, and that says nothing about what the real OAI-SearchBot receives.

We count those sites as inconclusive, not blocked, and we added the same Googlebot control to the AI Bot Readiness Checker as a result of this study.

Methodology

• Sample: Tranco list 64VVX (created October 16, 2026): the top 1,000 domains, plus 500 random domains from ranks 1,001–10,000 and 500 random domains from ranks 10,001–100,000 (random seed 20261017).
• Exclusions: 591 domains with no website at their root (infrastructure and API domains such as gstatic.com), 119 that timed out, 375 that refused or errored on a normal browser request, and 21 whose robots.txt couldn't be fetched, leaving 894 websites. Sites whose live results changed between runs were left out of the firewall counts.
• robots.txt: fetched and parsed following RFC 9309, checking 15 AI bots and 6 search engine crawlers. "Blocks" means the bot is disallowed from the whole site.
• Live access: each homepage was requested as a normal browser, as Googlebot, and as 8 AI bots (OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot and Perplexity-User), using each operator's published user agent where available. Every site named in this article gave the same result in two separate runs.

Limitations

• IP-based blocking is invisible to this method. Some firewalls, including Cloudflare's AI bot blocking, identify real AI crawlers by their IP address. Our requests don't come from those addresses, so a site can block the real GPTBot while letting our test through. Firewall blocks are therefore likely undercounted.
• One page, one moment. Each homepage was tested in October 2026 (twice for flagged sites). Settings change, and some sites treat inner pages differently.
• robots.txt can vary by visitor. A few large sites serve different robots.txt files to different crawlers; this study saw what a standard request receives.
• Popular sites only. Tranco ranks domains by popularity, so very small websites may look different.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. Blocking GPTBot only opts your content out of OpenAI's model training. Whether your pages can appear in ChatGPT search answers is controlled by OAI-SearchBot, which OpenAI treats as a separate setting.

What happens if I block OAI-SearchBot?

According to OpenAI, sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. In our study, 6.8% of the top 1,000 websites block OAI-SearchBot in robots.txt.

Is blocking AI bots in robots.txt enough?

robots.txt is a request, not an enforcement mechanism. Most major AI crawlers say they respect it, but user-request fetchers may not: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. To enforce a block, use your firewall or CDN.

How many websites block AI crawlers in 2026?

In our October 2026 study of 894 popular websites, 8.9% blocked at least one AI search bot and 12.4% blocked GPTBot in robots.txt.

How can I check which AI bots can access my site?

Use CrawlBeacon's free AI Bot Readiness Checker. It analyzes your robots.txt for 18 AI bots, runs a live test as 8 of them (from OpenAI, Anthropic and Perplexity), and tells you when your firewall's bot verification makes a result inconclusive.

Using this data?

You're welcome to quote these findings. Please credit "CrawlBeacon AI Bot Access
Study (October 2026)" and link to this page. Journalists and researchers can contact us for the summary tables behind these findings.