Blocking GPTBot costs you 40% of your AI citations. The data is in.
The most common GEO advice is "check your robots.txt." Most site owners assume they are fine because they never wrote a Disallow rule for GPTBot. The data says 25.8% of them are wrong. I ran a pre-registered calibration
The most common GEO advice is "check your robots.txt." Most site owners assume they are fine because they never wrote a Disallow rule for GPTBot. The data says 25.8% of them are wrong.
I ran a pre-registered calibration study in October 2026. The goal was to test whether the GEO score predicts AI citations. It did not. But the study also collected a variable I did not expect to be this important: whether the site blocks at least one AI bot in robots.txt.
341 pages, sampled from the Tranco list, audited with GEO Optimizer 4.19.3. For each page, the tool reads robots.txt and checks whether any of the 27 tracked AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and 23 more) are explicitly disallowed. The study also queried Perplexity Sonar with two templates per page and recorded whether the domain appeared in the source list.
The result was clear and not part of the pre-registered hypothesis:
- Sites that block at least one AI bot: cited 28.4% of the time (25 out of 88)
- Sites that block none: cited 47.0% of the time (119 out of 253)
That is an 18.6 percentage point gap. Sites that block AI bots get cited roughly 40% less often than sites that do not.
Where the blocks come from
Most of these blocks are not deliberate decisions to keep content out of AI search. The audit data shows three common sources:
Legacy WordPress security plugins. Wordfence and Sucuri ship default rules that block unrecognized user agents. If you installed one of these in 2019 and forgot about it, GPTBot may be getting a 403 right now.
Inherited robots.txt files. Many sites copy a robots.txt from a template or a previous project. If that template was written before AI crawlers existed, it may not have Allow rules for GPTBot or OAI-SearchBot. The crawler falls through to a wildcard Disallow.
CDN bot protection. Cloudflare Bot Fight Mode, when enabled, blocks non-browser user agents at the edge. It does not touch robots.txt, so the block is invisible to any tool that only reads robots.txt. I found that 34.3% of Cloudflare-fronted sites in a 636-site sample block at least one AI crawler at the HTTP level, compared to 12.0% of non-Cloudflare sites.
GPTBot is the most blocked
In a fresh CDN-level scan of 513 reachable sites, the per-bot blocking rates are:
- GPTBot: 17.7% (91 sites)
- PerplexityBot: 15.2% (78 sites)
- Applebot: 13.8% (71 sites)
- OAI-SearchBot: 12.9% (66 sites)
- Googlebot: 12.5% (64 sites)
- Claude-SearchBot: 12.3% (63 sites)
GPTBot is blocked more than any other AI crawler. 12 sites block only GPTBot and nothing else. That selectivity means someone read an article about OpenAI training data and added a Disallow, without realizing that GPTBot also powers ChatGPT's live web search. You can block training and allow citations by separating GPTBot (training) from OAI-SearchBot (search). Most people block both or neither.
14 sites block both GPTBot and PerplexityBot but allow Claude. That is a pattern: site owners who heard about the two most discussed crawlers and blocked them by name, without checking the full list.
What this means for you
If you have never checked whether AI crawlers can reach your site, this is the first thing to fix. Not llms.txt. Not schema. Not content structure. If the crawler gets a 403, every other optimization is invisible.
The fix is a robots.txt audit. Check which of the 27 AI crawlers are explicitly disallowed. If you use Cloudflare, check whether Bot Fight Mode or a WAF rule is intercepting non-browser user agents. If you use a security plugin, check its bot protection settings.
The calibration study is open. The raw data is in the geo-optimizer-skill repo. 341 rows, no domain names, shuffled order. You can reproduce the analysis.
If you want to check your site, run a free audit on geoready.dev. The audit tests all 27 AI crawlers at the robots.txt level and sends live HTTP requests as GPTBot, OAI-SearchBot, PerplexityBot, and Claude-SearchBot to check for CDN-level blocking.
The CLI is on GitHub, MIT licensed, 1,137 stars. pip install geo-optimizer && geo audit https://yoursite.com.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.