AI crawlers explained: which bots to block, which to allow

Originally published on CheckSEO.

The short answer

AI crawlers are the bots AI companies send to read web pages, and they do three different jobs. Training crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot collect content for future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI answers can cite and link to them. User-triggered fetchers such as ChatGPT-User and Perplexity-User load a page live because someone asked about it.

Each has its own robots.txt token, so blocking AI isn't all-or-nothing. The common middle ground is to disallow the training bots and allow the search bots, which opts you out of training while keeping you citable in AI answers. Just remember robots.txt is a request, not a lock.

The AI crawler list, grouped by purpose

Every token below comes from the operator's own documentation, checked in September 2026. Two entries aren't crawlers at all: Google-Extended and Applebot-Extended are robots.txt tokens that control how content fetched by Googlebot and Applebot may be used, so they never appear in server logs.

What blocking each group actually costs

Blocking training crawlers is the low-cost option. OpenAI documents GPTBot and OAI-SearchBot as independent settings, so disallowing GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search. It only works going forward: Anthropic describes a ClaudeBot block as excluding a site's future materials from its training datasets.

Blocking search crawlers takes you out of the answers. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.

Blocking user-triggered fetchers is the least predictable. Anthropic says disallowing Claude-User stops Claude retrieving your pages for user queries, but OpenAI and Perplexity say their fetchers act for a person and may not follow robots.txt.

Google works differently. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. AI Overviews and AI Mode are part of Search, so blocking Google-Extended won't remove you from them.

Copy-paste robots.txt rules

The file goes at the root of every host you want covered, since each subdomain needs its own. Allow about a day for changes to register. One trap: a crawler that finds a group naming it ignores the User-agent: * group entirely, so copy any path rules you rely on, such as Disallow: /admin/, into the named groups too.

Block training, allow AI search (the middle ground most sites want):

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# Stay visible in AI search answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Allow everything:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt is a request, not enforcement

Nothing technically stops a bot ignoring robots.txt. If a bot must stay out, enforce it at the edge:

1. Verify who is really visiting. User agent strings are trivial to fake. OpenAI, Anthropic, Perplexity, Common Crawl and Apple publish their IP addresses, so match both the name and the source before trusting or blocking a request.
2. Block by behaviour at your CDN or firewall. Cloudflare's AI bot policies block Search, Agent and Training bots separately. Other firewalls can match user agent plus IP range.
3. Keep robots.txt in place anyway. Anthropic warns that blocking its IP addresses can stop it reading your robots.txt, so an IP block alone may not register as a lasting opt-out.

Check which AI crawlers your site blocks

I built a free AI Crawler Access Checker that fetches a domain's robots.txt and evaluates it for 16 user agents. For each one it reports allowed, blocked or partly restricted, plus the exact rule that decided it. No signup.