Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

Is robots.txt enough to let AI crawlers in? (Usually not)

More sites now ask a simple question: can ChatGPT, Claude and Perplexity actually read my pages? Most people open robots.txt, see nothing scary, and stop there. That check is not enough. Two layers can block an

Is robots.txt enough to let AI crawlers in? (Usually not)

More sites now ask a simple question: can ChatGPT, Claude and Perplexity actually read my pages? Most people open robots.txt, see nothing scary, and stop there. That check is not enough.

Two layers can block an AI crawler

  1. robots.txt. A Disallow rule for a user agent like GPTBot or PerplexityBot. Polite crawlers respect it.
  2. The firewall or CDN. Cloudflare's "Block AI bots" toggle, a host's bot protection, or a WordPress security plugin. These return a 403 to the crawler no matter what robots.txt says.

The second layer is invisible if you only read robots.txt. Your file says "allowed", the bot gets blocked anyway, and you never find out.

Training bots vs search bots

Blocking every AI crawler is rarely what people actually want. The user agents fall into three groups:

Type Examples Block it and...
Training GPTBot, ClaudeBot, Google-Extended, CCBot your pages aren't used to train models. You can still appear in AI answers.
AI search OAI-SearchBot, Claude-SearchBot, PerplexityBot you usually disappear from that tool's answers.
User request ChatGPT-User, Claude-User, Perplexity-User the AI can't open your page when a user asks it to.

So a common setup is "search yes, training no": allow the search and user-request bots, disallow the training bots.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

I keep the full list of 20 user agents and three ready-made templates in a small open repo: ai-crawlers-robots-txt on GitHub.

Testing the firewall layer yourself

You can fake a crawler's request with curl and look at the status code:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://yoursite.com/

A 200 means the request got through. A 403 or a challenge page means something in front of your site is blocking that user agent. Repeat with OAI-SearchBot, PerplexityBot and ClaudeBot.

One caveat: some CDNs verify real crawlers by IP, so a spoofed user agent can be treated differently from the real bot. It's still a quick way to catch the most common problem, a blanket "block AI bots" rule.

What two big sites look like

On 16 Nov 2026 I ran both layers against two well-known sites:

  • nytimes.com: only 4 of 20 AI bots allowed in robots.txt, and only 2 of 9 live test requests got through. A deliberate choice for a publisher.
  • hubspot.com: all 20 allowed and all 9 live requests got through.

AI visibility check for nytimes.com: 4 of 20 AI bots allowed in robots.txt, 2 of 9 live requests got through

If you're a business that wants to be recommended by AI tools, your result should look like the second one.

A faster check

If you don't want to script it, we built a free checker at CodedSEO that does both layers for 20 bots and also looks for llms.txt: AI Visibility Checker. No sign-up.

Either way, check the firewall, not just the file.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.