Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check

More and more people find websites through AI assistants instead of a classic list of blue links. If an assistant can't fetch your pages, it can't quote them, summarize them or link to them. The annoying part: you can be blocked without ever deciding to block anyone.

I ran into this while building BotReady, a set of free checks for AI crawler access. Below is the manual version of the same checklist, so you can verify your own site with nothing more than a browser and a terminal.

1. Know which bots you are dealing with

AI companies usually run several user agents, and they do different jobs. Blocking one does not block the others.

The practical takeaway: "I don't want my content used for training" and "I want to show up when someone asks ChatGPT about my product" are two different robots.txt decisions. Many sites block everything with one rule and lose visibility they actually wanted.

Also note that user-triggered fetchers (a person pasting your URL into a chat) are treated differently by some vendors and may not follow robots.txt the same way a crawler does. Check each vendor's docs for the current behaviour.

2. Read your robots.txt like a bot would

Open https://yourdomain.com/robots.txt and look for three things:

1. A blanket User-agent: * + Disallow: / (blocks everyone, including AI bots).
2. Explicit Disallow rules for the bots above, often added by a CMS plugin or a security preset you forgot about.
3. Groups that accidentally match. A bot follows the most specific User-agent group that matches it and ignores the rest, so a GPTBot group without an Allow can override your generous * group.

A common "search yes, training no" setup looks like this:

# Allow AI search / answer engines
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This is a policy choice, not a recommendation: decide what you want, then make the file say exactly that.

3. Check the layer robots.txt can't see: your CDN or firewall

This is the one that surprises people most. robots.txt is a polite request. A CDN or WAF rule is a hard block that returns 403 before your server is even involved.

• Cloudflare has a one-click setting to block AI bots, and newer zones can have AI crawler blocking turned on by default. Look in the dashboard under the bots / AI crawl control section.
• Hosting "bot protection" and security plugins sometimes challenge any unknown user agent with a JavaScript or CAPTCHA page. A crawler can't solve that, so it sees an empty page.

A quick first test from your terminal:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://yourdomain.com/

Repeat with ClaudeBot, PerplexityBot, etc. 200 is good; 403, 429 or 503 deserves a look.

One caveat: spoofing the user agent from your laptop is not the same as a real crawler request. Many firewalls verify bots by IP range, so a fake GPTBot may be blocked while the real one is allowed, or the other way around. The test tells you whether UA-based rules exist; your logs tell you the truth (step 5).

4. Make sure there is something to read without JavaScript

Most AI crawlers fetch raw HTML and do not run your JavaScript. If your page is a client-side SPA, they may get an empty <div id="root"></div>.

curl -s https://yourdomain.com/your-page | grep -i "a sentence from your page"

If the text you care about isn't in the raw HTML, consider SSR, static generation or prerendering for content pages. While you're there, check that the page has a clear <title>, one <h1>, a meta description and valid structured data. These help any machine reader, not only AI ones.

5. Look at your server logs

This is the only place you see what really happened. Search access logs for the bot names:

grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user" access.log \
  | awk '{print $9}' | sort | uniq -c

That prints the status codes each bot received. Lots of 403s mean something is blocking them. No hits at all may mean they never got past the CDN (check CDN logs) or simply haven't discovered the site yet. Anyone can fake a user agent, so verify important hits against the vendor's published IP ranges.

6. Optional: llms.txt

/llms.txt is a proposed Markdown file that gives AI tools a short map of your most important pages. It's not a standard that crawlers are required to use, and it doesn't replace robots.txt or a sitemap. It's cheap to add, though, and handy for documentation sites.

The short version

1. Decide separately on training and AI search access.
2. Make robots.txt say exactly that, per user agent.
3. Check the CDN/WAF, because it overrides everything else.
4. Serve real content in the HTML.
5. Confirm in your logs.

If you'd rather not do it by hand, the AI Crawler Checker and AI Visibility Checker on BotReady run these checks for a URL in one go, and the robots.txt generator writes the per-bot rules for you. Free, no sign-up for the first checks.

What did you find on your own site? I'm curious how many people discover a forgotten firewall rule. 👀