How to check whether AI crawlers can read your site (robots.txt and server logs)

AI answer engines only recommend or cite what they can fetch. Before you invest in GEO, confirm the crawlers can actually read your site. Here is a quick check.

The user agents to know

• GPTBot – OpenAI's crawler, used to collect content that may be used to train models.
• ClaudeBot – Anthropic's crawler.
• PerplexityBot – indexes pages so Perplexity can cite them in answers.
• Google-Extended – a robots.txt token (not a separate bot) that controls whether Google may use your content for Gemini and AI features.

Note that OAI-SearchBot and ChatGPT-User also exist for search and live browsing, so check them if you want ChatGPT to cite you.

Step 1: read your robots.txt

Open https://yourdomain.com/robots.txt. A blanket Disallow: / under User-agent: *, or a rule copied from a security plugin, often blocks AI bots by accident. To explicitly allow them:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you prefer to keep some areas private, add Disallow: /private/ inside the relevant block.

Step 2: check the server access logs

robots.txt is a request, not enforcement, and CDNs or firewalls (Cloudflare bot rules, WAFs) can block bots before they ever see it. Your access log shows what really happens:

grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | tail -n 20

Count hits per bot and status code:

grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Look at the status codes. 200 means the bot read the page. 403 or 429 means your firewall or rate limiter is rejecting it. No hits at all over a couple of weeks may mean it is blocked upstream, or simply that your site is not yet discovered. Note that Google-Extended will not appear in logs, since Googlebot does the fetching.

Step 3: fix and re-test

Whitelist the bots in your CDN or WAF, keep pages server-rendered (many AI crawlers do not run JavaScript), and make sure your sitemap lists the key pages. Then re-check the logs after a week.

We wrote a longer glossary of overseas-ad and AI-search terms here: https://www.pandawm.com/blog/overseas-advertising-glossary.html