Can ChatGPT access your website? A 2-minute check (robots.txt, firewall, HTML)

Originally published on pagelens.io.

If ChatGPT can't read your pages, it can't quote them, recommend them or link to them. Most sites never check. Three things decide whether an AI assistant gets in, and you can test all three for free in a couple of minutes.

1. robots.txt: are AI bots allowed?

Open yoursite.com/robots.txt. Look for groups that name AI user agents, or a User-agent: * group with Disallow: /, which blocks everyone. The ones that matter for answers today:

• OAI-SearchBot builds the index ChatGPT search uses. Blocked here, you're unlikely to be cited in ChatGPT answers.
• ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks the assistant about it.
• GPTBot and ClaudeBot collect training data. Blocking them is a choice about training, not about appearing in answers.

Reading robots.txt by hand is easy to get wrong: a specific group overrides *, and the most specific path wins. A free AI crawler checker reads it the way crawlers do and shows each AI bot as allowed or blocked.

2. Your CDN or firewall

robots.txt is a request. A firewall is a wall. Many sites block AI bots at the CDN without knowing it:

• Cloudflare has a one-click setting that blocks AI bots, and since mid-2025 it's offered by default to new sites. Check the bot and AI crawler settings in your dashboard.
• Bot protection on other CDNs and WAFs, and some WordPress security plugins, challenge unknown agents with a JavaScript or CAPTCHA page that AI fetchers can't pass.

Quick test: ask ChatGPT (search on) or Claude to "open https://yoursite.com/pricing and summarise it". If it can't access the page, or summarises something generic, the request was probably stopped before it reached your content. Try a deep page, not only the home page.

3. Is the content in the HTML?

AI crawlers download the HTML and mostly don't run JavaScript. If your page is a single-page app that renders everything in the browser, the bot may see an empty <div id="root">.

curl -s -A "ChatGPT-User" https://yoursite.com/pricing | grep -i "your headline"

Nothing back? Render that page on the server or pre-render it at build time (Next.js, Nuxt, Astro and SvelteKit all can).

4. Check that they actually come

Allowed doesn't mean visited. AI bots never show up in JavaScript analytics, so you need server or CDN logs. I wrote up five ways to see AI crawlers from grep to automatic, and what the first week looked like on a brand new site.

Checklist

I build PageLens Analytics (pagelens.io), which shows AI crawler reads per page. The checks above work with any stack and cost nothing.