I built a free crawler audit tool: is your site invisible to ChatGPT, Claude and Gemini?

ChatGPT, Claude, Gemini and Perplexity are quietly becoming a second traffic source for many websites. But before any of them can cite you, their crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider — need to actually fetch your pages.

I built CrawlCheck to answer one question in under 30 seconds: can AI crawlers see your site, or are you accidentally blocking them? It is free and needs no signup.

Why sites get blocked without knowing it

While testing the tool on real sites, three failure modes came up again and again:

1. Stale CDN firewall rules. A WAF rule written years ago to stop scrapers now matches the user-agent strings that legitimate AI crawlers use. The site owner has no idea.
2. Copy-pasted robots.txt. Plenty of templates floating around the web include a blanket "User-agent: * disallow-all" block or a leftover CCBot ban from the "AI panic" era.
3. JS-only content. Some AI crawlers do not render JavaScript. If your key content only exists after hydration, they see an empty shell.

What the audit checks

• robots.txt directives for every major AI agent
• meta robots tags and X-Robots-Tag headers
• rendered DOM vs raw HTML (to catch JS-only content)
• response codes and redirect chains

Per-crawler reference pages

We documented 55 of the most common AI and search crawlers — what each one does, which pages it hits, and how to allow or block it. For example: GPTBot, ClaudeBot and CCBot.

Try the audit yourself: https://crawlcheck.dev — paste a URL and get the full report in seconds.

Feedback is welcome — which other checks would be useful to you?