Check if your site or Cloudflare WAF is silently blocking GPTBot
We noticed a lot of domains have User-agent: GPTBot Allow: / in their robots.txt, but Cloudflare Managed Rules or Super Bot Fight Mode still return a 403 challenge at the edge before the crawler ever touches the origin.
We noticed a lot of domains have User-agent: GPTBot Allow: / in their robots.txt, but Cloudflare Managed Rules or Super Bot Fight Mode still return a 403 challenge at the edge before the crawler ever touches the origin. Built a quick, no-signup tool that tests live cURL responses across major AI crawlers (GPTBot, ClaudeBot, PerplexityBot). Feedback and bug reports welcome!
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.