A Third of the AI Crawlers Hitting Our Site Were Fake

I wanted to know whether AI assistants actually read our site, so I grepped the nginx logs for the user agents. The numbers looked encouraging:

ClaudeBot       1136
GPTBot           420
PerplexityBot    356
Applebot         230
ChatGPT-User     213
OAI-SearchBot    196

Then I checked where those requests actually came from, and a third of them were not who they said they were.

The companies publish their IP ranges

This is the part most people do not know exists. Every serious crawler operator publishes the address ranges their bots use, specifically so that you can verify them:

• openai.com/gptbot.json — the training crawler
• openai.com/searchbot.json — the one that builds the index ChatGPT searches
• openai.com/chatgpt-user.json — live fetches when a user's question triggers browsing
• search.developer.apple.com/applebot.json — Applebot, which serves Spotlight, Siri and Apple Intelligence

Each file is a list of CIDR prefixes. Checking a log line against them is twenty lines of code and no dependencies:

import ipaddress, json, urllib.request