Fake GPTBot: How to Tell a Real AI Crawler From an Impostor
A fake GPTBot costs one line of configuration: a scraper that copies the User-Agent string OpenAI documents looks like the crawler in an access log, and the same holds for ClaudeBot, PerplexityBot and Googlebot. Anything
A fake GPTBot costs one line of configuration: a scraper that copies the User-Agent string OpenAI documents looks like the crawler in an access log, and the same holds for ClaudeBot, PerplexityBot and Googlebot. Anything more reliable has to come from what the client cannot type: the address its connection came from, and what DNS says about that address.
The User-Agent is the claim. The source address is evidence, and DNS can confirm it. A request earns a crawler's name only when that evidence agrees with it.
Fake GPTBot: three layers of evidence
- The User-Agent says who the client claims to be. The strings are published and anyone can paste them, so it proves nothing.
- The source IP against the operator's published list shows whether the connection came from an address the operator says its crawler uses. OpenAI, Anthropic, Perplexity, Google, Bing and Apple all publish such lists.
- Reverse DNS, confirmed by a forward lookup, shows whether the address's owner and the operator's own domain agree on a name. Google, Bing and Apple document it. OpenAI, Anthropic and Perplexity do not.
A signature on the request would outrank all three. OpenAI signs the requests of ChatGPT's cloud browser, an agent that acts for a user and is not a crawler, but the crawler pages of OpenAI, Anthropic and Perplexity describe IP lists and robots.txt rules and nothing signed.
GPTBot IP ranges: what a list match proves
Operators serve their ranges as JSON files: a creation time and a list of prefixes, which are address blocks written like 20.171.206.0/24. OpenAI publishes one file per bot: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. The GPTBot and ChatGPT-User files share no address, so a request that claims GPTBot from a ChatGPT-User address came from the right company and the wrong bot. Each name has to be matched against its own file, and the full guide to verifying AI crawlers lists the files of seven operators next to the bots they cover.
A match is strong evidence because the prefixes are narrow, but the network around them says little. In the files read on 19 November 2026, all 18 GPTBot prefixes sit inside Microsoft's Azure ranges, 13 of the 14 entries in Anthropic's file are single addresses inside Google Cloud ranges, and all 31 entries in Perplexity's two files sit inside Amazon's ranges. Anyone can rent a machine next door in each of those clouds. A check that stops at "this address is in Azure" proves nothing, and the exact prefix is what counts.
Lists carry a date, and the dates differ. On 19 November 2026 Google's files carried that day's date, OpenAI's GPTBot file dated from 22 September and its search file from 2 January, Anthropic's file from 20 October, PerplexityBot's from 18 November, and the Bingbot file reads January 2024 although Bing's help page asks sites to refresh it daily, so a date is only a clue. A copy downloaded once and kept goes stale either way: an address the operator gave up can end up with another customer of the same cloud, and a new pool looks like a fake. PerplexityBot's file shows the second case: it held 13 addresses on 7 November and 21 on 19 November, eight of them new.
A miss is weaker than a match. None of the three AI crawler pages promises that its list is complete, so for those operators an address off the list is unconfirmed, not proven fake.
A new Internet-Draft from October 2026, SAIL (Source Address Identity List), aims at exactly that gap. An operator serves a file at /.well-known/sail with the prefixes each of its services uses, and can add two promises: a service uses no other addresses, and nothing else the operator runs uses one particular pool of addresses. With those promises, a miss counts as proof that the request is fake. claude.com already serves such a file for Claude-Agent, its hosted browser, with both set. The draft is at revision 00 and not a standard, and a check on 19 November found no such file on openai.com, google.com, bing.com or perplexity.ai.
Forward-confirmed reverse DNS, step by step
Reverse DNS maps an address back to a name through a PTR record, and whoever holds the address block decides what that record says. The host behind a fake Googlebot can claim to be crawl-66-249-66-1.googlebot.com. The forward lookup exposes the claim, because only the owner of googlebot.com decides which addresses that name resolves to.
- Take the client address from the log and look up its name. The reverse DNS lookup on packet.guru returns the PTR names for an address you paste in.
- Check the name against the domains the operator names: googlebot.com or google.com for Google's crawlers, search.msn.com for Bing, applebot.apple.com for Apple.
- Look the name up forward, as an ordinary name-to-address query (A for IPv4, AAAA for IPv6), and see whether the answer is the address from step 1.
- Count a pass as confirmation. A missing name or a mismatch is no confirmation.
Two places where it breaks:
- OpenAI, Anthropic and Perplexity document no names. Reverse lookups of the first address in each published prefix returned no name for all 296 OpenAI entries, default AWS names for all 31 Perplexity entries and default Google Cloud names for 13 of Anthropic's 14. With no documented name there is nothing for a forward lookup to confirm.
- googleusercontent.com is a shared domain. Google's own steps accept it, and random addresses from Google's published Google Cloud ranges all carry a default name under bc.googleusercontent.com whose forward lookup returns the same address. A check that compares only the domain passes a rented VM. Compare the whole pattern Google documents for the crawler class, such as crawl- plus the address under googlebot.com, or fall back on the IP list.
When the client address is not the client's
Behind a CDN, proxy or load balancer, the address in the access log is the proxy's. The client's travels in a header such as X-Forwarded-For, which a client can also send itself, so only the part your own proxies add can be trusted. A rule written on the raw header lets a fake paste a Googlebot address into it, and if the server is also reachable directly from the internet, no part of the header is safe. Tell the web server which proxies it trusts (nginx has a trusted-address setting for this) so that the logged address is the real one before any list or DNS check runs.
What to do on the server
- Log the client address and the User-Agent on every request, plus the forwarded address when a proxy is in front, so a line can be checked later.
- Write an allow rule as the name plus the address range from that bot's own file. A rule on the name alone admits every fake, and one on a whole cloud admits every tenant of it.
- Re-download the lists on a timer, and when a download fails, go on using the previous version.
- Give confirmed crawlers their own rate limit and everything else the default one. ChatGPT-User, Perplexity-User and Google's user-triggered fetchers may not follow robots.txt, so a limit is often the only control left for them.
- Keep opt-outs in robots.txt. Blocking an operator's addresses can keep a well-behaved crawler from reading it.
- Paste a suspicious address, with its User-Agent, into the Crawler Verifier. It matches the address against each operator's published IP list and reverse DNS, and the result shows the list's date.
One log line, one verdict
| What the log shows | Verdict | Action |
|---|---|---|
| The name matches and the address is in that bot's own list, or its documented DNS name resolves back | Confirmed | Apply your policy for that bot |
| Claims GPTBot, address in the ChatGPT-User list | Right company, wrong bot | Apply the rule for the bot the address belongs to |
| Claims Googlebot, address in no Google list and no googlebot.com or google.com name that resolves back | Fake | Refresh the lists, check again, then block or challenge |
| Claims GPTBot, ClaudeBot or PerplexityBot from a cloud address outside that bot's list | Unconfirmed | Treat as an unknown client: default limits, outside every allow rule |
| Claims a bot whose operator publishes no list and no DNS rule | Unverifiable | Same as unconfirmed, and do not call it fake |
| Shows your own CDN's address | Cannot be judged | Fix the client address first |
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.