Which AI crawlers does your robots.txt really block? An RFC 9309 audit in stdlib Python
Most "does my site block GPTBot?" checks are a grep for the bot name. That misses how crawlers actually read robots.txt: a bot obeys only the most specific User-agent group that names it, falls back to * only when nothin
Most "does my site block GPTBot?" checks are a grep for the bot name. That misses how crawlers actually read robots.txt: a bot obeys only the most specific User-agent group that names it, falls back to * only when nothing names it, and inside a group the longest matching path rule wins (RFC 9309). Python's built-in urllib.robotparser doesn't do the longest-match part, which matters more than you'd think.
This post builds a small stdlib-only script that audits a list of domains against 13 AI crawlers, grouped by what they're for: model training, AI search citations, and user-triggered fetches (an assistant opening a link a person asked about). Blocking one group is a different decision from blocking the others, and many sites block more than they meant to.
The trap in urllib.robotparser
import urllib.robotparser as rp
rules = "User-agent: *\nDisallow: /\nAllow: /blog/\n"
p = rp.RobotFileParser()
p.parse(rules.splitlines())
print(p.can_fetch("GPTBot", "/blog/post")) # False
Under RFC 9309 /blog/post is allowed: Allow: /blog/ (7 characters) is a longer match than Disallow: / (1 character). robotparser returns the first rule in file order instead, and it doesn't support * / $ wildcards in paths either. Fine for a polite crawler that errs on the side of not fetching; wrong for an audit.
The script (~90 lines, no dependencies)
import re, sys, urllib.request, urllib.error
BOTS = { # token: purpose
"GPTBot": "training", "ClaudeBot": "training", "CCBot": "training",
"Google-Extended": "training", "Applebot-Extended": "training",
"Meta-ExternalAgent": "training", "Bytespider": "training",
"OAI-SearchBot": "search", "PerplexityBot": "search", "Claude-SearchBot": "search",
"ChatGPT-User": "user", "Claude-User": "user", "Perplexity-User": "user",
}
def fetch_robots(domain):
req = urllib.request.Request(f"https://{domain}/robots.txt",
headers={"User-Agent": "robots-audit/1.0"})
try:
with urllib.request.urlopen(req, timeout=15) as r:
return r.read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
if 400 <= e.code < 500:
return "" # RFC 9309: 4xx means no restrictions
return None # 5xx: treat as disallow all
except Exception:
return None
def parse(text):
groups, agents, rules, last_was_ua = [], [], [], False
for line in text.splitlines():
line = line.split("#", 1)[0].strip()
if ":" not in line:
continue
key, val = (s.strip() for s in line.split(":", 1))
key = key.lower()
if key == "user-agent":
if not last_was_ua and agents:
groups.append((agents, rules)); agents, rules = [], []
agents.append(val.lower()); last_was_ua = True
elif key in ("allow", "disallow"):
rules.append((key, val)); last_was_ua = False
else:
last_was_ua = False
if agents:
groups.append((agents, rules))
return groups
def rules_for(groups, bot):
bot = bot.lower()
named = [r for a, r in groups if bot in a]
if named:
return [x for r in named for x in r], "named"
star = [r for a, r in groups if "*" in a]
return [x for r in star for x in r], ("*" if star else "none")
def matches(pattern, path):
rx = "".join(".*" if c == "*" else "$" if c == "$" and i == len(pattern) - 1
else re.escape(c) for i, c in enumerate(pattern))
return re.match(rx, path) is not None
def allowed(rules, path="/"):
best = ("allow", -1)
for kind, pat in rules:
if not pat and kind == "disallow":
continue # "Disallow:" with no path allows everything
if matches(pat, path):
n = len(pat)
if n > best[1] or (n == best[1] and kind == "allow"):
best = (kind, n)
return best[0] == "allow"
for domain in sys.argv[1:]:
text = fetch_robots(domain)
if text is None:
print(f"{domain}: robots.txt unreachable -> treat as disallow all"); continue
groups = parse(text)
blocked = {}
for bot, purpose in BOTS.items():
rules, via = rules_for(groups, bot)
if not allowed(rules):
blocked.setdefault(purpose, []).append(f"{bot}({via})")
print(domain)
for purpose in ("training", "search", "user"):
print(f" {purpose:8} blocked: {', '.join(blocked.get(purpose, [])) or '-'}")
What it does, in order:
-
Fetch
https://<domain>/robots.txt. A 4xx means "no restrictions"; a 5xx or an unreachable host is treated as "disallow all", as RFC 9309 asks crawlers to do. -
Parse into groups. Consecutive
User-agentlines share one group, which is how sites write "block all of these" lists. -
Pick the group: every group that names the bot (merged), else the
*group, else nothing. -
Match with
*and$wildcards, longest pattern wins,Allowwins a tie. An emptyDisallow:allows everything.
Real output (run on 19 November 2026)
$ python3 ai_bots.py nytimes.com github.com anthropic.com theguardian.com stackoverflow.com example.com
nytimes.com
training blocked: GPTBot(named), ClaudeBot(named), CCBot(named), Google-Extended(named), Applebot-Extended(named), Meta-ExternalAgent(named), Bytespider(named)
search blocked: OAI-SearchBot(named), PerplexityBot(named), Claude-SearchBot(named)
user blocked: ChatGPT-User(named), Claude-User(named), Perplexity-User(named)
github.com
training blocked: -
search blocked: -
user blocked: -
anthropic.com
training blocked: -
search blocked: -
user blocked: -
theguardian.com
training blocked: ClaudeBot(named), CCBot(named), Applebot-Extended(named), Meta-ExternalAgent(named), Bytespider(named)
search blocked: PerplexityBot(named), Claude-SearchBot(named)
user blocked: Claude-User(named)
stackoverflow.com
training blocked: -
search blocked: -
user blocked: -
example.com
training blocked: -
search blocked: -
user blocked: -
Two things worth noticing:
-
Grouped user-agents block more than the training bot. The Guardian's file lists
ClaudeBot,Claude-SearchBotandClaude-Userin one group, so the search and user-fetch agents are blocked along with the training crawler. That may be exactly the intent, but it's the most common way sites drop out of AI search answers without meaning to. -
(named)vs(*)tells you why. If a bot is blocked via*, your general rules caught it; if it's(named), someone wrote a rule for it on purpose.
The script only checks /. To audit specific sections, call allowed(rules, "/blog/") with the paths you care about.
Limits (be honest with yourself about what this proves)
- robots.txt is a request. A CDN, WAF or bot-management setting can block a crawler that robots.txt allows, and a crawler that ignores robots.txt isn't stopped by it. Check your CDN's AI-crawler settings and server logs too.
- Bot tokens change. Confirm the list against each operator's own documentation before relying on it.
- It doesn't read
Content-Signallines (e.g.ai-train=no) orllms.txt.
No-code version
If you just want the answer for one domain, the free AI crawler checker on siftwright.com (no signup) runs the same RFC 9309 matching for 24 crawlers, groups them by training / search / user, lists Content-Signal lines and validates llms.txt, including links in llms.txt that your own robots.txt blocks. It also has copy-paste snippets for "block training, stay in AI search".
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.