Dev.to WebDev 🛠 Dev 👁 0 📖 5 min read

Which AI crawlers does your robots.txt really block? An RFC 9309 audit in stdlib Python

Most "does my site block GPTBot?" checks are a grep for the bot name. That misses how crawlers actually read robots.txt: a bot obeys only the most specific User-agent group that names it, falls back to * only when nothin

Most "does my site block GPTBot?" checks are a grep for the bot name. That misses how crawlers actually read robots.txt: a bot obeys only the most specific User-agent group that names it, falls back to * only when nothing names it, and inside a group the longest matching path rule wins (RFC 9309). Python's built-in urllib.robotparser doesn't do the longest-match part, which matters more than you'd think.

This post builds a small stdlib-only script that audits a list of domains against 13 AI crawlers, grouped by what they're for: model training, AI search citations, and user-triggered fetches (an assistant opening a link a person asked about). Blocking one group is a different decision from blocking the others, and many sites block more than they meant to.

The trap in urllib.robotparser

import urllib.robotparser as rp

rules = "User-agent: *\nDisallow: /\nAllow: /blog/\n"
p = rp.RobotFileParser()
p.parse(rules.splitlines())
print(p.can_fetch("GPTBot", "/blog/post"))   # False

Under RFC 9309 /blog/post is allowed: Allow: /blog/ (7 characters) is a longer match than Disallow: / (1 character). robotparser returns the first rule in file order instead, and it doesn't support * / $ wildcards in paths either. Fine for a polite crawler that errs on the side of not fetching; wrong for an audit.

The script (~90 lines, no dependencies)

import re, sys, urllib.request, urllib.error

BOTS = {  # token: purpose
    "GPTBot": "training", "ClaudeBot": "training", "CCBot": "training",
    "Google-Extended": "training", "Applebot-Extended": "training",
    "Meta-ExternalAgent": "training", "Bytespider": "training",
    "OAI-SearchBot": "search", "PerplexityBot": "search", "Claude-SearchBot": "search",
    "ChatGPT-User": "user", "Claude-User": "user", "Perplexity-User": "user",
}

def fetch_robots(domain):
    req = urllib.request.Request(f"https://{domain}/robots.txt",
                                 headers={"User-Agent": "robots-audit/1.0"})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            return r.read().decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        if 400 <= e.code < 500:
            return ""              # RFC 9309: 4xx means no restrictions
        return None                # 5xx: treat as disallow all
    except Exception:
        return None

def parse(text):
    groups, agents, rules, last_was_ua = [], [], [], False
    for line in text.splitlines():
        line = line.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, val = (s.strip() for s in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if not last_was_ua and agents:
                groups.append((agents, rules)); agents, rules = [], []
            agents.append(val.lower()); last_was_ua = True
        elif key in ("allow", "disallow"):
            rules.append((key, val)); last_was_ua = False
        else:
            last_was_ua = False
    if agents:
        groups.append((agents, rules))
    return groups

def rules_for(groups, bot):
    bot = bot.lower()
    named = [r for a, r in groups if bot in a]
    if named:
        return [x for r in named for x in r], "named"
    star = [r for a, r in groups if "*" in a]
    return [x for r in star for x in r], ("*" if star else "none")

def matches(pattern, path):
    rx = "".join(".*" if c == "*" else "$" if c == "$" and i == len(pattern) - 1
                 else re.escape(c) for i, c in enumerate(pattern))
    return re.match(rx, path) is not None

def allowed(rules, path="/"):
    best = ("allow", -1)
    for kind, pat in rules:
        if not pat and kind == "disallow":
            continue            # "Disallow:" with no path allows everything
        if matches(pat, path):
            n = len(pat)
            if n > best[1] or (n == best[1] and kind == "allow"):
                best = (kind, n)
    return best[0] == "allow"

for domain in sys.argv[1:]:
    text = fetch_robots(domain)
    if text is None:
        print(f"{domain}: robots.txt unreachable -> treat as disallow all"); continue
    groups = parse(text)
    blocked = {}
    for bot, purpose in BOTS.items():
        rules, via = rules_for(groups, bot)
        if not allowed(rules):
            blocked.setdefault(purpose, []).append(f"{bot}({via})")
    print(domain)
    for purpose in ("training", "search", "user"):
        print(f"  {purpose:8} blocked: {', '.join(blocked.get(purpose, [])) or '-'}")

What it does, in order:

  1. Fetch https://<domain>/robots.txt. A 4xx means "no restrictions"; a 5xx or an unreachable host is treated as "disallow all", as RFC 9309 asks crawlers to do.
  2. Parse into groups. Consecutive User-agent lines share one group, which is how sites write "block all of these" lists.
  3. Pick the group: every group that names the bot (merged), else the * group, else nothing.
  4. Match with * and $ wildcards, longest pattern wins, Allow wins a tie. An empty Disallow: allows everything.

Real output (run on 19 November 2026)

$ python3 ai_bots.py nytimes.com github.com anthropic.com theguardian.com stackoverflow.com example.com
nytimes.com
  training blocked: GPTBot(named), ClaudeBot(named), CCBot(named), Google-Extended(named), Applebot-Extended(named), Meta-ExternalAgent(named), Bytespider(named)
  search   blocked: OAI-SearchBot(named), PerplexityBot(named), Claude-SearchBot(named)
  user     blocked: ChatGPT-User(named), Claude-User(named), Perplexity-User(named)
github.com
  training blocked: -
  search   blocked: -
  user     blocked: -
anthropic.com
  training blocked: -
  search   blocked: -
  user     blocked: -
theguardian.com
  training blocked: ClaudeBot(named), CCBot(named), Applebot-Extended(named), Meta-ExternalAgent(named), Bytespider(named)
  search   blocked: PerplexityBot(named), Claude-SearchBot(named)
  user     blocked: Claude-User(named)
stackoverflow.com
  training blocked: -
  search   blocked: -
  user     blocked: -
example.com
  training blocked: -
  search   blocked: -
  user     blocked: -

Two things worth noticing:

  • Grouped user-agents block more than the training bot. The Guardian's file lists ClaudeBot, Claude-SearchBot and Claude-User in one group, so the search and user-fetch agents are blocked along with the training crawler. That may be exactly the intent, but it's the most common way sites drop out of AI search answers without meaning to.
  • (named) vs (*) tells you why. If a bot is blocked via *, your general rules caught it; if it's (named), someone wrote a rule for it on purpose.

The script only checks /. To audit specific sections, call allowed(rules, "/blog/") with the paths you care about.

Limits (be honest with yourself about what this proves)

  • robots.txt is a request. A CDN, WAF or bot-management setting can block a crawler that robots.txt allows, and a crawler that ignores robots.txt isn't stopped by it. Check your CDN's AI-crawler settings and server logs too.
  • Bot tokens change. Confirm the list against each operator's own documentation before relying on it.
  • It doesn't read Content-Signal lines (e.g. ai-train=no) or llms.txt.

No-code version

If you just want the answer for one domain, the free AI crawler checker on siftwright.com (no signup) runs the same RFC 9309 matching for 24 crawlers, groups them by training / search / user, lists Content-Signal lines and validates llms.txt, including links in llms.txt that your own robots.txt blocks. It also has copy-paste snippets for "block training, stay in AI search".

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.