We read the robots.txt of 2,347 MCP hosts: Python 3.12's parser gets 252 endpoints wrong

Two crawlers that log MCP servers compared notes on 1 November, and one server didn't add up. www.apier.no had changed its tool definitions twice in a day. Our log, HISTOR, recorded both changes. ScraperWatch, which honours robots.txt, skipped the server entirely. Its robots.txt looked clear enough (an excerpt; the group has twelve more Allow lines):

User-agent: *
Allow: /api/mcp$
Disallow: /api/

The site even explains the order in a comment: Allow lines come first so that first-match crawlers let the public paths through. ScraperWatch's crawler used Python's urllib.robotparser, and it still said no. ScraperWatch switched to an RFC 9309 reader the same day.

We wanted to know whether that was one unlucky server or a pattern. So we fetched the robots.txt of every host on ScraperWatch's panel, 2,347 hosts behind 2,995 MCP endpoints drawn at random from the official MCP registry. Then we asked two versions of Python the same question about each endpoint.

What the standard says, and what Python did

RFC 9309, the robots.txt standard since September 2022, has two rules that matter here:

• The most specific rule wins. Of all the Allow and Disallow lines that match a path, the one with the longest path applies; on a tie, Allow wins. The order of the lines doesn't matter.
• * and $ are special. * matches any run of characters, and a $ at the end means the path must end there. Allow: /api/mcp$ allows /api/mcp and nothing below it.

Python's urllib.robotparser did neither. It took the first matching line in file order, and it read * and $ as ordinary characters. Someone reported the ordering half in January 2020 (bpo-39187). The fix, gh-138907 by Serhiy Storchaka, was merged on 4 May 2026 and backported:

We checked each row against the tagged source. The 3.12 line only takes security fixes now, so it will not change. Ubuntu 24.04 LTS ships Python 3.12. The python3 on the Mac this post was written on is 3.9.6.

You can check your own interpreter in four lines:

from urllib.robotparser import RobotFileParser
p = RobotFileParser()
p.parse(["User-agent: *", "Allow: /", "Disallow: /api/"])
print(p.can_fetch("ExampleBot", "https://example.com/api/mcp"))

Python 3.12.13 and 3.9.6 print True; 3.14.8 prints False. Under RFC 9309, /api/ is longer than /, so the Disallow wins.

The survey

On 1 November between 19:06 and 19:10 UTC we made one request to https://<host>/robots.txt for each of the 2,347 hosts. We followed redirects and stopped at 10 seconds. Then we ran the same script under two interpreters. Each asked urllib.robotparser whether a crawler with no group of its own, so the * group applies, may fetch each endpoint:

As a third opinion we ran HISTOR's own RFC 9309 reader, written separately for our cross-checks. It agrees with Python 3.14.8 on all 1,764.

The disagreement runs almost entirely one way. On 250 endpoints, the old parser says "go ahead" where the site said no. On 2 it says no where the site said yes.

ScraperWatch found the same in its own log. Once the new parser ran, it checked what the old one had done: from 30 October to 1 November it had read about 250 endpoints of this panel whose robots.txt disallows them. It removed those reads and the change records they produced, and its page now says so. The miss we started with, apier.no, was the rare direction, an allowed endpoint refused. The common one, disallowed endpoints read, went unnoticed until the parser changed.

"Allow: /" first

218 of the 252 come down to rule order, and nearly all of them look like this:

User-agent: *
Allow: /
Disallow: /api/
Disallow: /dashboard/

The author meant "everything except the API and the dashboard". Under RFC 9309, /api/mcp hits Disallow: /api/, the longer rule. The old parser reaches Allow: / first, stops, and allows the whole site. 619 of the 1,315 files put Allow: / before at least one Disallow. On those hosts, a crawler on Python 3.12 never reaches any Disallow line after Allow: /.

On the 250 endpoints the old parser lets through, the rule the site actually applies is a /api prefix on 159 of them. On 85, it is a Disallow that names the MCP path itself: /mcp, /mcp/, /mcp$ and the like. Those sites singled out their MCP endpoint for crawlers to leave alone, and a crawler on old Python reads it anyway.

$ written for agents

The other 34 involve * or $. The two endpoints that the old parser wrongly refuses are the most interesting, because both sites wrote their rules for AI agents on purpose:

• www.apier.no: Allow: /api/mcp$ before Disallow: /api/.
• api.adplug.app: nine lines like Allow: /mcp/linkedin-ads$, one per MCP endpoint, then Disallow: /mcp.

Both open exactly the MCP endpoint and nothing under it. To the old parser, /api/mcp$ is a literal path with a dollar sign at the end, which no URL has, so the Allow never matches and the broad Disallow wins. The sites that tried hardest to welcome agents are the ones a Python 3.12 crawler turns away.

What this means for MCP crawlers

Most of those Disallow lines were written for search engines: /api/, /dashboard, /admin. Whether a monitor or an MCP client should honour them for an endpoint the publisher has listed in the MCP registry is a fair question. ScraperWatch honours robots.txt. Until 2 November HISTOR didn't consult it for registry-listed servers; since then it does, reading the histor group, or * if there is none, with RFC 9309's longest match. On the registry snapshot of 26 October, 4,708 of 33,520 remote endpoints (14%) disallow it. HISTOR's page for each of them says so and names the rule. Where robots.txt allows it, HISTOR still reads the server's MCP server card, the operator's own list of its tools.

If your crawler does honour robots.txt, it should read the file the way the site meant it. Concretely:

• Run Python 3.13.14+, 3.14.5+ or 3.15. On 3.12 or older, use a parser that implements RFC 9309 rather than urllib.robotparser.
• Test with Allow: / first and with a $ rule. The four lines above cover the first; Allow: /api/mcp$ before Disallow: /api/ covers the second.
• If you publish an MCP endpoint and want agents to reach it, Allow: /api/mcp$ is correct, and the longest-match rule makes it work in any order. Crawlers on old Python will still miss it. A plain Allow: /api/mcp without the $, placed before the Disallow, works in both, at the cost of also allowing everything below that path.

Data

Everything is at https://histor.modelmarket.dev/datasets/robots-mcp-hosts-2026-11-01/:

• the panel;
• the fetch script and the 2,347 responses;
• the script that runs under each interpreter;
• the 252 disagreements with the rules that caused them.

The panel itself is ScraperWatch's: https://scraperwatch.com/mcp-changes/panel.txt.

Limits: one fetch per host from one network on one day, and only the * group. A crawler with a named group in a file can get a different answer.

Disclosure: we build HISTOR. AI assistance was used for the analysis, the code and the writing. Python versions were checked against CPython's tagged sources, and every count comes from the scripts above.