Python got a 403 and curl didn't. It came down to one case-sensitive string

A few days ago I wrote about getting your public IP from the command line. In the Python section I mentioned, almost in passing, that the default urllib gets a 403 from ipcomet while curl doesn't. I wrote that line from memory. It kept bothering me, because "bot protection rejects it" isn't an explanation. So I tested it properly.

Same laptop, same connection, same IP, same endpoint:

curl https://ipcomet.com/api/ip              -> 200
Python urllib, default settings              -> 403

Something differs between those two requests. Finding out exactly what took less time than I expected, and the answer is narrower than "Python gets blocked".

First suspect: the headers

If a server tells two clients apart, it's looking at what they send. So I wrote a tiny server that echoes back the request headers it receives, ran it on localhost (no third party involved), and hit it with four clients.

# echo_headers.py: returns the headers each client sends, in order
import json
from http.server import BaseHTTPRequestHandler, HTTPServer

class Echo(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps(list(self.headers.items())).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8777), Echo).serve_forever()

What came back, in the order the headers arrived:

Nothing here looks more "bot-like" for urllib than for curl. curl sends the fewest headers of all and sails through, so the header count isn't it. urllib does send two things curl doesn't by default: Accept-Encoding: identity, which tells the server not to compress anything, and Connection: close. Neither looks unusual enough to justify a block.

That left the User-Agent, the one header whose value differs in an obvious way.

Same client, different User-Agent

To separate the header from everything else, I stopped changing the client and changed only the string. Same curl, same machine, one request per User-Agent, 1.5 seconds apart, against https://ipcomet.com/api/ip:

Twenty strings, 22 requests. Exactly one family was blocked: anything containing Python-urllib, with that capitalisation. The lowercase version passed. So did urllib alone and my-urllib-script/1.0, and so did every other non-browser client I tried, including an empty User-Agent.

Two reverse checks agree. curl sending Python-urllib/3.9 got the 403 all three times I tried it, and Python's urllib sending my-script/1.0 got a 200.

What the 403 looks like

The response is a 403 with a body of 17 bytes that contains the code 1010. That is Cloudflare's error for access denied based on the browser's signature, and ipcomet is served through Cloudflare. I didn't trace which rule produced it. The matrix already tells a client everything it needs to know about what the rule matches. It's a plain string match on a known default, which is why it's so easy to hit by accident and so confusing to debug. Nothing about the response says "change your User-Agent".

What I'd do about it

• Set a descriptive User-Agent on every script, something like my-tool/1.0 ([email protected]). It costs nothing, and it gives a site owner someone to contact instead of a reason to block you.
• Don't copy a browser's string to get around a block. A blocked default is a hint to identify yourself properly, and pretending to be Chrome sends the wrong signal.
• When a script gets a 403 with a tiny body, check the User-Agent first. It takes ten seconds.

In urllib the fix is one argument:

import urllib.request

req = urllib.request.Request(
    "https://ipcomet.com/api/ip",
    headers={"User-Agent": "my-tool/1.0 ([email protected])"},
)
print(urllib.request.urlopen(req, timeout=10).read().decode().strip())

Run the same test on your own site

This is the script I used, rewritten to use only the standard library. Only point it at a site you own or have permission to test. It sends one request per string with a pause between them.

import sys, time, urllib.error, urllib.request

URL = sys.argv[1]
AGENTS = [
    "curl/8.7.1", "Python-urllib/3.9", "Python-urllib/3.12", "Python-urllib",
    "python-urllib/3.9", "urllib", "my-urllib-script/1.0", "python-requests/2.31.0",
    "python-httpx/0.27.0", "Go-http-client/1.1", "Wget/1.21.4", "node",
    "axios/1.6.0", "okhttp/4.12.0", "Java/17.0.2", "PostmanRuntime/7.36.0", "my-script/1.0",
]

def status(agent):
    req = urllib.request.Request(URL, headers={"User-Agent": agent})
    try:
        return urllib.request.urlopen(req, timeout=20).status
    except urllib.error.HTTPError as e:
        return e.code
    except Exception as e:  # timeouts and resets are results too
        return type(e).__name__

for agent in AGENTS:
    print(f"{agent:<26} {status(agent)}")
    time.sleep(1.5)

Run it with python3 ua_matrix.py https://your-site.example/. I ran exactly this version against ipcomet as well, with urllib doing the asking instead of curl, and every status matched the table above.

What this doesn't tell you

• It's one site's configuration. Other sites, including most of the ones you scrape, will behave differently. This maps one rule.
• I tested the libraries by their strings. requests and httpx weren't installed on this machine, so those rows come from curl sending their User-Agent. The real libraries also send other headers of their own, which could matter on a stricter site.
• Rules change. This is what one site did when I tested it.

If your lookups to ipcomet fail with a 403, the API reference already lists this under client notes.

I maintain ipcomet, a free IP checker. AI-assisted: I used Claude to help draft this post and to run the tests. The results come from my own machine, and every request in the matrix went to my own site.