What a server knows about you before it reads a single header

Last week I sent the same HTTPS request to the 500 most popular websites from eight countries, and a fifth of them refused it everywhere. The refusals were not geographic, they were not rate limits, and they arrived just as readily from my flat in Sofia as from a rack in Singapore, which ruled out most of the explanations I had gone in with and left me fairly sure that something about the request itself was the problem.

I said at the time that I could not tell whether those sites were blocking programs in general or blocking my particular program, because my client was Python's urllib wearing a Chrome user agent, and that is a conspicuous combination. This post is me going and finding out what a server actually sees when my script knocks on the door.

The short version is that the user agent is close to the least interesting thing about a request, and a server knows what you are long before it reads a single header.

The handshake gives you away first

Before any HTTP exists, before a header is sent, before your user agent gets a chance to lie on your behalf, your client sends a TLS ClientHello. That message is not a formality. It is a detailed statement of what your TLS library can do, in the order it prefers to do it, and different libraries make that statement very differently.

Here is the cipher list my Python sends, and the one Chrome sends, to the same server seconds apart:

chrome : 4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53
python : 4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192- ...

Chrome offers fifteen cipher suites. Python offers thirty. Both start with the same three TLS 1.3 suites, but not in the same order: Chrome leads with 4865, Python with 4866. Nothing there is wrong or insecure. It is simply a different library with different defaults, and that difference is stable enough to be a name badge.

The industry name for hashing that badge is JA3, and the newer one is JA4. Running my two clients against a service that reports what it sees:

You can read those without a decoder ring. t13 is TLS 1.3. The four digits after the d are the number of cipher suites and the number of extensions: Chrome offers 15 and 17, Python offers 30 and 14. Then comes the ALPN, the protocol the client says it wants to speak. Chrome says h2, meaning it would like HTTP/2. Python says h1, because urllib has no HTTP/2 implementation to offer.

That last field on its own is close to conclusive. Essentially every browser in use speaks HTTP/2, and has for years. A client announcing in its first packet that it can only manage HTTP/1.1 has effectively introduced itself.

There is a second fingerprint above the first

If the connection does get as far as HTTP/2, there is another signature waiting, because HTTP/2 makes clients state their preferences up front too. The SETTINGS frame carries buffer sizes and limits, then there is the window update, the priority scheme, and the order the pseudo-headers arrive in.

Chrome's, in the compact notation these tools use:

1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p

Those numbers are the header table size, whether server push is enabled, the initial window size and the maximum header list size, followed by a window increment, a priority value, and finally m,a,s,p — method, authority, scheme, path, in that order. Another client might send the same four pseudo-headers in a different order and be perfectly correct while looking nothing like Chrome.

My Python client has no HTTP/2 fingerprint at all. Not a different one. None, because it never speaks HTTP/2, so the question never arises.

A digression about GREASE, which broke my first attempt at measuring this

I ran Chrome's handshake five times, expecting five identical fingerprints, and got five different ones instead:

8d6c93cfb40166f6899fac8d3b9a5650
585f0d40adbc5a1c63f3acca112db20c
f74e76d2f7aa21502b28bff6f6b3b9d4
655ddf67c23fab2041d5abbef66304d9
a1713b0d46fcc281cffbd6657d92ea5a

Five JA3 hashes from one browser on one machine inside a single minute, which sent me back to check the capture code twice on the assumption that I had broken something.

It was not. Chrome deliberately inserts randomly chosen nonsense values into its cipher and extension lists, a mechanism called GREASE, specifically so that middleboxes and servers cannot come to depend on the exact shape of a Chrome handshake, which means every connection it makes looks slightly different from the last one by design.

The same five handshakes produced exactly one JA4 string, because JA4 sorts the list and strips the GREASE values before hashing. That is the entire reason JA4 exists: JA3 turned out to be too easy to scramble, so the fingerprint was redesigned to survive the scrambling, which is a slightly deflating outcome for a feature originally built to make fingerprinting harder.

What this means for the thing I measured last week

When I reported that a fifth of the top 500 refuse a scripted request, I was describing a client that announced itself in at least four ways before the server read a byte of HTTP. It offered twice as many ciphers as a browser, in a different order, with a different extension set, announced that it could only manage HTTP/1.1, and then some way further up the stack claimed in a header to be Chrome 140, contradicting everything it had already said about itself.

Whether those sites are hostile to programs in general, or merely hostile to that specific contradiction, is not something the previous experiment could answer. The user agent is trivially editable and that is why nobody serious is looking at it. The handshake underneath is a property of the software you built with, and changing it means changing your TLS stack rather than a string.

So that is the next experiment, and the numbers from it are in the second half of this, which goes up a few hours after this one. I took the same 500 domains, the same servers in New York, Frankfurt and Singapore, and gave the script a real Chrome TLS fingerprint — the same cipher list in the same order, the same extensions, the same HTTP/2 settings, the same pseudo-header ordering — while keeping everything else identical.

If the refusals were about the handshake, the number should move a long way. If they were about the address the request came from, it should barely move at all. I genuinely did not know which before I ran it.