Your analytics said 32 visitors yesterday. Nine of them were people.

Yesterday my dashboard said 32 visitors, 20 unique. I spent the evening taking it apart. Nine of them were people.

Not nine out of twenty. Nine out of thirty-two page views. The unique-visitor number was inflated by 55% and the page-view number by 34%, on a day with no marketing push, no post, nothing that would attract a spike.

The interesting part is not that bots were in the numbers. That is old news. The interesting part is why my bot filter missed them, because the answer is a problem every site now has and almost nobody has fixed.

What my filter actually did

The backend classifies each hit by user agent:

const UA_BOT = /bot|crawl|spider|slurp|headless|phantom|puppeteer|playwright|selenium|curl\/|wget|python-|go-http-client|java\/|okhttp|axios|node-fetch|scrapy|semrush|ahrefs|mj12|dotbot|petalbot|yandex|facebookexternalhit|applebot|discordbot|telegrambot|whatsapp|uptime|monitor|scanner|nmap|masscan|googleother|google-inspectiontool|apis-google|mediapartners|feedfetcher|builtwith|dataforseo|lighthouse/i;

function classifyUA(ua) {
  const s = String(ua || '').slice(0, 200);
  if (!s) return '?';
  if (UA_BOT.test(s)) return 'bot';
  if (UA_TB.test(s)) return 'tb';
  if (UA_MB.test(s)) return 'mb';
  if (UA_PC.test(s)) return 'pc';
  return '?';
}

That regex is long. It catches Applebot, GoogleOther, facebookexternalhit, headless Chrome, and about thirty things I have never seen on my own traffic. On paper it looks thorough.

Here is a hit it let through, straight from the log:

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51
Mobile Safari/537.36 MicroMessenger/7.0.1

Read it word by word. Android 6.0. Nexus 5. Chrome/99. MicroMessenger/7.0.1 — it claims to be WeChat on a phone. There is no bot, no crawl, no headless, nothing my regex could possibly match. It walked in as mb — mobile human.

Thirty-one different IP addresses sent that exact string over twelve days.

WeChat has over a billion users. The odds that thirty-one of them are on a Nexus 5 running Android 6.0, sharing a build number from 2015, are not small — they are zero. That UA is the default device profile of a browser automation tool, and whoever runs it knows the filter only reads words.

The trap I nearly fell into myself

Then there was the opposite mistake, which is worse.

Another fourteen hits in a single day carried a version number ending in three zeros:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36

Chrome/145.0.0.0. My first instinct was: nobody's real Chrome reports .0.0.0, that is obviously forged.

Wrong. Since Chrome 110, the browser reduces its own UA. 145.0.0.0 is what genuine Chrome sends. There are no minor or patch digits any more, for privacy. My instinct would have deleted real Mac users from the report and I would have called it a job well done.

So the string is real, and simultaneously useless: every Chrome 145 user on macOS now presents the identical UA. Real visitors and fake ones are indistinguishable by fingerprint alone.

That is the actual problem. UA reduction did not just remove entropy from the header — it gave crawlers a crowd of legitimate users to hide in. A UA blacklist is now structurally unable to separate them, no matter how many keywords you add.

The question that does work

Stop asking "is this UA a robot?" Ask "is this UA too popular for the number of IPs using it?"

Real people are metered two ways: their IP is stable (one address, many page views) or they appear once and leave (one address, one view). A fleet rotating through proxies looks like neither — it looks like many addresses, each visiting exactly once, all presenting the same header.

So for every UA string I compute three numbers across the whole dataset:

• ipn — how many distinct IP addresses used it
• pv — total page views from it
• per — pv / ipn

and flag it when ipn >= 3 and per <= 1.6.

// Cloudflare Worker / D1 — run once, cache the result
async function fingerprintReuse(db) {
  const { results } = await db.prepare(`
    SELECT ua,
           COUNT(*)                       AS pv,
           COUNT(DISTINCT ip)             AS ipn,
           COUNT(DISTINCT d)              AS days
    FROM hits
    WHERE ua IS NOT NULL
    GROUP BY ua
  `).all();

  const fleet = new Map();
  for (const r of results) {
    const per = r.pv / r.ipn;
    fleet.set(r.ua, {
      ipn: r.ipn,
      pv: r.pv,
      per,
      squad: r.ipn >= 3 && per <= 1.6,
    });
  }
  return fleet;
}

Then the per-hit decision becomes a lookup instead of a regex:

function judge(ua, fleet) {
  const s = String(ua || '').toLowerCase();
  if (s.includes('meta-webindexer')) return 'bot';   // Facebook preview, not in any list
  const f = fleet.get(ua);
  if (f && f.squad) return 'bot';                    // fleet fingerprint
  return classifyUA(ua);                             // fall back to keywords
}

The two conditions are doing different work. ipn >= 3 requires the coincidence to happen more than twice. per <= 1.6 is what separates a fleet from a shared office: ten colleagues behind one NAT gateway is one IP with hundreds of views, so per goes up, not down.

What the data looked like after

The top offenders on a site with a few hundred views a day:

Fifty-six addresses, fifty-six page views, thirteen days, exactly one view each. No human population behaves like that. Every one of those hits had already passed the keyword filter.

A second signal shows up in the same table if you look at the version numbers. My real visitors arrive on Chrome 145 and 156. The fleet arrives on Chrome 75, 79, 83, 88, 92, 99 — build numbers from 2019, on three separate IPs each, one view per IP. Twelve crawler fingerprints, not one of them running anything newer than 2021.

That is a cheap heuristic on its own: when a UA claims a browser version that has been obsolete for four years, and only three addresses ever used it, believe neither. A slow-moving UA library is the fingerprint of automation, because browsers auto-update and people do not stay on Chrome 79 for four years.

The catch is that this cannot be your only rule either — a real visitor on a six-year-old Android phone looks exactly the same. Reuse across IPs is the part that survives that objection; version staleness is a supporting signal, not a verdict.

Re-running the classifier with fingerprint reuse, day by day:

Across the whole dataset: 452 reported page views became 300. 245 reported unique visitors became 110.

Nearly every day my dashboard had been off by a third, and the one day it reported 42 unique visitors the true number was 20. For two weeks I had been making decisions about content and distribution on a number that was, on average, half fiction.

Where this approach still fails

Two honest limits, because both will bite you.

It cannot see a crawler that keeps one IP. A single-address scraper has ipn = 1 and never triggers. If that matters, the only reliable signal is request rate per IP per minute, which means keeping raw request logs, not just visits.

It will occasionally misclassify a real person. Someone on a mobile carrier with CGNAT, browsing once, on a version of Chrome where every user shares a UA string, is structurally identical to a fleet member. My threshold (ipn >= 3) is set to be conservative — it leaves some bots in rather than deleting real readers. If your traffic is large enough that ipn >= 3 is common for genuine users, raise the threshold and inspect the strings by hand before trusting the filter.

Aggregate across days, not within one. A fleet may only send two hits a day. Judged on a single day it looks harmless; judged over two weeks the reuse is obvious. My first attempt ran the query per-day and flagged nothing, because 56 addresses spread over 13 days is only 4 per day.

Why I bothered

I run a construction-calculator site — concrete volumes, rebar weights, formwork areas. It is a small site, a few hundred page views a day, and the only thing I have to steer by is where the visitors come from and what they look at. If a third of that is noise, every conclusion drawn from it is noise too: which page to expand, which channel to drop, whether that flurry on Tuesday was worth repeating.

The number on the dashboard is not traffic. It is a claim about traffic, and it is worth checking about as often as you check a supplier's invoice.

The site is jianzhujisuan.com — Chinese-language calculators for construction quantity takeoff. The analytics code above runs on a Cloudflare Worker with D1 behind it; no third-party script, no cookie, which is also why there was nowhere else to look when the numbers went wrong.