5 Ways to Block AI Crawlers — What Actually Works (October 2026)

Everyone's blocking AI crawlers this month. Robots.txt files are getting edited at 2am. Cloudflare dashboards are getting toggled by people who have never opened the Security tab before.

Here's the uncomfortable bit: most of the blocks don't actually block anything.

I spent the last week digging through crawler docs, Cloudflare's bot-management behaviour, and the reports from people who actually watch their server logs. Not lab-tested claims — docs plus what site operators are seeing in the wild. This is the comparison I wish I'd found before I started toggling things.

If you haven't read it, last week's field guide — GPTBot vs OAI-SearchBot vs ChatGPT-User, what each bot actually does — is the companion to this one. That piece tells you which bot is knocking. This one tells you which lock actually holds.

The short version

Nobody sells you the one perfect switch. The honest answer is layers, and which layers depends on what you're actually afraid of.

1. robots.txt — the polite "please don't"

What it is: a text file at your domain root that says which crawlers may fetch which paths.

What it actually stops: the well-behaved ones. GPTBot, OAI-SearchBot, CCBot (Common Crawl), Google-Extended — they all check it and they all honour it. If your worry is "my content training the next model," this one line does most of the work:

User-agent: GPTBot
Disallow: /

The catch: it is a request. Scrapers running headless browsers, SEO tools, and the "we're definitely a real Chrome browser, promise" crowd don't read it. They fetch anyway. Blocking AI training and blocking scraping are two different problems, and this only solves the first.

Best for: everyone. It costs nothing, takes two minutes, and catches the biggest training pipelines. Just don't believe you're done.

2. Server-level user-agent blocks — free, brittle, and kind of fun

What it is: nginx/apache rules (or a WordPress plugin doing the same) that 403 any request whose user-agent string matches a known bot list.

What it actually stops: whatever's on your list, today. GPTBot announces itself. So do most of the honest ones.

The catch: this is a treadmill. Bots that want in rotate user agents — there are public lists of crawler UAs and equally public ways to spoof them. Every few weeks someone reports a "new" AI scraper using a generic Chrome UA string, and your blocklist quietly becomes decoration. Also: block the wrong string and you nuke a monitoring service, an uptime checker, or a reader's RSS app. I've seen people 403 their own health checks. Ask me how I know.

Best for: small static sites where you control the server config and like maintaining things. Not a strategy — a supplement.

3. Cloudflare Bot Fight Mode — the free bouncer

What it is: on Cloudflare's free plan, one toggle that serves JavaScript challenges (Managed Challenge) to traffic its heuristics flag as automated.

What it actually stops: the rude ones — scrapers, credential-stuffing scripts, the botnet-adjacent junk that doesn't bother identifying itself. It doesn't care about your robots.txt and it doesn't care about user-agent strings; it watches behaviour. In that sense it's the first method on this list that actually blocks rather than asks.

The catch: it's a blunt instrument. Bot Fight Mode doesn't distinguish "AI training crawler I'd like to allow" from "scraper stealing my content." It's binary: looks automated, gets challenged. Legit users on odd networks (corporate VPNs, some mobile carriers, privacy-hardened browsers) occasionally eat challenges too. And sophisticated scrapers with real browser automation sometimes pass the challenge — it's a filter, not a wall.

Also worth knowing: Bot Fight Mode can interfere with legitimate API traffic if your API sits behind the same zone. People discover this at the worst possible moment.

Best for: anyone on Cloudflare's free tier getting hammered by obvious junk bots. Turn it on, watch the Security events for a week, and decide whether the false-positive rate is acceptable for your audience.

4. Super Bot Fight Mode — the $25 bouncer with a guest list

What it is: the Pro-plan ($25/month at time of writing) upgrade: definite-bot blocking, static-resource protection, and — the important part — a verified-bots allowlist.

What it actually stops: everything Bot Fight Mode stops, plus it can hard-block "definitely automated" traffic instead of merely challenging it. The verified-bots list means known-good crawlers (search engines, uptime monitors) bypass the challenges instead of getting caught in them.

The catch: $25 a month is real money for a side project, and the heuristics still need watching. "Definitely automated" is a confidence score, not a fact — new legitimate tools occasionally land on the wrong side of it. And if your goal is specifically AI training control, this still isn't a scalpel: it sees automation, not intent. Pair it with robots.txt for the polite bots; don't expect it to replace that file.

Best for: sites with real traffic and real scraping costs — the ones where bot bandwidth shows up on the bill or in the analytics.

5. WAF custom rules + behavioural blocking — the actual answer, at a price

What it is: Cloudflare WAF custom rules (Business plan, $250/month) or your own stack: rate limiting per IP/ASN, blocking datacenter ASNs, fingerprinting TLS/JA3, tarpitting repeat offenders. Blocking by behaviour instead of by name.

What it actually stops: the most, by far. A scraper can rotate its user agent in one line of code. Rotating its ASN, its request cadence, and its TLS fingerprint at the same time is actual work. Behavioural rules are the only thing on this list that survives contact with a determined adversary.

The catch: the price — either $250/month or your own engineering hours building and tuning the equivalent with fail2ban, CrowdSec, or CDN edge rules. And tuning is the whole game: too aggressive and you block a university NAT with 400 students behind it; too loose and you're back to decoration. This is a part-time job, not a toggle.

Best for: businesses where scraping has a dollar cost — content sites with paywalls, APIs with usage-based billing, anyone whose logs show the same datacenter IPs chewing through pages daily.

What I'd actually do

For a personal blog or side project: robots.txt for the polite training bots (two minutes, free), plus Cloudflare's free Bot Fight Mode if junk traffic is actually bothering you. That's it. Don't pay $25/month to protect a blog nobody's scraping.

For a content site with real traffic: add Super Bot Fight Mode on Pro and actually read the Security events weekly for the first month. Keep robots.txt maintained — the polite bots are still the biggest training-data pipelines, and they're the cheapest to stop.

For anything where scraping costs money: behavioural WAF rules, and accept that this is now a maintenance task like backups. There is no set-and-forget.

And one thing from the field guide worth repeating: before you block anything, decide whether you want to block training or fetching. Blocking ChatGPT-User doesn't stop model training — it breaks "summarise this URL" for your own readers. I keep seeing people nuke their ChatGPT referral traffic and then wonder where it went. Block the training bots. Leave the user-initiated fetches alone.

Your turn

What are your logs actually showing? Is it GPTBot politely knocking, or is it some unnamed scraper chewing through your sitemap at 3am? I'm curious which of these you've tried — and which one burned you. If this saved you a 2am robots.txt edit, a reaction helps it reach the next person staring at their server logs.