Your traffic spike might be one AI newsletter bot: find it, verify crawlers, rate-limit the rest
Sammy Straus posted a side project to Hacker News, got about a hundred upvotes, and watched the site get slower for days. The culprit was not GPTBot or ClaudeBot. Nearly half of all traffic came from one client, a servic
Sammy Straus posted a side project to Hacker News, got about a hundred upvotes, and watched the site get slower for days. The culprit was not GPTBot or ClaudeBot. Nearly half of all traffic came from one client, a service that turns website changes into an AI newsletter, hitting the site many times a minute.
I expect a lot of small sites to hit the same thing, and his write-up is on the Hacker News front page today.
The fix they landed on is the right one: find out who is actually calling you, block the abusers, and rate limit the rest. Here are three tested pieces: a log analyzer that also catches fake "GPTBot" traffic, a robots.txt that slows crawlers without hiding you from AI search, and a Next.js middleware that returns 429s to anyone over budget.
TL;DR
- The biggest offender often does not call itself a bot. Rank clients by requests per minute, not by user agent.
- Check claimed crawlers against their published IP ranges. Anyone can send
GPTBotin a header. - Blocking AI search bots can make you invisible in ChatGPT, Claude and Perplexity answers. Rate limit training crawlers instead of blocking everything.
- An in-memory rate limiter in middleware is a good first line. It is per instance, so put a WAF rule in front when traffic is real.
Step 1: find out who is really calling
This script reads a standard nginx or Apache access log, sorts every request into a category, and verifies the big crawlers against the IP lists OpenAI, Anthropic and Google publish. It uses only the Python standard library.
#!/usr/bin/env python3
"""Who is really hitting your site? Classify an access log by bot type,
check that "GPTBot", "ClaudeBot" and "Googlebot" come from their published IP
ranges, and flag single clients that hammer you.
Usage: python3 botlog.py access.log (nginx/Apache "combined" format)
"""
import collections, ipaddress, json, re, sys, urllib.request
LINE = re.compile(r'(?P<ip>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+)[^"]*" '
r'(?P<status>\d{3}) (?P<bytes>\d+|-) "[^"]*" "(?P<ua>[^"]*)"')
# user-agent token -> (category, published IP list or None)
BOTS = {
"GPTBot": ("AI training", "https://openai.com/gptbot.json"),
"ClaudeBot": ("AI training", "https://claude.com/crawling/bots.json"),
"CCBot": ("AI training", None),
"Bytespider": ("AI training", None),
"OAI-SearchBot": ("AI search", "https://openai.com/searchbot.json"),
"Claude-SearchBot": ("AI search", "https://claude.com/crawling/bots.json"),
"PerplexityBot": ("AI search", None),
"ChatGPT-User": ("AI user fetch", "https://openai.com/chatgpt-user.json"),
"Claude-User": ("AI user fetch", "https://claude.com/crawling/bots.json"),
"Googlebot": ("Search engine", "https://developers.google.com/static/search/apis/ipranges/googlebot.json"),
"bingbot": ("Search engine", None),
}
SCANNER_PATHS = re.compile(r"wp-(admin|login|content|includes)|phpinfo|\.env|\.git/|xmlrpc|/cgi-bin/", re.I)
_ranges = {}
def ranges(url):
if url not in _ranges:
try:
with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "botlog"}), timeout=15) as r:
data = json.load(r)
_ranges[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
for p in data["prefixes"]]
except Exception as e:
print(f"warning: could not load {url}: {e}", file=sys.stderr)
_ranges[url] = None
return _ranges[url]
def classify(ip, ua, path):
if SCANNER_PATHS.search(path):
return "Scanner", None
for token, (cat, url) in BOTS.items():
if token.lower() in ua.lower():
if url and (nets := ranges(url)) is not None:
addr = ipaddress.ip_address(ip)
if not any(addr in n for n in nets):
return "Spoofed bot", token
return cat, token
if re.search(r"bot|crawler|spider|scrapy|python-requests|curl|go-http|headless", ua, re.I):
return "Other bot", None
return "Human or unknown", None
reqs, size = collections.Counter(), collections.Counter()
per_client, per_minute, spoofed = collections.Counter(), collections.Counter(), collections.Counter()
total = 0
for raw in open(sys.argv[1], errors="replace"):
m = LINE.match(raw)
if not m:
continue
total += 1
cat, token = classify(m["ip"], m["ua"], m["path"])
b = 0 if m["bytes"] == "-" else int(m["bytes"])
reqs[cat] += 1
size[cat] += b
client = f'{m["ip"]} {m["ua"][:60]}'
per_client[client] += 1
per_minute[(client, m["ts"][:17])] += 1 # dd/Mon/yyyy:HH:MM
if cat == "Spoofed bot":
spoofed[(token, m["ip"])] += 1
print(f"{total} requests\n")
print(f"{'category':18} {'requests':>9} {'share':>7} {'MB':>8}")
for cat, n in reqs.most_common():
print(f"{cat:18} {n:9} {n/total:7.1%} {size[cat]/1e6:8.1f}")
print("\nTop clients")
for client, n in per_client.most_common(5):
peak = max(v for (c, _), v in per_minute.items() if c == client)
print(f"{n:7} req peak {peak:4}/min {client}")
if spoofed:
print("\nClaims to be a known bot but is outside its published IP ranges")
for (token, ip), n in spoofed.most_common(10):
print(f"{n:7} req {token:12} {ip}")
I generated a test log of 6,330 requests that mirrors the pattern in the post: real GPTBot, ClaudeBot and Googlebot traffic from addresses inside their published ranges, a fake GPTBot from an address outside them, a vulnerability scanner probing /wp-content/phpinfo.php, a change-monitoring service polling an API endpoint, and normal browsers. The output:
6330 requests
category requests share MB
Human or unknown 4000 63.2% 59.7
AI training 1600 25.3% 51.5
Spoofed bot 300 4.7% 9.8
Search engine 250 3.9% 8.1
Scanner 180 2.8% 0.0
Top clients
2600 req peak 13/min 3.91.20.77 SiteWatch/2.3 (+https://example-monitor.io; AI change digest
900 req peak 6/min 20.171.206.15 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatibl
700 req peak 6/min 216.73.216.40 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatibl
300 req peak 3/min 45.83.12.9 Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gpt
250 req peak 6/min 66.249.66.1 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) Ap
Claims to be a known bot but is outside its published IP ranges
300 req GPTBot 45.83.12.9
Two things to notice. The heaviest client, with 2,600 requests, lands in "Human or unknown" because its user agent never says "bot". Only the top clients list exposes it. And 300 requests claim to be GPTBot but come from an address that is not in OpenAI's published list, so they are not OpenAI at all.
Step 2: tell crawlers what you want in robots.txt
Each AI company runs separate bots for training, for search and for fetching a page when a user asks. They are independent, and blocking the wrong one costs you visibility:
- OpenAI: GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User fetches pages on a user's behalf. OpenAI says robots.txt may not apply to ChatGPT-User, because a person started the request.
-
Anthropic: ClaudeBot collects training data, Claude-SearchBot indexes for search, and Claude-User fetches for users. Anthropic says its bots honor robots.txt and support the non-standard
Crawl-delaydirective.
Here is the version that keeps search bots and keeps crawlers out of your API:
User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /api/
User-agent: GPTBot
Disallow: /api/
User-agent: *
Disallow: /api/
A crawler obeys only the most specific group that names it, so ClaudeBot and GPTBot ignore the * group. That is why each named group repeats Disallow: /api/. Crawl-delay is a hint, not a guarantee: Google does not support it, and OpenAI does not document it, so GPTBot is slowed by the limiter in step 3. Anything that ignores robots.txt needs the next step.
Step 3: rate limit in Next.js middleware
A token bucket per client, with different budgets for search engines, AI search, AI training crawlers and unknown bots:
// lib/bot-limit.ts: a token bucket per client, tuned per kind of visitor.
export type Rule = { perMinute: number; burst: number };
export const RULES: Record<string, Rule> = {
search: { perMinute: 120, burst: 60 }, // Googlebot, bingbot: never starve these
aiSearch: { perMinute: 60, burst: 30 }, // OAI-SearchBot, Claude-SearchBot, PerplexityBot
aiTraining: { perMinute: 20, burst: 10 },// GPTBot, ClaudeBot, CCBot, Bytespider
unknownBot: { perMinute: 10, burst: 5 }, // anything else that says bot, curl, python...
default: { perMinute: 120, burst: 60 }, // browsers
};
export function kindOf(ua: string): keyof typeof RULES {
const u = ua.toLowerCase();
if (/googlebot|bingbot/.test(u)) return "search";
if (/oai-searchbot|claude-searchbot|perplexitybot/.test(u)) return "aiSearch";
if (/gptbot|claudebot|ccbot|bytespider/.test(u)) return "aiTraining";
if (/bot|crawler|spider|scrapy|python-requests|curl|go-http|headless/.test(u)) return "unknownBot";
return "default";
}
const buckets = new Map<string, { tokens: number; at: number }>();
/** Returns 0 if allowed, or the seconds to wait before retrying. */
export function check(key: string, rule: Rule, now = Date.now()): number {
const b = buckets.get(key) ?? { tokens: rule.burst, at: now };
const refill = ((now - b.at) / 60_000) * rule.perMinute;
b.tokens = Math.min(rule.burst, b.tokens + refill);
b.at = now;
if (b.tokens >= 1) {
b.tokens -= 1;
buckets.set(key, b);
return 0;
}
buckets.set(key, b);
return Math.ceil(((1 - b.tokens) / rule.perMinute) * 60);
}
And the middleware that uses it, which also drops scanner probes before they reach your app:
// middleware.ts (Next.js): rate limit by visitor kind, drop scanner probes early.
import { NextResponse, type NextRequest } from "next/server";
import { RULES, check, kindOf } from "@/lib/bot-limit";
const SCANNER = /wp-(admin|login|content|includes)|phpinfo|\.env|\.git\/|xmlrpc|\/cgi-bin\//i;
export function middleware(req: NextRequest) {
const { pathname } = req.nextUrl;
if (SCANNER.test(pathname)) return new NextResponse(null, { status: 404 });
const ua = req.headers.get("user-agent") ?? "";
const ip = req.headers.get("x-forwarded-for")?.split(",")[0].trim() ?? "unknown";
const kind = kindOf(ua);
const wait = check(`${kind}:${ip}`, RULES[kind]);
if (wait > 0) {
return new NextResponse("Too many requests", {
status: 429,
headers: { "Retry-After": String(wait) },
});
}
return NextResponse.next();
}
// Skip static files so images and JS never count against the limit.
export const config = { matcher: ["/((?!_next/static|_next/image|favicon.ico|.*\\.(?:png|jpg|webp|svg|css|js)$).*)"] };
I built this into a fresh Next.js 15 app, ran next start, and sent 14 requests in a row as each visitor:
# Scanner probe
curl -o /dev/null -w "%{http_code}" localhost:3000/wp-content/phpinfo.php
# 404
# 14 requests as GPTBot
200 200 200 200 200 200 200 200 200 200 429 429 429 429
# HTTP/1.1 429 Too Many Requests
# retry-after: 3
# 14 requests as Googlebot
200 200 200 200 200 200 200 200 200 200 200 200 200 200
GPTBot gets its burst of 10, then a 429 with a Retry-After header that well-behaved crawlers honor. Googlebot never notices the limiter.
What this does not do
- The buckets live in memory, per server instance. On serverless or several replicas, each instance keeps its own count. For real enforcement, add a rate-limiting rule at your CDN or WAF, or move the counter to Redis.
-
x-forwarded-forcan be forged if your app is reachable without a proxy in front. Read the client IP from the header your platform guarantees. -
Missing IPs share one bucket. Without a proxy setting
x-forwarded-for, every visitor falls into the "unknown" bucket. In production, skip limiting when no IP is found. The Map is also never pruned, so add a periodic cleanup on long-running servers. -
On Next.js 16 the file is
proxy.tsand the export is namedproxy. - The user-agent check is cheap, not secure. Combine it with the IP verification from step 1 before you trust anything that claims to be a search engine.
The decision I would make for most sites
Keep search and AI search bots allowed, because being cited in AI answers is now a real source of visitors. Slow the training crawlers. Find the anonymous heavy hitters in your logs, block them by IP or behavior, and put every unknown bot on a small budget. Run the log script weekly, because the worst client next month will not be the one you blocked this month.
Axionry's own site allows AI crawlers on purpose, so this is the toolkit I keep ready for the day one of them, or something pretending to be one, gets greedy. Logging and rate limits like these belong in the SaaS MVP tech stack from day one. If bandwidth from bots is showing up on your bill, it is also one of the first things I check in a cloud cost review.
Sources: I posted to HN, and then my website was flooded with AI bot traffic, Hacker News discussion, OpenAI crawlers and IP ranges, Anthropic on ClaudeBot, Claude-SearchBot and Claude-User, Googlebot IP ranges. Scripts tested on 14 October 2026 against a generated log and a Next.js 15.5 production build.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.