An AI training crawler was 58% of my site's traffic. I charged it $0.001 a page and it left.
My database is on a free tier with 5 million reads a day, and it was sitting at 4.8 million. I couldn't figure out why, so I watched live traffic for 75 seconds. 52 of 89 requests (58%) came from one user agent: Elevenl
My database is on a free tier with 5 million reads a day, and it was sitting at 4.8 million. I couldn't figure out why, so I watched live traffic for 75 seconds.
52 of 89 requests (58%) came from one user agent: ElevenlabsCrawler/1.0.
ElevenLabs documents it as the crawler that collects content that may be used to train its models. It was walking all 20,000 pages of my catalog one by one. Pages were cached, but a crawler almost never asks for the same page twice, so nearly every visit missed the cache and cost database reads.
What I did
I pointed my own pay-per-crawl tool at it. AI crawlers on those pages now get HTTP 402 Payment Required with a price: $0.001 in USDC, payable over x402. Pay it and you get the page.
- People and Google see exactly the same page as before.
- My home page and docs stay free for every bot, so I still show up in AI answers.
- Crawlers are recognized by user agent, and by the IP ranges their companies publish (OpenAI, Perplexity, DuckDuckGo and Common Crawl publish theirs).
- The 402 says in plain words how to pay, including without a wallet.
What happened
| Before | After | |
|---|---|---|
| ElevenlabsCrawler share of requests | 52 of 89 (58%) | 0 of 49, 20 minutes later; still 0 of 33 an hour later |
| Database rows read, last 24h | 4.81M | 3.96M and still falling |
| Paid by the crawler | $0 |
Did it pay? No. It stopped. Most crawlers can't pay yet, so right now a price works like a block on free bulk copying, without blocking people, search engines or the AI assistants you want citing you. When agents start carrying wallets, the same response starts earning.
These are short samples, not a study. The database number is a rolling 24 hours, so it keeps dropping for a day after a change.
The honest limit
It only catches bots that say who they are, or that come from IP ranges their company publishes. Anthropic, ByteDance and Meta don't publish theirs. A scraper pretending to be Chrome from some random server still gets the page, same as with robots.txt.
Two things I learned
- Your biggest visitor may not be a person. I only found this by reading raw traffic. Most analytics tools hide bots.
- A crawler's user agent is a calling card. ElevenLabs links its bot docs from its user agent, which is how I learned who it was. If you run a bot, do the same.
Try it
- Full write-up with the numbers: https://aipaygen.com/notes/ai-crawler-paywall
- Check which AI crawlers your site allows: https://aipaygen.com/ai-crawlers
- Verify whether a "GPTBot" in your logs is real: https://aipaygen.com/verify-bot
- The WordPress plugin and Cloudflare, Nginx, Apache, Next.js and Express snippets are open source: https://github.com/Damien829/pay-per-crawl
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.