How to charge AI crawlers per page with HTTP 402 (and keep your site free for people)

If you run a website, a growing share of your traffic isn't people. It's AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider and a long tail of others, fetching pages to train models or answer questions.

Until now you had two choices:

1. Let them in for free. You pay the bandwidth; they get the content.
2. Block them in robots.txt. You save the bandwidth, and you disappear from AI answers.

This post shows a third option: answer them with 402 Payment Required and a price, so a crawler that wants the page can pay a fraction of a cent for it, straight to your wallet. People and search engines see no difference.

The status code that waited 28 years

HTTP has had 402 Payment Required since 1997, marked "reserved for future use". The web never used it because charging a cent was impossible: a card payment costs around 30 cents plus a percentage.

x402 finally gives 402 a working flow:

1. Ask. The client sends a normal request.
2. Price. The server answers 402 with a payment-required header: amount, token (USDC), network (Base), the wallet to pay.
3. Pay. The client signs a payment authorization for exactly that amount and retries the same request with it attached.
4. Deliver. A facilitator verifies the signature and settles it on-chain in about a second; the server returns the page.

No accounts, no API keys, no subscriptions. That's exactly what an automated client needs.

You can see a real 402 quote from your browser on this demo page.

The idea: route AI crawlers to a payment gateway, on the same URL

The site itself shouldn't have to implement payments. So the setup is:

• Requests from known AI crawlers (or anything already carrying an x402 payment) are passed to a gateway.
• Unpaid, the gateway answers with your price.
• Paid, the gateway verifies the payment, fetches the page from your site with a secret header, settles the payment to your wallet, and returns the page.
• Everyone else is served by your site as usual.

The money goes crawler -> your wallet. If your site errors, nobody is charged.

The code: a Cloudflare Worker

If your site is on Cloudflare, put this Worker on your site's route:

// Pay-per-crawl. Set the secret: npx wrangler secret put AIPAYGEN_SECRET
const GATEWAY = "https://aipaygen.com/g/<your-route-id>";
const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider|Amazonbot|meta-externalagent/i;

export default {
  async fetch(req, env) {
    // The gateway fetching a page it was paid for: serve it.
    if (env.AIPAYGEN_SECRET && req.headers.get("x-aipaygen-gateway-secret") === env.AIPAYGEN_SECRET) return fetch(req);
    const paying = req.headers.get("payment-signature") || req.headers.get("x-payment");
    if (!AI_BOTS.test(req.headers.get("user-agent") || "") && !paying) return fetch(req); // people browse free
    const u = new URL(req.url);
    return fetch(GATEWAY + u.pathname + u.search, { method: req.method, headers: req.headers, body: req.body });
  },
};

(The full list of crawlers is longer; the page linked below has it.)

Three details worth calling out:

• The secret header closes the loop. Without it, a crawler could hit your origin directly with a browser user agent. With it, your origin only serves gateway traffic that has been paid for.
• Search engines are not on the list. Googlebot and Bingbot aren't AI-training crawlers, so your rankings don't change.
• Same URL, both ways. A crawler that understands x402 just retries the URL it asked for with a payment. Nothing special to discover.

The code: Express

Not on Cloudflare? The same thing as middleware (Node 18+):

const GATEWAY = "https://aipaygen.com/g/<your-route-id>";
const AI_BOTS = /GPTBot|ClaudeBot|PerplexityBot|CCBot|Bytespider/i;

app.use(async (req, res, next) => {
  if (process.env.AIPAYGEN_SECRET && req.get("x-aipaygen-gateway-secret") === process.env.AIPAYGEN_SECRET) return next();
  const paying = req.get("payment-signature") || req.get("x-payment");
  if (!AI_BOTS.test(req.get("user-agent") || "") && !paying) return next();
  const headers = { ...req.headers }; delete headers.host;
  const r = await fetch(GATEWAY + req.originalUrl, { method: "GET", headers });
  r.headers.forEach((v, k) => { if (!["content-encoding", "content-length", "transfer-encoding"].includes(k)) res.set(k, v); });
  res.status(r.status).send(Buffer.from(await r.arrayBuffer()));
});

Getting a route id

I built this as a free gateway, AIPayGen. On the pay-per-crawl page you enter your site, a price per page and your wallet address on Base, and it gives you the route plus the Worker code with your id filled in. No fees from us; the payment goes directly to you.

Then check it:

curl -i -A GPTBot https://your-site.com/
# HTTP/2 402
# payment-required: eyJ4NDAyVmVyc2lvbiI6Mi...

Honest limits

• User agents can be spoofed. A crawler that pretends to be Chrome gets the free page, same as with robots.txt. This targets the crawlers that identify themselves, which is most of the big ones.
• Most crawlers can't pay yet. Today this mostly works like a block with a price tag. But agent frameworks are adding x402 support quickly, and this gives them a way to say yes instead of being shut out.
• You need a wallet on Base to receive USDC.

If you try it, I'd love to hear which crawlers hit your site the most and what price per page feels fair.