I built a Next.js testbed to see what AI crawlers can actually read

AI assistants are becoming a front door to the web. People ask ChatGPT, Claude, Perplexity, Gemini or Grok a question, and the answer is built from whatever the assistant's fetcher managed to read on your page. That raised a question I couldn't find a solid answer to: does the way a page is rendered decide whether an AI assistant ever sees its content?

Googlebot runs JavaScript. Many of the newer AI fetchers seem to read only the HTML the server sends. If that's true, a client rendered page might as well be empty to them, and even server rendering isn't the end of the story once streaming and partial prerendering come in. I wanted to measure this rather than guess, so I built a testbed.

What the testbed is

It's a Next.js 16 site that serves the same product pages through six rendering strategies:

Every page has the same structure: a product description, specs, price and stock, reviews, an FAQ and so on. Each piece arrives at a different time depending on the strategy, which makes it possible to see exactly which parts a given fetcher picked up.

All products, brands, prices and reviews are fictional. That's deliberate. If an AI assistant ever repeats a fact from the site, it can only have come from the testbed.

How it's measured

A small proxy in front of the app logs every request: time, path, user agent and IP. The IP is used only to check that a request claiming to be from a known crawler really comes from that company's published address ranges. A tiny inline script reports whether the visitor ran JavaScript, so I can tell a plain HTML fetch apart from a real render. There are no cookies, no accounts and no third party trackers, and everything is reported in aggregate.

There are two parts to the study:

1. Lab runs on my own machine, fetching every page under each strategy with and without JavaScript and at different timeouts. These give exact numbers for what a fetcher would see.
2. A 45 day field study on the live site, watching which real crawlers show up, what they fetch, and how long they wait. It runs until mid December, so I'm keeping the field results to myself until it ends.

Try it

The site is live at geo-classic.vercel.app. You can browse every strategy from /browse, and /about explains what's logged.

I'd like your input

• Is there anything about the setup that makes the comparison unfair? Things a typical production Next.js site does that I've left out?
• If you've looked at AI crawler traffic on your own sites, what have you seen, especially around client rendered or streamed content?

Leave a comment here, I'll write up the results once the study closes.