I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.

Everyone "knows" that the web is locking out AI crawlers. Cloudflare flips the default to block, robots.txt blocklists go viral, every SEO newsletter screams about GPTBot.

So I ran the actual test. I pulled robots.txt, llms.txt, sitemap.xml, and the homepage JSON-LD from 30 well-known sites — and the results were the opposite of the narrative.

Only 1 site out of 30 blocks GPTBot. Zero block ClaudeBot or PerplexityBot.

And the most uncomfortable finding: the sites AI engines cite the most — Wikipedia, GitHub, MDN, Stack Overflow — are the worst at "AI optimization." If technical setup decided citations, that shouldn't be possible.

Here's the full data, the method, and what it actually means if your site isn't Wikipedia.

The method (reproducible in 10 minutes)

Two groups of 15, chosen for a reason:

• Group A — heavy AI-citation sites: Wikipedia, GitHub, Stack Overflow, MDN, Python.org, React, Next.js, Tailwind, Kubernetes, Stripe, AWS docs, Microsoft Learn, Ruby, Go, Rust
• Group B — commercial SaaS sites: Asana, ClickUp, Notion, Linear, Atlassian, Salesforce, Zoho, Wix, Typeform, Calendly, Loom, Miro, Basecamp, Zendesk, GitLab

For each site I fetched with a plain browser user agent (no special treatment), and recorded:

1. Does robots.txt explicitly block GPTBot / ClaudeBot / PerplexityBot / CCBot?
2. Does /llms.txt return 200?
3. Does /sitemap.xml return a valid urlset?
4. Does the homepage ship JSON-LD ("@type")?

Nothing exotic. You can re-run this yourself; the script is a couple of curls per domain.

Finding 1: the AI-crawler blockade is mostly a myth at the top

One. Loom was the only site in either group with an explicit GPTBot block. Everyone else — including every major media-adjacent property you'd expect to be defensive — allows all four crawlers by default.

The "web is closing itself to AI" story is real at the margins (some news sites, some forums), but among the sites that actually matter for citations, the door is wide open.

Which leads directly to the uncomfortable part.

Finding 2: the most-cited sites are the least "optimized"

Cross the two groups and the pattern flips everything GEO content tells you:

The SaaS group has spent the last two years buying "AI visibility" — and it shows in their plumbing. The sites LLMs actually cite? Wikipedia and Stack Overflow don't ship llms.txt at all. Most dev-doc sites have no JSON-LD on the homepage.

If llms.txt and schema markup drove citations, Wikipedia would be invisible. It is the single most-cited domain in AI answers.

Finding 3: what actually separates cited from ignored

Put the three findings together and one model explains all of it:

Citations are decided by third-party trust, not on-site configuration.

ChatGPT cites Wikipedia because thousands of independent sources corroborate it. It cites GitHub and Stack Overflow because developers quote them constantly across the open web. The model doesn't need your schema.org markup to know what these sites are — the rest of the web has already told it, thousands of times.

The SaaS group's pristine llms.txt files are the entry ticket, not the win condition. They make a site parseable; they don't make it trustworthy. Nothing on your own domain can substitute for other people talking about you.

I saw the same thing from the other direction in an earlier experiment — logging every URL five models cited across ~1,400 citation events for 200 buyer prompts. The pattern was infrastructure, not prose: extractable pages, consistent entities, crawlability. But all of it only mattered after the brand existed in third-party sources the models already trusted.

So what should a normal site actually do?

If you're not Wikipedia, your leverage looks like this, in order:

1. Get your robots.txt out of the way — it's the cheapest check you'll ever pass, and being the one site that blocks GPTBot is the one way to guarantee zero AI traffic.
2. Get into third-party sources the models already trust — this is where citations are actually decided. Dev.to, Hashnode, community answers, comparison pages, directory listings. One well-placed answer in a trusted community beats a month of on-site tweaks.
3. Make pages extractable — answers in one piece, consistent brand entity, clean chunks. This is what converts a mention into a citation.
4. llms.txt and JSON-LD — do them, they cost an afternoon, but understand what they are: plumbing, not strategy.

The uncomfortable summary: on-site "GEO optimization" is table stakes that the biggest winners barely bother with, because their real asset lives on other people's domains.

Data collected 2026-11-11 with plain browser user agents, all requests reproducible with curl. Two of the 30 sites (Wikipedia, GitHub) return errors to some automated fetches for robots.txt; their crawler permissions are documented publicly and are permissive — I've excluded them from the llms.txt percentage rather than guess.

If you want to run this audit against any domain — crawlability, entity consistency, extractability, share of voice against competitors — that's what I built Bidvix for. The 5-minute AI visibility audit checklist covers the same four checks I used above.