Do the Top 1,000 Sites Actually Use robots.txt, Open Graph and llms.txt?

Every SEO tool and checklist assumes a stack of files matters — robots.txt, canonical tags, Open Graph, structured data, and now llms.txt. But assumption isn't evidence. So we scanned the Tranco top 1,000 sites and just looked at what they actually ship.

First surprise: of the 1,000, only 554 served a browsable HTML homepage at all. The rest are CDN endpoints, DNS infrastructure and API hosts with nothing to crawl. Every percentage below is over those 554 real homepages, not the raw list.

The basics are table stakes

Among sites that have a homepage, the fundamentals are close to universal:

• meta description — 82%
• robots.txt — 82%
• Open Graph — 73%
• canonical — 69%

If your site is missing any of these, you're behind the top of the web. They're the floor, not an edge — which is worth knowing before you spend a sprint "optimising" them.

Then it falls off a cliff

• Structured data (JSON-LD) — 41%. Fewer than half of the biggest sites ship any.
• FAQPage schema — just 2%.

That 2% is the headline. Rich results and FAQ schema get endless attention in SEO content, and almost none of the largest sites actually implement FAQPage. Whatever advantage it confers is wide open, because the field is nearly empty.

The 2026 shift is on the AI side

The genuinely new movement isn't in classic SEO — it's in how sites treat AI crawlers:

• 27% now name at least one AI crawler in robots.txt to block it — GPTBot on 20%, ClaudeBot 19%, Google-Extended 17%. Blocking the AI bots has quietly become a mainstream default at the top of the web.
• 16% already publish an llms.txt — the proposed "here's how to read my site" file for LLMs. Adoption is concentrated in developer-tool and infrastructure companies: Cloudflare, GitHub, Stripe, Shopify. The audience that would build against an LLM is the audience shipping the file for one.

The honest caveat

The Tranco top 1,000 skews heavily technical — SaaS, infra, developer platforms — so these numbers run ahead of the general web, not level with it. Read them as "what the most sophisticated sites do," not "what the average site does." The llms.txt figure especially is a leading indicator, not a mainstream one.

Reproduce it

The scan is a public domain list plus a short fetch-and-parse script — no private data, no paywalled crawler. Point it at any list and you'll get comparable numbers. And to check your own site's files, the SEO tools cover each signal directly: a robots.txt tester, an Open Graph and schema generator, an llms.txt generator and an AI-crawler control builder — all in the browser.

Originally published on LK Forge.