I measured 58 stores registered in October. Two block AI crawlers.
Every piece of advice for a new store says the same first thing: check that AI crawlers are allowed in. So I took 58 online stores whose domains were registered in October — brand new, one to six weeks old — and checked
Every piece of advice for a new store says the same first thing: check that AI crawlers are allowed in. So I took 58 online stores whose domains were registered in October — brand new, one to six weeks old — and checked that first, as GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot.
Two of the 58 were blocked. Fifty-six were not.
So access was almost never the problem, which matches everything else I have measured this month. The interesting part is where the loss actually sits, and one number in this sample changed how I think about the problem.
Where each store loses the quote
Each site is counted once, by the weakest of four groups: access (can a crawler fetch it), discovery (does anything tell a crawler the other pages exist), facts (is the price and detail present as text), structure (does anything say which paragraph answers which question).
| Where it fails | Share | Count |
|---|---|---|
| Structure | 60% | 35 |
| Facts | 22% | 13 |
| Discovery | 14% | 8 |
| Access | 3% | 2 |
The median store passed 11 of the 15 checks. The worst passed 1.
The number I did not expect
Thirty-three of the 58 already serve an llms.txt.
These are new stores. More than half had already added the file that every guide published in the last year tells you to add, and fifty of the 58 had a working sitemap. Two had FAQ markup.
That is the whole finding in one line: the easy half is being adopted and the hard half is not.
A file is an afternoon and, increasingly, a plugin or a theme toggle. What is missing is somebody writing the sentences: who this is for, what it sells, what it costs, stated once, in plain text, and marked up so a machine knows that this paragraph answers that question.
A plugin cannot write those sentences, because they are the part of the site that only the owner knows.
What to do about it, concretely
Three things, in this order, and none of them is a redesign.
- Put three sentences near the top, as text. What it is, who it is for, what it costs. Not in an image, not behind a tab, not loaded by a script.
- Shape the headings as the questions a buyer asks. "What does delivery cost to Ireland?" beats "Shipping" for a machine trying to match a question, and costs nothing but a rewrite.
- Mark up the questions the page already answers. If the page has an FAQ, wrap it. If it has no FAQ, write three questions. The markup is five minutes; the writing is the afternoon.
Then fetch your own homepage with curl and count the words in the response. If it comes back at 300 words or fewer, the page is a shell to anything that does not run JavaScript, and no amount of markup fixes that.
curl -sL https://yourstore.com | sed 's/<[^>]*>/ /g' | wc -w
Caveats, because the number is a snapshot
Newly registered domains from a public daily list, English-language storefronts only, one run each on one afternoon from one host. Not a sample of e-commerce; a sample of shops that are two to six weeks old, which is a different and more interesting population: these are the sites whose owners are still changing things.
Two measurement notes that matter for anyone repeating this. First, six of the original 58 turned out to have addresses on domains with no MX record at all, or addresses on a different domain from the site, so they never left my queue — 58 measured, 42 genuinely reachable. Second, a crawler that receives a 200 to a borrowed user agent proves only the absence of user-agent blocking; it does not prove your real crawler gets in from OpenAI's IP range.
The check itself is free, runs fifteen machine-decidable measurements from outside in about ten seconds, and needs no account: citeai.tech. If you want the version that runs on your own machine, on your own model key, it is an agent skill: agensi.io/skills/ai-readability-standard.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.