What 9,483 small business websites actually serve to an AI crawler

Over two weeks I pointed a crawler at 11,060 small business websites with JavaScript off, a self-identifying user agent and three pages per site, and stored what came back. This is what the schema, the robots files and the served HTML look like across 9,483 sites that responded, and why an assistant reading them has so little to work with.

I did this because gaash.ai sells one thing, measuring whether AI assistants
recommend a business, and I wanted to know how much of the problem is the
assistants and how much is the websites. The answer is mostly the websites.

Two thirds have nothing an assistant can quote. 61.5% of the sites have no
question and answer content of any kind: no FAQ section, no FAQPage schema,
nothing shaped like the thing a buyer asks. Assistants lift answers from pages
that already contain answers. A homepage that says "Quality service since 1998"
gives them nothing to lift. Oddly, about half the sites (50.7%) do have at least
one heading phrased as a question. They just don't answer it in a way a machine
can isolate.

Reviews exist, but not where a machine looks. 64.7% of sites show no review
signal a crawler can read: no rating markup and no review quoted in the page
text. Only 8.9% carry rating schema. Most of these businesses have dozens of
Google reviews. The assistant reading their site does not know that, because
the site does not say so.

The business can't be pinned down. 43.1% are inconsistent about who and
where they are. 36.9% publish more than one phone number across their own three
main pages. 34.1% have no postal address a machine can parse anywhere. An
assistant deciding whether to name "Smith Plumbing" wants one name, one number
and one address, repeated. It often gets three of each.

Structured data is mostly present and mostly useless. Only 17.8% have no
schema.org markup at all, which surprised me. But look at what the other 82%
have: WebSite, WebPage, BreadcrumbList, SearchAction, ImageObject. The page
builder put it there. It describes the website, not the business. LocalBusiness
appears on roughly a quarter of sites. A further 13.7% have business schema
that is missing the fields that would make it usable.

The no-schema rate is not evenly spread. Real estate agents (33.6%),
photographers (32.7%), tax advisors (31.0%) and wedding planners (30.1%) are
the worst. Carpenters (15.8%) and moving companies (16.1%) are the best of the
trades with a hundred or more sites in the sample. Markets barely differ: the
EU (20.5%) and UAE (20.0%) are a little worse than the US (17.5%) and Canada
(17.2%).

Hardly anyone blocks AI on purpose. 3.2% disallow AI crawlers in
robots.txt. 7.2% sit behind bot protection that served my crawler a challenge
page instead of the site, which blocks assistants just as well without anyone
deciding to. 3.0% render their content only in JavaScript, so a crawler that
does not execute scripts sees an empty page. 2.1% keep their real content (the
price list, the service menu) in a PDF.

The pages are not thin. Median homepage: 880 words. Only 3.9% are under
150 words. The problem is not that these sites say too little. It is that what
they say is not in a form a machine can attribute, quote or verify.

llms.txt is nowhere. 63.6% have no /llms.txt. I include this last because I
am not sure it matters yet; it is a proposed convention, not a standard the
assistants have committed to. But it costs an hour and it is a clean signal of
whether anyone has thought about machine readers at all.

Do assistants actually name these businesses?

I also asked. For 9,240 of the businesses I put one buyer question from their
market and trade to Claude with web retrieval on ("Who are the best
electricians in Brampton?" and the like) and checked whether the business was
named. 14.9% were. Among the 5,928 questions where the assistant answered with
a clearly differentiated list rather than a hedge, 17.7%.

Treat that as a directional number, not a rate. It is one question per
business, one assistant, one day. The full question panel I use for paying
clients is ten to twelve questions; this was a tenth of that, run at scale. A
proper study would run the panel. What I can say is that the businesses that
were named were, on inspection, the ones whose sites had the answer-shaped
content and the consistent identity described above, and that is the whole
reason I keep measuring this.

One more thing from those 9,240 answers that I did not expect. Of every name
the assistant gave, 0.6% were directories or aggregators (Yelp, Angi and the
rest). 99.4% were actual businesses. Google's first page for these questions is
mostly aggregators. The assistant's answer is mostly not. If you are a small
business that has spent ten years failing to outrank Yelp, that is a
different game, and it is being decided by what your own site serves.

What to do with this

If you run one of these sites, the order that follows from the data is: put
your name, address and phone in one form and repeat it exactly on every page;
mark up the business, not the website, with LocalBusiness or the specific type
and fill in the address, phone, opening hours and area served; put your
reviews on the page as text and as AggregateRating; write the five questions
your customers ask on the phone, as headings, with real answers under them.
Then check what a crawler actually receives, because 7% of you are behind a
bot wall you did not know was there.

If you want to check what your own site serves, the free visibility check at
gaash.ai does exactly the fetch described here and sends back the result within
48 hours.

Method

Crawler user agent gaash-audit/1.0 (+https://gaash.ai/methodology), no
JavaScript execution, following redirects, fetching /, the contact page and
the about page where linked from the homepage, plus /robots.txt and
/llms.txt. Schema detection covers JSON-LD and microdata. "NAP inconsistent"
means more than one distinct phone number across the fetched pages or no
parseable postal address. The probe used Claude (Opus) with retrieval enabled,
one question per business, drawn from a panel built per metro and trade. Sample
drawn from businesses discovered per metro and trade, not from a random frame;
percentages describe this sample. Full methodology at gaash.ai/methodology.

Originally published at gaash.ai.