Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 12 min read

How to run a free AEO audit from your terminal: AI crawler checks, then 20 buyer questions on 5 engines

A free AEO audit answers two questions: can AI crawlers read your pages, and when a buyer asks ChatGPT, Gemini or Perplexity about the problem you solve, does the answer name you, recommend you or cite you? This guide ru

How to run a free AEO audit from your terminal: AI crawler checks, then 20 buyer questions on 5 engines

A free AEO audit answers two questions: can AI crawlers read your pages, and when a buyer asks ChatGPT, Gemini or Perplexity about the problem you solve, does the answer name you, recommend you or cite you? This guide runs both halves from the terminal, step by step, with supereo's free audit tools and with plain robots.txt reading, curl and a spreadsheet.

The examples are real recorded output from two public sites, canva.com and Wikipedia, and each step says what the finding means and what to fix first. The commands come from two open repos, superagnt/supereo-audit and superagnt/supereo-mcp.

What a free AEO audit has to answer

Two questions. Can AI engines read your pages at all? And when a buyer asks an engine about the problem you solve, does the answer name you, recommend you or cite you?

A lot of free graders answer one of the two. HubSpot's AI Search Grader describes itself as "a free, one-time check that reveals what ChatGPT, Perplexity, and Gemini say about you based on their training data" (hubspot.com, read Nov 18, 2026). That's what a model remembers about you, which is useful, but it isn't what ChatGPT finds when your buyer asks and it goes and searches the web. Perplexity made the opposite point when I asked it this exact question: a technical audit that only scans your site can check crawlability, schema and structure, and still can't tell you whether AI recommends you.

So a real audit has two halves:

  1. The crawler half. robots.txt per AI crawler, what your CDN actually serves each one, and what a page looks like without JavaScript. Any page, any site, results in seconds.
  2. The answers half. Your buyers' questions on the engines they use, every answer read for whether it named you, recommended you and cited you, and which pages it cited instead.

The six steps of a free AEO audit: three crawler checks on any page, then 20 buyer prompts on 5 AI engines and a fix order
The crawler half runs on any page with supereo's free tools; the answers half is supereo's free audit of your site.

Step 1: run the crawler checks from your terminal

The crawler half runs on any page with supereo-audit, a CLI from supereo. supereo (supereo.app) is an AI agent that does everything it takes to get your brand recommended by ChatGPT, Gemini, Perplexity, Claude and Google, on your site and off it: site fixes, new pages, roundups, placements and video. It proves what worked against control prompts. Made by superagnt.

You need Node 20 or later, and git (npm installs a github: package with it):

npx -y github:superagnt/supereo-audit acme.com

The first run opens supereo in your browser to sign in. The tools are free: you get a code by email, and your first sign-in makes your account. Then the CLI calls three of supereo's free MCP tools against the page and prints one report:

  • check_ai_access reads robots.txt for each AI crawler (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot), plus Google-Extended, Googlebot and Bingbot, and returns the rule that decided each one and whether the site serves an llms.txt.
  • fetch_as_bot requests the page with each crawler's own user agent, so a CDN or firewall that turns one away shows up.
  • audit_page reads the raw HTML the way a crawler that doesn't run JavaScript does: status, title, headings, word count, structured data, newest date and canonical, with flags for a JavaScript shell, a noindex, a canonical pointing away, a crawler block or a missing date.

Three flags worth knowing. --bots all fetches the page as every crawler instead of the default four (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot). --query 'best crm for dentists' scores the page against the question it should answer. --json prints the raw tool answers, so you can pipe them into whatever you already use.

The CLI is MIT-licensed, has no dependencies and only reads robots.txt, llms.txt and the page. Each page fetch counts toward a limit of 120 an hour per person. If you'd rather wire the three tools into an MCP client yourself, supereo-audit mcp serves them on stdio:

claude mcp add --scope user supereo-audit -- npx -y github:superagnt/supereo-audit mcp

By hand: open /robots.txt, find the group that names each crawler (or User-agent: * when none does), and apply the longest matching path rule, with Allow winning a tie. Then curl -A the page once per crawler user agent and compare status codes.

Step 2: read robots.txt crawler by crawler

Here's a real check_ai_access call on canva.com, recorded on Nov 19, 2026 on supereo's MCP server (its staging environment). Trimmed: the aiCrawler flag on each row, and the edge and note fields.

{
  "origin": "https://www.canva.com",
  "path": "/",
  "crawlers": [
    { "bot": "GPTBot", "allowed": false, "rule": "Disallow: /", "group": "GPTBot" },
    { "bot": "OAI-SearchBot", "allowed": true, "rule": "no rule matches the path", "group": "*" },
    { "bot": "ChatGPT-User", "allowed": true, "rule": "no rule matches the path", "group": "*" },
    { "bot": "ClaudeBot", "allowed": false, "rule": "Disallow: /", "group": "ClaudeBot" },
    { "bot": "Claude-SearchBot", "allowed": true, "rule": "no rule matches the path", "group": "*" },
    { "bot": "Claude-User", "allowed": true, "rule": "no rule matches the path", "group": "*" },
    { "bot": "PerplexityBot", "allowed": true, "rule": "Allow: /", "group": "PerplexityBot" },
    { "bot": "Google-Extended", "allowed": true, "rule": "no rule matches the path", "group": "*" },
    { "bot": "Googlebot", "allowed": true, "rule": "no rule matches the path", "group": "*" },
    { "bot": "Bingbot", "allowed": true, "rule": "no rule matches the path", "group": "*" }
  ],
  "llmsTxt": { "served": false, "status": 403, "chars": 0, "excerpt": null }
}

Canva gives GPTBot and ClaudeBot their own groups and shuts both out. Those two collect training data. Everything that puts a page into an answer (OAI-SearchBot for ChatGPT search, ChatGPT-User when someone asks ChatGPT to open a link, Claude-SearchBot, Claude-User, PerplexityBot) falls through to * and gets in.

That's a deliberate setup, and it's the split a lot of audit checklists get wrong. OpenAI documents GPTBot and OAI-SearchBot as separate crawlers, so blocking GPTBot keeps your pages out of training and leaves ChatGPT search alone. Anthropic splits ClaudeBot from Claude-SearchBot the same way. Google-Extended only decides whether Gemini may train on and ground in your pages: Google says it has no effect on inclusion in Google Search, and Googlebot's rules are what govern AI Overviews and AI Mode.

So the finding that matters is a Disallow on OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot. A Disallow on GPTBot or ClaudeBot is a training choice you can keep. And the 403 on llms.txt isn't urgent: Google says its AI features need no special AI text file.

Step 3: fetch the page as each crawler

robots.txt is a request. Your CDN is what answers. In the same session, fetch_as_bot asked for canva.com's home page as each of OpenAI's two crawlers. Trimmed to the fields that matter here (the full results carry every response header and the first 1,000 characters of HTML):

{ "bot": "GPTBot", "status": 403, "headers": { "cf-mitigated": "challenge", "server": "cloudflare" }, "htmlChars": 755073 }
{ "bot": "OAI-SearchBot", "status": 200, "headers": { "cf-cache-status": "DYNAMIC", "server": "cloudflare" }, "htmlChars": 512027 }

Canva's firewall agrees with its robots.txt: GPTBot gets a Cloudflare challenge page, and OAI-SearchBot gets the site. On your own site, the case to look for is the two disagreeing, where robots.txt allows a search crawler and the CDN challenges it anyway. Cloudflare's setting that blocks AI training also applies to crawlers used for both search and training, and Perplexity's docs note that Bot Fight Mode can still challenge its bots even when a firewall rule allows them. Neither shows up in robots.txt.

canva.com AI crawler audit: robots.txt blocks GPTBot and ClaudeBot, allows the search crawlers; GPTBot got a 403 challenge, OAI-SearchBot a 200
Recorded with check_ai_access and fetch_as_bot on supereo's MCP server, 19 November 2026.

One limit, straight from the tool's own note: it shows "what the site served this user agent from supereo's address. A site that checks crawler IP addresses may answer the real crawler differently." Whether the real crawler gets through shows in your CDN's logs or firewall events, for requests from the crawler's published IP ranges.

Step 4: read the page the way a crawler without JavaScript does

Vercel's study of AI crawler traffic found that none of the major AI crawlers render JavaScript; Gemini, which runs on Googlebot's infrastructure, is the exception. So audit_page reads only the HTML that comes back. Recorded on Nov 19, 2026, on Wikipedia's article on generative engine optimization, scored for the question what is geo. Trimmed: the counts of headings, lists, links and figures, and the openingText cut after the eleventh language.

{
  "url": "https://en.wikipedia.org/wiki/Generative_engine_optimization",
  "status": 200,
  "signals": {
    "title": "Generative engine optimization - Wikipedia",
    "h1": "Generative engine optimization",
    "words": 2259,
    "jsShell": false,
    "questionHeadings": 0,
    "structuredData": ["Article", "Organization", "ImageObject"],
    "newestDate": { "date": "2026-11-17T20:11:58.000Z", "source": "last_modified", "ageDays": 1 },
    "noindex": false,
    "queryInTitle": 0,
    "queryInOpening": 1,
    "openingText": "23 languages Π‘ΡŠΠ»Π³Π°Ρ€ΡΠΊΠΈ বাংলা Bosanski CatalΓ  ČeΕ‘tina Deutsch Ελληνικά EspaΓ±ol Eesti فارسی FranΓ§ais ...",
    "aiCrawlersBlocked": []
  },
  "issues": []
}

No issues: 2,259 words in the HTML, no JavaScript shell, a date a day old. But look at openingText. The first thing a crawler reads on that page is the language menu, and queryInTitle is 0 because the title says generative engine optimization while the buyer typed geo. Wikipedia has enough authority to get away with it. Your pricing page doesn't. If its first 200 characters are a nav menu, a cookie banner or a hero slogan, the sentence an engine would lift as the answer is buried.

Per page, check four things: the answer in the first lines, the question's own words in the title, a visible updated date, and prices and comparisons in plain text rather than images or tabs that only render in the browser.

Step 5: ask the engines your buyers' questions

Steps 1 to 4 tell you whether AI can read you. This step tells you whether it recommends you. In Claude Code, install supereo's plugin and connect its MCP server once:

/plugin install supereo --marketplace superagnt/supereo-audit
claude mcp add --scope user --transport http supereo https://mcp.supereo.app/mcp

Run /mcp to sign in through your browser; there's no API key to copy. Then ask Claude to audit your site. The plugin's ai-visibility-audit skill calls start_audit, which runs supereo's free audit as you in about a minute, and get_audit for the results. In Codex it's codex mcp add supereo --url https://mcp.supereo.app/mcp, then codex mcp login supereo, and the configs for Cursor, VS Code, Gemini CLI, Windsurf and Zed are in the supereo-mcp repo. No agent handy? The same audit runs in your browser on supereo's AEO audit guide in about two minutes.

What the audit does: supereo works out what you sell and who buys it, writes the 20 prompts those buyers would type, and runs them on ChatGPT, Gemini, Google AI Mode, AI Overviews and Perplexity. Every answer lands as it comes in, with who each engine recommended when it wasn't you. You can edit the prompts and the competitors after.

How to read the results:

  • Score every answer three ways: named, recommended, cited. An answer can name you in a list and recommend someone else, or cite your docs while it recommends a competitor.
  • Keep the prompts that name you apart. A question like is acme any good can't show whether an engine finds you on its own, so supereo reports those beside your rate, never in it.
  • Treat it as a snapshot. The free audit runs each prompt once on each engine: single answers, not rates. SparkToro's advice for knowing an AI's recommendations is to ask at least 60 to 100 times and average. One pass shows you where to dig.
  • List the pages each engine cited. Those are the pages to beat or to get onto: roundups, review sites, Reddit threads, docs pages.

By hand, it's a spreadsheet: 20 questions in your buyers' words (best X for Y, X vs Y, alternatives to X, how to fix the problem you solve), a fresh chat in each app, one row per question and engine. Use the apps rather than the APIs. Claude's app searches when it decides it helps, and through OpenAI's API the model searches only once web search is switched on, and even then chooses whether to, so an API-only audit can miss what buyers actually see.

Step 6: what to fix first, and how to tell it worked

The order I'd fix things in:

  1. A search crawler blocked in robots.txt (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot). It's one line, and nothing else on this list matters until it's gone.
  2. A CDN or firewall that challenges a crawler your robots.txt allows.
  3. Pages behind your most valuable questions that come back as a JavaScript shell.
  4. Those same pages' openings: the answer first, the question's words in the title, a date, prices in text.
  5. Your place on the pages the engines cited in step 5: the roundups and threads that list your competitors and not you.

Then prove each fix, because answers move on their own: a model update, a competitor's launch, a slow week. supereo grades every change it makes against control prompts. A change gets 1 to 5 target prompts, read on every engine supereo tracks against the brand's other tracked prompts, and what the change added is how much its targets moved minus how much the controls moved, read as recommendations per 100 answers. It's graded from day 14 and lands by day 60, on a range where a Worked comes up by luck at most 1 in 20. Worked needs the whole range above zero and at least +10 recommendations per 100 answers, and anything still unclear at day 60 is reported as Too close to call instead of rounded up. Every number it runs on is published at supereo's How supereo measures page.

No free audit does that part, because a one-time snapshot has nothing to compare against. By hand: keep a set of questions no fix targets, rerun both sets on the same days, and compare how much each moved.

Free AEO audits, compared on what they ask

Each row comes from the tool's own page: supereo from its repos (read Nov 21, 2026), HubSpot, Ahrefs and Catalyst from their audit pages (read Nov 18, 2026).

Free audit Live answers or training data Engines Crawler and page checks Every answer shown From your terminal or agent
supereo Live: 20 buyer prompts ChatGPT, Gemini, Google AI Mode, AI Overviews, Perplexity Per crawler, plus the page without JavaScript Yes CLI, Claude Code plugin, MCP server
HubSpot AI Search Grader Training data, by its own description ChatGPT, Perplexity, Gemini Not on its page A score out of 100 with a written interpretation Web form
Ahrefs AI Visibility Checker Live: search-backed prompts ChatGPT, Gemini, Perplexity, Microsoft's assistant, AI Overviews Not on its page A limited preview, top 5 per section Web form
Catalyst free search analysis Live: twice each, as two different buyers ChatGPT, Claude, Gemini, Perplexity, AI Overviews Crawler blocks and pages AI can't parse Not on its page Web form, work email

The others fit their own jobs. HubSpot's grader is best for teams that run their marketing and CRM in HubSpot, and it's a fast read of what three models remember about you. Ahrefs' checker is best for Ahrefs customers who want prompt tracking beside their SEO data. Catalyst's analysis takes about 5 minutes and asks each question twice.

As for what an AI visibility audit costs: every audit in this table is free. What you pay for later is repetition, the same prompts on a schedule, often enough that a rate means something.

The short version

Run supereo's free AEO audit first: the CLI for the crawler half on any page, then the 20-question audit on five engines for the answers half, both free and with no card. Connect the MCP server with the command in step 5, ask your agent to run start_audit on your site, and fix whatever step 1 turned up before anything else.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.