Dev.to WebDev πŸ›  Dev πŸ‘ 0 πŸ“– 14 min read

AEO over MCP, step by step: check AI crawlers, read AI answers and test fixes from Claude Code

An MCP server for AEO should let your agent do three jobs: see your pages as each AI crawler gets them, read what the engines answer and cite, and tell a fix that worked from noise. This tutorial runs each job by hand wi

AEO over MCP, step by step: check AI crawlers, read AI answers and test fixes from Claude Code

An MCP server for AEO should let your agent do three jobs: see your pages as each AI crawler gets them, read what the engines answer and cite, and tell a fix that worked from noise. This tutorial runs each job by hand with curl and a little statistics, then from Claude Code over supereo's MCP server, with a recorded session and its real output.

AEO (answer engine optimization) is getting your brand named in the answers ChatGPT, Gemini, Perplexity, Claude and Google's AI features give. An engine can only cite a page its crawler could read, so the work starts at robots.txt.

Step 1: Check which AI crawlers robots.txt lets in

Each AI company sends more than one crawler, and each reads robots.txt under its own token. The ones worth checking: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI; ClaudeBot, Claude-SearchBot and Claude-User for Anthropic; PerplexityBot; and Google-Extended, Google's token for Gemini. Googlebot and Bingbot still matter, since the search indexes they build feed AI answers too.

The rule people get wrong is group matching. A crawler obeys the group whose User-agent line matches its token and ignores every other group, * included. Only when no group names it does it fall back to *. Take this file:

User-agent: GPTBot
Disallow: /pricing

User-agent: *
Disallow: /admin

GPTBot can't read /pricing but can read /admin, because the * rules don't stack on top of its own group. ClaudeBot has no group, so it gets *: /pricing is open to it and /admin isn't. Inside a group the longest matching path wins, and when an Allow and a Disallow match equally, Allow wins (RFC 9309, the robots.txt standard). Pull the file and read it once per bot:

curl -s https://example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txt

The second line tells you whether the site serves an llms.txt, the plain-text summary some sites publish for language models.

Step 2: Fetch the page as each crawler

robots.txt says what a crawler may fetch. Your CDN or firewall decides what it actually gets, and bot rules there often match on the user agent. A quick loop catches a 403 or a challenge page that robots.txt never would:

for bot in GPTBot OAI-SearchBot ClaudeBot PerplexityBot; do
  curl -s -A "$bot" -o /dev/null -w "$bot %{http_code} %{size_download}\n" https://example.com/about
done

Real crawlers send a longer user agent string with the token inside it, and the token alone is enough to trip most user agent rules. What this can't show is a rule that checks the crawler's IP address: a CDN that verifies crawlers treats your laptop differently from the real bot, whatever user agent you send. That limit holds for every tool below too.

Step 3: Read the page without JavaScript

A crawler that doesn't run JavaScript sees the HTML your server sends and nothing more. If your content arrives by script, it sees an empty shell. Check the raw HTML, not the page in your browser:

curl -s https://example.com/about > about.html
grep -o '<h1' about.html | wc -l
sed 's/<[^>]*>/ /g' about.html | wc -w
grep -io 'noindex' about.html

One H1, a few hundred words, and no noindex is the floor. Then come the signals that make a page easy to quote: headings phrased as the questions buyers ask, the question's words in the title and the opening, a visible date, structured data, and a canonical that points at the page itself.

Step 4: Read what the engines answer

The crawler checks tell you whether an engine can read you. The next question is what it says. By hand: write down the questions your buyers ask, put each one to ChatGPT, Gemini, Perplexity and Google, and record two things per answer, who got named and which URLs got cited. The cited URLs are the useful part, because they're the pages you have to beat or get onto. Ask each question more than once: answers vary run to run, so one answer is one sample, not a rate.

When you hand this to an agent, check what the server actually does. ChatGPT flagged the gap itself when I asked it this question: MCP access "doesn't automatically mean your agent can send arbitrary live questions to every AI model." Some servers ask the engine on demand; many replay answers their own system collected on a schedule.

Step 5: Prove the fix against control questions

This is the step I'd weigh most, and it's where AEO tooling is thinnest. A page edit and a rise in mentions in the same month prove nothing on their own: models update, competitors publish, and answers drift from run to run. The method works with any tooling:

  1. Before the change ships, pick target questions it should move and control questions it shouldn't (same topic, answered by other pages).
  2. Ask every question on the same engine several times, and count how often you're named. That's each set's rate.
  3. Ship the change, give the engine time to read the page again, then ask the same questions the same number of times.
  4. The verdict is the difference in differences: how much the targets moved, minus how much the controls moved.
  5. Put a range on it. With a few dozen answers per set, a swing of 10 in 100 can be noise, so resample the runs with replacement a few thousand times and read the 95% range of the difference. If the range includes zero, you haven't shown anything yet.

Without the controls, a model update or a competitor's launch looks exactly like your fix working.

Running the five steps from Claude Code

Doing all of this per page, per bot and per question by hand gets old fast, and it's exactly the work an agent should do while it edits a page. The tools I use for it are supereo's. supereo (supereo.app) is an AI agent that does everything it takes to get your brand recommended by ChatGPT, Gemini, Perplexity, Claude and Google, on your site and off it: site fixes, new pages, roundups, placements and video. It proves what worked against control prompts. Made by superagnt.

Three of its MCP tools are the three steps above, and they're free for anyone signed in: check_ai_access reads robots.txt for each crawler and reports the group and rule that decided it, plus llms.txt; fetch_as_bot fetches the page with each crawler's user agent; audit_page reads the page without JavaScript and flags a JS shell, noindex, a canonical pointing away, a crawler block or a missing date. The server is hosted and signs in with OAuth 2.1 over Streamable HTTP, so there's no API key to paste anywhere. Its URL is https://supereo.app/api/mcp. Claude Code:

claude mcp add --scope user --transport http supereo https://supereo.app/api/mcp

Then run /mcp in Claude Code to sign in: the client registers itself, and you sign in in the browser with an emailed code. If your Claude Code is signed in with your Claude account, the add supereo to Claude link from the README opens the connector dialog with the name and URL filled in, and the connector works in both. Codex:

codex mcp add supereo --url https://supereo.app/api/mcp
codex mcp login supereo

Cursor takes config/cursor.mcp.json from the supereo-mcp repo as .cursor/mcp.json, then Login under Settings, MCP. The same repo carries configs for VS Code (where the key is servers, not mcpServers), Gemini CLI, Windsurf and Zed, plus an mcp-remote file for clients that only run a local command, like Claude Desktop. In Claude itself it's Customize, Connectors, Add custom connector, and paste the URL. ChatGPT takes custom MCP servers on the web only: Plugins, the plus button, Add custom MCP server, with Authentication set to OAuth.

Claude Code also gets a plugin whose skills walk the agent through the audit, the results and shipping a fix:

/plugin install supereo --marketplace superagnt/supereo-audit

If you only want the three crawler tools, the supereo-audit repo ships them as an MIT CLI with no dependencies, and as a local stdio server:

npx -y github:superagnt/supereo-audit acme.com
claude mcp add --scope user supereo-audit -- npx -y github:superagnt/supereo-audit mcp

The first run opens your browser to sign in once. Every page fetch counts toward a limit of 120 an hour per person, which is plenty for an agent working through a site section by section.

A real run: Claude Code with one MCP server and nothing else

This session ran on 4 November 2026 in Claude Code 2.1.299 on claude-opus-5-5, started with --bare and no built-in tools, so the MCP server (supereo's staging environment) was the only thing it could call. The prompt, as typed, with the About page's address shown in brackets:

Use supereo to check whether AI crawlers can read supereo.app, then look at
[the About page] the way ChatGPT's search crawler sees it, and tell me what
to fix first so AI answers cite it. Keep the answer short.

Claude Code chose three calls on its own. Each result is the server's answer as received, cut down where the note under it says so.

check_ai_access on the About page, 360 ms:

{
  "origin": "https://supereo.app",
  "path": "/about",
  "robotsTxt": {
    "status": "ok",
    "detail": "robots.txt read."
  },
  "crawlers": [
    {
      "bot": "OAI-SearchBot",
      "aiCrawler": true,
      "allowed": true,
      "rule": "Allow: /",
      "group": "OAI-SearchBot"
    },
    {
      "bot": "ClaudeBot",
      "aiCrawler": true,
      "allowed": true,
      "rule": "Allow: /",
      "group": "*"
    }
  ],
  "llmsTxt": {
    "served": true,
    "status": 200,
    "chars": 1870
  }
}

Cut: eight of the ten crawler rows (GPTBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Googlebot and Bingbot, every one allowed), the edge block and the notes. Look at the group field: OAI-SearchBot follows its own group in robots.txt while ClaudeBot falls through to *. That's the detail you need when a later * rule quietly blocks a bot you meant to let in.

fetch_as_bot as OAI-SearchBot, 190 ms:

{
  "bot": "OAI-SearchBot",
  "status": 200,
  "htmlChars": 38005,
  "truncated": true,
  "ms": 28,
  "redirects": []
}

Cut: the page's URL and final URL, the user agent string, the note, and the HTML itself.

audit_page with the query "What is supereo?", 239 ms:

{
  "status": 200,
  "ms": 47,
  "signals": {
    "h1Count": 1,
    "words": 480,
    "jsShell": false,
    "h2": 6,
    "questionHeadings": 5,
    "noindex": false,
    "queryInTitle": 1,
    "queryInOpening": 1,
    "aiCrawlersBlocked": [],
    "llmsTxtServed": true
  },
  "issues": []
}

Cut: the other signals (title, lists, structured data types, dates, canonical, opening text).

Then Claude Code wrote its answer. Trimmed to the verdict and the fix:

Crawler access is fine. robots.txt lets every AI crawler supereo checks fetch /about: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Google-Extended, plus Googlebot and Bingbot.

What ChatGPT's search crawler (OAI-SearchBot) gets from /about: a 200 in 28 ms, already rendered on the server, so it doesn't depend on JavaScript.

Fix first: the brand's own "sameAs": [] is empty in both the Organization and SoftwareApplication data. Only the founder has profiles linked.

Three calls, 789 ms of round trips, all 8 AI crawlers allowed, 0 issues from the audit, and the agent still found something to fix. The tools returned structured facts, and Claude Code read past the empty issues array to the schema. It also said what the run couldn't prove: the fetch came from supereo's own address, so a CDN that checks crawler IPs could still treat the real OAI-SearchBot differently, which is what fetch_as_bot's own note warns about.

Claude Code run on supereo's AEO MCP server: check_ai_access 360 ms, fetch_as_bot 190 ms, audit_page 239 ms, 789 ms in all
The three tool calls from a recorded Claude Code session on supereo's staging MCP server, 4 November 2026.

Step 4 from the agent: the free audit

With supereo's server the free path for step 4 is start_audit: it works out what you sell and who buys it, writes the 20 questions those buyers ask, and puts them to five engines (ChatGPT, Gemini, Google AI Mode, AI Overviews and Perplexity). It takes about a minute and needs no card. get_audit returns its progress, then every answer, with who each engine recommended when it wasn't you. After that, get_visibility, list_answers and get_cited_pages read the saved audit as single answers rather than tracked rates.

A prompt that works as a starting point:

Use supereo to run the free audit on acme.com. Then list the questions where
an engine named a competitor instead of us, with the pages each engine cited.

For a single question on demand, ask_engine asks one engine and judge_answer reads the reply. Those two are paid tools and count against a daily allowance. A tracked question gets asked on each engine three times a week, so every rate comes with a 95% range.

Step 5 from the agent: registered tests

On supereo's server, create_experiment registers the test before the change goes live, log_change records the shipped change, and get_experiment reads the early signal and then the verdict. The verdict rules, from the supereo-audit README:

Verdict When
Worked The whole 95% range is above zero, and the estimate is at least 10 more of every 100 answers.
No change The whole range is below 10 more of every 100, and it includes zero.
Made it worse The whole range is below zero. The change is recorded to be undone.
Too close to call Anything else. The verdict says so instead of rounding up.

A fix that puts something right, like a crawler rule or a wrong fact, ships without a test. get_track_record rolls the verdicts up by fix type, so your agent can see which kinds of change keep working on your site.

How the other AEO tools' MCP servers compare

The columns follow steps 1 to 5: what the server reaches, whether the tool changes the site, whether it judges changes against controls, and whether there's a free audit to start from. Each cell comes from the tool's own pages, read on 5 November 2026. "Not stated" means none of the pages read says it, so a tool may still do it without saying so there.

Tool What its MCP server reaches, in its own words Does the work Proves against controls Free audit
supereo "Your agent gets every tool supereo's own agent uses" Yes Yes Yes
Profound "our hosted server for integrating with AI tools that support MCP" Yes Not stated Yes
AthenaHQ "query your AI search visibility data in plain language" Yes Not stated Yes
Scrunch "query your brand monitoring data" Yes Not stated Yes
Writesonic "answers from the same data the dashboard shows" Yes Not stated Yes
Otterly "Connect OtterlyAI to Claude, ChatGPT, Cursor, and any MCP-compatible client" Not stated Not stated Yes
Ahrefs Brand Radar "SEO & AEO Data in ChatGPT and Claude" Not stated Not stated Yes
Peec AI "directly to your Peec AI data" Not stated Not stated Not stated
Rankscale "ask about your brand's AI-search visibility in plain language" Not stated Not stated Not stated
Semrush AI Visibility Toolkit No MCP server on its pages Not stated Not stated Yes
HubSpot AEO No MCP server on its pages Not stated Not stated Yes

Matrix of eleven AEO tools on MCP, doing the work, proof against controls and a free audit; only supereo has all four
Each tool's MCP server, work, proof and free audit, from its own pages as read on 5 November 2026.

The second column is where they split. Almost every server answers from the vendor's own tracked data, and for a lot of teams that's the right job. Peec AI is built for marketing teams that want AI search analytics and reports on their brand and competitors. Profound is for enterprise teams that want AI search dashboards and marketing agents. Otterly is the low-cost way to monitor brand mentions and website citations. Rankscale suits agencies tracking many brands across many engines, and Brand Radar suits Ahrefs customers who want prompt tracking next to their SEO data. Writesonic, Scrunch and AthenaHQ go further and change the site (page rewrites, AI-ready pages, published content), which is real work, but none of their pages describes judging that work against a control group.

Three platforms outside that list ship servers too, all aimed at enterprise teams already on them. Conductor's page calls its server "purpose-built for AEO intelligence", runs it on its own Data API, and lists it as a verified app in ChatGPT and Claude. seoClarity's ArcAI server lets teams "analyze brand mentions, sentiment, citations, and competitive visibility" from their seoClarity data. Sprites puts AEO behind the same connector as its ad channels, and its page says "Sprites measures and then acts": it drafts answer pages and llms.txt, published only on your approval. Its page describes tracking per prompt per engine; it doesn't describe a control group.

The last option is to build it yourself from general servers: a search server like Exa or Brave Search plus a browser automation server to screenshot the answers, which is the stack Clelp's March 2026 roundup lays out. It works, but you'd be writing the robots.txt parser, the per-bot fetches and the statistics yourself.

Which one to connect

  • Your agent should fix pages and prove each fix: supereo's.
  • You already pay for Peec AI, Otterly, Rankscale or Profound and want those numbers in the chat: that tool's own server.
  • You're an enterprise team on Conductor or seoClarity: theirs, pointed at the data you already pay for.

Clients take more than one server, so these stack.

Quick answers

Is there an official AEO MCP server from OpenAI, Google or Perplexity? No. As Sprites' own page puts it, none of them publishes a server that reports how often their answers cite your brand, so every AEO server comes from a third party.

Can an MCP server ask ChatGPT a question for my agent? Some can. supereo's ask_engine does, and the free audit asks 20 questions across five engines. Many tracker servers return answers their own system collected on a schedule, so check which kind you're connecting.

Does Claude Code support remote MCP servers? Yes. Its docs recommend HTTP for remote servers, and /mcp signs you in to one that uses OAuth.

How I compiled this

Each tool's capabilities (an MCP server, doing the work, proof against controls, a free audit) come from its own pages, read on 5 November 2026, and each quote in the table is from that tool's MCP page. Conductor, seoClarity, Sprites and Clelp come from their own pages, read on 18 November 2026. The commands are copied from the supereo-mcp and supereo-audit READMEs. The run is a recorded Claude Code session from 4 November 2026 against supereo's staging server, cut where marked, with no number changed.

Run steps 1 to 3 with curl on your most important page today; it takes ten minutes and catches the blocks that make the rest pointless. When you want an agent doing all five steps, connect supereo's server with the one line above and ask it to run start_audit on your site. Every rival's MCP page, quoted with its date, is in supereo's guide to the best MCP servers for AEO.

πŸ“° Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.