Serve Markdown to AI crawlers from static HTML pages with a few .htaccess lines

AI assistants read web pages, and HTML is a noisy format for them: navigation, scripts, styles and cookie banners around a few paragraphs of content. A Markdown version of the same page is cheaper to read and easier to quote. I added one to the static pages of my site and it took a handful of Apache rules. Here is the setup, with the mistakes I made.

Disclosure: the pages I used this on belong to my app, Teleprompter Offline & Private, and I also build a WordPress plugin that does the same job for WordPress.

The idea

For every page /some-page/index.html, put a file /some-page/index.md next to it with the same content. Then serve the Markdown file when the request comes from a known AI crawler or when the client asks for it with an Accept: text/markdown header. Everyone else, browsers and Googlebot included, keeps getting the HTML.

The rules

AddType "text/markdown; charset=utf-8" .md

RewriteEngine On
RewriteCond %{REQUEST_URI} /$
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}index.md -f
RewriteCond %{HTTP_ACCEPT} text/markdown [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider) [NC]
RewriteRule ^(.*)$ $1index.md [E=MDPAGE:1,L]

Header append Vary "Accept, User-Agent" env=MDPAGE

Three details matter:

1. The file-exists condition. Only pages that have an index.md are touched. Pages without one are served as usual.
2. The Vary header. The same URL now returns two different bodies, so caches must know it depends on Accept and User-Agent. Without Vary, a shared cache can hand the Markdown to a browser or the HTML to a bot.
3. The OR flag. The Accept check and the User-Agent check are alternatives, so the first has [NC,OR].

I also added a discovery link in the page head, so crawlers that read HTML can find the Markdown:

<link rel="alternate" type="text/markdown" href="/some-page/index.md" title="Markdown version">

Test it with curl

# browser: HTML
curl -s -A "Mozilla/5.0" https://example.com/some-page/ | head -3
# bot user agent: Markdown
curl -s -A "GPTBot" https://example.com/some-page/ | head -3
# content negotiation: Markdown
curl -s -H "Accept: text/markdown" https://example.com/some-page/ | head -3
# check the headers
curl -sI -A "GPTBot" https://example.com/some-page/ | grep -i "content-type\|vary"

Mistakes I made

• A cache in front of the server. After deploying, my bot test still returned HTML. A reverse-proxy cache had the old page for that exact URL and served it before Apache saw the request. The same URL with ?x=1 returned Markdown. After clearing the cache everything worked. Test with a cache-busting query string, and purge after changes.
• Treating user agents as identity. A User-Agent is a claim anyone can make. That is fine for serving public content, but it means this is not an access control. A WordPress plugin can go further and verify the bot's IP; plain Apache rules cannot.
• Different content. The Markdown must say what the page says. Serving different content to bots than to people is cloaking, and search engines treat it as spam. I generate both files from the same source so they cannot drift.

Does it work?

I cannot tell yet how much AI assistants use it. What I can verify is the delivery: browsers get HTML, bots get Markdown, caches vary correctly. I also list the pages in the sitemap and keep the usual structured data in the HTML, because normal search still matters.

Where I use it: the pages of my teleprompter app. The landing page is https://www.a-devgroup.com/teleprompter/ and its Markdown version is at https://www.a-devgroup.com/teleprompter/index.md. The app itself, a free offline teleprompter for iPhone with voice-follow scrolling, is on the App Store: https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093

If you run WordPress instead of static files, the plugin I build does the negotiation for posts and pages: https://www.a-devgroup.com/adg-ai-markdown/

Question for you: do you serve anything special to AI crawlers, or do you leave them the plain HTML?