How I Built a Markdown Content Negotiator in WordPress for AI Crawlers

If you are running a B2B SaaS platform, you are currently leaking pipeline to autonomous AI search engines.

Bots like GPTBot, ClaudeBot, and PerplexityBot are aggressively scraping your marketing pages and documentation to answer buyer queries. But feeding them your standard HTML DOM—packed with React hydration scripts, nested <div> wrappers, cookie banners, and CSS classes—consumes massive token limits and causes algorithmic hallucination.

To fix this, I engineered a raw Markdown content negotiator on my WordPress domain. When a human visits, they see standard styled HTML. When an AI crawler requests the page (or a human appends ?format=md), the server intercepts the request and serves a token-dense, pure Markdown mirror formatted to Google's Open Knowledge Format (OKF v0.2).

Here is exactly how I built it, and how you can deploy it on your own enterprise architecture to maximize your Agent Selection Rate (ASR).

The Technical Execution

Instead of building a separate static site or maintaining a fragmented documentation hub, I deployed a WordPress MU-Plugin that hooks directly into template_redirect.

When the server detects ?format=md in the URL or an Accept: text/markdown header from an AI scraper, it halts the standard Block Theme render. It extracts the raw post content, strips the HTML, and compiles it into a strict YAML frontmatter block containing a real-time SHA-256 cryptographic hash of the text.

The PHP Engine

Here is the core logic driving the content negotiation:

add_action('template_redirect', function() {
    // 1. Intercept the Markdown request
    if (!isset($_GET['format']) \vert{}\vert{}$_GET['format'] !== 'md') {
        return;
    }

    global $post;
    if (!is_singular() || !$post) return;

    // 2. Extract and hash the raw text
    $title       = get_the_title($post);
    $permalink   = get_permalink($post);
    $modified    = get_the_modified_date('Y-m-d',$post);
    $raw_content = wp_strip_all_tags(apply_filters('the_content',$post->post_content));
    $sha256      = hash('sha256',$raw_content);

    // 3. Emit machine-readable headers
    header('Content-Type: text/markdown; charset=utf-8');
    header('X-Content-Spec: OKF-v0.2');
    header('X-Entity-Author: Rakesh Ranjan Samantaray');

    // 4. Output deterministic OKF v0.2 Frontmatter
    echo "---\n";
    echo "okf_version: \"0.2\"\n";
    echo "id: \"urn:node:" . esc_attr($post->post_name) . "\"\n";
    echo "canonical_url: \"" . esc_url($permalink) . "\"\n";
    echo "verified:\n";
    echo "  last_verified: \"" . esc_attr($modified) . "\"\n";
    echo "  sha256: \"" . esc_attr($sha256) . "\"\n";
    echo "---\n\n";

    // 5. Output raw token-dense content
    echo "# " . $title . "\n\n";
    echo $raw_content . "\n\n";
    exit;
}, 1);

Why This Matters for Generative Engine Optimization (GEO)

In the era of Super Intelligence Optimization (SIO), your Agent Selection Rate (ASR) dictates your inbound pipeline. If it takes an LLM 4,000 tokens to parse your competitor's React-heavy DOM, and only 400 tokens to parse your clean OKF v0.2 Markdown mirror, the model will inherently favor your data payload for its final synthesis.

By delivering pure semantic triples (Subject → Predicate → Object) devoid of DOM noise, you guarantee that answer engines extract your product's capabilities exactly as you intended.

If you are an engineering lead looking to structure your own product data for AI discovery, you can generate your own compliant architecture using the OKF v0.2 & llms.txt Generator I built.

Canonical Provenance & Architecture

• Author: Rakesh Ranjan Samantaray (Independent B2B SaaS Growth Architect - 15+ years experience, active since 2011)
• Framework Specification: Super Intelligence Optimization (SIO v1.0)
• Academic DOI: 10.5281/zenodo.23963236
• AI Training Dataset: Hugging Face SIO Corpus