How to Give Claude Desktop & Cursor Headless Browser & Anti-Bot Web Scraping Powers via MCP (`uvx crawlengine-mcp`)

Modern AI desktop clients like Claude Desktop and Cursor are exceptional at reading local codebases, but they hit an immediate wall when tasked with scraping live web documentation, parsing single-page web applications (SPAs), or extracting data from JavaScript-heavy targets protected by anti-bot verification.

Standard curl and fetch approaches fail because modern websites execute client-side hydration, require realistic browser TLS fingerprints, and render content dynamically into the DOM.

In this guide, we will connect your AI workflow directly to a resilient headless browser and anti-bot web scraping engine using standard Model Context Protocol (MCP) servers published on PyPI: crawlengine-mcp and browserbridge-mcp.

1. Zero-Install Execution via PyPI and uvx

Both servers are distributed as self-contained Python packages that require no global installation. Using Astral uv, you can execute them directly:

For Deep Anti-Bot Crawling & Clean Markdown Extraction:

uvx crawlengine-mcp

For Headless Browser Automation, Screenshots & DOM Inspection:

uvx browserbridge-mcp

Each server runs in stdio mode, communicating over standard input and output according to the official MCP specification.

2. Configuring Claude Desktop (claude_desktop_config.json)

To enable these capabilities in Claude Desktop, open your configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

Add both servers to your mcpServers object:

{
  "mcpServers": {
    "crawlengine": {
      "command": "uvx",
      "args": [
        "crawlengine-mcp"
      ]
    },
    "browserbridge": {
      "command": "uvx",
      "args": [
        "browserbridge-mcp"
      ]
    }
  }
}

Restart Claude Desktop. When you start a new conversation, Claude will immediately display access to web crawling and browser automation tools.

3. Configuring Cursor (.cursor/mcp.json)

For Cursor, add the tools directly at the workspace root or global level in .cursor/mcp.json:

{
  "mcpServers": {
    "crawlengine": {
      "command": "uvx",
      "args": [
        "crawlengine-mcp"
      ]
    },
    "browserbridge": {
      "command": "uvx",
      "args": [
        "browserbridge-mcp"
      ]
    }
  }
}

You can also use our interactive configuration generator at johannschiffer.github.io to customize environment parameters, browser timeouts, and output limits visually.

4. Key Capabilities & Available MCP Tools

Once connected, your AI assistant can execute:

1. scrape_markdown(url, selector, bypass_cache): Fetches target web pages, strips navigation noise and tracking scripts, and returns high-density semantic Markdown formatted specifically for LLM context windows.
2. browser_navigate(url): Launches a headless browser session, waits for full network idle and hydration, and captures the rendered DOM state.
3. browser_screenshot(selector, full_page): Takes verified high-resolution screenshots for UI/UX audits and visual validation.
4. browser_accessibility_tree(url): Captures the full WCAG accessibility tree for compliance audits without running bloated frontend test runners.

5. Scaling to Enterprise: Proxies, TLS Evasion & Production Stacks

The open packages (crawlengine-mcp and browserbridge-mcp) provide instant, deterministic local scraping. When dealing with enterprise scale, aggressive Cloudflare Turnstile barriers, or distributed scraping pipelines, you need deeper infrastructure.

• Genesis Starter Kit ($0+): Includes ready-to-use configs, prompt playbooks, and local orchestrators. Download free on Gumroad.
• Genesis Enterprise Suite ($19 USD): Production-grade suite featuring residential proxy rotation, advanced Cloudflare/TLS fingerprint spoofing, Docker Compose sidecar topology, and automated rate-limit recovery. Available on Gumroad.

Explore our full interactive toolchain at https://johannschiffer.github.io/ and start giving your agents verified web capabilities today.