Dev.to WebDev 🛠 Dev 👁 0 📖 1 min read

How to Scrape React & Next.js Sites with Claude Desktop (Without Puppeteer)

If you’ve tried to build an AI agent that scrapes modern websites, you’ve probably hit this wall: standard fetch() tools return blank HTML. Why? Because sites built with React, Vue, or Next.js render on the client. To a

If you’ve tried to build an AI agent that scrapes modern websites, you’ve probably hit this wall: standard fetch() tools return blank HTML.

Why? Because sites built with React, Vue, or Next.js render on the client. To actually get the data, you need a headless browser.

Historically, this meant writing a Puppeteer or Playwright script, containerizing it in Docker, and hosting it on a VPS. That is a massive amount of overhead just to extract a pricing table or read documentation.

Here is how to solve this instantly in Claude Desktop using the x402-cluster MCP server—zero Docker or Puppeteer config required.

The Fix: Serverless Headless Browsers

Instead of running a heavy local browser, you can route the request through a serverless Edge browser using the x402 API.

You send it a URL and a JSON schema, it spins up a headless browser on Cloudflare's infrastructure, waits for the React DOM to render, and returns perfectly typed JSON.

How to configure Claude Desktop

Open your claude_desktop_config.json file and add this block:


json
{
  "mcpServers": {
    "x402-cluster": {
      "url": "[https://x402-cluster.ian-944.workers.dev/v1/mcp.json](https://x402-cluster.ian-944.workers.dev/v1/mcp.json)",
      "headers": {
        "Authorization": "Bearer sk_live_YOUR_API_KEY"
      }
    }
  }
}
📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.