Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 5 min read

Give your AI agent real Google rankings with a self-hosted MCP server

Most agents can already "search the web". What they get back is a handful of sources picked by their own search provider, not what Google shows for a query: no positions, no AI Overview, no "People also ask", and no say

Give your AI agent real Google rankings with a self-hosted MCP server

Most agents can already "search the web". What they get back is a handful of sources picked by
their own search provider, not what Google shows for a query: no positions, no AI Overview, no
"People also ask", and no say in which country the results are for. For SEO work, rank checks or
competitor research, that is the part you need.

This walkthrough plugs Google's actual results page into Claude Code, Cursor, Gemini CLI or
Claude Desktop as a tool, using SERP Maven, an
open-source Google scraper with a built-in MCP server. It runs on your machine through your
residential proxy, so there is no per-search bill. Setup takes about five minutes.

What you need

  • Python 3.10+, or uv (it brings its own Python).
  • A residential proxy with sticky sessions. Any provider works. We build SERP Maven at NodeMaven, so NodeMaven proxies have a one-command login; others take a proxy URL.

1. Install

uv tool install serpmaven
serpmaven install

The second command downloads the browser SERP Maven drives, about 150 MB, once per machine.
Do it now: an MCP server cannot stop and ask you about a download.

2. Save your proxy login

For NodeMaven, enter the proxy username and password from the dashboard:

serpmaven login

For any other provider, give the proxy URL with {session} where the provider expects a
session id in the username:

serpmaven login --proxy "http://USER-session-{session}:[email protected]:7000"

The login is stored in a file only your user can read. The MCP server picks it up from there,
so no password goes into your editor's config.

3. Try one search from the terminal

serpmaven search "best running shoes" --country us

The first search takes a few minutes. SERP Maven opens a real Chrome window and browses a few
ordinary pages on the first proxy IP before it asks Google anything, because a fresh browser
on a fresh IP is the thing Google refuses most. Later searches take seconds, and new proxy IPs
start from the cookies Google already accepted. The run ends with a summary of pages served
and traffic used.

4. Connect it to your agent

Claude Code. One command:

claude mcp add google-search -- serpmaven mcp --country us --prewarm

Cursor. Add this to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "google-search": {
      "command": "serpmaven",
      "args": ["mcp", "--country", "us", "--prewarm"]
    }
  }
}

Gemini CLI. The repository is also a Gemini CLI extension:

gemini extensions install https://github.com/nodemaven/serpmaven

Claude Desktop. The same JSON block goes into Settings β†’ Developer β†’ Edit Config. Restart
Claude Desktop and the google_search tool appears in the tools menu:

Claude Desktop asking to run google_search with the query, country, language and links

If a client says it cannot find serpmaven, it was started without your shell's PATH. Put the
full path from which serpmaven (where serpmaven on Windows) into command.

--prewarm warms the first proxy IP as soon as the server starts. That takes three to five
minutes, longer than some clients wait for a tool, so give it those minutes after a restart
before the first question. Later searches take under ten seconds. With NodeMaven, --country
picks the country of the proxy IP; the tool's gl argument sets the country Google answers for
on each call.

What the agent gets back

The server exposes one tool, google_search(query, pages, hl, gl, links):

  • query - the search;
  • pages - result pages to fetch, about 10 results each (default 1);
  • hl, gl - interface language and country, e.g. de and de;
  • links - how many top results get their full URL (default 5); fewer answers sooner.

Ask your agent something like "Who ranks on Google in the US for 'best python web scraping
library'?"
and the tool returns compact JSON with Google's own order. A real answer, trimmed:

{
  "results": [
    {
      "position": 1,
      "title": "Scrapy, a fast high-level web crawling & scraping ...",
      "link": "https://github.com/scrapy/scrapy",
      "snippet": "Scrapy is a web scraping framework to extract structured data from websites..."
    },
    {
      "position": 2,
      "title": "Advanced Web Scraping Tutorial! (w/ Python Beautiful Soup ...",
      "link": "https://www.youtube.com/watch?v=DcI_AZqfZVc"
    },
    {
      "position": 3,
      "title": "High Level Web Scraping Library for Python",
      "link": "https://www.reddit.com/r/Python/comments/1iy78rw/high_level_web_scraping_library_for_python/",
      "snippet": "Hi, I started working on an open source Python library that handles common web scraping tasks..."
    }
  ],
  "ai_overview": "There is no single best Python web scraping library because the right choice depends on your target site and scale. For small static pages, Beautiful Soup paired with Requests or HTTPX is the easiest starting point...",
  "related_searches": ["Best python web scraping library free", "Best python web scraping library github", "Web scraping tools"]
}

Results past the first five carry the site instead of the full link (raise links if the agent
needs them). Organic results and related searches come with every page; "People also ask",
Google's AI Overview and the map pack (name, rating, address, hours) come when Google shows them.

What it costs

The proxy is the only cost. The first IP's warm-up uses about 60 MB once. After that a search
is about 0.4 MB, so 1000 searches are roughly 0.4 GB of proxy traffic. Keep pages at 1 unless
you need more: every page is traffic on your plan.

New to NodeMaven? Sign up
and enter GOOGLENM for 2 GB of free traffic.

When something goes wrong

  • The server shows as failed or disconnected in your client: it stops at start when there is no saved proxy login or no browser. Run serpmaven login and serpmaven install once, then restart the client.
  • The first search times out right after the client starts: the proxy IP is still warming up. Wait a few minutes and ask again; the warm-up keeps going in the background.
  • A captcha or a block is never read as "no results". SERP Maven drops the proxy IP Google refused and asks again on another one; a search that still fails comes back with an error, so the agent can tell "Google refused" from "nothing found".
  • Results for a different query are Google's quiet way of turning a scraper away. When a proxy IP starts answering like that, SERP Maven drops it and asks the query again on a fresh one.
  • On a server without a screen, run it under xvfb-run -a, or use the Docker image ghcr.io/nodemaven/serpmaven.
  • serpmaven doctor starts the browser on a blank page and reports what a website would see. Run it first if searches keep failing.

The same thing without an agent

The MCP server is one of four ways to run SERP Maven. The others:

  • serpmaven search -f queries.txt -o results.jsonl for a batch;
  • serpmaven serve for a local HTTP API with SERP-API-style JSON;
  • from serpmaven import Scraper in Python.

Code, options and issues: github.com/nodemaven/serpmaven.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.