Dev.to WebDev 🛠 Dev 👁 0 📖 4 min read

Track AI Overviews and People Also Ask for your keywords every day (Python)

A rank tracker answers one question: where is my URL? It does not answer whether the SERP now has an AI Overview, or which People Also Ask questions Google is suggesting. Those are the parts that change fastest, and they

A rank tracker answers one question: where is my URL? It does not answer whether the SERP now has an AI Overview, or which People Also Ask questions Google is suggesting. Those are the parts that change fastest, and they are usually the reason a page that "still ranks" stops getting clicks.

This is a small loop that records those two features per keyword per day. It uses one POST per keyword, stores the result in SQLite, and leaves the raw JSON alone so you can re-parse it later.

What to record

For each keyword, I want four things:

  • Did an AI Overview appear?
  • Which PAA questions were shown?
  • Which country and language the check used?
  • The request id, so a weird result can be traced back.

The response from SerpBase's /google/search already has ai_overview and people_also_ask fields when Google shows them. Both are optional. A missing AI Overview is not an error; it is data.

One warning before the code: the shape of ai_overview can vary. Sometimes it is a string, sometimes an object with a summary and sources. PAA items are usually objects, but the key holding the question can differ. The helpers below handle the common shapes without pretending the schema never changes.

The fetch and extract helpers

import os
import requests

API_KEY = os.environ["SERPBASE_API_KEY"]
ENDPOINT = "https://api.serpbase.dev/google/search"


def fetch_serp(keyword: str, gl: str = "us", hl: str = "en") -> dict:
    resp = requests.post(
        ENDPOINT,
        headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
        json={"q": keyword, "gl": gl, "hl": hl},
        timeout=30,
    )
    data = resp.json()
    if data.get("status") != 0:
        raise RuntimeError(
            f"status={data.get('status')} request_id={data.get('request_id')}"
        )
    return data


def paa_questions(data: dict) -> list[str]:
    out = []
    for item in data.get("people_also_ask") or []:
        if isinstance(item, str):
            out.append(item)
        elif isinstance(item, dict):
            q = item.get("question") or item.get("title")
            if q:
                out.append(q)
    return out


def has_ai_overview(data: dict) -> bool:
    value = data.get("ai_overview")
    if isinstance(value, str):
        return bool(value.strip())
    return bool(value)

status is the field that matters. HTTP 200 alone does not mean the request succeeded. Failed calls are not charged, and request_id is what support will ask for if a response looks wrong.

Store one row per keyword per day

SQLite is enough for a personal tracker. The primary key prevents double-counting if the cron runs twice.

import json
import sqlite3
from datetime import date

con = sqlite3.connect("serp_features.db")
con.execute(
    """
    CREATE TABLE IF NOT EXISTS checks (
        day TEXT NOT NULL,
        keyword TEXT NOT NULL,
        gl TEXT NOT NULL,
        hl TEXT NOT NULL,
        has_ai_overview INTEGER NOT NULL,
        paa_count INTEGER NOT NULL,
        paa_json TEXT NOT NULL,
        request_id TEXT,
        PRIMARY KEY (day, keyword, gl, hl)
    )
    """
)

def save_check(keyword: str, data: dict, gl: str = "us", hl: str = "en") -> None:
    questions = paa_questions(data)
    con.execute(
        """
        INSERT OR REPLACE INTO checks
        (day, keyword, gl, hl, has_ai_overview, paa_count, paa_json, request_id)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
        """,
        (
            date.today().isoformat(),
            keyword,
            gl,
            hl,
            int(has_ai_overview(data)),
            len(questions),
            json.dumps(questions, ensure_ascii=False),
            data.get("request_id"),
        ),
    )
    con.commit()

Run it over a list. Keep the list small at first; 30 keywords is enough to see whether the idea is useful.

keywords = [
    "serp api",
    "serp features tracking",
    "google answer box api",
]

for kw in keywords:
    data = fetch_serp(kw)
    save_check(kw, data)
    print(kw, has_ai_overview(data), len(paa_questions(data)))

The query that makes it useful

A week of daily rows turns into a feature timeline:

SELECT
  keyword,
  day,
  has_ai_overview,
  paa_count
FROM checks
WHERE day >= date('now', '-7 day')
ORDER BY keyword, day;

The interesting signal is not the absolute count. It is the transition: a keyword that had no AI Overview for six days and gets one on day seven is a content opportunity. So is a PAA question that disappears after you publish an answer.

If you want a weekly digest, group by keyword and compare the first and last row. Do not average the boolean; report the change.

Cost and limits

Every /google/search call is 1 credit. One hundred keywords checked daily is about 3,000 requests a month. At the $3 / 10,000 credit Starter Boost rate, that is roughly $0.90/month. The new-account balance includes 500 free credits, which is enough to validate the loop on a few dozen keywords before scheduling it.

A few things that keep the bill predictable:

  • Pin gl and hl. A "missing" feature is often a different country or language.
  • Do not paginate for this job. The first page is enough for AI Overview and PAA, and page two is another request.
  • Store the raw response if you plan to re-parse later. Re-fetching to test a parser is the expensive way to learn.
  • Treat ai_overview as optional. Google does not show it on every query, and the API should not be expected to invent one.

Why not scrape the page?

You can scrape Google's HTML for the same signals until the markup changes. The failure mode is quiet: the selector returns nothing, your table says "no AI Overview", and you conclude the feature disappeared when the parser just stopped matching. Structured JSON does not remove every ambiguity, but it gives you typed fields and a request_id when something is off.

The longer version of this monitoring model, including how to diff the SERP features week over week, is here: https://serpbase.dev/blog/serp-feature-monitoring-track-the-search-page-not-just-rankings

The endpoint parameters and response fields are documented at https://serpbase.dev/docs/google_web_search

If you only track rankings today, add the two columns first. The AI Overview and PAA history is usually more interesting than another position number.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.