Dev.to Security 🔐 Cybersecurity Fri, 13 Nov 2026 08:16:47 +0000 👁 0 📖 5 min read

How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python)

If you've ever filtered your access logs for Googlebot to see how Google crawls your site, there's a good chance some of what you looked at wasn't Google. The User-Agent header is just text. Scrapers, SEO tools, vulnerab

If you've ever filtered your access logs for Googlebot to see how Google crawls your site, there's a good chance some of what you looked at wasn't Google. The User-Agent header is just text. Scrapers, SEO tools, vulnerability scanners and badly behaved bots routinely claim to be Googlebot, because some sites let Googlebot through rate limits, paywalls and bot protection that would block anyone else.

That matters for two reasons. First, any crawl analysis built on fake Googlebot traffic is wrong: you'll see "Google" hitting URLs it never requests, at rates it never uses. Second, if your app treats the Googlebot User-Agent specially, you've built a bypass that anyone can use.

Google documents two ways to verify a crawler. This post covers both, wraps them in a script you can run over a log file, and ends with what to do once you know.

Method 1: reverse DNS, then forward DNS

The classic check has two steps, and both matter:

Do a reverse DNS lookup on the client IP. A real Google crawler resolves to a hostname under googlebot.com, google.com or googleusercontent.com.
Do a forward DNS lookup on that hostname and confirm it resolves back to the same IP.

Step 2 is the part people skip, and it's the part that stops spoofing. Anyone who controls the reverse DNS for their own IP range can make it say crawl-1-2-3-4.googlebot.com. They can't make Google's DNS resolve that name back to their IP.

import socket

GOOGLE_SUFFIXES = (".googlebot.com", ".google.com", ".googleusercontent.com")

def verify_by_dns(ip: str) -> bool:
    try:
        host, _, _ = socket.gethostbyaddr(ip)
    except (socket.herror, socket.gaierror):
        return False
    if not host.endswith(GOOGLE_SUFFIXES):
        return False
    try:
        _, _, addrs = socket.gethostbyname_ex(host)
    except socket.gaierror:
        return False
    return ip in addrs

gethostbyname_ex only returns IPv4 addresses. If your logs include IPv6 clients, use socket.getaddrinfo(host, None) and compare against every returned address instead.

Method 2: published IP ranges

Google also publishes the IP ranges its crawlers use as JSON files (for the common crawlers, special-case crawlers and user-triggered fetchers). Checking an IP against a list is faster than two DNS lookups per address and doesn't depend on DNS being reachable from wherever you run the script.

import ipaddress, json, urllib.request

RANGE_URLS = [
    "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "https://developers.google.com/static/crawling/ipranges/special-crawlers.json",
    "https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json",
]

def load_ranges():
    nets = []
    for url in RANGE_URLS:
        data = json.load(urllib.request.urlopen(url, timeout=20))
        for p in data["prefixes"]:
            nets.append(ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")))
    return nets

def verify_by_range(ip: str, nets) -> bool:
    addr = ipaddress.ip_address(ip)
    return any(addr in n for n in nets if n.version == addr.version)

Check Google's crawler verification documentation for the current list of files before relying on these exact URLs; they have moved before (the older /search/apis/ipranges/ paths now redirect here). The ranges change over time, so download them fresh each run, or cache them for a day.

Checking a whole log

Put it together: read a combined-format access log, keep requests whose User-Agent claims to be Googlebot, verify each unique IP once, and report.

import re, sys, collections

LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "(?:\S+) (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

def main(path):
    nets = load_ranges()
    claims = collections.Counter()
    paths = collections.defaultdict(collections.Counter)
    for line in open(path, errors="replace"):
        m = LINE.match(line)
        if not m or "googlebot" not in m.group(4).lower():
            continue
        ip, url = m.group(1), m.group(2)
        claims[ip] += 1
        paths[ip][url] += 1
    real = fake = 0
    for ip, n in claims.most_common():
        ok = verify_by_range(ip, nets) or verify_by_dns(ip)
        real += n if ok else 0
        fake += 0 if ok else n
        if not ok:
            top = ", ".join(u for u, _ in paths[ip].most_common(3))
            print(f"FAKE {ip:<40} {n:>7} requests  e.g. {top}")
    total = real + fake
    if total:
        print(f"\n{total} requests claiming Googlebot: {real} verified, {fake} fake ({fake/total:.0%})")

if __name__ == "__main__":
    main(sys.argv[1])

The range check runs first because it's cheap; DNS is the fallback for anything the ranges miss. On a large log, unique IPs are usually a small fraction of requests, so even the DNS path finishes quickly.

What the results usually look like

On a typical public site, you'll often find:

A handful of IPs with a fake Googlebot User-Agent responsible for a large share of "Googlebot" requests, usually hitting search pages, feeds, or URLs with query strings no real crawler would generate.
Fake bots ignoring robots.txt, which real Googlebot respects. If "Googlebot" is fetching a disallowed path, that's a strong hint before you even run the check.
Real Googlebot traffic that looks calmer than you expected once the fakes are removed.

What to do with it

Fix your crawl analysis first. Re-run whatever log analysis you rely on with only verified requests. Crawl frequency, status code breakdowns and "most crawled URLs" can change a lot.
Don't trust the User-Agent for access decisions. If your app relaxes rate limits, skips a paywall or bypasses bot protection for Googlebot, verify the IP the same way first, and cache the result.
Block or rate-limit the fakes like any other bot. Don't return something special to them; treat them as the unidentified client they are.
Don't block based on reverse DNS failures alone. DNS lookups fail for boring reasons. Use the range list as the primary check, and log rather than block when DNS is unavailable.

Verified-crawler log analysis is one of the things we lean on at seotto.com, because the rest of an SEO investigation is only as good as the traffic it starts from.