Dev.to WebDev 🛠 Dev 👁 0 📖 8 min read

I Built a Python Tool to Analyze Googlebot Server Logs

How to parse access logs, identify crawler requests, detect HTTP errors, and build a simple crawling diagnostics report with Python. When working on technical SEO audits, I often need to answer a question that cannot

I Built a Python Tool to Analyze Googlebot Server Logs

How to parse access logs, identify crawler requests, detect HTTP errors, and build a simple crawling diagnostics report with Python.

When working on technical SEO audits, I often need to answer a question that cannot be solved by looking at a website in a browser:

What are search engine crawlers actually requesting from the server?

A page may be listed in an XML sitemap but have no internal links. Another page may return HTTP 200 but contain almost no useful content. Some URLs may generate repeated server errors, while others may redirect through several intermediate addresses.

A regular website crawl is useful, but it does not show the complete history of requests made by search engine bots.

This is where server access logs become valuable.

In this tutorial, I'll build a small Python tool that analyzes Apache/Nginx access logs, extracts requests containing the Googlebot User-Agent, groups URLs by HTTP status code, and exports the results to CSV.

The goal is not to build a complete log analysis platform. I want a simple, reproducible tool that can help identify suspicious crawling patterns during a technical website audit.

1. What information can we get from access logs?

A typical Apache Combined Log Format entry looks like this:

66.249.66.1 - - [22/Nov/2026:08:15:10 +0300] "GET /catalog/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It contains several useful fields:

Field Description
IP address Client IP address
Timestamp Date and time of the request
HTTP method GET, POST, etc.
Request path Requested URL path
Protocol HTTP protocol version
Status code Server response
Response size Number of bytes sent
Referer Referring URL
User-Agent Client identification string

For crawling diagnostics, I am particularly interested in:

  • Which URLs were requested?
  • How frequently were they requested?
  • What HTTP status codes did they return?
  • When was each URL requested last?
  • Are there repeated server errors?
  • Are there suspicious URLs with parameters?
  • Are crawlers spending requests on unnecessary URL variations?

These questions help turn raw server logs into actionable technical information.

2. Important: User-Agent is not proof of Googlebot

Before writing the parser, there is one security detail worth mentioning.

It is easy to filter log entries containing the word Googlebot. However, User-Agent strings can be spoofed.

A request like this:

Mozilla/5.0 (compatible; Googlebot/2.1)

does not prove that the request came from Google.

For reliable identification, Google recommends verifying the requesting IP address using reverse DNS followed by forward DNS confirmation, or checking Google's published IP ranges.

Official documentation:

https://developers.google.com/search/docs/crawling-indexing/googlebot

In this first version, I will filter by User-Agent to demonstrate the parsing workflow. I will also show where IP verification should be added.

Important: The sample log entries in this tutorial are synthetic test data. They are not presented as verified Googlebot traffic.

3. Preparing the test environment

We need Python 3.10 or newer.

Create a project directory:

mkdir googlebot-log-analyzer
cd googlebot-log-analyzer

Create a file:

googlebot_analyzer.py

We will use only Python standard libraries:

  • re for parsing log lines;
  • csv for exporting reports;
  • datetime for timestamp processing;
  • collections for aggregating requests;
  • argparse for command-line arguments;
  • urllib.parse for handling URL paths and parameters.

No external packages are required.

4. Creating a sample access.log

Here are several example entries representing different crawling scenarios:

66.249.66.1 - - [22/Nov/2026:08:15:10 +0300] "GET /catalog/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.66.1 - - [22/Nov/2026:08:16:02 +0300] "GET /catalog/?page=2 HTTP/1.1" 200 19102 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.66.1 - - [22/Nov/2026:08:22:15 +0300] "GET /product/product-103/ HTTP/1.1" 301 312 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.66.1 - - [22/Nov/2026:08:25:40 +0300] "GET /product/product-104/ HTTP/1.1" 404 1620 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.66.1 - - [22/Nov/2026:08:27:12 +0300] "GET /product/product-105/ HTTP/1.1" 503 910 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

192.0.2.55 - - [22/Nov/2026:08:31:00 +0300] "GET /catalog/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The last entry uses a documentation-only IP address. It is deliberately included to demonstrate why filtering by User-Agent alone is insufficient.

Save the complete sample as access.log.

5. Building the Python log analyzer

Now let's implement the parser.

The script will:

  1. Read access.log.
  2. Parse Combined Log Format entries.
  3. Filter requests containing Googlebot in the User-Agent.
  4. Group requests by URL.
  5. Count HTTP response categories.
  6. Record the last request timestamp.
  7. Export the results to CSV.

Here is the complete implementation:

import argparse
import csv
import re

from collections import defaultdict
from datetime import datetime
from urllib.parse import urlsplit


LOG_PATTERN = re.compile(
    r'(?P<ip>\S+) \S+ \S+ '
    r'\[(?P<time>.*?)\] '
    r'"(?P<method>\S+) (?P<path>\S+) (?P<protocol>[^"]+)" '
    r'(?P<status>\d{3}) (?P<size>\S+) '
    r'"(?P<referer>[^"]*)" '
    r'"(?P<ua>[^"]*)"'
)


def parse_timestamp(value):
    return datetime.strptime(
        value,
        "%d/%b/%Y:%H:%M:%S %z"
    )


def analyze_log(logfile):

    stats = defaultdict(lambda: {
        "requests": 0,
        "2xx": 0,
        "3xx": 0,
        "4xx": 0,
        "5xx": 0,
        "last_seen": None,
        "last_status": None
    })

    parsed_lines = 0
    skipped_lines = 0

    with open(
        logfile,
        "r",
        encoding="utf-8",
        errors="replace"
    ) as file:

        for line in file:

            match = LOG_PATTERN.match(line)

            if not match:
                skipped_lines += 1
                continue

            row = match.groupdict()

            # This is only a User-Agent filter.
            # It does NOT authenticate Googlebot.

            if "Googlebot" not in row["ua"]:
                continue

            try:
                request_time = parse_timestamp(row["time"])
            except ValueError:
                skipped_lines += 1
                continue

            parsed_url = urlsplit(row["path"])

            url = parsed_url.path

            if parsed_url.query:
                url += "?" + parsed_url.query

            status = int(row["status"])

            item = stats[url]

            item["requests"] += 1

            if 200 <= status < 300:
                item["2xx"] += 1

            elif 300 <= status < 400:
                item["3xx"] += 1

            elif 400 <= status < 500:
                item["4xx"] += 1

            elif 500 <= status < 600:
                item["5xx"] += 1

            if (
                item["last_seen"] is None
                or request_time > item["last_seen"]
            ):
                item["last_seen"] = request_time
                item["last_status"] = status

            parsed_lines += 1

    return stats, parsed_lines, skipped_lines


def export_csv(stats, output_file):

    with open(
        output_file,
        "w",
        newline="",
        encoding="utf-8-sig"
    ) as file:

        writer = csv.writer(file)

        writer.writerow([
            "URL",
            "Requests",
            "2xx",
            "3xx",
            "4xx",
            "5xx",
            "Last Seen",
            "Last Status"
        ])

        sorted_items = sorted(
            stats.items(),
            key=lambda item: item[1]["requests"],
            reverse=True
        )

        for url, data in sorted_items:

            writer.writerow([
                url,
                data["requests"],
                data["2xx"],
                data["3xx"],
                data["4xx"],
                data["5xx"],
                data["last_seen"].isoformat(),
                data["last_status"]
            ])


def main():

    parser = argparse.ArgumentParser(
        description="Analyze crawler requests in access logs"
    )

    parser.add_argument(
        "logfile",
        help="Path to the access log"
    )

    parser.add_argument(
        "--output",
        default="googlebot_report.csv",
        help="Output CSV filename"
    )

    args = parser.parse_args()

    stats, parsed, skipped = analyze_log(args.logfile)

    export_csv(stats, args.output)

    print("Analysis completed")
    print("Parsed crawler requests:", parsed)
    print("Unique URLs:", len(stats))
    print("Skipped lines:", skipped)
    print("Report:", args.output)


if __name__ == "__main__":
    main()

6. Running the analyzer

Run the following command:

python googlebot_analyzer.py access.log

The script will generate:

googlebot_report.csv

The CSV file can be opened in Excel, LibreOffice, or imported into another analysis tool.

For larger projects, I would also consider processing rotated log files and combining logs from multiple web servers.

7. What does the report tell us?

A simplified example of the resulting report:

URL Requests 2xx 3xx 4xx 5xx
/catalog/ 1 1 0 0 0
/catalog/?page=2 1 1 0 0 0
/product/product-103/ 2 1 1 0 0
/product/product-104/ 1 0 0 1 0
/product/product-105/ 2 1 0 0 1

Now we can identify several things worth investigating.

Product 103: redirect

The URL returned HTTP 301 and was requested again with HTTP 200 in our sample.

This is not necessarily a problem. However, if redirects form long chains, I would investigate whether the internal links point directly to the final destination.

Product 104: 404

The server returned HTTP 404.

If the product was permanently removed, this may be the correct response. If the product is still available, the routing or application logic needs investigation.

Product 105: 503 followed by 200

This is more interesting.

The server temporarily returned HTTP 503, but a later request succeeded.

I would check whether the issue is related to application availability, backend performance, rate limiting, or infrastructure instability.

A single 503 response does not prove a crawling problem. Repeated patterns and their timing are much more useful.

8. What I would add next

This script is intentionally small. It is a starting point, not a complete crawler analytics platform.

For a more advanced version, I would add:

  • Reverse DNS and forward DNS verification of Googlebot IP addresses.
  • IPv6 support.
  • Sitemap parsing.
  • Internal URL inventory from a website crawl.
  • Detection of URLs missing from internal links.
  • Redirect-chain analysis.
  • Response-time analysis.
  • Daily crawling trends.
  • HTML and CSV reports.
  • Filtering by Googlebot type.

The most useful next step would be combining three datasets:

  1. URLs listed in XML Sitemap.
  2. URLs discovered through internal links.
  3. URLs requested by verified Googlebot.

This makes it possible to identify pages that are listed in Sitemap but have no discovered internal links or no recorded crawler requests during the selected period.

That does not prove Google has never discovered them. It gives us a list of URLs that deserve further investigation.

9. Why I use server logs in technical SEO audits

I don't consider server logs a replacement for Google Search Console or a website crawler.

Each tool answers a different question.

Tool What it helps investigate
Screaming Frog Internal link structure and URL discovery
XML Sitemap URLs submitted to search engines
Server access logs Actual HTTP requests received by the server
Google Search Console Google's reported crawling and indexing information
Python scripts Custom analysis and combining datasets

Combining these sources provides a much better understanding of how a website is exposed to search engine crawlers.

If you're interested in the broader technical side of crawling architecture, I have also written about crawl budget and Googlebot behavior here:

Crawl Budget Waste: How Googlebot Spends a Website's Crawl Budget — SEO PROF 74

Conclusion

Server access logs are one of the most useful sources of information when investigating crawling problems.

They help answer questions that cannot always be answered through a browser or a standard SEO crawler:

  • Which URLs did the bot request?
  • What responses did the server return?
  • Are there recurring HTTP errors?
  • Are unnecessary URL variations consuming requests?
  • Are important pages receiving crawler visits?

The Python script in this tutorial is deliberately simple, but it provides a useful foundation for more advanced crawling diagnostics.

Uploading image

My main takeaway: don't assume that a page is accessible to search engines just because it opens in your browser. Look at the actual requests, server responses, internal links, and indexing data.

That's where technical investigations become much more interesting.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.