Access Log Analysis: Rank IPs by Risk, Not Volume

Every access log analysis tool I've reached for ranks IPs the same way: by request count. GoAccess does it. The browser-based nginx log parsers do it. So does every awk | sort | uniq -c | sort -rn one-liner that has ever been pasted into a runbook.

That ranking answers "who is noisy." It does not answer "who is dangerous." In a real log, those are two different lists, and the gap between them is the whole problem.

TL;DR

• Request count ranks your crawlers and your monitoring, not your attackers.
• Two independent signal families exist: what an IP is (reputation) and what it did to you (behaviour). Neither works alone.
• Bulk reputation lookups are one POST for up to 50,000 addresses, so enriching a whole log is cheap if you cache.
• The step nobody implements is subtraction: verified crawlers and corporate gateway egress IPs need their scores pushed down, hard.
• Every point in the score should carry a reason string, or you can't tune it and you can't defend it.

Sorting by hits gives you a list of noisy addresses. Joining reputation data to behaviour you measured yourself gives you a list of dangerous ones. The second list is shorter, more useful, and about forty lines of Python away from the first.

What "top offender" actually ranks

Here are four addresses from a sample log, sorted the usual way. The hit counts are from a synthetic log. The reputation column is real, and you can check every value in the API docs.

The top of that list is an entire company's workforce sharing one egress address. Second place is a crawler you probably want indexing you. The two entries you'd actually want to look at are at the bottom, and on a real log with a few thousand unique IPs they're on page three.

Worse, the naive fix makes it worse. Block the top offender and you've locked out every employee at whatever company routes through that Zscaler IP. Block the runner-up and you've told an AI crawler your site is gone. Both of those are quiet failures. Nothing in your log says "you just banned a customer's head office."

Two signals, and why neither works on its own

What the IP is

This is IP reputation data, and it comes from outside your system. Is the address a Tor exit node, a residential proxy, a VPN endpoint, a known attacker, a declared crawler, a shared corporate gateway?

You've got real options here: AbuseIPDB for community-reported abuse, IPGeolocation, IPQualityScore, Spur, MaxMind's Anonymous IP database, IPLocate. I used IPGeolocation's IP Security API for this because of two fields that matter specifically for log work, is_known_good_bot and is_corporate_gateway, plus a bulk endpoint that takes 50,000 addresses in one request. The second field is what makes the subtraction step below possible. I haven't found the same flag anywhere else, though I'd be glad to be wrong about that.

A single lookup, so you can see the whole shape:

curl -s 'https://api.ipgeolocation.io/v3/security?apiKey=API_KEY&ip=87.58.66.106'

{
  "ip": "87.58.66.106",
  "security": {
    "threat_score": 5,
    "is_tor": false,
    "is_proxy": false,
    "proxy_provider_names": [],
    "proxy_confidence_score": 0,
    "proxy_last_seen": "",
    "is_residential_proxy": false,
    "is_vpn": false,
    "vpn_provider_names": [],
    "vpn_confidence_score": 0,
    "vpn_last_seen": "",
    "is_relay": false,
    "relay_provider_name": "",
    "is_anonymous": false,
    "is_known_attacker": false,
    "is_bot": false,
    "bot_confidence_score": 0,
    "bot_operator_name": "",
    "bot_type": "",
    "is_known_good_bot": false,
    "bot_last_seen": "",
    "is_spam": false,
    "is_cloud_provider": true,
    "cloud_provider_name": "Zscaler Switzerland GmbH",
    "is_corporate_gateway": true,
    "corporate_gateway_type": "secure_web_gateway",
    "corporate_gateway_provider_name": "Zscaler"
  }
}

Note is_anonymous is false on that one even though hundreds of people share the address. That's correct: a corporate gateway isn't an anonymity service. The range is published by the vendor and belongs to an identifiable company. Treating it like a VPN is exactly the mistake this article is about.

Here's how I read the flags when deciding what to do:

That bot_type row has a trap in it. The value scanner appears in both the good and the bad list, because Censys and a reconnaissance script do literally the same thing. is_known_good_bot is the field that separates them, not bot_type.

What the IP did to you

Reputation alone is not enough, for two reasons that pull in opposite directions.

It over-flags. Plenty of ordinary people browse through a VPN. If is_vpn is your blocking rule, you've built a rule that fires on privacy-conscious customers and misses anyone attacking you from a clean residential address.

It also under-flags. An address with a threat score of zero that just made 200 failed SSH attempts against your box in four minutes is dangerous right now, whatever a global feed thinks of it. Your log knows something the feed doesn't.

So you measure behaviour yourself: failed authentication attempts, requests to paths that only a scanner would ask for, the ratio of 4xx and 5xx responses, 404 volume, request rate. None of that needs an API. It's all sitting in the file already.

Parse first, spend second

Parse before you enrich, for a boring but real reason: you pay per address, so you want the deduplicated set of public IPs, not every line.

import ipaddress
import re
from collections import defaultdict

COMBINED = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[[^\]]+\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+)[^"]*" (?P<status>\d{3})'
)

SENSITIVE = ("/wp-login.php", "/.env", "/.git/", "/phpmyadmin", "/xmlrpc.php", "/admin")


def is_public(value: str) -> bool:
    """Private and bogon addresses get rejected with HTTP 423, so drop them here."""
    try:
        addr = ipaddress.ip_address(value)
    except ValueError:
        return False
    return not (
        addr.is_private or addr.is_loopback or addr.is_reserved
        or addr.is_multicast or addr.is_link_local or addr.is_unspecified
    )

The standard library settles the whole bogon question for you. Private, loopback, reserved, multicast, link-local and unspecified addresses are gone before they cost anything, which matters because the API charges 2 credits per valid address and rejects these with a 423 anyway.

Then accumulate per address:

def parse(path: str) -> dict[str, dict]:
    activity: dict[str, dict] = defaultdict(
        lambda: {"hits": 0, "errors": 0, "not_found": 0, "sensitive": 0, "auth_failures": 0}
    )
    try:
        handle = open(path, "r", encoding="utf-8", errors="replace")
    except OSError as exc:
        raise SystemExit(f"Cannot read {path}: {exc}") from exc

    with handle:
        for line in handle:
            match = COMBINED.match(line)
            if not match:
                continue  # unknown format or a truncated line; never kill the run over one line
            ip = match.group("ip")
            if not is_public(ip):
                continue  # saves 2 credits per address and avoids a guaranteed error response
            status = int(match.group("status"))
            row = activity[ip]
            row["hits"] += 1
            row["errors"] += status >= 400
            row["not_found"] += status == 404
            row["sensitive"] += any(p in match.group("path").lower() for p in SENSITIVE)
            if status in (401, 403):
                row["auth_failures"] += 1
    return dict(activity)

Forty-ish lines and you have per-IP behaviour for a combined-format log. Point it at auth.log instead and the shape of the output is identical, only the regex and the failure condition change.

Pitfall: if your app sits behind a load balancer or Cloudflare, match.group("ip") is your proxy, not your visitor. You want X-Forwarded-For, and specifically the left-most entry in the chain with any port suffix stripped. But only trust that header if your own proxy sets it. If anything upstream is reachable directly, a client can forge the entire chain and you will happily score an address that never sent you a packet. Getting this backwards is how people end up blocking their own CDN.

The full version of this on GitHub handles combined, common, JSON lines from nginx, Caddy, Traefik and CloudFront, plus auth.log, with format auto-detection and gzip support. The version above is the shape of it, not the whole thing.

Enrich in bulk without burning credits

Heads up: the Security API is not on the free plan. Both /v3/security and /v3/security-bulk return 401 for a free key. On a free tier you still get location and ASN data from /v3/ipgeo, but no threat signals, and nothing below this line will run. Better to know that now than forty minutes in.

Two things make the cost sane. One request handles up to 50,000 addresses. And lookups cost 2 credits per valid IP, with bogon, private and malformed addresses not counted at all, which is why filtering them locally is worth doing anyway.

import os
import time

import requests

SECURITY_BULK = "https://api.ipgeolocation.io/v3/security-bulk"
CHUNK = 500  # well under the 50,000 ceiling; smaller batches fail smaller


def enrich(ips: list[str], timeout: float = 20.0, retries: int = 3) -> tuple[dict, int]:
    """Look up security data for many IPs. Returns (by_ip, credits_charged)."""
    key = os.environ.get("IPGEOLOCATION_API_KEY")
    if not key:
        raise SystemExit("Set IPGEOLOCATION_API_KEY in your environment or .env file")

    results: dict[str, dict] = {}
    charged = 0

    for start in range(0, len(ips), CHUNK):
        batch = ips[start:start + CHUNK]
        for attempt in range(retries + 1):
            try:
                response = requests.post(
                    SECURITY_BULK,
                    params={"apiKey": key},
                    json={"ips": batch},
                    headers={"Content-Type": "application/json"},  # 415 without this
                    timeout=timeout,
                )
            except requests.RequestException:
                if attempt == retries:
                    raise
                time.sleep(1.5 ** attempt)
                continue

            if response.status_code == 401:
                raise SystemExit("401: the Security API requires a paid plan")
            if response.status_code == 429 and attempt < retries:
                time.sleep(float(response.headers.get("Retry-After", 1.5 ** attempt)))
                continue
            response.raise_for_status()

            charged += int(response.headers.get("X-Credits-Charged", 0))
            for entry in response.json():
                ip = entry.get("ip")
                if not ip:
                    continue  # bogon that slipped through: {"message": "... is a bogon IP address."}
                results[ip] = entry.get("security", {})
            break

    return results, charged

A few things in there are worth the keystrokes. The Content-Type header is not optional on the bulk endpoint, which will tell you so with a 415. Invalid entries come back as an object with only a message key and no ip, so iterate defensively. And X-Credits-Charged is how you find out what a run actually cost, which is the number you'll want in front of you the first time you point this at a month of logs.

You can also trim the payload with fields=security.threat_score,security.is_bot and friends if you only need a few flags. I don't bother, because the flags I'd have dropped are the ones the subtraction step needs.

Cache, or the second run costs the same as the first

Reputation data does not change minute to minute. A day-old answer for an address you saw yesterday is fine, and SQLite is plenty for this.

Give the table an IP, a JSON blob and a fetched_at timestamp, read anything newer than your TTL straight from disk, and only send the misses to the API. A 24-hour TTL is a reasonable default. If you're running hourly from cron, the cache is the difference between a bill and a rounding error.

The subtraction step nobody writes about

Every nginx log analyzer I looked at adds points. None of them take points away. That's the gap.

Two categories need their scores pushed down, and they need it for different reasons.

Verified good bots. Googlebot, Bingbot, the uptime monitor you're paying for, the security scanner you configured, the AI crawlers you decided to allow. These generate request volume that looks exactly like scraping, because it is scraping, with an operator who publishes their ranges and says so. is_known_good_bot comes back true with bot_operator_name filled in. I apply a credit of -45, which is enough to bury a crawler under anything genuinely interesting even when it's the noisiest thing in the file.

One condition on that credit: only apply it when is_known_attacker is false. A declared operator whose address later shows up in attack traffic should not get a permanent pass.

Corporate gateways. This is the one that will bite you. Enterprise secure web gateways like Zscaler and Netskope proxy employee browser traffic through the vendor's cloud and forward it to you from the vendor's address space. One IP, an entire workforce behind it. is_corporate_gateway is true, corporate_gateway_type tells you whether it's a standard gateway or remote browser isolation, and corporate_gateway_provider_name names the vendor.

The credit I apply is only -10, and the small number is deliberate. A gateway IP is not automatically safe. Somebody inside that company can still attack you through it. What the flag actually buys you is context: high request volume from a gateway is a hundred people reading your docs, three failed logins is somebody mistyping their password, and blocking the address is a support ticket from an enterprise customer asking why your site is down for their whole office.

Per-IP rate limits are the other thing to revisit. Whatever threshold you set for one human is wrong by two or three orders of magnitude for an address fronting a company.

Score it, and show your work

Start from the API's threat score, adjust with what you measured, clamp to 0 through 100. That final number is the IP risk score you sort on. Every adjustment appends a reason string, which is the difference between a tool you can tune and a number you have to trust.

Then bucket it: 80 and up is critical, 60 is high, 40 is medium, 20 is low, anything below that is informational.

ATTACK_BOTS = {"exploit", "credential_stuffing", "brute_force", "worm"}


def score(activity: dict, sec: dict) -> tuple[int, list[str]]:
    """Blend the API's view of the IP with what it did to you."""
    points = float(sec.get("threat_score", 0))
    reasons: list[str] = []

    def add(delta: int, why: str) -> None:
        nonlocal points
        points += delta
        reasons.append(f"{delta:+d}  {why}")

    if sec.get("is_known_attacker"):
        add(35, "Flagged as a known attacker")
    if sec.get("is_bot") and sec.get("bot_type") in ATTACK_BOTS:
        add(25, f"Attack bot activity: {sec['bot_type']}")
    elif sec.get("is_bot") and not sec.get("is_known_good_bot"):
        add(8, f"Undeclared bot ({sec.get('bot_type') or 'unknown type'})")
    if sec.get("is_tor"):
        add(15, "Tor exit node")
    if sec.get("is_spam"):
        add(12, "Associated with spam activity")
    if sec.get("is_anonymous") and not sec.get("is_tor"):
        add(10, "Anonymised connection")  # not double-counted on top of Tor
    if sec.get("is_residential_proxy"):
        add(8, "Residential proxy network")

    failures = activity.get("auth_failures", 0)
    if failures >= 5:
        add(40, f"{failures} failed or denied auth attempts")
        if failures >= 20:
            add(10, "Sustained brute force")
    if activity.get("sensitive"):
        add(30, f"{activity['sensitive']} requests to sensitive paths")
        if activity["sensitive"] >= 10:
            add(10, "Sustained scanner activity")
    hits = activity.get("hits", 0)
    if hits >= 10 and activity.get("errors", 0) / hits >= 0.5:
        add(15, "Over half of requests returned 4xx or 5xx")
    if activity.get("not_found", 0) >= 25:
        add(10, f"{activity['not_found']} x 404, looks like path enumeration")

    # The subtraction. Order doesn't matter; the fact that it happens at all does.
    if sec.get("is_known_good_bot") and not sec.get("is_known_attacker"):
        add(-45, f"Verified {sec.get('bot_operator_name') or 'good bot'}, de-prioritised")
    if sec.get("is_corporate_gateway"):
        provider = sec.get("corporate_gateway_provider_name") or "Corporate"
        add(-10, f"{provider} gateway egress, shared by many users, do not block")

    return max(0, min(100, round(points))), reasons

Run that over the four addresses from the opening table and the order inverts. The Zscaler IP with 85 hits lands at 0. The ChatGPT crawler lands at 0. The residential proxy that also carries a known-attacker and spam flag lands at 100 on reputation alone, before its behaviour is counted. The brute_force bot clears 100 as soon as its failed logins are added.

Every weight above lives in one file in the repo, which is on purpose. You will want to change them. The auth-failure weight in particular is tuned for a box that gets hammered on SSH, and if yours doesn't, 40 points is too aggressive.

Two things the snippet skips that the repo does: request rate, which needs timestamps that my regex above throws away, and the --no-trust-good-bots escape hatch for when you genuinely do want to see crawlers ranked with everything else.

Turn findings into something that acts

A ranked list you read on Tuesday is a report. A ranked list that does something is a tool.

def nginx_blocklist(findings: dict, minimum: int = 60) -> str:
    """Emit deny rules for HIGH and above. Candidates, not a deploy artifact."""
    lines = ["# generated by log triage, review before applying"]
    for ip, result in sorted(findings.items(), key=lambda kv: -kv[1]["score"]):
        if result["score"] < minimum:
            continue
        if result["security"].get("is_corporate_gateway"):
            continue  # belt and braces: never emit a shared office egress address
        top_reason = result["reasons"][0] if result["reasons"] else "no reason recorded"
        lines.append(f"deny {ip};  # {result['score']}/100 {top_reason}")
    return "\n".join(lines)

That corporate gateway check is redundant, because the -10 credit plus a low threat score usually keeps those addresses well under 60 anyway. I left it in because "usually" is doing a lot of work in that sentence, and the failure mode is locking an enterprise customer out of your product.

Past that, the useful outputs are a non-zero exit code so cron or CI can page you when something crosses a threshold, and NDJSON so your SIEM can ingest findings without a custom parser. An hourly cron entry with --fail-on high covers most of what a small team needs.

If you already run fail2ban, this isn't a fail2ban alternative and I wouldn't pitch it as one. fail2ban reacts in seconds to a pattern it already recognises, in-process, with no network call. This runs on a schedule, costs money per address, and answers a different question: out of everything that hit you yesterday, which handful deserves a human looking at it. The two compose fine. fail2ban handles the obvious, this finds the quiet 16-hit address that never tripped a threshold.

What this will not do

Worth being straight about the limits, because IP-level scoring gets oversold.

Shared addresses are a real problem, and corporate gateways are only the visible half of it. Carrier-grade NAT puts thousands of mobile users behind one address: a university, a hotel, a coffee shop: same shape. A high score on a shared address means somebody behind it is a problem, not that everyone behind it is.

Scores expire. Residential proxy pools rotate. An address that was laundering traffic last week can be a normal home connection this week, and the reverse happens just as often. Cache for a day, not a quarter, and treat a blocklist as something you regenerate rather than something you accumulate.

IPv6 changes the unit. A single /64 is one customer's entire address space. Scoring individual v6 addresses will produce a very long list of singletons and tell you nothing. You want to aggregate at the prefix.

This is triage, not prevention. It reads logs after the requests already happened. It is not a WAF, it is not rate limiting, and it does not replace either. What it gives you is a short list of addresses worth a human decision, which is the thing you don't have when your only sort key is request count.

Run it against last week's log before you wire it into anything. The first run mostly tells you how boring your traffic is, which is useful calibration, and whether your thresholds are anywhere near right for your box. Set --fail-on high, let it page you for a month, and resist auto-applying the blocklist until you've read enough of them to trust it. iplogsec is on GitHub if you want the version with format detection, the SQLite cache and the HTML report already written.