Isolation Forest Anomaly Detection on Login Logs in Python: Thresholds, Explanations, Drift

Stolen passwords just lost first place. Verizon’s 2026 Data Breach Investigations Report says 31% of breaches now start with software vulnerabilities, “beating stolen passwords as the top way attackers get in.” The same report still lists stolen credentials among the most frequent causes of breaches, and those attacks leave a trail in one boring table: the authentication log.

The pitch for machine learning here is familiar. Signature rules only catch what someone already wrote a rule for, and an unsupervised model learns “normal” and flags the rest. I wanted to see what that looks like on login data with real numbers, so I built the whole loop with Isolation Forest anomaly detection in Python and scikit-learn: a synthetic auth log with labeled attacks, per-user features, training, threshold selection, evaluation against a hand-written rule baseline, per-alert explanations, and a drift scenario that forces a retrain.

Short version of the results: the rules won on precision (0.906 vs 0.426), the forest won on recall (0.894 vs 0.780), and the forest was the only one that caught any of the quietest attack. Every number below comes from a real run of the anomaly detection Python code in this post.

What we’ll build:

1. A log generator with a fixed seed and four labeled attack types

2. Per-user, per-hour features that only look backward in time

3. An Isolation Forest and three ways to set its threshold

4. Precision, recall and PR-AUC, plus a comparison with SIEM-style rules

5. “Why did this fire?” explanations for single alerts

6. A drift week that breaks the threshold, and two ways to fix it

Stack: Python 3.13.18, scikit-learn 1.9.1, pandas 3.0.5, numpy 2.5.3, pytest 9.1.1. The full experiment runs in well under a minute on 2 CPUs.

What the attacks look like in an auth log

Before choosing among anomaly detection algorithms, it helps to write down what each attack leaves behind. These are the four I generate, mapped to MITRE ATT&CK where it applies:

OWASP describes credential stuffing as “the automated injection of stolen username and password pairs” and notes it works because people reuse passwords. MITRE’s detection guidance for T1110.004 starts where you’d expect: rapid failures across different users from one IP. Identity providers ship their own versions of the travel signal: Microsoft Entra ID Protection, for example, has both “atypical travel” and “impossible travel” risk detections. The fourth attack is there to keep the experiment honest. A careful attacker spreads attempts across residential proxies and stays under every per-IP and per-account threshold.

Pipeline from raw login events to hourly per-user windows, ten features, Isolation Forest score, alert-budget threshold and an explained alert, with a retraining loop

Step 1. Generate auth logs with labeled attacks

Real labeled login data is hard to get and impossible to share, so the generator does three things on purpose:

• Normal users have habits. Each of 400 users has a home city, 1-3 home IPs, 1-2 devices, a typical login hour and a login rate. About 8% of logins come from a mobile IP and 4% start with a mistyped password.

• Normal users also do odd things. Some take business trips, buy new phones or forget their password and fail 4-8 times in a row. These are the false positives any detector has to live with.

• The training period is not clean. Days 1-21 contain a quarter-scale copy of the attacks, unlabeled, because your historical logs aren’t clean either.

The attack injector is the interesting part:

def attacks(rng, users, day_from, day_to, scale=1.0):
    rows = []
    n = lambda k: max(1, int(round(k * scale)))
    # 1) Credential stuffing: a few datacenter IPs, many users, 1-2 tries each, rare success.
    for _ in range(n(2)):
        t0 = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=float(rng.uniform(0, 24)))
        ips = [_ip(rng, "45.95") for _ in range(3)]
        for u in rng.choice(users, size=n(40), replace=False):
            ts = t0 + pd.Timedelta(minutes=float(rng.uniform(0, 50)))
            for _ in range(int(rng.integers(1, 3))):
                ok = rng.random() < 0.05
                rows.append(_event(ts, u["user"], rng.choice(ips), "Frankfurt", "bot-" + str(rng.integers(1e6)),
                                   16509, ok, "credential_stuffing"))
                if ok:
                    break
    # 2) Brute force -> takeover: many failures on one account, then success from a new device abroad.
    for u in rng.choice(users, size=n(6), replace=False):
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=float(rng.uniform(0, 24)))
        city, ip = rng.choice(ATTACK_CITIES), _ip(rng, "91.200")
        for k in range(int(rng.integers(10, 40))):
            rows.append(_event(ts + pd.Timedelta(seconds=15 * k), u["user"], ip, city, "atk-dev", 9123, False, "brute_force"))
        rows.append(_event(ts + pd.Timedelta(minutes=12), u["user"], ip, city, "atk-dev", 9123, True, "brute_force"))
    # 3) Impossible travel: phished password used from another continent, no failures at all.
    for u in rng.choice(users, size=n(10), replace=False):
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=(u["peak_hour"] + 1.5) % 24)
        rows.append(_event(ts - pd.Timedelta(hours=1), u["user"], u["ips"][0], u["city"], u["devices"][0], u["asn"], True, "normal"))
        rows.append(_event(ts, u["user"], _ip(rng, "102.89"), rng.choice(ATTACK_CITIES), "atk-" + u["user"],
                           36873, True, "impossible_travel"))
    # 4) Low-and-slow stuffing via residential proxies in the victim's own city: one try per user.
    for u in rng.choice(users, size=n(25), replace=False):
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=(u["peak_hour"] + rng.normal(0, 2)) % 24)
        rows.append(_event(ts, u["user"], _ip(rng, "73." + str(rng.integers(50, 99))), u["city"], u["devices"][0],
                           u["asn"], False, "low_and_slow"))
    return rows

«Full generator (users, normal traffic, benign oddities)»:

def make_users(rng, n_users):
    users = []
    for u in range(n_users):
        city = rng.choice(HOME_CITIES, p=[.22, .18, .14, .14, .1, .12, .1])
        users.append({
            "user": f"u{u:04d}",
            "city": city,
            "ips": [_ip(rng, f"73.{rng.integers(0, 50)}") for _ in range(rng.integers(1, 4))],
            "devices": [f"d{u:04d}{k}" for k in range(rng.integers(1, 3))],
            "asn": int(rng.choice([7922, 701, 20115, 5650, 2856, 3320])),
            "peak_hour": float(rng.normal(14, 2.5)),   # UTC
            "rate": float(rng.uniform(0.8, 5.0)),      # logins per weekday
        })
    return users

def normal_traffic(rng, users, days, day0=0):
    rows = []
    for u in users:
        for d in range(day0, day0 + days):
            weekend = (START + pd.Timedelta(days=d)).dayofweek >= 5
            for _ in range(rng.poisson(u["rate"] * (0.3 if weekend else 1.0))):
                hour = (u["peak_hour"] + rng.normal(0, 2.0)) % 24
                ts = START + pd.Timedelta(days=d, hours=hour)
                ip = rng.choice(u["ips"]) if rng.random() > 0.08 else _ip(rng, "172.56")  # mobile
                dev = rng.choice(u["devices"])
                if rng.random() < 0.04:  # typo, then success
                    rows.append(_event(ts, u["user"], ip, u["city"], dev, u["asn"], False, "normal"))
                    ts += pd.Timedelta(seconds=int(rng.integers(5, 60)))
                rows.append(_event(ts, u["user"], ip, u["city"], dev, u["asn"], True, "normal"))
    return rows

def benign_oddities(rng, users, day_from, day_to):
    """Legit behavior that *looks* suspicious: trips, new phones, forgotten passwords."""
    rows = []
    for u in rng.choice(users, size=max(1, len(users) // 25), replace=False):   # business trips
        d = int(rng.integers(day_from, day_to))
        dest = rng.choice([c for c in HOME_CITIES if c != u["city"]] + ["Singapore", "Warsaw"])
        for k in range(3):
            ts = START + pd.Timedelta(days=d + k, hours=float(rng.uniform(7, 20)))
            rows.append(_event(ts, u["user"], _ip(rng, "185.20"), dest, u["devices"][0], 9009, True, "normal", "trip"))
    for u in rng.choice(users, size=max(1, len(users) // 30), replace=False):   # new phone
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=u["peak_hour"] % 24)
        rows.append(_event(ts, u["user"], u["ips"][0], u["city"], f"{u['user']}-new", u["asn"], True, "normal", "new_phone"))
    for u in rng.choice(users, size=max(1, len(users) // 40), replace=False):   # forgot password
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=u["peak_hour"] % 24)
        for k in range(int(rng.integers(4, 9))):
            rows.append(_event(ts + pd.Timedelta(seconds=20 * k), u["user"], u["ips"][0], u["city"],
                               u["devices"][0], u["asn"], False, "normal", "forgot_password"))
    return rows

def generate(seed=42, n_users=400, train_days=21, test_days=7):
    """Days [0, train_days) are 'history' (with a little unlabeled attack noise), the rest is test."""
    rng = np.random.default_rng(seed)
    users = make_users(rng, n_users)
    total = train_days + test_days
    rows = normal_traffic(rng, users, total)
    rows += benign_oddities(rng, users, 0, total)
    rows += attacks(rng, users, 3, train_days, scale=0.25)        # contamination nobody labeled
    rows += attacks(rng, users, train_days, total)
    df = pd.DataFrame(rows).sort_values("ts", kind="stable").reset_index(drop=True)
    df["is_attack"] = df["label"] != "normal"
    return df, users
python 3.13.18, scikit-learn 1.9.1, pandas 3.0.5, numpy 2.5.3
events: 27,481  users: 400
train windows (days 1-21): 16,023  unlabeled attack windows hiding inside: 20
test windows  (days 22-28): 5,403
label
normal                 5280
credential_stuffing      80
low_and_slow             25
impossible_travel        10
brute_force               8

The test week has 123 attack windows out of 5,403, a base rate of 2.3%. Keep that number in mind: a detector that flags everything gets precision 0.023.

Step 2. Feature engineering per user and time window

This isn’t classic time series anomaly detection on a single metric. Each row is one user in one clock hour, and every feature compares that hour with what the user did before it. Getting the “before” right matters more than the model choice.

def build_windows(events: pd.DataFrame, history: dict | None = None):
    """Aggregate events into (user, 1h) windows. Returns (windows, history) so the
    per-user state can be carried into the next batch (train -> test -> drift)."""
    ev = events.copy()
    ev["window"] = ev["ts"].dt.floor("1h")
    # Network-side feature: how many distinct users each source IP touched in that hour.
    fanout = ev.groupby(["ip", "window"])["user"].nunique().rename("ip_users")
    ev = ev.join(fanout, on=["ip", "window"])

    ev = ev.sort_values(["window", "user", "ts"], kind="stable")
    hist = history if history is not None else {}
    out = []
    recs = ev.to_dict("records")
    i = 0
    while i < len(recs):
        j = i
        while j < len(recs) and recs[j]["window"] == recs[i]["window"] and recs[j]["user"] == recs[i]["user"]:
            j += 1
        g, first = recs[i:j], recs[i]
        window, user = first["window"], first["user"]
        h = hist.setdefault(user, {"devices": set(), "countries": set(), "asns": set(),
                                   "hours": np.ones(24), "last": None})
        # Geo-velocity for every event against the previous *successful* login, including
        # successes earlier in the same window. My first version checked only the first event
        # of the window and read 0 km/h when the attacker logged in right after the real user.
        speed, prev = 0.0, h["last"]
        for e in g:
            if prev is not None:
                km = haversine_km(prev[1], prev[2], e["lat"], e["lon"])
                hours = max((e["ts"] - prev[0]).total_seconds() / 3600, 0.25)  # 15-min floor for IP-geo noise
                speed = max(speed, km / hours)
            if e["success"]:
                prev = (e["ts"], e["lat"], e["lon"])
        fails = [not e["success"] for e in g]
        attack = [e["label"] for e in g if e["is_attack"]]
        seen = lambda key, known: int(bool(known) and any(e[key] not in known for e in g))
        out.append({
            "window": window, "user": user,
            "n_attempts": len(g),
            "n_fail": sum(fails),
            "fail_ratio": sum(fails) / len(g),
            "n_ips": len({e["ip"] for e in g}),
            "new_device": seen("device", h["devices"]),
            "new_country": seen("country", h["countries"]),
            "new_asn": seen("asn", h["asns"]),
            "geo_speed_kmh": speed,
            "hour_rarity": float(1 - h["hours"][window.hour] / h["hours"].sum()),
            "ip_fanout": max(e["ip_users"] for e in g),
            "is_attack": bool(attack),
            "label": attack[0] if attack else "normal",
            "note": next((e["note"] for e in g if e["note"]), ""),
        })
        # Update state only from *successful* logins: failed attempts must not teach the
        # model that the attacker's device or country is "known".
        ok = [e for e in g if e["success"]]
        for e in ok:
            h["devices"].add(e["device"]); h["countries"].add(e["country"]); h["asns"].add(e["asn"])
        h["hours"][window.hour] += len(ok)
        if ok:
            h["last"] = (ok[-1]["ts"], ok[-1]["lat"], ok[-1]["lon"])
        i = j
    return pd.DataFrame(out), hist

Three decisions in there are worth calling out:

• State updates only from successful logins. If failed attempts updated “known devices,” a brute-force attacker would teach the model their own device during the failures and look familiar by the time they succeed. There’s a test for this.

• Geo-velocity is checked for every event in the window. My first version only checked the first event of each hour. When the attacker logged in right after the real user in the same hour, it compared the user with themselves and returned 0 km/h. The fixed version reports 21,034 km/h for that window.

• ip_fanout is the one network-level feature. It counts how many distinct users a source IP touched in that hour. Network anomaly detection usually starts from exactly this kind of cross-account view, and it’s what MITRE recommends for stuffing.

Heavy-tailed columns go through log1p before training:

def to_matrix(windows: pd.DataFrame) -> np.ndarray:
    """Heavy-tailed columns go through log1p (Isolation Forest draws split values uniformly
    between a node's min and max). In this dataset it barely moved PR-AUC; see gotchas."""
    X = windows[FEATURES].astype(float).copy()
    for c in ["n_attempts", "n_fail", "n_ips", "geo_speed_kmh", "ip_fanout"]:
        X[c] = np.log1p(X[c])
    return X.to_numpy()

Step 3. Train the Isolation Forest

The idea behind Isolation Forest, from Liu, Ting and Zhou’s 2008 paper, is that anomalies are “few and different.” Each tree picks a random feature and a random split value between that feature’s min and max, and keeps splitting. Rare points get isolated after a few splits, so their average path length across the forest is short. The scikit-learn docs put it as “noticeably shorter paths for anomalies.” The paper used 256-point subsamples and 100 trees. I use 300 trees for steadier scores; max_samples=256 matches scikit-learn’s "auto" setting at this data size.

def train(X_train, contamination="auto", seed=7):
    return IsolationForest(n_estimators=300, max_samples=256, contamination=contamination,
                           random_state=seed).fit(X_train)

def anomaly_score(model, X):
    """Higher = more anomalous. score_samples() returns the opposite sign."""
    return -model.score_samples(X)

One sign convention trips people up: score_samples() returns the opposite of the paper’s score, and lower means more abnormal. I flip it once so that “higher = worse” everywhere else.

Step 4. Pick a threshold: contamination vs. score threshold

This is where most Isolation Forest tutorials stop at contamination=0.01 and move on. Here are four options on the same trained model:

contamination='auto'  offset_ = -0.5 | alerts= 1080  precision=0.114  recall=1.000  PR-AUC=0.873
contamination=0.01    offset_ = -0.695 | 1st percentile of train score_samples = -0.695
contamination=0.01    alerts=  151  precision=0.669  recall=0.821  PR-AUC=0.873
budget  5/day -> score>=0.705   alerts=  132  precision=0.750  recall=0.805  PR-AUC=0.873
budget 10/day -> score>=0.691   alerts=  162  precision=0.623  recall=0.821  PR-AUC=0.873
budget 20/day -> score>=0.652   alerts=  258  precision=0.426  recall=0.894  PR-AUC=0.873
budget 40/day -> score>=0.617   alerts=  352  precision=0.321  recall=0.919  PR-AUC=0.873

What the output says:

• contamination="auto" is not “let the model decide.” The scikit-learn docs say the offset is fixed at -0.5, as in the original paper. On this data that means 1,080 alerts in a week, a 20% alert rate. The paper itself warns that when every score sits near 0.5, there may be no clear anomalies, and with mostly discrete, low-variance features many normal windows score near that line.

• contamination=0.01 is just a quantile of the training scores. The fitted offset_ (-0.695) equals the 1st percentile of score_samples on the training set, to three decimals. It’s not a statement about how many attacks exist. It’s a statement about how many alerts you’ll get on data like your training data.

• So set it from your alert budget. If the SOC can triage 20 login alerts a day, compute the cut-off that produced 20/day on the training period and use that score threshold directly:
  

def threshold_for_budget(scores, alerts_per_day, n_days):
    """Score cut-off that yields a fixed number of alerts on a reference period."""
    k = int(alerts_per_day * n_days)
    return np.sort(scores)[::-1][k - 1]

At 20/day the model produced 258 alerts in the test week: about 21 false positives per day, which matches the budget, plus the attacks on top. PR-AUC doesn’t depend on any threshold, which is why it’s identical in every line.

Step 5. Evaluate with precision, recall and PR-AUC

With 2.3% positives, ROC-AUC can look great while most alerts are noise, so I report precision, recall and PR-AUC. scikit-learn’s average_precision_score computes it as a recall-weighted mean of precisions, which “is not interpolated” and avoids the optimism of the trapezoidal rule. The model scores 0.873 against a no-skill baseline of 0.023.

Precision-recall curve of the Isolation Forest on the test week, with operating points for contamination auto, contamination 0.01, the 20-alerts-per-day budget, and the rule baseline sitting above the curve

Step 6. Compare with a rule baseline

An anomaly detection machine learning model should beat what a security engineer writes in ten minutes. Here’s that ten-minute version:

def rule_baseline(w: pd.DataFrame) -> np.ndarray:
    """What a SIEM correlation rule would typically do."""
    return ((w["n_fail"] >= 5)
            | (w["geo_speed_kmh"] > 1000)
            | ((w["new_country"] == 1) & (w["new_device"] == 1))
            | (w["ip_fanout"] >= 10)).to_numpy()

Five failures in an hour is an alert, not a lockout. For comparison, NIST SP 800-63B only requires the verifier to cap consecutive failed attempts on an account at no more than 100.

rules             alerts=  106  precision=0.906  recall=0.780
iforest@20/day    alerts=  258  precision=0.426  recall=0.894
rules OR iforest  alerts=  267  precision=0.412  recall=0.894

attack windows caught, by type:
                     rules  iforest  total
label                                     
brute_force              8        8      8
credential_stuffing     78       80     80
impossible_travel       10       10     10
low_and_slow             0       12     25

false positives of iforest@20/day by cause:
note
plain normal    144
new_phone         3
trip              1
  of the plain-normal ones: 72% have 3+ logins in the hour, 81% contain a mistyped password
false positives of rules by cause: {'plain normal': 6, 'trip': 4}
  rule false positives where the user's previous successful login was the attacker's: 4

The rules win on precision by a wide margin: 96 of their 106 alerts are real. They catch every brute-force and impossible-travel window and 78 of 80 stuffing windows. The forest catches all three of those types too, but pays for it with 148 false positives. Its one real advantage is the bottom row: it caught 12 of the 25 low-and-slow attempts. The rules caught none.

The false positives are revealing. The forest’s top complaint is “busy hour with a typo”: 72% of its plain-normal false positives have 3+ logins in one hour and 81% include a mistyped password. That’s rare, so the forest flags it, but rare isn’t the same as malicious. Four of the rules’ ten false positives are more interesting: the victim’s own next login after a takeover. Their “last known location” was now the attacker’s, so coming home looked like impossible travel. Under the labels these count as false positives, but an analyst would want to see them.

Two caveats before anyone quotes “rules beat ML.” First, I wrote the rules and the generator, so the rules know exactly what the attacks look like. That’s the best case for rules, and real attackers don’t send you their source code. Second, rules OR iforest adds nothing here, because every attack the rules catch is already in the forest’s top 258. A more useful hybrid is rules for paging and the forest’s top-N as a daily review queue for what the rules can’t express.

Matrix of attack types against detectors: rules and Isolation Forest catch counts per attack type, plus the top false-positive patterns for each

Step 7. Explain an alert

An alert that says “score 0.784” gets ignored. The cheapest model-agnostic explanation I know is ablation: reset one feature at a time to its training median and measure how much the anomaly score drops.

def explain(model, X_train, x, top=3):
    """Ablation: reset one feature at a time to its training median and see how much the
    anomaly score drops. Big drop = this feature is what made the window look odd."""
    med = np.median(X_train, axis=0)
    base = anomaly_score(model, x[None, :])[0]
    probes = np.repeat(x[None, :], len(FEATURES), axis=0)
    probes[np.arange(len(FEATURES)), np.arange(len(FEATURES))] = med
    drops = base - anomaly_score(model, probes)
    order = np.argsort(drops)[::-1][:top]
    return base, [(FEATURES[i], float(drops[i])) for i in order if drops[i] > 0]
u0090 09-23 14:00 label=impossible_travel score=0.784 alert=yes
    new_device     value=      1.0  score drop if typical: 0.041
    geo_speed_kmh  value=  21033.8  score drop if typical: 0.032
    n_ips          value=      3.0  score drop if typical: 0.028
u0336 09-22 05:00 label=credential_stuffing score=0.804 alert=yes
    new_device     value=      1.0  score drop if typical: 0.027
    n_attempts     value=      4.0  score drop if typical: 0.019
    n_fail         value=      3.0  score drop if typical: 0.017
u0214 09-22 12:00 label=normal/trip score=0.766 alert=yes
    n_attempts     value=      4.0  score drop if typical: 0.085
    geo_speed_kmh  value=  25540.6  score drop if typical: 0.048
    n_ips          value=      3.0  score drop if typical: 0.039

The impossible-travel alert reads well: new device plus 21,000 km/h. The stuffing alert points at the new device and the failures. Notice what’s missing, though: ip_fanout never shows up, even though it’s the strongest stuffing signal. The third alert is a business trip, and its explanation shows the same impossible-travel pattern as a real attack, because a hotel IP on another continent is geographically odd.

Single-feature ablation has a known blind spot: when two features are odd together, resetting either one alone may barely move the score. In the test suite, a synthetic 30-failure window got its explanation attributed to n_attempts rather than n_fail, because both were extreme together. Treat explanations as hints for the analyst, not proofs.

Step 8. Drift and retraining

Days 29-35 simulate a normal IT change week. Two things ship at once: a corporate VPN starts sending 40% of users through two gateways in Frankfurt, and a new SSO client silently re-authenticates half the users twice after every login, about 20 minutes apart. Days 29-30 have no attacks, so the model has a chance to re-baseline. Days 31-35 get a full set of attacks.

PSI train -> IT-change week (>0.25 usually = significant shift):
    ip_fanout      2.64
    n_attempts     0.57
    hour_rarity    0.30
    new_asn        0.11

old model, old threshold, days 31-35: alerts=  367  precision=0.313  recall=0.943  PR-AUC=0.868
    alerts/day: 73  (budget was 20)
retrained (days 10-30), new threshold:   alerts=  269  precision=0.416  recall=0.918  PR-AUC=0.898
IPs with >50 distinct users in the IT-change week: ['198.51.100.10', '198.51.100.11']
old model + egress allowlist feature:   alerts=  356  precision=0.323  recall=0.943  PR-AUC=0.870

attack windows caught in days 31-35, by type:
                     old  retrained  allowlist  total
label                                                
brute_force            7          7          7      7
credential_stuffing   80         80         80     80
impossible_travel     10         10         10     10
low_and_slow          18         15         18     25

alerts per day, old model: [77, 33, 26, 28, 8, 51, 35, 87, 66, 74, 82, 30, 36, 145]
attack windows per day:    [52, 5, 3, 7, 4, 44, 8, 0, 0, 6, 6, 13, 24, 73]
false positives/day: test week 21.1 | days 31-35 old 50.4 | retrained 31.4 | allowlist 48.2

Population Stability Index screams about ip_fanout (2.64): two IPs now serve over 50 users each. A common rule of thumb treats PSI above 0.25 as a significant shift, though Yurdakul’s 2018 thesis notes these cut-offs are conventions with no stated error rates. But the feature with the loudest PSI isn’t what broke the alerts. Fixing it with an egress allowlist barely helps (50.4 → 48.2 false positives/day). My reading: each split picks one of ten features at random, so one extreme feature moves the average path length less than you might expect. The real damage comes from n_attempts: background SSO re-auth makes a normal hour look like a busy hour, and “busy hour” was already the forest’s main false-positive pattern.

Retraining on a rolling 21-day window (days 10-30, so it includes two days of the new normal) brings false positives down to 31.4/day and PR-AUC up from 0.868 to 0.898. It also costs something: it catches 15 low-and-slow attempts instead of 18. That’s the trade-off with every unsupervised retrain. The model learns whatever the window contains, including attacks nobody labeled.

Two smaller findings. hour_rarity drifts on its own (PSI 0.30) because per-user hour histograms keep sharpening as history grows: a cold-start effect, not a behavior change. And the 77 alerts on day 22 aren’t a model problem: that day contains a full stuffing campaign (52 attack windows). An alerts-per-day chart without attack counts next to it would have sent me looking for a bug.

Alerts per day over two weeks, with the IT change marked: false positives per day go from 21.1 in the test week to 50.4 after the change, and 31.4 after retraining

Where the model gets it wrong

Collecting the failures in one place:

• Low-and-slow stuffing: 12 of 25 caught. One failed login from the right city on the right device during working hours is indistinguishable from a typo. Only cross-account features (many accounts, same password-failure pattern, same hour) can see it, and that’s a different model.

• Busy users: most false positives are legitimate heavy users who mistyped a password.

• Travelers: a business trip through a foreign hotel IP looks exactly like impossible travel.

• The victim’s return: after a takeover, the real user’s next login looks like an anomaly. That’s useful, but it breaks naive labeling.

• Silent masking: 20 attack windows sit in the training data. The paper calls this masking: a dense cluster of anomalies is harder to isolate. Real logs will have more of it than my 20.

Gotchas I hit

1. Window-boundary bugs in features. The geo-velocity bug above raised no error, and PR-AUC only moved from 0.870 to 0.873 after the fix, because new_device and new_country carried those windows anyway. Only the explanation exposed it. Write a unit test for a two-logins-in-one-hour case.

2. log1p didn’t matter here. Isolation Forest splits uniformly between min and max, so I expected heavy tails to hurt. PR-AUC with log transforms was 0.873; with raw counts, 0.870. Measure before you add preprocessing.

3. The “obvious” network feature didn’t help the score. Dropping ip_fanout moved PR-AUC from 0.873 to 0.875. The stuffing windows were already isolated by new_device and failures. For a single strong signal, a rule is still a better tool.

4. Random seeds move PR-AUC by about 0.01. Five random_state values gave 0.866 to 0.878. Don’t celebrate a 0.005 improvement from a feature change without checking several seeds.
   

PR-AUC with log1p on counts: 0.873   raw counts: 0.870
PR-AUC over 5 random_state values: 0.878 0.869 0.871 0.870 0.870
PR-AUC without the network feature ip_fanout: 0.875

Tests that pin the behavior

The suite checks the claims this post depends on: deterministic generation, causal features, failed attempts not whitelisting the attacker’s device, in-window impossible travel, offset_ equal to a training quantile, and explanations pointing at the right feature family.

def test_features_are_causal(data):
    """Appending future events must not change features of earlier windows."""
    cut = START + pd.Timedelta(days=8)
    w_full, _ = build_windows(data)
    w_past, _ = build_windows(data[data["ts"] < cut])
    early = w_full[w_full["window"] < cut].reset_index(drop=True)
    cols = [f for f in FEATURES if f != "ip_fanout"]  # fan-out is per hour, also causal
    pd.testing.assert_frame_equal(early[cols], w_past[cols])

def test_contamination_is_just_a_train_quantile(data):
    w, _ = build_windows(data)
    X = to_matrix(w)
    m = train(X, contamination=0.02)
    assert m.offset_ == pytest.approx(np.percentile(m.score_samples(X), 2))
    assert abs((m.predict(X) == -1).mean() - 0.02) < 0.002
.......                                                                  [100%]
7 passed in 2.98s

Production checklist

• Compute features causally, from state that only successful logins update, and test window boundaries explicitly.

• Set the threshold from an alert budget on recent data, not from contamination="auto".

• Report precision, recall and PR-AUC per attack type, plus false positives per day. A single F1 hides the low-and-slow row.

• Keep rules for known high-signal patterns (fan-out, failure bursts, impossible travel) and use the forest as a ranked review queue for the rest.

• Ship an explanation with every alert, and treat it as a hint.

• Monitor false positives per day, not only PSI. The loudest drifting feature may not be the one that hurts.

• Retrain on a rolling window after known changes, compare recall per attack type before and after, and allowlist infrastructure such as VPN egress IPs in the features.

• Treat user feedback (“that was me, I was traveling”) as labels, and eventually move to a supervised model on top of these features.

Wrapping up

Isolation Forest is a reasonable first anomaly detection algorithm for login logs: it needs no labels, trains in seconds and gives a ranked list. On this data it didn’t replace rules. It complemented them, catching half of an attack class the rules missed, at the cost of a noisier queue. Most of the work, and most of the bugs, were in the features and the threshold, not the model.

What does your team use for suspicious-login detection: vendor risk scores, hand-written SIEM rules, or your own models? And how do you get labels for the alerts nobody escalates?