A great rule, 99% false positives: base rates and careful tuning

The scenario and numbers are illustrative; the arithmetic was verified by script.

"This rule is useless, it's all false positives." Maybe the rule is good, and the math is the problem.

What you'll learn

• The confusion matrix as a shared language
• The base-rate fallacy, with numbers
• Tuning that doesn't add false negatives
• Why every exclusion needs FN verification

The story at a glance

Week 1: a rule for WINWORD -> powershell -enc floods the queue, mostly managed-software automation. Week 2: analysts start ignoring it. Week 3: the real attack is buried. They detected it and still couldn't respond.

Confusion matrix

Base rate, in numbers

1,000,000 events/day, 10 real attacks, zero misses, false positive rate 0.1%:

• TP = 10, FP = 999,990 x 0.1% = about 1,000
• Precision = 10 / 1,010 = 0.99%

Cut the false positive rate to 0.001% and precision rises to about 50%. Both figures were computed and verified in a script.

Tuning rules of thumb

1. Baseline before you exclude.
2. Exclusions: narrow and specific (full path of the managed product, decoded content, principal, signature), never "exclude powershell.exe".
3. -enc is a signal, not a verdict: always decode.
4. Correlate in sequence and time window.
5. After each exclusion, replay an attack and confirm it still alerts.

Operate with metrics

Precision, alert volume per day, top-firing rules, and a periodic review of exclusions.

Record why it was a false positive

"Looks like FP" is not a conclusion. Write the evidence, and escalate when unsure.

Next: running all this as a system, not heroics.

Adapted from chapter 32 of my Japanese book on server defense.