My SOC triage model had 0 false escalations. On real alerts it had 39 out of 100.

On my test set the model got 55 out of 55 verdicts right, with zero false escalations. Then I gave it 100 real alerts from a Wazuh lab. It escalated 39 of them for nothing, and invented incidents that never happened for some of them.

That gap shaped everything after it. Here is what I built, the numbers I got, and what those numbers do not prove. The code and weights are open: GitHub, Hugging Face.

The goal

Tier-1 SOC analysts drown in repetitive alerts, and cloud LLMs are often off the table because logs must not leave the network. I wanted a helper that runs on a laptop with 8 GB of VRAM, sends nothing out, and prepares a card the second tier can check.

It is a fine-tuned Gemma 4 E2B (Apache 2.0). Input: one alert as an evidence pack, E1..En. Output: JSON with one of four verdicts, a summary, claims that cite evidence ids, MITRE ATT&CK techniques, what is missing and what to check next. The citations exist so code can verify the output.

What the real stream changed

On high-frequency rules missing from my synthetic data (registry FIM, rootcheck), the model overrated severity and wrote a story around it. Two fixes:

Train on real noise. I added 215 real benign cases from the lab. On 100 held-out real alerts:

Stop calling the model when the result does not depend on it. Log injections and allowlisted operational noise are decided deterministically first. Context still escalates: a change to an autorun key or a security service falls out of the allowlist. On 3,000 lab alerts, 98.1% were decided without the model and 1.9% reached it.

A schema beat a new training run

Version 6 got more accurate on known types, but on unseen types 9 of 20 cards failed validation. The raw answers showed why: the model cited the field name ev1 instead of the id E1.

Instead of retraining, I now build the Ollama JSON schema per alert. Only that alert's evidence ids and the MITRE candidates from a local index are allowed. No GPU needed.

The cost is in the table. There are a few extra escalations, and citation checks are now formal: the id always exists, but the schema cannot say it supports the claim.

The hint changes the answer

This was the most uncomfortable result. The candidate list itself nudges the verdict toward escalation. With T1555.004 in the list, the earlier v5.1 described a routine Credential Manager read by Outlook as mass credential harvesting.

So I told the model that candidates are keyword matches and do not imply an attack. Recall on unseen types fell from 20/20 to 18/20: OAuth consent and IDOR got closed as routine. I reverted it. For a SOC, an extra escalation is cheaper than a missed attack, but that is a trade-off, not a fix.

What these numbers do not prove

• Recall on real attacks is unmeasured. The real set is only noise, and 20/20 covers five attack families.
• Some cases nearly match training. 30 of 76 known and 63 of 100 real cases match training data once digits are ignored. Outside those, verdicts under the previous schema were 44/46 and 34/37.
• The real-noise row does not test the tail. In the service the allowlist closes nearly all of that stream before the model runs.
• The real-noise labels are one analyst's judgment, not shift decisions.

That is why the service runs in assist mode: it closes only deterministic noise, and a human sees everything that reached the model. It is useful for noise reduction and draft cards. It is not validated as an autonomous escalation decider.

What is open

The repo has the Wazuh service (read-only), the per-alert schema, the ATT&CK index, training and evaluation scripts, and an operator guide. The v6 weights are on Hugging Face as a 4.9 GB GGUF for Ollama.

The dataset is not included. It relies on course material and alerts from a private lab, so you can run the model on your own alerts, but the numbers above cannot be reproduced from public files. docs/dataset.md explains how to build your own set.

Before releasing the weights, I checked whether the model leaks lab machine names. They appear in training inputs but not in answers, and the loss covers only answers. They did not appear in 306 attempts, and neither did a synthetic name that was in 144 training answers. This lowers the risk but does not rule out memorization.

Next comes controlled attacks on the lab, which is the only way to get a real recall number. If you run a SOC, I would like to know where a setup like this would break first for you.