Dev.to Security 🔐 Cybersecurity 👁 0 📖 3 min read

I asked an LLM to harden a real SigmaHQ rule. Its self-test said 19/19 PASS. The real compiler said otherwise.

I asked an LLM to harden a published SigmaHQ detection rule. It delivered a new version with its own test report: 19/19 PASS, both false positives closed. When I ran the same rule through the real Sigma compiler against

I asked an LLM to harden a published SigmaHQ detection rule. It delivered a new version with its own test report: 19/19 PASS, both false positives closed.

When I ran the same rule through the real Sigma compiler against real Windows event logs, nothing had changed. Both false positives still fired. One line of the rule had quietly turned its own filter into a condition that is always true.

This post is about that one line, why the model's self-test could not see it, and what happened after I reported it.

The setup

  • Rule: SigmaHQ's proc_access_win_lsass_memdump.yml (LSASS credential dumping, MITRE ATT&CK T1003.001).
  • Ground truth: 725 real .evtx files (37,841 events) from EVTX-ATTACK-SAMPLES and SigmaHQ's own regression data.
  • Engine: pySigma with the SQLite backend. Every rule version was compiled and executed, not just read.

The original rule detects 7 of the 11 documented LSASS dump samples in that corpus. I asked the model to recover the 4 it misses without losing the 7 and without opening false positives.

Round 1 recovered all 4, but opened 2 false positives on real files. The model's own test (17 cases, some invented) had reported 0.

The line

In round 2 I gave the model the two false-positive events and asked it to close them. It narrowed one branch of the rule to suspicious CallTrace paths:

CallTrace|contains:
  - '\Users\'
  - '\Temp\'
  - '\Downloads\'
  # (three more paths)
  - '\\'

The last value was meant to match UNC paths (\\server\share). The model ran sigma check (0 errors), ran its own Python test harness, and reported 19/19 PASS.

Here is what pySigma actually compiles that to:

CallTrace LIKE '%\\Users\\%' ESCAPE '\'
  OR CallTrace LIKE '%\\Temp\\%' ESCAPE '\'
  OR CallTrace LIKE '%\\Downloads\\%' ESCAPE '\'
  -- (three more paths)
  OR CallTrace LIKE '%\\%' ESCAPE '\'

The last branch means "contains one backslash". Every Windows path contains a backslash, so the whole list is always true and the narrowing does nothing.

Why the self-test passed

In Sigma, backslash is the escape character, so '\\' is one literal backslash. That is intended and documented behavior, not a pySigma bug.

The model believed '\\' meant two backslashes. Its test harness did not call pySigma. It loaded the YAML and did plain Python substring checks, which also treat '\\' as two backslashes. The test shared the author's wrong belief, so it passed.

That is the finding: the model validated its rule against its own idea of what the rule means, not against the engine that runs it.

When I showed the model the compiled SQL, with no further hint, it found the line, removed it, and switched its tests to the real compiler. The final version detects 11/11 with 0 false positives on this corpus.

Reproduce it in two minutes

pip install pysigma pysigma-backend-sqlite
from sigma.collection import SigmaCollection
from sigma.backends.sqlite import sqliteBackend

rule = r"""
title: repro
logsource: {category: process_access, product: windows}
detection:
  sel:
    CallTrace|contains:
      - '\Users\'
      - '\\'
  condition: sel
"""
print(sqliteBackend().convert(SigmaCollection.from_yaml(rule))[0])

Output (pySigma 2.1.0, pysigma-backend-sqlite 2.1.1):

SELECT * FROM logs WHERE CallTrace LIKE '%\\Users\\%' ESCAPE '\' OR CallTrace LIKE '%\\%' ESCAPE '\'

What happened next

I opened an issue on pySigma suggesting a validator for values like this, framed as an idea rather than a bug: SigmaHQ/pySigma#615.

  • Within 12 hours another contributor implemented it: pySigma#617.
  • He ran it over the 3,305 official SigmaHQ rules. It found a published rule where an empty value turns a filter into a catch-all, now reported, with a fix proposed, in sigma#6596 / sigma#6597.

As of this writing, both pull requests are open and awaiting maintainer review. Nothing is merged.

What this does not show

  • One rule, one technique, one backend. The mechanism is probably general; what is demonstrated is one instance.
  • 0 false positives on this corpus is not 0 false positives in production. The corpus has no clean enterprise telemetry.
  • This is not "LLMs can't write detections". Shown the real compiler output, the model fixed its own bug.

The practical rule I took from it: never accept an LLM's self-test for a detection rule. Compile it with the real engine and run it against data the model did not choose.

Full study, rules, compiled SQL and evaluation script: sigma_t1003_study.

This work was done with LLM assistance throughout: one model built the rules, a second audited them, a third independently re-ran every measurement. I ran the experiment and decided what counts as evidence.

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.