The False Positive Wasn't a SQLite Quirk: It Reopened on Elasticsearch Too
In the last post (link), an LLM-hardened Sigma rule passed the model's own self-test with 19/19 and reopened both real false positives, because one Sigma value β a bare '\\' β collapses to a single backslash inside pySig
In the last post (link), an LLM-hardened Sigma rule passed the model's own self-test with 19/19 and reopened both real false positives, because one Sigma value β a bare '\\' β collapses to a single backslash inside pySigma's parser and compiles to "contains at least one backslash": a condition every Windows path satisfies.
In the comments, Ahmet Γzel made a precise suggestion, the kind that costs you a weekend: preserve a minimal pair of events that differs only in the path form, run it through each backend you actually deploy, and record the backend and version with each result, so the evidence stays scoped instead of implied.
This post is that experiment. Short version: the false positive reopens on Elasticsearch exactly the way it reopens on SQLite. The tautology was never a SQLite story; it rides in the parsed rule and travels with it.
The pair
Four synthetic Sysmon EventID 10 events β same TargetImage (lsass.exe), same GrantedAccess (0x1410), same SourceImage (outside the rule's trusted-path filter, which is what makes the events eligible) β differing only in CallTrace:
-
E1 β local path, backslashes:
C:\Program Files\Vendor\agent.dll+0x1234β benign. Ifv2matches this, the false positive is back. -
E2 β UNC path:
\\filesrv\metrics\agent.dll+0x1234. What the bare'\\'was (wrongly) meant to catch: for a UNC you need two. -
E3 β forward slashes:
C:/Program Files/Vendor/agent.dll+0x1234β the control: same module content as E1, no backslash anywhere. If the "contains at least one backslash" branch is what fires, this event must stay silent on every rule version β this is what isolates the mechanism. -
E4 β Downloads path with backslashes:
C:\Users\dev\Downloads\agent.exe+0x1234β the explicit-pattern true positive the narrowing was supposed to catch.
And the two real rule versions from the study, unchanged: v2 (bare '\\' still in the folder list) and v3 (bare value removed).
Predictions before results
Everything was pre-registered before the runs, including the boring ones: v2 matches E1/E2/E4 and not E3 on SQLite (the study's own result, re-executed); v3 matches only E4. The interesting registrations were the Elastic ones:
-
P1: the Elasticsearch backend compiles the bare value to a wildcard pattern that, in the registration's own words, βrequires at most one literal backslashβ β observed as
*\\*: contains at least one backslash, so the E3 control (zero backslashes) must stay silent. (Confirmed in substance β the collapse passed through the second compiler unmodified. The registered wording βat most oneβ was imprecise: the observed term requires at least one backslash.) -
P2:
v2matches E1 on Elastic β the same false positive reopens. -
P3:
v3matches only E4. -
P4, the falsifiable alternative: Lucene escaping re-widens the value into a genuine two-backslash requirement,
v2does NOT match E1, and the result is a cross-backend divergence β same rule, same parser, different detection outcome. Registered as "would be a NEW finding if it happens", because it would have meant the operator's risk depended on the backend's escaping table, not just on the shared parser.
P4 did not happen. The pre-registration is in the artifacts with its hash chain, amended once, with a date, before any Elastic run.
The wrinkle pip forced on the design
"One environment, two backends" was the plan. pip disagreed: every released version of pysigma-backend-elasticsearch (0.1.0 through 2.1.1) constrains pysigma<2.0.0, and the study was compiled with 2.0.2. That's a ResolutionImpossible for the whole stack in one venv. So the pair ran on two environments:
| Side | Engine | Stack |
|---|---|---|
| SQLite | sqlite3 3.54.1 | pySigma 2.0.2 Β· backend-sqlite 2.1.1 Β· pipeline-sysmon 3.0.0 Β· pipeline-windows 3.0.0 β the study's exact stack, verified identical to the study's SQL |
| Elastic | Elasticsearch 7.17.4 Β· Lucene 8.11.1 | pySigma 1.5.1 Β· backend-elasticsearch 2.1.1 Β· pipeline-sysmon 2.0.0 Β· pipeline-windows 2.0.1 β the newest combination pip resolves under pysigma<2.0.0
|
A different compiler on the Elastic side could have explained any difference in results by itself β a confounder sitting on top of the thing under test. So before any run, the mechanism itself was verified at runtime in both compilers: SigmaString("\\\\").s == ['\\'] on 2.0.2 and on 1.5.1. Same collapse, both parsers; the split changes the pipeline surface, not the parsed pattern. It's registered as a dated amendment (E-01) to the pre-registration, not folded in silently. Scope honestly: the Lucene term behavior below is verified on Lucene 8.11.1 (the 7.17 line), not on Elastic 8.x / Lucene 9.
Results
| Event | Path form | v2 SQLite | v2 Elastic | v3 SQLite | v3 Elastic |
|---|---|---|---|---|---|
| E1 (benign) | local, backslashes | MATCH | MATCH | no | no |
| E2 | UNC | MATCH | MATCH | no | no |
| E3 (control) | forward slashes | no | no | no | no |
| E4 (TP) | Downloads | MATCH | MATCH | MATCH | MATCH |
The matrix is identical across engines. On the compiled Lucene side, the bare value arrives as the term below (the compiled Lucene query, JSON-unescaped):
(CallTrace:(*\\Users\\* OR ... OR *\\*))
*\\* is "contains at least one backslash" β E1 and E2 both qualify, E3 (no backslashes) does not, which is the isolation the control was registered for. v3, with the bare value removed, keeps the explicit-pattern horizon (E4) and lets go of everything else, on both engines.
The control earns its keep by staying silent: it never fires on any rule version on either engine, because after the collapse the branch still requires a backslash β the failure is backslash-scoped, not "matches everything on every input". A rule version that matched E3 (no backslashes at all) would point at a branch with no meaningful predicate left; the observed result is the narrower, more useful one: every Windows-style path with a backslash qualifies, and nothing else does.
What went wrong along the way
The register keeps the dirt, because clean stories are how evidence rots:
- The bulk generator was wrong. The run scripts β written by the verifier model (Mistral Vibe) β wrote one bulk action line for all four documents; the Elasticsearch bulk API needs one action line before each document. Caught mid-run; the file was regenerated with one action line per document, event lines unchanged, and the buggy one is kept, hashed, in the artifact set.
-
The compile scripts shipped with a broken import. The scripts β written by the verifier model (Mistral Vibe) β shipped with a broken import (
from sigma.rule import SigmaString; the class lives insigma.types); it passed the model's own syntax check (py_compile) and failed on the first real execution. I ran them without reviewing the import, so that's on my process too. Fixed before any result existed, with the errata dated in the scripts' docstrings. -
An overstated label. The SQLite side reported
byte_identical_to_study_sql: truewhile showing two different hash pairs β because the reference files end with a trailing newline and the compiled SQL did not. Same content, no byte equality claim. Label corrected to a precise taxonomy in the post-run script revision (v1.2); the published results are the executed version's output, kept as-is. - The engine is from 2022. The Homebrew tap yields Elasticsearch 7.17.4. Evidence scoped per engine version, as requested.
- The Elastic venv's full freeze was taken after the run, not during it. Same environment, no changes between run and freeze β declared, not assumed.
-
Running the engine needed a non-bundled JDK. The brew Elasticsearch ships its own JDK, which is Intel-only on this machine ("Bad CPU type"); the cluster ran on brew's OpenJDK 17 with
-E xpack.ml.enabled=false. This should not affect Lucene query semantics; recorded for reproducibility. -
The two sides did not run the same query shape. The SQLite side ran the study's full pipelined SQL,
ChannelandEventIDconditions included; the Elasticsearch run consumed the no-pipeline variant of the same rules, with the pipelined Lucene variants compiled and recorded but not executed against the index. The predicate under test β theCallTracebranch carrying the bare'\\'β is identical on both sides; the log-source conditions are not. A design asymmetry, recorded rather than smoothed over. - The Mac's sqlite3 is newer than the baseline's. 3.54.1 on the host against 3.46.1 in the independent baseline; the matrix came out identical on both, and the version is recorded per result.
-
The executed script versions no longer exist as files. The runs were produced by v1.1 of both scripts (sha256
a22612c2β¦/43cc079eβ¦, hashed by the operator at run time, before the results β thebyte_identical_to_study_sqlfield in the results is v1.1 output); the post-run revisions that became v1.2 were saved on top of the same filenames. The deltas are documented in each script'sFixesdocstring; no backup exists; v1.1 survives by hash only. Custody policy from now on: a new version gets a new file β never an in-place overwrite.
(Items 6β8 added 2026-11-21 after an independent second-opinion review; item 9 added the same day, in that review's wake, when the executed-version question was confirmed.)
All nine are in the repo's minpair folder with dates and hashes (repo).
What this does not show
- Four synthetic events are not production telemetry. Nothing here supports a false-positive rate claim; only an existence proof per engine.
- One rule, one technique, two of many backends. Splunk, KQL/EQL and Elastic 8.x were not run. The "probably general" from the first post is now "demonstrated on two engines", not "demonstrated everywhere".
- The two-compiler split (pySigma 2.0.2 vs 1.5.1) is a declared scoping condition, not a version comparison.
-
v3not matching E2 is a declared trade-off of the fix, recorded in the study's report where the removal of the UNC bare value was decided (none of the study's false negatives required UNC matching in that branch) β not in the rule's YAML description, and not a regression found here.
The rule I took away last time stands, with the reviewer's addition stamped on it: never accept a self-test β compile against the real engine and data the model did not choose β and scope every result to the backend and version that produced it.
Artifacts: pre-registration (with the dated amendment), events, compiled queries, raw engine outputs, the run scripts (v1.2, post-run revisions of the executed v1.1 β see deviation 9), and the full hash chain: minpair/.
This work was done with LLM assistance throughout: the original study ran a three-model loop (builder, attacker, verifier). This follow-up I executed myself, step by step on my machine: one model guided the run and read the outputs as I pasted them back, and the verifier that keeps the hash register (which also wrote the run scripts) checked the artifacts independently, from outside the machine. I ran the experiment, decided what counts as evidence, and kept the bugs in the record.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.