Detection Engineering Lab: Building a Wazuh SIEM and Writing a Custom Rule for SSH Brute Force

This write-up documents a small SIEM detection lab I built to practise detection engineering from the defender's side: deploying a SIEM, generating adversary activity against a monitored host, validating that it is detected, and writing a custom correlation rule mapped to MITRE ATT&CK.

Repository with the rule, configuration notes and all screenshots: https://github.com/devyaninandanwar/wazuh-siem-detection-lab

Objective

To validate end to end that a SIEM can ingest endpoint logs, surface attacker activity, and support a custom detection, and to practise the triage thinking that goes with it, including recognising a false positive.

Architecture

Three virtual machines in VirtualBox on a bridged network:

The Wazuh agent on the endpoint collects local logs (journald for SSH authentication events, and the Apache access log) and forwards them to the manager. The manager decodes the events, evaluates them against its ruleset, and indexes alerts for the dashboard.

Deployment issues

Two infrastructure problems are worth recording, since they affect anyone running this on a laptop.

Storage. The first manager VM had a 25 GB disk, and the all-in-one installer failed with "No space left on device". After rebuilding with 60 GB, the Ubuntu installer's LVM layout still allocated only about 29 GB to the root logical volume. The fix was lvextend -l +100%FREE on the logical volume followed by resize2fs, verified with df -h.

Memory. With 4 GB of RAM, the Wazuh indexer, manager and dashboard compete for memory, and after a reboot some services failed to start. Starting the indexer first and the dependent services afterwards resolved it.

Detection scenario 1: SSH brute force (MITRE ATT&CK T1110)

Simulated from the Kali host with Hydra against the endpoint's SSH service:

hydra -l ubuntu -P passwords.txt ssh://192.168.1.194

The agent's authentication logs triggered Wazuh's built-in rule 5760 (sshd: authentication failed) repeatedly. The event detail includes the targeted account, the source IP and port, and the raw log line, which is the data an analyst needs to start an investigation.

Detection scenario 2: Network reconnaissance (MITRE ATT&CK T1595)

A service-version scan from the Kali host:

sudo nmap -sV 192.168.1.194

This was detected through the Apache access log. Nmap's scripting engine requests non-existent URLs, producing 404 responses that Wazuh flagged under its web rules (31101). The raw log line includes the Nmap Scripting Engine user agent, which makes attribution of the tooling straightforward.

Detection engineering: a custom correlation rule

The built-in rule 5760 alerts on every failed login. A single failure is normal operational noise, so alerting on it alone produces low-value alerts. The detection of interest is a burst of failures. I wrote a frequency-based correlation rule that fires when rule 5760 triggers five times within 120 seconds:

<rule id="100002" level="10" frequency="5" timeframe="120">
  <if_matched_sid>5760</if_matched_sid>
  <description>SSH brute-force attack detected</description>
  <mitre>
    <id>T1110</id>
  </mitre>
  <group>authentication_failures,pci_dss_11.4,mitre,</group>
</rule>

Re-running the Hydra attack triggered rule 100002 at level 10, confirming the detection works against the simulated activity.

Alert triage: a rootcheck false positive

After deployment the SIEM raised 84 alerts for rule 510 (rootcheck), "Trojaned version of file detected", on system binaries such as /usr/bin/md5sum. On a freshly built host this warrants investigation before dismissal. Reviewing the event showed the rootcheck signature is a generic string pattern that matches any file containing certain references (for example /bin/sh), and no other indicators of compromise were present. My assessment was that this is a known false positive from generic rootcheck signatures.

The appropriate response in an operational environment is a documented, scoped tuning exception for those files, not disabling the rule. I documented this approach but did not implement the exception in this lab.

Roadmap

The lab is built to be extended. Planned next iterations:

• Same-source correlation. Tighten rule 100002 with same_source_ip so it alerts only when the failures come from a single address, then retest against the same Hydra run.
• Automated response. Use Wazuh active response to block the offending IP once the rule fires, taking the lab from detection to containment.
• Broader coverage. Add further ATT&CK techniques and log sources, such as file integrity monitoring and privilege escalation scenarios.
• Tuning. Implement the scoped exception for the rootcheck false positive and document the before and after alert volume.

Takeaways

Standing up a SIEM and observing its alerts against traffic you generated yourself makes the detection engineering workflow concrete: telemetry in, a rule that expresses a hypothesis about attacker behaviour, validation against simulated activity, and triage of whatever noise comes with it.

Which attack techniques would you simulate next in a lab like this? Share your ideas in the comments!