I Built and Attacked A SIEM

1. Introduction

There is a special kind of thrill in watching an attack hit a server you own, knowing you are safe because you are the one attacking it. In this post I am going to walk you through how I built my very first SIEM deployment with Wazuh, added a brand-new server to it, and then attacked that server with real offensive tools to see what my monitoring was actually made of. Along the way I learned the difference between seeing an attack and stopping one, and I have a strong opinion about which one surprised me more. Stick around for the Final thoughts section, where I break down the single coolest moment of the project, the exact second my own attack machine got banned from my own server while my SIEM watched the whole thing.

A quick intro before we dive in. My name is Damilola Tijani, and I am a cybersecurity junior analyst studying at TripleTen. This is my first-ever contribution to the cybersecurity community. I built this project to get hands-on with the tools real SOC analysts use every day, and I am publishing it in the hope that somebody else standing where I stood a few weeks ago finds it fun, useful, or at least entertaining.

2. Setup

By the time I started this project, my little lab already had a working SIEM. A Wazuh manager was collecting logs from two endpoints, a Windows Active Directory server called ad01 and a Linux server called observer, each running a Wazuh agent plus Sysmon for detailed event logging. My environment also had one machine doing absolutely nothing, a fresh Ubuntu Server 24.04 VM called play-server, provisioned specifically for students to use in this project. A blank server sitting outside my monitoring? That had to change.

My modification was to enroll play-server into the SIEM as a third monitored endpoint, then harden it, because a machine I am about to attack on purpose should at least put up a fight.

Enrolling play-server

The Wazuh dashboard makes this pleasantly simple. From the agent management area I clicked Deploy new agent, picked the DEB package, pointed it at my manager (wazuh-siem.megaquagga.local), and named the agent play-server. The dashboard generated a ready-made install command for me.

Image 1: The Deploy new agent page

Over on the play-server console I pasted the command, then started the service with three more commands.

sudo systemctl daemon-reload
sudo systemctl enable wazuh-agent
sudo systemctl start wazuh-agent

Image 2: the agent installed and the service started.

A minute later, play-server showed up in my Endpoints Summary as Active (agent 004), and a quick filter in Threat hunting > Events (agent.name is play-server) confirmed logs were flowing, 284 hits and climbing.

Image 3: Endpoints Summary with play-server Active.

Image 4: Events view filtered to play-server with hits.

Hardening with UFW and fail2ban

Since Experiment 3 was going to be me throwing a password-guessing attack at this box, I wanted two defenses in place first. UFW (the Uncomplicated Firewall) would slam every door except SSH.

sudo apt install -y ufw
sudo ufw allow ssh
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw enable

Note the order here. allow ssh comes before enable. Get that backwards on a remote server and you have just firewalled yourself out of your own machine.

Then fail2ban, which watches authentication logs and automatically bans any IP that fails login too many times.

sudo apt install -y fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd

Image 6: fail2ban-client status sshd output.

The mistake

Mine was putting the cart before the horse. The dashboard’s generated command downloads and installs the agent package, and the systemctl lines only configure and start it. In my excitement I ran the systemctl commands first, before the installer, and got slapped with Unit wazuh-agent.service could not be found. Of course it could not be found, because I had not installed anything yet. Ten seconds of re-reading fixed it, but the difference between installing a service and starting one is now permanently installed in my head (pun intended).

3. Experiment time

With play-server enrolled and hardened, it was time to answer the real question. If something happens on this machine, will my SIEM actually show it to me?

Experiment 1 - (the control)

Before attacking anything, I wanted a baseline of what “normal” looks like. So, I logged into play-server and did the most ordinary admin work imaginable.

sudo apt update
sudo touch /etc/blog-test.txt
echo "testing my SIEM" | sudo tee /etc/blog-test.txt
sudo rm /etc/blog-test.txt
sudo systemctl restart fail2ban

Image 7: terminal after running these commands.

Then I went hunting in Threat hunting > Events. With the agent.name is play-server filter in place, my boring session was all over the dashboard. The login itself fired rule 5501 (PAM: Login session opened, level 3), tagged with MITRE technique T1078 (Valid Accounts). Each sudo command fired rule 5402 (Successful sudo to ROOT executed, level 3), tagged T1548.003 (Sudo and Sudo Caching). Think about that for a second. One ordinary admin session produced two different MITRE technique tags automatically. Nothing I did was an attack, but every piece of it looked exactly like the opening moves of one. That is the whole point of a baseline.

Image 8: the events list, and the expanded rule details showing 5501 (T1078) and 5402 (T1548.003).

The experiment mistake

Here is where it got interesting. I searched blog-test.txt expecting the file to show up in Wazuh’s File Integrity Monitoring (FIM). I got 3 hits, and for a moment I thought everything was fine. Then I looked closer. All three were sudo events (rule 5402) matching the command text I had typed. The file monitor itself had zero record of the file. The only proof my file ever existed was the echo of the command that created it.

The culprit turned out to be a default setting. Wazuh’s syscheck module watches /etc, but it does so on a schedule, one scan every 12 hours by default. A file created at lunchtime simply sits there unobserved until the evening scan. The fix was one line in /var/ossec/etc/ossec.conf on play-server, switching the watched directories to real-time monitoring.

<directories realtime="yes">/etc,/usr/bin,/usr/sbin</directories>

One sudo systemctl restart wazuh-agent later, I deleted and re-created the exact same file, and this time a syscheck alert appeared within seconds. Rule 554 (File added to the system, level 5) landed about one second after my touch and named /etc/blog-test.txt by its full path.

One fun detail for fellow beginners. Guides often cite rule 550 for this. In my Wazuh 4.14 install, 554 is the file-added rule, while 550 is the “integrity checksum changed” rule you will meet in Experiment 2. Write down the IDs your own install fires, not the ones a blog post told you to expect. (Yes, I see the irony of writing that in a blog post.)

Image 9: the before/after pair. Before: the search returning only sudo events. After: rule 554 in the list.

The lesson burned into my brain is simple. When a detection does not fire, check the tool’s defaults before you assume the attack is invisible.

Experiment 2 - (T1136.001, Create Account)

Next up was a classic attacker move, creating a local account for persistence. My plan was to drive it with Atomic Red Team, a free library of safe, bite-size attack tests, each one mapped to a MITRE ATT&CK technique. From the ART Workstation I loaded the framework and listed the available tests.

Import-Module "C:\AtomicRedTeam\invoke-atomicredteam\Invoke-AtomicRedTeam.psd1" -Force
Invoke-AtomicTest T1136.001 -ShowDetailsBrief

That was my first plot twist. Every test in this build of T1136.001 was Windows-only. Create a user via command prompt, create a user via PowerShell, plus two Windows admin variants, but nothing for Linux. So I improvised and ran the same MITRE technique by hand. Over on play-server I typed two commands.

sudo useradd attacker-test
grep attacker-test /etc/passwd

The grep confirmed the account existed, printing attacker-test:x:1001:1001::/home/attacker-test:/bin/sh.

Full honesty, because this blog is about real learning. My first attempt at that command was typed into the Windows PowerShell window on the ART Workstation, which politely informed me that “sudo is not recognized”. Right command, wrong machine. It happens to everyone, and it happened to me twice in one day.

Image 10: the ShowDetailsBrief output listing the Windows-only tests (the reason for the manual pivot).

Image 11: the play-server terminal with the useradd and the /etc/passwd line.

And here is where Experiment 1’s fix paid off. At 02:03:49, the exact second I ran useradd, Wazuh produced a small fireworks show.

• Rule 5902 (New user added to the system) at level 8, with a companion “New group added to the system” alert at the same second. For comparison, my sudo commands were level 3. Out of the box, Wazuh treats “someone just created an account” as a far bigger deal than routine admin work, exactly as it should.
• Rule 550 (Integrity checksum changed) from the file monitor, naming /etc/passwd, with the md5 hash before and after side by side and syscheck.mode: realtime right there in the event. The fix from Experiment 1 is the only reason this alert existed at all.

Two independent sensors, syslog and the file monitor, telling the same story at the same second. That kind of corroboration is the entire job of a SIEM.

Image 12: the 5902 event cluster, and the expanded syscheck event showing /etc/passwd with mode “realtime”.

Afterward I cleaned up with sudo userdel attacker-test. The deletion rewrote those same account files, producing a second wave of checksum alerts. Even the attacker’s attempt to cover his tracks was on camera.

Experiment 3 - (T1110, SSH brute force)

Finally, the fun one. From my Blue Team Workstation (a Kali box), I built a tiny password list of six common guesses and pointed Hydra, a classic login-cracking tool, at play-server’s SSH service.

hydra -l sysadmin -P ~/passwords.txt ssh://10.170.0.22 -t 4 -V

Round 1 was over almost before it started. All 6 passwords in 6 seconds (02:43:05 to 02:43:11), zero valid logins. Cute, I thought. Then I checked fail2ban’s log on play-server and found a ban stamped 02:43:12, one second after my attack ended and roughly seven seconds after it began. My last guess had beaten the hammer by a single second.

Then the second plot twist. The banned IP was 172.16.20.200, which is not my Kali machine’s own address (10.160.0.10). play-server sits on a different subnet, so my attack traffic arrived through the lab gateway, and as far as play-server was concerned, the gateway was the attacker. fail2ban banned exactly what it saw. The real-world lesson is that in NAT’d environments, IP-based blocking punishes the shared gateway address, which can lock out everyone behind it, not just the offender. Something to think about before auto-banning anything on a corporate network.

I beefed the list up to 106 passwords and came back for round 2. The first connection was refused instantly. The round 1 ban was still active, and my own firewall rejected my second attack before a single password was tried. I have never been so happy to be locked out of my own server.

Once the ban expired, the real run began. This time Hydra got 28 attempts in about two minutes, and then the terminal told the story in three acts. Normal [ATTEMPT] lines, then a pile of [RE-ATTEMPT] retries against a dead connection, and finally this.

[ERROR] all children were disabled due to too many connection errors

Hydra, a tool whose entire job is persistence, gave up on my server. fail2ban logged the second ban at 03:10:12.544.

Image 13: the round 2 Hydra output showing the transition from attempts to connection errors.

Meanwhile in Wazuh, searching the attacker IP (172.16.20.200) with my play-server filter showed the whole attack from the defender’s chair.

• Rule 5760 (sshd: authentication failed, level 5) and rule 5503 (PAM: User login failed, level 5) for the individual failures
• Rule 5763 (sshd: brute force trying to get access to the system) at level 10, the highest-severity alert of my entire project

Rule 5763 is the clever one. It ignores single failures, because people typo passwords all the time, and only fires after 8 failures from the same source (the event literally shows rule.frequency: 8). It is tagged T1110 (Brute Force) under the Credential Access tactic, and stamped with compliance references for GDPR, HIPAA, PCI DSS, NIST 800-53 and TSC. One alert, simultaneously telling an analyst “you are under attack” and an auditor “here is your evidence trail”.

My favorite detail of the whole project. The level-10 alert is timestamped 03:10:12.901. The fail2ban ban is stamped 03:10:12.544. Detection and response, from two completely independent tools, inside the same second.

Image 14: the event list for the attacker IP with 5760, 5503 and 5763 visible; plus the expanded 5763 event showing the MITRE T1110 fields.

One more honesty note. Documentation and older guides will tell you SSH brute force fires rules 5710 and 5712. My Wazuh 4.14 install fired 5760 and 5763 for the same attack. Rules get renumbered between versions, so always verify against your own dashboard.

Cleanup was its own mini-lesson. fail2ban-client unbanip returned “Invalid command” on my version, and the working syntax turned out to be sudo fail2ban-client set sshd unbanip 172.16.20.200. And in a repeat of Experiment 2’s wrong-machine trick, I first typed the Hydra command into play-server’s own console. Attacking the victim from the victim is not a known MITRE technique

Image 15: the fail2ban log grep showing both Ban lines (02:43:12 and 03:10:12).

The SIEM saw the attack. fail2ban stopped it. Two different superpowers, working half a second apart.

4. Conclusion

Three experiments, three different answers to “is my monitoring working?”

• Experiment 1 proved the pipeline end to end. Ordinary admin actions showed up within seconds (rules 5501 and 5402, both carrying MITRE tags), and I learned that file monitoring runs on a 12-hour schedule by default until you set realtime="yes".
• Experiment 2 proved the SIEM catches quiet moves. One useradd produced a level-8 “new user added” alert (rule 5902) plus file-integrity alerts on /etc/passwd, /etc/shadow and /etc/group at the same second, and even my cleanup was logged.
• Experiment 3 proved it catches loud ones too, and that detection and prevention are different jobs. Hydra produced a burst of level-5 failures that correlation rule 5763 escalated into a single level-10 brute-force alert (T1110), while fail2ban banned the attacker in the same second the alert fired.

If you take one warning from my mistakes, take this one. Empty or odd search results do not mean nothing happened. Check your time range (a dashboard reload silently reset mine to yesterday), check your filters, and check the tool’s default scan intervals before you conclude an attack was invisible. And write down the rule IDs your own install actually fires, because the versions in the documentation may not match yours.

5. Final thoughts

The coolest thing I learned was not a command or a tool. It was watching the difference between detection and prevention happen live on two screens. On one screen, Wazuh dutifully recorded every failed login my Hydra attack produced, then correlated the pile into a single level-10 brute-force alert timestamped 03:10:12.901. On the other screen, fail2ban simply ended the attack by banning the source at 03:10:12.544. Detection and response, inside the same second, from two tools that know nothing about each other. Neither one alone is the whole job. The prevention stops this attack, and the detection teaches you about every attack, including the ones prevention misses. Seeing that distinction with my own eyes, on machines I own, rewired how I think about blue-team work.

My one piece of advice for anyone about to build their first SIEM is simple. Baseline the boring stuff first. Before you run a single attack, spend twenty minutes doing completely normal things on your monitored machine and find every one of those actions in your SIEM. It feels like a waste of time right up until your first attack produces zero alerts and you are staring at an empty dashboard wondering what is broken. If you already know what “normal” looks like and how long it takes to appear, you will debug in minutes instead of hours.

My favorite discovery of this project is Wazuh’s own Proof of Concept guides. I found them through a link in the main documentation while setting up log collection, and they are fantastic. Each one walks you through a complete attack scenario, brute-forcing SSH, catching malware, spotting vulnerabilities, including the exact commands to run and the exact alerts that should fire. It is the rare documentation that lets you play instead of just read.

Two thank-yous, both genuinely earned. First, to Florian Roth (@Neo23x0), whose sysmon-config repository provided the Sysmon configuration that ran on my Windows endpoint throughout this project. Reading the comments in that config file is a security education all by itself. Second, to the Red Canary team, who build and maintain Atomic Red Team and the Invoke-AtomicRedTeam framework. They turned “practice attacking yourself” from an intimidating idea into a Tuesday-afternoon activity, and even though their T1136.001 tests sent me on a detour, the detour taught me more than a clean run would have.

6. References

1. Deploying Wazuh agents on Linux endpoints - Wazuh documentation team (Wazuh, Inc.). Date not listed. My step-by-step reference for enrolling play-server. The dashboard’s Deploy new agent flow plus this page is all you need. https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-linux.html
2. File Integrity Monitoring (syscheck) - Wazuh documentation team (Wazuh, Inc.). Date not listed. Where I learned that scans are scheduled by default and how to switch a directory to real-time monitoring. https://documentation.wazuh.com/current/user-manual/capabilities/file-integrity/index.html
3. Detecting a brute-force attack - Wazuh documentation team (Wazuh, Inc.). Date not listed. The playbook for Experiment 3. Note that it cites rules 5710 and 5712, while my Wazuh 4.14 install fired 5760 and 5763 for the same attack, a good reminder to verify rule IDs on your own version. https://documentation.wazuh.com/current/proof-of-concept-guide/detect-brute-force-attack.html
4. Blocking SSH brute-force attacks with Active Response - Wazuh documentation team (Wazuh, Inc.). Date not listed. How to make Wazuh itself ban attackers, my stretch goal after seeing fail2ban do it. https://documentation.wazuh.com/current/user-manual/capabilities/active-response/ar-use-cases/blocking-ssh-brute-force.html
5. T1136.001 atomic tests - Red Canary. Date not listed. The technique’s atomic test definitions. My local build only shipped the Windows variants, which is why I executed the technique manually on Linux. https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1136.001/T1136.001.md
6. Invoke-AtomicRedTeam - Red Canary. Date not listed. The PowerShell framework that executes atomic tests, including against remote endpoints. https://github.com/redcanaryco/invoke-atomicredteam
7. T1136.001: Create Account - Local Account - MITRE (MITRE Corporation). Date not listed. The ATT&CK technique page explaining why attackers create local accounts and what defenders should watch for. https://attack.mitre.org/techniques/T1136/001/
8. T1110: Brute Force - MITRE (MITRE Corporation). Date not listed. The technique behind Experiment 3. https://attack.mitre.org/techniques/T1110/
9. fail2ban - Fail2ban developers, originally Cyril Jaquier. Date not listed. The tool that banned my own attack machine. The repository documents the sshd jail I used. https://github.com/fail2ban/fail2ban
10. THC-Hydra - van Hauser and THC (The Hacker’s Choice). Date not listed. The brute-force tool I ran from my Kali workstation. https://github.com/vanhauser-thc/thc-hydra
11. Dashboards Query Language (DQL) - OpenSearch contributors (The OpenSearch Project). Date not listed. The reference I kept open while writing search filters in the Wazuh dashboard. https://opensearch.org/docs/latest/dashboards/dql/
12. Security - Firewalls (UFW) - Ubuntu Server documentation (Canonical). Date not listed. The UFW rules I used to lock play-server down to SSH only. https://documentation.ubuntu.com/server/how-to/security-firewalls/