Dev.to Security 🔐 Cybersecurity 👁 0 📖 6 min read

I let a cloud AI audit my server logs without showing it a single real IP, username or key

I run a small VPS on the open internet. As part of learning to build with AI, I pulled a week of its logs into SQLite and asked two questions: who is hitting this box, and did anyone get in? I wanted a cloud model to wr

I run a small VPS on the open internet. As part of learning to build with AI, I pulled a week of its logs into SQLite and asked two questions: who is hitting this box, and did anyone get in?

I wanted a cloud model to write the analysis, but logs are full of things I won't paste into someone else's AI: IPs, usernames, my hostname, sometimes a key. So I set one rule. The cloud only sees masked data, and a script has to prove the masking worked before anything is sent.

Claude wrote the Python scripts, because I can't write them from a blank file yet. Then I ran every step by hand, read each output, and checked what each model claimed against my own data.

Two models, two jobs

Where Job Sees real data?
Qwen3 14B My Mac, offline, no tools Writes a masking plan, later reviews a masked sample A few sample rows, locally
Code My Mac Applies the plan, scans for leaks, blocks the cloud call Yes, locally
DeepSeek Cloud Writes the threat report from masked aggregates Never

The LLM never rewrites the logs itself. A model rewriting 3,000 lines will miss one or invent one, and you won't know which. Qwen decides what should happen to each column, plain code does the masking, and a separate check confirms nothing got through.

The workflow

  1. Pull seven days of sshd and kernel logs with journalctl, read-only. Noise is sampled, rare events are kept in full.
  2. Load everything into one SQLite table, with a flag saying whether a source is on my private allowlist.
  3. Qwen gets the schema and five sample rows and returns JSON only: keep, pseudonymise, generalise or drop for each column and each pattern.
  4. Code applies the plan. The same IP always becomes the same ID (IP_0042) via a keyed hash, and the key stays on my Mac. Emails, MACs, keys and fingerprints are dropped.
  5. A leak scan searches the masked file for every real value, then runs a separate regex sweep for anything shaped like an IP, email, MAC or key.
  6. The cloud script refuses to run unless the scan passed for that exact masked file.
  7. DeepSeek analyses the masked aggregates. I map the IDs back only on my Mac.

This is part of the plan Qwen returned in step 3:

"columns": {"ts": "generalise", "ip": "pseudonymise", "username": "pseudonymise",
            "src_class": "keep", "message": "pseudonymise"},
"message_patterns": {"ipv4": "pseudonymise", "key_token": "drop", "email": "drop"}

If Qwen ever says keep for an IP, username or host, the code overrides it. It never had to.

And this is the core of the gate in step 6. The scan stores a hash of the exact masked file it checked, and the cloud script compares it before sending anything:

report = json.load(open(SCAN_REPORT))
sha = hashlib.sha256(open(MASKED_CSV, "rb").read()).hexdigest()
if not report["pass"] or report["masked_file_sha256"] != sha:
    sys.exit("BLOCKED: leak scan has not passed for the current masked file")

The final scan found 0 of 833 real IPs and 0 of 219 testable usernames in the masked file, and the regex sweep had 0 hits. "Testable" means long enough, and different enough from fixed log words, that a string search means something. The other 13 usernames couldn't be checked that way.

What the report found

About 46,000 events in a week. Because the noise was sampled, my code scaled the counts back up before sending, and the report repeated those estimates. They landed close to my own full totals:

Estimate Real
Invalid-user attempts 3,905 3,936
Blocked probes to other ports 30,849 30,912
Connections ended after repeated failed authentication 149 149

That match checks my sampling and scaling code, not DeepSeek's arithmetic.

DeepSeek mapped the activity to two MITRE ATT&CK techniques: T1110 (password guessing and spraying, from about 400 sources) and T1595 (scanning, across about 720 ports).

Most of it was drive-by traffic. About 90% of sources stayed at low volume and appeared on fewer than three days. The interesting ones were the 25 sources that returned on three or more days. One made over 1,500 events with more than 50 usernames. Another came back on all 7 days and only ever tried one username. Eight other sources worked in bursts instead, each firing 700+ events within an hour while cycling through just 5 or 6 usernames.

The attackers also look like they share wordlists. Four usernames made up 47% of the sampled attempts, and 15 usernames were tried by three or more unrelated sources.

Every successful sign-in came from my own network, and I found none from an unfamiliar source. That's as far as logs go, though. They can't show what happens after a login.

Checking any of these claims is one query. This is the source that came back every day:

SELECT COUNT(*), COUNT(DISTINCT substr(ts,1,10)), COUNT(DISTINCT username)
FROM events WHERE ip = 'IP_0537' AND service = 'sshd';   -- 31 events, 7 days, 1 username

Both models made things up

Neither model leaked anything, but both said things the data can't support.

Qwen, as the reviewer, raised 22 issues on 25 masked rows. Nineteen were the same sentence: "this pseudonym could be linked to a real value." It called the pseudonyms sequential. They are numbered, but in keyed-hash order, so the number tells you nothing about the real IP. It also warned that some ports might reveal what software I run, but those are ports the attackers chose to probe. It found no real leak, which matched the scan.

DeepSeek, as the analyst, kept the numbers straight and got the stories wrong. It named attacker tools (Hydra, Medusa, Masscan) that masked logs can't identify. It guessed at weaknesses in a setup it never saw. It called the spikes "scheduled campaigns", when a few bursts landing in the same hours fits the data just as well. And it said the host was "only accessed by authorized sources". What the data shows is logins from my network, which is a weaker claim.

What I learned

The small offline model was enough to write a masking plan. The cloud model was better at analysis but should never see identities. Code beat both of them at being exact.

Qwen's review was mostly noise. The leak scan was the only judge I could trust, because it's dumb and deterministic: a real value is either in the file or it isn't.

The checker shouldn't share code with the thing it checks. An address:port string slipped past my masker, and the leak scan missed it too because it used the same IP pattern. The independent regex sweep caught it before anything was sent.

My first sample hid the events that mattered. Keeping 1 in 8 lines threw away rare ones like successful logins. Now I keep every rare event, sample only the noise, and multiply back.

And I only caught the made-up claims by checking the report against my own data. Someone still has to read the output.

Limits

This is pseudonymisation, not anonymisation. With my key it's reversible, and timing and volumes still go to the cloud. That's fine for my own VPS logs, not for regulated data.

The plan is built from a handful of sample rows, so a value type that isn't in them can slip through, and the regex sweep only knows the patterns I gave it. The cloud model also only sees behaviour, not my configuration. It can describe the attackers, but it can't audit me.

Cost: about 30 seconds for the plan, 2.5 minutes for the review, 26 to 32 seconds for the cloud report, and $0.0004 to $0.0008 for the cloud call.

The part I'd reuse is the gate: the cloud call won't run unless an independent check says zero. If you've put a masking step in front of a cloud model, I'd like to hear what you use as the final check, and whether it ever caught something your masker missed.

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.