My bot detector blocked 1 in 5 real humans. My tests said 1.4%.
My open-source bot detector had a test suite with 950+ passing tests, a 1D-CNN, a Rust core with SIMD, and a README full of green checkmarks. It was also blocking about 19% of real humans. Not bots. People. Moving a mo
My open-source bot detector had a test suite with 950+ passing tests, a 1D-CNN, a Rust core with SIMD, and a README full of green checkmarks.
It was also blocking about 19% of real humans.
Not bots. People. Moving a mouse normally.
This post covers how I found that out, why my tests couldn't see it, and what changed in Synapse Shield v0.11.0. Every number below was measured before and after the change, on the same seeds.
π€ What is Synapse Shield?
It's a self-hosted alternative to CAPTCHAs and Cloudflare Turnstile. It doesn't show puzzles. It looks at how a visitor moves, clicks and types, never what they type, and scores the session on your own server.
Setup is one line of middleware plus a small browser SDK:
from fastapi import FastAPI
from synapse_shield import SynapseShieldMiddleware
app = FastAPI()
app.add_middleware(SynapseShieldMiddleware, protected_paths=["/api/login"])
<script src="/static/synapse-sdk.js"></script>
<script>
SynapseShield.init();
SynapseShield.protectFetch("/api/login");
</script>
That's the pitch. Now the uncomfortable part.
π₯ The bug that hid behind a green test suite
The engine has five layers. One of them is a small 1D-CNN that looks at 60-step movement sequences.
On my machine its false-positive rate was 1.4%. That looked fine.
Then I replayed the public SapiMouse dataset (120 real people, real mouse recordings) through the exact weights that ship inside the package. Those are the weights a Docker deployment actually loads.
| Real-human windows blocked | What I thought | What production did |
|---|---|---|
| 6-second windows | 1.4% | 19.0% |
| 20-second windows | ~0.5% | 19.2% |
The cause took a while to see:
- The bundled weights had been trained only on synthetic Gaussian noise.
- My local runs loaded a different weights file from the working directory.
- A unit test (
test_train.py) kept overwriting that local file with a model so saturated it was effectively switched off.
So the "1.4%" number came from a model that did nothing. The model my users would get was blocking one person in five.
The fix:
- Weights now load in a fixed, explicit order: argument, then env var, then bundled file. A stray file in the working directory is ignored, with a warning.
- Retraining refuses to write a saturated model.
- The CNN now runs in shadow mode by default. It still scores every request and reports its raw score, but it can't block anyone until it earns that on real data.
Result: 19.0% β 1.4% (6 s) and 19.2% β 0.5% (20 s) on real people. Detection of the synthetic bots stayed at 100%.
π‘ Lesson: a test suite proves your code does what you think. It doesn't prove what you think is true. Only real data does that.
π οΈ 6 more things real users taught me
Once I started measuring against real humans, the same pattern kept showing up. The rule looked reasonable, the synthetic tests passed, and it was still unfair to someone.
1. Touchscreen laptops were treated as liars
If the browser said "I support touch" but no touch events arrived, the session was marked as a spoofer.
A stricter penalty I tried would have blocked 100% of touch-laptop users and 71% of iPad-with-keyboard users. Those are normal devices.
Now such sessions are simply scored as desktop. I verified that the touch flag gives a bot no advantage: scores are identical with and without it.
2. Everyone behind one NAT was punished for one bot
Offices, universities and mobile carriers (CGNAT) put thousands of people behind one IP. One bot on that IP raised everyone's risk.
Now the short-term penalty is keyed to a hashed IP + User-Agent pair, and only hard violations count toward an IP ban. Under 60 requests per minute of mixed traffic, human loss went 2.47% β 0.08%. Scripted bots are still banned on their 4th request.
3. Short mouse movements looked "too perfect"
A single short flick of the mouse has very few points. Too few for curvature and frequency rules to mean anything.
50% of real single movements were blocked. Those rules now carry a capped weight on short traces, and borderline cases get a silent proof-of-work instead of a block. Result: 50.0% β 4.9%.
4. Pressing Enter looked like a bot click
When you press Enter on a form, Chrome fires a synthetic click at (0, 0). The engine saw a teleporting cursor.
Synthetic clicks within 50 ms of a key press are now ignored. In that scenario blocks went 100% β 5.4%.
5. Replaying a real human recording worked every time
This one's the attacker's side. If you record a real person's mouse trace and replay it from a fresh IP and User-Agent, it really does look human.
v0.11.0 adds two defences, neither of which stores raw traces:
- a keyed 64-bit trajectory fingerprint, consumed only when a request is allowed;
- a session shape hash (SimHash) that survives scaling, noise and time warping.
A recorded trace replayed with fresh IPs was allowed 100% β 10% of the time, and only on its first use. In-session repeats are caught from the 3rd request. Measured human false positives: 0.02%.
6. Some client signals could only help the attacker
A few browser-side hints lowered the risk score, for example a flag claiming "this is a common letter pair". Anything the client sends can be forged.
The rule now: client signals can only raise risk, never lower it. Real-typist false positives on the CMU keystroke dataset didn't change (0.034%).
β What I didn't ship (and why)
Some ideas looked good on paper and failed when measured:
- Tremor detection via an FFT frequency band: it cost bot detection and didn't fix the people it was meant to help.
- Extra keystroke rules: at most 8% more bots caught at a 0.5% real-human false-positive cost. Not worth it.
- Fail-closed on a Redis outage: it sounds secure, but it turns a cache hiccup into a self-inflicted outage. Fail-open is now documented instead.
Deleting a feature you spent a day on is painful. Shipping one that blocks real people is worse.
π§Ύ Honest limits
I'd rather you hear these from me than find them in production:
- All bot numbers are synthetic. Human false-positive numbers come from real public datasets, but I have no labelled real-bot traffic yet. Read "100% detection" as "catches these generators".
- About 18% of very short (3β10 point) single movements are still blocked.
- Sophisticated replay still works partly. An attacker who records many real sessions and replays each one once with small changes looks human to the behavioural layer. The system raises their cost; it isn't a wall. Keep rate limits and MFA.
-
The plain-JS
protectFetchdoesn't auto-solve proof-of-work challenges yet. The React and Vue SDKs do (submitProtected()).
π Try it
pip install "synapse-shield[server]"
synapse-shield run --port 8000
Open http://127.0.0.1:8000 for the live dashboard, then run synapse-shield test in a second terminal to throw the built-in attack simulations at it.
Not ready to block anyone yet? Turn on shadow mode. It scores real traffic and reports what would have been blocked, per device cohort, without touching a single user.
π GitHub: 0xStoic-bit/Synapse_Shield (MIT, Python 3.10β3.14, prebuilt wheels for Linux, macOS and Windows)
π¬ Over to you
I'm curious about two things:
- Have you ever shipped a security rule that turned out to punish real users? How did you find out?
- Does anyone have labelled real bot traffic (even a small sample) that could be used to test bot detection honestly? It's the biggest gap in this project.
If this saved you from one more "select all traffic lights" puzzle, a β on GitHub helps other developers find it.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.