My bot detector was blocking most real users, and my tests never noticed

I maintain Synapse Shield, an open-source bot detection engine (Python with a native Rust core and JS client SDKs) that evaluates mouse and keyboard dynamics: jerk variance, Fitts' law terminal deceleration, path curvature, and keystroke flight times.

On paper, my test suite looked bulletproof: over 500 unit tests, mutation testing, and adversarial red-team suites. Every test case representing a "human" was passing cleanly.

Then I finally did what I should have done from day one: I ran actual recorded mouse movements from real people through the engine.

The experiment

Getting labeled interaction data is tedious, so I used recorded desktop sessions from a public academic dataset (120 participants). I converted the raw trajectories to match exactly what the frontend SDK emits (30 ms throttling, integer pixel coordinates, click events) and fed them directly into the full evaluation pipeline.

The numbers

Every single participant was blocked as a bot at least once.

My unit tests never caught this because my "human" fixtures were synthetic. They were sine waves with Gaussian noise, shaped by my own assumptions about how people move mice. The heuristic rules were tuned against a caricature of human movement, not reality.

The culprit: a division by zero in disguise

Nearly all the false positives came from a single rule: the differential curvature check.

κ = |v × a| / |v|³

The formula divides by speed cubed. When a real person aims for a button, they slow down and pause. While the cursor is resting, browsers do not emit mousemove events. When the user makes a tiny 1-pixel adjustment after that 200-500 ms pause, the recorded speed drops to about 0.003 px/ms.

Because |v|³ is in the denominator, κ shoots up to 30-100. The engine interpreted that sudden spike as chaotic trajectory jitter and slapped a heavy bot penalty on the session.

In other words, the rule was triggering precisely when a human did what humans naturally do: stop and adjust before clicking.

To make matters worse, that branch barely caught real bots. Against automated scripts it flagged only 5-13% of them, and only when the bots happened to produce a similar stop artifact.

The fix

The chaotic curvature check now filters out near-stationary points. It only evaluates steps where the cursor actually traveled a meaningful distance (at least 5 px on both neighboring steps, with a minimum of 10 such points).

Instead of guessing a threshold, I swept values against the real recordings:

Across the entire decision engine:

• 6 s windows: 68.0% → 1.4% false positives
• 20 s windows: 87.8% → 2.7% false positives
• Synthetic bot patterns: 100% detection maintained

What an external review caught

After drafting the fix, an independent code review found real problems:

1. Ineffective regression tests: My first tests for minimum-jerk bots did not actually exercise the new moving-point path (one fixture was a straight line, the other was too clean). I rewrote them with coordinates rounded the way real bots produce them, and the tests now also assert that the fixture is not smooth on all points, so only the new branch can pass them.
2. Mobile touch false positives: A polynomial acceleration rule meant for desktop mouse trajectories was also evaluating mobile taps. Taps spaced seconds apart collapsed acceleration variance to near zero, blocking simulated mobile users about 97% of the time. Adding a touch check brought that down to 2-3%.
3. Rollover typing: When fast typists hold Shift or overlap keypresses, the inter-keystroke interval can be negative or zero. Because the check only looked for < 25ms, negative numbers tripped the "superhuman typing frequency" penalty. Adding a strict lower bound (0 < interval < 25ms) fixed it, and the same bug turned out to live in two more places (the Brave privacy cap and the threat label), so the check is now a single shared function.

Every fix has a test that fails on the old code.

Known limitations

A few blind spots remain:

• Quantized Bézier curves: On integer coordinates, curvature analysis alone only catches about 23.5% of smooth Bézier bots. Other layers (FFT spectral analysis, terminal deceleration) catch the rest, but curvature is not a magic bullet on its own. That was true before the fix too.
• Fast typing false positives: The superhuman typing threshold still trips on roughly 1.4% of real typing sessions. That is next on the list.
• Simulated mobile data: The mobile numbers come from a simulation model, not in-the-wild mobile recordings. Keyboard-only and older users have no real data yet either.
• Adversarial tooling: Detection was evaluated against mathematical generators, not yet against full stealth headless browsers like Playwright with ghost-cursor.

Takeaways

1. Synthetic test fixtures share your blind spots. If you generated your own "human" test data with math, your heuristics will overfit to your own mental model of how people behave.
2. Measure false positives before tightening detection. A bot detector that catches 99% of bots is useless if it blocks most legitimate customers.
3. Watch out for singularities in kinematic formulas. Anything with velocity or time in the denominator will blow up when a user stops moving.

These fixes are shipped in v0.10.6.

I built this with AI assistance (Claude) for implementation and review. The detection approach and design decisions are mine.