This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

enki-harness: a free, MIT-licensed test harness for AI filters that refuses to flatter you.

You wrote a safety filter, you wrote its tests, it passes. That proves less than you think: the person who wrote the code wrote the tests, so the tests only check the cases you already thought of.

enki-harness builds the harness for you and adds the parts you'd skip:

• Twins: for every case it must catch, a near-identical innocent case it must not flag
• Controls: dumb stand-ins (always-allow, always-block) that must fail, or the harness itself is broken
• Rules off: switches each rule off, alone and in pairs, to find dead rules and rules that cancel each other out
• Attacker slice: written by a different generator than your cases (leet, spacing, stretching)
• Falsifier: you write down what result would prove you wrong before it runs, and it's timestamped

No score until every gate is green. Rows first, the number last.

This week I added add-on #1: hb judge, for when the filter isn't code but an AI model. It sends every case to the model several times and reports flag rates, not one lucky answer.

Demo

Video plus a one-click Windows install: https://enkilabs.co.uk/harness/

Code

How I Built It

• Plain Python standard library for the tool itself; nothing to pip install to run it.
• Runs as a local web page (localhost only) and as an MCP server, so you can just ask your coding agent: "test my filter in filters/swear.py with enki-harness". One command registers it with Claude Code, VS Code/Copilot, Cursor, Codex, Claude Desktop and Windsurf, backing up every config first.
• hb judge talks to open-weight models running locally through Ollama (I used Llama Guard 3 8B). There are no API keys, no per-call cost, and nothing leaves the machine.

Why Does Open Innovation Matter?

Because a safety model you can't run yourself is one you can't measure.

I pointed hb judge at the moderation step in my own product's image feature. The prompts it screens are fixed sentences we wrote ourselves, all describing ordinary adult portraits, with no user text anywhere. Run each one several times and the screen still flagged them at random, about 1 time in 40: a different sentence each time, never the same one twice in a row. Every flag was a false alarm turning away a real user.

With an open-weight model I could run it hundreds of times locally for free, see the rows, and prove the noise was the model, not my wording. Against a closed API, each of those checks costs money and you're guessing.

The Lazy Payoff

It saved me from the "obvious" fix. I was about to spend an evening rewording prompts to dodge the filter. The runs showed the plain cartoon style, already live and working, gets flagged just as often, so rewording would never have fixed it. That screen was only ever judging our own text, so we removed it from that one feature. The checks that actually protect people still run, on the finished image.

Two lessons now built into the add-on:

• A warm-up call: my first run voided itself because the local model was still loading when the must-pass control ran. The harness refused to score it, which is exactly what it should do.
• Progress output: a 400-call run that prints nothing looks hung. Now it prints a count every 25 calls.

Prize Categories

Repo:https://github.com/stephen487/enki-harness

The one-click Windows install is live now at https://enkilabs.co.uk/harness/