This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

Fixing a prompt by hand is a loop I hate. I rewrite it, test it by hand, redeploy, and find that I fixed two cases and broke a third. Then I do it again.

So this week I wrote the laziest prompt I could for a bank's message router, and I never edited it again. Open-weight models rewrote it in two optimizer iterations. I checked every result on messages the models never saw.

What I Built

PromptSloth is one Python file that runs a prompt-improvement loop on AcruxCore, the open-source LLM-ops platform I work on. The script talks to AcruxCore only through its MCP server. MCP (Model Context Protocol) is an open JSON-RPC protocol that lets a client call a server's tools.

The router's job is intent routing. A customer writes "I still have not received my new card", and the router must answer card_arrival, not card_delivery_estimate. The data is Banking77 from PolyAI: 13,083 real banking messages with 77 intents, under CC-BY-4.0.

The lazy prompt, v1, is two sentences: the list of 77 labels, then "Reply with the intent label only, nothing else."

The router never changes

AcruxCore stores each prompt as numbered versions, and an alias is a name that points at one version. My router asks for an alias, never a version number. Moving an alias changes the router's prompt without a code change or a redeploy. The aliases here are lazy (v1), fixed (v2) and fixed2 (v3).

Every model is open-weight

Demo

The recording shows three real commands. report prints the second iteration's scores. Then eval runs v1 and v3 on the same 24 held-out messages. With v1, most replies are wrapped in braces, and only 2 of 24 are both right and parseable. With v3, 15 of 24 are right, and every reply is a bare label.

In the second iteration, two of the three rewrites scored below v2 on the judge's training-set score:

fixed (v2, baseline)  57.1
candidate-A           64.3  beats baseline: True   <- promoted as v3
candidate-B           50.6  beats baseline: False
candidate-C           55.8  beats baseline: False

AcruxCore did not promote B or C, so the router never used them.

What each iteration fixed

Iteration 1 fixed the format

With the lazy v1, the 4B model often wrapped its answer, as in {card_arrival}. On 20% of all messages the label inside the braces was right, but a strict parser rejects the whole reply. The first rewrite told the model to "respond with only the exact label (no punctuation, no surrounding text)". The extra characters disappeared. The share of wrong labels stayed at about 44%, so iteration 1 did not improve the routing.

Iteration 2 fixed the routing

For the second iteration I wrote a new rubric, the paragraph of instructions that a dataset gives the optimizer and the judge. It asked for a one-line definition of every intent, saying how the intent differs from its nearest neighbour. The winning rewrite added 77 lines like these:

card_arrival = ordered card has not arrived yet.
card_delivery_estimate = user asks how long delivery will take before the card arrives.
activate_my_card = user wants to enable a newly received card.

The optimizer wrote those definitions from the 154 training examples. On held-out messages, wrong labels fell from 45% to 40%.

A reply counts as correct when it is the right label and nothing else. In every cell, the two runs are at most 3.2 points apart.

Is the gain real?

A paired test on every message

I compared each pair of prompts message by message, with an exact McNemar test. That test looks only at the messages that one prompt got right and the other got wrong.

• 4B, v1 → v3: v3 got 88 messages right that v1 got wrong, and 16 wrong that v1 got right (p < 0.0001).
• 4B, v2 → v3, routing only: v3 won 36 messages and lost 17 (p = 0.013). The definitions helped after the format was already fixed.
• 27B, v1 → v3: v3 won 29 messages and lost 13 (p = 0.02). Both iterations scored rewrites only on the 4B model, so the 27B gain comes from a prompt that was never tuned for 27B.

The judge agreed with the test it never saw

The judge scored v1, v2 and v3 at 36, 57 and 64 on the training messages. Exact match on the held-out messages gave 37%, 55% and 60%. The judge never saw a held-out message, and the two measurements still rank the three versions the same way.

The optimizer never sees a test message either. The 308 held-out messages come from Banking77's test split, and the 154 training messages come from its train split.

How I Built It

Only MCP, no SDK

promptsloth.py has six subcommands: setup, dataset, optimize, report, promote and eval. Every subcommand sends JSON-RPC to POST /api/v1/mcp through mcp.py, a 40-line client built on requests. The MCP guide shows the same server connected to Claude.

Four tool calls per iteration

1. create_prompt and commit_prompt_version store the lazy v1.
2. create_dataset and add_dataset_example load 154 training messages, two per intent, with the rubric.
3. optimize_prompt starts an iteration, and get_optimize_report returns the scores. The optimizer drafts three rewrites, and the judge scores each rewrite and the current version on the training messages.
4. promote_optimize_candidate commits the winner as a new version and moves an alias to it. AcruxCore refuses a rewrite that did not beat the version it was compared with.

The optimizer guide covers the same loop from the dashboard, and the evaluation page lists what else it scores.

Scoring without a judge

The eval step calls run_prompt once per held-out message at temperature 0 and compares the reply with the label in Python:

text, err = mcp.tool("run_prompt", {
    "prompt": "promptsloth-banking77", "alias": "fixed2",
    "model": "ps-gemma-3-4b-it", "variables": {"message": row["text"]},
    "temperature": 0, "max_tokens": 40,
})

sample.py draws both sets from Banking77 with seed 42: two messages per intent from the train split and four per intent from the test split.

The one step outside MCP

ps-gemma-3-4b-it is the name I gave Gemma 3 4B in AcruxCore's gateway, the component that routes every model call to a provider and records its cost. The MCP server has no tool to register a model, so I made one REST call per model.

What went wrong

• I first included a third size, Gemma 3 12B. One upstream provider returned empty replies for it, and one run needed 1,504 retries. Its scores changed with the provider that answered, so I removed 12B from the results. The script now retries an empty reply and counts each retry.
• The lazy prompt's wrapped replies hid what each iteration changed. The script now reports two scores: routing only, with the wrapping removed, and routing plus a clean reply. The routing-only score showed that iteration 1 fixed the format and iteration 2 fixed the routing.

The Lazy Payoff

My own writing was a prompt of two sentences and two rubric paragraphs. The optimizer drafted six rewrites, the judge scored 1,232 answers (4 versions × 154 messages × 2 iterations), and AcruxCore refused the two rewrites that scored below v2.

The gateway's usage report (get_gateway_usage) puts the whole day at $0.38 for 11,660 model calls. That total covers both iterations, every held-out run, the demo and the wasted 12B retries. Each v3 request is longer than a v1 request, about 1,380 tokens against 575. So the 4B router costs $0.046 per 1,000 messages with v3, against $0.018 with v1.

Why Does Open Innovation Matter?

The router can leave the cloud

Gemma 3 4B is small enough to run on a laptop. I ran it through OpenRouter for this measurement, not on my own machine. The accuracy is still the accuracy of the same weights you can download and host.

I could inspect every layer

When the 12B scores jumped between runs, the traces showed empty replies from one provider. When iteration 1's gain looked large, the raw replies showed braces, not better routing. A closed optimizer that returns only a score would have shown neither.

Anyone can repeat it

Banking77 is public, and the sampling uses a fixed seed. AcruxCore is Apache-2.0 and self-hostable, with its source on GitHub, and every model is open-weight.

Limits

• With v3, the 4B model (60%) still scores below the 27B model with the lazy v1 (69%).
• Banking77 is public, so the models may have seen it during training.
• Each cell has two runs. Even at temperature 0, two runs differed by up to 3.2 points.
• I wrote both rubrics. The first one named a few example confusions, such as card arrival against card activation.
• I tested one task. A task with fewer and more distinct labels may gain less from definitions.

Code

The repo, talhaanwarch/promptsloth (Apache-2.0), holds every script and every result file behind the charts. To run it, get an AcruxCore API key with the editor role and run export ACRUXCORE_API_KEY=.... Then run python3 sample.py and the subcommands in the order the script's docstring lists.

When you fix a prompt today, how do you check that the fix did not break other cases? I tested on a held-out split, and I would like to hear what works on your team. 🦥

Prize Categories

Best Use of Gemma. Gemma 3 4B and 27B route the messages, and Gemma 4 31B is the judge whose scores decided each promotion. OpenRouter served all three Gemma models.

AI assistance: I built PromptSloth with an AI coding assistant, and this post was drafted with an AI assistant from the project's own result files. I reviewed it, and every number comes from the repo.