The 2B model scored 10%. 93% of its answers were right

I expected small local models to be worse than a hosted decision API at Japanese classification. I did not expect most of the gap to be something other than judgement. On the hardest of three tasks one model scored 10.8% overall while getting 27 of the 29 answers it produced right. The other 221 rows it never answered.

What I ran, and what the accuracy number counted

Three Japanese tasks, 250 identical rows each. livedoor is 9-class news-topic classification (mean 1,308 characters), rakuten is 2-class review polarity (165), chabsa is 3-class financial-sentence polarity (100). Chance is 11.1%, 50.0% and 33.3%. Every system got the same prompt asking for one label as JSON, and a row counted correct only if the parsed label matched gold. Unparseable output counted as wrong.

How many rows actually produced an answer?

This is the column I should have read first:

Score only the rows that answered and the ranking moves. LFM2.5-2.6B's livedoor figure becomes 93.1% (27/29), and its rakuten figure 95.0%, above phi-4-mini's 89.6% and level with the hosted API's 94.4%. These models were not failing to judge. They failed to finish.

Why a small model returns nothing

I reproduced it on the mini-PC on 2026-10-26 (llama.cpp 60081bb, CPU-only) on the same livedoor row. With a 512-token budget the response body came back empty with finish_reason: length and 512 completion tokens spent. The tokens went to a populated reasoning_content field holding the model's step-by-step working: it ran out of budget before the answer.

Raise the budget to 2,048 and that row returns {"label": "MOVIE ENTER"}, correct, finish_reason: stop, 790 tokens. Same weights and prompt, temperature 0.

Raising the budget is not a complete fix. On the first 40 livedoor rows at 2,048: 28 answered (70%, against 11.6% at 512), but 11 still hit the cap and returned nothing, and one answered with an out-of-vocabulary label (IT Lifehack) that a strict parser rejects. Answer rate and answer validity are separate failure modes.

A reasoning-capable model spends tokens thinking, and a budget sized for the answer truncates it to nothing, which the harness scores as wrong. My earlier Gemma 4 E2B run shows the same shape at the format layer: 142 of 250 rows emitted valid JSON (56.8%), and 140 of those 142 were correct (98.6%).

Does a 2B model ever beat the API?

On accuracy, in my data, no. Paired McNemar on identical row ids, two-sided: phi-4-mini vs Jev −38.8pt on livedoor (p=1.2e-21), −4.8pt on rakuten (p=0.029), −2.4pt on chabsa (p=0.59); MiniCPM5-2B −40.4pt / −5.2pt (p=0.0072) / −12.4pt (p=0.0042); SemIf 4B −8.0pt (p=0.0012) / −2.4pt (p=0.070) / −10.0pt (p=0.014). The gap narrows as the task gets shorter and more lexical: 4.8 points on the 165-character polarity task, 39 to 40 on the long 9-class one.

Costs do not favour the small side either. Summed per-row latency over all 750 rows: MiniCPM5-2B 20 minutes, phi-4-mini 39, SemIf 4B 40, LFM2.5-2.6B 191.

Meanwhile 2,000 Jev requests across eight experimental arms produced zero unparseable responses. A closed-choice API cannot format-fail; a chat model can, and that difference dominates the table.

Measure answered rows before accuracy: a number averaging "wrong" and "silent" tells you nothing about which one you are buying. Background is in the zero-shot versus trained comparison and the CPU throughput limits.

Measured 2026-09-19 to 2026-10-26 on a Ryzen 9 7940HS mini-PC, llama.cpp 60081bb, Q4_K_M weights, temperature 0, 250 rows per task.

If you have sized a token budget for a local model's answers rather than its thinking, what budget actually cleared it?

Originally published at agentjournal.dev.