Mistral Small: 58 empty replies in 300 tool calls, and the retry loop

On 300 public function-calling tasks, Mistral Small (2603) returned an empty reply 58 times, even after two retries. That is 19.3% of tasks with nothing to parse, and it is why a model priced at $0.15 per million input tokens ended up at $0.000098 per successful task against $0.000051 for GPT-6 Luna, which passes 93.0% with no retries at all.

Here is the part of the harness that decides what a retry costs, because the number depends entirely on it.

What counts as a failure, and what gets retried. One rule: a reply that cannot be read is retried, a reply that can be read but is wrong is final. Production code cannot know an answer is wrong, so the harness doesn't either.

function parseJsonLoose(text) {
  if (typeof text !== "string" || text.trim() === "") return { ok: false, reason: "empty output" };
  let t = text.trim();
  const fence = t.match(/```
{% endraw %}
(?:json)?\s*([\s\S]*?)
{% raw %}
```/i);
  if (fence) t = fence[1].trim();
  const a = t.indexOf("{");
  const z = t.lastIndexOf("}");
  if (a === -1 || z <= a) return { ok: false, reason: "no JSON object in output" };
  ...

The loop. Every attempt is billed and recorded; the loop stops on the first reply that parses.

for (let a = 0; a <= config.retries; a++) {
  const res = await transport({ model, messages, max_tokens: maxTokens, temperature: config.temperature, extra: extra && extra.body });
  const { cost, estimated } = res.ok ? attemptCost(res, model, prices) : { cost: 0, estimated: false };
  const rec = { ms: res.ms, cost, cost_estimated: estimated, status: res.status };
  // (error-status branch omitted)
  const j = judge(config, res.text, row);
  rec.valid = j.valid;
  attempts.push(rec);
  if (j.valid) {
    last = { outcome: j.pass ? "pass" : "wrong", reason: j.reason, output: res.text };
    break;
  }
  last = { outcome: "invalid", reason: j.reason, output: res.text };
}
...
cost: attempts.reduce((s, x) => s + x.cost, 0),

config.retries is 2 and the temperature is 0, so a task gets at most three identical requests. Cost per successful task is every attempt's cost divided by the tasks that passed, not by the attempts that did.

What it did to Mistral Small. 434 attempts for 300 tasks: 134 retries. Fifty-eight tasks used all three attempts and still ended empty, so 116 of those 134 retries (58 tasks x 2; arithmetic on the stored counts) went to tasks that never recovered. Only 18 retries went to tasks that came back with something readable. Of the tasks that needed a retry, 58 never produced a readable reply and at most 18 did (18 retries on recovered tasks cannot cover more than 18 tasks), so at least three in four stayed unreadable (58 / (58 + 18) = 76.3%; arithmetic on the stored counts). Retries were 23.9% of the model's spend. Before them, its cost works out to $0.0000742 per success ($0.020673 x (1 - 0.239) / 212, a calculation on stored values), still above Luna's $0.000051: the retries are not the whole gap, but they are the part a first-attempt token count never shows.

The other eight models in the run: six had no retries at all, DeepSeek V4.1 Flash had 2, Qwen 3.8 Flash had 5, and none ended on an unreadable reply. So this is not "tool calling is hard for cheap models"; it is one model's behaviour on this prompt.

What the empty replies look like. Row tc-2, the easiest kind in the set:

tools: generate_random_number(min, max), calculate_loan_payment(loan_amount, interest_rate, loan_term)
user:  Hi, I need a random number between 1 and 100.
expected: {"name":"generate_random_number","arguments":{"min":1,"max":100}}
got:   (empty)   -- 3 attempts

tc-10, tc-19, tc-21 and tc-37 are empty in the same way: short conversations, one clearly matching tool. Mistral Small's other 30 misses are ordinary ones (14 right value in different words, 10 a different value, 5 a number written differently, 1 missing argument), and the other eight models make the same kinds of mistakes.

Run it yourself. The 300 rows, the prompt, the script and the prices are on the verdict page. To test only the row above:

grep '"id":"tc-2"' glaive-tool-call.jsonl > one.jsonl
export OPENROUTER_API_KEY=...
node rerun-glaive-tool-call.mjs --dataset ./one.jsonl

That sends one row to each of the nine models in the run for well under a cent (the whole run cost $0.503184 for 300 rows across nine models, about $0.0017 a row). Results at temperature 0 are not perfectly deterministic, so a second run is a result too.

Want the same verdict on your own data? Import 10-100 of your input/expected pairs, add your current model, and the page names the cheapest model that clears your bar, with cost per successful task, retries counted, and the failures it will have. Free during early access, you pay OpenRouter for the calls with your own key. Tool-call verdict, failures and re-run files: https://clearbar.grain64.com/verdicts/glaive-tool-call/?utm_source=devto&utm_campaign=devto-empty-reply

The stored result keeps the reply text but not the finish reason, so the page does not say why these replies are empty. If you run tc-2 and get a readable call from Mistral Small, or you know what finish_reason and usage.completion_tokens an empty reply from this model shows on OpenRouter, put it in the comments: that is the missing piece.

From Clearbar, by Grain64.