SWE-bench Multilingual on Hugging Face: 15 of 34 Scores Are Not a Count Out of 300 Tasks

TL;DR

• On November 9, 2026 the Hugging Face SWE-bench Multilingual board lists 34 rows from 34 repos, none verified. 30 rows are read from a pull request that is still open (all 30 opened by one of two accounts) and 4 from a repo's main branch. 32 rows cite a model card and 2 cite a paper.
• The benchmark has exactly 300 tasks, so a single run can only score a multiple of 0.333 points. 15 of the 34 listed values are not one, including ranks 1 and 2 (88.5 and 82.9). They may be averages over several runs or use a different denominator. The files do not say which.
• Two models also appear on the benchmark's official table, where every row ran the same open agent. GLM-5 is listed at 73.3 on the board and 69.7 officially (220 against 209 tasks). Kimi-K2.5 is 73.0 against 67.3 (219 against 202). The board names a harness for 2 of its 34 rows.
• On the official table's per-task data, the top two rows are 2 tasks apart (218 and 216 of 300), and the paired standard error of that gap is 2.45 points. A 7 point spread across ranks 2 to 12 on the board is about 21 tasks, which is a tier more than a ranking.
• The board pools nine languages that have between 12 and 44 tasks each. On the official table five different rows lead or tie for the lead in at least one language.

What does the SWE-bench Multilingual board on Hugging Face rank?

SWE-bench Multilingual is a set of 300 real GitHub issue and pull request pairs from 41 repositories in 9 languages. A system gets the repository at the commit before the fix and the issue text, produces a patch, and the task counts as resolved if the repository's own tests pass afterwards. The score is the share of the 300 tasks resolved.

The Hugging Face page for the dataset carries a leaderboard assembled from files that model repos publish themselves: a YAML file under .eval_results/ that gives a value, a source link and, optionally, a date and notes. The board JSON is public at /api/datasets/SWE-bench/SWE-bench_Multilingual/leaderboard, so anyone can pull the same rows I used. All values are higher-is-better percentages. Every row has verified: false.

Where the rows come from is worth a sentence. 30 of the 34 sit on pull requests that I checked through the discussions API and that are all still open. Two accounts opened all 30, 23 by one and 7 by the other. The other 4 rows (ranks 7, 14, 17 and 34) are on main branches. The source is a model card for 32 rows and a paper table for the other two (ranks 3 and 26). Six of the 34 rows come from one organization, five from a second and five from a third, so three organizations hold 16 of the 34 rows, on a board with 15 organizations.

Can a score on a 300-task benchmark be 88.5?

Only if it is not a single run. Each task is pass or fail, so one run produces k/300 for a whole number k, and k/300 expressed as a percentage moves in steps of 0.333. A value reported to one decimal place can come from a single run only if some whole k rounds to it.

Here is the check, along with the official comparison used later in this post:

import json, math, urllib.request

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    return json.loads(urllib.request.urlopen(req, timeout=120).read())

N = 300  # tasks in SWE-bench Multilingual
board = get("https://huggingface.co/api/datasets/SWE-bench/SWE-bench_Multilingual/leaderboard")

def counts_that_round_to(v):
    dec = max(1, len(repr(float(v)).split(".")[1]))  # 81 is stored as 81.0, one decimal
    half = 0.5 * 10 ** -dec
    return [k for k in range(N + 1) if abs(100 * k / N - v) <= half + 1e-9]

bad = [(r["rank"], r["modelId"], r["value"]) for r in board if not counts_that_round_to(r["value"])]
print(len(board), "rows;", len(bad), "cannot be k/300 at their stated precision")
for b in bad: print("  ", *b)

LB = "https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json"
official = [x for x in get(LB)["leaderboards"] if x["name"] == "Multilingual"][0]["results"]
off = {x["name"]: x for x in official}
for repo, name in (("zai-org/GLM-5", "GLM 5"), ("moonshotai/Kimi-K2.5", "Kimi K2.5")):
    hf = next(r["value"] for r in board if r["modelId"] == repo)
    k = counts_that_round_to(hf)[0]
    print(f"{repo}: board {hf} (= {k}/300), official {off[name]['resolved']} (= {round(off[name]['resolved'] * N / 100)}/300)")

a, b = (off[n]["per_instance_details"] for n in ("Gemini 3 Flash", "Claude 4.6 Opus"))
only_a = sum(a[i]["resolved"] and not b[i]["resolved"] for i in a)
only_b = sum(b[i]["resolved"] and not a[i]["resolved"] for i in a)
se = 100 * math.sqrt(only_a + only_b - (only_a - only_b) ** 2 / N) / N
print("official 1 vs 2: only-first", only_a, "only-second", only_b, "paired SE", round(se, 2), "points")

On November 9 it prints:

34 rows; 15 cannot be k/300 at their stated precision
   1 inclusionAI/Ling-3.1-flash 88.5
   2 tencent/Hy4-preview 82.9
   5 ornith-ai/Ornith-1.5-397B 79.6
   6 ornith-ai/Ornith-1.0-397B 78.9
   10 MiniMaxAI/MiniMax-M2.7 76.5
   11 deepseek-ai/DeepSeek-V4-Pro 76.2
   12 tencent/Hy3 75.8
   17 inclusionAI/Ling-3.0-flash 72.4
   19 ornith-ai/Ornith-1.5-35B-A3B 71.4
   23 nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 67.72
   25 Qwen/Qwen3.6-35B-A3B 67.2
   27 poolside/Laguna-XS-2.1 63.1
   29 ornith-ai/Ornith-1.5-9B 54.4
   32 nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 45.8
   34 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 36.47
zai-org/GLM-5: board 73.3 (= 220/300), official 69.7 (= 209/300)
moonshotai/Kimi-K2.5: board 73 (= 219/300), official 67.3 (= 202/300)
official 1 vs 2: only-first 28 only-second 26 paired SE 2.45 points

Seven of the top twelve ranks and four of the top six are on that list. A miss is informative and a hit is not: a one-decimal number lands on some k/300 about 3 times in 10 by luck, so the 19 rows that do fit prove nothing on their own. The 15 that do not fit cannot be one pass count.

That does not make them wrong. A model card often reports the mean of several runs, and a mean of five runs steps in units of 0.067 points. A card may also use a different denominator, for instance after dropping tasks whose environment failed. Both are legitimate, and neither is visible in the file. The YAML has a notes field and the two rows that fill it with harness details are ranks 4 (mini-SWE-agent, 256K context) and 26 (OpenHands). Nothing on the board tells a reader whether 88.5 is one run, an average, or something else, and that matters because a mean of several runs is usually a less noisy number than a single run.

How large is a gap of a few points on 300 tasks?

Smaller than the ranking implies. For a single run at 245 of 300 (81.67, rank 3), an unpaired 95% Wilson interval runs from 76.9 to 85.6. The row at rank 13, 227 of 300 or 75.7, has an interval of 70.5 to 80.2. The two overlap by more than 3 points even though 10 ranks separate them.

The two rows are different models scored on the same tasks, so an unpaired interval is the conservative choice. The official table lets me measure how much smaller the paired number is, because it publishes per-task results. For its top two rows, Gemini 3 Flash with 218 tasks and Claude 4.6 Opus with 216, 190 tasks are solved by both, 28 only by the first and 26 only by the second. The paired standard error of the 0.67 point difference is 2.45 points. A lead has to exceed roughly 5 points, about 15 tasks, before it clears 1.96 standard errors for two rows this similar. On the board, ranks 3 to 13 span 18 tasks.

The same pattern holds one level down. Among all 13 official rows, 30 of the 300 tasks (10%) are solved by none and 69 (23%) by every row, so the rows differ on the other 201 tasks. A ranking built from rows that agree on a third of the benchmark is mostly reading disagreement on the rest.

Do the same models score the same on the official table?

Two repos on the board also have rows on the benchmark's own table, which I read from the leaderboard data file in the benchmark's website repository. Every row there ran the same agent, mini-SWE-agent, and the file includes per-task outcomes. Its latest entry is dated February 20, 2026, so any new submission since then is not in it. It lists 13 rows, and 12 of them carry the maintainers' checked flag.

The model-card figures are higher in both cases, and the board does not say what harness produced them. A different agent, a different tool budget or a different prompt can all move a score by several points, and a model card is free to report the setup that suits it best. None of that is an error. It means the board mixes numbers measured under unlike conditions and presents them in one column.

The ordering is the more interesting part. Officially GLM-5 resolves 7 tasks more than Kimi-K2.5. On the board the same two rows are 1 task apart. The paired standard error between those two official rows is 2.47 points, so even the 7 task official gap is under one standard error. Neither table can say which is better. The other two open-weight rows on the official table, MiniMax M2.5 and DeepSeek V3.2, have no row on the board, which lists newer versions of both, so the overlap is two models.

Why do nine languages matter for one pooled number?

Because the languages do not carry equal weight, and each is a small sample. The dataset card lists the task counts: Ruby 44, Java 43, Rust 43, PHP 43, Go 42, C 30, JavaScript 26, TypeScript 17 and C++ 12. Five languages hold 215 of the 300 tasks, so the pooled score is mostly a Ruby, Java, Rust, PHP and Go score. C++ is 4 percent of the benchmark, with 11 of its 12 tasks coming from one repository.

I mapped each of the 41 repos to the languages in the card, and I checked that the counts reproduce the card's table exactly. Then I computed per-language rates for every official row from its per-task data. The chart shows the top two.

Claude 4.6 Opus is 92.3 on JavaScript (24 of 26) against 69.2 (18 of 26) for Gemini 3 Flash. On C++ the order flips: 91.7 (11 of 12) against 58.3 (7 of 12). Both are differences of 4 to 6 tasks, and each of them is larger than the 2 task gap that separates the two rows overall. Across all 13 official rows, five different rows lead or tie for the lead in at least one of the nine languages: Gemini 3 Flash, Claude 4.6 Opus, GLM 5, Kimi K2.5 and GPT 5.2 Codex. The overall leader leads or ties in 4 of the 9.

None of that says the pooled number is wrong. It says that a user who writes Go or C++ and picks from a pooled ranking is reading a number dominated by other languages, and that the per-language cells are small enough that one task moves them by 2 to 8 points.

What else on the board is worth knowing?

• Ties get different ranks. Ranks 14 and 15 are both 73.3, and ranks 21 and 22 are both 69.3. At 300 tasks that is the same count, 220 and 208.
• The size column is not always right. The board's num_parameters for ornith-ai/Ornith-1.0-35B is 664,944 and for ornith-ai/Ornith-1.0-9B it is 1,469,680. The Hub's own safetensors metadata gives the same numbers for these two repos, which have 16 and 4 weight shards. Treat the size column as a hint.
• Dates are mostly missing. The date field is filled for 9 of 34 rows, so a reader cannot tell a February score from an October one.
• The top row is a different generation from the official table's top row. Rank 1 at 88.5 sits 15.8 points above the official table's best of 72.7. The rows are different models, run by different parties with different agents, and the official table ends in February, so this is not a like for like gap.

What can a reader do with this board?

1. Convert to task counts. Multiply by 3 to get tasks out of 300. If the result is not close to a whole number, the row is an average or uses another denominator.
2. Read gaps in tasks, not points. Two rows 5 points apart are 15 tasks apart. For two models that agree on most tasks, the paired noise on that gap is around 2.5 points.
3. Look for a harness. Only 2 of 34 rows name one. Without it, a model-card figure and a table row are not the same experiment.
4. Cross-check with per-task data where it exists. The official file publishes which tasks each row solved, and it is the only place to see where two models actually differ.
5. Weight languages by what you write. If your code is in Go or C++, the pooled number tells you little about it, and C++ has 12 tasks.
6. Treat 0 verified as 0 verified. All 34 rows are self-reported, and all 30 pull request rows are still open.

For the board itself, three small changes would remove most of the ambiguity: show the number of runs and the denominator next to each value, show the harness, and show the date for every row.

FAQ

What is the SWE-bench Multilingual leaderboard on Hugging Face?
It is a board on the dataset page for SWE-bench Multilingual, which has 300 real issue and pull request tasks in 9 programming languages. Model repos publish YAML result files and the board lists them, sorted by percentage resolved.

How many tasks does SWE-bench Multilingual have, and why does it matter for scores?
300 tasks. A single run therefore gives a score in steps of 0.333 points, and a gap of 1 point is 3 tasks. A listed value that is not a multiple of that step is an average over runs or uses another denominator.

Are the Hugging Face scores comparable with the official SWE-bench table?
Not directly. The official table runs every submission with the same open agent and publishes per-task results. The board takes figures from model cards, names a harness for 2 of 34 rows, and has no verified rows. For GLM-5 and Kimi-K2.5, the two models I could compare, the board values are 3.6 and 5.7 points higher.

Does the top row on the board mean that model is the best at multilingual code repair?
It means the model's card reports the highest number. The top row's 88.5 cannot be a single count out of 300, and a lead of a few points between top rows is within the noise on 300 tasks.

How do I check a score on a small benchmark myself?
Multiply the percentage by the number of tasks and see whether you get a whole number. Then, where per-task data exists, count the tasks only one of two models solved and use that count to size the noise on their gap. The code above does both.

Further reading

• SWE-bench Verified on Hugging Face: 9 Self-Reported Rows Top the Official Table's Best, and 6 Repos Fill 16 of 78 Rows
• SWE-Bench Pro Has Two Versions: What the Hugging Face Leaderboard's 47 Rows Mix Together