Dev.to AI 🤖 Ai 👁 0 📖 9 min read

Claw-Eval on Hugging Face: The Top 7 Rows Are All Model-Card Reports, and 0 of 22 Rows Are Verified

TL;DR: The official Claw-Eval board on Hugging Face (claw-eval/Claw-Eval) lists 22 rows. On Nov 22, 2026, 0 of 22 are marked verified and 18 come from open pull requests. The top 7 rows (66.5 to 81.4) all cite a model ca

TL;DR: The official Claw-Eval board on Hugging Face (claw-eval/Claw-Eval) lists 22 rows. On Nov 22, 2026, 0 of 22 are marked verified and 18 come from open pull requests. The top 7 rows (66.5 to 81.4) all cite a model card, which means the vendor's own report. The best row that cites the benchmark's own leaderboard site is rank 8, at 64.0. One vendor family holds 6 of the top 10 rows. The board scores the 161-task general split, so one task is worth 0.62 points, and rows from the same checkpoint family, the same quantized repo or the same model in two modes share the table without a column that says so. Read the top of this board as "who reported what", not as one controlled race.

Claw-Eval board rows by rank: model-card sources in blue, leaderboard-site sources in grey

What does the Claw-Eval board on Hugging Face actually rank?

Claw-Eval is an agent benchmark: a model acts in a sandbox with tools, files and services, and a grader checks what it did. The dataset card describes three splits: general (161 tasks across 24 categories such as communication, finance, ops and productivity), multimodal (101 tasks) and multi_turn (38 tasks). That is 300 tasks in total. The paper is linked from the dataset page.

The Hugging Face board for this dataset is keyed to the general split. You can see it in the .eval_results files behind the rows: the leaderboard-sourced entries carry the note "Pass^3% | N=3 | 161 tasks", and a model card that reports a single unlabeled Claw-Eval number gets mapped to general by the person who wrote the YAML. One such file says so in plain words: "the exact task split is not specified."

Two details of that note matter for reading scores:

  • Pass^3 is a strict metric. A task counts only if all three independent runs pass it. It is not an average of three scores, so it is not the same quantity as an avg@3 or pass@1 number.
  • n = 161 means one task flips a score by 0.62 points. Scores come with one or two decimals, but the unit underneath is coarse.

The board JSON itself has no field for the metric, the number of runs or the split. Those facts live in free-text notes inside individual YAML files, and not every row has one.

Who reported the top of the Claw-Eval board?

Here is the top of the board's JSON, fetched on Nov 22, 2026:

Rank Model Score Source cited Open PR
1 ornith-ai/Ornith-1.5-397B 81.4 model card #1
2 ornith-ai/Ornith-1.0-397B 77.1 model card #6
3 MiniMaxAI/MiniMax-M3 74.5 model card none
4 dots-studio/dots3-note-prev 73.4 model card #5
5 ornith-ai/Ornith-1.5-35B-A3B 72.5 model card #1
6 ornith-ai/Ornith-1.0-35B 69.8 model card #8
7 ornith-ai/Ornith-1.5-9B 66.5 model card #1
8 XiaomiMiMo/MiMo-V2.5-Pro 64.0 claw-eval.github.io #10
9 inclusionAI/Ring-2.6-1T 63.82 model card none
10 ornith-ai/Ornith-1.0-9B 63.1 model card #3

Count the sources across all 22 rows and the split is even: 11 rows cite a Hugging Face model card, and 11 cite the benchmark's own site, claw-eval.github.io. But the two groups sit in different places. Ranks 1 to 7 are all model-card rows. Every leaderboard-site row is at rank 8 or lower, and the best of them is 64.0. Ring-2.6-1T at rank 9 is a model-card row, so the model-card group also reaches below the line.

That is not a flaw in any single number. A vendor reporting its own run on its own model card is normal, and the cards for these rows describe their settings. The Ornith-1.5-397B card, for example, gives the Claw-Eval footnote as "temp=0.6 and 256K context". What the footnote does not state is the split, the number of runs or whether the number is Pass^3. The rows from the benchmark's site state Pass^3 | N=3 | 161 tasks outright. So the top 7 and the rest of the table are different kinds of evidence sharing one rank column.

Gap to the next row down, in points

The gap chart shows how the table is shaped. Between rank 1 and rank 2 there are 4.3 points, about 7 tasks. After that the steps are small: 2.6, 1.1, 0.9, then 2.7, 3.3 and 2.5. From rank 8 down to rank 14 the total spread is 2.5 points (64.0 to 61.5), about 4 tasks. The only cliff is the last one, 49.7 to 6.8 for the Nemotron row, a 42.9-point drop that is more likely a harness or format mismatch than a measure of capability. I did not check that run, so treat that as a guess about the cause.

Is one vendor family crowding the top 10?

Six Ornith rows sit at ranks 1, 2, 5, 6, 7 and 10, which is 6 of the top 10 and 6 of 22 rows overall. They are two release generations (1.0 and 1.5) across several sizes (397B, 35B and 9B). Each is a separate model repo with its own card and its own row.

This says nothing bad about those models. It does change what "rank 5" means. Ranks 5, 6 and 7 are three different models from one lab, scored under one lab's setup, and they are separated from the next vendors' rows by margins that the table cannot tell apart from run noise (see the next section). If you want to know how many different vendors are in the top 10, the answer is 5: Ornith, MiniMax, dots, Xiaomi and inclusionAI. The table makes it look like 10 independent entries.

A quick way to count this yourself:

import json, math, urllib.request
from collections import Counter

URL = "https://huggingface.co/api/datasets/claw-eval/Claw-Eval/leaderboard"
req = urllib.request.Request(URL, headers={"User-Agent": "board-audit"})
rows = json.load(urllib.request.urlopen(req))

N = 161  # tasks in the general split
print("rows:", len(rows), "verified:", sum(r["verified"] for r in rows),
      "open PRs:", sum(1 for r in rows if r.get("pullRequest")))
print(Counter(r["source"]["name"].lower().endswith("model card") for r in rows))
top10 = rows[:10]
print(Counter(r["modelId"].split("/")[0] for r in top10))
print("one task =", round(100 / N, 2), "points")

def wilson(p, n=N, z=1.96):
    ph = p / 100
    c = (ph + z * z / (2 * n)) / (1 + z * z / n)
    h = z * math.sqrt(ph * (1 - ph) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return round((c - h) * 100, 1), round((c + h) * 100, 1)

for r in rows:
    tasks = r["value"] * N / 100
    print(r["rank"], r["modelId"], r["value"], round(tasks, 2), wilson(r["value"]))

Would the order survive a different draw of 161 tasks?

A fair question for any 161-task score is how far it would move on a different sample of similar tasks. A Wilson interval on a pass rate with n = 161 gives a rough scale. This is illustrative only: the board stores no per-task data, a Pass^3 score is not a plain pass rate, and models are compared on the same tasks, so real paired differences are tighter than these intervals suggest.

On that scale, rank 1 (81.4) has a range of 74.7 to 86.7. Rank 3 (74.5) has 67.3 to 80.6. Rank 7 (66.5) has 58.9 to 73.3. The rank 1 range overlaps the ranges of ranks 2 to 6. The rank 8 range (56.3 to 71.0) contains the scores of ranks 9 to 19 and also the rank 7 score. In plain terms, the rows from rank 8 to rank 19 are not separable from each other on this evidence, and ranks 3 to 7 are not clearly separable either.

Illustrative 95% ranges for ranks 1 to 12

Nothing here says the models are equal. A paired test on shared tasks could separate a 3-point real edge, and the Claw-Eval site publishes more than this one number. The point is narrower: the Hugging Face board gives you a single score per row and no paired data, so rank numbers alone cannot make that call.

Are the scores whole task counts?

If every row were one clean Pass^3 count over 161 tasks, each score would be a multiple of 0.621. Multiply each score by 161 and divide by 100. Fourteen of the 22 rows land within 0.1 of a whole number: 81.4 is 131.05 tasks, 74.5 is 119.94, 66.5 is 107.06, 62.7 is 100.95, 62.1 is 99.98, 58.4 is 94.02, 49.7 is 80.02. Eight do not: 77.1 is 124.13, 73.4 is 118.17, 72.5 is 116.72, 69.8 is 112.38, 63.82 is 102.75, 63.1 is 101.59, 58.2 is 93.7 and 57.7 is 92.9.

This is a weak test. With a step of 0.62 points, a random decimal lands within 0.1 of a whole task roughly one time in three, so 14 of 22 is above chance, but it does not prove which rows are clean counts. What it does support is that the board mixes rows whose protocol is documented with rows whose protocol is not, which fits the source split above. Ring-2.6-1T (63.82) is the clearest non-integer case, and its YAML note does say "Pass^3 | N=3 | 161 tasks" with "high reasoning effort", so a non-integer score can come from a documented setup too.

Why do some repos appear twice, or share a score?

Three patterns in the table are worth knowing before you quote a rank.

A derivative repo can carry the base score. Rank 11 is zai-org/GLM-5.1 at 62.7. Rank 12 is INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound, a quantized community repo, also at 62.7. Its YAML is the same entry as the base model's: same date, same note, same submitting user, and the parameter count in the board JSON is identical (753,864,139,008). A quantized checkpoint is a different artifact from the original, and the same number here is a copied entry rather than a separate run. The result is one measurement shown twice, which takes up a rank slot.

One repo can appear in two modes. openpangu/openPangu-2.0-Flash is at rank 16 (58.2) and rank 19 (57.7). Its model card lists a Thinking column and a Non-Thinking column for Claw-Eval, Pass^3, with 57.7 and 58.2. So the Non-Thinking mode scores 0.5 points above Thinking, and the board shows the same model id twice with no mode column. A reader who only sees the table cannot tell them apart.

Parameter metadata is not reliable. The board JSON's num_parameters field gives 664,944 for Ornith-1.0-35B (rank 6) and 1,469,680 for Ornith-1.0-9B (rank 10). Those are not model sizes, since the names say 35B and 9B. They look like counts picked up by the metadata scan from something other than the full model. If you filter or plot this board by parameter count, those two rows land at under 2 million parameters.

How should you use this board?

A short checklist that works for this board and for other Hugging Face official boards:

  1. Group rows by source before comparing. Model-card rows and leaderboard-site rows answer different questions. Compare within a group first.
  2. Check the split and the metric. For Claw-Eval, general with Pass^3 over N=3 and 161 tasks is the documented setting. A card that says only "Claw-Eval" has not said that.
  3. Convert a gap to tasks. Divide the gap by 0.62. Under 5 tasks (3 points), treat the rows as a tier.
  4. Collapse duplicates. Merge the same repo in two modes, and quantized copies that repeat the base score, before counting entries.
  5. Check the PR state. 18 of 22 rows come from open pull requests, and 0 are verified, so the platform has not signed off on any row.

FAQ

What does "verified" mean on a Hugging Face benchmark board?
It is a flag on each row. On the Claw-Eval board, 0 of 22 rows carry it on Nov 22, 2026, and 18 rows come from open pull requests rather than the main branch.

Is Claw-Eval's score on Hugging Face the same as its website score?
Not necessarily. The board maps entries to the general split, and entries from the site carry a Pass^3, N=3, 161-task note. Model-card entries may use another split, metric or run count, and some cards do not say.

How many tasks separate rank 1 from rank 2?
Rank 1 is 81.4 and rank 2 is 77.1, a 4.3-point gap, about 7 tasks of 161. Ranks 3 to 7 are separated by 8.0 points in total, about 13 tasks.

Does a quantized model have its own Claw-Eval score?
Only if someone ran it. The GLM-5.1 quantized repo on this board shows the same 62.7 as the base model, with an identical YAML entry, so it reads as an inherited number.

Why is the last row 6.8?
The Nemotron row sits 42.9 points below the next one. The board does not explain it, and the cause is not documented in the row. A harness or output-format mismatch is a plausible explanation, but that is a guess.

Further reading

Sources: the board JSON at https://huggingface.co/api/datasets/claw-eval/Claw-Eval/leaderboard (fetched Nov 22, 2026), the Claw-Eval dataset page, the Claw-Eval site, the Ornith-1.5-397B card, the openPangu-2.0-Flash card and the GLM-5.1 quantized repo.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.