DeepSWE on Hugging Face: Rank 1 Is an Agent Layer 21 Points Above Its Base Model, and 22 of 34 Rows Name No Scaffold
TL;DR On November 18, 2026 the Hugging Face board for the DeepSWE coding-agent benchmark lists 34 rows from 34 repos (19 organisations), each a single percentage over 113 tasks. 20 rows come from pull requests that ar
TL;DR
- On November 18, 2026 the Hugging Face board for the DeepSWE coding-agent benchmark lists 34 rows from 34 repos (19 organisations), each a single percentage over 113 tasks. 20 rows come from pull requests that are still open, 14 sit on
main, and all 34 carryverified: false. -
Rank 1 is a different kind of row. It is
PCL-RSI-lab/Seshat-agent, 90.3 (102 of 113 tasks), 13.3 points above rank 2. That is the biggest step on the board; the next biggest is 9.87. The repo holds a README and two YAML files and no weights, and its own note reads "Seshat + GLM-5.3". - The base model it wraps is on the same board at rank 12. GLM-5.3 shows 66.9 on its model card. The benchmark's own site shows 68.96 pass@1 and 87.6 pass@4 for it. The rank 1 row is 21.3 points above the official pass@1 and 3 tasks above the official best-of-4.
- The rest of the board is consistent with the official site. Five more rows can be matched to it, and they agree within 2.4 points (mean 1.1). So the board's data pipeline is fine; the problem is that one column holds different kinds of measurement.
- 22 of 34 notes name no agent scaffold. 16 of 33 steps between neighbouring rows are 1.0 point or less, and one task is 0.88 point. A single row's 95% interval is about 17 points wide.
What is the DeepSWE leaderboard on Hugging Face?
DeepSWE is a set of 113 long-horizon engineering tasks drawn from 91 repositories in 5 languages. The benchmark's own site (deepswe.datacurve.ai) runs every model through one open scaffold, mini-swe-agent, four attempts per task, and publishes two numbers: pass@1, the pass rate over all scored attempts, and pass@4, the share of tasks with at least one passing attempt.
The Hugging Face board is assembled differently. Model repos carry a file such as .eval_results/deep-swe.yaml with one value and a free-text notes field, and the board at GET /api/datasets/datacurve/deep-swe/leaderboard sorts those values in descending order. There is no column for scaffold, number of attempts, benchmark version or metric type. Whatever the author knows about those goes into notes, if anywhere.
That design is easy to join and easy to extend, and it works well when every row is the same kind of number. The question for this audit is how many of the 34 rows are.
Why is rank 1 on the DeepSWE board a different kind of row?
Four facts from public files, no inference needed:
- The board row is
PCL-RSI-lab/Seshat-agent. Its file list isREADME.md,.gitattributesand two eval YAML files. There are no model weights, and the repo shows 0 downloads. - The notes field says "Seshat + GLM-5.3; DeepSWE-v1.1 (113), resolve rate 102/113=90.3%". The README describes an agent layer that adds a reusable knowledge base around a base model, and it quotes the uplift on GLM-5.3 as 59.3% to 90.3%.
- The linked evidence dataset says
agent: mini-swe-agent,model: openai/GLM-5.3,reasoning_effort: max, 102 resolved, 11 unresolved, 4 empty patches, generated 2026-11-02. It contains patches and pass/fail flags only, with no trajectories or prompts. - Every other row is a model checkpoint with a score from its own card or from the official site.
So rank 1 is "an agent system built on GLM-5.3", ranked in the same column as the checkpoints it sits above. Nothing in the board marks the difference.
I am not claiming that 90.3 is wrong. Outputs-only evidence is a common way to publish, and a different layer around a model can legitimately move a score. The point is that a reader cannot tell from the board what the number measures, and the checkable comparison is awkward:
| Row | Board | Official pass@1 | Official pass@4 | Board minus pass@1 |
|---|---|---|---|---|
| GLM-5.3 (rank 12) | 66.9 | 68.96 | 87.6 | -2.06 |
| Kimi-K3 (rank 11) | 67.5 | 68.51 | 89.4 | -1.01 |
| GLM-5.3-Flash (rank 14) | 63.4 | 63.39 | 85.0 | +0.01 |
| GLM-5.2 (rank 28) | 46.2 | 43.78 | 77.0 | +2.42 |
| Kimi-K2.7-Code (rank 32) | 30.53 | 30.53 | 61.1 | 0.00 |
| Seshat + GLM-5.3 (rank 1) | 90.3 | 68.96 (GLM-5.3) | 87.6 (GLM-5.3) | +21.34 |
Official numbers are the max reasoning-effort configuration of each model on the benchmark site's page, which was generated on 2026-09-22. 90.3 is 102 tasks, and GLM-5.3's official pass@4 is 99 tasks (87.6% of 113). The rank 1 row is above what four tries of the plain model reach on the official scaffold.
One more detail from the repo's own README: it puts the starting agent at 59.3%, which is below both the model card's 66.9 and the official 68.96. The uplift the repo quotes (31.0 points) is therefore larger than the gap to the official base (21.3 points). Both are consistent with a real improvement, and both are consistent with a different baseline setup. The public files do not settle which.
Do the other DeepSWE rows agree with the official site?
Yes, and that is the useful finding about the board itself. Five rows can be matched one to one to an official configuration, and the differences are -2.06, -1.01, +0.01, +2.42 and 0.00 points. Kimi-K2.7-Code's row copies the official figure exactly, and it is the only row whose note says how the number was made: "Pass@1 over 452 attempts on 113 tasks, mini-swe-agent harness".
The model-card numbers are close to the official site's pass@1 which fits labs running the same open scaffold. The GLM-5.3 card, for example, states the mini-swe-agent harness with temperature=0.95, top_p=1.0, a 6-hour timeout and a 400K context window.
The gap between the two official numbers is where a column with no metric field loses information. Across the 65 model, scaffold and effort configurations on the official page, pass@4 is a median of 21.9 points above pass@1, and 57 of the 65 are more than 15 points apart. A row that reports a best-of-k figure and a row that reports a mean over attempts differ by about twice the span between ranks 2 and 12 (77.0 down to 66.9 is 10.1 points). Nothing on the board says which one a given row is.
How many of the 34 rows are comparable with each other?
Less than the sort order suggests. Four checks, all computed from the board JSON and the YAML files it points to:
-
Scaffold. 12 of 34 notes name one (7
mini-swe-agent, plus Claude Code, SWE-agent, LingCode, NexAU, and one "Claude Code / mini-SWE-agent (best of two)"). 22 name none, including rank 1 and one row that says only "the authors' preferred agent harness". The benchmark's own site runs one scaffold for everything, so a card score from another harness is a different experiment. -
Version. The official site has separate v1.1 and v1 leaderboards. 13 notes say v1.1, and 21 do not say. Every YAML uses the same
task_id,deep_swe, so the schema cannot separate them either. -
Source. 20 of 34 rows come from pull requests on model repos that are still open at fetch time, so the model owner has not merged them. 14 are on
main. -
Duplicates and lineage. Ranks 14 and 15 both show 63.4. Rank 15 is
inference-optimization/GLM-5.3-Flash-MXFP4xFP8, a quantized checkpoint whose YAML points at the original model's card as its source, with the same date and the same note. The board lists it with the original's score, and nothing says the quantized weights were run. The official site's number for the original is 63.39. Five of the 34 rows are DeepSeek snapshots, and ranks 21 and 22 also tie at 58.4.
Is a 1-point gap on a 113-task board a real difference?
No. One task is 0.88 point. 16 of the 33 steps between neighbouring rows are 1.0 point or less, and 2 are exact ties. A pass rate of 66.9% on 113 tasks (76 tasks) has a Wilson 95% interval of 58.2 to 75.2, about 17 points wide. Rank 1's 102 tasks gives 83.4 to 94.5. The official site reports a run-to-run 95% interval of plus or minus 3.0 points for GLM-5.3 (65.9 to 72.0, from four repeated passes), and 28 of the 33 steps on the board are 3.0 points or less.
The ordering that survives these checks is coarse: a top group of cards between 71.7 and 77.0, a middle between 53 and 68, and a tail below 47. Within each group, rank is mostly noise.
How should I read the DeepSWE board?
- Open the repo behind the row. Check whether it holds weights or only a result file, and read the notes field before the number.
- Find the metric. Mean over attempts, best of k and a single run are different quantities. On this benchmark the first two differ by about 22 points.
- Check the scaffold and version. If the notes do not name them, treat the row as unmatched.
- Compare to the benchmark's own site for the same model. Five of six matched here, and the exception is the one that deserves the question.
- Merge duplicates. A quantized checkpoint that carries the base model's card score is one measurement, not two rows.
- Do not rank within 2 points. On 113 tasks that is two to three problems.
The check takes a few lines of Python against the public API:
import json, math, re, urllib.request
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
return json.load(urllib.request.urlopen(req, timeout=60))
board = get("https://huggingface.co/api/datasets/datacurve/deep-swe/leaderboard")
named = re.compile(r"mini[- ]swe|claude code|swe-agent|lingcode|nexau", re.I)
print(len(board), "rows;", sum(1 for r in board if r.get("pullRequest")), "from open PRs;",
sum(1 for r in board if not named.search(r.get("notes") or "")), "name no harness")
top = board[0]
print("rank 1:", top["modelId"], top["value"], "|", top["notes"])
def wilson(k, n, z=1.96):
p = k / n; d = 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return round(100 * (c - h), 1), round(100 * (c + h), 1)
for r in board:
if r["modelId"] in ("zai-org/GLM-5.3", "PCL-RSI-lab/Seshat-agent"):
k = round(r["value"] * 113 / 100)
print(r["rank"], r["modelId"], r["value"], f"= {k}/113", "95% CI", wilson(k, 113))
Output on November 18, 2026:
34 rows; 20 from open PRs; 22 name no harness
rank 1: PCL-RSI-lab/Seshat-agent 90.3 | Seshat + GLM-5.3; DeepSWE-v1.1 (113), resolve rate 102/113=90.3%
1 PCL-RSI-lab/Seshat-agent 90.3 = 102/113 95% CI (83.4, 94.5)
12 zai-org/GLM-5.3 66.9 = 76/113 95% CI (58.2, 75.2)
The pullRequest field is present for rows read from a pull request. I confirmed their status separately through the discussions API, which showed all 20 as open.
What would make the board harder to misread?
The information needed is mostly already in the files and gets dropped when the board is assembled. Four optional fields in the eval YAML would remove most of the ambiguity: the metric (pass@1, pass@k, single run), the scaffold, the benchmark version, and a row type (model checkpoint or agent system). A board could then sort within a row type, show system rows in their own band, and fold a derivative checkpoint that reuses its base model's source into that model's entry. None of that changes what any author publishes; it changes what the reader is told.
FAQ
What is the DeepSWE leaderboard on Hugging Face?
It is the board on the dataset page for datacurve/deep-swe, a benchmark of 113 long-horizon software tasks. Model repos publish YAML result files and the board sorts the values. As of November 18, 2026 it has 34 rows from 34 repos.
Who is at rank 1 on the Hugging Face DeepSWE board?
PCL-RSI-lab/Seshat-agent with 90.3 (102 of 113 tasks), a repo that holds a README and result files and describes an agent layer around GLM-5.3. The plain GLM-5.3 model is at rank 12 with 66.9.
Is the 90.3 DeepSWE score wrong?
The public files do not show that. The evidence dataset contains patches and pass/fail flags but no trajectories, so a reader cannot re-derive it. What can be said is that it is 21.3 points above the official pass@1 of the base model and 3 tasks above its official pass@4, so it is not comparable with a plain model score.
What is the difference between pass@1 and pass@4 on DeepSWE?
On the official site, pass@1 is the pass rate over all scored attempts, and pass@4 is the share of the 113 tasks solved by at least one of four attempts. Across 65 configurations pass@4 is a median of 21.9 points higher.
Do Hugging Face model-card DeepSWE scores match the official site?
For the five matched rows, yes, within 2.4 points (mean 1.1). The mismatch on this board comes from a row of a different kind, not from card scores drifting.
How many tasks separate two DeepSWE rows that are 1 point apart?
About one. A single task is 0.88 point of 113, and a 66.9% result has a 95% interval of 58.2 to 75.2.
Further reading
- SWE-bench Verified on Hugging Face: 9 Self-Reported Rows Top the Official Table's Best, and 6 Repos Fill 16 of 78 Rows
- WildClawBench on Hugging Face: Rank 1 Is a Runtime, Because Scores, Minutes and Dollars Share One Ranked Column
Links
- Board API: https://huggingface.co/api/datasets/datacurve/deep-swe/leaderboard
- Benchmark site: https://deepswe.datacurve.ai/
- Rank 1 repo: https://huggingface.co/PCL-RSI-lab/Seshat-agent
- Rank 1 evidence dataset: https://huggingface.co/datasets/PCL-RSI-lab/DeepSWE-v1.1-Evaluation-Evidence
- GLM-5.3 model card: https://huggingface.co/zai-org/GLM-5.3
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.