SWE-bench Verified on Hugging Face: 9 Self-Reported Rows Top the Official Table's Best, and 6 Repos Fill 16 of 78 Rows

Orange bars are rows whose YAML file exists only in an open pull request. Blue bars are rows already on the model repo's main branch. The axis starts at 74, so bar lengths exaggerate the gaps; read the numbers.

SWE-bench Verified is a human-validated subset of 500 issue and pull request pairs from popular Python repositories. Evaluation runs the repo's unit tests against the patch an agent produces. Hugging Face lists it as a benchmark dataset, so a model repo can publish a result in.eval_results/*.yaml and the dataset's board collects it.

I pulled the board on November 5, 2026 (KST) from the JSON behind the leaderboard view. It has 78 rows. The dataset's own eval.yaml defines two task ids, swe_bench_%_resolved and swe_bench_average_cost, and names the swe-bench evaluation framework. It does not name an agent, a scaffold, a number of attempts or a context length. That matters because for this benchmark the scaffold is part of the measurement. A model alone does not edit a repository; an agent loop does, and each team brings its own.

Of the 78 notes fields, 54 are empty. Nine name a harness or an official run. So most rows tell the reader nothing about how the number was produced, and the card behind each row is the only place to look.

No row is. The verified field is false on all 78. I fetched the discussion behind every row that carries a pullRequest number.

All 51 pull requests are open. The oldest was opened on February 16, 2026, which is 262 days before my pull; the median age is 153 days, and 36 of the 51 are older than 90 days. The top row is one of them: Ornith-1.5-397B (86.0) comes from a pull request opened on August 19, 2026 by a Hugging Face account, not by the model's owner.

I am not criticizing the design. Opening a pull request with results taken from a model card is a reasonable way to bootstrap a board, and each file is labeled with its source. The cost is that the ranking blends results the owner has accepted with results someone else transcribed, and nothing in the rank column separates them. In the top ten, five rows are on main and five are open pull requests.

The swebench.com site publishes its own Verified table. The data file behind it has 180 entries, the newest dated February 26, 2026, and the highest value is 79.2, held by two agent submissions from December 2025 (Sonar Foundation Agent with Claude Opus 4.5, and live-SWE-agent with Claude Opus 4.5). Neither is marked as checked. The highest entry among the runs that use the site's bash-only mini-SWE-agent is 76.8 (Claude Opus 4.5, February 17, 2026).

On the Hugging Face board, 9 rows are above 79.2 and 18 are above 76.8. None of that proves anything is wrong. Newer models are stronger, and the site's table has not moved since February. But the two tables are not measuring the same thing. The site's mini-SWE-agent runs give every model the same minimal agent with one bash tool. The board rows come from whatever harness each team used, and the cards say so when you read them:

The anti-hacking measures in the rank 1 card are good practice. They also show that the number belongs to a specific environment. Among the top-ranked rows whose setup I could read, there are two different agent loops (OpenHands and Claude Code), and neither is the one the site uses.

The board itself contains the answer, because some repos have several rows.

NVIDIA's Nemotron-3-Super-120B card reports three SWE-bench Verified results for the same weights: 60.47 with the OpenHands harness, 59.20 with OpenCode and 53.73 with Codex. That is 6.74 points between harnesses, or about 34 tasks. The board shows them at ranks 60, 63 and 68. The RedHatAI repo with the same name carries the same three rows, with the source URL pointing at NVIDIA's card, at ranks 61, 64 and 69. Six rows, three measurements, two repos.

The gpt-oss repos use a different axis. OpenAI's model card gives low, medium and high reasoning effort, and the board lists all three. gpt-oss-120b has 47.9, 52.6 and 62.4 at ranks 75, 72 and 56. gpt-oss-20b has 37.4, 53.2 and 60.7 at ranks 77, 70 and 59. The spread is 14.5 points for the larger model and 23.3 for the smaller one. Nobody is misreporting here; the effort level is in the file name. But a reader who looks up gpt-oss-20b gets a different answer depending on which of its three rows they land on, and the best and worst rows are 18 ranks apart. openPangu-2.0-Flash has the same shape in smaller form: 63.1 thinking and 57.6 non-thinking, 5.5 points.

GLM-5 is the cleanest case. Its single YAML file contains two entries. One is 72.8, sourced to swebench.com with the note "high reasoning, official". The other is 77.8, sourced to the model card with the note "Z.ai reported number". The board shows 77.8 at rank 15 and 72.8 at rank 34, 19 ranks and 25 tasks apart.

The 72.8 matches the swebench.com data file exactly: mini-SWE-agent v2.0.0 with glm-5 at high effort, dated February 17, 2026. The same is true of MiniMax-M2.5 (75.8) and Kimi-K2.5 (70.8). So these three rows are the only ones on the board I could trace to a run under a common harness, and for GLM-5 that run scores 5.0 points under the maker's own figure. The maker's number may well come from a better scaffold, and I did not try to reproduce either one. The point is that the board puts both on the same scale.

Ranks 2 and 3 are Macaron-V1-Venti (on main) and Macaron-V1-Coding-Venti (an open pull request by nielsr), both 85.6. I did not check whether the two repos hold the same weights. If they do, two of the top three ranks are one result. If they do not, the identical value is a coincidence worth a footnote.

Ranks 46 and 50 are poolside/Laguna-XS.2 at 69.9 and MuVeraAI/Laguna-XS.2 at 68.2. The second repo's YAML gives the poolside model card as its source. That card's table currently says 69.9, mean pass@1 over 4 runs, with a 262,144 token context. The MuVeraAI repo carries its own copy of the card, last modified April 28, 2026, with a 131,072 token context and 68.2 in the same table. Its downloads field reads 16, against 905,143 for the poolside repo. So the 68.2 looks like an earlier version of a card that was later updated, copied into another repo, and given its own rank. I can see the mismatch between the source link and the number; I cannot see why the copy exists.

The benchmark has 500 tasks, so one task is 0.2 points. The arithmetic below is a sampling calculation for a single run on a fixed set. Prompt and scaffold variance is larger, as the Nemotron rows show.

Ranks 8 and 9 are tied at 80.2, and rank 6 is 2 tasks above them. A paired comparison on the same 500 tasks has a narrower interval than the independent one I show, but it needs per-task results, which a board row does not carry. So the top 14 rows, from 86.0 to 78.0, contain only a few differences that I would call larger than noise, and none of them is established by a verified run.

In February 2026 OpenAI said it would no longer treat SWE-bench Verified as a measure of frontier coding ability. Third-party summaries of that post, from Pebblous and Decrypt among others, report that of 138 hard tasks audited, 59.4% had flawed tests or problem statements, 35.5% of them from overly narrow tests, and that some tasks showed signs of leakage from training data. OpenAI's page returned HTTP 403 to my fetcher, so I am relaying the summaries and have not read the original; check it before you cite the percentages. At the time, OpenAI pointed to SWE-bench Pro as the alternative. I looked at the Pro board in an earlier post in this series, and it has its own mixing problem.

For a board reader, the implication is practical. A Verified score near 80 or 86 is a number on a benchmark whose main early adopter has questioned the grading. It can still rank models roughly, and the board is still useful as an index of claims. It is a weak basis for a procurement decision or a claim that one open model beats another by two points.

The script uses only the standard library. It counts rows by source, finds repos with several rows, and compares the board with the swebench.com data file.

import json, math, urllib. request, collections def get ( url ): req = urllib. request. Request ( url, headers = { " User-Agent ": " Mozilla/5.0 " }) return json. load ( urllib. request. urlopen ( req, timeout = 120 )) rows = get ( " https://huggingface.co/api/datasets/SWE-bench/SWE-bench_Verified/leaderboard " ) repos = collections. Counter ( r [ " modelId " ] for r in rows ) multi = { k: v for k, v in repos. items () if v > 1 } authors = collections.

Counter () for r in rows: pr = r. get ( " pullRequest " ) if pr: d = get ( f " https://huggingface.co/api/models/ { r [ ' modelId ' ] } /discussions/ { pr } " ) authors [( d [ " status " ], d [ " author " ][ " name " ])] += 1 print ( " rows: ", len ( rows ), " | repos: ", len ( repos ), " | verified: ", sum ( bool ( r [ " verified " ]) for r in rows )) print ( " open/merged PR rows by author: ", dict ( authors ), " | main-branch rows: ", sum ( not r. get ( " pullRequest " ) for r in rows )) print ( " repos with 2+ rows: ", len ( multi ), " carrying ", sum ( multi. values ()), " rows " ) print ( " empty notes: ", sum ( not ( r. get ( " notes " ) or "" ). strip () for r in rows )) n = 500 print ( " one task = %.1f points " % ( 100 / n )) for acc in ( 0.70, 0.80, 0.86 ): print ( " accuracy %.2f: 95%% half-width = %.2f points (illustrative, independent runs) " % ( acc, 1.96 * math. sqrt ( acc * ( 1 - acc ) / n ) * 100 )) lb = json. load ( urllib. request. urlopen ( urllib. request. Request ( " https://raw.githubusercontent.com/swe-bench/swe-bench.github.io/master/data/leaderboards.json ", headers = { " User-Agent ": " Mozilla/5.0 " }), timeout = 120 )) ver = next ( l for l in lb [ " leaderboards " ] if l [ " name " ] == " Verified " )[ " results " ] top = max ( v [ " resolved " ] for v in ver ) print ( " swebench.com Verified table: %d entries, max %.1f, newest %s " % ( len ( ver ), top, max ( v [ " date " ] for v in ver ))) print ( " HF rows above that max: ", sum ( r [ " value " ] > top for r in rows )) Enter fullscreen mode Exit fullscreen mode On November 5, 2026 it printed:

rows: 78 | repos: 68 | verified: 0 open/merged PR rows by author: {('open', 'SaylorTwift'): 28, ('open', 'nielsr'): 23} | main-branch rows: 27 repos with 2+ rows: 6 carrying 16 rows empty notes: 54 one task = 0.2 points accuracy 0.70: 95% half-width = 4.02 points (illustrative, independent runs) accuracy 0.80: 95% half-width = 3.51 points (illustrative, independent runs) accuracy 0.86: 95% half-width = 3.04 points (illustrative, independent runs) swebench.com Verified table: 180 entries, max 79.2, newest 2026-02-26 HF rows above that max: 9 Enter fullscreen mode Exit fullscreen mode The board endpoint is undocumented, so field names may change. To check a row, open its pullRequest discussion, read the YAML, and then read the footnotes of the source card. Notes that name a harness are rare; the card usually has them.

How many models are on the Hugging Face SWE-bench Verified leaderboard? On November 5, 2026 the board has 78 rows from 68 repos. Six repos have more than one row, carrying 16 rows between them.

Is the Hugging Face SWE-bench Verified leaderboard verified? No row has verified set to true. 51 rows come from open pull requests, opened by two Hugging Face staff accounts, and 27 are on the model repos' main branches.

Why does the Hugging Face board show higher scores than swebench.com? Nine rows are above the 79.2 maximum in the site's Verified data file, last updated February 26, 2026. The board rows are self-reported under each team's own harness, while the site's mini-SWE-agent runs use one minimal bash-only agent. Different harnesses move scores by several points, as the three Nemotron rows show.

Does the agent harness change a SWE-bench Verified score? Yes. NVIDIA's Nemotron-3-Super-120B card gives 60.47 with OpenHands, 59.20 with OpenCode and 53.73 with Codex. GLM-5 has 77.8 from its maker and 72.8 from a swebench.com mini-SWE-agent run.

Should I still use SWE-bench Verified to compare coding models? As a rough index, yes. As a deciding number, no. OpenAI said in February 2026 that it no longer treats it as a frontier measure, and the board mixes harnesses, efforts and copies. Run the model in your own agent on your own tasks.