Humanity's Last Exam on Hugging Face: At Least 23 of the Top 24 Rows Use Tools, and the Board Has No Column for It

HLE is a 2,500-question exam written by subject experts. The paper describes it as multimodal: roughly 14% of the questions need images, and about 24% are multiple choice, the rest short exact-match answers. Hugging Face lists the dataset as an official benchmark, so model repos can publish a score in a.eval_results/*.yaml file and the board collects them.

A row is therefore a self-report. In my last post on this series I walked through how those YAML files become a board; here the relevant part is small. A row has a value, a source link, a pull request or commit, and an optional notes string. There is no structured field for tool access, test subset, judge model or number of runs.

Two rows hint at what the board's default is. One submitter writes that their number is "with tools (full set)... not the no-tools default". Another says theirs is "evaluated with tool/browsing access enabled, not the closed-book default". I could not read the dataset's eval.yaml (it is gated), so I cannot confirm the default from the source, but if those two notes are right, the top of the board is made of off-default runs.

I classified every row by its note: "with tools", "w/ tool" or "with tool" means tools; "no tools" or "without tools" means closed-book; anything else is unlabeled.

Half the board does not say. That is not a criticism of any single submitter: the YAML format never asked.

Ten rows also say "text-only", which means they were scored on the subset without images. The Artificial Analysis HLE page uses 2,158 text-only questions out of the 2,500, so a text-only row and a full-set row are scored on different tests. Three rows state a judge: two say GPT-4o (one adds that the task default is o3-mini) and one says GPT-5.6-luna. The other 93 rows say nothing about the grader.

Sorted by value, the top 24 rows break down like this: 21 say "with tools", 3 are unlabeled, none say "no tools". The first explicit no-tools row, Ornith-1.5-397B at 44.6, is rank 25.

The three unlabeled rows in the top 24 are Kimi-K3 (rank 5, 56.0), Hy3 (rank 10, 53.2) and Inkling (rank 23, 46.0). I opened the model cards:

So the accurate statement is "at least 23 of the top 24". Row 1 is also worth a look: its note reads "With tools; harness not specified in the model card." The top score comes with the tool set and the harness both unspecified.

Several repos publish both numbers, so I paired rows that belong to the same model. Seven pairs have both rows explicitly labeled. Six more come from gpt-oss, where the paired rows share a reasoning level and the no-tools rows carry no tools label (the pairing follows the cited OpenAI paper, arXiv 2508.10925, so treat it as weaker).

For the seven explicit pairs the median uplift is 11.3 points with a range of 7.8 to 15.1. Two of those are the same Nemotron model in two precisions, so call it six independent pairs. Across all 13 pairs the range is 1.8 to 15.1, an eightfold spread.

That spread is the practical point. There is no single "tools constant" to subtract from a with-tools row to get its closed-book equivalent. The uplift depends on the model, its training for tool use, the tool set and the harness. A with-tools row cannot be turned into a closed-book row by arithmetic; it can only be replaced by a closed-book measurement.

For scale, the sampling noise of the test itself is small next to this. With n = 2,500 and accuracy near 45%, the standard error is 0.99 points and a 95% interval is about plus or minus 1.95. Two independent runs need to differ by about 2.8 points (3.0 on the 2,158-question subset) before the gap clears noise. Three of the gpt-oss uplifts (1.8, 2.1, 2.7) fall under that line, though the pairs are scored on the same questions, which lowers the real threshold by an amount I cannot compute without per-question results.

Take the closed-book numbers: the explicit no-tools rows plus Kimi-K3's 43.5 from its card.

The top six run from 44.6 down to 42.6, a spread of 2.0 points. Every interval overlaps every other. Three of the six are text-only rows scored on 2,158 questions, so their intervals are slightly wider than drawn, and the rows without a subset note may or may not be on the same test. The honest reading is a cluster, not a ranking. This matches what I found on the GPQA Diamond board: when the test is small relative to the model gaps, listing models in order of value implies precision that is not there.

The board JSON endpoint is undocumented and may change, so treat this as a snapshot tool. It reproduces the counts above.

import json, re, collections, urllib. request URL = " https://huggingface.co/api/datasets/cais/hle/leaderboard " # undocumented, may change rows = json. load ( urllib. request. urlopen ( urllib. request. Request ( URL, headers = { " User-Agent ": " Mozilla/5.0 " }), timeout = 60 )) def condition ( note ): note = ( note or "" ). lower () if re. search ( r " with tools?|w/ tool ", note ): return " tools " if re. search ( r " no tools?|without tools? ", note ): return " no_tools " return " unlabeled " for r in rows: r [ " cond " ] = condition ( r. get ( " notes " )) print ( len ( rows ), " rows; ", collections.

Counter ( r [ " cond " ] for r in rows )) print ( " top 24 by condition: ", collections. Counter ( r [ " cond " ] for r in rows [: 24 ])) by_model = collections. defaultdict ( list ) for r in rows: by_model [ r [ " modelId " ]]. append ( r ) for model, rs in by_model. items (): t = [ r for r in rs if r [ " cond " ] == " tools " ] n = [ r for r in rs if r [ " cond " ] == " no_tools " ] if len ( t ) == 1 and len ( n ) == 1: print ( f " { model }: { t [ 0 ][ ' value ' ] } vs { n [ 0 ][ ' value ' ] } (+ { t [ 0 ][ ' value ' ] - n [ 0 ][ ' value ' ]:. 1 f } ) " ) Enter fullscreen mode Exit fullscreen mode On November 1, 2026 it prints:

96 rows; Counter({'unlabeled': 50, 'tools': 34, 'no_tools': 12}) top 24 by condition: Counter({'tools': 21, 'unlabeled': 3}) ornith-ai/Ornith-1.5-397B: 56.1 vs 44.6 (+11.5) tencent/Hy4-preview: 55.4 vs 43.4 (+12.0) reflection/Beam-1.0-501B-A23B: 51.1 vs 36 (+15.1) nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16: 37.4 vs 26.7 (+10.7) nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4: 37.4 vs 26.1 (+11.3) ornith-ai/Ornith-1.5-35B-A3B: 33.4 vs 25.6 (+7.8) ornith-ai/Ornith-1.5-9B: 30.5 vs 20.2 (+10.3) Enter fullscreen mode Exit fullscreen mode The classifier reads free text, so it is a lower bound on labeled rows: a note phrased in some other way lands in "unlabeled". That is the same limitation a reader faces.

Do tools make an HLE score invalid? No. A with-tools score is a legitimate measurement of a model plus its tool harness. It is just not comparable to a closed-book score, and the board puts them in one sorted list.

Is the 63.9 at the top wrong? Nothing here shows it is wrong. It is self-reported, marked with tools, and its note says the harness is not specified, so it cannot be compared with rows run under a different harness.

How many HLE questions does the board's test have? The dataset has 2,500 questions. Some rows are scored on the text-only subset, which Artificial Analysis puts at 2,158 questions, so the denominator differs between rows.

Can I convert a with-tools score to a closed-book estimate? Not reliably. In the paired rows the uplift runs from 1.8 to 15.1 points, depending on the model and reasoning level.

Why do unlabeled rows matter more than labeled ones? They are 50 of 96 rows, and the two I could resolve at the top of the board turned out to be with-tools numbers.

All counts in this article are my own measurements from the board JSON and the cited cards on November 1, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The classification of rows is mine, based on free-text notes, and the board endpoint is undocumented and may change.