The first LLM router metric I check is the missing-row count

A router can choose a model for every prompt and still leave you with a very incomplete evaluation. Those are different achievements. I want to see both before I trust the quality chart. 🔎

AutoBench's new router audit makes that distinction unusually easy to inspect: it publishes the decisions as well as the scores. The author reports that none of the commercial settings in its headline comparison beats a particular static model on quality and cost together. That is the author's result, not a fresh model test from me.

My contribution here is smaller and reproducible: read the released decision ledger, count which decisions have scores, and check whether a supposedly adaptive policy mostly picks the same model.

Original chart computed from the pinned AutoBench decision export. Unscored means unavailable for scoring in this ledger; it does not establish a failed request.

In the exported azure_cost pass, all 252 decisions name a model, but only 35 have scored=True. Its top model receives 149 decisions. In orca_constrained_quality, the corresponding counts are 245 scored and 252 top-model selections, again out of 252 decisions.

The first pass raises a coverage question. The second raises a routing question. A single quality percentage hides that distinction.

Imagine printing a restaurant review after tasting only the starters. The score may describe the starters perfectly. It still needs a label. 🍽️

For a benchmark, I would keep a decision count, a scored count and a missing-cell explanation next to every quality number. A score calculated over available cells is a conditional result. It should not silently become a statement about the entire workload.

That does not mean every unavailable cell is a bad answer. The provider may have selected a model outside the stored score matrix. Penalizing every such cell as a miss is another evaluation choice, not an automatic repair.

Save the following as audit.py and run python3 audit.py with Python 3.10 or newer and internet access. It uses the standard library, pins the dataset revision and checks the downloaded file's digest before parsing it.

import csv import hashlib import io from collections import Counter, defaultdict from urllib.request import urlopen revision = ' 43755180be7d8097068982d7a3f37fc04cd15379 ' url = f ' https://huggingface.co/datasets/AutoBench/router-benchmark-2026-10/resolve/ { revision } /decisions.csv ' raw = urlopen ( url ). read () assert hashlib. sha256 ( raw ). hexdigest () == ' d000e5310744ae15f9c9d572ee1f88f84858fc0ba01fb54eef1e6a73a505d540 ' passes = defaultdict ( list ) for row in csv. DictReader ( io. StringIO ( raw. decode ())): passes [ row [ ' pass ' ]]. append ( row ) assert sum ( map ( len, passes. values ())) == 7046 assert len ( passes ) == 28 collapsed = 0 for rows in passes. values (): assert len ({ r [ ' prompt_id ' ] for r in rows }) == len ( rows ) top = Counter ( r [ ' selected_model ' ] for r in rows ). most_common ( 1 )[ 0 ][ 1 ] collapsed += 10 * top >= 9 * len ( rows ) print ( ' exported passes: ', len ( passes ), ' | collapsed: ', collapsed ) for name in [ ' orca_constrained_quality ', ' azure_cost ', ' nd_subset_k0 ', ' or_auto ' ]: rows = passes [ name ] top = Counter ( r [ ' selected_model ' ] for r in rows ). most_common ( 1 )[ 0 ][ 1 ] scored = sum ( r [ ' scored ' ] == ' True ' for r in rows ) print ( f ' { name }: scored { scored } / { len ( rows ) }, top-model picks { top } / { len ( rows ) } ' ) Enter fullscreen mode Exit fullscreen mode The output is:

exported passes: 28 | collapsed: 9 orca_constrained_quality: scored 245/252, top-model picks 252/252 azure_cost: scored 35/252, top-model picks 149/252 nd_subset_k0: scored 248/252, top-model picks 252/252 or_auto: scored 245/252, top-model picks 248/252 Enter fullscreen mode Exit fullscreen mode ✅ Here, nine of the 28 exported passes send at least 90% of decisions to one model. The threshold uses integer arithmetic, so a floating-point boundary cannot change the classification.

There is an easy trap here: the export's 28 passes are not the blog's 17 headline settings. The dump also contains secondary and reference passes. My count describes the whole exported ledger; it does not correct or contradict the author's narrower count.

A concentrated policy is not automatically useless. If one model is suitable for the workload, consistently choosing it may be sensible. Reliability requirements, contractual constraints and availability can also narrow a usable pool.

What concentration changes is the comparison I want. If a policy nearly always selects model A, compare it with always selecting A. Then ask what the remaining decisions contribute. Merely showing that it beats random selection does not isolate the value of adapting to the prompt. 💡

I would also compare routers on the same scored prompt set before interpreting a quality difference. Otherwise a router evaluated on difficult questions and another evaluated on an easier subset can trade places without either improving. Keep the missing rows visible, and report the full workload separately from that matched subset.

This is an offline audit of published decisions, not another round of router or model calls. I did not reproduce AutoBench's judges, bootstrap procedure or cost ranking. The source says its router calls predate this release; the ledger is not a guarantee about current vendor behavior.

The useful habit survives those limits: print coverage and concentration before celebrating a quality score. A routing graph gets much more interesting once you know which decisions it actually describes.

Peter Kruger, Do LLM Routers Actually Route?.

AutoBench, router benchmark dataset and methodology notes. Counts and original chart derived from its CC BY 4.0 decision table. The script pins the exact revision.