25 reproducible audits of the metrics and gold labels behind LLM benchmark scores

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0; the corpora, results and this write-up are CC BY 4.0. Every number below is reproduced by a small standard-library script that runs in a sealed container.

When a model scores 84.2 on a benchmark, two things have to be trustworthy for that number to mean anything: the scoring code that decides whether an answer is correct, and the gold labels it is scored against. Both are code and data written by people, and both have bugs.

Over the last few weeks I have been building eval-reliability-audits, a versioned catalogue of small, reproducible corpora that isolate one such defect at a time. Each audit pins the upstream scoring code or dataset, hand-traces a synthetic fixture and freezes its expected counts before any real data is touched, then measures the blast radius on the benchmark's own released data inside an isolated runner. There are now 25 of them. Here are a few of the clearest findings.

Natural Language Inference datasets pair a premise with a hypothesis and a gold label (entailment / neutral / contradiction). If the same (premise, hypothesis) string pair appears twice with two different gold labels, that pair is self-contradictory ground truth.

Censusing exact-duplicate string pairs with conflicting labels:

SNLI deliberately reuses a premise across labels, and prior work reports aggregate annotator agreement, but the specific defect of exact-duplicate (premise, hypothesis) string pairs carrying conflicting gold labels does not appear to have been catalogued before. Reassuringly, in all three corpora the held-out evaluation splits are clean; the conflicts live in train.

GSM8K's grade-school math solutions carry inline <<expr=result>> calculator annotations. Re-computing every one of them:

GSM8K also ships a "socratic" copy of every problem with a sub-question before each step. Two more audits check that copy against the main one:

And a contamination check for good measure: 0 questions are shared between GSM8K's train and test splits, and neither split contains internal duplicate questions.

The other half of the catalogue targets scoring code. A sampler:

The full catalogue, with every script, fixture and frozen results.json, is here: github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits. Corrections and additional datasets to audit are welcome.