Under review2026

Benchmark Accuracy Is Not an Identified Quantity

Rajveer Singh Pall

Most orderings on a published MATH-Hard leaderboard cannot be separated once the responses a scorer could not read are counted.

Leaderboard data ↗

The discovery in one figure

THE TOP THREE, AS PUBLISHED AND AS IDENTIFIED#1Qwen2-72B-Instruct35.1%#2Yi-1.5-9B-Chat25.4%#3Yi-1.5-34B-Chat23.3%0%20%40%60%80%Shaded: the accuracy all three ranges share,so no ordering of the three is identified.357 / 378orderings not identified with human-audited margins332 of 378 even at zero scorer errorALL 378 PAIRWISE ORDERINGS21 identified357 not identifiedRows and columns: the 28 models in published rank order.

Swipe to see the whole figure

Count the responses the scorer could not read, and 357 of 378 orderings on a published leaderboard can no longer be separated.

The paper in five minutes

A benchmark score is produced by two things: the model, and a scoring rule that decides what answer the model’s text contains. When the scorer cannot read a response, it quietly marks it wrong and throws the count away. This paper shows that the published accuracy is then consistent with a whole range of true accuracies, and on a real 28-model leaderboard most of the rankings people quote do not survive that range.

The research question

If only the model’s half of a benchmark score is reported, is the resulting leaderboard ordering statistically meaningful?

How it works

Reported accuracy is formalised as a partially identified quantity, bounded by the scorer’s own count of unparseable responses and two error margins. Both margins are measured on the same extractor and benchmark: a 400-item human audit specified and frozen before labelling, and an independent LLM adjudication of the same items cross-checked against two further vendors. The practice is then audited across lm-evaluation-harness and OpenCompass.

  1. 01
    Separate the two halvesthe model’s answer, and whether the scorer could read it
  2. 02
    Bound the true accuracypublished score, unreadable count, two measured error margins
  3. 03
    Measure the margins400-item human audit frozen before labelling, plus independent LLM adjudication
  4. 04
    Test every orderingall 378 pairs on a 28-model MATH-Hard leaderboard
  5. 05
    Check the ecosystemlm-evaluation-harness and OpenCompass scoring configurations

Experimental results

357 of 378 pairwise orderings (94.4%) are not identified with human-audited margins, and 87.8% are not identified even when the extractor is granted zero error. 64.9% of generative tasks in lm-evaluation-harness and 43.3% of readable OpenCompass scoring configurations coerce an unreadable response into a wrong answer. A purpose-built learned extractor still leaves 80.2% of orderings unidentified.

Share of MATH-Hard orderings that are not identified
Human-audited margins
94.4%
LLM-adjudicated margins
90.2%
Zero scorer error assumed
87.8%
Learned extractor, zero error
80.2%

Out of 378 model pairs. Even with the most favourable assumption, most rankings cannot be separated.

  • 357 / 378MATH-Hard orderings not identified (human-audited margins)
  • 87.8%not identified even at zero scorer error
  • 64.9%of lm-evaluation-harness generative tasks turn unreadable into wrong

Stated honestlyEach measured margin is applied as a single bound shared across every model on the board, an assumption the paper states and tests as far as the data allow.

What this changes

A leaderboard gap is only meaningful if it is larger than what the scorer cannot see. The paper proposes a five-field Scorer Card and a tested upstream patch that reports the one number the scorer already computes and throws away.

Resources

Leaderboard data ↗

Not public yetThe paper and its code are withheld while the manuscript is under double-blind review.

  • Benchmark Evaluation
  • NLP · LLMs
  • Deployment Shift

Citation

Manuscript under review. The citation will be posted on acceptance. Reach me at rajveerpall04@gmail.com.

Where this leads next