Which Leaderboard Orderings Survive a Human Reader?
A 400-item human audit certifies 174 of 351 pairwise orderings among 27 models under one stated assumption, and none without it.
The discovery in one figure
Swipe to see the whole figure
The paper in five minutes
A leaderboard score comes from two things: the model, and a verifier, the program that finds the answer in a response and decides whether it is right. Leaderboards name the model but not the verifier, and verifiers disagree. This work asks which rankings still hold when a person reads the responses instead.
The research question
Scores on benchmarks graded from generated text come from a verifier, and leaderboards report the model but not the verifier. Which orderings in a published ranking survive when a human reader decides what each response says?
How it works
On the archived MATH-Hard board of the Open LLM Leaderboard, three verifiers read all 35,748 published responses of 27 models. A human audit of 400 items bounds the accuracy gap between each pair of models under one stated assumption: where all three verifiers agree, the rate at which a reader overrules them differs across models by at most ε. Orderings are certified for all 351 pairs at once, exactly in finite samples, and the result is reported as a curve over ε rather than at one value.
- 01Read every response three waysthree verifiers, 35,748 published responses, 27 models
- 02Audit a sample by hand400 items read by a human
- 03Bound each pairwise gapone stated assumption, reported as a curve
- 04Certify orderingsall 351 pairs at once, exact in finite samples
Experimental results
The verifiers disagree on 1,159 responses, and on 891 of them a verifier that gave no credit had found no answer it could read. The audit certifies 174 of 351 orderings at ε = 0.01, 208 at ε = 0, and none without the assumption. When the leaderboard itself changed verifier in February 2025, 15.9% of the model pairs that both scores order strictly reversed (3,366 models, each compared on its own results file before and after, so on the same responses).
2081740Simultaneous over all 351 pairs. The reference tolerance was set after the audit’s labels existed; the curve between the two ends is the result.
8912681,159 of 35,748 responses.
- 174 / 351orderings certified at the reference tolerance (ε = 0.01)
- 1,159of 35,748 responses where the three verifiers disagree
- 15.9%of strictly ordered model pairs reversed when the leaderboard changed verifier
Stated honestlyThe reference tolerance ε = 0.01 is a judgement made after the audit’s labels existed, not a registered value, so the paper reports the whole curve. The audit covers 27 of the 3,751 models on the board.
What this changes
A published ranking should state the verifier that produced it and how many responses that verifier could not read.
Resources
Not public yetThe manuscript is not public yet; the code, data and results are.
- Benchmark Evaluation
- NLP · LLMs
Citation
Submitted manuscript. The citation will be posted on acceptance. Reach me at rajveerpall04@gmail.com.