Under review2026

Benchmarks Do Not Report Whether They Could Read the Answer

Rajveer Singh Pall

Two answer extractors shipped in the same evaluation harness disagree by 88.5 points on the same 400 MMLU responses.

The discovery in one figure

ONE FRONTIER MODEL · THE SAME 400 MMLU RESPONSES · TWO WAYS TO READ THEMThe task’s strict pipeline0.013The answer read wherever it appears0.93800.250.50.751ACROSS THE HARNESS AND PUBLISHED GENERATIONSGenerative tasks that score a format mismatch as wrong2,936 of 4,524 (64.9%)Hidden parse-failure rate across 28 models4% to 99.9%A judge that never saw the gold answer confirms 97 of 99 recovered answers.

Swipe to see the whole figure

Only the scoring rule changed, and one model’s MMLU score moved from 0.013 to 0.938.

The paper in five minutes

Whether a model gets credit on a benchmark depends partly on whether the scoring code can find its answer, and that half of the score is normally invisible. Auditing every task in a widely used evaluation harness shows that most generative tasks match an assumed output format, count a mismatch as wrong, and never report how often that happened.

The research question

Does the unreported half of a benchmark score, whether the scorer could read the answer, change what the benchmark says?

How it works

All 13,668 task configurations in lm-evaluation-harness are resolved with the framework’s own loader. On MMLU, 400 responses are scored under a task’s strict pipeline and under a robust reading, checked by a third-vendor judge blind to the gold answer and by three human labellers. Published leaderboard generations for 28 models are then rescored the same way.

  1. 01
    Resolve every taskall 13,668 configurations in lm-evaluation-harness
  2. 02
    Score the same text twicea task’s strict pipeline versus a robust reading, 400 MMLU responses
  3. 03
    Check the checkera third-vendor judge blind to the gold answer, three human labellers
  4. 04
    Rescore published resultsleaderboard generations for 28 models

Experimental results

2,936 of 4,524 generative tasks (64.9%) record a format mismatch as a wrong answer, 81.0% of them report a single number with no second pipeline, and none report how often the extractor failed. A frontier model scores 0.013 under a task’s strict pipeline and 0.938 when the answer is read wherever it appears. Across 28 models' published generations the hidden parse-failure rate spans 4% to 99.9%, and one published score of 0.000 is entirely an extraction artefact.

One frontier model on MMLU, same 400 responses
Task’s strict pipeline
0.013
Answer read wherever it appears
0.938

Only the scoring rule changed. The model ordering inverts, significant after Holm-Bonferroni correction.

  • 88.5 ptsdisagreement between two extractors in one harness
  • 2,936 / 4,524generative tasks score a format mismatch as wrong
  • 97 / 99recovered answers confirmed by a judge blind to the gold answer

What this changes

The fix costs one extra reported number per task, and it is already implemented in the newest evaluation code of one audited repository.

Resources

Not public yetThe paper and its code are withheld while the submission is under double-blind review.

  • Benchmark Evaluation
  • NLP · LLMs

Citation

Manuscript under review. The citation will be posted on acceptance. Reach me at rajveerpall04@gmail.com.

Where this leads next