Evaluation Integrity
A contamination and construct-validity audit of twenty-eight public benchmarks.
Abstract
Public benchmarks are the primary instrument through which capability claims reach non-specialist audiences. This paper audits twenty-eight widely cited benchmarks for training-set contamination, construct validity and ceiling effects. We find contamination indicators in the majority, construct-validity problems in a substantial minority, and four instruments where reported near-ceiling performance coexists with failure on trivially perturbed variants of the same tasks. We estimate the resulting overstatement of generalisable capability at between fifteen and forty percentage points on affected instruments, and set out the reporting practices that would address it.
Key findings
- 01Contamination indicators present in the majority of the twenty-eight benchmarks audited.
- 02Four instruments show near-ceiling reported performance alongside failure on trivially perturbed variants.
- 03Estimated overstatement of generalisable capability: 15 to 40 percentage points on affected instruments.
- 04Construction date is not published for the majority of instruments, which prevents contamination assessment by readers.
This paper is under internal review. A pre-review summary is published here; the full paper and dataset follow on completion.
A benchmark is a measurement instrument. Like any instrument it can be miscalibrated, it can measure something other than what it claims, and it can be gamed. The field has treated benchmarks as though they were none of these things, largely because the alternative — constructing and maintaining held-out instruments — is expensive and unglamorous.
What we tested
- 01Contamination: whether benchmark items or close variants appear in accessible training corpora.
- 02Construct validity: whether the instrument measures the capability it is described as measuring.
- 03Perturbation robustness: whether performance survives semantically neutral rewording, reordering or numeric substitution.
- 04Ceiling behaviour: whether reported scores near the maximum reflect capability or instrument exhaustion.
The perturbation results are the most striking. On four instruments, systems reporting above ninety-five per cent accuracy fell below sixty on variants that a competent human would regard as identical in difficulty. This is a signature of memorisation, not of capability.
The full audit, including per-instrument results and the perturbation battery, will be released when internal review concludes. We are publishing the summary now because compromised instruments continue to be cited.
Sources and references
- [1]Benchmark corpora and published results, 28 instruments.
- [2]Frontier Labs perturbation battery, revision 2.
Source assessments, provenance records and the full data appendix are described in the methodology. Corrections to this publication are recorded on the corrections page.
Keywords
- evaluation
- benchmarks
- contamination
- validity
Cite this publication
Frontier Labs Research Group (2026). Evaluation Integrity. Frontier Labs. https://frontierlabs.research/research/evaluation-integrity-audit
Research infrastructure powered by AGI-S.