Against single-number capability claims

A single benchmark percentage carries almost no information about what a system can do. It compresses performance across a task distribution that was chosen by the benchmark's authors, on an instrument whose construction date is usually unpublished, against a scale whose upper region is frequently saturated.

Yet a single number is what travels. It appears in launch materials, in coverage, in policy briefings, and eventually in analysis that treats it as a measurement.

We think three practices would fix most of this, and none of them are difficult. Publish the instrument's construction date. Report performance on semantically neutral perturbations alongside the headline. Give a distribution across task classes rather than an aggregate.

We apply these to our own reporting. Where we give a capability figure in a Frontier Labs publication, the instrument, its construction date and its perturbation behaviour are stated. Where we cannot state them, we do not give the figure.