The Data Wall

Every estimate of how much training data remains is really an estimate of what counts as data. The disagreement is definitional before it is empirical.

The data wall argument runs as follows: frontier models are trained on a large fraction of the high-quality text that exists; the stock is finite; production of new text is slow relative to consumption; therefore scaling by data will stop. Each step is plausible. The first and the last are where the argument is weakest.

Quality is a threshold, and the threshold moves

'High-quality text' is not a natural category. In practice it means text that survives a filtering pipeline, and filtering pipelines have become substantially more permissive as filtering itself has improved. Material discarded in 2022 as low quality is routinely retained in 2026 because better classifiers can extract the usable portion. The stock is therefore partly a function of processing capability rather than a fixed quantity.

The spread across these definitions exceeds two orders of magnitude in token count. Any claim about how close the field is to exhaustion is dominated by which of these definitions is being used, and published claims frequently do not say.

Access binds before availability

In every scenario we modelled, licensing and access restrictions become binding before physical stock does. Large portions of the highest-quality corpora — academic publishing, professional archives, proprietary technical documentation, much of the historical book record — are legally or practically inaccessible regardless of whether they physically exist.

This reframes the problem. The data question is substantially a question about licensing markets, and the trajectory of those markets is a matter of contract and litigation rather than of information theory.

Revised April 2026 to incorporate updated repeated-epoch results and to widen the interval on the permissive-filtering stock estimate.