The Inference Economy

What a model costs to serve and what it costs to buy have never been the same number. The gap between them is where most of the interesting economics lives.

Published prices for model inference reflect competitive positioning, customer acquisition strategy, and in several cases deliberate subsidy. They tell you what a firm has decided to charge. They tell you nothing reliable about what the service costs to provide. Since almost every claim about the economics of artificial intelligence depends on the latter, we built a model of it.

What the model includes

The unit is one million output tokens at a specified quality level, produced under a specified latency constraint. Cost components are:

• Accelerator capital, amortised on a four-year straight-line schedule with a residual value assumption we vary in sensitivity analysis.
• Host system capital: CPU, memory, local storage, chassis, networking interface.
• Fabric capital allocated per accelerator, including switching and optics.
• Electricity at contracted industrial rates, covering both IT load and cooling overhead at the site's measured power usage effectiveness.
• Facility cost, whether owned and amortised or leased and expensed.
• Operational overhead: staffing, monitoring, incident response, software engineering allocated to serving.
• Utilisation adjustment, which converts theoretical throughput into delivered throughput.

The model deliberately excludes training cost amortisation, research expenditure, sales and marketing, and margin. It answers the question 'what does it cost to run this' and not 'what does it cost to be in this business'.

The decline and its composition

Cost per million output tokens at constant quality fell from a figure we estimate in the low tens of dollars in Q1 2023 to a figure in the low tens of cents by Q2 2026. This is a large decline by any historical standard for a computing service. Its composition is not what most commentary assumes.

Four-fifths of the decline came from serving the same capability more efficiently. This is a different economic phenomenon from hardware improvement, and it has a different forward trajectory. Hardware improvement is a long-running exponential driven by an industry with a multi-decade record. Systems-engineering improvement is a one-time harvest of accumulated inefficiency, and the largest inefficiencies are removed first.

The slowdown

Our model shows the proportional rate of cost decline falling sharply. Between Q1 2023 and Q2 2024, fully loaded cost fell by approximately 89 per cent. Between Q2 2025 and Q2 2026 it fell by approximately 41 per cent. Both are large; the second is much smaller than the first, and the trajectory is monotone.

We want to be careful about what this implies. A slowing rate of decline in the cost of a fixed capability level is entirely compatible with rapid growth in the total amount spent on inference, and both are happening. Falling unit cost has expanded the range of economically viable applications faster than it has reduced spend. Aggregate inference expenditure has grown throughout the period.

Utilisation dominates

If we had to identify a single number that determines serving economics, it would be utilisation — the fraction of purchased accelerator time that is spent producing billable output. Serving workloads are bursty, latency-constrained and heterogeneous. Reserving capacity for peak demand while paying for it continuously is the central operational problem.

The spread between well-managed and poorly-managed fleets on identical hardware exceeds a factor of three in our modelling. Techniques that close the gap — continuous batching, disaggregated prefill and decode, speculative decoding, multi-tenant scheduling across latency classes, and shifting batch-tolerant work into troughs — are individually well documented and collectively difficult to operate.

The forward question this raises is whether the remaining systems-engineering headroom is large. Our assessment is that it is smaller than it was, that the highest-value remaining gains are in memory hierarchy management and in cross-request state reuse, and that absent a hardware discontinuity the cost curve for a fixed capability level will flatten considerably before 2028.