Interconnect
Networking fabrics and the collective communication ceiling on large-scale training.
Abstract
As training clusters have grown, the fraction of wall-clock time spent on collective communication rather than arithmetic has become the dominant determinant of achieved utilisation. This paper analyses the scaling behaviour of network fabrics in large training deployments, examining topology choice, collective algorithm selection, optical versus electrical switching, and the failure characteristics that determine effective cluster size. We find that the practical ceiling on synchronous training scale is set by fabric behaviour and failure rate rather than by accelerator count, and that reported cluster sizes therefore substantially overstate the compute available to any single coherent run.
Key findings
- 01Achieved utilisation on large synchronous runs falls approximately as the logarithm of node count under current fabric designs.
- 02Collective communication accounts for 18 to 34 per cent of wall-clock time on the largest runs we could characterise.
- 03Mean time between node failure at cluster scale sets an effective ceiling on synchronous run size well below installed capacity.
- 04Optical circuit switching materially improves large-scale behaviour but adds reconfiguration latency that penalises irregular communication patterns.
A cluster is not a computer. It is a network with computers attached, and the network is what decides how much of it you get to use.
Accelerator counts are the standard unit of account for training infrastructure. They are a poor one. Two clusters with identical accelerator counts can differ by a factor of two in the compute they deliver to a single synchronous training run, and the difference is almost entirely attributable to the network fabric and to failure behaviour.
The collective bottleneck
Synchronous data-parallel training requires that gradients be reduced across all participating devices at every step. The time this takes scales with the communication volume and inversely with the achievable bandwidth of the reduction, which depends on topology, on the collective algorithm, and on congestion behaviour under real traffic patterns.
Modern training does not use pure data parallelism — pipeline, tensor and expert parallelism all reduce the volume of the global reduction — but every hybrid strategy has a communication pattern, and each pattern stresses the fabric differently. Tensor parallelism demands very high bandwidth within a small group. Expert parallelism produces all-to-all traffic that is punishing on oversubscribed topologies. The engineering problem is to match parallelism strategy to fabric structure, and it is solved differently at every site.
Frontier Labs fabric simulation, release 3.
Failure sets the ceiling
At sufficient scale, the limiting factor is not bandwidth but reliability. If a single node failure halts a synchronous run, and the failure rate per node is fixed, then the expected interval between run-halting failures shrinks linearly with cluster size. Beyond a certain point, a run spends more time checkpointing and recovering than progressing.
Every operator at scale has responded with the same family of techniques: frequent asynchronous checkpointing, hot spares with rapid reconfiguration, and increasingly, training strategies tolerant of partial participation. These work, at a cost. The cost is rarely reported, which is why published cluster sizes should not be read as available synchronous compute.
| Nodes | MTBF per node | Expected run-halting events per day | Recovery overhead |
|---|---|---|---|
| 1,000 | 5 years | 0.55 | 1.2% |
| 8,000 | 5 years | 4.4 | 7.8% |
| 32,000 | 5 years | 17.5 | 24% |
| 32,000 | 15 years | 5.8 | 9.1% |
| 128,000 | 15 years | 23.4 | 31% |
Assumes 20-minute recovery, checkpoint interval optimised for the failure rate, and no partial-participation tolerance. Real deployments do better; the shape of the curve is the point.
The direction of travel is clear. Fabric bandwidth per accelerator has grown more slowly than accelerator throughput for four consecutive hardware generations. Unless that reverses, the share of training time spent communicating will continue to rise, and the practical ceiling on coherent run size will be set in the network rather than in the silicon.
Sources and references
- [1]Published systems papers on large-scale training, 2022–2026.
- [2]Frontier Labs fabric simulation, release 3.
- [3]Hardware failure rate disclosures, aggregated.
Source assessments, provenance records and the full data appendix are described in the methodology. Corrections to this publication are recorded on the corrections page.
Keywords
- networking
- interconnect
- collectives
- utilisation
Full edition
Read the complete report.
The full edition is released to named readers. Tell us who you are and it opens straight away.
- Extent
- 42 pages
- Price
- Free
Cite this publication
Infrastructure & Compute Desk (2026). Interconnect. Frontier Labs. https://doi.org/10.5281/frontierlabs.2026.0004
Research infrastructure powered by AGI-S.