Software Production After Assistants
Throughput, defect density and maintenance burden in machine-assisted engineering.
Abstract
Software engineering is the domain where the effects of machine assistance can be measured with revealed rather than self-reported data, at scale, and over enough elapsed time to observe second-order effects. This paper analyses repository-level telemetry covering 3.1 million merged changes across 1,400 codebases, comparing periods before and after sustained assistant adoption. We find substantial increases in change volume and reductions in time-to-first-draft, no significant change in post-merge defect rate at the aggregate level, and a measurable increase in review latency and in the proportion of changes subsequently reverted or substantially rewritten. We argue that first-order productivity studies overstate net effect by omitting review and maintenance cost.
Key findings
- 01Merged change volume per engineer rose 38 per cent following sustained adoption, with the increase concentrated in small and medium changes.
- 02Time from task start to first reviewable draft fell by a median of 52 per cent.
- 03Post-merge defect rate per change did not change significantly; defect rate per engineer-week rose in proportion to change volume.
- 04Review latency rose 21 per cent and the share of changes reverted or substantially rewritten within 90 days rose from 6.2 to 9.8 per cent.
Writing code faster is measurable. Whether the system as a whole produced more value is a harder question, and the answer depends on what happens after the code is written.
Studies of machine-assisted software engineering have generally measured the part of the process that is easiest to measure: how quickly a developer produces working code for a bounded task. This is a real effect and it is large. It is also only the first stage of a production system whose later stages — review, integration, operation, maintenance — carry most of the total cost.
Data and design
Our corpus covers 3.1 million merged changes across 1,400 codebases between 2022 and 2026, with matched defect tracking for a subset. We classify each codebase-quarter by adoption intensity using observable signals rather than self-report, and compare within-codebase periods to control for the large heterogeneity across projects.
Within-codebase comparison controls for project characteristics but not for time-varying confounders such as team growth or changes in release cadence. We report robustness checks against a matched control set of codebases with low adoption over the same calendar period.
Throughput rose substantially
Merged changes per engineer-week rose 38 per cent. The increase is concentrated in small and medium changes; the volume of large changes was essentially unaffected. Median time from task start to first reviewable draft fell by 52 per cent, which is the largest single effect in the study.
- Merged changes per engineer-week38
- Time to first reviewable draft−52
- Median change size (lines)−14
- Review latency21
- Post-merge defects per change3
- Reverted within 90 days58
Frontier Labs repository telemetry corpus, 2022–2026.
The costs move downstream
Two second-order effects are visible and material. Review latency rose 21 per cent — reviewers face more changes and, in interviews, describe assistant-generated changes as requiring more careful reading because plausible-looking errors are harder to spot than obviously wrong ones. And the proportion of changes reverted or substantially rewritten within ninety days rose from 6.2 to 9.8 per cent, a relative increase of 58 per cent.
Post-merge defect rate per change was essentially flat. This is the finding most often quoted in isolation, usually as evidence that quality is unaffected. Read alongside the volume increase, it means defects per engineer-week rose roughly in proportion to output. Quality per unit of code held; the amount of code grew.
| Stage | Direction | Magnitude | Confidence |
|---|---|---|---|
| Drafting | Benefit | Large | High |
| Review | Cost | Moderate | High |
| Integration | Neutral | Small | Moderate |
| Post-merge defect handling | Cost | Moderate | Moderate |
| Rework and revert | Cost | Moderate | High |
| Long-tail maintenance | Unknown | Not measurable yet | Low |
Net effect remains positive under every weighting we tested. The size of the net benefit is between one third and one half of the first-order drafting benefit.
The unresolved question is long-tail maintenance. Code written today imposes cost for years. Four years of data is not enough to observe whether assistant-generated code carries a different maintenance profile than human-written code of the same era. We have committed to re-running this analysis annually on the same corpus, which is the only way we know to answer it.
Sources and references
- [1]Repository telemetry corpus, 1,400 codebases, 2022–2026.
- [2]Incident and defect tracking records, matched subset.
- [3]Frontier Labs adoption classification methodology.
Source assessments, provenance records and the full data appendix are described in the methodology. Corrections to this publication are recorded on the corrections page.
Keywords
- software
- productivity
- defects
- code review
Full edition
Read the complete report.
The full edition is released to named readers. Tell us who you are and it opens straight away.
- Extent
- 53 pages
- Price
- Free
Cite this publication
Frontier Labs Research Group (2026). Software Production After Assistants. Frontier Labs. https://doi.org/10.5281/frontierlabs.2026.0011
Research infrastructure powered by AGI-S.