PaperlayerView exact revision on arXiv

Source-linked AI summary

The rising costs of training frontier AI models

Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, David Owen

arXiv:2405.21015v2cs.CY

TL;DR

Frontier AI training costs have been difficult to measure publicly despite their rapid growth and importance for access to large-scale innovation. The paper develops three complementary cost-estimation approaches covering hardware, energy, cloud rental, and staff expenses, finding that amortized compute costs grew 2.4x per year since 2016 and that R&D staff costs are a major component of development costs. The authors conclude that rising costs may concentrate frontier AI development among the largest organizations.

  • Problem

    Public evidence on the magnitude and growth of frontier AI training costs has been limited, despite the importance of these expenses for large-scale AI development.

  • Method

    The paper estimates frontier-model costs using complementary approaches based on amortized hardware and energy, cloud rental prices, and development-wide compute plus R&D staff costs.

  • Results

    2.4x per year: amortized compute costs for frontier training runs grew at this rate since 2016, while R&D staff costs comprised 29–49% of total amortized development costs for four selected models.

  • Takeaways & Limitations

    Frontier model development costs may make major AI advances accessible mainly to the largest companies and government institutions.

  • Takeaways & Limitations

    The analysis relies on public information, uncertain cost assumptions, and methods that omit potentially significant data-center infrastructure and training-data acquisition costs.

Abstract

from arXiv · show

The costs of training frontier AI models have grown dramatically in recent years, but there is limited public data on the magnitude and growth of these expenses. This paper develops a detailed cost model to address this gap, estimating training costs using three approaches that account for hardware, energy, cloud rental, and staff expenses. The analysis reveals that the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x). For key frontier models, such as GPT-4 and Gemini, the most significant expenses are AI accelerator chips and staff costs, each costing tens of millions of dollars. Other notable costs include server components (15-22%), cluster-level interconnect (9-13%), and energy consumption (2-6%). If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.

1 Introduction

The paper addresses limited public evidence on frontier AI training costs by combining hardware, energy, cloud-rental, and staff-cost analyses. It finds that costs are rising rapidly, with R&D staff and computing hardware forming major components of development expenses.

  • Costs and motivation: Few concrete public estimates previously measured the magnitude and growth of training costs for the largest ML models.The study builds on prior cloud-rental estimates with a more detailed account of hardware, energy, and R&D staff costs.
  • Cost components: R&D staff costs including equity represented 29–49% of total amortized development costs for GPT-3, OPT-175B, GPT-4, and Gemini Ultra.Computing hardware made up 47–64%, while energy comprised 2–6%.
  • Contribution: The analysis incorporates hardware purchase costs, energy consumption, and opaque R&D labor expenses to clarify the economics of AI development.It aims to illuminate both current costs and the economic hurdles associated with continued scaling.

2 Methodology

The methodology combines a database of notable models and historical hardware prices with three complementary approaches to estimate frontier-model costs. These approaches cover amortized hardware and energy costs, cloud rental prices, and development-wide compute plus R&D staff costs.

  • Model selection: 41 models were selected because they ranked among the top 10 in training compute at release.The source database documents 796 notable models, filtered to models published from 1 October 2015 through 31 December 2023.
  • Hardware and energy costs: Hardware costs include accelerator chips, servers, networking hardware, and depreciation based on a rate of r = 0.14 orders of magnitude per year.The amortized cost uses start value per chip and training chip-hours, with hardware failures neglected because their effect was judged small compared with hardware progress.
  • Hardware and energy costs: Energy costs combine energy price, hardware TDP, average power-to-TDP ratio, data-center PUE, and training chip-hours.TDP denotes thermal design power, while PUE accounts for power-distribution and cooling overhead.
  • Approach: Three approaches estimate frontier-model costs: amortized hardware CapEx plus energy, cloud rental prices, and development-wide compute plus R&D staff costs.The third approach was applied to GPT-3, OPT-175B, GPT-4, and Gemini Ultra.
  • Cloud compute cost: Cloud estimates multiply price per chip-hour by training chip-hours and may overestimate costs when developers use owned hardware.Prices were matched to hardware type, publication date, and likely cloud provider from a historical database.
  • Full development costs: Development-wide compute applies a multiplicative factor to final-run compute to include experiments, failed attempts, evaluation, and fine-tuning.The factor was sampled from a log-normal distribution with a 90% CI of 1.2x to 4x.
  • Full development costs: R&D staff costs include salaries and equity compensation for project researchers, engineers, and managers, excluding operations and data-center employees.Compensation estimates may vary significantly across AI labs, especially for small or early companies.

3 Results

Frontier-model training costs have risen rapidly across multiple estimation approaches, while hardware, energy, and staff analyses reveal distinct cost drivers and important uncertainty. The results also show that acquisition costs and projected future runs can create substantial capital barriers.

  • 3.2 Training costs may exceed $1 billion by 2027: $1 billion: extrapolating the 2.4× annual trend from GPT-4’s $40M amortized hardware and energy cost implies a largest publicly announced training run at about this cost by early 2027.The extrapolation uses GPT-4’s publication in March 2023 as the reference point.
  • 3.3 Hardware acquisition costs are one to two orders of magnitude higher than amortized costs: $800M versus $40M: estimated GPT-4 hardware acquisition cost was one to two orders of magnitude above its amortized hardware CapEx plus energy cost.Acquisition cost represents the upfront purchase of training hardware, whereas amortized cost spreads hardware use over its useful lifetime; their ratio depends on when and how long the model is trained.
  • 3.4 Half of amortized hardware CapEx + energy cost is for AI accelerator chips: 44%: AI accelerator chips were the largest average component of amortized hardware CapEx plus energy, followed by server components at 29% and cluster-level interconnect at 17%.Energy averaged 9% and varied across models, while the component breakdown excludes data-center infrastructure, personnel, and maintenance beyond servers and networking.
  • 3.5 R&D staff are a significant fraction of costs over the whole model development process: 29–49%: R&D staff costs including equity comprised this share of total amortized development costs for the selected models, while hardware comprised 47–64% and energy 2–6%.Excluding equity reduced the staff share to 21–33% and increased the hardware share to 61–76%; the analysis covers experiments, fine-tuning, evaluation, and final training.

4 Discussion

The paper compares cost-estimation approaches and argues that frontier AI development is becoming financially concentrated. It also emphasizes uncertainty from incomplete data, omitted costs, and limited generality.

  • 4 Discussion: If 2.4x annual cost growth continues, amortized frontier training runs will exceed $1 billion by 2027, with cloud estimates suggesting this could happen sooner.The authors caution that absolute cost estimates may be biased upward or downward even while growth trends remain informative.
  • 4 Discussion: Frontier AI development may concentrate among large companies and government institutions because only a handful possess the necessary financial resources.The paper links this concentration to reduced diversity in the perspectives and approaches represented in AI development.
  • 4 Discussion: The amortized hardware CapEx + energy approach and cloud rental approach agree on cost growth, but cloud estimates are about twice as large on average.The authors recommend the amortized approach when feasible because it better reflects practical costs and supports component-level analysis.
  • 4 Discussion: The most comprehensive approach includes R&D staff and compute for experiments, evaluations, and fine-tuning, but requires more evidence before becoming the default.For selected frontier models, R&D staff costs represent a major share of total development costs.
  • 4 Discussion: The estimates may be biased by public-data gaps, uncertain depreciation and pricing assumptions, omitted infrastructure costs, and limited generality beyond the selected frontier models.Rapid hardware and software efficiency improvements could also make historical cost trends difficult to extrapolate.

5 Conclusion

The paper finds that frontier AI compute costs have grown rapidly since 2016 and that R&D labor is a major component of total development costs. It warns that continued growth may create power and financial barriers to further scaling.

  • 5 Conclusion: 2.4x per year: amortized frontier compute costs grew at approximately this rate from 2016 onward, with a 90% confidence interval of 2.0x to 2.9x.The estimate is consistent across hardware-based and cloud-rental approaches.
  • 5 Conclusion: 29–49%: R&D staff costs comprised this share of total amortized development costs for GPT-3, OPT-175B, GPT-4, and Gemini Ultra.The result motivates further research on how R&D labor scales with computing power.
  • 5 Conclusion: Power capacity may become a bottleneck for larger computing clusters, making infrastructure constraints an important topic for future work.The authors identify this as a challenge alongside the rapid growth of training costs.
  • 5 Conclusion: $100 million+: frontier model development costs already exceed this level, suggesting that major advances may be accessible mainly to the largest companies and government institutions.The paper connects this concentration with questions about responsible development and deployment.

A Training cost estimation

The paper estimates hardware acquisition and amortized training costs by combining chip, server, networking, energy, and depreciation information. It uses separate approaches for GPUs and proprietary TPUs, with TPU costs inferred from manufacturing and equivalent-performance comparisons.

  • Hardware acquisition cost: Hardware acquisition cost sums chip, server, and networking expenses across the training cluster.The estimate includes the accelerator itself, other server components, chip-to-chip networking, and server-to-server networking.
  • Hardware acquisition cost: 19% of hardware acquisition cost is assigned to server-to-server networking, producing a 1.23× server-to-cluster adjustment factor.The authors note substantial uncertainty because this share varies with cluster architecture.
  • Cost of Google TPUs: TPU costs are estimated from low-end manufacturing costs and high-end equivalent-GPU prices, aggregated with a geometric mean.The resulting TPU cost estimates are about $5,000 per version for Google.
  • Amortization model: Amortized training cost equals the hardware start value multiplied by chip-hours and the depreciation rate, using a linear approximation.The approximation is considered valid when training time multiplied by r ln 10 is small; the analysis uses r = 0.14.
  • Amortization model: GPU start values use acquisition prices adjusted for hardware availability and training dates, while TPU timing uses announcement dates.For GPUs, the method applies a 90-day shipping buffer; when training time is unknown, it uses a median of approximately 33 days.

A.6 Accounting for compute used throughout model development

The cost model accounts for experiments, failed attempts, evaluation, and fine-tuning by scaling final-run compute to total model-development compute. Evidence from several models and Meta infrastructure supports a broad multiplier range.

  • Accounting for compute used throughout model development: Total model-development compute is estimated at 1.2× to 4× the final training run, with a median near 2×.The multiplier covers experiments, failed attempts, evaluation, and fine-tuning and is sampled from a log-normal distribution.
  • Accounting for compute used throughout model development: Smaller-model training and reported budgets show that substantial compute occurs outside the main training run.Smaller GPT-3 models account for at least 14% of compute, while BLOOM uses about 63% on smaller models; OPT-175B reports roughly twice the final-run cost.

A.7 Cost uncertainty analysis

The uncertainty analysis propagates uncertainty in input variables through simulated cost estimates. Hardware acquisition estimates are narrower than amortized hardware-and-energy estimates, especially for TPU-based models.

  • Cost uncertainty analysis: The simulation samples uncertain inputs with log-normal distributions, except for depreciation and utilization rates, which use normal distributions.An example input interval for per-GPU server overhead spans 1.3× to 2.1× at 90% credibility.
  • Cost uncertainty analysis: Hardware acquisition costs are generally accurate within a factor of two for GPU models and a factor of four for TPU models.TPU uncertainty is larger because equivalent TPU costs are harder to estimate.
  • Cost uncertainty analysis: Amortized hardware CapEx plus energy is generally accurate within factors of three or four for GPUs and five for TPUs.The estimates are most sensitive to GPU and TPU unit costs and training chip-hours.

A.8 Ground truth cost comparison

The authors compare estimates with independently reported costs for BLOOM-176B and OPT-175B. The cloud-price estimates align more closely with reported costs than amortized hardware-and-energy estimates in some cases, but both approaches are broadly comparable.

  • Ground truth cost comparison: The validation set is limited because few models publicly report both computing-resource usage and total cost.BLOOM-176B and OPT-175B are used because both quantities are independently known for these models.
  • Ground truth cost comparison: OPT-175B’s reported $1.98 million cost differs from the $1.5M cloud estimate by 25% and the $700K amortized estimate by 65%.The authors suggest the discrepancy may reflect a higher actual cluster cost rate or uncertainty in the developers’ reported hourly figure.

B Sensitivity analysis

Sensitivity analyses indicate that the estimated frontier-model cost trend is broadly stable across model-selection methods, top-N choices, and hardware depreciation assumptions.

  • Model selection: Selection methods generally produce similar results, but distance-from-trend selection yields a flatter cost trend by excluding many earlier models.
  • Model selection: Selecting the top 10 most compute-intensive models is preferred because it is more robust to dataset sampling than quantile- or trend-based alternatives.The database includes many models far from the compute frontier, which can distort methods based on quantiles or distance from the historic trend.
  • Top-N sensitivity: The estimated growth rate is moderately robust to top-N choice, remaining similar for N = 3, N = 5, and N = 20.
  • Hardware assumptions: Changing the GPU price-performance depreciation rate within 0.10–0.18 OOMs/year does not significantly change the amortized hardware CapEx + energy growth rate.These bounds decrease cost estimates by 15% or increase them by 10% on average, respectively.
  • Hardware assumptions: Testing an extreme depreciation rate of 0.3 OOMs/year also leaves the growth rate largely unchanged, while increasing estimated costs by 30% on average.

B.4 Varying the time between hardware acquisition and the start of training

The analysis varies dates governing hardware depreciation and reports their effect alongside the cost breakdown with equity excluded from R&D staff costs.

  • Varying depreciation timing: Removing the 90-day acquisition buffer and moving training start 15 days earlier decreases estimated costs by 4%.
  • Excluding equity: Excluding equity reduces R&D staff costs from 29–49% to 19–33% of total amortized model development costs.

C Power capacity for model training

The required power capacity of frontier-model training clusters is estimated from hardware quantity, hardware TDP, and data-center PUE, and is projected to rise sharply.

  • Measurement: Power capacity is calculated as hardware quantity × hardware TDP × data-center PUE, with hardware TDP covering all server hardware.
  • Power capacity trend: 2.2x per year is the estimated growth rate of training-cluster power capacity, with a 90% CI of 1.9x to 2.6x.
  • Power capacity trend: Around 35 MW is the estimated power capacity of Gemini Ultra, the largest among the models discussed.
  • Power capacity trend: 1 GW is the projected power demand of the most power-intensive training run at some point in 2028.The paper compares this projection with 3–7 GW capacities for the ten largest U.S. power plants.
Loading 2405.21015v2…