Source-linked AI summary
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, Jeff Dean
TL;DR
ML training’s growing carbon footprint has prompted concern, alongside evidence that published estimates can be substantially overstated. The paper describes four practices and accounting methods, finding large reductions in emissions while Google’s ML energy share remained below 15%; it concludes that broad adoption could make training emissions plateau and then shrink.
Problem
Growing ML workloads have raised concerns about carbon emissions, while some published studies significantly overestimated actual emissions.
Method
The paper identifies four best practices and estimates ML energy and carbon footprints using processor use, datacenter overhead, and energy carbon intensity.
Results
ML energy use remained below 15% of Google’s total energy use for three years, while GLaM produced 14x less CO2e than GPT-3 and improved model quality.
Takeaways & Limitations
If ML communities adopt these practices, the paper predicts that total carbon emissions from ML training will first plateau and then shrink.
Takeaways & Limitations
The analysis focuses on operational emissions; estimating lifecycle emissions is left to a larger future study.
Abstract
from arXiv · showhide
Machine Learning (ML) workloads have rapidly grown in importance, but raised concerns about their carbon footprint. Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use (across research, development, and production) held steady at <15% of Google's total energy use for the past three years. If the whole ML field were to adopt best practices, total carbon emissions from training would reduce. Hence, we recommend that ML papers include emissions explicitly to foster competition on more than just model quality. Estimates of emissions in papers that omitted them have been off 100x-100,000x, so publishing emissions has the added benefit of ensuring accurate accounting. Given the importance of climate change, we must get the numbers right to make certain that we work on its biggest challenges.
1. Introduction
ML training emissions have been substantially overstated, while four best practices can reduce energy and carbon footprints by orders of magnitude without sacrificing model quality. The paper argues that adopting these practices can keep ML’s energy share low and ultimately shrink training emissions.
- Field-level impact: ML’s energy share held below 15% of Google’s total energy use for the past three years despite growing workloads.The paper presents this as evidence associated with following best practices.
- Accounting: Published faulty estimates and extrapolations are 100x–100,000x higher than the real carbon footprints.The paper highlights accurate emissions accounting as a reason to report energy and carbon explicitly.
- Scope: The analysis focuses on operational emissions, while lifecycle emissions require a larger future study.Operational emissions cover energy used by ML hardware and datacenter overheads; lifecycle emissions also include manufacturing-related embedded carbon.
- Best practices: 100x lower energy and 1000x lower carbon emissions are achievable by applying four best practices instead of orthodox choices.The practices are model, machine, mechanization, and map.
- Best practices: 83x lower energy consumption and 747x lower CO2 emissions were achieved over four years while maintaining the same quality.The comparison models training the same Transformer under progressively improved choices.
- Evidence: 750x lower emissions without accuracy loss was shown for Transformer, while a larger GLaM model achieved 14x lower emissions and improved accuracy relative to GPT-3.These are two studies of best-practice impacts.
2. Overview of Energy and CO2e for ML Training
The paper estimates ML training energy and carbon by combining hardware use, datacenter overhead, and energy carbon intensity. It defines the main accounting terms and shows that datacenter location materially affects carbon intensity.
- Key terms: PUE is the ratio of total datacenter energy, including overheads such as cooling, to computing-equipment energy.The average 2020 industry PUE was 1.58, while cloud providers had PUEs of approximately 1.10.
- Key terms: Carbon intensity measures datacenter energy cleanliness in tCO2e per MWh.Average datacenter carbon emissions in 2020 were 0.429 tCO2e per MWh, with some Google datacenters 5x lower.
- Energy accounting: MWh equals training hours multiplied by processor count and average processor power before datacenter PUE is applied.The estimate includes server components and datacenter energy used to power and cool hardware.
- Carbon accounting: Carbon emissions are obtained by multiplying energy consumption by the carbon intensity of the energy supply.The paper uses carbon intensity for Google datacenters, derived from location-specific energy data.
3. Case Study 1: Transformer vs. Evolved Transformer vs. Primer
The Transformer family illustrates how model architecture, specialized hardware, datacenter efficiency, and location can sharply reduce training energy and CO2e while preserving quality. Across successive optimizations, the possible reduction reached 747x relative to the 2017 reference.
- Model: 4.2x faster training was achieved by Primer while matching the original Transformer’s quality scores.Evolved Transformer was ~1.3x faster than Transformer while also matching its quality scores.
- Machine: 13.7x lower energy consumption came from TPUv4 relative to unoptimized 2017 P100s.TPUv2 reduced energy by 5.7x in 2019; improvements reflect specialized logic, fabrication, and training-to-hardware mapping.
- Mechanization: 1.4x reduction came from Google Cloud’s better PUE compared with the average datacenter.
- Map: Datacenter location substantially affected carbon intensity, with Oklahoma scoring 96% CFE in 2019 and Iowa 93% in 2020.
4. Case Study 2: GPT-3 vs GLaM
The GPT-3 and GLaM comparison shows that a larger, sparse model can improve quality while reducing training resource use. GLaM used fewer accelerator years and less energy, and its lower-carbon location further reduced gross CO2e.
- 95B active parameters per token represented 8% of GLaM’s 7x larger parameter count, versus all 175B GPT-3 parameters being active.Selective expert activation provides GLaM’s sparsity.
- GLaM used 2.8x fewer accelerator years and 2.8x less energy than GPT-3 while improving quality.GLaM used TPUv4s; GPT-3 took 405 V100 years to train.
- ~5x lower carbon intensity in Oklahoma contributed to GLaM’s lower gross CO2e: 0.088 versus 0.429 tCO2e per MWh.
- ~14x lower gross carbon footprint was achieved by GLaM only 18 months after GPT-3 despite higher accuracy.Continuous improvements in algorithms and hardware make extrapolating current ML trends prone to overestimating future energy use and CO2e.
- Halving GLaM’s training time, energy, and CO2e would have been possible if it had matched rather than exceeded GPT-3’s quality scores.Google’s renewable energy purchases further reduce the impact to zero.
5. Overall ML Energy Consumption
Google’s measurements indicate that ML’s share of total energy remained stable despite growing ML usage. Efficiency improvements kept the share at 10%–15% across 2019–2021, while renewable-energy matching can make net carbon impact zero for some companies.
- 15.4 TWh was Google’s annual 2020 energy consumption, while GPT-3 training represented ~0.012% of Microsoft’s 2020 energy and GLaM ~0.004% of Google’s.
- ML energy estimates included TPUs, GPUs, associated servers and networking, plus CPU consumption attributable to inference.
- The CPU estimate likely overstates ML energy because some libraries are non-ML, host CPUs may be double-counted, and some GPU use is for graphics.
- 10%–15% of Google’s weekly total energy was attributable to ML in 2019, 2020, and 2021.ML represented 70%–80% of Google’s FLOPS, and about three-fifths of ML energy use was for inference.
- ML’s energy share was not increasing despite growing applications, with algorithmic and hardware improvements keeping growth comparable to Google’s overall energy growth.
- 100% renewable-energy matching means some companies’ net carbon impact from ML computations could be considered zero.The renewable energy need not be generated in the same location as consumption.
6. Additional Factors
Additional analyses address whether neural architecture search increases total energy use and whether client-side ML is significant. The evidence indicates that reused efficient models can offset search costs and that server-side ML dominates mobile-device ML energy.
- Neural Architecture Search: 15x the NAS energy cost was saved when the Evolved Transformer was used to train Meena.The Evolved Transformer search used 7.5 MWh; Primer’s search used 6.2 MWh.
- Neural Architecture Search: NAS is likely a net environmental gain when discovered models are trained more than a few times.Efficient models may be reused hundreds or thousands of times, allowing downstream savings to exceed the initial search effort.
- Client-side ML: <1% of a Google Pixel 6 phone’s daily energy was consumed by its Edge TPU for ML workloads.Client ML libraries and applications were similarly minor contributors.
- Client-side ML: 0.4 TWh was the estimated upper bound for ML energy on all 3.8 billion mobile phones in 2021.The estimate assumes nightly charging and accounts for charger inefficiency.
- Client-side ML: Server-side ML energy use was clearly dominant: Google’s 2020 ML server energy was ~6 times the conservative estimate for ML on all mobile phones.
7. Related Work
Prior studies developed tools and recommendations for understanding ML’s climate impact, but widely cited estimates substantially overstated actual emissions. The paper attributes these errors to missing information and incorrect assumptions about hardware, datacenter conditions, and NAS computation.
- A prior framework proposed a leaderboard and tool for collecting energy use and CO2e from preliminary training runs.
- FLOPs is not a reliable proxy for energy because runtime and energy can be uncorrelated with operation count.AutoML found faster models using 2.4x as many FLOPs.
- Using older, non-ML-optimized GPUs and average datacenter factors instead of Google-specific values explained another 5x difference.
- The NAS estimate was 18.7x too high because the search used a small proxy task rather than full-size tasks.
- Published estimates overstated the Evolved Transformer NAS footprint by 88x, reporting 284 tCO2e instead of the actual 3.2 tCO2e.The discrepancy arose from both computation and datacenter assumptions.
- The authors recommend calculating and publishing energy use and carbon footprint because retrospective estimation is difficult to make accurate.
8. Conclusion
The conclusion argues that technology improvements and operational best practices have offset rapid growth in ML workloads, while production decisions should consider training and serving together. It recommends transparent reporting and greener infrastructure, and identifies lifecycle manufacturing emissions as a likely larger computing challenge.
- ML energy use remained below 15% of Google’s total energy use for three years despite rapidly increasing workloads and computation.Technology improvements largely compensated for the increased load.
- GLaM produced 14x less CO2e than GPT-3 training and used 0.004% of Google’s annual energy while improving model quality.
- Training and serving emissions should be minimized together in large-scale production, because methods that raise training emissions can reduce serving emissions and total footprint.Neural architecture search is given as an example.
- Datacenter providers should publish PUE, %CFE, and CO2e/MWh by location to help customers reduce energy use and carbon footprint.
- ML practitioners should use effective processors in the greenest accessible datacenter, often in the Cloud.
- The authors predict ML training’s total carbon footprint will first plateau and then shrink if communities adopt the identified practices.
- Published studies overestimated training costs and carbon footprints because they lacked information or extrapolated without accounting for algorithmic and hardware improvements.
Short Author Bios
Maud Texier leads Google’s 24/7 carbon-free energy development and scaling effort for data centers. Her research covers carbon abatement, carbon-free energy, grid modernization, and decarbonization.
- Maud Texier is Google’s Head of Energy Development and leads development and scaling of 24/7 carbon-free energy for data centers.
- Her research interests include carbon abatement technologies, carbon-free energy technologies, grid modernization, and decarbonization.
- Texier holds an MS in engineering in Energy and Power Systems from École Centrale Paris.