Source-linked AI summary
Carbon Emissions and Large Neural Network Training
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, Jeff Dean
TL;DR
Growing ML computation increases energy and carbon costs, while estimating those costs remains difficult without detailed lifecycle information. The paper measures energy use and CO2e for large NLP models and examines architecture, hardware, datacenter, and location effects. It finds substantial efficiency opportunities and argues for explicit reporting and energy-aware evaluation.
Problem
Estimating the energy use and CO2e of large NLP models is difficult because key information about hardware, datacenters, energy mix, and training lifecycle is often unavailable.
Method
The paper calculates energy use and CO2e for five recent large NLP models, refines an Evolved Transformer NAS estimate, and analyzes model, processor, datacenter, and energy-mix factors.
Results
Large sparsely activated models can use less energy than dense models, while processor, datacenter, and energy-mix choices substantially change training emissions.
Takeaways & Limitations
ML papers using substantial computation should report energy consumption and CO2e, and efficiency should be evaluated alongside accuracy.
Takeaways & Limitations
Full model-development costs remain difficult to compare because setup attempts and other lifecycle expenses are rarely reported.
Abstract
from arXiv · showhide
The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.
1. Introduction
As ML models grow, their capabilities and computing demands increase, making energy and CO2e impacts difficult to estimate without detailed information. The paper evaluates recent large NLP models and recommends transparent reporting and efficiency-aware evaluation.
- Larger ML models generally become more accurate and capable while demanding more computation and energy.
- The study estimates energy use and CO2e for T5, Meena, GShard, Switch Transformer, and GPT-3.
- 88X lower CO2e was obtained for the earlier Evolved Transformer neural architecture search estimate by characterizing its actual hardware and datacenter.
- Accurate assessment is difficult because datacenter, hardware, energy mix, and other required information is often unavailable or hard to reconstruct retrospectively.
- The paper encourages researchers to report energy usage, CO2e, accelerator counts, and training time for computationally intensive models.
2. Energy Consumption and Carbon Footprint of an NLP Model
The paper models training carbon footprint from algorithmic, hardware, datacenter, and energy-supply factors, then uses measured examples to show how these choices change energy and CO2e. It emphasizes that model architecture, processors, datacenter efficiency, and location jointly determine emissions.
- Training energy depends on the algorithm, implementation, processor count, processor power and speed, datacenter efficiency, and energy supply mix.
- Inference can account for 80–90% of ML workload, but the paper focuses on training because its energy use is more isolated and easier to investigate.
- 57X overall CO2e improvement demonstrates the importance of selecting the DNN, processor, datacenter, and geographic location.
- Model improvement: 1.6X fewer FLOPS and 1.1X–1.3X less time let Evolved Transformer match Transformer Big accuracy while reducing training cost and CO2e.
- Processor improvement: 4.3X and 5.2X faster execution on TPU v2, with 5.6X and 6.2X performance/Watt gains, improves Transformer Big and Evolved Transformer efficiency respectively.
- Datacenter improvement: 1.4X better PUE characterized the Google Iowa datacenter than the 2018 US national average, while cloud datacenters are roughly 2X as energy efficient as typical enterprise datacenters.
- Energy mix: 5.4X lower net CO2e intensity in Iowa followed from matching local clean-energy purchases to the datacenter’s time and location.
3. Energy Usage and CO 2 e Emissions of Five Recent Large NLP Models
The section compares energy and CO2e for five large NLP models, emphasizing that sparse activation can substantially reduce computation, energy, and emissions relative to dense models.
- The analysis estimates energy use and CO2e for T5, Meena, GShard, Switch Transformer, and GPT-3.The models are listed in the section’s comparison table, with hardware and emissions assumptions documented in Appendix A.
- GShard-600B used 24 MWh and produced 4.3 net tCO2e, despite having 600B total parameters through sparse computation.Sparse models can contain many parameters while requiring much less computation than dense models.
- GPT-3 training consumed 1287 MWh and produced 552 tCO2e, with 175B parameters.
- Compared with dense GPT-3, sparsely activated GShard used ~55X less energy and reduced gross CO2e by ~115X.GShard also required ~45X fewer processor years.
4. Discussion
The discussion argues that neural architecture search costs are difficult to estimate retrospectively and that reporting the full development lifecycle would improve comparisons and energy-aware model selection.
- 4.1 Estimating the cost of neural architecture search (NAS): Evolved Transformer NAS cost nearly two orders of magnitude less than previously estimated because researchers used a small proxy task before scaling models up.The earlier estimate assumed the search used full-size tasks and was 18.7X too high for the average organization.
- 4.1 Estimating the cost of neural architecture search (NAS): Full model-development costs are rarely reported, making comparison with prior work and among efficiency approaches impossible or difficult.
- 4.1 Estimating the cost of neural architecture search (NAS): Evolved Transformer achieved higher accuracy at lower cost than comparable Transformer models, running 1.3X faster and producing 1.3X less CO2e.
- 4.2 There are more resources used for training than the only final training run: Many setup attempts before final training mean the final run does not represent total training cost.The paper discusses energy tracking and lifecycle measurement as ways to capture these preliminary costs.
4.3 Measurements are more interesting than extrapolations
This section favors measured experiments over hypothetical extrapolations because carbon estimates depend strongly on the actual hardware, datacenter, and workload assumptions.
- Hypothetical extrapolations should receive less attention than measurements from experiments that were actually conducted.
- Large NLP models may use smaller proxy tasks to explore the model space because full training can take a month.The cited AutoML example reports exploration computation roughly equal to final training cost.
- A hypothetical Transformer (Big) run on a non-ML-optimized supercomputer might use ~75 MWh and generate 32 tCO2e.The estimate is described as ~10,000 times larger than training on Google TPU v2s.
4.4 Standard ML algorithmic techniques can improve energy efficiency
Standard algorithmic techniques can reduce computation, energy use, and carbon emissions while preserving accuracy. Sparsely activated mixture-of-experts models are highlighted as especially effective.
- Algorithmic techniques can achieve the same accuracy with less computation or create lighter models from already-trained networks.These approaches reduce computational cost, energy use, and carbon emissions.
- Fine-tuning and transfer learning reuse trained representations instead of randomly initializing each NLP task.
- More than 10X reductions in computation and energy are possible with sparsely activated mixture-of-experts models.They can also provide significantly higher accuracy than dense Transformer or LSTM models at equivalent computational cost per token.
- Training location and timing are practical opportunities for reducing CO2e because datacenter choice can materially affect workload emissions.The authors report that colleagues switched to a Google datacenter with a smaller carbon footprint after reading early drafts.
4.6 Many have access to energy-optimized datacenters
Cloud computing has improved datacenter energy intensity and provides broad access to energy-optimized infrastructure. However, expensive training remains an economic and equity concern, while comparisons show substantial emissions associated with large digital activities.
- 20% annual decreases in datacenter energy intensity have accompanied the increasing use of cloud computing since 2010.Energy intensity is measured as energy use per compute instance.
- Alibaba, Amazon, Google, and Microsoft offer globally accessible cloud computing, including custom DNN processors through cloud services.
- Expensive NLP-model training limits access to state-of-the-art models and motivates efficiency improvements alongside environmental concerns.
- 180 tCO2e is the estimated emissions of a direct round trip by a whole passenger jet between San Francisco and New York.The paper uses this flight as a comparison point for ML and other activities.
4.9 Are the benefits of NLP models worth the energy cost?
The paper presents NLP models as delivering practical benefits while emphasizing that sparse multilingual models can improve translation quality efficiently. Examples include scientific literature search and improved search snippets across languages.
- BERT powers the COVID-19 Research Explorer, helping scientists find answers or evidence in COVID-19-related articles.Its training consumed ~2.8 MWh and produced 0.13 tCO2e.
- BERT significantly improved featured snippets in Hindi, Korean, and Portuguese.Featured snippets are short summaries displayed at the top of Google search results.
- Low-resource languages receive greater BLEU improvements than high-resource languages in the multilingual GShard comparison.The figure groups source languages by available training resources and compares translation to English across 100 languages.
- 1000 queries per day were served by the COVID-19 Research Explorer at launch despite its narrow scientific audience.Organizations including Pfizer, AstraZeneca, and the National Institutes of Health expressed interest in the tool.
- 13.5 BLEU is the average quality improvement reported for GShard-600B over the 2.3B dense model.The 600B sparse MoE model required ~10X fewer computational resources to train than the dense model despite higher translation quality.
5. Conclusion
The conclusion calls for more transparent reporting and stronger efficiency incentives in computationally intensive ML. It identifies model, hardware, datacenter, and geographic choices as important determinants of CO2e.
- Computationally intensive ML papers should report energy consumed, CO2e, training time, and processor count explicitly.
- ML conferences should reward efficiency improvements alongside traditional metrics, and MLPerf is expected to include power.
Appendix A. Details of CO 2 Estimates for Four Large NLP Models in Tables 1 and 4
The appendix documents how the CO2e values in Tables 1 and 4 were derived, including datacenter metrics and measurement choices. The 2020 datacenter measurements remain provisional pending validation.
- Datacenter CO2e estimates use gross and net emissions per kilowatt-hour, PUE, and measured average power.Google-specific inputs include datacenter PUE and measured system power rather than chip TDP alone.
- 2020 datacenter measurements are provisional and await final validation in May 2021.
Appendix B. Carbon Offset and 24/7 Carbon Free Energy
The appendix distinguishes conventional carbon offsets from Google’s 24/7 Carbon Free Energy framework. The latter accounts for carbon-free energy hourly at each location and can guide the timing and placement of compute.
- Carbon offsets: Conventional carbon offsets match electricity consumption with clean-energy certificates under Scope 2 accounting.Current accounting can compensate consumption in one place and month with clean energy in another place and month.
- 24/7 Carbon Free Energy: 24/7 Carbon Free Energy resets accounting every hour, counting shortfalls without granting extra credit for hourly oversupply.
- 24/7 Carbon Free Energy: Google reports datacenter gross CO2e from grid energy separately from net CO2e after applying local renewable-energy purchases.
- Workload scheduling: Google shifts non-urgent compute tasks toward periods and datacenters where carbon-free power is more plentiful.
Appendix C. Details of a CO 2 e Estimate for NAS in an Average Datacenter
The appendix reconstructs an NAS carbon estimate under average U.S. datacenter and grid assumptions. It reports a substantially lower estimate than the earlier estimate and identifies the calculation inputs.
- 626,155 pounds (284 tCO2e) was the earlier estimate for Evolved Transformer NAS under average U.S. P100 and grid assumptions.
- 33,544 lbs (15.2 t) is the appendix’s calculated CO2e estimate using one chip, estimated training time, and the stated emissions factor.
- Adjusting for the actual hardware and datacenter produces a further ~5X reduction in CO2e relative to the hypothetical average-datacenter setup.
Appendix D. Details of a CO 2 e Estimate for Google’s Actual NAS
The appendix adjusts the NAS estimate to reflect Google’s actual execution environment rather than average U.S. assumptions. The adjustments include a lower measured PUE and a different power accounting basis.
- Google’s Georgia datacenter PUE was 1.10, compared with the 2018 U.S. average PUE of 1.58.
- The earlier estimate used the 2018 U.S. average emissions intensity of 0.423 kg per kWh.
- The authors measured 296 Watts for P100 Transformer (Big), versus 189 Watts estimated for GPU, host CPU, and host memory.The 296-Watt measurement included additional rack components such as TPU memory, switches, fans, and power supplies.