PaperlayerView exact revision on arXiv

Source-linked AI summary

Carbon Emissions and Large Neural Network Training

David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, Jeff Dean

arXiv:2104.10350v3cs.LGcs.CY

TL;DR

Estimating the environmental impact of large NLP models is difficult when relevant details are unavailable. This paper estimates energy use and CO2e for recent large models and reports that GShard-600B is more efficient than other large NLP models.

  • Problem

    Estimating the environmental impact of large NLP models is difficult, partly because datacenter, hardware, and energy-mix information is rarely reported or publicly available.

    Source: §1 · pp. 1–2

  • Method

    The paper estimates energy use and CO2e for five recent large NLP models, refines the Evolved Transformer search estimate, and accounts for model, hardware, datacenter, and energy-supply factors.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    GShard-600B operates more efficiently than other large NLP models, and ML accelerators are more efficient than off-the-shelf hardware.

    Source: §5 · p. 15

  • Takeaways & Limitations

    The authors recommend that compute-intensive ML papers report energy use and CO2e explicitly and treat efficiency as an evaluation criterion.

    Source: §1 · p. 2

    Full model-development costs are rarely reported, preventing straightforward comparisons among different approaches.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

1. Introduction

The paper estimates operational energy use and CO2e for recent large NLP models, revises an earlier Evolved Transformer search estimate, and calls for transparent reporting and efficiency incentives.

Introduction

  • 88X lower CO2e was estimated for the Evolved Transformer neural architecture search after characterizing its actual hardware and datacenter use.The study also estimates energy use and CO2e for T5, Meena, GShard, Switch Transformer, and GPT-3; it covers computer and datacenter operation, not fabrication or recycling.Source: §1 · p. 1
  • Researchers should measure energy use and CO2e, or use a rough-estimate tool such as ML Emissions Calculator [Lac19], and publish the data.Precise estimates are difficult because datacenter, hardware, and energy-mix information is often unavailable and can be hard to reconstruct later.Source: §1 · p. 2
  • Efficiency should join accuracy as an evaluation criterion for computationally intensive ML research [Str19] [Hen20].The authors argue that sustainable energy is energy not used.Source: §1 · p. 2
  • Researchers should report accelerator counts and training time for computationally intensive models, following the recommendation in [Str19].The authors argue that reducing training time matters even if cloud-datacenter CO2e reaches zero, because it lowers costs and broadens participation.Source: §1 · p. 2

2. Energy Consumption and Carbon Footprint of an NLP Model

An ML model’s carbon footprint depends on its algorithm, processors, datacenter efficiency, and energy supply mix. The paper focuses on training, while noting that inference emissions are significant.

2. Energy Consumption and Carbon Footprint of an NLP Model

  • 57X overall CO2e improvement was reported for Evolved Transformer (Medium) on TPU v2s in Google Iowa versus Transformer (Big) on P100 GPUs in an average US datacenter.The comparison is shown in Figure 1; Table 1 provides the CO2e breakdown.Source: §2 · p. 2, Figure 1 · p. 3, Table 1 · p. 3

2.1 Algorithm/program improvement

  • Evolved Transformer (Medium) used 1.6X fewer FLOPS and 1.1X–1.3X less time than Transformer (Big), at slightly higher accuracy.Faster training saves researcher time, organizational money, and CO2e.Source: §2.1 · p. 4

2.2 Processor improvement

  • TPU v2 ran Transformer (Big) 4.3X faster and Evolved Transformer (Medium) 5.2X faster than P100 GPUs, with 5.6X and 6.2X performance/Watt gains, respectively.TPU v2 used 1.3X less power for Transformer (Big) and 1.2X less for Evolved Transformer (Medium).Source: §2.2 · p. 4

2.3 Datacenter improvement

  • The Iowa datacenter’s PUE of 1.11 was 1.4X better than the 2018 US national average PUE of 1.58.Cloud datacenters are roughly 2X as energy efficient as typical enterprise datacenters, though the paper quantifies improvement using PUE.Source: §2.3 · p. 4

2.4 Energy mix improvement

  • Net CO2e intensity in the Iowa datacenter was 0.080 kg CO2e/KWh, 5.4X better than the U.S. average gross intensity of 0.429 kg CO2e/KWh.Matching clean-energy purchases to the same local grid and hour supports the net accounting; Iowa’s wind portfolio lowered December 2020 gross intensity from 0.478 kg to 0.080 kg, a 6X reduction.Source: §2.4 · pp. 4–5

2.5 Summary: Formulas for energy consumption and carbon footprint of training

  • Practitioners can estimate training energy from hours, processor count, average processor power, and datacenter PUE, then estimate CO2e using energy and the datacenter’s CO2e intensity.When measured processor power or datacenter-specific data are unavailable, the paper suggests using averages from similar models, published PUE or the US average, and regional emissions tools.Source: §2.5 · p. 5

3. Energy Usage and CO 2 e Emissions of Five Recent Large NLP Models

Training energy and CO2e vary across large NLP models; sparsely activated models use many more parameters with lower total FLOPS, and GShard’s footprint is far below dense GPT-3’s.

3. Energy Usage and CO 2 e Emissions of Five Recent Large NLP Models

  • T5 [Raf19] used 86 MWh and produced 47 tCO2e; Meena [Adi20] used 232 MWh and produced 96 tCO2e.Meena is a 2.6B-parameter model; Evolved Transformer saved 48 tCO2e in developing Meena, against 3.2 net tCO2e development cost, a 15:1 return.Source: §3 · pp. 5–6, Table 4 · p. 6
  • GShard-600B [Lep20] [Sha17] used 24 MWh and produced 4.3 net tCO2e; Switch Transformer [Fed21] used 179 MWh and produced 59 net tCO2e.Switch Transformer has 1500B parameters but activates only 0.1% per token, enabling up to 7x pre-training speed with the same computational resources.Source: Table 4 · p. 6, §3 · p. 6
  • GPT-3 [Bro20] training consumed 1287 MWh and produced estimated emissions of 552 tCO2e.OpenAI measured GPT-3 performance, power, total FLOPS, and PUE; the estimate used the US average CO2e per kWh for Microsoft Azure.Source: Table 4 · p. 6, §3 · p. 7
  • Compared to dense GPT-3, sparsely activated GShard needs ~45X fewer processor years, uses ~55X less energy, and reduces gross CO2e ~115X and net CO2e ~130X.Sparsely activated models use many more parameters with much lower total FLOPS than Transformer (Big).Source: Figure 3 · p. 7, §3 · p. 8

4. Discussion · 4.1 Estimating the cost of neural architecture search (NAS)

The discussion revisits the cost of Evolved Transformer neural architecture search and shows how its energy and emissions costs compare with the benefits of reusing the resulting architecture.

4. Discussion

  • The discussion addresses additional carbon-emissions factors in NLP training, revisiting NAS estimates and presenting benefits of some NLP models.Source: §4 · p. 8

4.1 Estimating the cost of neural architecture search (NAS)

  • The actual Evolved Transformer NAS cost was nearly two orders of magnitude below the earlier estimate; proxy-task confusion made that estimate 18.7X too high for average organizations and 88X off for energy-efficient organizations [Str19].Researchers searched using a small proxy task and scaled the resulting models to full size, but the earlier estimate assumed full-size search tasks.Source: §4.1 · p. 8
  • NAS is generally run once per problem-domain and architectural-search-space combination, not once per model training; Evolved Transformer found for translation can also serve language modeling without a new search [So19] [Adi20].Source: §4.1 · p. 8
  • Reusing Evolved Transformer to train Meena saved 48.5 t CO 2 e, about 15X the energy cost of the search that discovered it.Source: §4.1 · p. 8
  • The Evolved Transformer NAS results are open-sourced and can be reused for NLP model training.Source: §4.1 · p. 9
  • Because full model-development costs are rarely reported, the Evolved Transformer cost-savings ratio cannot be compared with prior architecture work [Dod19] [Str19].The authors argue that measuring and reporting energy requirements would clarify cost-accuracy tradeoffs and support more informed model selection, potentially reducing emissions.Source: §4.1 · p. 9
  • At 131M and 210M parameters, Evolved Transformer achieves higher accuracy at less cost, running 1.3X faster and producing 1.3x less CO 2 e than plain Transformer models [So19].Figure 4 compares plain Transformer models with the open-sourced Evolved Transformer architecture found by NAS.Source: Figure 4 · p. 9

4.2 There are more resources used for training than the only final training run · 4.3 Measurements are more interesting than extrapolations

The final training run can omit substantial development costs, making lifecycle measurement important. The section favors measured experiments over hypothetical emissions estimates and cautions that theoretical efficiency and calculator outputs can differ from measured results.

4.2 There are more resources used for training than the only final training run

  • The final training run omits setup attempts, so papers reporting only that run cannot capture the full training cost.The passage describes tools intended to record training information and energy, including experiment-impact-tracker [Hen20] and CodeCarbon [Lac19].Source: §4.2 · p. 9
  • Researchers could study the total lifecycle to develop rules of thumb for estimating overall carbon footprint from final training cost, and detailed, checkable reporting would help deter fudging.The passage says an internal Google product plans to add energy consumption tracking to information originally collected for data provenance.Source: §4.2 · pp. 9–10

4.3 Measurements are more interesting than extrapolations

  • Actual experiments deserve more attention than hypothetical case studies because carbon-emissions extrapolations are relatively easy to produce.Source: §4.3 · p. 10
  • For large NLP models, developers may explore with smaller tasks and limited training time; AutoML work [Li21] reported exploration computation roughly equal to final-training cost.The passage notes that month-long training makes repeating the full task unaffordable.Source: §4.3 · p. 10
  • Theory overstates performance/Watt by 1.6X for TPUs and 3.5X for GPUs; V100 is 1.5X better than TPU v3 theoretically, while measured TPU v3 is 2.0X better.The comparisons are for large NLP DNNs.Source: Figure 5 · p. 10, §4.3 · p. 11
  • ML Emissions and Green Algorithms estimates differ from measured gross CO2e by 0.53–1.64 (geometric mean 0.92) and 0.91–2.42 (geometric mean 1.48), respectively.The calculators do not estimate net CO2e, which could be up to 10X lower; measurement is preferred, with a calculator as an alternative when measurement is difficult.Source: §4.3 · p. 11, Figure 6 · p. 11
  • A hypothetical CTS-1 estimate put Transformer (Big) at ~75 MWh and 32 tCO2e, ~10,000 times the emissions for TPU v2s at Google.The example concerns a supercomputer not optimized for ML training and is presented as a hypothetical, not a measurement.Source: §4.3 · p. 11

4.4 Standard ML algorithmic techniques can improve energy efficiency · 4.5 It matters which datacenter is used, even within the same organization

The paper describes algorithmic techniques that reduce ML computation and energy use, and argues that choosing where and when to train can lower CO2e without merely shifting work to dirtier datacenters.

4.4 Standard ML algorithmic techniques can improve energy efficiency

  • Pruning, quantization, and efficient coding can improve DNN energy efficiency 3X–7X.Source: §4.4 · p. 11
  • Distillation transfers knowledge from large models to smaller, efficient models with almost the same accuracy; fine-tuning and transfer learning reuse trained representations instead of random initialization.Source: §4.4 · pp. 11–12
  • Sparsely activated mixture-of-experts models can cut computation requirements and energy costs by more than 10X while achieving significantly higher accuracy than dense models at equivalent per-token computational cost.The comparison covers training and inference and names dense Transformer and LSTM-based models as baselines.Source: §4.4 · p. 12

4.5 It matters which datacenter is used, even within the same organization

  • Choosing where and when to train can be an easy path to lower CO2e; colleagues switched to a lower-carbon Google datacenter for a large NLP model.Source: §4.5 · p. 12
  • Datacenters are not always fully utilized, cloud sites can expand, and renewable energy supply can grow through investment, challenging proposed limits on greener training.Source: §4.5 · p. 12
  • ML training can be limited to a smaller set of green datacenters because ML supercomputers use separate interconnection networks from CPU-only tasks; user-facing inference still needs global distribution.Source: §4.5 · p. 12
  • Google reports carbon neutrality since 2007 and renewable procurement matching 100% of datacenter energy use since 2017, and targets matching hourly local carbon-free energy by 2030.Source: §4.5 · p. 12

4.6 Many have access to energy-optimized datacenters · 4.7 Reducing the cost of training matters too · 4.8 How does training a large NLP model compare to other activities?

Cloud datacenters are increasingly energy-efficient and broadly accessible, but reducing training cost and energy use remains important for research access and environmental impact. Comparisons with flights, Bitcoin, and Google’s total energy use put the scale of large-model training in context.

4.6 Many have access to energy-optimized datacenters

  • Cloud datacenter energy intensity has decreased 20% annually since 2010 [Has20], and public cloud business has grown by up to 50% annually since 2010.People worldwide can rent cloud computers, and Alibaba, Amazon, and Google offer custom DNN processors through their cloud services.Source: §4.6 · p. 13

4.7 Reducing the cost of training matters too

  • Reducing training cost remains important because expensive NLP training can make this research style unattainable for many researchers and limit access to state-of-the-art models.The passage gives a 4 to 552 tCO2e range for the training emissions it discusses and motivates energy-efficient models that match computationally hungrier counterparts.Source: §4.7 · p. 13

4.8 How does training a large NLP model compare to other activities?

  • Against a 180 tCO2e passenger-jet round trip, T5 training was ~26%, Meena 53%, GShard-600B ~2%, Switch Transformer 32%, and GPT-3 ~305%.The comparison is for a direct round trip between San Francisco and New York.Source: §4.8 · p. 13
  • A Bitcoin purchase transferring bitcoin uses ~700 KWh or ~0.3 tCO2e, equivalent to ~750,000 credit card swipes; 2021 mining estimates are ~78–121 TWh and ~37M–58M tCO2e.Bitcoin miners use custom chips that operate continuously until they fail.Source: §4.8 · p. 13
  • Even assuming all four large Google NLP models were trained in 2019, their combined training represented less than 0.005% of Google’s 12.2 TWh energy use that year.The passage concludes that these four model trainings were not a significant fraction of Google’s energy consumption.Source: §4.8 · p. 13
  • Bitcoin’s estimated annual carbon footprint equates to roughly 200,000–300,000 whole SF–NY passenger-jet round trips, while Google uses 1/10th of Bitcoin’s energy to serve billions of people and offsets all its energy use.The passage also notes that Bitcoin’s footprint would place it among the top 30 countries by CO2e, larger than Argentina.Source: §4.8 · p. 13
  • Google provides researchers worldwide free access to ~500 PetaFLOPS/second of TPU compute, while noting that cheaper training could increase energy use if more people train models.The passage presents that possible rebound as preferable to continued use of inefficient models.Source: §4.8 · p. 13

4.9 Are the benefits of NLP models worth the energy cost?

The section pairs NLP benefits—from supporting COVID-19 research to improving multilingual search and translation—with the energy and emissions incurred in training.

4.9 Are the benefits of NLP models worth the energy cost?

  • BERT significantly improved featured snippets in Google search results in Hindi, Korean, and Portuguese.The section describes this use as applying models that learn from improvements in English to other languages.Source: §4.9 · p. 14
  • +13.5 BLEU: GShard’s 600B multilingual MoE improves average translation quality by 7.4 BLEU over the 2.3B dense model [Lep20].Figure 7 shows gains across 100 languages, larger for low-resource languages. The sparse MoE needs ~10X fewer training compute resources than a dense DNN despite its higher translation quality; training used 24 MWh and emitted 4.3 tCO2e, with amortized per-user emissions below those of one text message for a model that could have 2B users.Source: Figure 7 · p. 14, §4.9 · p. 15
  • At launch, the COVID-19 Research Explorer served 1,000 queries per day; training it used ~2.8 MWh and emitted 0.13 tCO2e.It was trained for six days on 64 TPU v3s in Oklahoma and was used daily by Pfizer’s Director of Global Medical Epidemiology; the tool also drew interest from pharmaceutical, medical, and public-health organizations.Source: §4.9 · p. 14

5. Conclusion

The ML community can help curb the carbon footprint of computationally intensive models by making energy and emissions visible and rewarding efficiency alongside accuracy. These incentives could spur improvements across algorithms, systems, hardware, datacenters, and carbon-free energy.

5. Conclusion

  • The paper recommends reporting energy use and CO2e, rewarding efficiency improvements at ML conferences, and disclosing training time and processor counts for compute-intensive models.These disclosures would help readers understand training costs.Source: §5 · p. 15
  • Power in upcoming MLPerf benchmarks is described as an important step toward evaluating energy alongside traditional model metrics.The authors believe power will be included in upcoming MLPerf benchmarks.Source: §5 · p. 15
  • Competing on training quality and carbon footprint, not accuracy alone, could direct ML demand toward the most efficient datacenters and hardware.Publication incentives for emissions improvements alongside accuracy could reinforce this shift.Source: §5 · p. 15
  • A virtuous cycle of emissions-focused incentives and competition could slow ML’s carbon-footprint growth by accelerating efficiency and carbon-free-energy innovations.The proposed innovations span algorithms, systems, hardware, and datacenters.Source: §5 · p. 15
  • Gshard-600B operates more efficiently than other large NLP models, while ML accelerators are more efficient than off-the-shelf hardware.The passage frames efficiency improvements in algorithms, datacenters, hardware, and software as a longstanding business priority for Google and other Cloud companies.Source: §5 · p. 15

Appendix A. Details of CO 2 Estimates for Four Large NLP Models in Tables 1 and 4

Appendix A explains how the authors derived the energy and CO2e inputs and contextual comparisons reported in Tables 1 and 4. It combines datacenter-specific measurements and published information, with some GPT-3 values estimated from reported FLOPs and GPU performance.

Datacenter inputs

  • Google datacenter carbon intensity, PUE, and carbon-free energy inputs reflect conditions when the models ran, while Microsoft uses the 2020 US national average and GPT-3 uses a reported PUE of 1.10.Google net CO2e/KWh uses its 24/7 carbon-free energy methodology; 2020 datacenter measurements were provisional pending final validation in May 2021.Source: Appendix A · p. 18

Power and computation inputs

  • Google system power was measured rather than inferred from chip TDP, and performance was profiled with Xprof; total computation was derived from performance, chip count, and training time except for GPT-3.The Google system-power measurement includes memory, fans, the CPU host, and network interface; measured FLOPs/s equals computed operations divided by execution time.Source: Appendix A · p. 18

GPT-3 estimates

  • For GPT-3, the authors used 10,000 V100 GPUs, 330W per GPU, 3.14E+23 training FLOPs, and 24.6 TeraFLOPS/sec per V100 to estimate about 14.8 days of training [Bro20].The paper notes that CO2e is unchanged whether the work takes about two weeks on 10,000 GPUs or 20 weeks on 1,000 GPUs; 10,000 GPUs was used for Table 4 based on NVIDIA’s suggestion.Source: Appendix A · p. 18

Contextual comparisons

  • The appendix calculates each model’s energy use as a percentage of Google’s total 2019 energy consumption, 12.2 terawatt-hours, including models not run in Google datacenters or in 2019.Source: Appendix A · p. 18

Appendix B. Carbon Offset and 24/7 Carbon Free Energy

Conventional carbon accounting can match clean-energy purchases across locations and over an annual period, while Google’s 2030 24/7 CFE goal requires matching consumption to clean energy on the same local grid in each hour. The appendix describes Google’s current contracts and hourly accounting, and how regional clean-energy availability can affect compute planning.

Appendix B. Carbon Offset and 24/7 Carbon Free Energy

  • Google’s 24/7 CFE goal restricts purchases to the same local grid and requires hourly rather than annual accounting.Figure 8 contrasts hourly accounting in one location with annual accounting across locations.Source: Appendix B · p. 19, Figure 8 · p. 20
  • Conventional Scope 2 accounting can match 1 MWh of consumption in one place and month with 1 MWh of CFE elsewhere and later; Google achieved carbon neutrality using offsets starting in 2007.Google became the first major company to match 100% of annual electricity use with renewable energy in 2017 and did so for three consecutive years.Source: Appendix B · p. 19
  • Google uses long-term renewable-energy PPAs to cover electricity consumption and plans to combine generation and storage to supply clean energy at all hours.Long-term contracts provide generators with reliable income that can make projects easier to finance.Source: Appendix B · p. 19
  • Hourly 24/7 CFE accounting counts shortfalls against the goal but gives no extra credit for surplus, and datacenter CFE is reported after data are settled.Google estimates regional CFE from grid conditions and multi-year contracts, and uses that estimate to incentivize programs to run in the region.Source: Appendix B · p. 20
  • 95.6% net CFE was Oklahoma’s average for 2020, while Google’s large models ran in Georgia, where gross and net CO2e had historically differed little.Google planned to deploy many new TPU v4 accelerators in windy Oklahoma in 2021.Source: Appendix B · p. 20
  • Google shifted non-urgent compute tasks such as ML training in 2020 to times when carbon-free power was more plentiful, with a later system planned to move tasks between datacenters.Regions with 24/7 clean-energy supply and flexible purchasing options are more compelling for future compute growth from a carbon-impact perspective.Source: Appendix B · p. 20

Appendix C. Details of a CO 2 e Estimate for NAS in an Average Datacenter

The appendix revises [Str19]’s estimate of the Evolved Transformer NAS footprint, finding that a smaller proxy configuration reduced the estimated emissions substantially. It also explains why retrospective estimates are difficult and recommends reporting energy use and CO2e explicitly.

Appendix C. Details of a CO 2 e Estimate for NAS in an Average Datacenter

  • [So19]’s search evaluated 15K child models over 979M training steps; after hurdles, the top 20 models were trained for 300K steps each on one TPU v2 chip.More than 13K models failed the first hurdle, avoiding the 3.6B training steps estimated without hurdles for the same number of models.Source: Appendix C · p. 21
  • Because the information needed for an accurate estimate was scattered across [So19], the authors recommend that conferences encourage computationally expensive NLP papers to report energy use and CO2e.They note that missing details make retrospective estimates difficult.Source: Appendix C · p. 21
  • The overestimate arose because [Str19] assumed Transformer Big with batch size 32,768, while [So19] used Transformer Base at batch size 4,096, lowering cost per step by 2.3x and a further 8x.The calculation defines emissions per chip-hour as average chip power multiplied by PUE and pounds of CO2e per watt.Source: Appendix C · p. 21
  • 33,544 lbs (15.2 t CO2e) is the revised estimate for the NAS, versus 626,155 lbs (284 t CO2e) in [Str19], an overestimate of ~18.7x.The revised calculation uses one P100 and Transformer Base training speed; Appendix D further adjusts for the actual hardware and datacenter.Source: Appendix C · p. 21

Appendix D. Details of a CO 2 e Estimate for Google’s Actual NAS

Correcting the datacenter, carbon intensity, and hardware assumptions substantially reduces the estimated emissions of Google’s actual NAS search and shows why training energy and CO2e should be reported directly.

Appendix D. Details of a CO 2 e Estimate for Google’s Actual NAS

  • 3.2 tCO2e (7,096 lbs) is the estimate for So et al.’s [So19] Google NAS search, 88X smaller than Strubell et al.’s [Str19] estimate.The calculation uses Google Georgia’s PUE of 1.10, average emissions of 0.431 CO2e/KWh, and TPU v2 hardware.Source: Appendix D · p. 22
  • 0.6 tCO2e, 476X smaller than the Georgia estimate, is the projected result of rerunning the search on TPU v2s in Google’s Iowa datacenter with 24/7 local, real-time net CO2e reduction.Newer TPUs would reduce emissions further.Source: Appendix D · p. 22
  • 3.3X is the TPU v2 NAS performance-per-Watt improvement over P100, with the search taking 32,633 TPU v2 hours versus 117,780 P100 hours.Measured TPU v2 power was 208 W including memory, fans, network interfaces, and CPU host; Strubell et al. [Str19] estimated 189 W per P100, while the paper reports 296 W for P100 Transformer (Big) in Table 4 and notes differences in systems and implementations.Source: Appendix D · p. 22
  • ~3 passengers taking a San Francisco–New York round trip is the flight-emissions equivalent of 3.2 net tCO2e from the NAS search.The comparison uses 1.2 tCO2e per passenger round trip, based on Google Flights’ historical average occupancy.Source: Appendix D · p. 22

Related papers

Loading 2104.10350v3…