Source-linked AI summary
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, Joelle Pineau
TL;DR
Machine learning research lacks systematic energy and carbon reporting despite growing computational demands. The paper introduces experiment-impact-tracker to simplify measurement and reporting, then uses it for efficiency analysis and mitigation strategies, including a leaderboard and location-based emissions comparisons.
Problem
Machine learning papers rarely report energy or carbon metrics, limiting systematic estimates of ML’s broader climate impacts.
Method
The paper presents experiment-impact-tracker, a lightweight framework that tracks energy, compute, and carbon impacts and generates standardized online appendices.
Results
Up to 30x lower CO2eq emissions were observed by running experiments in locations powered by more renewable energy sources.
Takeaways & Limitations
Energy leaderboards, efficient configurations and environments, carbon-aware locations, reproducibility, and consistent reporting are proposed to support more energy-efficient ML research.
Takeaways & Limitations
Cloud-machine carbon intensities may not reflect regional intensities, and customized cloud hardware may lack energy-accounting interfaces.
Abstract
from arXiv · showhide
Accurate reporting of energy and carbon usage is essential for understanding the potential climate impacts of machine learning research. We introduce a framework that makes this easier by providing a simple interface for tracking realtime energy consumption and carbon emissions, as well as generating standardized online appendices. Utilizing this framework, we create a leaderboard for energy efficient reinforcement learning algorithms to incentivize responsible research in this area as an example for other areas of machine learning. Finally, based on case studies using our framework, we propose strategies for mitigation of carbon emissions and reduction of energy consumption. By making accounting easier, we hope to further the sustainable development of machine learning experiments and spur more research into energy efficient algorithms.
1. Introduction
Machine learning’s growing compute demands create climate-impact concerns, while energy and carbon metrics remain underreported. The paper introduces a lightweight framework for systematic accounting and proposes reporting, mitigation, and incentive strategies.
- Machine learning’s increasing compute and energy demands may significantly contribute to carbon emissions.
- Systematic energy and carbon measurements are needed, yet most machine learning papers do not regularly report these metrics.
- The experiment-impact-tracker provides a lightweight interface for consistent, easier, and more accurate reporting of energy, compute, and carbon impacts.
- Floating point operations are often uncorrelated with energy consumption in image-classification experiments.
- CO2eq emissions can be reduced by up to 30x by running experiments in locations powered by more renewable energy sources.
- The proposed strategies include energy leaderboards, carbon-friendly regions, efficient resources and environments, reproducibility, and consistent reporting.
2. Related Work
Prior work estimates or models the environmental impacts of computation, including machine learning, but application heterogeneity and incomplete reporting limit aggregate assessment. This paper extends that work with easier accounting and additional mitigation strategies focused on ML compute.
- Prior studies examine carbon impacts across global, national, industrial, and application-specific scales.
- Estimating aggregate climate impacts of machine learning requires many assumptions because reporting and accounting are currently incomplete.
- Studies of machine learning propose reporting training time, hyperparameter sensitivity, equitable computation access, and computationally efficient hardware and algorithms.
- Existing tools estimate emissions from GPU type, experiment length, cloud provider, or computational metrics, but their methods may be inaccurate for combined CPU and GPU workloads.
- The paper focuses mainly on accurate accounting of machine learning compute rather than conference-travel emissions.
3. Background
The background defines energy and carbon accounting for ML experiments and reviews commonly reported computational metrics. It shows that indirect estimates depend on simplifying assumptions and can be substantially inaccurate.
- 3.1 Energy Accounting: Energy consumption is measured in Joules or Watt-hours, while life-cycle manufacturing energy is largely excluded because per-experiment attribution is difficult.
- 3.2 Carbon Accounting: Carbon accounting here measures emissions associated with ML projects while ignoring carbon offsetting and financial carbon accounting.
- 3.2 Carbon Accounting: The framework focuses on carbon emissions from energy consumption, calculated using the energy grid’s grams of CO2eq emitted per kWh.
- 3.3 Current State of Reporting in Machine Learning Research: Reported ML metrics include energy, power, PFLOPs-hr, FPOs, parameters, utilization, processor-hours, inference time, training time, and hardware-time combinations.
- 3.3 Current State of Reporting in Machine Learning Research: Among 100 sampled NeurIPS 2019 papers, 1 measured energy, 45 measured runtime, 46 reported hardware, 17 reported computational complexity, and 0 reported carbon metrics.
- 3.3 Current State of Reporting in Machine Learning Research: Estimating energy by multiplying runtime and GPU TDP assumes 100% GPU utilization and omits CPU usage, while the paper reports significant inaccuracies from such assumptions.
4. A New Framework for Tracking Machine Learning Impacts
The experiment-impact-tracker provides an easy-to-deploy framework for reporting carbon impact summaries and detailed energy, carbon, and compute metrics. It automates metric collection, standardized appendix generation, and reproducibility-oriented logging while supporting extensibility and fault tolerance.
- The framework provides carbon impact summaries and appendices containing detailed energy, carbon, and compute metrics.
- The design emphasizes usability, interpretability, extensibility, reproducibility, and fault tolerance to support systematic reporting.Modular metric support, version logging, raw-information retention, and hardware and package logging enable extension, recreation, and comparison of accounting results.
- A simple code change launches background monitoring that gathers compatible system metrics and logs them alongside the machine-learning process.Collected metrics include software versions, hardware information, timing, grid region, carbon intensity, power draw, and per-process CPU/GPU utilization.
- Generated HTML pages present graphs and tables for the collected metrics and can aggregate results across experiments with averages and standard error.
- The framework abstracts vendor-specific energy tools, using Intel tools for CPU/DRAM power, nvidia-smi for GPU power and utilization, and psutil for CPU utilization.It also attributes resource use to experiment processes on shared machines, excluding unrelated background processes.
- California experiments run during the day use roughly 2/3 of the energy of night-time experiments as solar generation changes realtime carbon conditions.
5. The Importance and Challenges of Accounting: Why a New Framework?
The paper examines why common partial proxies and estimation methods can misrepresent machine-learning energy and carbon impacts. Its case studies show that detailed accounting is needed because relationships and estimates vary across architectures, workloads, and regions.
- FPOs are not strongly correlated with energy or runtime across different neural-network architectures, but are almost perfectly correlated within one architecture.Across architectures, energy correlation is R2 = 0.083 and Pearson 0.289; within VGG versions, energy correlation is R2 = .999 and Pearson 1.0.
- 50,000 inference rounds on one sampled image measured kWh, experiment time, FPOs, and parameters across four random seeds.
- Partial estimation methods can differ significantly from tracking all data in real time.GPU-only estimates omit memory and CPU effects, while average-region assumptions omit regional energy-grid differences.
- Without detailed accounting, extrapolating from partial information can severely over- or underestimate carbon and energy emissions.The comparison used carbon-emission and kWh estimates for a Pong PPO experiment.
6. Encouraging Efficiency and Mitigating Carbon Impacts: Immediate Mitigation Strategies
The section presents energy-aware reinforcement-learning comparisons and argues that both algorithm choice and execution region offer immediate ways to reduce energy and carbon impacts. It also recommends standardized reporting and leaderboards to make efficiency trade-offs more visible.
- Reporting and incentives: Standardized reporting can ease the burden of tracking energy, compute, and carbon impacts in ML experiments.The framework is intended to make reporting consistent, easy, and more accurate.
- Reporting and incentives: Energy-aware leaderboards can promote a balance between model performance and efficiency.The paper proposes using carbon and energy metrics to inform comparisons between computationally efficient and larger models.
- A Deep RL Energy Leaderboard: The Deep RL Energy Leaderboard compares PPO, A2C, A2C with VTraces, and DQN across PongNoFrameskip-v4 and BreakoutNoFrameskip-v4.Experiments train for 5M timesteps and evaluate average return against total energy in kWh.
- A Deep RL Energy Leaderboard: 888 kWh of energy would be saved in a 235-student class by using PPO instead of DQN for the described Pong assignment.The estimate assumes each student runs the algorithm five times and that performance is similar.
- Caveats and mitigation: The leaderboard rankings may change with different codebases and hyperparameters, and the paper focuses on energy rather than directly comparing carbon efficiency.Carbon comparisons are confounded by time of day and region; a fixed carbon-intensity factor is suggested for approximate regional comparisons.
- Running In Carbon-Friendly Regions: Moving training jobs to carbon-efficient energy grids can immediately reduce emissions, with Quebec estimated at 30x lower emissions than Estonia.The framework provides region-emissions information to support cloud-region decisions.
7. Discussion: Systemic Changes
The discussion links accurate accounting to systemic changes in ML research and deployment, while identifying practical mitigation strategies and tooling limitations.
- Systemic changes: Energy leaderboards can inform energy-efficient configurations, but making efficient settings framework defaults could spread those choices more broadly.The paper specifically discusses mixed-precision computing and integrating efficient components into frequently used frameworks.
- Mitigation strategies: Deployment decisions should weigh model performance gains against energy costs, including routing energy-hungry models only to data subsets where they improve performance.The paper also raises the broader trade-off between training costs and lifetime deployment costs.
- Reproducibility and reporting: Accurate accounting supports cost-benefit analyses, while releasing code and models can reduce energy spent on replication when release is socially and ethically responsible.The paper notes that safety considerations may justify withholding resources in some cases.
- Reporting and incentives: The paper recommends standardized energy and carbon reporting, Carbon Impact Statements, and tooling that exposes these metrics alongside compute information.It also proposes climate-friendly conference badges to recognize mitigation efforts without punishing compute-intensive experiments.
- Limitations: The framework does not eliminate measurement barriers: driver support, administrative access, unsupported container measurements, and unknown cloud-provider energy mixes remain constraints.Cloud carbon intensities may differ from regional intensities when providers purchase clean energy directly for data centers.
8. Concluding Remarks and Recommendations
The conclusion presents the framework and associated tools as aids for consistent accounting, then recaps recommendations for reducing ML energy and carbon impacts.
- Conclusions: The framework eases consistent reporting of energy, compute, and carbon metrics and is intended to support research on energy-efficient algorithms.The authors encourage contributions to expand the framework and highlight the Deep RL Energy Leaderboard.
- Researchers and companies: Report model-related energy metrics and make energy-efficient configurations more accessible through reported results.The paper recommends consistent reporting regardless of which accounting tool is used.
- Researchers and companies: Create energy leaderboards, work on energy-efficient systems, and release code and models when safe to do so.For production systems, the paper also recommends internal release to encourage reuse.
- Researchers and companies: Integrate energy-efficient operations as defaults in frameworks and baseline implementations.The recommendation is framed as making efficient configurations more accessible across downstream projects.
- Researchers and companies: Run cloud jobs in low-carbon regions and make launch configurations and documentation point to those regions.The recommendations identify moving training jobs to low-carbon regions as an immediate action.
- Researchers and companies: Consider energy-based costs versus benefits when deploying new models and encourage climate-friendly initiatives at conferences.The authors present these recommendations as part of broader responsible ML research and practice.
Carbon Impact Statement
The Carbon Impact Statement quantifies the paper's electricity use, carbon contribution, and estimated social cost, while the surrounding passages compare ML impacts with conference travel.
- Carbon Impact Statement: 8.021 kg of CO2eq was attributed to this work, alongside 24.344 kWh of electricity use.The statement also reports a USA-specific social cost of carbon of $0.38, with a range of $0.00 to $0.95.
- Broader impacts: 801 kg CO2eq per conference participant is one reported estimate for conference travel emissions.Other cited estimates are around 883 kg and 910 kg CO2eq per participant.
- Broader impacts: The cited 2019 ML conference attendances include 13,500 participants for NeurIPS and 9,227 for CVPR.The passages also report 41% year-over-year growth for NeurIPS from 2018 to 2019.
Appendix B. NeurIPS Sampling on Metric Reporting
A sample of 100 NeurIPS 2019 papers shows that energy and carbon reporting was rare, while runtime, hardware, and computational-complexity metrics were more common. The appendix also discusses how regional electricity factors affect carbon accounting.
- 1 of 100 sampled NeurIPS 2019 papers measured energy, while 0 reported carbon metrics.
- 45 papers reported runtime, 46 provided hardware details, and 17 reported computational-complexity measures such as FPOs or parameters.
- The sampling procedure combined keyword checks with contextual classification and manual validation of related metrics.
- Carbon accounting depends on local grid composition, although renewable-energy certificates and power-purchase agreements can support additional renewable generation.
Appendix D. ImageNet Experiments
The ImageNet experiments compare pretrained image-classification architectures using measured computational, energy, and runtime quantities. Across architectures, parameter counts show little relationship with energy use or runtime.
- 50,000 inference rounds on a single image were run for pretrained models including AlexNet, DenseNet, GoogLeNet, MobileNetv2, and VGG.
- R2 = 0.002 and Pearson −0.048 describe the relationship between parameter count and energy consumption.
- R2 = 0.14 and Pearson −.373 describe the relationship between parameter count and runtime.
- The authors note that runtime results may differ from Canziani et al. because the model sets have different architectures.
Appendix F. Reinforcement Learning
The reinforcement-learning appendix evaluates four algorithms on Pong and Breakout under a common, energy-conscious experimental protocol. It reports return over training and cautions that the DQN comparison is limited by hyperparameter and training-budget choices.
- Four baseline algorithms—PPO, A2C, A2C with VTraces, and DQN—were evaluated on PongNoFrameskip-v4 and BreakoutNoFrameskip-v4.
- Training used 5M timesteps, fewer than prior work, to encourage energy efficiency.
- Evaluation used 25 separate rollouts every 250k timesteps to maintain parity between on-policy and off-policy methods.
- Average and asymptotic return were reported, alongside plots relating experiment runtime to asymptotic and average return.
- DQN underperformed PPO in this setting, but better hyperparameters or a longer training regime could change the comparison.
Appendix G. Possible Sources of Error, Limitations, and Overheads
The framework’s measurements have hardware, API, sampling, cloud-region, and accounting limitations. Its reported overhead was low in one CPU-only regression experiment, but carbon comparisons require controlled intensity assumptions.
- Hardware power APIs can contain chipset- and workload-dependent errors, and Nvidia documentation states readings are accurate within +/- 5 watts.
- Polling may misestimate short or highly erratic workloads because an unrepresentative power sample can determine the energy estimate.
- Cloud-region carbon estimates lack exact provider carbon intensities and PUEs and may also inherit errors from IP-based and manually gathered grid information.
- Runtime overhead was less than 1% in a 200-epoch CPU-only regression experiment and around .5% for 500 epochs.
- The framework supports only some driver and hardware combinations, with most metrics limited to Linux systems and Mac OS support restricted to specified Intel Power Gadget cases.
- The tool records CPU, GPU, and DRAM power but omits disk I/O energy, power-conversion overhead, and voltage-regulator overhead.
- Carbon-emissions comparisons require holding carbon intensity constant and using multiple runs on comparable machines.