Source-linked AI summary
Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
Lasse F. Wolff Anthony, Benjamin Kanding, Raghavendra Selvan
TL;DR
DL’s rapidly increasing computational demand raises energy use and potential climate impact, while practitioners need visibility into training footprints. The paper introduces Carbontracker to track and predict energy and emissions, evaluates its predictions, and advocates reporting footprints alongside performance metrics. The tool enables practitioners to identify their footprint and take steps such as shifting training to lower-carbon periods.
Problem
Rapidly increasing DL compute requires extensive training on specialized accelerators, raising energy demand and potential climate impact.
Method
Carbontracker tracks and predicts total training duration, energy consumption, and carbon emissions using monitored epochs and carbon-intensity forecasts.
Results
Across evaluated models and datasets, one-epoch predictions had energy errors of 4.9–19.1%, CO2eq errors of 7.3–19.9%, and duration errors of 0.8–4.6%.
Takeaways & Limitations
Reporting training energy and carbon footprints alongside performance metrics can support awareness, responsible computing, and research into energy-efficient deep neural networks.
Abstract
from arXiv · showhide
Deep learning (DL) can achieve impressive results across a wide variety of tasks, but this often comes at the cost of training models for extensive periods on specialized hardware accelerators. This energy-intensive workload has seen immense growth in recent years. Machine learning (ML) may become a significant contributor to climate change if this exponential trend continues. If practitioners are aware of their energy and carbon footprint, then they may actively take steps to reduce it whenever possible. In this work, we present Carbontracker, a tool for tracking and predicting the energy and carbon footprint of training DL models. We propose that energy and carbon footprint of model development and training is reported alongside performance metrics using tools like Carbontracker. We hope this will promote responsible computing in ML and encourage research into energy-efficient deep neural networks.
1. Introduction
Deep learning’s rapid growth has sharply increased computational demand, energy use, and potential climate impact. Carbontracker addresses this by tracking and predicting training footprints while supporting intervention and responsible reporting.
- DL compute grew 300000-fold from 2012 to 2018, increasing reliance on specialized hardware accelerators.
- Continued exponential growth in DL compute could make ML a significant contributor to climate change.
- Carbontracker tracks and predicts the energy consumption and carbon emissions of training DL models.
- The tool supports proactive intervention by allowing users to stop training when predicted environmental cost exceeds a chosen threshold.
- Carbontracker supports clusters, desktop computers, and Google Colab notebooks for plug-and-play use.
- Experiments evaluate prediction accuracy across CNN architectures and medical image-segmentation datasets, alongside recommendations for reducing emissions.
2. Design and Implementation
Carbontracker is designed as an interpretable, flexible Python tool that minimally integrates with existing ML workflows. It predicts training duration, energy, and carbon footprint from monitored epochs and carbon-intensity forecasts.
- Carbontracker follows Pythonic, usable, extensible, flexible, performance-conscious, and interpretable design principles.
- The tool promotes interpretability by converting carbon footprints expressed in gCO2eq into more familiar measures.
- Carbontracker is an open-source Python package distributed through PyPI and implemented as a multithreaded program.
- Separate threads collect power measurements and fetch real-time carbon intensity without disrupting model training in the main thread.
- The tool predicts total training duration, energy, and carbon footprint using a user-specified number of monitored epochs, defaulting to one.
- Carbon-intensity forecasts from supported APIs are combined with predicted training duration to estimate carbon footprint.
- A simple linear model is used for predictions following preliminary research.
3. Experiments and Results
Experiments assess Carbontracker on medical image-segmentation models and datasets, showing generally close predictions but sensitivity to carbon-intensity estimation. Accounting for CPU and DRAM is also necessary for complete footprint estimates.
- Experiments use U-net and lungVAE models on the DRIVE, LIDC, and CXR medical image datasets.
- 37.445 kWh of electricity and 3.166 kg of CO2eq were estimated for training the models, equivalent to 26.296 km travelled by car.
- After monitoring one epoch, energy prediction errors were 4.9–19.1%, CO2eq errors 7.3–19.9%, and duration errors 0.8–4.6%.
- Figure 1 compares predicted with measured full-session energy, emissions, and duration, using the diagonal as the equality reference.
- Figure 2 compares carbon-intensity estimation methods for U-net training on LIDC in Denmark and Great Britain.
- Country- or region-wide average intensities may severely overestimate or underestimate emissions, making real-time or forecasted intensity important for accuracy.
- GPU use accounts for around 50–60% of total energy, while CPU and DRAM also contribute substantially.
- Counting only GPU consumption considerably underestimates the total energy and carbon footprint of DL models.
4. Reducing Your Carbon Footprint
Carbon emissions from DL training can be reduced through regional and temporal carbon-intensity choices, efficient hardware and settings, and more efficient algorithms. Carbontracker highlights substantial variation in emissions across locations and training times, while energy accounting should include CPU and DRAM as well as GPUs.
- Energy Accounting: GPU consumption represents around 50–60% of training energy, while CPU and DRAM also account for significant shares across models and datasets.Accounting only for GPU consumption therefore omits substantial energy use from other components.
- Low Carbon Intensity Regions: Training in Estonia may emit more than 61 times the CO2eq of equivalent training in Sweden.For the U-net model on LIDC, emissions were 17.7 gCO2eq in Sweden versus 1087.9 gCO2eq in Estonia.
- Low Carbon Intensity Regions: Large-scale, long-running multi-GPU training can produce several megagrams of carbon emissions, making low-carbon-intensity regions especially important.The absolute difference can remain significant even between relatively green regions such as Sweden and France.
- Training Times: Training during low-carbon-intensity hours may reduce emissions to one quarter of peak-hour emissions in Denmark and by 2-fold in Great Britain.Carbon intensity varies with energy demand, capacity, and available energy sources, so the appropriate training period is region-specific.
- Efficient Hardware and Algorithms: More efficient hardware, power-management settings, and algorithms can reduce the compute resources and carbon emissions required for DL training.Examples include GPUs with higher FLOPS per watt, DVFS, random or Bayesian search instead of grid search, and Hyperband.
5. Discussion and Conclusion
DL’s rapidly increasing compute demand raises environmental concerns, while energy efficiency is rarely reported alongside standard performance metrics. The paper introduces Carbontracker to track and predict training energy and emissions, supporting awareness and action to reduce ML’s footprint.
- Discussion: GPT-3 required an estimated 28000 GPU-days to train, excluding research and development, illustrating DL’s rapidly increasing compute demand.The paper presents this trend as a potential environmental concern if it continues.
- Discussion: The paper proposes reporting total energy and carbon footprint alongside accuracy and similar performance metrics.This is intended to promote responsible computing and research into energy-efficient deep neural networks.
- Conclusion: Carbontracker is an open-source tool for tracking and predicting total energy consumption and carbon emissions during DL model training.The tool is intended to make practitioners aware of their training footprint.
A. Implementation details
Carbontracker is a multithreaded Python tool that tracks and predicts training energy and carbon emissions while logging measurements for reproducibility. Its estimates account for selected hardware components, data-center overhead, and training assumptions, but omit some infrastructure and lifecycle factors.
- Implementation: Carbontracker is implemented as a multithreaded Python program with separate threads for monitoring and carbon-intensity retrieval.The design aims to collect measurements in parallel without disrupting model training in the main thread.
- Assumptions: The estimates include static power during reserved-resource use but ignore life-cycle energy and carbon offsets or Renewable Energy Credits.The paper treats users as accountable for static consumption and bases its reported footprint solely on training energy.
- Power accounting: The tool monitors selected components including GPUs, CPUs, and DRAM, while multiplying measured power by data-center PUE for supporting infrastructure.The authors use PUE to account for energy overhead from cooling, power delivery, storage, and networking.
- Limitations: Power monitoring remains limited to a few components, and using an average PUE can produce inaccurate estimates in energy-efficient data centers.The paper cites Google’s fleetwide PUE of 1.10 in 2020 as an example of this boundary.
A.3. Power and Energy Tracking
Carbontracker continuously samples device power and records epoch durations, while separately retrieving carbon intensity in real time. It converts these measurements into energy and carbon estimates, logs training information, and is constrained by regional carbon-intensity data availability.
- Power and energy tracking: The carbontracker thread continuously samples instantaneous power for each available device and stores each epoch’s duration.After collecting samples for all devices, it sleeps for a fixed interval before sampling again.
- Carbon conversion: Carbon intensity is fetched every 900 s and used with energy consumption to estimate carbon emissions.The tool dynamically determines location and retrieves regional intensity data through external APIs.
- Supported devices: The tool supports NVIDIA GPUs through NVML and Intel CPUs and DRAM through Intel RAPL.These interfaces expose power measurements for the supported hardware components.
- Limitations: Carbon-intensity estimation is limited because no globally accurate, free, public real-time database exists; direct regional support covers Denmark and Great Britain.Other regions default to the 2017 EU-28 average.
- Logging: Carbontracker logs power, energy, epoch durations, monitored devices, early stopping, and predictions, and can aggregate multiple training sessions.The parser module converts logs into Python dictionaries and supports combined impact estimates.
B. Models and Data
The evaluation uses two CNN models across three medical-image segmentation datasets, with dataset-specific training configurations and runs.
- The experiments cover U-net and lungVAE models on DRIVE, LIDC, and CXR medical-image segmentation datasets.Measurements were taken across these model–dataset combinations in the evaluation.
- U-net DRIVE: U-net DRIVE uses 15 training images from the DRIVE retinal-vessel dataset, with 300 epochs, batch size 4, and learning rate 10^-3.The images are 768 by 584 pixels and JPEG compressed.
- U-net CXR: U-net CXR uses 528 training and 176 validation chest X-rays for 200 epochs with batch size 12 and learning rate 10^-4.The model uses no data augmentation and weight decay of 10^-5, with Adam optimization.
- U-net LIDC: U-net LIDC is trained on preprocessed LIDC-IDRI CT data from 1,018 scans for 100 epochs with batch size 64 and learning rate 10^-3.Training uses annotations from a single radiologist.
- lungVAE CXR: lungVAE CXR uses the same CXR dataset as U-net CXR, with 528 training and 176 validation images and early stopping for a maximum of 200 epochs.Two runs lasted 90 and 97 epochs, respectively, using batch size 12 and learning rate 10^-4.
C.1. Performance Impact of Carbontracker
Monitoring every training epoch with Carbontracker had a small average effect on epoch duration, while larger isolated fluctuations also appeared in baseline runs.
- 0.19% on DRIVE, 1.06% on LIDC, and −0.58% on CXR are the mean epoch-duration changes across two U-net runs.These values are relative increases compared with training without Carbontracker.
- Up to 5% on LIDC and 22% on DRIVE were observed for individual relative epoch-duration increases.The passage attributes these isolated increases more likely to stochastic epoch-duration variation than Carbontracker.
- Figure 7 summarizes monitoring overhead with box plots using Tukey whiskers and outliers based on 1.5 times IQR.The plotted quantity is relative epoch-duration increase versus a baseline without Carbontracker.
- The duration fluctuations are also present in baseline runs and therefore are not caused by Carbontracker.
Energy and Carbon Footprint of GPT-3
The GPT-3 estimate translates reported compute into single-GPU runtime and energy, then combines energy with regional carbon intensity to estimate emissions.
- 3.14·10^23 FPOs divided by 130·10^12 FLOPS yields 2,415,384,615.38 seconds, or 27,955.84 days, on one GPU.The calculation assumes NVIDIA V100S PCIe performance of 130 TFLOPS.
- About 310 GPUs running non-stop for 90 days is equivalent to the estimated GPT-3 compute time.
- Using V100 thermal design power and a PUE of 1.125, the analysis estimates the electricity consumed by training.The PUE assumption is the average reported for new Microsoft data centers in 2015.
- Figure 8 reports EU-28 countries’ 2016 average carbon intensity as emissions from public electricity production divided by gross electricity production.The data comes from the European Environment Agency.
- 449.06 gCO2eq/kWh is the assumed average USA carbon intensity in 2017 for converting estimated electricity use into emissions.The calculation multiplies this intensity by 188,701.92 kWh.