Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training

Jie You; Jae-Won Chung; Mosharaf Chowdhury

Source-linked AI summary

Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training

Jie You, Jae-Won Chung, Mosharaf Chowdhury

arXiv:2208.06102v2Aug 12, 2022cs.LGcs.AIcs.DC

TL;DR

DNN training’s energy demand is substantial, while existing optimization largely emphasizes faster completion rather than energy efficiency. Zeus uses online configuration tuning and just-in-time profiling to navigate the energy–time tradeoff, reducing energy consumption across diverse workloads while adapting to changing workloads.

Problem
Existing DNN-training optimization largely ignores energy efficiency despite growing GPU energy demand and a tradeoff between training energy and time.
Method
Zeus automatically tunes batch size and GPU power limit using online exploration–exploitation and just-in-time energy profiling for recurring DNN training jobs.
Results
15.3%–75.8% lower energy consumption was achieved across diverse workloads compared with maximum batch size and maximum GPU power limit.
Takeaways & Limitations
Zeus enables users to navigate the energy–time Pareto frontier while adapting to workload changes such as data drift.
Takeaways & Limitations
Zeus currently supports only single-node training; distributed multi-GPU support is identified as future work.

Abstract

from arXiv · show

Training deep neural networks (DNNs) is becoming increasingly more resource- and energy-intensive every year. Unfortunately, existing works primarily focus on optimizing DNN training for faster completion, often without considering the impact on energy efficiency. In this paper, we observe that common practices to improve training performance can often lead to inefficient energy usage. More importantly, we demonstrate that there is a tradeoff between energy consumption and performance optimization. To this end, we propose Zeus, an optimization framework to navigate this tradeoff by automatically finding optimal job- and GPU-level configurations for recurring DNN training jobs. Zeus uses an online exploration-exploitation approach in conjunction with just-in-time energy profiling, averting the need for expensive offline measurements, while adapting to data drifts over time. Our evaluation shows that Zeus can improve the energy efficiency of DNN training by 15.3%-75.8% for diverse workloads.

1 Introduction

DNN training’s growing energy demand exposes a gap in performance-focused optimization. Zeus addresses the energy–time tradeoff through online, adaptive tuning of batch size and GPU power limit.

Motivation: DNN training increasingly relies on powerful GPUs, while existing training literature largely ignores energy efficiency.GPT-3 training consumed 1,287 MWh, illustrating the scale of the energy challenge.
Zeus: Zeus automatically tunes job- and GPU-level configurations to navigate the energy–training-time tradeoff for recurring DNN jobs.It uses online exploration and exploitation instead of per-job offline profiling or prediction-model training.
Zeus: Zeus handles stochastic training outcomes and heterogeneous models and GPUs through online optimization and just-in-time energy profiling.The batch-size problem is formulated as a Multi-Armed Bandit using Thompson Sampling, while the profiler finds the power limit during training.
Evaluation: 15.3%–75.8% lower energy consumption and 60.6% lower training time were achieved across speech, vision, NLP, and recommendation workloads.The comparison is against selecting the maximum batch size and maximum GPU power limit.
Evaluation: Zeus integrates with existing DNN workflows with little code change and negligible overhead while adapting to workload dynamics.The paper presents these properties as contributions of the implemented optimizer.

2 Motivation

Batch size and GPU power limit create substantial opportunities to reduce DNN training energy, but energy optimization can trade off against time-to-accuracy. Zeus’s motivating measurements expose this tradeoff and its Pareto frontier.

2.1 DNN Training: Energy usage is evaluated against a baseline using maximum power limit and a default or maximum consistently feasible batch size on NVIDIA V100.Figure 1 normalizes DNN-training energy against this baseline.
2.1 DNN Training: DNN training repeatedly processes datasets in epochs and measures performance using time-to-accuracy, while GPUs account for around 70% of training energy.Batch size affects model accuracy, throughput, and energy consumption.
2.1 DNN Training: Production pipelines retrain DNNs frequently, creating recurring jobs where configuration choices repeatedly affect energy and performance.Retraining intervals can be as short as every hour.
2.1 DNN Training: 3.4%–65.0% lower energy consumption is possible with the energy-optimal batch size than with the default choice at the same target accuracy.The result comes from sweeps across six diverse workloads on NVIDIA V100.
2.2 Opportunities for Improving Energy Efficiency: A lower-than-maximum GPU power limit can reduce energy consumption by 3.0%–31.5%, because the limit triggers dynamic voltage and frequency scaling.The sweep covered power limits from the minimum to maximum allowed for NVIDIA V100.
2.2 Opportunities for Improving Energy Efficiency: 23.8%–74.7% energy reduction is achievable by jointly optimizing batch size and GPU power limit.Similar opportunities were observed across other GPU generations.
2.3 Energy-Performance Tradeoffs: Energy optimization can affect time-to-accuracy, motivating characterization of the energy-to-accuracy versus time-to-accuracy tradeoff.Energy-to-accuracy is defined as energy consumed until the target accuracy is reached.
2.3 Energy-Performance Tradeoffs: The ETA optimum can differ from the TTA optimum: (b =32, p =100W) versus (b =48, p =250W).This demonstrates that optimizing energy and training time selects different configurations.

3 Zeus Overview

Zeus navigates the energy–time tradeoff in recurring DNN training by configuring batch size and GPU power limit through an online feedback loop. Its cost metric lets users express the relative importance of energy efficiency and training performance.

Framework scope: The framework automatically configures batch size and GPU power limit for recurring DNN jobs without requiring per-job offline profiling or prediction-model training.Zeus jointly considers job- and GPU-related configurations.
Optimization metric: Zeus minimizes an energy-time cost that combines average GPU power and time to accuracy, weighted by user parameter η.η = 0 emphasizes time consumption, while η = 1 emphasizes energy consumption.
Online optimization: Zeus uses a just-in-time profiler and Multi-Armed Bandit with Thompson sampling to optimize under stochastic and changing workloads.The profiler measures energy characteristics online, while the bandit adapts to uncertainty and data drift.
Workflow: Zeus predicts a configuration from execution history, launches the job, collects training and power statistics, and feeds the results back into the optimizer.Jobs terminate when they reach the target metric or exceed a Zeus-determined stopping threshold.

4 Zeus Algorithm Design

Zeus reduces a large, uncertain configuration search to online optimization of batch size and power limit. It profiles power limits during training and uses Thompson Sampling to balance exploration with exploitation across recurring jobs.

Problem formulation: Zeus minimizes cumulative recurring-job cost while exploring feasible batch sizes B and power limits P.This objective captures the exploration–exploitation tradeoff.
Problem formulation: Decoupling batch size and power-limit optimization makes the search more tractable without losing optimality.For a given batch size, power and throughput can be profiled quickly, while epochs to target are unaffected by power limit.
Power-limit optimization: Zeus profiles all feasible power limits during the first epoch of an unseen batch size and then selects the power limit with minimum cost.The JIT profiler partitions the epoch into iteration-boundary slices and changes the GPU power limit between slices.
Batch-size optimization: Thompson Sampling supports exploration without exhaustive grid search by quickly reducing attention to batch sizes with clearly high observed costs.This addresses stochastic training costs while limiting exploration overhead.
Batch-size optimization: Zeus models batch-size selection as a Multi-Armed Bandit whose arms are batch sizes and whose stochastic costs are updated over job recurrences.Thompson Sampling samples estimated mean costs and selects the batch size with the smallest sample.
Batch-size optimization: Early stopping prunes expensive batch-size trials when their cost exceeds β times the best observed cost, with default β = 2.The threshold accommodates run-to-run variation in training time to accuracy.

15 end

Zeus extends its online optimizer to pruning, concurrent job submissions, and data drift. These mechanisms reduce wasted exploration and preserve adaptation when training conditions change.

Batch-size pruning: Pruning explores smaller and larger batch sizes around a default, retaining only sizes that reach the target metric.The default is then updated to the lowest-cost observed size and pruning is repeated.
Batch-size pruning: The pruning strategy exploits the observed convexity of the batch-size ETA curve to rule out noticeably suboptimal sizes.Very large sizes may increase epochs or lose accuracy, while very small sizes may produce noisy gradients.
Concurrent submissions: Thompson Sampling naturally mitigates duplicate exploration when recurring jobs overlap because its randomized predictions can select different batch sizes without new observations.This is especially useful early, when belief variances are large.
Concurrent submissions: During initial pruning, concurrent submissions use the best-known batch size, distributing extra exploration opportunities toward sizes already showing convergence.The best-known choice is updated throughout exploration.
Data drift: Data drift makes each batch-size cost distribution non-stationary, motivating Zeus’s use of recent observations for adaptation.Windowing discards older observations and estimates recent cost variance directly.

5 Zeus Implementation

Zeus is implemented as a Python library integrated with PyTorch through ZeusDataLoader. It profiles power and throughput online, determines power limits automatically, and offers an observer mode for estimating potential benefits without changing execution.

Integration: ZeusDataLoader integrates with PyTorch and configures or profiles GPU power limits using NVML.It profiles by slicing epochs at iteration boundaries.
Online profiling: Five seconds of profiling for each power limit has been observed to produce stable results.The profiler uses online measurements of power and throughput to determine the optimal limit.
Training workflow: The training-loop integration supports epoch iteration, batch processing, evaluation, and reporting a validation metric.The reported metric is used alongside the target condition in the training workflow.
Observer Mode: Observer Mode profiles power limits and reports estimated time and energy under the optimal limit while keeping execution at maximum power.This avoids affecting the job’s time or energy consumption during observation.

6 Evaluation

Zeus is evaluated across diverse workloads and hardware, reducing energy while navigating the energy-time tradeoff. It converges quickly, adapts to data drift, incurs low profiling overhead, and extends to multi-GPU settings.

6.2 Zeus Performance: Up to 60.1% lower training time is achieved for some workloads, while others incur a 12.8% increase because of the energy-time tradeoff.Workloads whose default batch size already minimizes training time have little room for further TTA improvement.
6.2 Zeus Performance: Zeus reaches the cumulative-regret plateau earlier than Grid Search, with Grid Search producing 72× more cumulative regret in the worst case until convergence.Zeus uses significantly fewer resources to converge and learns configurations online without prior application information.
6.3 Trace-Driven Simulation Using the Alibaba Trace: 7%–52% lower training energy is observed in trace-driven simulations, while training time increases by at most 16% and decreases by up to 33% in many cases.Zeus’s early stopping and online power optimization reduce exploration costs compared with the baselines.
6.4 Data Drift: Spikes in ETA and TTA trigger exploration of a different batch size as workload behavior drifts across successive data slices.The BERT sentiment-analysis experiment uses a sliding-window workload with a window size of 10.
6.5 Overhead: 0.01% higher energy and 0.03% higher time are measured for JIT profiling on DeepSpeech2, demonstrating negligible overhead for long training epochs.The profiling time for all power limits is less than one minute, while one epoch typically spans hours.
6.6 Scaling to Multi-GPU: 21% less energy is used with Zeus than Pollux on four NVIDIA A40 GPUs for DeepSpeech2, although Zeus takes 12% more time.Zeus also lets users select different energy-time tradeoff points through η.
6.7 GPU Models and Early Stopping: Zeus achieves consistent energy reductions across four NVIDIA GPU generations, and β = 2.0 gives the lowest geometric mean across jobs.Lower β can prematurely stop exploration, whereas higher β increases exploration cost by weakening early stopping.

7 Discussion

Zeus selects batch size and GPU power limit to balance control granularity against search cost, while extending its formulation toward constrained training settings.

Configuration knobs: Zeus optimizes batch size and GPU power limit because together they balance control granularity with search-space size.Finer-grained frequency and voltage controls could improve efficiency but would prolong exploration.
Configuration knobs: Batch size affects end-to-end energy through both training time and average power consumption, unlike learning rate, which affects only training time.
Hyperparameter optimization: If users fix the feasible batch-size set to one value, Zeus can still reduce energy consumption by searching for the optimal GPU power limit.
Supporting distributed training: Zeus currently supports single-node training, but distributed extensions can apply one power limit across identical GPUs and sum time and energy across devices.
Supporting heterogeneous GPUs: Zeus assumes each recurring training job uses the same GPU type, although heterogeneous-GPU support can translate cost values using GPU-specific power and throughput profiles.

8 Related Work

Related work improves DNN training speed, measures or models energy, and studies energy optimization, while Zeus jointly optimizes job and GPU configurations online for training energy-to-accuracy.

DNN training: Most DNN training studies optimize time-to-accuracy through kernels, placement, or communication while overlooking energy impact.
DNN training: Pollux dynamically changes batch size for multi-GPU training using Gradient Noise Scale, whereas Zeus trades off time-to-accuracy and energy-to-accuracy by tuning batch size.
Energy measurement for Deep Learning: Energy research covers DNN training consumption, environmental impact, device benchmarking, and software-based measurement or prediction.
Energy optimization for Deep Learning: Prior energy-optimization work addresses inference, training strategies, multi-GPU scheduling, and task mapping; Zeus complements these solutions transparently.
Energy optimization for Deep Learning: GPU DVFS and power-configuration studies often rely on offline modeling and profiling, while Zeus uses online optimization of end-to-end energy-to-accuracy.
Energy optimization for Deep Learning: BatchSizer optimizes inference energy with batch size, whereas Zeus jointly optimizes GPU and job configurations for DNN training.

9 Conclusion

Zeus frames DNN training energy and time as a tradeoff, then uses online tuning to find Pareto-efficient configurations across workloads and GPU generations.

9 Conclusion: Zeus identifies an energy–training-time tradeoff and tunes batch size and GPU power limit for recurring DNN jobs.It continuously adapts to dynamic workload changes such as data drift.
9 Conclusion: Four NVIDIA GPU generations show sufficient potential for energy savings, motivating Zeus.
9 Conclusion: Figure 16 presents Pareto fronts for six workloads against a default configuration using an NVIDIA V100 GPU.
9 Conclusion: Sweeps of batch size and power limit produce ETA curves whose batch-size convexity supports Zeus’s pruning exploration algorithm.
9 Conclusion: Zeus generally converges to a better configuration than Grid Search while exploring faster.
9 Conclusion: Decoupling batch size and power limit lets Zeus navigate the two-dimensional configuration space with less exploration energy and time than Grid Search.

F Additional Sensitivity Analysis

Zeus’s sensitivity analysis shows that the objective weight controls whether optimization prioritizes energy consumption or training time.

F Additional Sensitivity Analysis: Higher η prioritizes reducing energy consumption over training time, while lower η shifts priority toward time reduction.The resulting energy improvement factor rises with higher η and falls when time receives greater priority.

G Performance of Zeus on All GPUs

Figure 23 compares energy consumption and training time across workloads and four NVIDIA GPU generations, normalized against Default. The accompanying figures examine Zeus’s regret, search path, and the impact of priority knob η on ETA and TTA.

Optimization behavior: Figures 19–22 compare Zeus with Grid Search using cumulative regret and search paths across all workloads.The figures also show the impact of priority knob η on ETA and TTA.
Energy and time consumption: The evaluation covers four NVIDIA GPU generations: Ampere A40, Volta V100, Turing RTX6000, and Pascal P100.The overall trends hold across all four GPUs.
Optimization behavior: The plotted regret visualizations encode lower regret with darker shading.The supplied figure labels identify regret of configuration as the plotted quantity.
Energy and time consumption: Figure 23 reports energy consumption and training time normalized against Default for DNN training workloads.The figure includes energy and time measurements across multiple GPU configurations.