PaperlayerView exact revision on arXiv

Source-linked AI summary

The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study

Zhenheng Tang, Yuxin Wang, Qiang Wang, Xiaowen Chu

arXiv:1905.11012v1cs.PF

TL;DR

GPU-based DNN training consumes substantial energy, and the energy effects of GPU frequency settings are non-trivial. The paper empirically evaluates DVFS across GPU architectures and DNN configurations, finding that optimal core frequencies can save energy relative to defaults in both training and inference.

  • Problem

    GPU DNN training consumes substantial energy, and the effect of GPU DVFS on training energy is non-trivial because energy depends on power and running time.

    Source: §I · p. 1, A. Convolutional Neural Networks · p. 2

  • Method

    The study empirically evaluates DNN performance and energy under GPU DVFS settings across GPU architectures, DNN configurations, and convolution algorithms.

    Source: §I · p. 1

  • Results

    Compared with default settings, optimal core frequencies saved an average 23.1%, 14.5%, and 8.7% energy for DNN training on three GPUs, and 26.4%, 22.3%, and 19.6% for inference.

    Source: §I · p. 1

  • Takeaways & Limitations

    The authors conclude that GPU DVFS has potential for energy-efficient DNN processing schemes without significant performance degradation.

    Source: §V · p. 5

Abstract

from arXiv · show

Over the past years, great progress has been made in improving the computing power of general-purpose graphics processing units (GPGPUs), which facilitates the prosperity of deep neural networks (DNNs) in multiple fields like computer vision and natural language processing. A typical DNN training process repeatedly updates tens of millions of parameters, which not only requires huge computing resources but also consumes significant energy. In order to train DNNs in a more energy-efficient way, we empirically investigate the impact of GPU Dynamic Voltage and Frequency Scaling (DVFS) on the energy consumption and performance of deep learning. Our experiments cover a wide range of GPU architectures, DVFS settings, and DNN configurations. We observe that, compared to the default core frequency settings of three tested GPUs, the optimal core frequency can help conserve 8.7%$\sim$23.1% energy consumption for different DNN training cases. Regarding the inference, the benefits vary from 19.6%$\sim$26.4%. Our findings suggest that GPU DVFS has great potentials to help develop energy efficient DNN training/inference schemes.

I. INTRODUCTION

The paper examines GPU DVFS as a way to balance DNN performance with energy use, a trade-off made important by the electricity consumed by high-throughput GPU systems. It evaluates frequency settings and convolution approaches, reporting energy savings at optimal core frequencies.

I. INTRODUCTION

  • The study addresses the often-overlooked energy use of high-throughput GPU training, alongside the performance focus of prior work.Large-scale distributed systems used to accelerate complex DNN training also consume significant electricity.Source: §I · p. 1
  • GPU DVFS presents a non-trivial energy trade-off because raising core or memory frequency can consume more power without improving performance when that component is underused.Energy depends on both system power and running time.Source: §I · p. 1
  • Compared with default settings, optimal core frequencies saved an average 23.1%, 14.5%, and 8.7% energy for DNN training on three GPUs, and 26.4%, 22.3%, and 19.6% for inference.The reported energy savings also vary among GEMM, FFT, and Winograd convolution algorithms.Source: §I · p. 1

A. Convolutional Neural Networks

  • Convolution energy efficiency merits comparison across matrix-multiplication, Fourier-transform, and Winograd implementations, all implemented in cuDNN.These approaches respectively use matrix multiplication, point-wise multiplication in the Fourier domain, and transforms that reduce multiplications.Source: A. Convolutional Neural Networks · p. 2

B. GPU DVFS

The paper situates GPU DVFS within prior work on GPU energy scaling and identifies a need to examine its effects on deep learning. Its experiments compare GPU architectures, frequency settings, DNN structures, and convolution algorithms.

B. GPU DVFS

  • GPU computational capacity is also used for AutoML searches that can run across hundreds or thousands of GPUs for weeks.The passage gives an example of 800 GPUs used to search for an RNN for language modeling.Source: B. GPU DVFS · p. 2
  • GPU DVFS research has examined balancing GPU performance and energy efficiency, while prior work also applied DVFS-based conservation to high-performance-cluster scheduling.The paper notes related studies of DNN scalability and early investigations of GPU DNN energy characteristics.Source: B. GPU DVFS · p. 2

III. METHODOLOGY

  • The experiments are designed to compare GPU architecture, DVFS setting, DNN structure, and convolution algorithm.The paper presents this coverage as a way to explore DVFS effects on deep learning.Source: §III · p. 2

A. Hardware Setup

  • Three GPUs are tested across the frequency options listed in Table I, but the experiment controls frequency while the NVIDIA driver adjusts voltage automatically.The GPU vendor’s limited support prevents direct control of voltage in these experiments.Source: A. Hardware Setup · p. 2, Table I · p. 2

B. Network Setup

The experiments cover training and inference across four DNNs and three convolution algorithms. Performance, power, and energy are measured or derived to compare frequency settings.

B. Network Setup

  • The network setup tests AlexNet, VggNet-16, GoogleNet, and ResNet-50, with batch sizes chosen according to GPU memory availability.The study fixes GEMM, FFT, or Winograd convolution algorithms by revising Caffe.Source: B. Network Setup · p. 2, Table II · p. 3

C. Performance and Power Measurements

  • Training performance is measured in images per second, while inference performance uses forwarding time.Training iteration time is averaged over 120 repetitions, and performance is obtained by dividing by batch size.Source: C. Performance and Power Measurements · p. 2
  • Power is sampled through NVML during processing, and energy per image is represented by power divided by performance.A sampling thread runs at 2 ms intervals; measurements outside DNN execution are intercepted before averaging.Source: C. Performance and Power Measurements · p. 3
  • For GTX 2080Ti, the study uses geometric means across memory-frequency samples to analyze core scaling, and applies similar aggregation for memory scaling and convolution algorithms.Figure 1 presents the impact of different core-frequency settings on training performance and energy consumption.Source: C. Performance and Power Measurements · p. 3, Figure 1 · p. 3

IV. EXPERIMENTAL RESULTS

The results section highlights selected experimental findings, with complete experimental data provided in the appendix.

IV. EXPERIMENTAL RESULTS

  • The results discussion highlights selected findings; the complete experimental data are in the appendix.Source: §IV · p. 3

A. Impact of GPU DVFS on Performance and Energy Efficiency

The best core frequency depends on the GPU and workload: higher frequencies improve performance, while the most energy-efficient setting varies. Compared with default settings, optimal frequencies conserve energy in both training and inference, and DNNs differ in their performance and energy profiles.

A. Impact of GPU DVFS on Performance and Energy Efficiency

  • 17.4%∼38.2% performance improvement on GTX 2080Ti training follows increasing core frequency from 1350 MHz to 2050 MHz.The paper reports this range across different DNNs.Source: A. Impact of GPU DVFS on Performance… · p. 3
  • P100 and V100 reach their best energy efficiency at middle core frequencies, while GTX 2080Ti benefits more from higher frequency; Tesla GPU power rises sharply above 1,000 MHz.The paper attributes the difference to the Tesla GPUs’ increasing power consumption above 1,000 MHz, which GTX 2080Ti does not show.Source: A. Impact of GPU DVFS on Performance… · p. 3
  • Across four DNNs, AlexNet has the best performance and energy efficiency, while ResNet-50 has the lowest throughput and highest energy per image but the best convergence and classification accuracy.The comparison reflects different DNN characteristics and convolution-layer counts.Source: A. Impact of GPU DVFS on Performance… · pp. 3–4
  • 23.1% (P100), 14.5% (V100), and 8.7% (GTX 2080Ti) average training energy conservation; inference benefits are 26.4%, 22,3%, and 19.6%, respectively, versus default settings.Table III summarizes energy conservation for different CNNs.Source: Table III · p. 4, A. Impact of GPU DVFS on Performance… · p. 4

B. Impact of GPU DVFS on Convolution Algorithms

Core-frequency scaling affects convolution algorithms differently across GPUs. P100 and V100 show a middle-frequency energy sweet spot, while Turing’s IPC GEMM performance scales faster than FFT Tile and Winograd.

B. Impact of GPU DVFS on Convolution Algorithms

  • P100 and V100 show similar performance increases and a middle-frequency energy sweet spot across the three algorithms, alongside a sharp power rise above 1,000 MHz.The paper links the Tesla GPUs’ energy valley to the power increase above 1,000 MHz.Source: B. Impact of GPU DVFS on Convolution… · p. 4
  • On GTX 2080Ti, IPC GEMM accelerates more than FFT Tile and Winograd, whose power changes negligibly with core frequency; memory-frequency scaling is also not significant.The authors suggest FFT Tile and Winograd may not fully use Turing’s computational resources.Source: B. Impact of GPU DVFS on Convolution… · p. 4
  • 14.5% for GEMM, 12.6% for FFT, and 15.8% for Winograd are the average energy conservation results from optimal core frequency settings versus default.Table IV reports training energy conservation for different convolution algorithms.Source: Table IV · p. 4, B. Impact of GPU DVFS on Convolution… · p. 4

C. Discussion

The paper finds that core-frequency performance gains depend on GPU core utilization, which is shaped by data loading and GPU kernels. Energy savings depend on the joint changes in performance and power as frequency rises, illustrated by a V100 AlexNet case where power rises sharply above 1,000 MHz.

C. Discussion

  • Performance gains from higher core frequency depend on GPU core utilization, which is influenced by data-loading latency during training and by the GPU kernels used.Data loading occurs outside the GPU, and whether its latency is hidden affects core utilization.Source: C. Discussion · p. 4
  • Energy conservation depends on how both performance and power change as core or memory frequency increases.Source: C. Discussion · p. 5
  • For V100 AlexNet, performance rises approximately 83% from 510 MHz to 945 MHz while average power rises 66%; above 1,000 MHz, higher core voltage supports a sharp power increase and energy rises again.Source: C. Discussion · p. 5

V. CONCLUSION

The study concludes that choosing an optimal GPU core frequency can improve DNN performance and conserve energy, and identifies layer-wise DVFS and multi-task scheduling as future directions.

V. CONCLUSION

  • Up to 33% performance improvement can accompany energy savings of up to 23.1% for training and 26.4% for inference with optimal core frequency.The conclusion also states these schemes can operate without significant performance degradation.Source: §V · p. 5
  • GPU DVFS has potential for energy-efficient DNN processing schemes without significant performance degradation.Source: §V · p. 5
  • Future work could explore layer-wise DVFS and use GPU DVFS to improve system-wide throughput and decrease energy in multi-task training schedules.The paper notes that different layers may have different energy-conservation benefits from DVFS settings.Source: §V · p. 5

APPENDIX

The appendix compares core- and memory-frequency scaling across convolution algorithms, GPUs, DNNs, and batch sizes; implicit GEMM shows different energy-efficiency patterns on Tesla GPUs and the GTX 2080Ti.

APPENDIX

  • The appendix reports results for four DNNs on three GPUs across GEMM, Winograd, and FFT, examining core- and memory-frequency effects; some large batch sizes are unsupported due to GPU memory limits.For GTX2080Ti memory-frequency scaling, it calculates the geometric mean of each memory frequency’s data across all memory frequency sets.Source: APPENDIX · p. 7

A. Using implicit GEMM algorithm

  • Below 1,000 MHz on P100 and V100, performance mostly grows faster than power; above 1,000 MHz, power jumps and best energy efficiency usually lies midrange.AlexNet and VggNet-16 performance rarely changes with batch size, while GoogleNet and ResNet-50 gain throughput at larger batch sizes.Source: A. Using implicit GEMM algorithm · p. 7
  • On GTX 2080Ti, the highest core frequency mostly gives the best energy efficiency, while low memory frequency saves energy because memory-frequency increases barely improve training performance.Source: A. Using implicit GEMM algorithm · p. 7

B. Using Winograd algorithm

Winograd’s energy-efficiency patterns differ by GPU: Tesla P100 and V100 favor middle core frequencies, while GTX 2080Ti shows varying gains from core scaling and little energy benefit from higher memory frequency.

B. Using Winograd algorithm

  • On P100 and V100, Winograd generally achieves better performance at nearly the same power as GEMM, giving it better energy efficiency.The best energy-efficient core frequency usually lies in the middle interval.Source: B. Using Winograd algorithm · p. 7
  • Winograd requires more GPU memory than GEMM, and only a few batch sizes are supported for the four DNNs on P100 and V100.Source: B. Using Winograd algorithm · p. 7
  • On GTX 2080Ti, core-frequency increases produce DNN- and batch-dependent performance gains; higher batch sizes raise power for GoogleNet and ResNet-50, while memory-frequency increases do not help conserve energy.AlexNet power remains nearly unchanged across batch sizes.Source: B. Using Winograd algorithm · p. 7

C. Using FFT algorithm

FFT has constrained batch-size support on Tesla GPUs and limited benefits on GTX 2080Ti; for the tested DNNs, its small convolution kernels also make it difficult to outperform GEMM and Winograd.

C. Using FFT algorithm

Loading 1905.11012v1…