PaperlayerView exact revision on arXiv

Source-linked AI summary

Compute and Energy Consumption Trends in Deep Learning Inference

Radosvet Desislavov, Fernando Martínez-Plumed, José Hernández-Orallo

arXiv:2109.05472v2cs.LGcs.AI

TL;DR

The paper asks whether exponential growth in deep-learning parameters translates into exponential energy consumption. It analyzes inference costs, model performance, and hardware efficiency across computer vision and NLP, finding softer energy growth when consolidated efficient models are considered. The conclusion is qualified by the multiplicative effect of increasing AI penetration.

  • Problem

    The paper asks whether reported exponential growth in deep-learning model size also produces exponential growth in energy consumption.

  • Method

    The study compares inference FLOPs and energy across computer-vision and NLP models while incorporating hardware efficiency and later consolidated implementations.

  • Results

    General-use models achieve systematic performance growth at almost constant energy consumption, unlike the faster growth associated with cutting-edge models.

  • Takeaways & Limitations

    Inference energy grows more softly than parameter trends suggest because algorithmic improvements, hardware specialization, and hardware efficiency partially compensate for growth.

  • Takeaways & Limitations

    The conclusion assumes a constant multiplicative factor, although increasing AI penetration may escalate total energy consumption.

Abstract

from arXiv · show

The progress of some AI paradigms such as deep learning is said to be linked to an exponential growth in the number of parameters. There are many studies corroborating these trends, but does this translate into an exponential increase in energy consumption? In order to answer this question we focus on inference costs rather than training costs, as the former account for most of the computing effort, solely because of the multiplicative factors. Also, apart from algorithmic innovations, we account for more specific and powerful hardware (leading to higher FLOPS) that is usually accompanied with important energy efficiency optimisations. We also move the focus from the first implementation of a breakthrough paper towards the consolidated version of the techniques one or two year later. Under this distinctive and comprehensive perspective, we study relevant models in the areas of computer vision and natural language processing: for a sustained increase in performance we see a much softer growth in energy consumption than previously anticipated. The only caveat is, yet again, the multiplicative factor, as future AI increases penetration and becomes more pervasive.

Introduction

The paper asks whether growing deep-learning models imply exponential inference energy growth, examining models, hardware efficiency, and consolidated implementations across computer vision and NLP.

  • Motivation: Inference costs can exceed training costs in deployed systems because each trained model is applied repeatedly.Inference is estimated to account for up to 90% of costs.
  • Research question: The study addresses whether exponentially growing network size necessarily produces exponentially growing energy consumption.Hardware specialization, declining energy per unit of compute, and later efficient implementations may offset model growth.
  • Scope: The analysis covers image classification in computer vision and GLUE language-understanding results in natural language processing.These domains were selected because of historical data availability and the centrality of language understanding in NLP.
  • Approach: The method combines inference FLOPs per input with historical hardware data to estimate energy required for one prediction.Hardware data are used to estimate FLOPs obtained per Joule for each chip.
  • Contributions: Better DNN performance reflects algorithmic improvements as well as computing power, while efficient post-breakthrough models can keep inference energy low.The paper also examines how hardware improvements and specialization reduce energy consumption.
  • Resources: The paper provides its collected data and estimations publicly through the appendixes and a GitHub repository.

Background

Prior work documents growing AI compute and energy concerns, but inference costs have received less attention despite their substantial repeated-use contribution.

  • Prior work: Existing studies frequently analyze training computation, energy, and carbon emissions across AI applications and model families.Examples include analyses of NLP training and reinforcement-learning energy and carbon footprints.
  • Compute trends: AI models have doubled the computational power used every 3.4 months since 2012.
  • Research gap: Inference-cost research is comparatively sparse despite inference using a more representative share of compute and energy.
  • Inference studies: Prior inference studies have measured accuracy, memory, parameters, operations, time, power, or Joules per image for selected models.
  • Terminology: The paper distinguishes FLOPS as floating-point operations per second, clarifying terminology around compute and hardware speed.

Methodology

The methodology compiles model information from papers, implementations, existing databases, and measurement tools to estimate architectures’ compute and parameters.

  • Data collection: The researchers manually inspected original papers and official repositories, supplementing missing information with additional sources.
  • Data sources: Related papers provide comparative information about newly introduced models and previously existing models.
  • Data sources: Model implementations such as PyTorch supply pretrained models and reported performance data.
  • Data sources: Existing compilations and public databases contribute information about deep-learning architectures and their benchmarks.
  • Measurement: When reliable sources were unavailable, ptflops or similar tools calculated model FLOPs and parameters when implementations existed.
  • Scope: The general methodology is followed by detailed selection procedures for computer-vision, NLP, and hardware data.

CV Models Data Compilation

The dataset prioritizes popular, accurate vision models and available GLUE inference estimates, while tracking Nvidia GPU performance and efficiency over time.

  • CV Models Data Compilation: Image-classification models were selected using popularity and accuracy to represent influential and high-performing models over time.Popularity was assessed through citations and mentions in comparative analyses.
  • NLP Models Data Compilation: GLUE was chosen for NLP because it is representative and has performance information for many architectures.The dataset includes models since 2017 for which inference compute estimates were found.
  • Hardware Data Compilation: Nvidia GPU data from 2010 to 2021 include FLOPS, memory size, TDP, and launch date.The study emphasizes recent server GPUs because they are common in AI applications.
  • Hardware Data Compilation: GPU efficiency is estimated as peak FLOPS divided by TDP, expressed as GLOPS/Watt under full utilization.Actual efficiency may vary with workload, but the estimate is considered adequate for trend analysis.
  • Hardware Data Compilation: Google TPUs were considered but excluded because insufficient public information was available about hardware that is provided as a service.

Computer Vision Analysis

Computer vision models achieved major accuracy gains, but peak-accuracy models required exponentially more inference FLOPs. Over time, algorithmic improvements produced models with similar compute and much higher accuracy, while newer models also improved energy-relevant efficiency.

  • Accuracy and compute: ImageNet Top-1 accuracy rose from 56% for AlexNet in 2012 to 90.2% for EfficientNet-L2 in 2021, while forward-pass compute increased from 1.42 to 1040 GFLOPs.These values refer to single-model, one-crop evaluation.
  • Accuracy and compute: For the highest-accuracy model each year, inference compute followed an exponential growth trend, although many newer models required substantially less compute.The exponential trend is represented by a linear fit because the y-axis is logarithmic.
  • Algorithmic progress: At similar FLOPs to AlexNet, EfficientNet-B1 reached 79.1% Top-1 accuracy in 2019, while NoisyStudent-B1 reached 81.5% using extra data.The comparison covers a seven-year period and illustrates higher accuracy at similar computation.
  • Algorithmic progress: Models released after a SOTA breakthrough often achieved similar accuracy with far fewer FLOPs, and these efficient successors became popular in industry applications.The paper attributes better general-use DNN results partly to algorithmic improvements rather than only increased computing power.
  • Accuracy and compute: Extra training data can place models on the accuracy–GFLOPs Pareto frontier, but it does not change inference GFLOPs.The Pareto frontier consists mainly of newer models, including models trained with extra data.

Natural Language Analysis

NLP models show a trade-off between GLUE performance and inference compute: cutting-edge models require rapidly increasing GFLOPs, while later efficient models can match scores with fewer operations.

  • Natural Language Analysis: GLUE performance is analyzed against GFLOPs for single inputs of length 128.GLUE covers nine sentence-understanding tasks across broad domains.
  • Natural Language Analysis: Soon after high-compute models appear, other NLP models reach the same GLUE score with far fewer GFLOPs.Efficient architectures include MobileBERT and SqueezeBERT, while older models become inefficient relative to newer ones.
  • Natural Language Analysis: Cutting-edge NLP models show clear exponential growth in GFLOPs, while the overall model population does not scale as aggressively.Figure 6 distinguishes the most demanding released models from all models and accounts for differing input sequence lengths.
  • Natural Language Analysis: GLUE was selected instead of SUPERGLUE to provide a sufficiently long temporal window for the analysis.

Hardware Progress

The hardware analysis measures GPU performance and efficiency, showing that specialized inference hardware can substantially improve FLOPS per Watt, with realistic tensor-core performance below theoretical peaks.

  • Hardware Progress: FLOPS measures hardware performance, while FLOPS/Watt measures hardware efficiency across GPU precision formats and tensor cores.Theoretical tensor-core FLOPS are high, but realistic inference estimates for V100, A100, and T4 are lower.
  • Hardware Progress: Figure 7 presents theoretical Nvidia GPU GFLOPS per Watt, with source data reported in an appendix table.
  • Hardware Progress: GPU efficiency estimates produce good linear fits on a logarithmic scale for both computer vision and NLP hardware data.The 2018 T4 mixed-precision point is exceptionally efficient because the GPU was designed specifically for inference.

Energy Consumption Analysis

Inference energy is estimated by combining model FLOPs with hardware efficiency. Hardware progress softens energy growth, but high-accuracy models still show exponential growth, while the broader model trend is much flatter.

  • Energy Consumption Analysis: Inference energy is calculated by dividing a model’s forward-pass FLOPs by the FLOPS per Watt estimated for hardware available at release.The resulting quantity is the energy consumed by one inference, measured in Joules.
  • Energy Consumption Analysis: Figures 9 and 10 estimate Joules consumed by one forward pass for computer vision and NLP models, respectively.
  • Energy Consumption Analysis: Hardware progress softens the growth in FLOPs-related energy consumption for computer vision models.The dashed trend covers the highest-accuracy model per year, while the dotted line shows yearly average Joules.
  • Energy Consumption Analysis: High-accuracy models retain a clearly exponential energy-growth trend, whereas the all-model trend is nearly horizontal on the logarithmic plot.The same overall picture appears for NLP models in Figure 10.

Forecasting and Multiplicative Effect

The paper finds that efficiency gains do not fully offset compute growth for cutting-edge models, while the broader trend is less clear. Future energy impact also depends on how frequently AI inference is performed.

  • Forecasting and Multiplicative Effect: Performance and efficiency both continue improving, but they do not cancel out for cutting-edge models; cancellation is less clear for regular models.
  • Forecasting and Multiplicative Effect: Estimating future AI energy requires knowing how many inference passes occur per person, but the paper leaves that quantity for future work.The paper notes that smart devices, Internet of Things systems, self-driving cars, and surveillance cameras increase inference ubiquity and may perform many passes per second.
  • Forecasting and Multiplicative Effect: The energy of one best-model computer-vision inference approaches human body energy consumption in one second but remains far below total human energy use in one second.The reference values are approximately 100 Joules per second internally and 10,000 Joules per second externally.
  • Forecasting and Multiplicative Effect: NLP best-model energy grows faster than the computer-vision trend, while energy for regular NLP models may decrease.

Discussion and Future Work

The paper argues that inference energy grows much more slowly than parameter counts suggest because algorithmic improvements and hardware efficiency partly offset increasing model demands. However, total energy can still rise as AI deployment becomes more widespread, while broader analysis is limited by sparse reporting of inference costs.

  • Discussion and Future Work: Inference deserves emphasis because models are trained once but may execute millions of forward passes, making aggregate deployment costs much larger.The paper estimates inference accounts for up to 90% of costs.
  • Discussion and Future Work: Energy associated with inference grows more slowly than parameter counts because algorithmic improvements, hardware specialisation, and hardware efficiency partially compensate for model growth.The paper contrasts this pattern with the unbridled exponential growth often reported for parameter counts.
  • Discussion and Future Work: General-use models that settle one or two years after a breakthrough can sustain systematic performance growth at almost constant energy consumption.The paper distinguishes these models from new cutting-edge models, whose energy consumption still increases exponentially.
  • Discussion and Future Work: Increasing AI penetration can still escalate total energy consumption even if energy use per model remains controlled.The conclusion assumes a constant multiplicative factor for repeated use across devices and services.
  • Discussion and Future Work: Future analyses should incorporate AI penetration, energy and device costs, carbon footprint per Joule, and socioeconomic indicators, while improving reporting of FLOPs and inference energy.The paper notes that scientific publications often omit complete inference-cost information, limiting the number of benchmarks and models that can be studied.

Appendix

The appendix supplies supplementary methodological and data details supporting the paper’s analyses of compute, models, benchmarks, hardware, and energy consumption.

  • Appendix: The appendix covers FLOPs versus FLOPS, CV and NLP methodology, benchmarks, hardware precision, NLP performance and compute, FLOPS estimation, GLUE results, and GPU consumption data.These topics provide supporting detail for the paper’s model and energy analyses.

FLOPs vs FLOPS

The paper standardises compute accounting by distinguishing task-level FLOPs from hardware-level FLOPS and carefully normalising model, input, precision, and resolution measurements. It then compares performance and inference compute across computer vision and NLP models, including efficient models that reach similar scores with fewer operations.

  • FLOPs vs FLOPS: FLOPS measures hardware performance in floating-point operations per second, whereas FLOPs measures the operations required for a task such as one inference pass.A fused multiply-add is counted as 2 FLOPs in this work.
  • FLOPs vs FLOPS: Accuracy and FLOPs comparisons account for sampling procedures such as AlexNet’s ten-crop prediction, avoiding comparisons between unequal inference workloads.The analysis also standardises measurements around a forward pass and records the relevant input resolution.
  • FLOPs vs FLOPS: ImageNet classification models are selected using popularity and accuracy so yearly progress can be analysed across prominent and high-performing models.ImageNet provides a substantial historical record for computer vision progress.
  • FLOPs vs FLOPS: Hardware analysis incorporates tensor cores, mixed precision, and GPU throughput rather than relying only on theoretical FLOPS.The study uses experimental speedups because tensor-core theoretical performance does not fully reflect practical DNN inference utilisation.
  • FLOPs vs FLOPS: GLUE comparisons use single inputs of length 128 and represent inference GFLOPs as bubble size alongside GLUE scores over time.The benchmark contains diverse NLP tasks, while the analysis includes models from 2017 onward when inference compute could be estimated.
  • FLOPs vs FLOPS: SOTA NLP models require many FLOPs, but later models can reach the same GLUE score with substantially fewer FLOPs.This evolution is reported as similar to the pattern observed for ImageNet.
  • FLOPs vs FLOPS: EfficientNet FLOPs are estimated from a scaling relation when model papers do not report them, while ViT-G/14 FLOPs are extrapolated from reported resolutions.The ViT-G/14 calculation assumes the 518 × 518 finetuning resolution is also used for testing.
Loading 2109.05472v2…