PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Do the Joules Go? Diagnosing Inference Energy Consumption

Jae-Won Chung, Ruofan Wu, Jeff J. Ma, Mosharaf Chowdhury

arXiv:2601.22076v2cs.LGcs.DC

TL;DR

As energy becomes a critical bottleneck in AI infrastructure, measuring inference energy alone does not explain why configurations differ or guide optimization. This paper studies inference time and energy across generative AI workloads and develops a framework linking configuration choices to latent factors and end metrics, finding order-of-magnitude differences across tasks and systems.

  • Problem

    Measuring inference energy is necessary but insufficient for understanding why configurations consume different amounts of time or energy and for guiding optimization.

  • Method

    The paper combines a large-scale study of 46 models across 7 tasks and 1,858 configurations on NVIDIA H100 and B200 GPUs with controlled comparisons and a framework for reasoning about latent energy drivers.

  • Results

    Task type, generation modality, and GPU utilization produce order-of-magnitude energy variation, including 25× differences across LLM tasks, over 100× higher video-generation energy than image generation, and 3–5× gaps from utilization differences.

  • Takeaways & Limitations

    Inference time and energy can be analyzed through latent factors such as memory, compute, utilization, and application constraints rather than treated as black-box measurements.

  • Takeaways & Limitations

    MLLM analyses are limited by CPU-side bottlenecks, which make it difficult to isolate factors affecting GPU energy.

Abstract

from arXiv · show

Energy is now a critical ML computing resource. While measuring energy consumption and observing trends is a valuable first step, accurately understanding and diagnosing why those differences occur is crucial for optimization. To that end, we begin by presenting a large-scale measurement study of inference time and energy across the generative AI landscape with 46 models, 7 tasks, and 1,858 different configurations on NVIDIA H100 and B200 GPUs. Our empirical findings span order-of-magnitude variations: LLM task type can lead to 25$\times$ energy differences, video generation sometimes consumes more than 100$\times$ the energy of images, and GPU utilization differences can result in 3--5$\times$ energy differences. Based on our observations, we present a framework for reasoning about the underlying mechanisms that govern time and energy consumption. The essence is that time and energy are determined by latent metrics like memory and utilization, which are in turn affected by various factors across the algorithm, software, and hardware layers. Our framework also extends directly to throughput per watt, a critical metric for power-constrained datacenters.

1. Introduction

Energy consumption is becoming a critical bottleneck, but measuring it alone does not explain why configurations differ. This paper studies those differences and develops a framework linking end metrics to underlying factors.

  • Energy demand is becoming a critical bottleneck because AI compute growth outpaces the time required to build new power sources.
  • Measuring energy is necessary but insufficient for explaining configuration differences or guiding optimization.
  • 46 models, 7 tasks, and 1,858 configurations on NVIDIA H100 and B200 GPUs reveal order-of-magnitude energy variation.
  • 25× energy differences arise across LLM tasks, video can exceed image energy by 100×, and GPU utilization can produce 3–5× gaps.
  • Lower precision is not always faster or more efficient, and additional GPUs can reduce total energy by unlocking memory capacity.
  • The framework attributes time and energy to latent factors such as memory, utilization, and application constraints, extending to power-constrained datacenter capacity.

2. Methodology

The study measures GPU inference energy using production-grade serving systems across current models and NVIDIA H100 and B200 nodes. It derives per-token or per-request energy from steady-state or batched measurements while sweeping batch size and GPU count.

  • The methodology focuses on GPU energy because AI accelerators account for 50–70% of datacenter power or energy.
  • The benchmark uses Zeus, current LLMs and diffusion models, vLLM and xDiT, and NVIDIA H100 and B200 nodes with NVSwitch.
  • LLM energy per token is computed from steady-state total energy divided by tokens generated during steady state.
  • Diffusion per-request energy divides total batch energy by batch size, while batch sizes and GPU counts are swept with BF16 as the default precision.

3. Energy by Architecture

Energy varies substantially across LLM, multimodal, and diffusion workloads because output length, memory pressure, preprocessing, denoising steps, resolution, and frame count shape utilization and work. Across architectures, task and runtime factors can matter as much as model size.

  • Large Language Models: Longer reasoning sequences increase KV-cache memory use, restrict batch size, and raise energy per token.For Qwen 3 32B, Problem Solving uses 2.1× higher energy per token at task-specific maximum batch sizes and 23× more energy per response.
  • Large Language Models: 10× more output tokens in Problem Solving than Text Conversation contribute to 25× higher energy per response.Problem Solving averages 6,988 tokens versus 717 and 4,625 J versus 184 J per response.
  • Multimodal LLMs: CPU-side vision preprocessing can bottleneck multimodal inference even when GPU memory has spare capacity.
  • Multimodal LLMs: Image inputs use 1.1–5.2× and video inputs 1.3–15.0× the text energy per token.Vision encoders, increased input length, and CPU-side preprocessing can reduce batch size and utilization.
  • Diffusion Models: 20× energy variation occurs across text-to-image models, and denoising steps can outweigh parameter count.Hunyuan-DiT 1.2 consumes more energy than SD 3.5 Large despite having fewer parameters, largely because it uses 50 versus 28 denoising steps.
  • Diffusion Models: 26 kJ to 1.16 MJ is required for one generated video, one to two orders of magnitude more than images.Resolution and frame count help explain why smaller models can consume more energy than larger ones.

4. Deeper Dive into Energy

Energy varies with batch size, model architecture, GPU generation, precision, and multi-GPU configuration. These effects reflect tradeoffs among utilization, memory capacity, communication overhead, latency, and energy.

  • 4.1. Batch Size: 3–5× energy per token reduction can result from increasing batch size, although latency and power also increase.Throughput rises with batch size but eventually shows diminishing returns as GPU utilization saturates.
  • 4.2. Model Size and Architecture: MoE models generally consume less energy than dense models with similar total parameters because active parameters are lower.30B A3B uses 3.56× less energy per token than dense 32B, while total parameters still affect memory requirements.
  • 4.3. GPU Generation: B200 wins 88% of LLM comparisons and 86% of Text-to-Image comparisons at matched latency constraints, but tight LLM constraints can favor H100.For Text-to-Video, B200 wins 79% of comparisons; small diffusion models can underutilize B200.
  • 4.4. Precision: At batch sizes 8–16, FP8 can consume up to 56% more energy and be up to 26% slower than BF16 before gaining advantages at larger batches.Extra quantization operations and GPU underutilization delay FP8’s benefits until the workload better saturates available compute.
  • 4.5. Multi-GPU Scaling: Adding GPUs at the same batch size usually reduces latency but increases energy because communication overhead and lower per-GPU utilization offset parallelism.Across H100 configurations, energy increases in 93% of cases while latency always decreases.
  • 4.5. Multi-GPU Scaling: More GPUs can reduce energy when additional memory capacity enables much larger batches; GPT OSS 120B achieves 68% lower minimum energy on two H100s.This benefit occurs when the original configuration is memory-capacity bound rather than when extra GPUs merely add parallelism.

5. Reasoning about Energy Consumption

The paper presents a framework for explaining inference time and energy through latent factors shaped by algorithmic, software, and hardware choices. It connects these mechanisms to configuration tradeoffs, datacenter throughput per watt, and root-cause diagnosis.

  • Mechanisms: Energy consumption reflects interacting factors across hardware, software, and algorithms, including computation, memory, utilization, and application constraints.The framework distinguishes configuration properties and knobs, latent variables, and observed end metrics.
  • Mechanisms: B200 generally consumes less energy than H100, consistent with newer architectures delivering more operations per joule.The paper attributes this to microarchitectural improvements and technology-node shrinks.
  • Utilization: CPU, network, disk, or other system bottlenecks can leave GPUs underutilized, wasting static power and increasing energy per token.Multimodal LLMs illustrate this when CPU-side vision preprocessing limits batch size and GPU utilization.
  • Utilization: Lower arithmetic intensity can reduce GPU utilization and offset energy savings from lower-precision arithmetic, especially at smaller batch sizes.FP8 extra operations and smaller matrix multiplications can both have lower arithmetic intensity.
  • Time–Energy Tradeoff Frontier: When GPUs are the bottleneck, batch size, memory capacity, application constraints, and GPU count shape the time–energy tradeoff frontier.Larger batches can reduce energy per work but increase per-request latency; memory capacity limits feasible batch sizes.
  • Time–Energy Tradeoff Frontier: A proper time–energy tradeoff frontier may not exist when GPU underutilization allows both time and energy to fall through improved utilization.This boundary distinguishes configuration tradeoffs from system-bottleneck remediation.
  • Time–Energy Tradeoff Frontier: Memory capacity can make additional GPUs reduce energy by enabling larger batch sizes, while adding GPUs otherwise may add overhead.The relevant condition is whether aggregate memory capacity unlocks larger batches.
  • Datacenter Capacity: Throughput per watt is the inverse of energy per fixed work, extending the framework to power-constrained datacenter capacity.Examples include tokens per second per watt and images per second per watt.

6. Related Work

Existing tools measure inference energy under realistic or industry-relevant conditions, but they differ in scope and generalizability. The paper identifies a deeper gap: mapping latent factors and causal structures to observed energy differences.

  • Coverage and Gaps: Existing benchmarks and disclosures provide energy or power measurements, but vary in model coverage, batch-size assumptions, deployment scope, and generalizability.MLPerf Power covers fewer models and tasks, the Hugging Face score fixes batch size to 1, and Google reports internal-system measurements.
  • Coverage and Gaps: Existing work does not go deeply into mapping latent factors and causal structures that manifest as energy differences.This motivates the paper’s focus on explaining, rather than only measuring, inference energy.

7. Conclusion

The paper argues that systematic optimization requires understanding inference energy, not merely measuring it. Its framework maps configuration choices through latent factors such as compute, memory, utilization, and application constraints to end metrics.

  • Conclusion: Systematic optimization requires understanding inference energy consumption in addition to measuring it.The conclusion frames this need in the context of scaling AI infrastructure.
  • Conclusion: Time and energy are governed by latent factors that mediate between configuration choices and observed end metrics.The cited examples include compute, memory, utilization, and application constraints.

A. List of Tasks and Models

The paper lists the benchmark tasks, their request datasets, and the models used in the measurement studies.

  • Benchmark Inventory: Table 2 lists benchmark tasks and request datasets, while Table 3 lists the models used in the measurement studies.These tables support the studies in Sections 3 and 4.

B. B200 vs H100 for Text to Video

Figure 11 compares B200 and H100 energy for Text to Video diffusion models under a 500 s generation-latency constraint, with annotations showing B200’s energy reduction relative to H100.

  • B200 vs H100 for Text to Video: Figure 11 compares B200 and H100 energy for Text to Video diffusion models at 500 s generation latency.The comparison is made under a fixed latency constraint.
  • B200 vs H100 for Text to Video: The figure’s annotations indicate the percentage by which B200 reduces energy relative to H100.The annotations express the comparison as an energy reduction percentage.
  • B200 vs H100 for Text to Video: The comparison focuses on energy consumption while holding generation latency to the stated 500 s constraint.This frames the hardware comparison at matched latency.

C. Throughput per Watt

Throughput per watt is evaluated alongside energy per token for four LLMs on B200 across batch sizes. Increasing batch size lowers energy per token and raises throughput per watt, which are inverse measures of efficiency.

  • Throughput per Watt: Figure 12 reports energy per token and throughput per watt for four LLMs on B200 across batch sizes.The figure examines both efficiency metrics across changing batch sizes.
  • Throughput per Watt: As batch size increases, energy per token decreases, indicating better efficiency.The passage explicitly associates lower energy per token with improved efficiency.
  • Throughput per Watt: As batch size increases, throughput per watt increases, meaning more work is done per unit power.Throughput per watt is described as the inverse of energy per work.
Loading 2601.22076v2…