PaperlayerView exact revision on arXiv

Source-linked AI summary

The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization

Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, Mosharaf Chowdhury

arXiv:2505.06371v2cs.LGcs.AI

TL;DR

Generative-AI services face an energy bottleneck, while energy remains a secondary consideration in ML systems. The paper presents the ML.ENERGY Benchmark for realistic inference-energy measurement and latency-aware optimization recommendations. Early-2025 results show energy varies across models and tasks, with design trade-offs and opportunities for automated savings.

  • Problem

    Energy is a growing bottleneck for generative-AI services, but remains secondary to traditional optimization objectives such as time and accuracy.

    Source: §1 · p. 1

  • Method

    The ML.ENERGY Benchmark measures inference energy in realistic service settings and recommends energy-efficient configurations based on application-specific performance constraints.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 5

  • Results

    Early-2025 results show energy varies across generative-AI models and tasks, with energy–time–architecture trade-offs and savings opportunities from automated optimization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Energy consumption is affected by design choices across application, model, software, and hardware layers, supporting cross-layer rather than siloed optimization.

    Source: §6 · p. 10

Abstract

from arXiv · show

As the adoption of Generative AI in real-world services grow explosively, energy has emerged as a critical bottleneck resource. However, energy remains a metric that is often overlooked, under-explored, or poorly understood in the context of building ML systems. We present the ML$.$ENERGY Benchmark, a benchmark suite and tool for measuring inference energy consumption under realistic service environments, and the corresponding ML$.$ENERGY Leaderboard, which have served as a valuable resource for those hoping to understand and optimize the energy consumption of their generative AI services. In this paper, we explain four key design principles for benchmarking ML energy we have acquired over time, and then describe how they are implemented in the ML$.$ENERGY Benchmark. We then highlight results from the early 2025 iteration of the benchmark, including energy measurements of 40 widely used model architectures across 6 different tasks, case studies of how ML design choices impact energy consumption, and how automated optimization recommendations can lead to significant (sometimes more than 40%) energy savings without changing what is being computed by the model. The ML$.$ENERGY Benchmark is open-source and can be easily extended to various customized models and application scenarios.

1 Introduction

Generative AI services face growing energy constraints, while energy remains secondary to traditional optimization objectives. The paper introduces an extensible benchmark and leaderboard for realistic inference-energy measurement and optimization, and presents early-2025 findings on model and task variation, design trade-offs, and optimization opportunities.

1 Introduction

  • Energy is a critical bottleneck for generative AI services, yet it remains secondary to objectives such as time and accuracy.The paper notes that inference reportedly accounts for 80–90% of total compute demand.Source: §1 · p. 1
  • The ML.ENERGY Benchmark measures inference energy for generative AI models across varied tasks and realistic deployment environments, and provides automated optimization recommendations.The corresponding leaderboard reports benchmark results; the benchmark is open-source and designed to be extended.Source: §1 · p. 2
  • Early-2025 results examine energy variation across models and tasks, energy–time–architecture trade-offs, and savings opportunities from automated optimization.The paper frames these as results from the ML.ENERGY Leaderboard iteration.Source: §1 · p. 2

2 Design Principles

The design principles aim to make generative-AI energy benchmarks representative of real-world services and their measurements accurate, reusable, and actionable.

2 Design Principles

  • The principles collectively target measurements that represent real-world generative-AI deployments and remain accurate, reusable, and actionable.The benchmark’s stated overarching goal is to produce representative measurement results.Source: §2 · p. 2

2.1 Generalizability and Portability

  • Software-based GPU energy measurement supports portability without hardware modification, while GPU standardization makes measurements useful across systems using the same GPU.The authors motivate this choice by GPUs’ role as dominant workers and energy consumers in ML systems.Source: §2.1 · p. 3

2.2 Representing Real-World Deployments

  • Production-grade stacks and representative workloads, measured in a saturated-batch steady state, are used to approximate long-term serving deployments.The steady state is the period when batch size reaches the server’s configured maximum.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3 Energy Measurement at the Right Granularity

  • Energy is reported per complete generation response, capturing model verbosity and task-specific output-length distributions that per-token measures can miss.The intended granularity avoids measurements that are too coarse for runtime insight or too fine to capture service-level characteristics.Source: §2.3 · p. 3

2.4 Actionable Measurement Results

  • Users specify application-specific latency constraints, and the benchmark recommends a configuration that meets them together with its expected energy savings.The authors argue that generalized measurements allow these recommendations to inform optimization across a wide range of systems.Source: §2.4 · p. 3

3 The ML.ENERGY Benchmark

The ML.ENERGY Benchmark measures generative-AI inference energy and recommends configurations that satisfy application-specific performance constraints. Its design supports service-aware accounting and user extensions.

3 The ML.ENERGY Benchmark

  • The benchmark combines realistic energy measurement with automated recommendations and extension points for customized application scenarios.Its overall flow includes service-aware measurement and accounting alongside optimization recommendations.Source: §3 · p. 4

3.1 Benchmark Flow

  • The benchmark flow selects a model, request dataset, and configurations, measures each configuration, then recommends an energy-optimal option that meets a latency target.Users set the target after benchmarking, and recommendations use the time–energy Pareto frontier.Source: §3.1 · p. 4

3.2 Energy Measurement and Service-Aware Energy Accounting

  • For diffusion, batch energy is allocated across requests; for LLMs, steady-state energy per token is scaled by output-token counts, while batch-size sweeps capture differing utilization and energy–time points.The LLM steady state is the interval with the server’s maximum configured batch size, intended to approximate well-utilized long-term service.Source: §3.2 · pp. 4–5

3.3 Automated Optimization Recommendation

  • The benchmark constructs an energy–time Pareto frontier and selects the minimum-energy configuration that satisfies the user’s latency constraint.The constraint depends on the application: for example, whole-generation time for images or code, versus average time per output token for conversational AI.Source: §3.3 · p. 5

3.4 Extending the Benchmark

  • Users can extend the benchmark to customized models and scenarios, vary configuration sweeps, run on supported hardware, and derive context-dependent power, throughput, cost, and carbon metrics.Additional configuration dimensions and finer sweeps take longer to benchmark but can reveal more energy-efficient configurations.Source: §3.4 · pp. 5–6

4 Results Highlight

Early-2025 results span 40 models and six tasks, revealing wide variation in inference energy and trade-offs involving model design, latency, and energy. The benchmark’s time–energy frontier identifies lower-energy configurations that meet application latency targets.

4 Results Highlight

  • The early-2025 evaluation covered 40 models and six tasks on A100 and H100 GPUs using vLLM and Diffusers.Source: §4 · p. 6

4.1 Energy Measurements

  • Diffusion models’ per-request energy can match larger LLMs because they draw more power and long service latency limits concurrency, while TDP estimates overstate measured use by up to 4.1×.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Energy Implications of ML Design Decisions

  • Reasoning models generate one to two orders of magnitude more output tokens per request than standard chat models, raising energy per generation and per token as long outputs limit batch size.Source: Table 1 · p. 7, §4.2 · p. 8
  • LLMs of similar sizes consume different energy partly because response lengths vary; Mistral Large responses averaged 36% longer than Mixtral 8×7B.Source: §4.2 · pp. 7–8
  • As batch size grows, 7B Phi-3 Small can use less energy than 3.8B Mini because Mini’s MHA KV cache uses 3× more memory and limits batching.Source: Figure 4 · p. 8, §4.2 · p. 8
  • Llama 3.1 70B draws much less than GPU maximum power, while Stable Diffusion 3 Medium approaches it; low-intensity LLM decoding is bottlenecked by VRAM bandwidth.Source: Figure 5 · p. 8, §4.2 · p. 8
  • Stable Diffusion XL Turbo’s lower 512×512 resolution changes its energy use relative to SDXL at 1024×1024, while each additional denoising step linearly increases energy.Source: Figure 6 · p. 9, §4.2 · p. 9

4.3 Automated Energy Optimization Recommendation

  • A 100 ms average-TPOT target yields a 77 ms frontier point using 44% less energy per generation than latency-minimizing; for Stable Diffusion 2.1, a 5-second target yields 3.63-second latency and 21% less energy than latency-minimizing.Source: §4.3 · pp. 9–10

5 Related Work

Prior inference-energy efforts leave gaps in task coverage, deployment realism, accessibility, or reproducibility; ML.ENERGY combines broader generative-AI measurement with automated optimization. Related optimization systems address training or inference clusters, while ML.ENERGY recommendations are described as complementary.

Prior energy measurement

  • Existing energy leaderboards have distinct scope limits: LLM-Perf omits task-specific output lengths, AI Energy Score fixes batch size at 1, and MLPerf Power requires physical analyzer installation and covers few architectures per task.Google's disclosed service measurements use internal workloads, hardware, and models, limiting generalizability and reproducibility.Source: §5 · p. 10

ML.ENERGY's positioning

  • ML.ENERGY is presented as the first inference-energy benchmark for modern generative AI models and supports both measurement and optimization.Source: §5 · p. 10

Related optimization

  • Zeus, EnvPipe, and Perseus optimize training energy, while µ-Serve and DynamoLLM target inference clusters; ML.ENERGY recommendations are complementary to these efforts.Source: §5 · p. 10
  • The authors argue that the results support cross-layer energy optimization spanning model, software, and hardware rather than optimization siloed within one layer.Source: §5 · p. 10

6 Conclusion

The conclusion presents ML.ENERGY as a generative-AI benchmark that pairs realistic energy measurements with automated, constraint-aware configuration recommendations. Its measurements show energy is affected by choices across the AI stack, motivating cross-layer rather than single-layer optimization.

Conclusion

  • ML.ENERGY provides realistic energy measurements and automatically suggests energy-optimal configurations based on user- and application-specific performance constraints.Source: §6 · p. 10
  • Measurement results show that energy consumption is impacted by design choices across application, model, software, and hardware layers.Source: §6 · p. 10
  • The authors conclude that automated cross-layer energy optimization is important instead of optimizations siloed within a single layer.Source: §6 · p. 10

NeurIPS Paper Checklist

The checklist asks whether the introduction accurately reflects the paper's contributions and scope, and records an affirmative justification. Its guidance emphasizes accurately stating contributions, assumptions, limitations, and generalizability.

NeurIPS Paper Checklist

  • The checklist's question asks whether the abstract and introduction accurately reflect the paper's contributions and scope.Source: NeurIPS Paper Checklist · p. 16
  • The recorded justification says the abstract and introduction reflect the paper's contributions and scope.Source: NeurIPS Paper Checklist · p. 16
  • The checklist guidance calls for claims to match results, reflect expected generalization, and distinguish aspirational goals from attained contributions.Source: NeurIPS Paper Checklist · p. 16

2. Limitations

The supplied checklist material identifies a cost-related limit on repeated benchmark runs and says limitations are discussed in Appendix E. It also reports public access to benchmark code and leaderboard data, with experimental details distributed across the paper, appendix, and repository.

Limitations

  • The checklist says limitations are discussed in Appendix E.Source: §2 · p. 16

Reproducibility

  • Benchmark code and result data are open-source and documented, while important experimental details appear in the paper and appendix and full details are in the code repository.Source: §2 · p. 18

Limitations

  • The checklist records that the paper does not report suitable error bars or other statistical-significance information.Source: §2 · p. 18
  • The authors could not run the benchmark multiple times because even one run had high monetary cost.Source: §2 · p. 18

10. Broader impacts

The accompanying checklist notes the use of external assets and directs readers to Appendix F for the paper’s societal-impact discussion.

10. Broader impacts

  • The checklist directs readers to Appendix F for the discussion referenced in this section.Source: §10 · p. 19
  • The benchmark uses models and datasets created by others, credits their authors through citations, and recommends default request datasets without bundling specific models or datasets.The appendix also includes a comprehensive table of assets.Source: §10 · p. 20
  • The benchmark code is open-source, documented, and released under the Apache-2.0 license.Source: §10 · p. 21
  • The paper states that it does not involve crowdsourcing or research with human subjects.Source: §10 · p. 21

A Tasks, Model Architectures, and Default Request Datasets

The benchmark supports varied model architectures and tasks, and the accompanying system-configuration analysis shows that inference energy depends on how requests are served. Under overload, the tested preemption mechanism changes energy use.

A Tasks, Model Architectures, and Default Request Datasets

  • The benchmark supports model architectures and default request datasets across tasks, and also supports fine-tuned versions of its supported models.It is designed to be extensible.Source: A Tasks, Model Architectures, and… · p. 23

B Energy Implication of System Parameters

B.1 Request Preemption Mechanism

  • Recomputation discards temporary request state and rebuilds it on restoration; Swapping moves that state to CPU memory and returns it to the GPU.The better choice depends on the hardware, software configuration, and model being served.Source: §B.1 · p. 23
  • Under overload, Swapping consistently uses less energy per generation than Recomputation, while energy generally rises as overload and preemptions increase.Recomputation repeats work when restoring requests; Swapping moves request state between GPU and CPU memory.Source: §B.1 · p. 24, Figure 8 · p. 24

B.2 Tensor Parallelism Scaling

Energy changes with system configuration: multi-GPU communication can offset reduced per-GPU computation, while disaggregation and chunked-prefill settings show different energy patterns.

B.2 Tensor Parallelism Scaling

  • Energy barely changes when scaling Llama 3.1 8B from one GPU to two, but rises with further scaling as communication offsets reduced per-GPU computation.More GPUs can slow execution of the same computation and increase energy; the impact may be larger without sufficient or state-of-the-art networking.Source: §B.2 · pp. 24–25

B.3 Prefill–Decode Disaggregation

  • Decode consumes most energy in the tested prefill–decode disaggregation setup, and configurations have little impact on absolute energy or its split when phase throughputs are reasonably balanced.Source: §B.3 · p. 25, Table 4 · p. 25

B.4 Chunked Prefill

  • With chunked prefill, batching more sequences or tokens lowers energy per generation through better energy amortization, with diminishing returns beyond a point.Table 5 reports Llama 3.1 8B measurements on a synthetic long-context dataset running on H100 GPUs.Source: §B.4 · p. 25, Table 5 · p. 25

C Power Consumption Analysis

The power analysis finds different usage patterns across model types and phases: LLM decoding draws relatively little power, while diffusion denoising approaches GPU TDP. The timelines also connect power variation to the work performed by the serving system.

C Power Consumption Analysis

  • LLMs and VLMs generally draw less power than GPU TDP, while diffusion models approach maximum GPU power at non-small batch sizes.VLMs draw slightly more power than LLMs; the paper relates these patterns to their compute intensity.Source: C Power Consumption Analysis · p. 26, Figure 11 · p. 27
  • LLM prefill has high compute intensity and power draw, whereas the repeated decode phases have lower compute intensity and power draw.The initial power spike corresponds to back-to-back prefills; subsequent drops recur when CPU-side scheduling leaves the GPU idle.Source: C Power Consumption Analysis · pp. 26–27, Figure 12 · p. 27
  • Stable Video Diffusion XT’s denoising phase draws power close to GPU TDP, while decode includes VRAM power increases as memory is allocated and written.The timeline shows 25 local GPU-power peaks per batch, each corresponding to a denoising step.Source: C Power Consumption Analysis · pp. 27–28, Figure 12 · p. 27
  • The power breakdown shows that memory operations draw less power than computation, and that software and hardware components in the system stack affect energy use.Source: C Power Consumption Analysis · p. 28

D The ML.ENERGY Leaderboard and Benchmark

Since its July 2023 launch, ML.ENERGY has expanded as an inference-energy leaderboard and benchmark, though the presented snapshot is not exhaustive. The authors argue that better measurement and optimization could deepen understanding and reduce generative-AI service costs.

D The ML.ENERGY Leaderboard and Benchmark

  • Launched in July 2023, ML.ENERGY began as an inference-energy leaderboard and benchmark for generative AI and has expanded to include more tasks, models, and datasets.The paper presents its results as a snapshot and says updates will continue as resources allow.Source: D The ML.ENERGY Leaderboard and… · p. 28

E Limitations

  • The benchmark is non-exhaustive across tasks, models, datasets, and hardware, with costly batch updates and current support focused mainly on flagship NVIDIA GPUs.The authors collect new advances within a time window rather than adding each model or task as it appears.Source: E Limitations · p. 28

F Broader Impacts

  • The authors argue that optimizing energy can lower generative-AI service infrastructure costs and help democratize access.They also believe accurate measurement can enhance understanding in research and industry and fuel energy-optimization work.Source: F Broader Impacts · pp. 28–29

Related papers

Loading 2505.06371v2…