PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Energy Efficiency of Large Language Models Using vLLM

K. Pronk, Q. Zhao

arXiv:2509.08867v1cs.SEcs.AI

TL;DR

Inference-energy studies have often relied on lab conditions, despite concerns about LLM energy use and emissions. This paper introduces a vLLM-based benchmark for realistic serving conditions and reports how energy efficiency varies with request load, model size, and architecture.

  • Problem

    Inference-energy research has often been limited to lab conditions, while LLM energy use and associated emissions raise environmental concerns.

    Source: §I · p. 1

  • Method

    The LLM Efficiency Benchmark uses vLLM to evaluate models under realistic serving conditions and examines request volume, model size, and architecture.

    Source: §I · p. 1, 1) Request load · p. 3

  • Results

    Energy efficiency varied with request load, model size, and architecture: per-request energy plateaued beyond 100 simultaneous requests, same-architecture efficiency declined near-linearly with size, and similar-size architectures showed no significant effect.

    Source: §IV · p. 3, A. Request amount · p. 3, §V · pp. 4–5

  • Takeaways & Limitations

    The benchmark demonstrates that realistic LLM serving scenarios can be simulated, and a future energy-efficiency database is intended to support developers' model choices.

    Source: §VI · p. 6

    The study does not account for model accuracy or other metrics such as perplexity.

    Source: B. Limitations · p. 5

Abstract

from arXiv · show

The prevalence of Large Language Models (LLMs) is having an growing impact on the climate due to the substantial energy required for their deployment and use. To create awareness for developers who are implementing LLMs in their products, there is a strong need to collect more information about the energy efficiency of LLMs. While existing research has evaluated the energy efficiency of various models, these benchmarks often fall short of representing realistic production scenarios. In this paper, we introduce the LLM Efficiency Benchmark, designed to simulate real-world usage conditions. Our benchmark utilizes vLLM, a high-throughput, production-ready LLM serving backend that optimizes model performance and efficiency. We examine how factors such as model size, architecture, and concurrent request volume affect inference energy efficiency. Our findings demonstrate that it is possible to create energy efficiency benchmarks that better reflect practical deployment conditions, providing valuable insights for developers aiming to build more sustainable AI systems.

I. INTRODUCTION

LLM growth is accompanied by rising energy use and CO2 emissions, while inference-energy research has often used lab conditions. The paper introduces a vLLM-based benchmark designed to reflect modern serving environments.

  • Growing LLM energy consumption and associated CO2 emissions raise concerns about environmental impact.Source: §I · p. 1
  • Recent inference-energy research has often been limited to lab conditions that do not reflect modern LLM serving.Source: §I · p. 1
  • The LLM Efficiency Benchmark uses vLLM to evaluate models under realistic conditions reflecting modern serving environments.vLLM is described as a high-performance backend optimized for large request volumes.Source: §I · p. 1

II. RELATED WORK

The related work reviews AI energy and carbon impacts, carbon-offset estimation, and strategies for reducing emissions. It notes that location- and time-based strategies are less practical during inference than training.

II. RELATED WORK

  • The related-work section surveys carbon-offset estimation, energy measurement, prior LLM power studies, and vLLM as a serving backend.Source: §II · p. 1

A. Carbon Offset

  • Carbon offset can be estimated from power consumption and local grid carbon intensity, with PUE accounting for additional data-center infrastructure power.The text represents greenhouse-gas effects as CO2eq.Source: A. Carbon Offset · pp. 1–2
  • Changing data-center location or operating hours can reduce emissions without reducing power use, but these strategies are less useful for inference.Inference server location choices face privacy, cost, and latency issues, and services require continuous availability unlike batch training.Source: A. Carbon Offset · p. 2

B. Energy consumption measurement

The reviewed literature describes component-level energy measurement tools and limitations in existing LLM efficiency studies. Those studies often use lab setups and energy-per-token metrics rather than modern serving technology.

B. Energy consumption measurement

  • Hardware component power can be measured with power meters or embedded sensors, accessed through tools such as NVIDIA-smi and powermetrics.Source: B. Energy consumption measurement · p. 2
  • System-energy tools combine low-level interfaces such as RAPL with power measurement, while carbon estimates depend on local grid-intensity data.The passage notes that estimation accuracy varies with local transparency.Source: B. Energy consumption measurement · p. 2

C. LLM Efficiency measurements

  • Prior LLM efficiency work often uses parameter size as an estimate, although reported research finds no correlation across architectures and a sub-linear relation within one architecture.These studies also use energy-per-token metrics, whose values may be affected by tokenizer-specific token sizes.Source: C. LLM Efficiency measurements · p. 2
  • Existing studies use lab experiments and PyTorch or HuggingFace Transformers scripts, which provide fine-grained control but omit modern serving optimizations.Source: C. LLM Efficiency measurements · p. 2

D. LLM serving

The paper describes vLLM as a production-oriented serving backend that uses hardware efficiently. The methodology section covers benchmark design, experimental parameters, and component-level power measurement with CodeCarbon.

D. LLM serving

  • vLLM supports high-throughput, low-latency model serving through efficient hardware use and paged attention, which makes memory management more flexible.Paged attention stores values in non-contiguous sequences.Source: D. LLM serving · p. 2

III. METHODOLOGY

  • The methodology covers benchmark design, energy-measurement strategy, and experimental test parameters.Source: §III · p. 2

A. Measurement strategy

  • CodeCarbon estimates component power from low-level sensor readings taken every 15 seconds, multiplying voltage measurements by the time interval.Source: A. Measurement strategy · p. 2

B. Benchmark design

The benchmark runs inference with vLLM and varies model, request amount, and request rate. Its tests use HellaSwag phrases on specified hardware, without evaluating accuracy.

B. Benchmark design

  • vLLM was selected for inference because of its high performance and open-source nature.The benchmark design uses it to execute the tests.Source: B. Benchmark design · p. 2
  • Each run varies the model, request amount, and request rate; tests send all requests simultaneously without a request rate.Before each run, 200 warm-up requests are sent, and CodeCarbon tracks energy until all requests return.Source: B. Benchmark design · pp. 2–3

C. Test parameters

  • The tests run on a PC with two NVIDIA GeForce RTX 3090 GPUs, an Intel Core i9-13900K, and 128 GB of DDR5 RAM.Source: C. Test parameters · p. 3
  • Tests use HellaSwag phrases, but the model’s results are not collected because accuracy is outside the experiment’s focus.HellaSwag supplies incomplete phrases for the model to complete.Source: C. Test parameters · p. 3

1) Request load:

The evaluation varies simultaneous request load and compares model sizes and architectures. Pythia models provide a basis for isolating parameter amount, while architecture comparisons use similarly sized models.

1) Request load:

  • Request loads range from 5 to 5,000 simultaneous requests to examine how model efficiency changes with load.Source: 1) Request load · p. 3
  • Model size is measured by parameter count, which affects inference operations and the memory needed to load a model.Source: 1) Request load · p. 3
  • The selected Pythia models are intended to isolate parameter amount rather than optimize each model size.Table I lists the selected Pythia models.Source: 1) Request load · p. 3, Table I · p. 3
  • Architecture comparisons use models of the same parameter size, selected to enable comparison with an earlier study.Table II lists the selected models for architecture testing.Source: 1) Request load · p. 3, Table II · p. 3

IV. RESULTS

The results examine request amount, model size, and architecture. Across the tested models, energy per request falls as simultaneous requests rise, then levels off at 100.

IV. RESULTS

  • The results address three factors influencing energy consumption: request amount, model size, and model architecture.Source: §IV · p. 3

A. Request amount

  • The request-load results report GPU consumption per request in joules against the number of simultaneous requests sent to vLLM.Source: A. Request amount · p. 3
  • Energy per request decreases as simultaneous requests increase, then plateaus at 100 for both the Pythia suite and different architectures.Figure 1 covers Pythia models from 70 million to 6.9 billion parameters; Figure 2 covers models close to 3B parameters.Source: A. Request amount · pp. 3–4, Figure 1 · p. 4, Figure 2 · p. 4

B. Model size

Energy per request rises with model size, approximately linearly in most cases, while comparable-sized models of different architectures show no significant efficiency differences.

B. Model size

  • Energy per request increases with parameter size, with a close-to-linear relationship for most models.The 410-million-parameter model is an exception, consuming roughly the same energy per request as the 1-billion-parameter model.Source: B. Model size · p. 4, Figure 3 · p. 4

C. Model Architecture

  • Models close to 3B parameters show no significant GPU energy-per-request differences across architectures.Figure 4 compares four similarly sized models.Source: C. Model Architecture · p. 4

V. DISCUSSION

The discussion finds that energy efficiency declines close to linearly with parameter size within an architecture, but comparable-sized architectures have no significant efficiency differences. It also notes that vLLM and the per-request metric make the benchmark more production-like while complicating comparisons with earlier work.

V. DISCUSSION

  • Model efficiency decreases close to linearly with parameter size within the same architecture, except that the 410-million- and 1-billion-parameter models have roughly equal efficiency.The discussion attributes this parity possibly to layer counts: 410 million parameters has 24 layers, compared with 16 for 1 billion; sequential layer processing limits parallelization.Source: §V · pp. 4–5
  • Comparable-sized architectures show no significant efficiency effect here, unlike Argerich et al.’s reported 47% gain between the least and most efficient models in the category.The studies differ in metric and serving library: this paper uses joules per sent request and vLLM, while Argerich et al. use joules per token and Huggingface Transformers,,.Source: §V · p. 5
  • Whether vLLM explains the architecture result remains unclear, and the paper calls for further research.The authors suggest vLLM’s serving optimizations could make architecture-related efficiency gains negligible, but state that no clear explanation has been established.Source: §V · p. 5

A. Validity

  • Using vLLM better reflects modern serving, but its real-time optimizations introduce variables that complicate attribution of measured efficiency to models rather than the backend.The authors describe automatic batching, speculative decoding, and GPU acceleration as backend optimizations.Source: A. Validity · p. 5
  • The study measures energy per request rather than the more common energy per token, which can vary in consistency across models.Tokenizers can encode the same text differently, and models may produce replies of different lengths.Source: A. Validity · p. 5

B. Limitations

The study omits accuracy, and its future-work agenda calls for broader model and serving-system tests and benchmarks that examine energy alongside accuracy.

B. Limitations

  • Accuracy is the study’s most prominent omitted parameter, and its relevance depends on the developer’s use case.The authors note that different accuracy measures may matter for different applications.Source: B. Limitations · p. 5

C. Future work

  • Future work should test more LLMs and a wider variety of serving techniques, including TGI and TensorRT,.The authors recommend larger tests to further validate findings about model size and architecture.Source: C. Future work · p. 5
  • Future benchmarks should measure energy efficiency alongside accuracy because lower energy does not necessarily imply higher accuracy.The authors note that optimized lower-parameter models can outperform larger models on specific accuracy benchmarks.Source: C. Future work · pp. 5–6

VI. CONCLUSION

The LLM Efficiency Benchmark shows that benchmarks can simulate realistic LLM serving scenarios; the authors plan to build an energy-efficiency database to support developers selecting models for production.

VI. CONCLUSION

  • The LLM Efficiency Benchmark demonstrates that an energy-efficiency benchmark can simulate realistic LLM serving scenarios.The authors plan to create an extensive LLM energy-efficiency database to support developers selecting more energy-conscious models for production.Source: §VI · p. 6
Loading 2509.08867v1…