PaperlayerView exact revision on arXiv

Source-linked AI summary

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

arXiv:2604.09048v1cs.DCcs.AI

TL;DR

System operators lack broad energy-aware evidence for choosing hardware for LLM inference across heterogeneous GPUs and deployment scenarios. Watt Counts provides an open benchmark and dataset and uses them to study inference across models and GPUs. The study finds that efficient hardware choices vary by model and scenario, with reported energy reductions of up to 70% in server and up to 20% in batch scenarios.

  • Problem

    Open evidence for systematically comparing LLM inference energy across diverse models, heterogeneous GPUs, and batch and server scenarios is lacking.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    Watt Counts combines an open energy-aware benchmark and dataset with a system-level study of LLM inference across heterogeneous GPUs.

    Source: §1 · p. 1

  • Results

    GPU energy-efficiency rankings vary by model and deployment scenario; batch efficiency is shaped mainly by throughput and TDP, while server efficiency is dominated by idle and maximum power.

    Source: §7 · pp. 10–11

  • Takeaways & Limitations

    Hardware-aware placement matters because energy-efficient GPU choices differ across models and deployment scenarios.

    Source: §7 · p. 10

    The evaluation is limited to single-GPU deployments; multi-GPU system effects may alter the energy-efficiency rankings observed.

    Source: §6 · p. 9

Abstract

from arXiv · show

While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.

1. Introduction

Watt Counts addresses the lack of system-level evidence and tools for comparing energy trade-offs across LLMs, heterogeneous GPUs, and deployment scenarios. It combines an open benchmark and dataset with a system-level study to support hardware-aware deployment decisions.

1. Introduction

  • System-level evidence and tools are lacking for comparing energy-performance trade-offs across LLM characteristics, GPU architectures, and deployment scenarios.GPUs can account for up to 90% of total system power draw in AI inference,.Source: §1 · p. 1
  • The open dataset covers 50 LLMs on ten GPUs across five NVIDIA architectures, with 5K+ experiments on 370 LLM-GPU pairs.It also contains 14M+ power-draw samples and multiple energy and performance metrics.Source: §1 · p. 1
  • Watt Counts provides a customizable, open-source benchmark for reproducible energy and performance measurement in bare-metal and cloud instances without external power meters.The benchmark is designed to let users contribute to a live dataset.Source: §1 · p. 1
  • The system-level study finds that GPU, model, and deployment-scenario properties jointly shape energy-efficiency and performance trade-offs.The dataset and study are intended to support deployment decisions that leverage heterogeneous GPU architectures.Source: §1 · pp. 1–2
  • Up to 70% less energy in server scenarios with negligible user-experience impact, and up to 20% less in batch scenarios, is reported as achievable.These are the stated reductions from applying the paper’s data and insights.Source: §1 · p. 2

2. Related Work

Prior work measures LLM energy, estimates emissions or energy, and benchmarks selected workloads, but the available evidence does not span diverse models, heterogeneous GPUs, and both batch and server scenarios. Watt Counts addresses this coverage gap with an open dataset and benchmark.

Energy and carbon measurement studies during training and inference

  • Prior measurement work covers training and inference, while MLPerf Power reports measurements across ML workloads and TokenPowerBench focuses on 15+ open-source models on H100 clusters in batch scenarios.TokenPowerBench’s dataset and benchmark implementation were not public at the time of writing.Source: §2 · p. 2

Tools and benchmarks for energy estimation

  • Energy-estimation tools and workload analyses cover varied settings, but many optimization studies focus on specific GPUs and models; Watt Counts offers empirical data across a wider hardware space.The cited approaches include software estimators, an energy-efficiency leaderboard, and modeling work tested on three GPUs and seven LLMs.Source: §2 · p. 2

Energy-aware datasets for LLM inference

  • Existing datasets lack open, systematic coverage of diverse models, heterogeneous GPUs, and both batch and server scenarios, which Watt Counts is designed to provide.Some studies cover a narrow GPU range or batch scenarios, while MLPerf Power reports aggregate system-level rather than GPU-level energy measurements.Source: §2 · p. 2

3. Benchmarking Methodology

The benchmark uses question-answering workloads in batch and server modes, collects energy and performance metrics, and is designed for reproducible and modular execution. Its research questions and measurement choices target comparisons across GPU, model, and deployment characteristics.

3.1. Research Questions

  • The research questions examine how GPU architecture, LLM properties, and batch-versus-server deployment relate to inference energy efficiency.They ask whether GPU rankings vary across LLMs, which model properties matter, and whether optimal LLM-GPU pairs change by scenario.Source: §3.1 · p. 3

3.2. Benchmark Design

  • The benchmark evaluates question-answering inference in batch and server scenarios, guided by reproducibility, modularity, and practical relevance.It is open-source and publicly available for community adoption and extension.Source: §3.2 · p. 3

3.2.1. Deployment Scenarios

  • Batch mode submits 1,000 SQuAD v1.1 queries synchronously; server mode sends asynchronous SQuAD requests at Poisson arrival rates λ=0.017 and 0.3.Batch energy covers only the finite workload, whereas server requests model low- and high-load conditions.Source: §3.2.1 · p. 4

3.2.2. Energy Metering

  • GPU energy is measured with NVML at board level; 10 Hz sampling produced less than 1% integrated-energy error versus 200 Hz, with less than 0.5% execution-time overhead.In validation, the GPU accounted for 91.2–92.5% of three-component energy in batch and high-load server runs, and 78.7% in low-load server runs.Source: §3.2.2 · pp. 4–5

3.2.3. Energy Metrics

  • Batch comparisons use energy per token, while server comparisons use mean power draw to account for idle and active periods.The paper notes that server energy per token depends on request arrival rates and idle-power assumptions.Source: §3.2.3 · p. 5

3.2.4. Additional Performance Metrics

  • The benchmark collects system and LLM performance measures, including throughput, TTFT, and response-time percentiles, but excludes task-accuracy evaluation.Those performance indicators support balancing energy use against service-level requirements; task correctness is outside the benchmark’s scope.Source: §3.2.4 · p. 5

4. Dataset Construction

The dataset is built to compare LLM energy and performance across diverse GPU architectures, with model and hardware selections shaped by compatibility and deployment-relevant variation.

4. Dataset Construction

  • The dataset includes 50 open-source models up to 30B parameters, chosen to fit on at least two evaluated GPUs for cross-GPU comparison.The study focuses on single-GPU setups to avoid variability from communication, synchronization, sharding, and interconnect effects.Source: §4 · p. 5
  • The selected NVIDIA GPUs span five architectures, with memory capacities from 16GB to 141GB and TDP from 70W to 600W.The hardware set includes server-class and consumer-grade devices.Source: §4 · p. 5

5. Dataset Analysis and Insights

Across 5,000+ experiments, the analysis finds that energy efficiency depends on the LLM, GPU, and deployment scenario. It identifies hardware-aware placement as important for balancing energy use, throughput, and response time.

5. Dataset Analysis and Insights

  • The analysis uses more than 5,000 experiments across 50 LLMs and 10 GPUs; batch data is used for GPU and model-property comparisons, while server analysis accounts for idle periods.Mean idle time is 2% in batch, compared with 53% at high server load and 96% at low load.Source: §5 · p. 5, §5.1 · pp. 5–6, §5.3 · p. 8

5.1. RQ1: GPU Architecture Efficiency Across LLMs

  • Throughput and energy efficiency do not align consistently: the H200 has the highest mean throughput, while TFLOPS per Watt does not reliably predict lower energy per token.For an Xlarge model, an H100 can save 20% energy relative to the H200, which would be selected to optimize throughput.Source: §5.1 · pp. 6–7
  • The H100 has the lowest batch energy per token in 90% of models, but GPU rankings vary with model size and no GPU is universally most efficient.The H200 ranks second for most models; the L4 leads for three models, including two of the smallest, while low-TDP A30 and L4 GPUs can suit smaller models.Source: §5.1 · pp. 6–7, Table 3 · p. 6

5.2. RQ2: LLM Efficiency Across GPUs

  • Model size is the dominant energy factor, while key-value and attention head counts increase energy and wider hidden dimensions reduce it, conditional on size.The size coefficients are 0.237 and 0.282 (both p<0.001); key-value head count has coefficient 0.121 (p<0.001), and hidden size −0.098 (p=0.026).Source: §5.2 · pp. 7–8

5.3. RQ3: Deployment Scenario Sensitivity

  • Server GPU rankings differ by load: high load favors the L4 and T4, while low load favors the T4 and A30, with idle and maximum power driving these results.Throughput remains relevant because faster processing reduces active time, but its effect is secondary to idle and maximum power; TDP is a poor proxy for server power draw.Source: §5.3 · pp. 8–9, Table 4 · p. 9
  • Up to 70% energy savings are possible for a medium-sized Llama 3.1 8B under low load using an A30 rather than an H200, with TTFT below 70 ms.The same passage reports up to 60% savings versus an H100 in that scenario; for large models under high load, an A100 saves 10% versus an H100.Source: §5.3 · p. 9, Table 5 · p. 9

6. Discussion & Limitations

The study is bounded to single-GPU deployments, 16-bit weights, and vLLM; the authors note that quantization or other inference engines may change results. Accuracy and whole-host energy are outside the evaluation, and the paper reports the experiments’ own energy use and emissions.

Discussion & Limitations

  • The analysis covers single-GPU deployments, avoiding communication, synchronization, and interconnect confounds; multi-GPU effects could alter the observed energy-efficiency rankings.The authors note that the insights directly apply to edge-cloud and resource-constrained deployments.Source: §6 · p. 9
  • All evaluated models use 16-bit weights; quantization could shift GPU rankings and model-property coefficients, especially where memory bandwidth is the primary constraint.The authors expect qualitative insights to generalize, but specific rankings and coefficients may differ for quantized workloads.Source: §6 · p. 10
  • The study uses vLLM, so results may differ with TensorRT-LLM or SGLang, particularly for GPU-model combinations affected by engine-specific optimizations.The authors chose vLLM so observed trade-offs primarily reflect architecture and workload rather than implementation inefficiencies.Source: §6 · p. 10
  • Task accuracy is out of scope, and optimizing deployments solely for energy may trade off output quality.The authors identify joint optimization of energy, latency, and accuracy as a future framework goal.Source: §6 · p. 10
  • 78.7–92.5%: GPU energy is dominant and correlates 0.995-1.0 with total energy, but host components and cooling are unmeasured; cooling may add 30–50%.The reported cooling contribution depends on facility power usage effectiveness.Source: §6 · p. 10
  • 180 kWh: the experiments used approximately this much energy and had estimated emissions of 17 kgCO2; most ran on servers sourced with a 99% share of renewable energies.The authors consider the potential savings from reduced redundant experimentation and better deployment decisions to outweigh this one-time footprint.Source: §6 · p. 10

7. Conclusion

Watt Counts reports that energy-efficient GPU choices vary with model size and deployment scenario, and that model architecture and service-level requirements also shape inference trade-offs. It releases a dataset intended to support future work on heterogeneous deployments, scheduling, and benchmarking.

7. Conclusion

  • Watt Counts provides an open dataset spanning 50 models and 10 heterogeneous GPUs in batch and server scenarios, released to support future work on heterogeneous deployments, scheduling, and benchmarking.Source: §7 · pp. 10–11
  • GPU energy-efficiency rankings vary with model size and deployment scenario: H100 leads for most batch models, while L4 and A30 outperform it for small models, and server rankings differ.These findings highlight the importance of hardware-aware model placement.Source: §7 · p. 10
  • Model size is the primary factor in energy consumption, but depth, hidden size, and key–value head configuration also matter; TFLOPS per Watt poorly predicts full-workload system energy efficiency.The paper identifies memory bandwidth as a main inference bottleneck.Source: §7 · p. 11
  • Within the evaluation scope, throughput and TDP are the main energy-efficiency factors in batch, while idle and maximum power dominate in server deployments.Manufacturers rarely disclose idle and maximum power, reporting TDP instead.Source: §7 · p. 11
  • Energy efficiency must be considered alongside service-level requirements such as response time, because low-power GPUs might have response times that limit practical use.Source: §7 · p. 11

9. Use of Generative AI

The authors used generative AI tools to refine language and clarity, reviewed and edited the suggestions, and take responsibility for the paper.

9. Use of Generative AI

  • The authors used Claude and ChatGPT to assist with language refinement and clarity.Source: §9 · p. 11
  • The authors carefully reviewed and edited all AI-generated suggestions.Source: §9 · p. 11
  • The authors state that they take full responsibility for the accuracy, integrity, and originality of the published work.Source: §9 · p. 11

A. Full List of Evaluated Large Language Models

Table 11 lists the evaluated models and classifies them by model category as discussed in Section 5.1.

A. Full List of Evaluated Large Language Models

B. Validation of GPU as the Primary Energy Consumer

The appendix measures CPU, DRAM, and GPU energy on one bare-metal instance and reports component energy shares and correlations by deployment scenario. GPU energy accounts for 92.5–78.7% of total energy across scenarios and has the highest correlation in all cases.

B. Validation of GPU as the Primary Energy Consumer

  • EnergyMeter measures CPU and DRAM energy with Intel RAPL and GPU energy with NVML on a bare-metal instance with an Intel Core i9-9900K, 32 GB RAM, and one RTX 4090.Source: B. Validation of GPU as the Primary… · p. 12
  • Table 7 reports mean energy share and Pearson correlation with total system energy by component and deployment scenario.Source: Table 7 · p. 13
  • GPU accounts for 92.5–78.7% of total energy consumption across scenarios and has the highest correlation with total energy in every case.Source: B. Validation of GPU as the Primary… · p. 13

C. Outlier Analysis: Energy per Token vs. Model Size

The energy-per-token outliers include older-GPU cases and two model-specific exceptions on newer hardware, with proposed explanations tied to hardware support, architecture, and memory pressure.

C. Outlier Analysis: Energy per Token vs. Model Size

  • Most outliers run on V100 or T4 GPUs, whose limited memory bandwidth and reduced support for modern kernels are linked to disproportionately high energy per token.The passage identifies these as structural patterns among the outliers.Source: C. Outlier Analysis: Energy per Token… · p. 13
  • SmolLM2-1.7B-Instruct is an outlier on L40S, RTX 3090, T4, and V100, but performs within expectations on newer Ada Lovelace and Hopper GPUs.The authors hypothesize device-specific kernel inefficiencies on the former GPUs.Source: C. Outlier Analysis: Energy per Token… · p. 13
  • SOLAR-10.7B-Instruct-v1.0 is an outlier on RTX 3090, RTX 4090, and A30; its depth-upscaling architecture duplicates transformer layers and increases energy consumption.The authors also hypothesize that 24GB GPU memory limits and the model's 10.7B parameters—whose weights alone are estimated to use 21GB—create memory pressure and degraded throughput, increasing energy per token.Source: C. Outlier Analysis: Energy per Token… · p. 13

D. Full Mixed-Effects Model Results for LLM Properties and Energy Per Token

The section reports mixed-effects analyses of energy per token and presents results alongside measurements of energy, throughput, power, and TTFT across GPUs, model sizes, and scenarios.

D. Full Mixed-Effects Model Results for LLM Properties and Energy Per Token

  • Model A analyzes log energy per token using log parameters and model type, while Model B adds residualized hidden size, layer count, attention heads, and key-value heads; GPU is a random intercept in both.The reported log-likelihoods are 55.81 for the first model and 104.02 for the second; both models converged.Source: Table 9 · p. 13, D. Full Mixed-Effects Model Results… · pp. 13–14, Table 10 · p. 14
  • Each model's data points are means over five repeated runs for each LLM–GPU–scenario configuration to reduce measurement noise.Model A focuses on active parameters, while Model B includes residualized architectural features.Source: D. Full Mixed-Effects Model Results… · p. 13
  • The energy-per-token and throughput figures compare GPUs across model-size categories, with missing bars marking categories no GPU could run because of memory or compatibility limits.The energy-per-token figure shows 95% confidence intervals; the throughput figure also reports TFLOPS performance.Source: Figure 2 · p. 14, D. Full Mixed-Effects Model Results… · pp. 14–15, Figure 3 · p. 15
  • Additional figures show a log–log energy-per-token outlier envelope, power draw by GPU and scenario, and mean GPU power draw against 95th-percentile TTFT.The regression's gray region is a 95% prediction interval; the power-draw plot separates batch, server-low, and server-high load and reports TDP and idle power for reference.Source: Figure 4 · p. 15, Figure 5 · p. 16, Figure 6 · p. 16
  • The evaluated set comprises 50 LLMs grouped by size category, with MoE models listed separately and GPU coverage indicated for each model.Size categories are based on minimum GPU memory required; models are sorted by parameter count within groups.Source: Table 11 · p. 17
Loading 2604.09048v1…