PaperlayerView exact revision on arXiv

Source-linked AI summary

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

arXiv:2607.02391v1cs.DCcs.LG

TL;DR

Operators lack a practical way to identify efficient LLM-GPU combinations without profiling each candidate, limiting evidence for deployment choices. WattGPU predicts power draw and ITL from public LLM and GPU data and is evaluated on unseen GPUs and LLMs. Its models achieve low prediction errors on unseen GPUs and outperform analytical baselines in server scenarios.

  • Problem

    Identifying efficient LLM-GPU deployment pairs requires costly profiling, while existing approaches have limited evaluation on unseen GPUs.

    Source: §1 · p. 1

  • Method

    WattGPU uses two predictive models for mean GPU power draw and ITL, relying on public LLM metadata and GPU specifications rather than target-pair profiling.

    Source: §1 · p. 2

  • Results

    On unseen GPUs, mean power draw MdAPE is ≤3.4% offline and ≤13.5% server, ITL MdAPE is ≤8.5% server, and server GPU rankings reach Kendall τ≥0.76.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The predictions support energy-aware GPU or model selection, capacity planning across heterogeneous hardware, and advance deployment energy estimates.

    Source: §6 · p. 9

    Offline ITL prediction is harder than server ITL prediction (LOGO MdAPE 24.9% vs. 8.5%), and evaluation is limited to dense models in single-GPU configurations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce \textit{WattGPU}, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of $\leq3.4\%$ for offline and $\leq13.5\%$ for server scenarios on unseen GPUs, while the latency model achieves $\leq8.5\%$ in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall $\tau\geq0.76$). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4$\times$ on unseen LLM-GPU combinations for server scenarios or approximately 2$\times$ for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.

1. Introduction

The introduction argues that choosing the right GPU can reduce LLM inference power, but finding suitable LLM-GPU pairs currently requires costly profiling. WattGPU predicts mean power draw and inter-token latency from public metadata, and is evaluated on unseen GPUs and LLMs.

1. Introduction

  • GPU choice is a consequential energy lever: for Llama 3.1 8B at low load, an A30 can reduce power draw by up to 43% versus an H100.The example concerns deployments where the model fits in GPU memory.Source: §1 · p. 1, Figure 1 · p. 2
  • Finding efficient LLM-GPU pairs currently requires exhaustive profiling, physical hardware access, specialized expertise, and significant resources that smaller operators often lack.The introduction notes that smaller deployments are less optimized and their aggregate energy footprint is growing.Source: §1 · pp. 1–2
  • WattGPU predicts mean GPU power draw and ITL using publicly available LLM metadata and GPU specifications, without profiling or hardware access.The introduction presents separate predictive models for the two metrics and makes their training and evaluation pipelines open source.Source: §1 · p. 2
  • On unseen GPUs, mean power draw error is ≤3.4% offline and ≤13.5% server, while server ITL error is ≤8.5%; server GPU rankings reach Kendall τ≥0.76.Against the strongest baselines, median absolute percentage error falls by approximately 4× on unseen LLM-GPU combinations and 2× on completely unseen GPUs in server scenarios.Source: §1 · p. 2

2. Related Work

Prior measurement and prediction approaches provide useful inference energy or performance estimates, but typically rely on profiling and offer limited evidence of generalization to unseen GPUs or models.

2. Related Work

  • Existing measurement and prediction approaches generally require profiling the target LLM-GPU pair, and their generalization to unseen GPUs has not been evaluated.This limits their usefulness for choosing among deployment options that have not yet been profiled.Source: §2 · p. 3
  • Measurement efforts include instrumented studies and leaderboards and TokenPowerBench, but still require meters or profiling and cannot estimate unprofiled hardware or models.Krupp et al. [12] instead infer GPU use from API inference time to estimate energy when the hardware is unknown.Source: §2 · p. 3
  • Predictive energy and latency models use learned, analytical, or simulation-based approaches [16] [18] [21], but often depend on profiling data or have narrow validation scope.Examples include single-GPU evaluation, H100-only estimates, and validation on one unseen LLM with known GPUs [21].Source: §2 · p. 3

3. Background

The background defines offline and server inference settings and the power and latency metrics used to characterize them. It treats server efficiency as a multi-objective problem rather than combining mean power and ITL into one per-token measure.

3. Background

  • The paper frames its background around inference scenarios, energy and latency metrics, and the dataset used for training and evaluation.The supplied scenario passages describe offline and server operation; the dataset is introduced as part of the section but its details are not included here.Source: §3 · p. 3

3.1. Inference Scenarios

  • Offline inference processes many concurrent requests at maximum throughput, whereas server inference handles arriving requests under latency requirements, with load affecting GPU utilization and energy per token.Server requests may follow a load pattern such as a Poisson distribution.Source: §3.1 · pp. 3–4

3.2. Energy Metrics

  • Mean power draw is average GPU power during inference, while ITL is the average time between consecutive generated tokens; server ITL excludes TTFT, whereas offline ITL amortizes prefill across generated tokens.Because server mean power includes idle and busy intervals while ITL describes active generation, the paper treats server efficiency as a multi-objective problem.Source: §3.2 · p. 4

4. Method

WattGPU uses regression models and public LLM and GPU information to predict power draw and ITL without runtime profiling. It engineers hardware- and model-derived features, normalizes power targets by GPU power limits, and selects XGBoost models through cross-validation.

4. Method

  • WattGPU implements separate regression models for mean power draw and ITL using public data, avoiding runtime metrics and enabling pre-deployment estimates without profiling runs.The models predict characteristics of LLM-GPU pairs.Source: §4 · p. 4

4.1. Feature Engineering and Selection

  • Features combine manufacturer GPU specifications and Hugging Face LLM architecture metadata, with derived features for clock boosting, weight-reading time, and theoretical compute latency.The selected features were refined using LOGO and LOLO performance and are summarized for both models in Table 1.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2. Model Selection and Training

  • The power target is normalized by GPU TDP or configured power limit to learn scale-invariant behavior; XGBoost performed best among evaluated algorithms during LOGO cross-validation.Table 2 lists the NVIDIA GPUs and technical characteristics used in the study.Source: §4.2 · p. 5, Table 2 · p. 6

5. Experimental Evaluation

WattGPU is evaluated on a filtered public inference dataset using three validation splits that test known pairs, unseen GPUs, and unseen LLMs. It improves absolute prediction accuracy over analytical baselines, while offline ITL and LLM power ranking remain challenging.

5. Experimental Evaluation

  • The evaluation uses grouped 5-fold CV for unseen GPU–LLM combinations, LOGO for unseen GPUs, and LOLO for unseen LLMs.It reports MdAPE, Pearson r, and GPU and LLM Kendall τ; the Kendall measures assess ranking quality.Source: §5 · p. 6

5.1. The Watt Counts Dataset

  • The Watt Counts dataset contains measurements for 50 LLMs on 10 NVIDIA GPUs, with offline and server scenarios and two server request rates.The study retains 8 server-grade GPUs and 42 dense LLMs (125M–27B parameters), excluding two consumer GPUs and leaving eight MoE models for future work.Source: §5.1 · p. 6

5.2. Mean Power Draw Prediction

  • XGBoost achieved MdAPE of 1.6% offline and 5.5% server in CV, compared with Load-Scaled TDP’s 26.0% server MdAPE.The plain TDP baseline’s server MdAPE was 190.1%; Table 3 reports XGBoost and TDP baseline performance.Source: Table 3 · p. 7, §5.2 · p. 7
  • Offline LLM power ranking is difficult because near-full GPU utilization makes power draw approach TDP, with small-model exceptions on H100 and H200.The model reaches LLM Kendall τ=0.33 in this case, versus τ=0 for random ranking; Figure 2 shows power draw distributions across LLMs, GPUs, and scenarios.Source: §5.2 · p. 7, Figure 2 · p. 8
  • For power prediction, scenario is the most important feature, followed by memory bandwidth and bandwidth latency; GPU features outweigh LLM features apart from scenario.For ITL, bandwidth latency is most important, ahead of scenario; adding the roofline estimate as a feature did not improve performance.Source: §5.2 · pp. 7–8, Figure 3 · p. 8, §5.3 · p. 9

5.3. ITL Prediction Results

  • On CV, for unseen (GPU, LLM) combinations, our approach reduced MdAPE from the roofline baseline’s 80.2% to 12.8% offline and 29.6% to 4.9% server, with higher GPU Kendall τ.Source: §5.3 · p. 8
  • Offline ITL remained difficult: LOGO MdAPE=24.9%, GPU τ=0.72, LLM τ=0.83; LOLO MdAPE=15.6%, GPU τ=0.97, LLM τ=0.66.Generalization to unseen GPUs and LLMs increases MdAPE and decreases ranking capability; the roofline model can still rank well, while the learned model captures additive overheads.Source: §5.3 · p. 9

6. Discussion & Future Work

Across 42 dense LLMs and 8 server-grade GPUs, WattGPU predicts mean power draw accurately and ranks GPUs well on unseen hardware, while server-scenario ITL estimates and rankings are reliable. Offline ITL prediction remains harder, and the evaluation is limited to dense models in single-GPU configurations.

6. Discussion & Future Work

  • Across 42 dense LLMs and 8 server-grade GPUs, the power model ranks unseen GPUs well, while ITL estimates and GPU and LLM rankings are reliable for server workloads.The evaluation dataset comprises 42 dense models, as listed in Table 5.Source: §6 · p. 9, Table 5 · p. 10
  • Offline ITL prediction remains harder than server prediction, with LOGO MdAPE of 24.9% versus 8.5%.The authors attribute the gap to static features’ limited capture of vLLM continuous batching and suggest workload-aware features as future work.Source: §6 · p. 9
  • The evaluation covers dense models in single-GPU configurations; MoE, quantized models, and multi-GPU deployments require separate modeling and remain future work.Workload-aware features would need to be operator-assumed or estimated from static data to preserve generalization.Source: §6 · p. 9

7. Conclusion

WattGPU predicts mean GPU power draw and inter-token latency for LLM-GPU pairs from public data without target-pair profiling. Its predictions support placement decisions, including an example where an A30 uses less power than an H100 while meeting an ITL constraint; MoE and quantized support remains future work.

7. Conclusion

  • On unseen GPUs, mean power draw MdAPE is ≤3.4% offline and ≤13.5% server, while server ITL MdAPE is ≤8.5%; offline ITL is the hardest case at 24.9%.The models were tested across 42 LLMs and 8 GPUs from different architectures.Source: §7 · p. 10
  • WattGPU supports energy-aware GPU and model selection, capacity planning, and deployment energy estimates using predictions made before target-pair profiling.Future work includes supporting MoE and quantized models and integrating the models into workload placement and request routing.Source: §7 · pp. 10–11
  • For low-load Llama 3.1 8B with a 25 ms ITL SLA, the A30 draws 43% less power than the H100 among compliant GPUs.The L4 violates the SLA despite its 72 W TDP, illustrating why TDP heuristics can mislead GPU selection.Source: §7 · p. 11
Loading 2607.02391v1…