PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton, John Paul Shen

arXiv:2608.28044v1cs.PFcs.DCcs.LG

TL;DR

Token pricing does not capture GPU energy consumed over inference windows, so token-normalized metrics can fall even as total request energy rises. The paper develops a fixed-plus-step energy model and evaluates request and token energy across request shapes on H100 and H200 GPUs. It finds that output length and batching can lower token energy through amortization while increasing total energy, with context-bounded batching gains and stronger MoE sensitivity.

  • Problem

    Token-normalized energy metrics do not reveal whether lower J/token reflects reduced GPU energy or amortization of request-window energy.

  • Method

    The paper decomposes inference-window energy into fixed prefill and generation-setup energy plus marginal energy for each output-token step, evaluating H100 and H200 GPUs across request-shape parameters.

  • Results

    Longer outputs and larger batches can reduce token energy by spreading fixed energy across more tokens while increasing total request energy; batching gains shrink with context length and MoE models are especially sensitive.

  • Takeaways & Limitations

    Energy-aware serving should jointly optimize request energy and token energy rather than relying only on per-token energy cost.

Abstract

from arXiv · show

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

I. INTRODUCTION

LLM inference is priced and evaluated in token-visible terms, while GPU energy is consumed across inference windows. The paper therefore decomposes request energy and characterizes when lower token energy reflects amortization rather than lower total energy.

  • Motivation: Token-normalized energy can conceal whether lower J/token reflects reduced GPU energy or fixed inference-window energy spread across more tokens.The service-visible unit is the generated token, whereas the hardware-visible energy unit is the inference window.
  • Request-shape abstraction: Request shape is represented by (M, P, B, C, N): model, phase, batch size, context length, and output length.These parameters describe the operating point and are visible to serving systems.
  • Key findings: Longer outputs and larger batches can reduce average token energy by amortizing fixed costs while increasing total request energy.Batching shares fixed costs across requests, while longer outputs amortize them across more generated tokens.
  • Energy model: Measured request energy is decomposed into fixed request-level energy plus marginal energy for each output-token generation step.The fixed term includes one-time prefill and fixed generation setup; each output-token step contributes marginal energy.
  • Key findings: Batching gains shrink with context length, very long outputs can become step-energy dominated, and MoE models are especially sensitive to batching.Routing, dispatch, and fragmented expert execution add overhead that can be amortized across the request shape.
  • Measurement and scope: The study combines NVML request-energy measurements with resident-floor and isolated-kernel replay measurements to attribute floor and kernel-active energy.This targets short, low-concurrency windows where floor energy and orchestration can dominate active kernel work.

A. LLM Inference Execution

LLM inference separates into prompt-processing prefill and sequential token-generation decode, while serving systems schedule these phases to manage latency and throughput. Prior studies likewise analyze energy across workload shape, hardware, runtime, and serving configuration.

  • LLM Inference Execution: Prefill processes the full prompt in one forward pass and populates the KV cache, whereas decode generates one token per sequential forward pass.Prefill exposes substantial token-level parallelism; decode depends on the KV state from previous steps.
  • LLM Inference Execution: PagedAttention enables continuous batching, DistServe separates prefill and decode workers, and Sarathi-Serve chunks long prefills alongside decode work.These systems primarily optimize latency and throughput, but their packing decisions also change inference request energy.
  • Related Work: Prior LLM energy studies report energy or power across language tasks, serving stacks, GPU generations, batch sizes, context lengths, engines, and parallelism.The cited work includes Words-to-Watts, Chen et al., and TokenPowerBench.
  • Related Work: Workload-shape and serving studies model per-LLM energy, identify nonlinear input/output-length regimes, and design heterogeneous-instance or frequency-scaling policies.These studies position energy as dependent on workload and system configuration rather than a static per-token quantity.
  • Related Work: Host orchestration, dispatch, CUDA-library, kernel-launch, and elevated-power-state effects can create fixed energy within an inference window.Such overheads are especially relevant when visible kernel activity is low.

C. This Work

This work characterizes request energy and token energy as distinct outcomes of LLM inference request shape. It combines a fixed-plus-step energy model with NVML measurements over complete inference windows.

  • C. This Work: Request energy and token energy are treated as separate metrics because longer outputs or larger batches can lower token energy while increasing total request energy.The study frames batching and output length as an energy-control surface rather than optimizing only token-normalized cost.
  • C. This Work: Measurements vary request shape and use cumulative NVML energy differences across inference-window boundaries as the primary GPU-energy metric.Power samples are retained as smoothed diagnostics because they cannot resolve short decode windows and host/runtime gaps as precisely.
  • C. This Work: The model represents complete batched-request energy as a fixed term plus marginal energy for each generated output token.The fixed term includes one-time prefill and generation setup, while the step term captures output-length-dependent generation energy.
  • C. This Work: Each configuration uses warmup and profiled runs, with energy measured over the final five profiled runs and normalized per inference.Mean latency is reported over profiled runs, while cumulative-counter energy is used to maintain tractable measurement runtimes.
  • C. This Work: Deterministic shapes are enforced with fixed random-token prompts, actual attended context lengths, disabled EOS termination, and exactly N generated tokens.The same input tensor is reused across warmup and measured iterations, and context is clamped to the model maximum.

D. Diagnostic Floor and Kernel Attribution

The study uses resident-floor and isolated-kernel replay measurements to interpret fixed and active energy components. Platform comparisons are system-level because the H100 and H200 configurations differ beyond the GPU memory subsystem.

  • D. Diagnostic Floor and Kernel Attribution: Resident-floor power is measured after model loading, runtime initialization, CUDA synchronization, and issuing no kernels.An idle cumulative-energy measurement provides a cross-check, with the energy-counter-derived value used as the primary resident-floor estimate.
  • D. Diagnostic Floor and Kernel Attribution: Replay reliability requires at least two effective windows and no more than 5% inter-window variation in energy-derived net power.Smoothed Hopper power readings are not treated as an independent consensus channel for the 500 ms replay windows.
  • D. Diagnostic Floor and Kernel Attribution: Isolated kernel replay estimates active energy from energy-counter-derived net power, mean production duration, and production dispatch count.Unique kernels are replayed in a child process after warmup and measured over repeated 500 ms windows.
  • D. Diagnostic Floor and Kernel Attribution: The diagnostic residual is computed as measured NVML energy minus floor and active energy, but it may be signed because production overlap prevents strict additivity.Because isolated replay cannot perfectly reproduce production inference windows, attribution supports interpretation rather than kernel-level redistribution.
  • D. Diagnostic Floor and Kernel Attribution: H100 and H200 results represent complete platform configurations rather than an isolated memory experiment.The systems differ in host CPU, GPU form factor, and nominal TDP in addition to memory capacity and bandwidth.

B. System Software

The evaluation uses dense and MoE models with specified software and expert-routing configurations, then studies output length, platform, phase, and attribution effects. It also extends the analysis to reasoning utility and energy.

  • B. System Software: Runs use Python 3.13, PyTorch 2.9.0+cu128, CUDA 12.6, Transformers 4.57.3, and default BFloat16 eager execution.FlashAttention-2 is enabled only for its corresponding sweep, while NVML measures request energy and torch.profiler collects traces.
  • B. System Software: The benchmark includes dense Llama models and MoE models with different expert organizations and routing schemes.OLMoE uses top-8 routing without shared experts, whereas Qwen1.5-MoE uses top-4 routing with four shared experts.
  • B. System Software: The experiments vary output length and batch size, compare H100 with H200, contrast prefill with decode, and use kernel attribution to interpret floor-dominated cases.The study closes with scheduler-facing implications before extending the analysis to capped reasoning budgets and utility per joule.
  • B. System Software: Figure 3 compares mJ/token across output lengths under low-load and high-load H200 operating points on a log-scaled y-axis.The corresponding latency analysis is presented separately in Figure 4.

A. Output Length Is a First-Class Energy Axis

Output length strongly affects token energy, but its effect depends on concurrency and workload regime. Longer outputs can amortize fixed request energy while total request energy still rises, with very long outputs eventually becoming step-energy dominated.

  • Low-load output-length scaling: 1.1× reduction in token energy occurs for Llama-3.2-1B at (B=1, C=512) over the same N sweep.The near-flat response reflects low-concurrency underutilization and limited amortization of fixed energy.
  • MoE scaling: MoE models show stronger token-energy reductions with increasing output length than the dense models in the low-concurrency regime.Routing, dispatch, and fragmented expert execution create more output-length-amortizable overhead.
  • High-load output-length scaling: 5.7× reduction in token energy occurs for Llama-3.2-1B at (B=16, C=2048) when N increases from 10 to 512.Token energy falls from 2,198 to 386 mJ/token even though the decode window consumes more total energy.
  • Energy decomposition: Erequest(N) = Efixed + NEstep fits complete inference-window energy over N ∈ {10, 128, 512}.The intercept estimates output-length-independent energy, while Estep is the incremental energy of one full-batch decode step.
  • Model scope: At N=10, the affine model can have substantial relative error, so it is used only as a local decomposition over the measured range.It is not evidence that marginal step energy remains constant outside that range.

4) Batch gain is context-bounded:

Batching reduces token energy by spreading fixed costs, but context growth increases step energy and weakens the gain. MoE models are especially sensitive to batching, while very long outputs can enter a step-energy-dominated regime.

  • Very long outputs: 337 mJ/token at N=128 rises to 1,600 mJ/token at N=8192 for Llama-3.2-3B/H200 at (B=16, C=512).This marks a second regime in which step energy overtakes the fixed component.
  • Dense–MoE contrast: At low load, OLMoE-1B-7B costs 22,476 mJ/token versus 2,431 mJ/token for active-parameter-comparable Llama-3.2-1B.The gap is consistent with routing, dispatch, and fragmented expert-execution overheads.
  • Dense–MoE contrast: At high load, OLMoE-1B-7B costs 3,440 mJ/token versus 2,198 mJ/token for Llama-3.2-1B, while Qwen1.5-MoE-A2.7B is 1.11× lower than Llama-3.2-3B.Batching narrows or reverses some dense–MoE comparisons, depending on the baseline.
  • MoE batching sensitivity: 12.6× token-energy reduction occurs for OLMoE-1B-7B when B increases from 1 to 16 at C=512 and N=10.The corresponding Llama-3.2-1B reduction is 6.3× at the same request shape.

1) H200 lowers token energy for dense models:

H200 generally lowers token energy for the measured dense models, but request shape remains the dominant structural influence. MoE results are model- and shape-dependent, so platform-specific calibration is still required.

  • Dense models: 0.61–0.84× of H100 token energy is reached by matched dense-model configurations on H200.For Llama-3.2-3B at B=1, C=512, energy drops from 6,741 to 4,093 mJ/token, a 1.65× reduction.
  • Platform behavior: 312.5 W on H100 versus 216.4 W on H200 is the median smoothed NVML power across plotted complete N=10 configurations.Request shape still determines active duration, shared requests, and KV-state access.
  • Figure scope: Fig. 5 compares H100 and H200 at N=10 across batch-size and context-length configurations.The short-output setting makes fixed request-level energy more visible when separating platform scaling from request-shape structure.
  • MoE models: 0.80× and 0.66× H200/H100 ratios occur for OLMoE-1B-7B and Qwen1.5-MoE-A2.7B at (B=1, C=2048), respectively.At B=16, C=512, OLMoE-1B-7B instead has a 1.22× ratio, showing that MoE direction is not universal.
  • Implication: H200 preserves the qualitative trends of expensive low-batch, low-output decode and context-eroded batching gains.Absolute energy still requires per-platform scheduling calibration.

2) Additional generation dominates long-output request energy:

TTFT-oriented and long-output requests occupy distinct operating points: prefill dominates the former, while repeated generation determines much of the latter’s energy. Free batching can improve prompt efficiency, but the benefit ends when larger contexts make execution compute-heavy.

  • Long-output energy: 2,839 J accounts for 89.9% of Llama-3.2-1B’s N=512 batched-window energy, relative to N=1 at B=16, C=2048 on H200.For Llama-3.1-8B, the additional 10,427 J accounts for 91.0% of the N=512 total.
  • Free batching: 85.49 to 220.32 prompt tokens/J is the increase when B rises from 1 to 4 at C=512 while TN=1 remains nearly constant.This identifies a free-batching region for complete N=1 request windows.
  • Bounded batching: At B=8, expanding C from 512 to 4096 raises TN=1 from 45.27 to 1,175.80 ms and lowers prompt efficiency from 120.81 to 47.70 prompt tokens/J.The same transition raises ENVML from 33,905 to 686,698 mJ and is consistent with a deeper device queue.
  • Energy attribution: 61.1% floor share and 13.6% replay-active share shift to 27.6% and 60.4% at the compute-heavy endpoint.The diagnostic comparison moves from an under-filled, floor-dominant window toward compute-heavy execution.

E. Scheduler Implications and Energy-Aware Serving Design

Energy-aware serving should calibrate model/runtime behavior, choose batching based on request shape, and track request-level energy alongside token-normalized metrics. The same joint energy-accounting principle extends to reasoning budgets, where additional tokens can increase energy without proportionate utility gains.

  • Model and runtime calibration: Model size alone is not a reliable energy predictor, so schedulers need measured model/runtime profiles rather than parameter-count heuristics.For Llama-3.2 models on H200, the 3B/1B token-energy ratio is 1.68× at B=1, C=512, N=10, while the 8B/1B ratio is 2.47×.
  • Model and runtime calibration: FlashAttention-2 reduces Llama-3.2-3B token energy relative to eager execution by 13.4% to 51.9% across tested context and batch settings.The largest reported reduction occurs at B=16, C=2048 on H100.
  • Request-level accounting: 51.6% of measured request energy is the median floor share, rising to 79.1% in low-load short-decode regimes and falling to 31.0% at high-load long-output shapes.These shares show why sampled power and token-normalized metrics can mislead when reported alone.
  • Batching policy: Batching is most valuable when fixed energy dominates, but its benefit is reduced when long-context KV-cache traffic or very long generation makes step energy dominant.The batching knee is model- and context-dependent; at C=512, B=4 captures 79% of the B=16 gain for Llama-3.2-1B/H200 and 81% for OLMoE-1B-7B.
  • Batching policy: Schedulers should expose request energy, token energy, and performance per watt together, treating batch size as an energy-control knob dependent on context, output length, and model family.Request shape can lower token energy by spreading fixed energy across more tokens while increasing total request energy as the inference window extends.
  • Utility-aware reasoning: 1024-token reasoning caps produce the highest accuracy for every evaluated model-benchmark pair but maximize utility per joule for none.For Qwen3-8B on MATH-500, expanding from 64 to 1024 tokens raises accuracy from 26.0% to 68.8% on H100 while increasing request energy 7.42× and reducing utility/J approximately 2.8×.
  • Utility-aware reasoning: Reasoning budgets should be selected jointly by model, workload, and service objective rather than by accuracy or token count alone.Short caps fit saturated or poorly matched settings, moderate caps fit cases where marginal accuracy offsets energy, and larger caps suit accuracy-prioritized settings.

VII. SUMMARY AND CONCLUSION

The paper characterizes LLM inference energy as a request-shape problem rather than a static per-token cost. Its measurements and utility analysis show that request energy, token energy, performance per watt, and task utility must be considered jointly when configuring serving and reasoning budgets.

  • Energy model and evaluation: The study evaluates a decomposed LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts models.It reports request and token energy as functions of model type, phase, batch size, context length, and output length.
  • Energy model and evaluation: Longer outputs and larger batches can reduce token energy by spreading fixed energy across more tokens while increasing total request energy as the inference window extends.The paper also identifies a bounded free-batching regime for prefill before larger contexts become compute-heavy.
  • Serving implications: Energy-aware serving should expose request energy, token energy, and performance per watt alongside latency, throughput, batch size, context length, and output length.The paper frames these request-shape variables as an energy-control surface for scheduling.
  • Utility-aware reasoning: Utility per joule operationalizes whether reasoning tokens produce successful task completion rather than merely efficient token generation.The conclusion notes that reasoning tokens can become wasteful when accuracy saturates at certain token counts.
Loading 2608.28044v1…