PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: LLM Inference Should Be Evaluated as Energy-to-Token Production

Xiang Liu, Shimiao Yuan, Zhenheng Tang, Peijie Dong, Kaiyong Zhao, Qiang Wang, Bo Li, Xiaowen Chu

arXiv:2605.11733v1cs.CEcs.DC

TL;DR

LLM inference is commonly evaluated with model and software metrics that omit deployment-scale constraints on quality-conditioned token production. The paper proposes a Token Production Function bounded by compute and energy ceilings, then frames system optimizations as energy-to-token levers and calls for standardized energy-aware reporting. Its evidence and recommendations are directional and bounded by non-controlled comparisons and complementary infrastructure constraints.

  • Problem

    Current inference evaluation does not fully capture quality-conditioned token production under joint compute, power, cooling, PUE, and utilization constraints.

  • Method

    The paper formalizes inference with a dimensionally consistent Token Production Function whose token rate is bounded by compute-per-token and energy-per-token ceilings at fixed quality and service targets.

  • Results

    System optimizations are framed as energy-to-token levers that reduce FLOPs/token, joules/token, memory traffic, or utilization losses; reported adaptive reasoning reduces token output by 22.4% with no quality loss.

  • Takeaways & Limitations

    Inference papers and benchmarks should report Joules/token, the active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency.

  • Takeaways & Limitations

    The directional etok compilation is not a controlled head-to-head benchmark because rows differ in serving stack and workload mix.

Abstract

from arXiv · show

LLM inference is still evaluated mainly as a model or software problem: accuracy, latency, throughput, and hardware utilization. This is incomplete. At deployment scale, the relevant output is a quality-conditioned token produced under joint constraints from effective compute, delivered data-center power, cooling capacity, PUE, and utilization. We argue that the ML community should treat inference as \emph{energy-to-token production}. We formalize this view with a dimensionally consistent Token Production Function in which token rate is bounded by both compute-per-token and energy-per-token ceilings. Listed API prices vary by over an order of magnitude across providers, but we use price dispersion only as directional motivation, not as causal evidence of marginal cost. The core physical question is instead: under fixed quality and service targets, when does the binding constraint move from theoretical peak compute toward delivered power, cooling, and operational efficiency? Under this framing, system optimizations -- latent KV-cache compression, sparse or heavily compressed attention, quantization, routing, and difficulty-adaptive reasoning -- are not merely local engineering tricks. They are energy-to-token levers because they reduce FLOPs/token, joules/token, memory traffic, or utilization losses under fixed $(q^{*},s^{*})$. We therefore call for inference papers and benchmarks to report Joules/token, active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency.

1 Introduction

The paper argues that inference evaluation should treat tokens as quality- and service-conditioned industrial output constrained by compute, power, cooling, and utilization. It formalizes this perspective and calls for energy-aware reporting alongside conventional ML metrics.

  • AI deployments convert electricity, accelerators, memory bandwidth, cooling capacity, and software organization into quality- and service-constrained token output.
  • Accuracy, latency, throughput, and MFU do not answer how many quality-conditioned tokens a deployment can produce from fixed compute, power, cooling, and utilization.
  • KV-cache compression, sparse attention, quantization, routing, and scheduling shift the energy-to-token frontier rather than merely improving local benchmark metrics.
  • Posted API prices span over an order of magnitude across comparable per-million-token units, but the paper uses this dispersion only as directional motivation, not causal cost evidence.
  • The Token Production Function bounds token output by compute-per-token and energy-per-token ceilings under fixed quality and service targets.
  • The proposed evaluation agenda reports Joules/token, active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency.
  • The claim is bounded: delivered power and cooling enter the ML evaluation objective, but electricity alone is not claimed to determine prices, capability, or geopolitical outcomes.

2 The Token Production Function

The Token Production Function models quality- and service-conditioned token production as the realized minimum of compute and power ceilings, adjusted for system mechanisms and utilization. Its fixed-proportions formulation identifies the active bottleneck and links optimizations to measurable energy, compute, memory, and scheduling effects.

  • The function converts effective compute and IT-delivered power into token rate through the minimum of compute-per-token and energy-per-token ceilings.
  • Both ceilings have units of tokens/sec, while total output Q_token integrates token rate over time; comparisons require fixed quality and service targets.
  • K_eff denotes effective compute throughput after hardware, kernel, and memory-stall losses, whereas U captures demand, scheduling, routing, and operational utilization losses.
  • The Leontief minimum treats compute and delivered power as co-required inputs under negligible short-run substitution, providing a local binding-constraint approximation.
  • Φ_system parameterizes measurable mechanisms that change c_tok and e_tok, including memory, scheduling, and architecture effects, rather than acting as one free multiplier.
  • The crossover diagnostic compares ρ = P_IT/K_eff with workload demand ρ*; ρ > ρ* indicates compute scarcity, while ρ < ρ* indicates delivered-power scarcity.
  • K_eff should default to realized serving throughput at the disclosed operating point, with peak throughput reported only as an upper-bound calibration.
  • A 50% reduction in measured J/token raises the power-side token ceiling under the same power cap without adding infrastructure, provided quality, latency, reliability, and freshness remain within target.

3 When Power Becomes the Binding Constraint

The paper frames inference history as a shift from abundant compute toward power and cooling constraints, using the Token Production Function to interpret how system improvements alter token production. Its illustrative calibration shows measured and projected energy-efficiency gains, while emphasizing that the comparisons are directional rather than controlled validation.

  • Framework: The Token Production Function organizes inference history into epochs by relating system-efficiency changes to power and token-production constraints.The framework uses public annual facility-power proxies and qualitative Φsystem step-changes rather than causal empirical identification.
  • Calibration: Table 2 is an illustrative compilation rather than a controlled head-to-head benchmark because configurations differ in serving stack, workload mix, and energy-accounting boundary.The 65B / 100 ms setting is a nominal anchor, not a normalized ceteris-paribus comparison.
  • Calibration: The measured A→C spread is ∼3×, while composing INT4 onto MLA implies an additional projected 3× and makes ∼10× plausible.The paper identifies the ∼3× measured spread as conservative evidence and the ∼10× figure as a projection requiring controlled cross-stack benchmarking.
  • Epoch 2 (2023–2024): FlashAttention, PagedAttention/vLLM, and quantization reduced memory movement or stretched capacity, changing both compute-per-token and energy-per-token.FlashAttention is described as reducing attention memory movement from O(N^2) to O(N), while PagedAttention enables dynamic KV-cache allocation.
  • Historical illustration: Figure 2 presents global data-center electricity alongside a qualitative Φsystem proxy, showing energy growing roughly linearly while Φsystem rises over an order of magnitude.The proxy is normalized to 2020=1 and is not a fitted measurement.
  • Power constraint: 415 TWh in 2024 and 945 TWh projected for 2030 mark the growth of global data-center electricity as the Power Wall emerges.US data centers are separately projected at 325–580 TWh by 2028.

4 System Optimizations Are Energy Multipliers

Inference optimizations become energy-to-token multipliers when they reduce memory traffic, compute, or energy intensity while preserving fixed quality and service targets. Their reported benefits are configuration-dependent and should not be read as unified head-to-head results.

  • Energy-to-token levers: Φsystem choices reduce FLOPs/token, joules/token, memory traffic, or utilization losses under fixed quality and service targets.The paper treats these changes as production-function levers rather than only benchmark optimizations.
  • Evidence boundary: Figure 3 compares reported gains across heterogeneous systems papers and developer reports rather than a unified benchmark.Its architectural levers are comparable only under fixed quality and SLO assumptions.
  • Memory boundary: Memory-side optimizations can support higher batch sizes, million-token contexts, and improved hardware utilization when memory traffic is the binding bottleneck.A developer report lists approximately 27% of V3.2 single-token FLOPs and approximately 10% of V3.2 KV cache for a 1M-token serving target, pending third-party replication.
  • Latent compression: 50% cache reduction is reported for semantic-preserving eviction and offloading under task-specific quality constraints.The DeepSeek lineage extends this direction with learned compression and sparse top-k selection.
  • Sparse and adaptive computation: 22.4% token-output reduction is reported for difficulty-adaptive token budgets with no quality loss.The method allocates compute according to per-token entropy; reported sparse-attention speedups of 6–11× are single-source and configuration-dependent.
  • Energy-to-token levers: KV-cache compression, sparse attention, routing, and adaptive computation can increase token output per unit of delivered power when quality remains comparable.Compression is counted as a gain only when retrieval, reasoning, latency, and reliability stay within the fixed envelope.

5 Divergent Energy-to-Token Trajectories

The production-function framing distinguishes infrastructure-constrained ecosystems from efficiency-optimized ones. Their trajectories differ because power, cooling, PUE, and Φsystem shape token economics alongside compute supply.

  • Infrastructure-constrained: Infrastructure-constrained ecosystems may scale compute rapidly while grid bottlenecks and legacy infrastructure constrain power, with PUE of 1.5–2.0.Limited Φsystem investment is associated in the paper’s archetype with rising token prices as delivered power becomes binding.
  • Efficiency-optimized: Efficiency-optimized ecosystems combine careful compute scaling, power expansion, aggressive Φsystem maximization, and PUE of 1.1–1.2.The paper presents this archetype as tending toward lower token prices under comparable quality and SLO targets.
  • Strategic interpretation: The paper interprets ecosystem strategy as accumulating K/P/Φsystem capacity before providers compete on price, latency, and quality.It further describes marginal token cost as shaped by PUE and energy-related factors, though the supplied passage is truncated.

6 Alternative Views

The paper addresses alternative explanations by bounding the role of hardware, grid expansion, vertical integration, demand elasticity, and token heterogeneity. Its position is that physical production constraints persist, while comparisons require quality- and service-conditioned measurement.

  • Hardware efficiency: Hardware improvements may raise performance per watt, but 18–36-month hardware cycles lag 3–6-month model and context cycles.The paper invokes rebound: efficiency gains can be absorbed by larger models, longer contexts, and higher request volumes.
  • Grid expansion: Grid-scale additions can relax facility power over 5–10 years, but they do not directly resolve PUE, cooling, utilization, or routing inefficiencies.The paper therefore presents Φsystem as complementary to renewable and grid expansion.
  • Stylized trajectories: The trajectories in Figure 4 are stylized, anchored to a 3×–30× listed-price spread, and do not forecast 2030 outcomes.Path A is infrastructure-constrained; Path B is efficiency-optimized despite tighter compute supply.
  • Price signals: Vertical integration and cross-subsidized APIs can decouple posted prices from marginal cost, but infrastructure still requires electricity, cooling, interconnect, and utilization.API prices are therefore directional motivation rather than causal cost measurements.
  • Demand elasticity: A persistent 2–3× cost advantage may shift market share despite API lock-in, migration costs, and compliance constraints.The paper treats elasticity as affecting monetization rather than removing the underlying physical advantage.
  • Measurement boundary: Token counts are not comparable without fixed quality and service targets plus disclosed workload, batching, hardware, and energy-accounting boundaries.The proposed measure is J/token at fixed (q*, s*), not raw tokens per joule.

7 Conclusion and Call to Action

The paper argues that LLM inference can become power-, cooling-, or utilization-bound rather than compute-bound, and calls for energy-to-token evaluation under fixed quality and service targets.

  • The binding constraint can shift from compute K toward delivered power P, cooling, and utilization.
  • Φsystem optimizations can expand token capacity without infrastructure expansion.
  • Benchmarks should report J/token, active binding constraint, PUE-adjusted power, and utilization at disclosed (q∗, s∗) alongside accuracy and latency.
  • Energy-normalized serving tracks and disclosure of ρ, PUE, and Φmem should become reviewable evaluation requirements.

A Scope, Limitations, and What This Paper Does Not Claim

The paper bounds its framework as a short-run, directional evaluation lens rather than a structural macroeconomic model or causal account of prices and outcomes.

  • The Leontief min(·, ·) form is a short-run binding-constraint approximation, not a long-run substitution model.
  • Tables 2 and 3 provide illustrative, independently sourced anchors conditioned on six disclosed measurement dimensions rather than controlled ceteris-paribus benchmarks.
  • The ρ −ρ∗ diagnostic depends on whether Keff uses peak or realized serving throughput, so both the convention and operating point must be disclosed.
  • The paper does not treat API prices as causal marginal-cost evidence or predict geopolitical outcomes, capability rankings, or ecosystem winners.
  • The framework surfaces its caveats in the main text and makes reviewer-requested control dimensions part of the reporting agenda.

B Worked Example: ρ −ρ∗on H100

The H100 example shows that the same accelerator can be classified as power-bound or effective-compute-bound depending on how effective compute is measured.

  • Under a peak-throughput denominator, ρ < ρ∗ classifies the deployment as power-bound.
  • A 5–10× reduction in realized effective serving throughput raises ρ toward ρ∗ and reclassifies the same hardware as effective-compute-bound.
  • The ρ −ρ∗ diagnostic requires reporting whether Keff is peak or realized and specifying context length, batch size, and quality target.

E MLA Worked Example: Bandwidth Derivation

The MLA example connects KV-cache compression to lower memory traffic and energy per token, while the broader comparison distinguishes measured results, projections, and capability trade-offs.

  • Bandwidth derivation: In MHA decoding, a 65B-class workload with L = 1024 generates approximately 32 MB of KV traffic per layer per step and is bandwidth-limited before other costs.
  • Bandwidth derivation: MLA uses a low-rank latent and reduces KV-cache bandwidth by approximately 16× relative to full MHA at the same head count.
  • Bandwidth derivation: The reduction maps to lower etok through Φmem because lower HBM traffic means fewer watt-seconds per token at the same compute utilization.
  • Bandwidth derivation: Within the same power envelope, MLA empirically enables 2–3× higher sustainable batch sizes and correspondingly higher token output at fixed PIT.
  • Configuration comparison: Quantization alone reduces etok by ∼66% while batch sizes remain largely unchanged because the KV cache remains the bandwidth bottleneck.
  • Configuration comparison: MLA without quantization reduces etok by ∼54% and roughly triples batch size.
  • Configuration comparison: Rows A–F show roughly a 10× etok spread within the nominal hardware envelope, while capability-tradeoff projections widen the illustrative range toward 30×.
  • Configuration comparison: The comparison is directional rather than controlled, and matched benchmarks at fixed q∗ and s∗ remain future work.

F Cross-Vendor Listed-API Pricing (April 2026)

Table 4 compiles listed per-million-token input/output prices across major Chinese and US vendors as of late April 2026, while documenting important normalization limits. The observed ∼3–30× output-price gap spans multiple vendors but is treated as directional evidence consistent with several possible infrastructure and market differences, not causal proof.

  • Table 4 reports listed per-million-token input/output prices for frontier reasoning models from major Chinese and US vendors as of late April 2026.DeepSeek input prices use cache-miss rates, while output prices are converted from RMB to USD.
  • The table does not normalize prices for quality, latency SLOs, context window, caching policy, batch discounts, exchange-rate movement, or promotional pricing.DeepSeek cache-hit inputs are cheaper, and the Pro discount is time-limited.
  • The ∼3–30× output-price gap appears across at least four independent Chinese vendors and three independent US vendors.This makes a single-firm pricing-strategy explanation incomplete, while leaving multiple infrastructure and market explanations in play.
  • The observed pricing gap is treated as consistent with infrastructure-level Φsystem differences alongside quality, latency-SLO, caching, business-model, subsidy, and exchange-rate differences, not as causal identification of a specific cost component.
Loading 2605.11733v1…