PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: LLM Inference Should Be Evaluated as Energy-to-Token Production

Xiang Liu, Shimiao Yuan, Zhenheng Tang, Peijie Dong, Kaiyong Zhao, Qiang Wang, Bo Li, Xiaowen Chu

arXiv:2605.11733v1cs.CEcs.DC

TL;DR

Inference evaluation centered on accuracy and hardware utilization does not capture quality-conditioned token production under power and cooling constraints. The paper formalizes a token-production framework and argues that the binding constraint can shift from compute toward delivered power, cooling, and utilization.

  • Problem

    Inference evaluation centered on accuracy and hardware utilization leaves quality- and service-conditioned token production under power and cooling constraints insufficiently characterized.

    Source: §1 · p. 2

  • Method

    The paper formalizes quality- and service-conditioned token output with a dimensionally consistent production function and maps inference optimizations to the physical variables they change.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The paper concludes that LLM inference’s binding constraint can shift from compute toward delivered power, cooling, and utilization.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Inference papers and benchmarks should report Joules/token, the active binding constraint, PUE-adjusted power, and utilization alongside accuracy and latency.

    Source: §7 · p. 10

    The 65B-class energy-per-token calibration is illustrative, not a controlled head-to-head benchmark, with rows differing in serving stack and workload mix.

    Source: §3 · p. 5

Abstract

from arXiv · show

LLM inference is still evaluated mainly as a model or software problem: accuracy, latency, throughput, and hardware utilization. This is incomplete. At deployment scale, the relevant output is a quality-conditioned token produced under joint constraints from effective compute, delivered data-center power, cooling capacity, PUE, and utilization. We argue that the ML community should treat inference as \emph{energy-to-token production}. We formalize this view with a dimensionally consistent Token Production Function in which token rate is bounded by both compute-per-token and energy-per-token ceilings. Listed API prices vary by over an order of magnitude across providers, but we use price dispersion only as directional motivation, not as causal evidence of marginal cost. The core physical question is instead: under fixed quality and service targets, when does the binding constraint move from theoretical peak compute toward delivered power, cooling, and operational efficiency? Under this framing, system optimizations -- latent KV-cache compression, sparse or heavily compressed attention, quantization, routing, and difficulty-adaptive reasoning -- are not merely local engineering tricks. They are energy-to-token levers because they reduce FLOPs/token, joules/token, memory traffic, or utilization losses under fixed $(q^{*},s^{*})$. We therefore call for inference papers and benchmarks to report Joules/token, active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency.

1 Introduction

The paper frames LLM inference as energy-to-token production: quality- and service-conditioned token output is constrained by compute, delivered power, cooling, and operational efficiency. It argues that evaluation should measure these physical limits and the system optimizations that change them, alongside accuracy and latency.

1 Introduction

  • Tokens are framed as metered AI-factory output, converting electricity, accelerators, memory bandwidth, cooling, and software organization into model output under quality and service constraints.The paper situates this framing amid rising data-center electricity demand and vendors' tokens-per-watt descriptions.Source: §1 · p. 1
  • KV-cache compression, sparse attention, quantization, routing, and scheduling are interventions that can shift the energy-to-token frontier, not merely local benchmark optimizations.The relevant levers reduce FLOPs/token, joules/token, memory traffic, or utilization losses without proportional infrastructure expansion.Source: §1 · p. 2
  • A dimensionally consistent Token Production Function bounds quality- and service-conditioned token output by both compute-per-token and energy-per-token ceilings.The paper maps optimizations to FLOPs/token, joules/token, memory traffic, and utilization losses under fixed quality and service targets.Source: §1 · p. 2
  • Inference evaluation should report Joules/token, the active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency.The paper diagnoses accuracy- and MFU-centered evaluation as incomplete under regional power and cooling constraints.Source: §1 · pp. 1–2
  • The paper argues delivered power and cooling have become binding enough to enter the ML evaluation objective, without claiming electricity alone determines prices, capability, or geopolitical outcomes.Posted API prices span over an order of magnitude across comparable per-million-token units, but are used only as motivation, not causal cost measurements.Source: §1 · p. 2

2 The Token Production Function

The Token Production Function measures quality- and service-conditioned token output as the utilization-adjusted minimum of compute-limited and delivered-power-limited token rates. Its local Leontief form treats compute and delivered power as co-required constraints, with realized output depending on utilization and system efficiency.

2 The Token Production Function

  • At fixed quality and service targets, token rate is U(t) × min(K_eff(t)/c_tok(t), P_IT(t)/e_tok(t)); integrating over T gives total tokens.Both terms in the minimum are tokens per second, and token output is comparable across systems only at the same targets.Source: §2 · p. 2
  • K_eff captures effective hardware and execution throughput, while U captures realized use after queueing, batching, routing, and operational frictions; c_tok and e_tok define distinct compute and power ceilings.U and Φsystem are distinct: utilization reflects deployed capacity in use, while Φsystem reflects architectural energy efficiency under load.Source: §2 · pp. 2–3
  • The min operator is a local Leontief approximation: compute and delivered power are co-required, and system gains shift both limits by reducing c_tok or e_tok or raising U.The formulation is a short-run binding-constraint approximation, not a claim that long-run technological substitution is impossible.Source: §2 · p. 3
  • Φsystem is decomposed into workload-specific serving mechanisms; Φmem < 0.3 indicates memory-bound operation, while Φsched ≪1 indicates scheduling or batching overhead.The proposed estimates use hardware specifications for bandwidth, decode-phase tokens, and SM-activity counters relative to an ideal-batch reference.Source: §2 · p. 3
  • The binding constraint switches at ρ = P_IT/K_eff versus workload ρ*; K_eff should default to realized serving throughput, with peak throughput reported only as an upper-bound calibration.The same deployment can appear power-bound using peak compute but effective-compute-bound using realized throughput; context length and grid headroom also affect the regime.Source: §2 · pp. 3–4
  • When delivered power binds, a 50% reduction in J/token raises the power-side token ceiling under the same cap; report J/token, the active constraint, PUE-adjusted power, and utilization-adjusted output.Such gains count only if quality, latency, reliability, and freshness remain within disclosed tolerances at the stated operating point.Source: §2 · p. 4

3 When Power Becomes the Binding Constraint

The section uses the Token Production Function to organize inference history into three epochs and argues that system-efficiency gains can partly decouple token output from energy, while presenting its calibration as illustrative rather than causal or controlled.

3 When Power Becomes the Binding Constraint

  • The framework maps annual facility-power and effective-compute proxies to public data and treats API price and regional comparisons as directional, not causal.The three-epoch account and its Φsystem step-changes are a theoretical illustration, not empirical validation; Table 1 supplies order-of-magnitude calibration anchors.Source: §3 · pp. 4–5
  • The measured A→C spread is ∼3×, while another 3× from composing INT4 onto MLA is projected; ∼10× is plausible, pending controlled cross-stack J/token measurement.Rows differ in serving stack and workload mix, so the comparison is illustrative rather than a matched-stack head-to-head benchmark.Source: §3 · p. 5, Table 2 · p. 5

3.1 Epoch 1 (2020–2022): The Pre-Cambrian Era

  • In the early phase, compute K(t) and power P(t) were abundant relative to demand, and GPT-3-scale models ran on concentrated clusters with Φsystem ≈1, without sophisticated memory management or scheduling.Scaling laws suggested strong returns from parameters and compute [31] [32] [33], while energy costs were buried in operational budgets.Source: §3.1 · p. 5

3.2 Epoch 2 (2023–2024): The LLM Explosion

  • ChatGPT triggered exponential K(t) growth [34] alongside first-wave Φsystem improvements, while API pricing remained relatively uniform and energy was not yet the binding constraint.FlashAttention [35] reduced attention memory movement from O(N^2) to O(N). Profiling training, fine-tuning, and inference on commodity hardware found memory traffic, not raw FLOPs, dominated real-world LLM throughput [39].Source: §3.2 · p. 5

3.3 Epoch 3 (2025–2026): The Context War and Power Wall

  • By 2025–2026, 1M+ token contexts and rising data-center electricity demand marked a Power Wall that had become binding in some regions.Global data-center electricity reached 415 TWh in 2024 and is projected to reach 945 TWh by 2030; the section says API price divergence is consistent with differing power constraints.Source: §3.3 · p. 6

4 System Optimizations Are Energy Multipliers

System optimizations can reduce compute/token and energy/token under fixed quality and service targets, making them energy-to-token levers. Their gains are configuration-dependent and count only when quality and service requirements are preserved.

4.1 Latent Compression Moves the Memory Boundary

  • DeepSeek-V4’s hybrid stack targets 1M-token serving and reports ∼27% of V3.2 single-token FLOPs and ∼10% of V3.2 KV cache, pending third-party replication.The lineage combines MLA low-rank KV compression [30] and learned sparse selection [46]; V4’s stack includes compressed and heavily compressed attention, low-precision indexing, and heterogeneous KV placement [47].Source: §4.1 · p. 6
  • With task-relevant evidence preserved, memory-side optimizations enable higher batch sizes, million-token inputs, and less memory-stall time; reported million-token-context direction is an order-of-magnitude reduction in etok and ctok.Prior semantic-preserving eviction and offloading work reports up to 50% cache reduction under task-specific quality constraints [48] [49] [55]. These gains are not pure efficiency improvements if retrieval quality degrades.Source: §4.1 · p. 7

4.2 Sparse and Hybrid Attention Reduce Wasted Work

  • Sparse attention, hybrid O(N)/O(N^2) routing, and difficulty-adaptive budgets reduce wasted work; reported examples include 6–11× speedups on 64K+ sequences [46] and 22.4% fewer tokens with no quality loss [57].These figures are configuration-dependent, and compression or adaptive computation must preserve relevant quality and capability targets [20] [25] [59] [60].Source: §4.2 · pp. 7–8
  • At energy-constrained sites, system improvements can stretch delivered power into more quality-conditioned tokens and help maintain capacity at fixed quality and service targets.Long generations and adaptive depth can dominate per-query energy under realistic service targets.Source: §4.2 · p. 8

5 Divergent Energy-to-Token Trajectories

The section frames two stylized, non-exhaustive trajectories that real ecosystems may blend: infrastructure-constrained premium tokens and efficiency-optimized cost-efficient tokens.

5.1 Path A: Infrastructure-Constrained Trajectory

  • Rapid K(t) growth amid grid bottlenecks, legacy infrastructure, and PUE 1.5–2.0 makes delivered power binding, raising token prices and emphasizing frontier capability.Limited Φsystem investment characterizes this path.Source: §5.1 · p. 8

5.2 Path B: Efficiency-Optimized Trajectory

  • Careful K(t) scaling alongside renewable power, grid modernization, Φsystem maximization, and PUE 1.1–1.2 tends to support lower token prices at comparable quality/SLO targets.As an early directional signal, not a global census, OpenRouter reports growth in open-source and China-developed open-weight model token share and heavy coding and agentic-workflow use on a 100T-token sample.Source: §5.2 · p. 8
  • Divergence may persist as export controls, energy endowments, and sovereignty rules shape conditions, while switching costs can turn early adoption into installed-base advantage.Source: §5.2 · p. 8

6 Alternative Views

The section argues that hardware efficiency, grid expansion, silicon access, integration, and market pricing do not remove the physical constraints on quality-conditioned token production. It concludes that energy-to-token comparisons require fixed quality and service targets and disclosed workload and energy-accounting conditions.

Alternative Views

  • A 2× more efficient accelerator can be absorbed by larger models, longer contexts, and higher request volumes as hardware and product cycles diverge, while efficiency gains stimulate rebound in token demand.Hardware cycles span 18–36 months while model scale and context lengths move on 3–6 month cycles [66]; per-token prices for GPT-4-equivalent capability have fallen sharply since 2023 while aggregate token consumption expanded faster.Source: §6 · p. 8
  • Grid expansion can relax Pfacility, but its 5–10-year buildout lags 3–6-month LLM release cycles and does not directly ease PUE, cooling, utilization, or routing constraints.The section frames grid expansion as complementary: Φsystem determines how many quality-conditioned tokens each new megawatt produces.Source: §6 · p. 8
  • Figure 4 contrasts infrastructure-constrained and efficiency-optimized ecosystems, arguing that silicon access alone does not determine token costs or quality-conditioned output [65, 71].Its trajectories are stylized, anchored to a 3×–30× listed-price spread observed in April 2026, and the 2030 endpoint is illustrative rather than a forecast.Source: Figure 4 · p. 9
  • Vertical integration and cross-subsidized APIs can decouple posted prices from marginal cost [70], but do not remove electricity, cooling, interconnect, or utilization constraints.API prices are treated as directional motivation only; the framework applies at the infrastructure layer where physical constraints persist.Source: §6 · p. 9
  • A persistent 2–3× cost advantage can shift market share despite sticky incumbent workloads, because API lock-in, migration costs, and compliance constraints limit elasticity.Tiered pricing may compete away some energy-cost advantage at the margin, but elasticity affects monetization rather than the underlying physical advantage.Source: §6 · p. 9
  • Because token quality varies [24, 25] and posted prices are not marginal costs, comparisons should report J/token at fixed (q*, s*) and disclose workload, batching, hardware, and energy-accounting boundary.With those controls, energy-to-token production becomes measurable; without them, token counts are not comparable.Source: §6 · p. 9

7 Conclusion and Call to Action

The conclusion argues that LLM inference’s binding constraint can shift from compute to delivered power, cooling, and utilization, making energy-to-token production a first-class evaluation target. It calls for energy-aware reporting and benchmarks at disclosed quality and service targets.

Conclusion and Call to Action

  • The Leontief min(·, ·) is a short-run binding-constraint approximation, etok anchors are directional under six disclosed measurement dimensions, and ρ −ρ∗ depends on the Keff convention.These scope choices make disclosure of reviewer-requested fixed dimensions part of the reporting agenda.Source: §7 · p. 9
  • LLM inference’s binding constraint can shift from compute toward delivered power, cooling, and utilization, while Φsystem optimizations expand capacity without infrastructure expansion.The conclusion also argues that hardware alone cannot escape the Power Wall by Jevons Paradox.Source: §7 · p. 9
  • Papers and benchmarks should report J/token, the active binding constraint, PUE-adjusted power, and utilization at disclosed (q∗, s∗), alongside accuracy and latency.Source: §7 · p. 10
  • Conferences and leaderboards should add energy-normalized tracks, while funders, operators, and reviewers should treat Φsystem-shifting work as first-class contributions.The absence of ρ, PUE, and Φmem disclosures should be a reviewable gap.Source: §7 · p. 10

A Scope, Limitations, and What This Paper Does Not Claim

The paper bounds its position as a short-run diagnostic and measurement agenda, not a structural forecast or controlled cross-stack benchmark. Its caveats specify the conditions needed for comparison and identify claims and topics outside its scope.

Scope and limitations

  • The Leontief min(·, ·) form is a short-run binding-constraint lens, not a claim that compute and delivered power are generally non-substitutable or a prediction of equilibrium outcomes.The CES family is the appropriate generalization once substitution elasticities become measurable; the min operator identifies which factor binds in a given measurement window.Source: A Scope, Limitations, and What This… · p. 16
  • The ∼10× spread in Tables 2 and 3 is a directional upper bound, not a ceteris-paribus result, across six disclosed measurement dimensions.The tables compile independent sources; a matched cross-stack J/token benchmark is future work, while the paper proposes reporting and leaderboard standards to make such measurements comparable.Source: A Scope, Limitations, and What This… · p. 16
  • The same H100 can be classified as power-bound or effective-compute-bound depending on whether peak or realized throughput is the denominator.Comparisons require disclosure of the Keff measurement convention and (q*, s*); isolated J/token values without that context are not comparable.Source: A Scope, Limitations, and What This… · p. 16
  • The paper does not predict geopolitical outcomes, capability rankings, or which ecosystem wins, and API prices are directional motivation rather than causal evidence of marginal cost.Training-time energy is out of scope except where serving-side Φsystem amortizes training cost across more tokens; the paper does not redo prior carbon-accounting work [9] [10] [12].Source: A Scope, Limitations, and What This… · p. 16
  • The paper presents each limitation as bounded by design, with its measurement agenda already requiring disclosure of the dimensions that qualify comparisons.The caveats are surfaced in the main text and Appendix B, including the Leontief choice, price evidence, regime-flip example, and (q*, s*) conditioning; tighter controls therefore make the position more defensible.Source: A Scope, Limitations, and What This… · p. 16

B Worked Example: ρ −ρ∗on H100 · C Token Export: The Invisible Commodity Flow · D Token Abundance, Verification, and Value

The worked example shows that an H100 deployment’s binding constraint depends on how effective compute is measured. The following sections connect token production to location-dependent costs, market structure, and the growing importance of verification.

B Worked Example: ρ −ρ∗on H100

  • With peak-throughput Keff, ρ < ρ* classifies the H100 deployment as power-bound: workload energy per FLOP exceeds peak delivery.The anchors are about 3.5 J/token and 4×10^11 FLOPs/token for the workload, versus an H100 at about 700 W and 10^15 peak BF16 FLOPs/s.Source: B Worked Example: ρ −ρ∗on H100 · pp. 16–17
  • A 5–10× reduction in Keff from serving overhead raises realized ρ to the same order of magnitude as ρ*, classifying the hardware as effective-compute-bound.Memory stalls, insufficient batching, kernel-launch overhead, and utilization losses can reduce throughput, particularly at long context with KV-cache pressure.Source: B Worked Example: ρ −ρ∗on H100 · p. 17
  • Because ρ −ρ* depends on measurement convention as well as physics, reproducible diagnostics must report peak or realized Keff, context length, batch size, and quality target.The passage presents this as necessary for comparisons across studies.Source: B Worked Example: ρ −ρ∗on H100 · p. 17

C Token Export: The Invisible Commodity Flow

  • Industrial electricity tariffs differ 2–3× across major regions, and U.S. state tariffs vary widely; with PUE and utilization, location shapes token marginal cost, not causal identification.The pricing patterns are described as consistent with binding-constraint divergence, not as a clean identification result.Source: C Token Export: The Invisible… · p. 17
  • Application migration costs grow super-linearly with system complexity, while energy-optimized ecosystems’ market share suggests Φsystem advantages can offset K(t) constraints.Each developer integrated into an ecosystem represents future token demand locked into that infrastructure.Source: C Token Export: The Invisible… · p. 17
  • Data-localization requirements act as trade barriers, fragmenting global token demand into regional pools and favoring ecosystems with domestic energy abundance.Source: C Token Export: The Invisible… · p. 17

D Token Abundance, Verification, and Value

  • Projected scarcity of high-quality human-generated text during 2026–2028 [75] shifts scarcity toward verification, curation, provenance, and quality assurance, while Φsystem supports generation and validation at scale.The operational question expands from how cheaply tokens can be generated to how reliably they can be filtered, validated, and routed.Source: D Token Abundance, Verification, and… · p. 17

E MLA Worked Example: Bandwidth Derivation · E.1 Extended Directional Comparison

The appendix derives the bandwidth mechanism behind MLA’s KV-cache compression and extends the main-body comparison with directional evidence across configurations. It frames the reported differences as illustrative rather than controlled, with a matched-quality, matched-service benchmark still needed.

E MLA Worked Example: Bandwidth Derivation

  • Approximately 32 MB of KV traffic per layer per decoding step at the stated 65B-class, L = 1024, FP16 setup would cap one-layer decoding at roughly 105 tokens/s on H100 HBM, before other costs.Standard MHA reads keys and values for all prior context positions; the estimate excludes all layers, batching, cache layout, and compute.Source: E MLA Worked Example: Bandwidth… · p. 17
  • MLA’s dc = 512 low-rank latent cuts KV-cache bandwidth by approximately 16× versus MHA at the same head count, reducing etok through lower HBM traffic per token at the same compute utilization.Source: E MLA Worked Example: Bandwidth… · pp. 17–18

E.1 Extended Directional Comparison

  • Roughly 10× separates etok across rows A–F under the same nominal hardware envelope; including capability-tradeoff projections widens the illustrative range toward 30×.The comparison is not a controlled test; under tight power budgets, the analysis favors low-etok configurations at fixed quality, with F identified as the capability-preserving frontier and J and K trading capability for further energy gains.Source: Table 3 · p. 18, §E.1 · p. 19

E MLA Worked Example: Bandwidth Derivation

  • 2–3× higher sustainable batch sizes within the same power envelope raise token output by the same factor under fixed PIT; at 100 MW IT load, MLA can produce 2–3× more intelligence tokens per hour without added infrastructure.Source: E MLA Worked Example: Bandwidth… · p. 18
  • Up to 50% KV-cache reduction without quality degradation was demonstrated by semantic-preserving eviction methods ChunkKV [48] and H2O [49]; MLA extends this principle through learned compression.Source: E MLA Worked Example: Bandwidth… · p. 18

E.1 Extended Directional Comparison

  • ∼66% lower etok from quantization alone (A→C) leaves batch sizes largely unchanged because the KV cache remains the bandwidth bottleneck.Source: §E.1 · p. 18
  • ∼54% lower etok from MLA without quantization (A→D) roughly triples batch size, consistent with KV-compression headroom reported by [30].Source: §E.1 · p. 18
  • A→F projects a ∼10× reduction by composing MLA and INT4, while distillation projects a further 10–35× reduction with a model-capability tradeoff.The V4 long-context developer report gives relative compute and KV-cache reductions at 1M-token context, not a normalized J/token measurement, so it is mechanism evidence rather than an energy benchmark.Source: Table 3 · p. 18, §E.1 · p. 19

F Cross-Vendor Listed-API Pricing (April 2026)

Listed output prices show an approximately 3–30× gap across Chinese and US vendors, but the comparison is not quality- or service-normalized and does not identify specific cost causes.

F Cross-Vendor Listed-API Pricing (April 2026)

  • Table 4 compiles listed per-million-token input/output prices for frontier reasoning models across major Chinese and US vendors as of late April 2026.The table supports the cross-vendor pattern referenced in §4.1, not a controlled head-to-head comparison.Source: F Cross-Vendor Listed-API Pricing… · p. 19
  • DeepSeek rows use cache-miss input prices and output prices converted from RMB to USD; cache-hit inputs are cheaper and the Pro discount is time-limited.Prices are not normalized for quality, latency SLOs, context window, caching policy, batch discounts, exchange-rate movement, or promotional pricing.Source: F Cross-Vendor Listed-API Pricing… · p. 19
  • ∼3–30× output-price gaps span at least four Chinese and three US vendors, making single-firm pricing an incomplete explanation.The gap is consistent with infrastructure-level Φsystem differences shaping marginal API economics, alongside quality, latency-SLO, caching, business-model, subsidy, and exchange-rate differences; no specific cost component is causally identified.Source: F Cross-Vendor Listed-API Pricing… · p. 19

Related papers

Loading 2605.11733v1…