Source-linked AI summary
The Cost of Context: Profiling the Energy Footprint of Input Tokens in Large Language Models
Boris Ruf, Marcin Detyniecki
TL;DR
Input-token energy is difficult to characterize because common estimates often focus on output generation, despite input-heavy applications and contrasting reports about prefill costs. The paper profiles GPU energy across context lengths and finds that engine optimizations shift the efficiency threshold, with up to an 8.9× energy-efficiency advantage at 112k tokens.
Problem
Common inference-cost estimates rely on output-token counts and omit input tokens, leaving context-length effects on energy costs and the prefill phase poorly characterized.
Source: §1 · p. 1Method
The study uses high-resolution GPU power telemetry aligned to inference events and synthetic prompts with exact token lengths to profile prefill energy.
Source: §5.1 · p. 2, §5.2 · p. 2Results
Up to 8.9× energy-efficiency advantage at 112k tokens for optimized engines versus baselines, with optimizations delaying the efficiency threshold.
Source: §6.1 · p. 3, §8 · p. 5Takeaways & Limitations
The paper concludes that prefill energy scaling is primarily an orchestration challenge rather than an inherent arithmetic limitation.
Source: §7 · p. 4The threshold depends on GPU capacity for simultaneous parallel execution, beyond which attention chunks are processed sequentially.
Source: §4 · p. 2
Abstract
from arXiv · showhide
The rapid adoption of Retrieval-Augmented Generation (RAG) and long-context LLMs has shifted the computational burden toward the prefill phase---the initial stage where the model processes all input tokens in parallel before generating a response---yet its energy impact remains poorly characterised. We investigate the contradiction between massive hardware-level power spikes and the sub-linear marginal cost of input tokens reported in system-level studies. Through high-resolution telemetry of Llama-3.1 and Qwen-2 models on NVIDIA A100 hardware, we identify the efficiency threshold where $O(n^2)$ attention complexity overcomes GPU parallelisation. Our results quantify how optimised memory orchestration (PagedAttention) delays this scaling bottleneck across three orders of context scaling, yielding up to an 8.9$\times$ energy efficiency advantage at 112,000 tokens compared to standard baselines. Furthermore, we characterise the energy disparity between input and output tokens, demonstrating that optimised serving (vLLM) causes this ratio to narrow (from $\approx$850$\times$ to 750$\times$) as context scales due to high decode efficiency, whereas unoptimised baselines show a much smaller, roughly constant gap ($\approx$150--230$\times$) because their prefill is itself expensive. This work establishes context-scaling laws that differentiate engine-level optimisations from inherent model constraints, providing an empirical framework for sustainable LLM deployment.
1 Introduction
LLM energy estimates often emphasize sequential output generation, although many applications use input-heavy contexts and input-token energy remains poorly characterized. The paper asks how context length, memory orchestration, and serving-engine optimization shape prefill efficiency and the input–output energy disparity.
1 Introduction
- RAG, long-document summarization, and coding assistants are examples where input volume significantly outweighs generated output.This motivates examining input-token energy rather than relying only on output-token counts.Source: §1 · p. 1
- Hardware benchmarks report up to 3× higher GPU power draw during vLLM prefill, while usage studies find only marginally greater total energy for very large contexts.The contrast creates the paper’s “Prefill Paradox”: high instantaneous power appears decoupled from total energy expenditure.Source: §1 · p. 1
- The paper identifies a gap in understanding how context-length scaling affects LLM energy costs and calls for prefill and decode to be characterized independently.Existing approaches often estimate costs from output-token counts while omitting input tokens.Source: §1 · p. 1
- The paper asks where O(n^2) attention overcomes GPU parallelisation, how memory orchestration shifts that threshold, and how engine optimizations affect input–output energy disparity.These questions distinguish context scaling across engines from the changing energy relationship between input and output tokens.Source: §1 · p. 1
2 Background
Prefill processes the prompt in parallel and builds a KV cache, whereas decode generates tokens sequentially while attending to that cache. Their different computational profiles produce different power and runtime patterns.
2 Background
- Prefill processes all input tokens in parallel and computes a KV cache encoding contextual relationships through self-attention.This phase is compute-bound and fully uses the GPU’s parallel arithmetic units.Source: §2 · p. 1
- Decode generates output tokens sequentially, with each new token attending to the full KV cache built during prefill.The sequential dependency makes decode memory-bound, as the GPU spends most of its time loading weights and cache entries.Source: §2 · p. 1
- Prefill draws high instantaneous power but completes quickly, while decode draws lower power across many sequential steps.The paper studies how this balance changes with context length.Source: §2 · p. 1
3 Related Work
Prior work measures inference power, studies energy footprints and token disparities, models phase-specific costs, and examines system bottlenecks and serving optimizations. These approaches provide context for evaluating energy across inference phases and deployment constraints.
3 Related Work
- TokenPowerBench isolates prefill and decode to measure instantaneous power, while Caravaca et al. report that output energy can exceed input energy in output-heavy prompts.MELODI reports weak correlation between energy consumption and prompt length (0.075), but strong correlation with response length (0.846).Source: §3 · pp. 1–2
- Wilkins et al. [16] and Wu et al. [17] model energy using input and output token counts, while Nguyen et al. [9] discuss sustainable-serving trade-offs.Other studies identify memory pressure and KV-cache availability as bottlenecks; multimodal prefill can cost up to 15× text-only baselines [3].Source: §3 · p. 2
- Serving strategies include PagedAttention [7], phase-splitting [11], chunked prefills [1], KV-cache materialization [13], and new scheduling paradigms [18].Sinha et al. caution that utilization alone is not a sufficient sustainability proxy.Source: §3 · p. 2
4 Theoretical Analysis
The analysis hypothesizes that self-attention’s quadratic complexity eventually dominates energy as GPU parallel efficiency gives way to compute saturation and memory-bandwidth limits.
4 Theoretical Analysis
- For small contexts, parallel processing amortizes fixed startup and weight-loading costs, so energy per token effectively decreases as input tokens increase.This is the high-parallel-efficiency phase in the paper’s hypothesis.Source: §4 · p. 2
- When context size exceeds the GPU’s capacity for simultaneous parallel execution, attention chunks are processed sequentially and the O(n^2) energy tax becomes measurable.The analysis calls this transition the compute saturation cliff.Source: §4 · p. 2
- For massive sequences, the N × N attention-probability matrix drives quadratic VRAM-to-core data movement, eventually saturating memory bandwidth.The paper’s hypothesis attributes the resulting energy cost to off-chip transfers’ high Joule-per-bit cost.Source: §4 · p. 2
5 Experimental Design
The experimental design combines fine-grained GPU telemetry with controlled context-length sweeps to measure prefill energy and compare input- and output-token intensity. It evaluates two model families on NVIDIA A100 hardware across standard and optimised inference engines.
5 Experimental Design
5.1 Instrumentation and Telemetry
- GPU power is sampled through NVML every 1 ms and integrated between prefill_start and first-token completion to calculate E_prefill.Each sample is aligned with inference-engine event markers.Source: §5.1 · p. 2
5.2 Workload Configuration
- A logarithmic context sweep identifies the efficiency threshold where O(n^2) complexity breaks hardware parallelisation, using exact-length synthetic prompts, model-specific tokenisers, and bfloat16 on NVIDIA A100 (80GB) instances.Prompts repeat high-entropy text and truncate to exact token counts; vLLM enforces declared context limits, while the HF Baseline permits extrapolation beyond its context boundary.Source: §5.2 · p. 2
5.3 Controlled Inference Scenarios
- The controlled comparison uses HF Transformers with Eager SDPA and contiguous KV-cache allocation versus vLLM with PagedAttention, isolating prefill with one generated token.Configurations receive a warm-up pass; each deterministic context length is measured once.Source: §5.3 · p. 3
5.4 Comparative Intensity Protocol
- With output length fixed at 128 tokens, subtracting isolated prefill energy from total energy yields mean energy per output token for comparison with input-token intensity.The resulting E_out/E_in ratio is measured across the context sweep.Source: §5.4 · p. 3
6 Experimental Evaluation
Across Llama-3.1 8B and Qwen-2 7B, the evaluation compares HF SDPA with vLLM PagedAttention as context length scales. It finds a large energy-efficiency advantage for vLLM at long contexts, distinct input/output intensity patterns across engines, and latency-driven energy growth under saturation.
6.1 Energy Intensity Scaling
- At approximately 112,000 tokens, vLLM used 1,005 J versus the HF baseline’s 7,600 J for Llama-3.1, a 7.6× energy-efficiency improvement; Qwen-2 reached 8.9×.Llama-3.1 processing times were 3.54 seconds for vLLM and 25.54 seconds for the baseline.Source: Table 1 · p. 3, §6.1 · p. 3
6.2 Input vs. Output Proportion
- In vLLM, output-token intensity was approximately 850× input-token intensity at 1k context and ≈750× at 112k, while HF ratios were ≈155× to ≈220× for Llama-3.1 and ≈220× to ≈160× for Qwen-2.The baseline’s smaller gap reflects prefill costs ≈9× higher per token than vLLM’s, while decode costs were 1.5–2.6× higher.Source: §6.2 · p. 4, Figure 2 · p. 4
6.3 Component-Wise Attribution
- Beyond 16,384 tokens, A100 power plateaus at approximately 280–310 W, while baseline energy intensity rises mainly with latency extension and attention-matrix traffic.vLLM’s FlashAttention and PagedAttention reduce memory traffic, and its normalised latency remains nearly flat until L>32k.Source: §6.3 · p. 4, Figure 3 · p. 4
7 Discussion
The prefill-scaling contradiction is attributed primarily to orchestration rather than an inherent arithmetic limit. In optimized serving, parallel execution amortizes fixed hardware costs, while high arithmetic intensity supports low energy per token even at high instantaneous power.
7 Discussion
- A flat J/T profile across three orders of magnitude in vLLM is attributed to amortising fixed hardware costs through parallel execution.This orchestration is presented as the explanation for the apparent contradiction in prefill energy scaling.Source: §7 · p. 4
- The Prefill Paradox is framed as an orchestration challenge, not an inherent arithmetic limitation.Source: §7 · p. 4
- High arithmetic intensity lets prefill reach maximum parallel efficiency, minimising power per token despite high instantaneous power draw.Source: §7 · p. 4
- Input tokens remain effectively negligible relative to output tokens beyond 100,000 tokens for modern optimised text-only stacks, though multimodal inputs or future architectures may shift this balance.Source: §7 · p. 4
8 Conclusion
The profiling finds that prefill energy scaling depends on engine orchestration: optimized memory strategies delay efficiency thresholds, while unoptimized baselines incur an early quadratic energy tax. The results also quantify context-dependent input-output energy disparities.
8 Conclusion
- Up to 8.9× energy efficiency advantage at 112k tokens is reported for PagedAttention and FlashAttention, which delay the efficiency threshold.Source: §8 · p. 5
- GPU power draw saturates at ≈16k tokens, while unoptimised baselines incur an O(n^2) energy tax almost immediately once parallel overhead is exceeded.Source: §8 · p. 5
- Optimised engines maintain an efficient linear J/T profile until L > 32k.Source: §8 · p. 5
- Unoptimised systems show a roughly constant input-output energy disparity of ≈150–230:1 because their prefill is itself expensive.Source: §8 · p. 5
- Optimised systems show a narrowing input-output energy intensity gap as they manage memory effectively.Source: §8 · p. 5
9 Future Work
Future energy-aware LLM research should examine batching, precision-energy trade-offs, and scalability beyond the evaluated model scale.
9 Future Work
- High-throughput amortisation studies should investigate how request and continuous batching compress decode-phase energy disparity by amortising weight-loading costs.Source: §9 · p. 5
- Precision-energy studies should test whether FP8 or INT4 memory-traffic reductions outweigh specialised low-precision tensor-unit power overhead.Source: §9 · p. 5
- Scalability sweeps should extend to Mixture-of-Experts and 70B+ parameter models to test whether scaling bottlenecks shift predictably with parameter count and activated memory footprints.Source: §9 · p. 5