PaperlayerView exact revision on arXiv

Source-linked AI summary

The Real Cost of Context: Predicting Compute Time and Energy Consumption in LLM Prefill and Decode Phases

Lars Krupp, Daniel Geißler, Francisco M. Calatrava-Nicolas, Stefan Gerd Fritsch, Paul Lukowicz, Jakob Karolus

arXiv:2610.39621v1cs.AI

TL;DR

Energy estimation for proprietary API models is difficult because their serving hardware and power use are hidden, while existing approaches often overlook input context. The paper profiles input and output lengths and links API compute-time models to locally measured energy; it finds prefill matters for large contexts and compute time can serve as a proxy for energy.

  • Problem

    Energy estimation for proprietary API models is difficult because their hardware and power consumption are hidden, and output-focused approaches inadequately model input length for large-context workloads.

    Source: Introduction · pp. 1–2

  • Method

    The study profiles input and output token lengths, fits GLMs for compute time, and matches API compute-time behavior to locally measured compute-time and energy GLMs.

    Source: Introduction · p. 2

  • Results

    The prefill phase significantly impacts compute time and energy, and compute time serves as a reliable proxy for estimating API energy through locally equivalent models.

    Source: Conclusion · p. 10

  • Takeaways & Limitations

    Prefill must be considered for accurate energy estimation as context sizes grow and agentic AI systems expand.

    Source: Conclusion · p. 10

    Experiments covered only a limited number of hardware-software configurations and used batch size one, unlike likely higher-batch proprietary APIs.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The integration of Large Language Models (LLMs) into agentic architectures has substantially increased token consumption, raising concerns about their energy footprint. Yet, estimating the energy consumption of proprietary API-based models remains challenging, and existing approaches focus primarily on generated output tokens while overlooking the impact of the large input contexts characteristic of agentic workflows. To address this gap, we systematically profile compute time and energy consumption across a wide range of prefill (input) and decode (output) token lengths. Using generalized linear models, we show that decode compute time and energy scale linearly with output length, whereas prefill compute time and energy exhibit both linear and quadratic scaling with input length. These findings demonstrate that input context becomes a relevant factor for large-context workloads. Furthermore, we show that compute time serves as a reliable proxy for energy consumption when direct energy measurements are unavailable. Our mapping framework enables precise energy estimation and supports more sustainable deployment decisions for agentic AI.

Introduction

Estimating energy for proprietary API models is difficult because their hardware and power use are hidden, while large-context workloads are poorly represented by output-focused estimates. The paper profiles input and output lengths and uses GLMs to model compute time and connect it to energy estimates.

Introduction

  • Proprietary API models expose little information about their hardware, serving configuration, or power consumption, making their energy footprint difficult to estimate.Open-weight models on accessible hardware can instead be measured with software telemetry or external power meters.Source: Introduction · p. 1
  • Existing estimates largely focus on output tokens despite agentic workflows repeatedly including histories, instructions, retrieved documents, tool outputs, and intermediate reasoning in their inputs.These workflows can substantially increase input token consumption.Source: Introduction · pp. 1–2
  • Input tokens have both linear and quadratic effects on compute time, so output-focused estimates may miss an important cost component for long-context requests.The paper systematically varies prefill and decode token lengths and uses the resulting observations to fit a GLM.Source: Introduction · p. 2
  • Paired local GLMs connect API-observable compute-time behavior to energy measurements from local configurations, supporting energy estimation without access to proprietary serving infrastructure.The local compute-time GLM most similar to the API GLM identifies the paired energy GLM used for the estimate.Source: Introduction · p. 2

Related Work

Prior work has studied training and inference energy, as well as the serving mechanisms and infrastructure that shape it. The paper identifies a gap in modeling long input contexts for API-based inference, where hardware and power information is hidden.

Related Work

  • Earlier environmental-impact research moved from training costs and carbon-estimation dependencies to inference measurements and deployment trade-offs Strubell et al. [2019], Patterson et al. [2021], Luccioni et al. [2023], Samsi et al. [2023], Husom et al. [2025].The reviewed work includes model, accelerator, workload, and datacenter factors.Source: Related Work · pp. 2–3
  • Inference energy depends on model and request characteristics as well as serving hardware and infrastructure, while commercial APIs conceal infrastructure and power details.Prior studies measured deployment energy, hardware and workload effects, and energy attributed to serving components.Source: Related Work · pp. 2–3
  • Research has distinguished compute-intensive prefill from decode and described serving choices such as batching, hardware allocation, and parallelism as relevant to inference energy.Commercial APIs expose request-level information such as token counts and response time, limiting estimators that require direct measurements or detailed system knowledge Krupp et al. [2026b].Source: Related Work · p. 3
  • EcoLogits models GPU energy using output tokens, active parameters, and assumed batch size, but does not explicitly represent input length Rincé and Banse [2025].Earlier evaluations mostly used short inputs; longer prompts up to 32,768 tokens were studied for local models on known hardware Fernandez et al. [2025].Source: Related Work · p. 3
  • The paper addresses limited characterization of long-context API inference by varying inputs up to approximately 200,000 tokens while controlling output length and measuring duration, compute time, and energy.The cited prior prompt lengths include 8 tokens Argerich and Patiño-Martínez [2024], 28–134 Luccioni et al. [2023], and approximately 369 AI Energy Score [2025].Source: Related Work · p. 3

Method

The method links API-observable compute-time behavior to locally measured energy through matched GLMs. It varies input and output lengths, removes estimated network latency, and fits interpretable models for compute time and energy.

Method

  • The API compute-time GLM is matched to the closest local compute-time GLM, whose paired energy GLM estimates API energy from locally measured power and compute time.The evaluation uses Mistral-Small-2603, a 119B-parameter MoE model with 6B active parameters, and three local hardware configurations.Source: Method · p. 3

Benchmark Construction

  • Prompts combine a nonce to inhibit prefix caching, expandable random sentences for input length, and a fixed prompt for output generation; reruns address variable output lengths.The benchmark also estimates network latency to reduce noise from API calls.Source: Benchmark Construction · pp. 3–4, Method · p. 3

Evaluating the Impact of Prefilling

  • The benchmark tests 20 input lengths from 10,000 to 200,000 tokens against five output lengths from 100 to 2,000 tokens, with 10 repeats per combination.Network latency is estimated by pinging the server every second and subtracting it from total response time; other serving overhead is treated as negligible for these large-token runs.Source: Evaluating the Impact of Prefilling · p. 4

Predicting Compute Time

  • A quasipoisson GLM with an identity link models compute time because its variance increases with the mean and its additive form reflects sequential prefill and decode operations.Each generated output token incurs a flat additive GPU execution cost; the fitted model is also tested on unseen interpolated and extrapolated configurations.Source: Predicting Compute Time · p. 4

Predicting Energy Consumption

  • For each local configuration, the study fits compute-time and energy GLMs using GPU power measurements, then uses numerical integration to estimate energy per prompt.The model and measurement workflow is repeated locally with Mistral-Small-2603 hosted using vLLM across H100 and H200 configurations.Source: Predicting Energy Consumption · p. 5

Results

Across input and output lengths, the results show nonlinear input-related compute-time growth and energy scaling, while paired local models reveal a linear relationship between compute time and energy. A locally matched configuration provides a lower-bound estimate for API energy, and the estimates exceed EcoLogits increasingly as input length grows.

Results

Predicting Compute Time

  • Input tokens have significant linear and quadratic compute-time effects, output tokens scale at β3 = 6.71 ms/token, and the input–output interaction raises decode time as context grows.The input coefficients are β1 = 0.018 ms/token and β2 = 2.70 × 10^-7 ms/token^2; the interaction coefficient is β4 = 1.23 × 10^-5 ms/token^2.Source: Predicting Compute Time · pp. 5–6

Predicting Energy

  • Energy GLMs show linear and quadratic input-token effects, linear output-token and input–output interaction effects, and a maximum unseen-data difference of 2.48%.GPU utilization had a median of 100% across configurations.Source: Predicting Energy · p. 7

Time as Indicator for Energy Consumption

  • Paired local compute-time and energy GLMs have a linear relationship, supporting compute time as an energy proxy; the H200 2GPU, TP=2 match yields a lower-bound API energy estimate.That local configuration is the closest match to the API behavior, though not a perfect one.Source: Time as Indicator for Energy Consumption · p. 7, Figure 2 · p. 8

Comparing Energy Estimates

Discussion

The GLM pairings predict compute time across interpolated and extrapolated inputs, with explainability aligned to transformer architecture. The discussion argues that prefill matters for large-context energy estimates, while noting practical constraints in the tested configurations.

Discussion

  • The GLM pairings are described as minimal-data, generalizable, and explainable, with behavior aligned to transformer architecture.Source: Discussion · p. 9

Prefill Time Must Not Be Neglected

  • For inputs beyond commonly benchmarked lengths, established approaches can systematically underestimate compute time and energy.The authors note that benchmarks rarely exceed 4,000 input tokens, which they believe has contributed to prefill energy being deemed negligible.Source: Prefill Time Must Not Be Neglected · pp. 9–10

Scaling Our Approach Through Local Equivalents

  • GLMs predict interpolated and extrapolated data points, with explainability and alignment to transformer architecture supporting real-world benchmarking.The authors contrast this with purely data-driven methods.Source: Scaling Our Approach Through Local… · p. 10
  • More comparisons with local hardware equivalents can improve API-model estimates, but exploring the configuration space has an associated cost.The accuracy depends on finding local hardware with compute-time behavior close to the API-accessed model.Source: Scaling Our Approach Through Local… · p. 10

Limitations

  • Experiments covered only a limited number of hardware-software configurations because of limited computational resources.The tested configurations did not include a perfect fit with the proprietary Mistral-Small-2603 API version.Source: Limitations · p. 10
  • Batch size one enabled phase separation; the authors report that larger batches increase prefill’s energy contribution without changing the overall energy trend.They also note that batching reduces decode energy by a constant margin.Source: Limitations · p. 10
  • Accuracy improves with more GLM pairs, motivating a planned community website to aggregate models and recordings across configurations.The planned aggregation would include different batch sizes.Source: Limitations · p. 10

Conclusion

The conclusion identifies prefill as a significant factor in LLM compute time and energy, especially as contexts grow. It presents GLMs and local equivalents as a way to estimate energy when direct measurements for proprietary LLMs are unavailable.

Conclusion

  • The prefill phase significantly impacts LLM compute time and energy, making it critical to accurate estimation as contexts grow.Source: Conclusion · p. 10
  • GLMs predict compute and energy behavior from recorded data, and compute time serves as a proxy for proprietary models’ energy estimates via local equivalents.Source: Conclusion · p. 10
  • The findings offer a scalable, reproducible way to estimate energy consumption and advance transparency for proprietary LLMs.Source: Conclusion · p. 10

Related papers

Loading 2610.39621v1…