Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

Molka Chkir; Syed Muhammad Danish; Jos Höll; Arghavan Asad

Source-linked AI summary

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

Molka Chkir, Syed Muhammad Danish, Jos Höll, Arghavan Asad

arXiv:2608.25096v1Aug 25, 2026cs.LGcs.AI

TL;DR

Inference energy is an important sustainability concern, but the effect of attention architecture on decode-phase energy scaling has not been systematically characterized. This study empirically evaluates four open-source LLMs across attention mechanisms and workloads, finding that attention architecture governs energy scaling while model size determines absolute consumption and batching improves efficiency.

Problem
Inference may account for up to 90% of lifecycle energy consumption, yet attention mechanisms’ effects on decode-phase energy scaling remain systematically uncharacterized.
Method
The study measures decode-phase GPU energy for four open-source LLMs using MHA, GQA, and GQA with SWA across context lengths, KV-cache growth, generation positions, and batch sizes.
Results
Attention architecture primarily governs energy scaling: MHA increases 17–18% from 128 to 1800 tokens, versus 3–4% for GQA-based models, while GQA with SWA remains nearly constant.
Takeaways & Limitations
Model size primarily determines absolute energy consumption, while batching reduces energy per generated token and request latency by up to 87%.

Abstract

from arXiv · show

The growing adoption of large language models (LLMs) has raised increasing concerns about the energy consumption and environmental impact of inference. This paper presents a systematic empirical study of decode-phase energy consumption across representative open-source LLMs employing Multi-Head Attention (MHA), Grouped Query Attention (GQA), and Grouped Query Attention with Sliding Window Attention (SWA) to characterize how attention architecture influences decode-phase energy consumption under varying inference workloads. We evaluate four models across different context lengths, batch sizes, and generation workloads while measuring GPU energy using NVIDIA hardware counters. We examine the effects of context length, attention mechanism, Key-Value (KV) cache growth, and batching on decode-phase energy consumption. Results show that attention mechanism is the primary factor governing how decode energy scales with context length. MHA models exhibit substantially steeper energy growth than GQA models, whereas GQA with SWA maintains nearly constant energy consumption. We further show that model size primarily determines absolute energy consumption, while batching reduces both energy per generated token and request latency by up to 87%. These findings provide practical guidance for selecting energy-efficient LLM architectures and inference configurations.

I. INTRODUCTION

Inference energy is an important sustainability concern, but prior work offers limited evidence about how attention architecture shapes decode-phase scaling. This study systematically evaluates context length, attention design, KV-cache growth, and batching across representative open-source LLMs.

Inference may account for up to 90% of a model’s total lifecycle energy consumption, making efficient serving important for sustainable deployment.
Prior studies often measured training or inference energy using few models and fixed workloads, limiting insight into the factors governing inference energy.
MHA, GQA, and GQA with SWA differ in KV-state storage and access, but their effects on decode-energy scaling had not been systematically characterized.
The study examines how context length, attention mechanism, KV-cache growth during generation, and batching affect decode-phase energy.
Across four models, MHA shows steeper energy growth, GQA remains more stable, and GQA with SWA maintains nearly constant energy as context increases.The evaluated models are OPT-1.3B and Phi-3 Mini with MHA, Gemma-2-2B with GQA, and Mistral-7B with GQA plus SWA.

II. RELATED WORK

Prior research established the importance of inference energy but generally did not isolate decode-phase scaling across attention architectures and jointly varying workloads. This work addresses that gap with controlled comparisons of MHA, GQA, and GQA with SWA.

Inference may represent up to 90% of lifecycle energy after deployment at scale, motivating greater attention to operational energy use.
Earlier inference benchmarks primarily reported aggregate measurements and did not isolate the decode phase or systematically study attention-related scaling.
MHA, GQA, and SWA were introduced as architectural approaches that respectively provide standard attention, shared key-value heads, and fixed local attention windows.
This study isolates decode from prefill and jointly evaluates context length, token position, batching, and attention mechanism under identical hardware, precision, and measurement conditions.
The findings provide quantitative evidence and practical guidance for selecting energy-efficient architectures and inference configurations.

III. METHODOLOGY

The methodology compares four open-source LLMs spanning MHA, GQA, and GQA with SWA using decode-energy, token-latency, and request-latency metrics. The design supports systematic comparisons across attention mechanisms and model sizes.

The methodology is presented as an overall study workflow whose components are described in subsequent subsections.
The study evaluates OPT-1.3B, Gemma-2-2B, Phi-3 Mini, and Mistral-7B to represent different modern attention mechanisms.OPT-1.3B and Phi-3 Mini use MHA, Gemma-2-2B uses GQA, and Mistral-7B combines GQA with SWA.
Energy per generated token, latency per token, and latency per request are the three metrics used to evaluate decode-phase efficiency and performance.Energy per generated token is calculated by dividing total decode energy by the number of generated tokens.

C. Inference Setup

Inference uses fixed, tokenized prompts and separates prefill from autoregressive decode, while experiments run on dedicated NVIDIA hardware with an experimental configuration summarized in Table I. Figure 2 reports decode energy per generated token across context lengths.

A fixed English-text prompt is tokenized and truncated to each target context length to keep inputs consistent across models and configurations.
Prefill populates the KV cache and is excluded from measurement, whereas decode generates tokens autoregressively using the cached states.
Experiments run on a dedicated NVIDIA A100-SXM4-40GB GPU within a controlled software and workload environment.
Table I summarizes the complete experimental configuration.
Figure 2 compares decode energy per generated token across context lengths for the evaluated models.

E. Measurement and Analysis

GPU energy is measured with NVIDIA's cumulative energy counter immediately before and after decoding, with the difference converted to Joules.

GPU energy consumption is measured using NVIDIA Management Library through the nvmlDeviceGetTotalEnergyConsumption() API.The API reports cumulative GPU energy in millijoules.
Measurements are taken immediately before the decode phase and after the final output token is generated.
The difference between the two readings, converted to Joules, represents total decode-phase energy.

IV. RESULTS

The results section analyzes evaluation results across four research questions concerning LLM inference energy and efficiency.

The evaluation results address RQ1, RQ2, RQ3, and RQ4.
The section presents results from evaluations of various large language models.
The results are analyzed to address the paper's research questions.

A. Effect of Context Length on Decode Energy and Latency

Decode-energy scaling depends strongly on attention architecture: MHA grows substantially with context length, while GQA-based designs remain much flatter. Latency stays comparatively stable, so longer contexts primarily raise GPU energy rather than execution time.

3–4%: GQA-based models show much smaller decode-energy increases across the same context-length range.Gemma-2B increases 3.62%, and Mistral-7B increases 3.32%; Mistral reflects combined GQA and SWA.
MHA models attend over the full KV cache, while GQA shares key-value heads and SWA bounds attention to a local window.These designs produce different computation and memory-access behavior as context expands.
10.84–26.17 ms: latency per generated token remains comparatively stable across context lengths for the evaluated models.The reported ranges cover OPT-1.3B, Phi-3 Mini, Gemma-2B, and Mistral-7B.
Longer contexts primarily increase GPU energy rather than execution time because energy rises without a comparable latency increase.
17–18%: MHA models increase decode energy per generated token from 128 to 1800 context tokens.OPT-1.3B rises 17.92%, while Phi-3 Mini rises 16.98%.

B. Interplay Between Model Size and Attention Architecture

Model size and attention architecture affect different aspects of inference efficiency. Model size primarily sets baseline energy and latency, while attention design determines the additional cost of longer-context decoding.

Model size primarily determines the baseline energy required to generate each token.
1.5960 J: OPT-1.3B has the lowest energy consumption per token at context length 128.The ordering of absolute energy follows model size more closely than attention mechanism.
4.8905 J: Mistral-7B has the highest absolute energy consumption at context length 128 despite its efficient attention architecture.Its larger parameter count drives higher absolute energy than the other evaluated models.
10.84 ms: OPT-1.3B has substantially lower token latency than the other models, which require approximately 24–26 ms per token.
Model size sets baseline energy and latency, whereas attention mechanism determines scaling efficiency for longer contexts.The paper frames these as complementary factors for deployment selection.

C. Impact of Token Position on Decode Energy

Decode energy increases during generation for MHA models as the KV cache grows, while GQA produces flatter profiles and GQA with SWA nearly eliminates within-sequence drift.

Within-sequence energy drift is measured as the percentage change between token positions 50 and 950.Energy was measured every 100 generated tokens at context lengths 128, 512, and 1024, with each configuration repeated 10 times.
MHA models show increasing decode energy as generation progresses because each step attends over the expanding KV cache.OPT-1.3B drift ranges from 9.64% at context length 128 to 2.81% at 1024; Phi-3 Mini remains between 5.41% and 6.43%.
Gemma-2B's GQA profile remains nearly constant, with energy drift of 2.12%, 0.99%, and −0.32% across the three context lengths.Head sharing limits the additional computation introduced as the KV cache grows.
Mistral-7B has the flattest energy profile, with drift values of +1.02%, −0.69%, and −0.83%, within measurement variability.Its combination of GQA and SWA limits decoding to a fixed local attention window.
Across the evaluated mechanisms, MHA shows measurable drift, GQA reduces it, and GQA with SWA effectively eliminates it.The pattern applies both across initial context lengths and throughout token generation.

D. Impact of Batching on Energy and Latency Efficiency

Batching substantially improves decode efficiency across the evaluated models and contexts, reducing both energy per generated token and latency per request. Its benefits remain strongest for GQA-based models under long-context workloads.

Increasing batch size from 1 to 8 decreases energy consumption by more than 80% in nearly all configurations.Batching benefits both MHA- and GQA-based models, although the magnitude varies by architecture and context length.
MHA energy reductions decline from approximately 84% at context length 128 to approximately 67% at context length 1800.Gemma-2B and Mistral-7B maintain reductions above 82% across all evaluated contexts.
Longer contexts limit MHA batching gains because larger KV caches increase memory demand and reduce GPU parallelism.GQA alleviates this bottleneck by reducing the number of key-value heads, while GQA with SWA further limits attention work.
Batching reduces latency per request by approximately 86–87% at context length 128.The improvement occurs alongside energy savings through increased GPU utilization.
Batching is the most effective deployment-level optimization evaluated and requires no changes to the underlying model.It consistently improves energy efficiency and latency across the evaluated models and contexts.

V. CONCLUSION

This study characterizes decode-phase energy across four open-source language models with different attention mechanisms under identical hardware and measurement conditions. Attention architecture governs energy scaling, while batching consistently improves energy efficiency and latency.

The study systematically characterizes decode-phase energy across four open-source models with different attention mechanisms under identical hardware and measurement conditions.
Attention mechanism design is a primary factor determining how decode energy scales with context length.MHA models show steeper growth than GQA and GQA+SWA models.
Model size dominates absolute energy consumption, with smaller models consuming less energy per token regardless of attention mechanism.
MHA shows measurable within-sequence energy increases, GQA remains nearly flat, and SWA eliminates drift as the KV cache grows.
Batching consistently and dramatically reduces energy per token and latency per request across models and context lengths.The conclusion identifies batching as the most impactful optimization for deployed inference systems.