PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

Molka Chkir, Syed Muhammad Danish, Jos Höll, Arghavan Asad

arXiv:2608.25096v1cs.LGcs.AI

TL;DR

Inference energy is an important sustainability concern, but the effect of attention architecture on decode-phase energy scaling has not been systematically characterized. This study empirically evaluates four open-source LLMs across attention mechanisms and workloads, finding that attention architecture governs energy scaling while model size determines absolute consumption and batching improves efficiency.

Abstract

from arXiv · show

The growing adoption of large language models (LLMs) has raised increasing concerns about the energy consumption and environmental impact of inference. This paper presents a systematic empirical study of decode-phase energy consumption across representative open-source LLMs employing Multi-Head Attention (MHA), Grouped Query Attention (GQA), and Grouped Query Attention with Sliding Window Attention (SWA) to characterize how attention architecture influences decode-phase energy consumption under varying inference workloads. We evaluate four models across different context lengths, batch sizes, and generation workloads while measuring GPU energy using NVIDIA hardware counters. We examine the effects of context length, attention mechanism, Key-Value (KV) cache growth, and batching on decode-phase energy consumption. Results show that attention mechanism is the primary factor governing how decode energy scales with context length. MHA models exhibit substantially steeper energy growth than GQA models, whereas GQA with SWA maintains nearly constant energy consumption. We further show that model size primarily determines absolute energy consumption, while batching reduces both energy per generated token and request latency by up to 87%. These findings provide practical guidance for selecting energy-efficient LLM architectures and inference configurations.

I. INTRODUCTION

Inference energy is an important sustainability concern, but prior work offers limited evidence about how attention architecture shapes decode-phase scaling. This study systematically evaluates context length, attention design, KV-cache growth, and batching across representative open-source LLMs.

  • Inference may account for up to 90% of a model’s total lifecycle energy consumption, making efficient serving important for sustainable deployment.Source: §I · p. 1
  • Prior studies often measured training or inference energy using few models and fixed workloads, limiting insight into the factors governing inference energy.Source: §I · p. 1
  • MHA, GQA, and GQA with SWA differ in KV-state storage and access, but their effects on decode-energy scaling had not been systematically characterized.Source: §I · p. 1
  • The study examines how context length, attention mechanism, KV-cache growth during generation, and batching affect decode-phase energy.Source: §I · p. 1
  • Across four models, MHA shows steeper energy growth, GQA remains more stable, and GQA with SWA maintains nearly constant energy as context increases.The evaluated models are OPT-1.3B and Phi-3 Mini with MHA, Gemma-2-2B with GQA, and Mistral-7B with GQA plus SWA.Source: §I · pp. 1–2

II. RELATED WORK

Prior research established the importance of inference energy but generally did not isolate decode-phase scaling across attention architectures and jointly varying workloads. This work addresses that gap with controlled comparisons of MHA, GQA, and GQA with SWA.

  • Inference may represent up to 90% of lifecycle energy after deployment at scale, motivating greater attention to operational energy use.Source: §II · p. 2
  • Earlier inference benchmarks primarily reported aggregate measurements and did not isolate the decode phase or systematically study attention-related scaling.Source: §II · p. 2
  • MHA, GQA, and SWA were introduced as architectural approaches that respectively provide standard attention, shared key-value heads, and fixed local attention windows.Source: §II · p. 2
  • This study isolates decode from prefill and jointly evaluates context length, token position, batching, and attention mechanism under identical hardware, precision, and measurement conditions.Source: A. Novelty of this work · p. 2
  • The findings provide quantitative evidence and practical guidance for selecting energy-efficient architectures and inference configurations.Source: A. Novelty of this work · p. 2

III. METHODOLOGY

The methodology compares four open-source LLMs spanning MHA, GQA, and GQA with SWA using decode-energy, token-latency, and request-latency metrics. The design supports systematic comparisons across attention mechanisms and model sizes.

  • The methodology is presented as an overall study workflow whose components are described in subsequent subsections.Source: §III · p. 2, Figure 1 · p. 3
  • The study evaluates OPT-1.3B, Gemma-2-2B, Phi-3 Mini, and Mistral-7B to represent different modern attention mechanisms.OPT-1.3B and Phi-3 Mini use MHA, Gemma-2-2B uses GQA, and Mistral-7B combines GQA with SWA.Source: A. Selection of LLMs · p. 3
  • Energy per generated token, latency per token, and latency per request are the three metrics used to evaluate decode-phase efficiency and performance.Energy per generated token is calculated by dividing total decode energy by the number of generated tokens.Source: B. Sustainability Metrics · p. 3

C. Inference Setup

Inference uses fixed, tokenized prompts and separates prefill from autoregressive decode, while experiments run on dedicated NVIDIA hardware with an experimental configuration summarized in Table I. Figure 2 reports decode energy per generated token across context lengths.

  • A fixed English-text prompt is tokenized and truncated to each target context length to keep inputs consistent across models and configurations.Source: C. Inference Setup · p. 3
  • Prefill populates the KV cache and is excluded from measurement, whereas decode generates tokens autoregressively using the cached states.Source: C. Inference Setup · p. 3
  • Experiments run on a dedicated NVIDIA A100-SXM4-40GB GPU within a controlled software and workload environment.Source: D. Measurement Environment · p. 3
  • Table I summarizes the complete experimental configuration.Source: Table I · p. 3
  • Figure 2 compares decode energy per generated token across context lengths for the evaluated models.Source: Figure 2 · p. 4

E. Measurement and Analysis

GPU energy is measured with NVIDIA's cumulative energy counter immediately before and after decoding, with the difference converted to Joules.

IV. RESULTS

The results section analyzes evaluation results across four research questions concerning LLM inference energy and efficiency.

  • The evaluation results address RQ1, RQ2, RQ3, and RQ4.Source: §IV · p. 4
  • The section presents results from evaluations of various large language models.Source: §IV · p. 4
  • The results are analyzed to address the paper's research questions.Source: §IV · p. 4

A. Effect of Context Length on Decode Energy and Latency

Decode-energy scaling depends strongly on attention architecture: MHA grows substantially with context length, while GQA-based designs remain much flatter. Latency stays comparatively stable, so longer contexts primarily raise GPU energy rather than execution time.

B. Interplay Between Model Size and Attention Architecture

Model size and attention architecture affect different aspects of inference efficiency. Model size primarily sets baseline energy and latency, while attention design determines the additional cost of longer-context decoding.

C. Impact of Token Position on Decode Energy

Decode energy increases during generation for MHA models as the KV cache grows, while GQA produces flatter profiles and GQA with SWA nearly eliminates within-sequence drift.

  • Within-sequence energy drift is measured as the percentage change between token positions 50 and 950.Energy was measured every 100 generated tokens at context lengths 128, 512, and 1024, with each configuration repeated 10 times.Source: C. Impact of Token Position on Decode… · p. 5
  • MHA models show increasing decode energy as generation progresses because each step attends over the expanding KV cache.OPT-1.3B drift ranges from 9.64% at context length 128 to 2.81% at 1024; Phi-3 Mini remains between 5.41% and 6.43%.Source: C. Impact of Token Position on Decode… · p. 5
  • Gemma-2B's GQA profile remains nearly constant, with energy drift of 2.12%, 0.99%, and −0.32% across the three context lengths.Head sharing limits the additional computation introduced as the KV cache grows.Source: C. Impact of Token Position on Decode… · p. 5
  • Mistral-7B has the flattest energy profile, with drift values of +1.02%, −0.69%, and −0.83%, within measurement variability.Its combination of GQA and SWA limits decoding to a fixed local attention window.Source: C. Impact of Token Position on Decode… · p. 6
  • Across the evaluated mechanisms, MHA shows measurable drift, GQA reduces it, and GQA with SWA effectively eliminates it.The pattern applies both across initial context lengths and throughout token generation.Source: C. Impact of Token Position on Decode… · p. 6

D. Impact of Batching on Energy and Latency Efficiency

Batching substantially improves decode efficiency across the evaluated models and contexts, reducing both energy per generated token and latency per request. Its benefits remain strongest for GQA-based models under long-context workloads.

  • Increasing batch size from 1 to 8 decreases energy consumption by more than 80% in nearly all configurations.Batching benefits both MHA- and GQA-based models, although the magnitude varies by architecture and context length.Source: D. Impact of Batching on Energy and… · p. 7
  • MHA energy reductions decline from approximately 84% at context length 128 to approximately 67% at context length 1800.Gemma-2B and Mistral-7B maintain reductions above 82% across all evaluated contexts.Source: D. Impact of Batching on Energy and… · p. 7
  • Longer contexts limit MHA batching gains because larger KV caches increase memory demand and reduce GPU parallelism.GQA alleviates this bottleneck by reducing the number of key-value heads, while GQA with SWA further limits attention work.Source: D. Impact of Batching on Energy and… · p. 7
  • Batching reduces latency per request by approximately 86–87% at context length 128.The improvement occurs alongside energy savings through increased GPU utilization.Source: D. Impact of Batching on Energy and… · p. 7
  • Batching is the most effective deployment-level optimization evaluated and requires no changes to the underlying model.It consistently improves energy efficiency and latency across the evaluated models and contexts.Source: D. Impact of Batching on Energy and… · pp. 7–8

V. CONCLUSION

This study characterizes decode-phase energy across four open-source language models with different attention mechanisms under identical hardware and measurement conditions. Attention architecture governs energy scaling, while batching consistently improves energy efficiency and latency.

  • The study systematically characterizes decode-phase energy across four open-source models with different attention mechanisms under identical hardware and measurement conditions.Source: §V · p. 8
  • Attention mechanism design is a primary factor determining how decode energy scales with context length.MHA models show steeper growth than GQA and GQA+SWA models.Source: §V · p. 8
  • Model size dominates absolute energy consumption, with smaller models consuming less energy per token regardless of attention mechanism.Source: §V · p. 8
  • MHA shows measurable within-sequence energy increases, GQA remains nearly flat, and SWA eliminates drift as the KV cache grows.Source: §V · p. 8
  • Batching consistently and dramatically reduces energy per token and latency per request across models and context lengths.The conclusion identifies batching as the most impactful optimization for deployed inference systems.Source: §V · p. 8

Related papers

Loading 2608.25096v1…