PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models

Soham Poddar, Paramita Koley, Janardan Misra, Sanjay Podder, Niloy Ganguly, Saptarshi Ghosh

arXiv:2502.05610v2cs.CL

TL;DR

Inference energy has received less attention than training energy, despite repeated deployment calls contributing to a model’s lifetime energy requirements. This paper benchmarks inference across diverse LLMs and NLP tasks, finding that energy tracks response time and is more strongly influenced by output than input length. It also identifies deployment approaches, including quantization, batching, and targeted prompt phrases, that can reduce energy use.

  • Problem

    Inference energy has been studied less than training energy, even though repeated deployment calls can make it significant over a model’s lifetime.

    Source: §1 · p. 1

  • Method

    The study benchmarks LLM inference across diverse NLP tasks and analyzes model, task, prompt, and system-related factors.

    Source: §4 · p. 9

  • Results

    Inference energy strongly correlates with response time, and output length has a stronger influence on energy use than input size.

    Source: §4 · p. 9

  • Takeaways & Limitations

    Response time can serve as an energy proxy, while input compression, output optimization, targeted phrases, and quantization are recommended for resource-constrained settings.

    Source: §4 · p. 9

    The controlled benchmarks may not capture real-world deployment variability, other domains, network latency, or hardware-specific optimizations.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment.

1 Introduction

Inference energy merits separate study because repeated deployment calls can make its lifetime energy impact significant, although prior work focused mainly on training. This paper benchmarks diverse models, tasks, prompts, and system factors, extending earlier studies that covered fewer settings.

1 Introduction

  • Repeated deployment inferences can make energy use significant over a model’s lifetime, motivating study beyond the training phase.Earlier work focused mainly on training energy.Source: §1 · p. 1

1.1 Literature survey

  • Prior inference-energy studies were often limited in model and task diversity, and preliminary work mostly ignored the performance-energy tradeoff.Earlier work included studies of question answering, Llama models on different GPUs, and fine-tuning versus few-shot learning.Source: §1.1 · pp. 1–2

1.2 Our contributions & differences with prior works

  • The study benchmarks encoder-decoder and decoder-only LLMs on discriminative and generative NLP tasks, varying models, tasks, prompts, and system factors.It analyzes token lengths, response time, model size and complexity, batch size, quantization, and prompt editing.Source: §1.2 · p. 2
  • The authors examine accuracy-energy tradeoffs across tasks and models and provide efficiency guidelines, including combined quantization and larger batches under fixed memory constraints.Their analysis also varies input and output lengths separately to compare their influence on inference energy.Source: §1.2 · p. 2

2 Experimental setup

The experiments cover ten LLMs from two architecture families and a varied set of NLP tasks, using a common sample set per dataset. Energy is measured on GPU and CPU, with a normalized accuracy metric for cross-task model comparisons.

2.1 Models

  • The benchmark includes six decoder-only GPT-style models and four encoder-decoder Flan-T5 models, spanning 248M to 13B parameters.The decoder-only models range from Tiny-Llama to Llama-2-13B; the encoder-decoder models range from Flan-T5-base to Flan-T5-xxl.Source: §2.1 · pp. 2–3

2.2 Tasks and Datasets

  • Tasks span generative, question-answering, classification, and single-sentence NLP, with 1,024 randomly selected samples per dataset reused across experiments.Summarization uses the average of ROUGE-1, ROUGE-2, and ROUGE-L; other tasks use an F1 variant.Source: §2.2 · p. 3, Table 2 · p. 3
  • Normalized Accuracy applies per-dataset Z-score normalization and a sigmoid, then averages model scores across datasets and scales them to 100.The metric depends on which models are included, so it supports comparisons within the evaluated model set.Source: §2.2 · p. 3

2.3 Hardware and Energy metrics

  • Experiments primarily ran on one A6000 server; energy tools tracked GPU and CPU use over 1,024 samples, with batch size 8 except on CNN-DM and SAMSUM.CodeCarbon also estimates RAM power; PUE was fixed at 1.0, so actual energy use is higher than reported.Source: §2.3 · pp. 3–4

3 Factors Affecting Energy / Accuracy

This section examines how response time, token lengths, task complexity, model family and size, batching, quantization, and prompt wording relate to inference energy and accuracy. Energy patterns vary across these factors, with output length and system choices especially informative.

3.1 Response time

  • Energy strongly correlates with response time (Pearson r = 0.996, Spearman rs = 0.968), which may proxy energy when location, grid, and model are fixed.The relationship’s slope differs across datasets.Source: §3.1 · p. 4

3.2 Input and Output token length

  • Output length correlates more strongly with energy than input length (Pearson r = 0.952 vs 0.697), while response time also correlates strongly with energy (Pearson r = 0.996).The authors attribute the stronger output-length relationship to sequential generation, compared with input processing that can use KV caching and GPU parallelism.Source: §3.1 · p. 4, §3.2 · p. 5
  • Energy increases linearly with generation length: generating 2 tokens takes only about 12% more energy than generating 1, because the first step processes the entire input while later steps use cached K-V computations.Source: §3.2 · pp. 5–6

3.3 Task complexity

  • With input and output lengths held fixed, energy differences were below 1% for VAX-STANCE versus CAVES and below 1.3% for summarization versus returning three sentences.The authors attribute the small differences to fixed per-token computational steps without additional task-dependent branches.Source: §3.3 · p. 6, Table 3 · p. 6

3.4 Model family and size

  • Energy rises linearly with model size, while encoder-decoder models typically use less energy than decoder-only models with comparable parameter counts.The authors attribute the family difference to encoder-decoder models having a decoder half the size during autoregressive decoding; model performance also varies by size and family.Source: §3.4 · pp. 6–7, Figure 4 · p. 7

3.5 Batch size and Quantization

  • Increasing batch size reduces per-sample inference energy, but GPU VRAM limits feasible batches and makes the energy-efficient batch size model- and dataset-dependent.The study reports a 48GB A6000 GPU and out-of-memory errors for some larger model and dataset combinations.Source: §3.5 · p. 7, Figure 5 · p. 8
  • 4-bit quantization alone raises energy to almost 2×, but pairing it with batch size 256 reduces energy to about 0.33× that of the original 16-bit model at batch size 64.Quantization changes performance by less than 5% for some models.Source: §3.5 · p. 8

3.6 Effect of targeted phrases in prompts on inference energy

  • Targeted prompt phrases reduce energy for Mistral and Llama-2, but performance often degrades; the authors identify Mistral-7B and Llama-2-13B as useful without affecting performance.Benefits are less pronounced when outputs are already short, and prompts can raise energy if they add input without reducing generation.Source: §3.6 · pp. 8–9, Figure 6 · p. 8

4 Concluding discussions

The study benchmarks inference power consumption across diverse NLP tasks and identifies energy patterns and deployment choices relevant to resource-constrained settings.

4 Concluding discussions

  • Inference energy strongly correlates with response time; output length has a stronger influence than input size, while task complexity has little independent impact.The takeaways also recommend response time as a lower-overhead proxy for estimating energy.Source: §4 · p. 9
  • Encoder-decoder models are recommended for energy efficiency and performance, while batch size should be optimized for each model-task pair within GPU memory limits.Quantization can permit larger batches and reduce energy without much accuracy degradation.Source: §4 · p. 9
  • Targeted phrases reduce energy for older decoder-only models on discriminative tasks by restricting their outputs.Source: §4 · p. 9
  • In energy-constrained settings, input compression and output optimization are recommended, along with response time as a lower-overhead energy proxy.Removing energy-tracking libraries and recording system time reduced inference time by 15%–50% in different scenarios.Source: §4 · p. 9

Limitations

The study’s controlled benchmarking may not capture real-world deployment variability, other domains, or all factors that affect inference energy.

Limitations

  • Controlled experiments may not reflect deployment variability across hardware, infrastructure, and operational contexts.Source: Limitations · p. 9
  • Results may not generalize beyond the specific NLP tasks studied to domains such as vision or time-series analysis.Source: Limitations · p. 9
  • The study omits potential energy influences including network latency and hardware-specific optimizations.Source: Limitations · p. 9

Ethical Considerations

The paper reports substantial energy use during its experiments and reviews prior sustainability, measurement, and LLM inference-energy work. It positions its benchmark as broader in model, task, and configuration coverage than earlier studies.

Ethical Considerations

  • About 3000 kWh were used across repeated and pilot experiments, so the authors limited testing to 1024 sampled examples per dataset.The experiments involved 1024 inferences for 11 datasets over 10 models in several configurations, with multiple repetitions.Source: Ethical Considerations · p. 9

A Additional Literature Survey

  • Prior work addresses energy measurement tools and reduction approaches including prompt directives, model cascades, approximation, and quantization.Tool studies compare software-based meters with physical power meters and report differences in their agreement with wattmeter readings.Source: A Additional Literature Survey · pp. 12–13
  • Earlier LLM inference-energy studies covered particular models, tasks, or GPU architectures, and preliminary work often omitted performance-energy trade-offs.Source: A Additional Literature Survey · p. 13
  • The paper presents a comprehensive benchmark spanning token lengths, response time, model size and family, task complexity, batch size, quantization, and prompt features.Source: A Additional Literature Survey · p. 13

G Testing on other systems

Limited experiments on two systems compared representative encoder-decoder and decoder-only models over four datasets. Energy trends were similar across systems, although magnitudes differed.

G Testing on other systems

H Energy proxy for black-box models

For black-box API models, response time alone cannot reliably estimate energy because network latency, concurrent requests, and batch processing vary; averaging repeated runs may help compare strategies but does not remove latency effects.

H Energy proxy for black-box models

  • API energy is difficult to estimate from response time alone because network latency, concurrent requests, and batch scheduling or processing vary, and providers do not report energy metrics.The paper identifies inference time as a possible basis for comparing black-box strategies, not as a reliable absolute energy estimate.Source: H Energy proxy for black-box models · p. 14
  • Averaging response times across repeated runs may help compare black-box strategies under the same hardware, but it does not eliminate network-latency effects.The paper notes that latency is not a mean-zero random variable and is influenced by complex hardware-related factors.Source: H Energy proxy for black-box models · p. 14
  • Almost 2× energy for 4-bit precision versus default at the same batch size; at BS=256, a 4-bit model uses about 0.33× the energy of the default model at BS=64.Source: H Energy proxy for black-box models · p. 17

Related papers

Loading 2502.05610v2…