PaperlayerView exact revision on arXiv

Source-linked AI summary

From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally

arXiv:2310.03003v1cs.CLcs.DC

TL;DR

LLM inference creates substantial computational and energy costs, but these costs have received less attention than training costs. The paper benchmarks LLaMA across model sizes, datasets, GPUs, and distributed sharding, reporting large hardware requirements and an energy–latency trade-off from power capping. Its results provide preliminary measurements for understanding inference resource use and motivate broader benchmarking.

  • Problem

    Inference compute and energy costs for increasingly deployed LLMs are less studied than training costs, despite the need to understand their resource utilization.

  • Method

    The study benchmarks different LLaMA sizes on NVIDIA V100 and A100 GPUs using Alpaca and GSM8K, including model-sharded inference across multiple GPUs and nodes.

  • Results

    At least 8 V100 GPUs or 4 A100 GPUs are required for meaningful LLaMA 65B inference, while reducing power from 250W to 175W increases inference time by 6.7% and reduces total energy by 23.21%.

  • Takeaways & Limitations

    The measurements support further analysis of LLM inference resource use across hardware, data, and optimization strategies, while indicating that appropriately applied power capping can reduce inference energy.

Abstract

from arXiv · show

Large language models (LLMs) have exploded in popularity due to their new generative capabilities that go far beyond prior state-of-the-art. These technologies are increasingly being leveraged in various domains such as law, finance, and medicine. However, these models carry significant computational challenges, especially the compute and energy costs required for inference. Inference energy costs already receive less attention than the energy costs of training LLMs -- despite how often these large models are called on to conduct inference in reality (e.g., ChatGPT). As these state-of-the-art LLMs see increasing usage and deployment in various domains, a better understanding of their resource utilization is crucial for cost-savings, scaling performance, efficient hardware usage, and optimal inference strategies. In this paper, we describe experiments conducted to study the computational and energy utilization of inference with LLMs. We benchmark and conduct a preliminary analysis of the inference performance and inference energy costs of different sizes of LLaMA -- a recent state-of-the-art LLM -- developed by Meta AI on two generations of popular GPUs (NVIDIA V100 \& A100) and two datasets (Alpaca and GSM8K) to reflect the diverse set of tasks/benchmarks for LLMs in research and practice. We present the results of multi-node, multi-GPU inference using model sharding across up to 32 GPUs. To our knowledge, our work is the one of the first to study LLM inference performance from the perspective of computational and energy resources at this scale.

I. INTRODUCTION

LLM deployment has expanded rapidly, but inference requires substantial time, computation, and energy that has received less attention than training. The paper studies these costs through LLaMA inference experiments and aims to support broader benchmarking of real hardware.

  • LLM deployment has expanded across education, government, engineering, law, finance, and other domains.
  • Prior work has estimated training compute and energy costs, while this paper focuses on the resources required for LLM inference.
  • The study benchmarks LLaMA inference to characterize compute performance and energy use on real hardware.
  • The authors seek to encourage further benchmarking across large models, hardware, data, and optimization strategies.
  • LLaMA is presented as a recent state-of-the-art model whose scale and complexity resemble contemporaneous systems such as Bard and GPT-4.

A. Large Language Models & Large Foundation Models

LLMs are resource-intensive language models within the broader category of multimodal foundation models. The paper uses LLaMA, including its largest 65B variant, as a representative state-of-the-art model for studying inference.

  • LLMs typically contain hundreds of millions to billions of parameters and are trained on extremely large text datasets.
  • Large foundation models extend beyond language to modalities such as images, video, and image generation.
  • LLaMA is a Meta AI transformer-based LLM whose performance rivaled or exceeded GPT-3 on many NLP benchmarks.
  • The study compares LLaMA 7B and 13B as smaller baselines while focusing on LLaMA 65B for inference analysis.
  • LLaMA 65B has 65 billion parameters, 80 layers, 64 attention heads, and was trained over 1.4 trillion tokens.

III. EXPERIMENTAL SETUP

The experiments use the MIT Supercloud heterogeneous HPC system and evaluate inference on NVIDIA GPU-equipped compute nodes. Each listed node contains two CPUs and two GPUs.

  • Experiments were conducted on the MIT Supercloud HPC system, which contains 448 compute nodes.
  • Each compute node has dual Intel Xeon Gold 6248 CPUs, 384 GB of RAM, and two NVIDIA V100 GPUs with 32 GB each.
  • The compute-node configuration table lists hardware types used for inference evaluations, with two CPUs and two GPUs per node.

A. Models

The evaluation varies datasets, model variants, decoding settings, and GPU monitoring measures to characterize inference performance and energy. Energy metrics are computed from aggregate GPU energy and generated outputs.

  • A. Models: Inference uses decoder temperature τ = 0.8 and top-p = 0.95 with model sharding across multiple GPUs and nodes.
  • B. Datasets: The experiments use Alpaca instruction-following tasks and GSM8K grade-school mathematics problems to evaluate diverse data effects.
  • B. Datasets: For both datasets, the study samples 4,096 inputs rather than running the complete datasets.
  • A. Models: The analysis evaluates LLaMA 65B and includes limited comparisons with smaller variants under bare-minimum hardware settings.
  • C. Evaluation: Inference performance is measured by words, tokens, and responses generated per second.
  • C. Evaluation: GPU utilization, energy, and power draw are monitored with nvidia-smi and NVIDIA DCGM during experiments capped at 250 Watts unless stated otherwise.
  • C. Evaluation: Energy per second, token, and response are calculated by dividing total aggregate GPU energy by runtime, decoded tokens, or responses.

1) Inference Performance:

The study compares LLaMA inference performance across model sizes and GPU generations under minimum hardware settings. A100 GPUs outperform V100s, especially for smaller models, while 65B gains are smaller because sharding adds latency.

  • Baseline setup: The baseline compares LLaMA 7B, 13B, and 65B under the minimum resources needed for inference.The analysis examines performance and energy trade-offs for users and researchers with limited computational resources.
  • Baseline setup: LLaMA 7B uses one GPU, 13B uses two GPUs, and 65B uses 8 V100 or 4 A100 GPUs.The experiments use batch size 64 and maximum generation length 256 under bare-minimum settings.
  • GPU comparison: A100 GPUs outperform V100s on Alpaca and GSM8K across words per second, tokens per second, and responses per second.The reported improvement ranges from 2 times for 7B to 1.25 times for 13B.
  • GPU comparison: The 65B model shows a much smaller A100-over-V100 improvement because sharding across multiple nodes can add latency to each forward pass.The 65B model requires sharding across two A100 or four V100 compute nodes at minimum.
  • Dataset comparison: A100 throughput improves more for Alpaca than GSM8K for LLaMA 7B, consistent with differences in input complexity between the datasets.

2) Inference Energy:

The paper measures inference energy per second across LLaMA sizes, datasets, GPU types, batch sizes, and shard counts. A100 throughput gains can require higher wattage, while 65B inference reaches roughly 300 Watts to 1 Kilowatt as sharding increases.

  • Baseline energy: Figure 3 compares energy per second for LLaMA 7B, 13B, and 65B on V100 and A100 GPUs using Alpaca and GSM8K.Bars show average quantities, lines show error bars, and values are plotted on a log scale.
  • Baseline energy: A100 inference uses considerably more energy per second than V100 inference across all LLaMA sizes, with the largest increase for 7B.Energy is averaged over maximum generation lengths of 256, 512, and 1024 because trends were nearly identical across lengths.
  • Baseline energy: For 65B, higher A100 energy per second is less clearly justified by its small throughput improvement over V100.The comparison concerns words per second, tokens per second, and responses per second.
  • 65B scaling: Increasing the number of shards always increases wattage for LLaMA 65B, including at the same batch size.Energy per second rises on average with shard count, while its relationship with batch size is slight.
  • 65B scaling: LLaMA 65B inference consumes about 300 Watts to 1 Kilowatt across configurations ranging from 8 to 32 GPUs.Increasing maximum generation length from 512 to 1024 tends to increase energy per second, but the overall effect is less consistent.

C. Energy per Decoded Token: LLaMA 65B

For LLaMA 65B, energy per decoded token generally increases with the number of shards, while maximum generation length from 512 to 1024 has little effect. GSM8K shows a conditional 16-shard batch-size sweet spot at length 512 that disappears at length 1024.

  • Energy per output token increases as the number of shards increases.
  • 3-4 Joules per output token is observed at maximum generation length 512, with little average change at length 1024.
  • GSM8K exhibits a 16-shard sweet spot where increasing batch size can reduce energy per token at maximum generation length 512.
  • At maximum generation length 1024, increasing batch size increases GSM8K energy per token at 16 shards, eliminating the shorter-length sweet spot.
  • The energy-per-token estimates cover batch sizes of 64, 128, 256, and 512 across 8, 16, and 32 shards on Alpaca and GSM8K.

E. Effects of GPU Power Capping on LLaMA 65B

The paper evaluates GPU power capping for LLaMA 65B inference and finds that a moderate cap can reduce total energy with a smaller time penalty. Broader recommendations remain constrained because a static cap may not work equally across workloads.

  • 23.21% average total-energy reduction accompanies a 6.7% average inference-time increase when the power cap drops from 250W to 175W.
  • 19.49% average inference-time increase occurs when the power cap is reduced to 150W.
  • A static power cap may not be equally effective across GPU workloads, so broader recommendations require additional experimentation.
  • The power-capping experiments use LLaMA 65B on GSM8K with batch size 64, output lengths 256, 512, and 1024, and NVIDIA A100 GPUs.

F. GPU Resource Utilization under Distributed Inference

Distributed LLaMA 65B inference uses GPU compute resources heavily but leaves substantial memory capacity unused. The observed memory under-utilization suggests opportunities to co-locate models or workloads on the same GPUs.

  • The utilization study uses batch size 64 and maximum generated output length 256, including four A100 GPUs and 8, 16, or 32 V100 GPUs.
  • 94%-95% streaming-multiprocessor utilization is observed across the evaluated distributed-inference configurations.
  • A100 SM utilization rises to 98% when maximum generated output length increases to 2048.
  • 23%-27% memory utilization is observed on four 80GB A100 GPUs, varying with maximum generated output length.
  • Memory under-utilization may permit co-locating multiple models to increase aggregate throughput or improve system utilization.

V. DISCUSSION

The discussion frames LLaMA inference benchmarking across model sizes, datasets, hardware, and distributed configurations, emphasizing substantial hardware requirements and resource-sharing opportunities. It also identifies power capping as a promising but preliminary energy intervention and calls for broader benchmarking.

  • The benchmark compares 7B, 13B, and 65B LLaMA models across varying model parameters, input data, and hardware configurations.
  • Dataset complexity can affect model performance under a given set of hyperparameters and hardware configuration.
  • 8 V100 GPUs with 32 GB each or 4 A100 GPUs with 80 GB each are required for meaningful LLaMA 65B inference.
  • 20%-25% of GPU memory is used during the reported 65B experiments despite evenly sharding the model across all GPUs.
  • The preliminary analysis suggests GPU power capping can reduce inference energy, while broader recommendations require further experiments.
Loading 2310.03003v1…