PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

Pol G. Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Torres, Josep Ll. Berral

arXiv:2503.08311v2cs.DCcs.LG

TL;DR

Large-batch LLM inference plateaus despite batching, and the paper investigates whether this reflects a transition to compute-bound execution. Through GPU profiling, it identifies persistent DRAM-bandwidth saturation, then proposes BCA to reduce unnecessary memory allocation and use freed capacity for concurrent replicas.

  • Problem

    Large-batch inference exhibits throughput plateaus and rising latency, but whether this reflects a transition from memory-bound to compute-bound execution has not been rigorously validated.

  • Method

    The paper combines GPU tracing with a Batching Configuration Advisor that selects batch size and memory allocation under latency constraints, then evaluates concurrent model replicas.

  • Results

    Large-batch inference remains memory-bound because DRAM bandwidth saturation leaves compute resources underutilized, while replication increases throughput by 33.72% for OPT-1.3B and 12.78% for OPT-2.7B.

  • Takeaways & Limitations

    Freed GPU memory from BCA can support concurrent workloads through model replication, improving resource utilization and overall serving throughput.

  • Takeaways & Limitations

    The memory-versus-compute classification depends on arithmetic intensity, with low ratios indicating memory-bound execution and high ratios indicating compute-bound execution.

Abstract

from arXiv · show

Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference. While batching is commonly used to increase throughput, performance gains plateau beyond a certain batch size, especially with smaller models, a phenomenon that existing literature typically explains as a shift to the compute-bound regime. In this paper, through an in-depth GPU-level analysis, we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck. To address this, we propose a Batching Configuration Advisor (BCA) that optimizes memory allocation, reducing GPU memory requirements with minimal impact on throughput. The freed memory and underutilized GPU compute capabilities can then be leveraged by concurrent workloads. Specifically, we use model replication to improve serving throughput and GPU utilization. Our findings challenge conventional assumptions about LLM inference, offering new insights and practical strategies for improving resource utilization, particularly for smaller language models. The code is publicly available at https://github.com/FerranAgulloLopez/vLLMBatchingMemoryGap.

I. INTRODUCTION

Large-batch LLM inference plateaus because decoding remains memory-bound: DRAM bandwidth saturation limits attention despite increased batching. BCA selects a balanced batch size and reallocates freed memory to concurrent model replicas.

  • Small models can fit hundreds or thousands of requests, but throughput gains diminish beyond a knee point while latency increases.
  • Over 50% of attention kernel cycles stall on data access, and attention remains memory-bound across all tested batch sizes.Matrix multiplication gains arithmetic intensity with batching, whereas xFormers and FlashAttention attention kernels remain nearly constant.
  • DRAM bandwidth saturation consumes substantial GPU memory without producing proportional throughput gains.
  • BCA recommends an optimal batch size and memory allocation subject to latency constraints, avoiding the throughput plateau.
  • Freed memory can host concurrent model replicas, increasing GPU utilization and throughput by overlapping operations and mitigating idle time.

II. BACKGROUND

Decoder-only LLMs generate tokens autoregressively through transformer blocks containing self-attention and feed-forward layers. During decoding, the KV cache avoids recomputation but creates substantial memory-transfer demands relative to computation.

  • Decoder-only models generate tokens autoregressively from an input prompt using next-token prediction.
  • Transformer blocks use self-attention to model token relationships and feed-forward layers to process representations.
  • Prefill processes input tokens in parallel, whereas decode generates subsequent tokens one at a time using previously generated tokens.
  • The KV cache stores key-value pairs so decode can reuse attention states and reduce attention computation from matrix-matrix to matrix-vector multiplication.
  • Decode transfers substantial key-value pairs and model weights despite minimal computation, creating a primary inference bottleneck.

B. Memory vs. Compute Performance Limitations

Arithmetic intensity distinguishes memory-bound from compute-bound operations, but the usual explanation of large-batch plateaus as compute-bound is challenged by profiling evidence. Throughput plateaus while latency rises and only part of the KV cache is needed to reach near-maximum throughput.

  • Arithmetic intensity is the FLOP-to-memory-byte ratio: low values indicate memory-bound execution, while high values indicate compute-bound execution.
  • The conventional stage-level view classifies prefill as compute-bound and decode as memory-bound, encouraging a compute-bound interpretation of large-batch plateaus.
  • Profiling tools expose GPU activity and kernel-level execution to investigate the plateau’s underlying bottleneck.
  • Beyond a knee point, larger batches provide diminishing throughput gains while significantly increasing inter-token latency.
  • OPT-1.3B reaches almost maximum throughput using 40% of its KV cache, while OPT-2.7B requires 50%.

A. Serving Language Models

LLM serving systems combine scheduling, batching, and memory or compute optimizations to improve throughput and latency. The paper emphasizes that batching has a knee point and that KV-cache allocation is difficult because output lengths are unpredictable.

  • Serving systems such as Orca, TGI, DeepSpeed-FastGen, Sarathi-Serve, and vLLM target efficient language-model deployment.
  • Figure 2 compares input and output token throughput with inter-token latency across batch sizes for four models, marking KV-cache capacity exceedance.
  • Figure 3 compares throughput with maximum KV-cache usage as maximum batch size increases across the same four models.
  • Quantization, mixture-of-experts, sparsity, offloading, speculative decoding, and multi-query attention reduce memory or computation through different trade-offs.
  • Unpredictable output lengths make efficient KV-cache preallocation difficult and can cause fragmentation when memory is reserved for maximum outputs.

D. LLM Inference Profiling

The paper profiles large-batch LLM inference with GPU tracing and finds that the throughput plateau is caused primarily by DRAM saturation in attention, not a shift to compute-bound execution.

  • Profiling methodology: GPU tracing with Nsight Systems and Nsight Compute is used to characterize the large-batch throughput plateau and its underlying causes.The experiments use vLLM on a single NVIDIA H100 across four models and online/offline workloads.
  • Main finding: DRAM saturation in the attention mechanism is the primary cause of the throughput plateau, challenging the assumption that large batches become compute-bound.Batching raises matrix-multiplication arithmetic intensity, but attention arithmetic intensity remains nearly constant, leading to memory-bandwidth saturation.

A. Decode vs Prefill

Decode dominates inference time and becomes increasingly constrained by attention, DRAM access, and CPU gaps as batch size grows. These effects explain the throughput plateau and leave substantial GPU capacity underutilized.

  • A. Decode vs Prefill: Decode accounts for most inference time across tested batch sizes and models, while prefill remains below 5% at maximum batch size for OPT-2.7B.The analysis therefore focuses on decode as the primary serving bottleneck.
  • A. Decode vs Prefill: 6x slowdown occurs at the largest OPT-2.7B batch size after execution time begins increasing beyond 32 requests.Throughput rises from 225 tokens per second at batch size 1 to 7,607 tokens per second at batch size 256, an approximate 33.8x increase rather than 256x.
  • B. Decode Kernels: At large batch sizes, average GPU utilization remains significantly below 50% even when peak DRAM and compute utilization approach saturation.The pattern indicates stalls that prevent full resource utilization during decoding.
  • B. Decode Kernels: In OPT-1.3B, attention rises from approximately 5% to over 40% of decode-step execution time as batch size grows, while matrix multiplications fall from around 50% to under 10%.CPU computations also reach up to 30% at batch size 512, adding to GPU underutilization.
  • B. Decode Kernels: DRAM read saturation occurs exclusively during attention kernels at larger batch sizes and correlates with warp unallocation in Llama-2-7B.This links the growing attention cost to a DRAM bottleneck that prevents available GPU warps from running.

C. Attention Kernel

The attention mechanism remains memory-bound across batch sizes because DRAM reads, rather than computation, limit performance. Increasing the batch size therefore cannot overcome bandwidth saturation, while cache inefficiency and idle cycles leave compute capacity underused.

  • Memory-bound behavior: Both xFormers and FlashAttention remain memory-bound across batch sizes, with performance orders of magnitude below the hardware roofline.Arithmetic intensity stays nearly constant at 0.5–1 operations per byte, unlike matmul kernels whose intensity increases with batching.
  • Memory access inefficiency: L1 cache hit rates average no more than 12%, while L2 hit rates average no more than 2%, and both decline as batch size increases.These low hit rates indicate increasingly inefficient memory access patterns.
  • Memory access inefficiency: At maximum batch size, more than 50% of attention cycles remain idle while waiting for data, exceeding 80% for xFormers across all models.The measurements compare batch size 1 with the maximum batch size across tested models and attention implementations.
  • Memory-bound behavior: DRAM bandwidth saturation in the attention mechanism prevents further throughput improvement at larger batch sizes.At maximum batch size, attention aligns with the DRAM bandwidth limit.

VI. BATCHING CONFIGURATION ADVISOR

The Batching Configuration Advisor selects a batch size that balances throughput against latency and memory use. It uses offline profiling to avoid the throughput plateau while satisfying user-defined service constraints.

  • Advisor design: BCA determines the optimal batch size Bopt by jointly considering the throughput plateau and a user-defined latency constraint.The advisor is computed offline before online deployment.
  • Advisor design: BCA selects a batch size that maximizes throughput, avoids diminishing returns, and satisfies a predefined Service Level Objective.The selected configuration also reduces GPU memory usage and frees resources for other workloads.
  • Optimization objective: Bopt maximizes T(B) subject to L(B) meeting the SLO and throughput relative to T(1) ∗B remaining above threshold ϵ.T(B) and L(B) are obtained by benchmarking the model at each batch size, while SLO and ϵ are user-defined.

A. Evaluation of BCA

BCA identifies batch sizes that meet throughput and latency constraints while reducing memory use, with the largest benefits for smaller models. Its memory savings diminish as output sequences become unusually long.

  • BCA evaluation: BCA evaluation uses ϵ = 0.1 under strict and relaxed latency SLOs to identify batch sizes that avoid diminishing throughput returns.The strict SLO is 2× the latency at batch size 32, while the relaxed SLO is 4× that latency.
  • Memory savings: BCA benefits smaller models most, while Llama-2-13B uses all available memory and does not reach the throughput plateau in the evaluated hardware setup.Effectiveness depends on model size, available GPU memory, and achievable batch size.
  • Sequence-length effects: For OPT-1.3B, 520 requests use 20% of the KV cache at 130 output tokens but over 80% at 520 output tokens.Thus, BCA’s memory gains diminish for unusually long outputs.

B. Model Replication

Model replication uses memory freed by BCA to overlap replica execution and improve GPU utilization. Replication can increase throughput, but its benefits depend on available memory, CPU gaps, and the number of replicas.

  • Replication rationale: Replicas can overlap DRAM-bound attention periods with execution phases of other replicas, increasing overall performance.The strategy uses multiple model servers, each allocated an equal portion of GPU memory.
  • Replication configurations: FCFS replication fills GPU gaps during CPU computation, while NVIDIA MPS runs replica kernels in parallel.These are the two evaluated replication configurations.
  • Memory constraints: Llama-2-7B and Llama-2-13B cannot be replicated at their BCA batch sizes because duplicating the required memory exceeds GPU capacity.The number of replicas is increased incrementally until available GPU memory is fully utilized.
  • Performance trade-offs: Replication increases inter-token latency by an average of 28% but decreases end-to-end latency while increasing overall output token generation.The inter-token increase remains below that of the maximum batch size configuration.
  • Resource utilization: Two replicas reduce CPU time by an average of 78% across the evaluated models, while scaling from two to four replicas in OPT-1.3B yields diminishing throughput gains.The limited additional gain follows substantial CPU-gap reduction with two replicas.

VII. DISCUSSION

The study finds that large-batch inference remains limited by DRAM bandwidth saturation rather than becoming compute-bound, leaving compute resources underutilized. BCA selects latency-constrained batch sizes and enables replication to improve resource utilization, while the analysis remains focused on smaller single-GPU models.

  • Bottlenecks: DRAM bandwidth saturation remains the principal bottleneck as batch size increases, leaving most GPU compute resources underutilized.The study attributes the slowdown beyond the batch-size knee point to nearly constant attention-kernel arithmetic intensity and saturated DRAM bandwidth.
  • Bottlenecks: CPU overhead can reach 30% of total execution time in some cases, further limiting scalability at large batch sizes.The reported CPU overhead contributes an additional constraint beyond GPU memory-access stalls.
  • Batching Configuration Advisor: BCA estimates a throughput-maximizing batch size subject to a user-defined latency constraint and allocates only the memory needed for that configuration.The advisor operates offline under an assumption that all requests arrive simultaneously, making it an upper-bound estimator for variable request patterns.
  • Concurrent serving: 33.72% higher throughput for OPT-1.3B with four replicas and 12.78% for OPT-2.7B with two replicas demonstrate the benefit of concurrent model instances.Replication primarily mitigates GPU idle cycles caused by CPU bottlenecks at large batch sizes.
  • Concurrent serving: Replication and freed memory from BCA support more efficient GPU sharing for concurrent workloads, including multi-model serving in shared cloud environments.The paper identifies heterogeneous workloads and larger models across multiple GPUs as directions requiring further study.

VIII. CONCLUSION

The paper identifies DRAM bandwidth saturation as the main cause of large-batch throughput plateaus and proposes BCA and GPU sharing to improve efficiency. These strategies target smaller-model serving by selecting efficient batch sizes and using freed resources for concurrent instances.

  • Findings: Large-batch inference remains memory-bound because DRAM bandwidth saturation leaves significant GPU compute resources underutilized.This challenges the assumption that large batches transition to a compute-bound regime and fully utilize compute resources.
  • Approach: BCA determines an optimal batch size and avoids unnecessary GPU memory allocation for more efficient serving.The freed memory can support concurrent workloads through GPU sharing techniques such as time-sharing and MPS.
  • Implications: Replicating smaller LLMs improves GPU utilization and serving throughput by overlapping operations and mitigating DRAM saturation.The conclusion presents GPU sharing as a practical strategy for improving resource utilization.
Loading 2503.08311v2…