PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Inference Unveiled: Survey and Roofline Model Insights

Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, Kurt Keutzer

arXiv:2402.16363v6cs.CLcs.AI

TL;DR

Efficient LLM inference research is growing, but it lacks a concise, practical framework for unified analysis. The paper surveys the field and introduces a Roofline-based approach with LLM-Viewer to identify deployment bottlenecks. It organizes methods across model, algorithmic, system, and hardware dimensions, while highlighting reported advances and explicit scope limitations.

  • Problem

    The expanding efficient-LLM-inference literature lacks a systematic, practical framework for unified analysis and comprehensive solution development.

  • Method

    The paper develops a Roofline-based LLM-Viewer and surveys efficiency strategies across model compression, decoding algorithms, compiler/system optimization, and hardware optimization.

  • Results

    The survey provides a practice-driven overview and uses Roofline analysis to pinpoint deployment bottlenecks across efficient-inference methods.

  • Takeaways & Limitations

    The framework supports selecting more specific deployment strategies by relating LLM inference methods to hardware and system bottlenecks.

  • Takeaways & Limitations

    QLoRA is restricted to at most 4-bit quantization during fine-tuning, while lower-bit quantization such as 2-bit can significantly deteriorate performance.

Abstract

from arXiv · show

The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems, such as why LLMs are memory-bound, how much memory and computation they need, and how to choose the right hardware. We systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as model compression (e.g., Knowledge Distillation and Quantization), algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Our survey stands out by analyzing these methods with roofline model, helping us understand their impact on memory access and computation. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The analyze tool, LLM-Viewer, is open-sourced.

1. Introduction

LLM inference research is expanding rapidly, but existing literature lacks a unified, practical framework for analyzing deployment challenges. This survey addresses the gap with a Roofline-based analysis, LLM-Viewer, and a four-part organization of efficiency strategies.

  • Motivation: Models with tens of billions of parameters create substantial inference challenges across resource-constrained devices and advanced hardware.Their scale also raises concerns about energy consumption, scalability, and accessibility.
  • Research gap: The rapidly growing efficient-inference literature lacks a systematic and practical framework for unified analysis and comprehensive solution development.The survey positions this absence as a critical gap amid an expanding research field.
  • Contribution: The survey combines a practice-driven overview with a specifically developed Roofline model for analyzing deployment bottlenecks.The model is presented as supporting practical application and optimization.
  • Contribution: LLM-Viewer analyzes bottlenecks when deploying any LLM architecture on various hardware platforms.The tool is introduced as the survey’s practical analysis component.
  • Survey scope: The survey organizes efficient-inference strategies into model compression, fast decoding algorithms, compiler/system optimization, and hardware-level optimization.This categorization provides the paper’s framework for reviewing the field.

2. Delve into LLM Inference and Deployment

LLM inference uses a Transformer decoder with Prefill and Decode stages, while deployment efficiency depends on both model operations and hardware memory access. The Roofline model and LLM-Viewer analyze these constraints across layers, hardware platforms, and whole networks.

  • LLM architecture: Transformer decoder LLMs comprise an embedding layer, sequential Transformer layers with MHA and MLP, and a prediction head.
  • Inference stages: Inference separates into Prefill, which builds KV caches from the prompt, and Decode, which repeatedly generates tokens using those caches.
  • Decode stage: During Decode, attention loads existing KV caches, appends newly computed key and value pairs, and sends its output through the MLP toward next-token prediction.
  • Roofline model: The Roofline model compares layer arithmetic intensity with hardware peak computation and memory bandwidth to identify computation- or memory-bound execution.
  • Roofline model: In Llama-2-7b on an Nvidia A6000, Prefill is mostly compute-bound, whereas every Decode computation is memory-bound and below the GPU’s computational capacity.
  • LLM-Viewer: LLM-Viewer performs network-wide analysis of operations, memory footprint, peak memory usage, and inference time across LLM architectures and hardware platforms.

3. Model Compression

LLM compression addresses the deployment difficulties created by the models’ formidable size and computational demands. The survey reviews techniques including quantization, pruning, and knowledge distillation.

  • Model Compression: Model compression is presented as a straightforward response to LLM deployment limits, especially in resource-constrained environments.
  • Model Compression: The survey’s compression review includes established techniques such as quantization, pruning, and knowledge distillation.

3.1. Quantization

Quantization reduces LLM storage, memory access, and computational complexity, but its benefits depend on tensor coverage, hardware support, and whether inference is memory- or compute-bound.

  • Quantization transforms floating-point LLM values into discrete representations, reducing storage requirements and computational complexity while introducing potential precision loss.Careful quantization designs aim to limit that loss.
  • Hardware support determines whether low-bit computation delivers its potential: unsupported formats require conversion, and INT8 weights cannot exploit INT8 computation with FP16 activations.For example, the H100 converts INT4 tensors to higher-bit formats for computation.
  • Quantizing weights and KV caches can reduce memory consumption, while temporary activations contribute relatively little because their short-lived storage can be released.KV-cache memory persists through answer generation and grows with batch size and sequence length.
  • Quantization reduces data movement and increases arithmetic intensity, alleviating memory-access pressure when layers remain memory-bound.Its impact is therefore tied to the layer’s position relative to the Roofline model’s memory and compute regions.
  • When decoding uses small batches, quantization improves performance and reduces inference time, but lower-bit weights provide no further decoding-time decrease once large batches make the network compute-bound.During prefill, quantization helps at short sequence lengths but may provide little improvement at long lengths when computation dominates.
  • PTQ avoids retraining and is practical for large models, whereas QAT adapts models to low precision but can be costly; QLoRA is limited to at most 4-bit fine-tuning.The survey describes a shift from PTQ toward QAT and Q-PEFT as performance bottlenecks receive greater attention.

3.2. Pruning

Pruning compresses LLMs by removing parameters, with unstructured methods favoring accuracy and structured methods offering hardware-compatible regularity at a potential performance cost.

  • Pruning removes unnecessary or redundant parameters to reduce the model’s computational demand and size without necessarily sacrificing substantial performance.
  • 3.2.1 Unstructured pruning: Unstructured pruning removes individual weights or neurons, producing sparse but irregular networks that may require specialized software or hardware handling.
  • 3.2.1 Unstructured pruning: SparseGPT can process a 175-billion-parameter model in a few hours on one GPU and induce 50–60% sparsity without significant accuracy loss or fine-tuning.
  • 3.2.2 Structured pruning: Structured pruning removes entire neurons or layers, producing regular structures compatible with conventional hardware but potentially harming performance more substantially.
  • 3.2.2 Structured pruning: LLM-Pruner uses one-shot pruning with first-order and estimated Hessian information, followed by LoRA fine-tuning to restore weights while reducing computation and memory requirements.

3.3. Knowledge Distillation

Knowledge distillation transfers capabilities from larger teacher models to smaller students, using either internal teacher information or only teacher-generated input-output behavior.

  • Knowledge distillation transfers capabilities from a larger teacher model to a smaller student model with reduced computational-resource requirements.
  • White-box distillation: White-box distillation assumes access to the teacher’s architecture and weights, allowing students to learn outputs, internal representations, and decision processes.
  • White-box distillation: MiniLLM combines single-step regularization, teacher-mixed sampling, and length normalization to address training challenges in LLM distillation.
  • White-box distillation: GKD trains students on sequences they generate themselves and uses teacher probabilities as guidance, avoiding backpropagation through student sampling.
  • Black-box distillation: Black-box distillation uses only teacher-produced input-output pairs and does not require access to the teacher’s internal information.
  • Black-box distillation: Black-box distillation can transfer chain-of-thought prompting capabilities, including targeted mathematical reasoning, from larger models to smaller students.

3.4. Factorization

Factorization compresses neural networks by representing parameters in lower-rank forms, with LLM-specific methods addressing activation outliers, layer sensitivity, and large embedding storage.

  • Low-rank matrix decomposition compresses large neural networks by representing them with fewer parameters and has become a widely studied approach.
  • ASVD compresses LLMs by adjusting weight matrices to activation distributions, managing activation outliers and calibrating decomposition choices across layers.
  • TensorGPT compresses an LLM embedding layer through Tensor-Train Decomposition, storing embeddings in a low-rank tensor format with substantially fewer parameters.

4. Algorithmic Methods for Fast Decoding

LLM decoding is inefficient because autoregressive token generation repeatedly loads enormous decoder weights, making inference memory-bound and slow. Algorithmic methods therefore reduce the parameters used per token or decode more tokens per forward propagation.

  • Autoregressive decoding repeatedly loads decoder weights for every token, creating a memory-bound process with low hardware utilization and high latency.This is especially problematic for applications requiring fast or real-time responses.
  • Algorithmic optimization considers either minimizing parameters used for each decoded token or maximizing tokens decoded per forward propagation.

4.1. Minimum Parameter Used Per Token Decoded

Minimum-parameter methods reduce inference cost by dynamically selecting only the parameters needed for each input token. Early exiting skips depth, contextual sparsity reduces width, and Mixture-of-Experts activates sparse expert structures, with roofline effects depending on the method.

  • Dynamic parameter selection can reduce latency while preserving token accuracy because not all model parameters are needed for every input token.
  • Early Exiting: Early exiting uses confidence estimates to stop decoder computation at intermediate layers, while decoder-specific methods must preserve sequence-level consistency and quality.Confidence can be derived from softmax responses, hidden-state saturation, or classifiers inserted at each layer.
  • Early Exiting: Some early-exit methods propagate hidden states across skipped layers, whereas SkipDecode avoids state propagation by enforcing monotonically decreasing maximum layers for later positions.
  • Contextual Sparsity: Contextual sparsity can reach 80%, allowing most weights to be omitted dynamically while preserving original performance; Deja Vu reports over 2x inference speedup by reducing memory I/O.
  • Mixture-of-Experts: Mixture-of-Experts decouples total parameter count from computation FLOPs, while contextual sparsity instead exploits dynamic sparsity in pretrained dense models.
  • Roofline Analysis: Early exiting proportionally reduces computation, memory access, and inference time by skipping complete Transformer layers.
  • Roofline Analysis: Contextual sparsity and Mixture-of-Experts produce operation-dependent changes in arithmetic intensity, so their computation and memory-access reductions affect inference time differently.

4.2. Maximum Tokens Decoded Per LLM Forward Propagation

Maximum-token methods relax autoregressive decoding by generating multiple candidate tokens per forward propagation and evaluating them with the LLM. Speculative decoding uses a draft model, while other methods directly construct parallel predictions or exploit language structure.

  • Speculative Decoding: Speculative decoding uses a computationally efficient draft model to propose several tokens, which the large LLM evaluates instead of generating each token directly.This improves hardware utilization because the smaller model requires less weight loading.
  • Speculative Decoding: Speculate-then-evaluate methods preserve autoregressive decoding quality while enabling inference speedup through candidate acceptance and resampling.
  • Speculative Decoding: Speculative decoding speedup is modest because accepted draft lengths are bounded and rejected future tokens are discarded.
  • Speculative Decoding: Knowledge distillation aligns the draft model with the LLM, with reported latency-speedup improvements of 10-45%, although the best distillation loss varies by model.
  • Direct Parallel Decoding: Direct parallel decoding methods predict multiple future tokens from one LLM forward pass without requiring a small Transformer draft model.
  • Language Structure: Skeleton-of-Thoughts parallelizes expansion of independently structured outline points and achieves approximately 2x speedup, but does not easily generalize to all text generation.
  • Non-Autoregressive Transformers: Non-Autoregressive Transformers iteratively decode all output tokens together, deriving speedup by producing more than one token per decoder forward pass.

5. Compiler/System Optimization

Compiler and system optimizations target redundant data movement, memory placement, operator execution, and multi-request serving. Their benefits depend on whether execution is memory- or compute-bound and on changing sequence, batch, and hardware conditions.

  • Operator Fusion: Operator fusion combines connected operators, eliminating intermediate representations and reducing data movement, memory consumption, and memory access.A linear operator followed by SiLU is an example of a fusible pattern.
  • Operator Fusion: Roofline analysis indicates that kernel fusion increases arithmetic intensity and improves performance in memory-bound regions, but provides little benefit when operators are compute-bound.
  • Operator Fusion: Operator fusion is constrained when intermediate outputs are reused, on-chip buffers are insufficient, or frameworks and hardware restrict supported fusion patterns.
  • Operator Fusion: FlashAttention and Flash-Decoding fuse self-attention matrix multiplications with softmax, avoiding storage and loading of potentially large intermediate attention matrices.The resulting reductions differ between prefill and decode stages.
  • Memory Management: PagedAttention manages variable-length KV caches by dividing each sequence into fixed-token blocks and mapping logical blocks to physical GPU-memory blocks.
  • Memory Management: Offloading can place model data in CPU DDR, GPU GDDR/HBM, or disk, whose bandwidths differ; transferring data from CPU DDR to GPU can outperform CPU computation under suitable conditions.
  • Parallel Serving: Increasing decode-stage batch size significantly improves throughput but also increases response latency and memory consumption.
  • Parallel Serving: Continuous batching combines inferences from different users, while chunked prefills and decode-maximal batching increase arithmetic intensity and throughput.

6. Hardware Optimization

Hardware optimization for LLM inference must address changing arithmetic intensity across prefill and decode stages. The survey examines spatial architectures, processing-in-memory, and lower-precision formats as responses to memory and computation bottlenecks.

  • Hardware and workload characteristics: Prefill typically has high arithmetic intensity with GEMM operators, whereas decoding processes one token at a time using GEMV or lean GEMM operators.Batch size and sequence length further change arithmetic intensity and memory-access overhead, including KV-cache reads.
  • Memory-bound decoding: Decoding is costly because each generated token requires accessing large amounts of weights and KV cache, producing low arithmetic intensity.The resulting memory-bound behavior motivates architectures that improve data movement and bandwidth.
  • Spatial architectures: Spatial architectures distribute computation across multiple processing elements with direct memory access, increasing bandwidth and improving inference performance.Figure 20 reports that linear-layer decoding performance significantly increases as total memory bandwidth rises.
  • Spatial architectures: Groq’s spatial LPU system achieves over 300 tokens per second per user on Llama-2-70b.Graphcore’s IPU is presented as another spatial architecture for efficient LLM execution.
  • Processing in memory: HBM-PIM provides 2TB/s internal memory bandwidth and 1.2TFLOPS peak throughput, targeting operators with 1-2 Ops/Byte arithmetic intensity.A GPU+HBM-PIM system achieves 3.24× speedup over a traditional monolithic GPU for KV-cache processing.
  • Processing in memory: DRAM-based PIM remains constrained by limited computation power and capacity, making it suitable mainly for small-batch inference or KV-cache processing.The passage states that computation-intensive large-batch inference still requires a powerful host, while DRAM-PIM capacity is typically 50% lower.
  • New data formats: FP8 can quantize FP16-pretrained models without significant accuracy degradation while reducing hardware overhead relative to high-precision floating-point arithmetic.LLM-specific work also explores FP4, FP6, and FP8 weight or activation quantization for efficient hardware execution.

7. Discussion

Compression improves practical LLM efficiency but can alter reliability, especially for safety and subgroup robustness. The discussion recommends preserving precision in knowledge-critical layers and validating compressed models before deployment.

  • Reliability: Compression methods can improve inference and training efficiency while causing subtle changes in model reliability.The discussion focuses on hallucination, safety alignment, and reliability as affected dimensions.
  • Hallucination: Quantization and structured compression should selectively preserve precision in important FFN layers because these layers often store factual knowledge.Retaining higher precision in knowledge-critical layers is presented as a way to mitigate hallucination risk.
  • Hallucination: Pruning should retain important FFN layers to preserve the model’s ability to recall and process factual knowledge.The stated goal is to reduce the likelihood of hallucinated outputs while streamlining the model.
  • Safety alignment: Moderate compression such as 8-bit quantization does not significantly compromise safety capabilities but may increase susceptibility to certain jailbreak attacks.The authors recommend comprehensive red teaming before deploying compressed models.
  • Safety alignment: Knowledge-transfer approaches can substantially weaken model safety, motivating re-finetuning smaller models after knowledge transfer.This recommendation is stated specifically for models produced through knowledge transfer.
  • OOD generalization: Quantization compression can produce subgroup performance disparities, while KV-cache compression may further increase reliance on decision shortcuts in long-tail subgroups.The discussion recommends robustness enhancements for downstream scenarios.

7.5. Long Context Modeling

Long-context LLM inference is limited by insufficient training context and the computation and memory costs of attending to all past tokens. Existing approaches reduce or restructure context use through summarized attention, recurrence, retrieval, or modified position encoding.

  • Motivation: General pretraining corpora often lack sufficiently long examples, challenging long-context language modeling and reasoning in chatbot and document-summarization applications.The section frames long-context capability as a deployment challenge for LLMs.
  • Attention costs: With KV cache, attending to all past keys and values for long contexts creates both computation and memory bottlenecks.This bottleneck motivates methods that avoid processing every past token at each step.
  • Attention reduction: Landmark Attention summarizes token blocks and lets queries first select relevant landmarks, reducing attention computation.The method uses landmark tokens to determine whether tokens inside each block are needed for next-word prediction.
  • Recurrence: Transformer-XL extends dependency distance through segment-level recurrence by carrying the previous segment’s last-layer output into the current segment.Segatron and Compressive Transformer are described as extensions of this recurrence idea.
  • Retrieval: Retrieval-augmented approaches place past KV cache in secondary memory and retrieve specific key-value pairs only when needed.This exploits the observation that not every past token is required for current-token generation.
  • Position encoding: Long-context methods also modify attention bias or RoPE scaling to address position-encoding limits on unfamiliar sequence lengths.Examples include ALiBi, NTK-aware Scaled RoPE, Positional Interpolation, Dynamic-NTK, and YaRN.

8. Conclusion

The paper reviews efficient LLM inference through a practice-driven survey and a roofline model for identifying deployment bottlenecks. It covers weight optimization, decoding algorithms, and hardware and system-level optimizations.

  • Conclusion: The survey develops a roofline model to pinpoint bottlenecks in LLM deployments and guide more specific deployment strategies.The project also assembles recent work on weight optimization, decoding, and hardware and system-level optimization.
  • Conclusion: The project is intended to be updated and maintained as research on efficient LLM inference develops.
Loading 2402.16363v6…