PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Efficient Inference for Large Language Models

Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, Yu Wang

arXiv:2404.14294v3cs.CLcs.AI

TL;DR

Efficient LLM inference is difficult because large models, quadratic attention, and autoregressive decoding impose substantial computational and memory costs in resource-constrained deployments. This survey analyzes these bottlenecks, organizes research into data-, model-, and system-level optimization, and evaluates representative methods. Its experiments include a 3.47∼3.72× end-to-end speed-up for Eagle, while the survey identifies remaining performance and deployment limitations.

  • Problem

    LLM inference imposes substantial computational and memory requirements that hinder deployment in resource-constrained scenarios.

  • Method

    The survey analyzes inference bottlenecks, builds a three-level taxonomy, reviews studies, and experimentally evaluates representative methods and serving frameworks.

  • Results

    Eagle achieves a 3.47∼3.72× end-to-end speed-up across multiple LLMs in the speculative decoding comparison.

  • Takeaways & Limitations

    The survey provides practical suggestions and identifies research avenues for efficient LLM inference across data, model, and system techniques.

  • Takeaways & Limitations

    Efficient inference research does not adequately address security, and deploying LLMs on extremely resource-constrained edge devices remains challenging.

Abstract

from arXiv · show

Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. Efforts within the field have been directed towards developing techniques aimed at enhancing the efficiency of LLM inference. This paper presents a comprehensive survey of the existing literature on efficient LLM inference. We start by analyzing the primary causes of the inefficient LLM inference, i.e., the large model size, the quadratic-complexity attention operation, and the auto-regressive decoding approach. Then, we introduce a comprehensive taxonomy that organizes the current literature into data-level, model-level, and system-level optimization. Moreover, the paper includes comparative experiments on representative methods within critical sub-fields to provide quantitative insights. Last but not least, we provide some knowledge summary and discuss future research directions.

1 INTRODUCTION

LLMs achieve broad capabilities but are difficult to deploy efficiently because inference demands substantial computation, memory, and storage. This survey organizes existing efficiency research and supplements its review with experiments, recommendations, and future directions.

  • LLMs support tasks including language understanding, generation, reasoning, and code generation, enabling applications such as ChatGPT, Copilot, and Bing.
  • Inference efficiency challenges arise from high computational cost, memory access cost, and memory usage, harming latency, throughput, power consumption, and storage.
  • A 70-billion-parameter model may be impractical on personal laptops, while low throughput can substantially reduce search-engine profits.
  • The survey classifies efficient inference studies into data-level, model-level, and system-level optimization.
  • Experiments on representative methods consolidate knowledge, offer practical recommendations, and guide future research.

2 PRELIMINARIES

Transformer-based LLM inference relies on self-attention and autoregressive generation, whose computational, memory, and latency costs grow with model and sequence size. KV caching separates inference into prefilling and decoding stages but does not remove the underlying deployment burden.

  • Transformer architecture: Transformer blocks typically combine multi-head self-attention, a feed-forward network, and layer normalization, processing tokenized input through stacked blocks.
  • Self-attention: Self-attention projects inputs into queries, keys, and values, captures long-range dependencies, and has quadratic computational complexity in input length.
  • Feed-forward network: The FFN follows attention and applies two linear transformations with a nonlinear activation function to the attention output.
  • Inference process: Autoregressive decoding generates tokens one by one, repeatedly using the full input and prior outputs, so generation time grows with sequence length.
  • Inference process: KV caching stores and reuses prior key-value pairs, dividing inference into prefilling, which builds the cache, and decoding, which updates it token by token.
  • Efficiency analysis: A LLaMA-2-70B model requires 140 GB of FP16 VRAM, while generating hundreds of tokens can require more than 10 seconds on two NVIDIA A100 GPUs.
  • Efficiency analysis: The main inference bottlenecks are model size, quadratic attention during prefilling, and autoregressive decoding with repeated weight loads and growing KV cache.

3 TAXONOMY

The survey classifies efficient LLM inference methods into data-, model-, and system-level optimization according to where efficiency is improved. These levels differ in training requirements and their typical effect on model performance.

  • The taxonomy is motivated by three inference bottlenecks: model size, attention operation, and decoding approach.
  • Data-level Optimization: Data-level optimization improves efficiency by compressing input prompts or organizing outputs, generally without changing the original model.
  • Model-level Optimization: Model-level optimization designs efficient structures or compresses pretrained models, often requiring training and typically introducing lossy performance changes.
  • System-level Optimization: System-level optimization improves inference engines or serving systems without costly model training and is typically lossless in model performance.

4 DATA-LEVEL OPTIMIZATION

Data-level optimization improves LLM inference by shortening inputs or organizing outputs for parallel generation. The survey groups input compression into four categories and discusses output-organization methods that reduce generation latency.

  • Data-level optimization comprises input compression and output organization, which shorten inputs or enable batch inference to improve hardware utilization and reduce latency.
  • Input Compression: Input prompt compression is categorized into prompt pruning, prompt summary, soft prompt-based compression, and retrieval-augmented generation.
  • Input Compression: Prompt pruning removes unimportant tokens, sentences, or documents using predefined or learned importance indicators, including dynamic example selection and perplexity-based pruning.
  • Input Compression: Prompt summary condenses an entire prompt into a shorter semantic summary, while soft prompt methods replace longer prompts with learnable continuous tokens.
  • Output Organization: SoT generates a concise answer skeleton, expands its points simultaneously, and achieves up to a 2.39× speedup on 12 recently released LLMs.SoT also considers sharing the common prompt-prefix KV cache to mitigate extra-prompt overhead.
  • Output Organization: APAR dynamically triggers parallel decoding with [fork] control tokens, achieving an average 1.4∼2.0× speed-up with negligible impact on answer quality.It also combines its decoding approach with Medusa and vLLM to further improve inference latency and system throughput.
  • Output Organization: Output organization exploits independent points or dependency graphs to facilitate batch inference, improve hardware utilization, and reduce end-to-end generation latency.
  • Future Direction: FrugalGPT uses a cascade of differently sized LLMs and halts inference early when confidence is sufficient, performing dynamic inference at the pipeline level.

5 MODEL-LEVEL OPTIMIZATION

Model-level optimization improves inference by changing model structure or data representation. The survey distinguishes efficient structure design from compression of pre-trained models based on training overhead.

  • Model-level optimization targets model structure or data representation to improve LLM inference efficiency.
  • Efficient Structure Design: Efficient structure design creates more efficient architectures and typically requires training from scratch.
  • Model Compression: Model compression compresses pre-trained models and generally requires only minimal fine-tuning to restore their abilities.

5.1 Efficient Structure Design

Efficient structure design targets the Transformer components that dominate inference costs: FFNs, quadratic attention, and the broader sequence-modeling architecture. The survey organizes these efforts around sparse expert computation, efficient attention variants, and Transformer alternatives.

  • Motivation: Transformer inference bottlenecks arise from FFN parameter and memory costs and attention’s quadratic complexity in input length.FFNs account for 63.01% of LLaMA-7B parameters and 71.69% of LLaMA-70B parameters.
  • Efficient FFN Design: Efficient FFN designs commonly use Mixture-of-Experts to allocate different computational budgets to input tokens through selectively activated expert networks.MoE models combine parallel FFN experts with a trainable routing module.
  • Efficient FFN Design: MoE routing research addresses load imbalance, which wastes under-utilized expert capacity and adds computation through padding for batched matrix multiplication.Switch Transformers introduces a load-balancing loss to encourage balanced token assignment across experts.
  • Efficient FFN Design: MoE training methods target instability and routing inconsistency using auxiliary losses or more consistent routing procedures.SE-MoE penalizes large routing logits, while StableMoE fixes a learned routing strategy during backbone training and inference.
  • Efficient Attention Design: Efficient attention designs reduce inference memory by sharing key-value caches or reduce sequence-length complexity through kernel and low-rank approximations.MQA shares K and V across attention heads; kernel-based attention reduces complexity to O(nd^2), while low-rank attention can reduce it to O(nkd).
  • Transformer Alternates: Transformer alternatives use sub-quadratic sequence modeling and can avoid caching previous-token features during decoding, reducing memory access costs.The survey compares architectures including state-space, convolutional, and recurrent alternatives.

5.2 Model Compression

Model compression improves LLM inference efficiency by changing tensor representations or model structure. The survey covers quantization, pruning, factorization, dynamic sparsification, neural architecture search, and knowledge distillation, with workflow and latency analyses.

  • Quantization: Quantization reduces computational and memory costs by converting FP16 weights and activations into lower-bit representations.Scaling factor and zero-point parameters determine the mapping from FP16 values to low-precision integers.
  • Compression taxonomy: Representative compression methods include structure factorization, weight pruning, sparse attention, neural architecture search, and knowledge distillation.Knowledge distillation is divided into white-box and black-box approaches, while the taxonomy also includes token- and sample-level dynamic methods.
  • Quantization: Weight-only quantization targets decoding latency by reducing memory-access cost, while de-quantizing weights to FP16 for computation.It is performed offline and primarily accelerates the memory-bound decoding stage.
  • Quantization: GPTQ reduces quantization overhead by using a uniform left-to-right row order and reusing one computed Hessian across subsequent rows.LUT-GEMM separately accelerates de-quantization with a look-up table and non-uniform quantization.
  • Quantization: Weight-only quantization can accelerate decoding and end-to-end latency but may increase prefilling latency because de-quantization adds computation.Prefilling is primarily computation-bound, whereas decoding is strongly affected by loading large weight tensors from HBM.
  • Sparsification: Dynamic attention pruning selectively omits attention calculations through input-dependent token grouping or token-level selection.Reformer uses locality-sensitive hashing, Routing Transformer uses spherical k-means, Sparse Sinkhorn Attention uses learned sorting, and H2O performs token-level pruning.
  • Knowledge distillation: White-box knowledge distillation uses teacher structure and parameters to transfer intermediate features and output logits to smaller student models.Examples include reverse-KLD distillation, on-policy data, task-aware layer-wise filters, and MoE students.

5.3 Knowledge, Suggestions and Future Direction

The survey identifies important trade-offs and open directions for efficient LLM compression. Quantization is widely used, but long-context and emergent abilities can degrade; sparse attention, pruning, NAS, and hybrid compression remain active challenges.

  • Efficient structure design: Non-Transformer architectures show competitive performance across various tasks, but their shortcomings relative to Transformers and integration with attention remain open questions.The survey highlights these issues as directions for further investigation.
  • Quantization: Quantization is predominant in LLM deployment because it compresses models conveniently and can reduce memory consumption and inference speed with minor trade-offs.PTQ can compress a seven-billion-parameter LLM within minutes.
  • Quantization: Quantization may compromise emergent abilities such as self-calibration or multi-step reasoning and can significantly degrade performance on long contexts.The survey recommends carefully selecting quantization methods for specialized scenarios.
  • Sparse attention: Sparse attention can process very long contexts efficiently, but approaches may sacrifice critical information and degrade performance.StreamingLLM is cited as processing four million tokens by restoring only several attention sink tokens.
  • Weight pruning: Current state-of-the-art weight-pruning methods can experience considerable performance degradation even at relatively low sparsity ratios.Maintaining LLM performance under pruning remains an emerging research direction.
  • Structure optimization: Neural Architecture Search can require extensive computational resources, limiting its practical application to LLM compression.Low-rank factorization also faces a balance between compression ratio and task performance; ASVD achieves 10% to 20% compression without compromising reasoning capabilities.
  • Hybrid compression: Combining compression methods can leverage complementary advantages to reduce memory requirements, quantization errors, memory footprint, and memory-access cost.Examples combine factorization with MoE expert compression, sparsity with outlier protection, and low-rank factorization with pruning or quantization.

6 SYSTEM-LEVEL OPTIMIZATION

System-level optimization improves the LLM forward pass by targeting operators and decoding characteristics that dominate inference runtime, especially attention and linear operators.

  • System-level optimization: System-level optimization primarily targets the LLM forward pass, where attention and linear operators dominate runtime.It also considers the distinctive characteristics of the attention operator and autoregressive decoding approach.

6.1 Inference Engine

Inference-engine optimizations accelerate model execution through operator and graph optimization, speculative decoding, and offloading. The survey reviews these techniques and compares representative speculative-decoding methods, including their speedups and overheads.

  • Optimization Scope: Operator and computational-graph optimizations target the main components of LLM forward execution, while speculative decoding accelerates generation and offloading reduces memory pressure.The survey identifies these as core inference-engine optimization directions.
  • Runtime Bottlenecks: Attention and linear operators often account for more than 75% of inference runtime, making them primary targets for operator-level optimization.The profiling uses HuggingFace implementations across models and context lengths.
  • Attention Optimization: As context length grows, attention occupies a larger runtime proportion, increasing computational and memory demands for long sequences.Customized attention operators are therefore used to reduce standard GPU attention overhead.
  • Graph and Operator Optimization: FlashAttention fuses attention into one kernel, while ByteTransformer and DeepSpeed fuse lightweight operators to reduce memory-access and kernel-launch overhead.The cited methods also fuse query, key, and value projections into a single linear operator.
  • Speculative Decoding: Speculative decoding uses draft-token construction followed by target-model verification in one inference step, with acceptance rate serving as a key metric.The method is designed to preserve output equivalence with standard autoregressive decoding.
  • Speculative Decoding: 3.47∼3.72× end-to-end speed-up is achieved by Eagle across multiple LLMs, while draft-model generation costs make speed-up lower than acceptance rate.The comparison also identifies token-tree verification as effective for improving speculative decoding.

6.2 Serving System

Serving-system optimizations improve asynchronous request handling through memory management, batching, scheduling, and distributed execution. The survey highlights KV-cache paging, continuous batching, chunked prefilling, and prefilling–decoding disaggregation as key approaches.

  • Serving-System Scope: Serving-system optimization targets asynchronous requests by increasing memory capacity, improving batching and scheduling, and exploiting distributed computational resources.These methods focus on system efficiency rather than model retraining.
  • Memory Management: KV-cache storage dominates serving memory usage for long contexts, while preset maximum-length allocation wastes storage when requests terminate early.Paged storage mitigates redundant memory waste but introduces irregular attention access.
  • Continuous Batching: Continuous batching adds new requests as earlier requests finish, releasing spare storage and computation while maintaining iteration-level batching.ORCA introduced iteration-level batching, and vLLM extended it to attention with different KV-cache lengths.
  • Prefilling Optimization: Chunked prefilling partitions prefilling requests along the sequence dimension to prevent bottlenecks for other requests while preserving mathematical equivalence.The approach relies on attention depending only on prior tokens.
  • Scheduling: Long requests can cause head-of-line blocking because rapidly increasing memory usage impedes subsequent requests when system memory is exhausted.Request execution order therefore affects serving throughput.
  • Distributed Systems: Disaggregating prefilling and decoding can improve inference-service efficiency because the two stages interfere with each other on distributed platforms.Splitwise, TetriInfer, and DistServe demonstrate this direction.

6.3 Hardware Accelerator Design

Hardware-accelerator designs address the memory-intensive decoding stage through model compression, sparse computation, mixed precision, and reusable hardware kernels. The surveyed results show gains in speed, energy efficiency, and cost efficiency across FPGA and GPU comparisons.

  • Prior Accelerator Designs: Earlier Transformer accelerators often optimized attention or used sparse methods for FPGA deployment, but were not tailored for generative LLMs.FACT improves energy efficiency through mixed-precision quantization and algorithm–hardware co-design.
  • FPGA Acceleration: ALLO combines model-compression insights with composable, reusable HLS kernels for efficient FPGA deployment across prefilling and decoding.The implementation reports higher generation speed-up than DFX during prefilling and improved energy efficiency and speedup over NVIDIA A100 during decoding.
  • FPGA Acceleration: FlightLLM uses configurable sparse DSP chains and an always-on-chip mixed-precision decode scheme to improve computational efficiency and memory-bandwidth utilization.The design supports various sparsity patterns.
  • Reported Results: 6.0× higher energy efficiency, 1.8× better cost efficiency, and 1.2× higher decoding throughput are reported for FlightLLM versus NVIDIA V100S or A100 baselines on Llama2-7B.The reported efficiency and throughput figures use different GPU comparison baselines.

6.4 Comparison of LLM Frameworks

The survey compares open-source inference engines and serving systems under fixed Llama2-7B inference conditions and on ShareGPT serving workloads. FlashDecoding++ and TensorRT-LLM lead inference throughput, while serving systems use fine-grained KV-cache storage and continuous batching.

  • Evaluation Setup: Inference throughput is measured with Llama2-7B at batch size 1, 1k input tokens, and 128 output tokens on one NVIDIA A100 80GB GPU.Serving performance is measured as maximum throughput on ShareGPT.
  • Inference Throughput: FlashDecoding++ and TensorRT-LLM outperform other frameworks in inference throughput by optimizing predominant operators and the computational graph.Speculative decoding is disabled for all inference-throughput measurements.
  • Serving Performance: Serving frameworks use fine-grained, discontiguous KV-cache storage and continuous batching to improve system throughput.DeepSpeed, vLLM, LightLLM, and TensorRT-LLM provide serving functions for asynchronous multi-user requests.

6.5 Knowledge, Suggestions and Future Direction

The survey identifies serving-system scheduling as difficult because real-world systems balance job completion time, throughput, and fairness, while current predictors remain imperfect.

  • System-level optimization: System-level optimization improves inference efficiency without accuracy degradation and is therefore prevalent in LLM inference practice.Inference optimizations also apply to serving systems, including prefix caching and speculative-decoding verification.
  • Serving scheduling: Serving schedulers must balance job completion time, system throughput, and fairness when request lengths are indeterminate.The combination of multiple objectives and unpredictable request lengths makes scheduling design intricate.
  • Serving scheduling: Current request-length predictors fall short of ideal standards, leaving serving scheduling strategies open to refinement and optimization.The survey specifically identifies predictor efficacy as a remaining development opportunity.

7 DISCUSSIONS OF KEY APPLICATION SCENAR-

The survey discusses application scenarios that both increase demand for efficient LLM inference and create opportunities for optimization, especially in agents, long contexts, and edge deployment.

  • Agent and Multi-Model Framework: Agent and multi-model frameworks increase LLM computational demands while introducing parallelism that can support output-organization and system-level optimizations.The survey connects these frameworks’ output structure with opportunities for data-level and system-level techniques.
  • Long-Context LLMs: Long-context LLMs remain constrained by self-attention’s quadratic complexity in context length during training and inference.The survey lists input compression, sparse attention, low-complexity structures, and operator optimization as responses.
  • Long-Context LLMs: Non-Transformer architectures with sub-quadratic or linear complexity are attracting interest, but their competitiveness with Transformers remains under scrutiny.Open questions include in-context learning and long-range modeling abilities.
  • Edge Scenario Deployment: Deploying LLMs on extremely resource-constrained edge devices remains challenging despite substantial efficiency research.Smaller 1B–3B-parameter models reduce inference resource costs and may offer competitive generalization and performance, but development methods remain under-explored.
  • Edge Scenario Deployment: Recent small-model approaches include hyperparameter studies, pruning-informed initialization, deep-and-thin architectures, and cross-layer weight sharing.These methods seek efficient and powerful smaller language models, including designs that increase depth without additional memory costs.
  • Edge Scenario Deployment: MLC-LLM demonstrates mobile deployment of LLaMA-7B by using compilation techniques such as fusion, memory planning, and loop optimization.The project targets lower inference latency and memory cost on mobile phones.

8 CONCLUSION

The survey reviews efficient LLM inference as a multi-level field focused on reducing computational and memory costs, and it organizes the literature to support evaluation, recommendations, and future research.

  • Scope: Efficient LLM inference targets computational, memory-access, and memory costs while optimizing latency, throughput, storage, power, and energy.These efficiency metrics define the survey’s central scope.
  • Evaluation and guidance: The survey provides insights, recommendations, and future directions for key efficient-inference techniques.Its recommendations are based on the literature review and comparative analyses.
  • Taxonomy: The survey organizes research into data-level, model-level, and system-level optimization through a hierarchical taxonomy.It examines studies at each level and sub-field using this taxonomy.
  • Evaluation and guidance: Experiments evaluate representative methods in established areas such as model quantization and efficient serving systems.The analyses support practical suggestions and identify research opportunities for practitioners and researchers.
Loading 2404.14294v3…