Source-linked AI summary
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, Zhihao Jia
TL;DR
Generative LLM serving faces substantial computational and memory demands, making low latency, scalability, and accuracy-efficiency trade-offs central concerns. This survey synthesizes solutions from algorithmic modifications to system designs, reviews serving frameworks and benchmarks, and outlines future research directions. It aims to help researchers and practitioners understand existing methodologies and make informed deployment decisions.
Problem
LLM serving is constrained by substantial computational requirements, memory demands, latency needs, scalability requirements, and accuracy-efficiency trade-offs.
Method
The survey systematically examines efficient-serving strategies spanning algorithmic innovations, system architectures, serving frameworks, and benchmarks.
Results
The survey consolidates current methodologies and identifies future directions for efficient LLM serving.
Takeaways & Limitations
The synthesis is intended to support informed decisions when deploying LLMs in real-world environments.
Takeaways & Limitations
The survey primarily focuses on research anchored in GPU-based deployment despite the broader range of available hardware platforms.
Abstract
from arXiv · showhide
In the rapidly evolving landscape of artificial intelligence (AI), generative large language models (LLMs) stand at the forefront, revolutionizing how we interact with our data. However, the computational intensity and memory consumption of deploying these models present substantial challenges in terms of serving efficiency, particularly in scenarios demanding low latency and high throughput. This survey addresses the imperative need for efficient LLM serving methodologies from a machine learning system (MLSys) research perspective, standing at the crux of advanced AI innovations and practical system optimizations. We provide in-depth analysis, covering a spectrum of solutions, ranging from cutting-edge algorithmic modifications to groundbreaking changes in system designs. The survey aims to provide a comprehensive understanding of the current state and future directions in efficient LLM serving, offering valuable insights for researchers and practitioners in overcoming the barriers of effective LLM deployment, thereby reshaping the future of AI.
1 INTRODUCTION
Generative LLMs enable broad language and AI applications but impose substantial serving costs that limit deployment. This survey organizes efficient-serving solutions from algorithmic innovations through system architectures.
- Generative LLMs have demonstrated strong performance across language tasks and are increasingly used beyond NLP, including programming and scientific applications.
- Their model size and computational complexity create serving requirements that raise concerns about energy consumption, scalability, and accessibility.
- The survey examines strategies for efficient LLM serving across algorithmic innovations and system architectures to optimize inference.
- Its taxonomy covers decoding algorithms, architecture design, model compression, low-bit quantization, parallel computation, memory management, request scheduling, and kernel optimization.
- The paper is organized around background, a taxonomy of approaches, serving frameworks, benchmarks, related literature, and future research directions.
2 BACKGROUND
Transformer-based LLM inference combines self-attention, computationally intensive feed-forward networks, and autoregressive decoding. Efficient serving must balance latency, memory, throughput, hardware utilization, and accuracy across diverse deployment settings.
- 2.1 Transformer-based LLM: Self-attention transforms input-derived queries, keys, and values into outputs that capture dependencies across the sequence.
- 2.1 Transformer-based LLM: The feed-forward network applies two linear transformations with a nonlinear activation and contributes substantially to model parameters, memory footprint, and computational load.
- 2.2 GPUs and Other Accelerators: GPUs accelerate LLM computation through parallel processing, shared memory, high bandwidth, specialized precision units, and increased floating-point throughput.
- 2.2 GPUs and Other Accelerators: The survey primarily focuses on GPUs, although LLM deployment also spans CPUs, mobile and edge devices, ASICs, TPUs, FPGAs, and other accelerators.
- 2.3 LLM Inference: Autoregressive inference predicts one token at a time, appends it to the sequence, and conditions subsequent predictions on the generated context.
- 2.3 LLM Inference: Inference optimization depends on algorithm configurations and system setups because different strategies can be optimal under different conditions.
- 2.4 Challenges: Efficient serving must address low latency, memory footprint, scalability, throughput, hardware compatibility, acceleration, and accuracy-efficiency trade-offs.
3 TAXONOMY
The taxonomy organizes efficient LLM serving into algorithmic innovations and system optimizations, with this section emphasizing decoding and architecture techniques for reducing inference cost while balancing quality.
- 3 TAXONOMY: The survey classifies LLM inference advancements into algorithmic innovations and system optimizations.The taxonomy provides the organizing framework for reviewing approaches to serving efficiency.
- Decoding algorithms: Non-autoregressive decoding generates output tokens in parallel by weakening or modeling token dependencies, but often produces less reliable quality than autoregressive decoding.Semi-autoregressive variants attempt to recover quality by modeling output dependencies.
- Decoding algorithms: Speculative decoding uses draft token sequences or trees and verifies them with the target model to improve generation efficiency.Adaptive draft-tree construction is explored to improve speculation efficiency.
- Decoding algorithms: Early exiting reduces computation by allowing internal model layers to emit predictions for easier requests, but limited representations can reduce prediction fidelity.These methods are also described as adaptive computation because they vary computation per request.
- Architecture design: Alternative architecture designs seek a balance among model size, performance, and inference efficiency beyond the original Transformer.The survey also compares attention simplification methods in efficient Transformers and recent LLMs.
3.2 System Optimization
System optimization improves LLM serving without changing computation semantics by combining quantization, parallelism, memory and request management, kernel techniques, and deployment-level trade-offs.
- 3.2 System Optimization: System optimization refines inference systems and frameworks to improve efficiency without modifying LLM computation semantics.The reviewed techniques target the underlying serving stack rather than the model’s computational meaning.
- Quantization: Low-bit quantization represents weights and activations with fewer than 32 bits, reducing memory consumption and accelerating inference.The survey distinguishes Quantization-Aware Training from Post-Training Quantization.
- Parallel computation: Parallel computation distributes model work across cores or devices, with tensor and pipeline model parallelism supporting multi-device inference.Tensor parallelism splits internal dimensions, while pipeline parallelism assigns consecutive layers to separate devices.
- Memory management: KV-cache management is a central memory target because cache size changes dynamically during incremental decoding, making fixed contiguous allocation inefficient.Offloading systems move parameters or KV cache to slower memory tiers to reduce GPU memory pressure.
- Request scheduling: Iteration-level scheduling and selective batching better accommodate variable output lengths and improve hardware utilization.Later systems also address heterogeneous workloads with chunked prefill, disaggregation, and multi-level scheduling.
- Kernel optimization: Kernel fusion reduces launch and memory-access overhead by combining compatible operations, while compilers help discover efficient implementations across hardware platforms.Examples include fusing same-shape GEMMs and combining bias with residual, normalization, or activation operations.
- Deployment trade-offs: Efficient serving requires balancing speed and accuracy according to application needs, since real-time dialogue may tolerate small accuracy losses while medical diagnosis demands precision.The survey presents algorithmic and system techniques as complementary dimensions of this trade-off.
4 SOFTWARE FRAMEWORKS
The survey compares open-source GPU-based serving frameworks as full-stack systems, focusing on parallelism, scheduling, attention kernels, and differentiated latency behavior rather than a universal best solution.
- 4 SOFTWARE FRAMEWORKS: The survey analyzes representative open-source GPU-based LLM serving systems while excluding specialized hardware solutions and systems built atop other frameworks.This scope defines the basis for its framework comparison.
- Framework capabilities: Most compared systems support tensor parallelism, while some add pipeline parallelism or offloading for multi-node and resource-constrained inference.These features address different deployment environments and hardware availability.
- Framework capabilities: Some frameworks inherit Orca’s iteration-level scheduling, reflecting the shift from request-level scheduling toward generation-aware execution.The comparison identifies scheduling granularity as a meaningful system difference.
- Kernel implementations: Flash and Paged attention implementations differ across frameworks, including imported libraries, Triton kernels, TVM-generated kernels, and modified fused kernels.Kernel implementation choices therefore vary even when systems expose similarly named attention mechanisms.
- Performance metrics: Initial-phase processing speed and per-iteration decoding time are distinct metrics, motivating system designs tailored to responsiveness or long prompts rather than one universal solution.Adaptive kernel libraries and compilers generate optimized kernels for input configurations.
5 BENCHMARKS
The survey identifies the lack of a convincing, comprehensive benchmark for LLM serving systems and calls for evaluations spanning workloads, hardware, and competing efficiency objectives.
- 5 BENCHMARKS: Existing evaluations commonly use dynamic request-arrival patterns from production traces, but the field lacks a convincing benchmark for comparing LLM serving systems.The challenge stems from many interacting evaluation settings, including model configuration, hardware, and request load.
- Benchmark design: A comprehensive benchmark must calibrate throughput, latency, and cost-efficiency across bursty, unpredictable, and sustained high-load workloads.It should also vary request patterns and hardware configurations while isolating inference performance from system overhead.
6 CONNECTION WITH OTHER SURVEYS
This survey focuses specifically on generative LLM serving, distinguishing it from broader Transformer and accelerator surveys while complementing related empirical and environmental studies.
- The survey focuses on generative LLM serving, an area not central to prior studies of general Transformer models and domain-specific accelerators.
- Related work includes experimental studies of LLM inference efficiency on GPUs and novel accelerators, which provide empirical insights relevant to serving efficiency.
- The survey primarily examines performance-oriented efficiency, whereas LLMCarbon adds an environmental perspective through deployment impacts such as carbon footprints.
7 FUTURE DIRECTION
The survey identifies future directions spanning hardware, decoding, long-context workloads, architectures, deployment environments, and automatic adaptation to application requirements.
- Development and Enhancement of Hardware Accelerators: Future hardware accelerators could improve LLM serving through hardware–software co-design, closer memory placement, and architectures tuned to attention and tensor operations.
- Efficient and Effective Decoding Algorithms: Generalized speculative inference and related decoding methods could increase generation speed while preserving generation quality.
- Long Context/Sequence Scenarios Optimization: Long-sequence serving requires addressing algorithmic and system challenges, including length generalization failure beyond training-time sequence lengths.
- Investigating Alternative Architectures: Alternative architectures, including attention-free and pure-MLP approaches, may offer computational-efficiency advantages and create new inference-engine opportunities such as KV-cache management.
- Exploration of Deployment in Complex Environments: Future deployment research should cover cloud alternatives including edge, hybrid, decentralized, and spot-instance environments to improve robustness, scalability, and efficiency.
- Automatic Adaptation to Specific Requirements: Application-specific requirements motivate automatic integration of serving techniques across the LLM lifetime, from data processing and model management to resource allocation and monitoring.
8 CONCLUSION
The survey consolidates research on algorithms and systems for efficient LLM serving to support informed real-world deployment and encourage further innovation.
- The survey provides researchers, practitioners, and developers with a comprehensive understanding of existing efficient LLM serving methodologies.
- By consolidating findings on algorithms and systems, the survey aims to accelerate progress toward highly efficient LLM serving solutions.
- Efficient LLM serving is presented as a step toward democratizing access to advanced AI technologies.