PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Inference Serving: Survey of Recent Advances and Opportunities

Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari

arXiv:2407.12391v1cs.DCcs.AI

TL;DR

LLM serving's compute and memory demands, alongside a fast-growing body of research, create a challenge for practitioners tracking deployment-relevant advances. This survey reviews selected system-level work that does not alter core decoding mechanisms and organizes advances across four areas. It highlights innovations and practical considerations for deploying and scaling LLMs.

  • Problem

    LLMs' substantial compute and memory demands can strain even high-performance GPU servers, while the rapid growth of serving research makes it difficult for practitioners to track deployment-relevant work.

    Source: §I · p. 1

  • Method

    The survey selects high-quality LLM serving research from January 2023 through June 2024, focusing on system-level enhancements that do not alter core decoding mechanisms.

    Source: §I · p. 1

  • Results

    The survey organizes recent advances into KV-cache and memory management, computation optimization, cloud deployment, and emerging fields including RAG and MoE.

    Source: §I · p. 1

  • Takeaways & Limitations

    The survey highlights innovations and practical considerations for deploying and scaling LLMs in production environments.

    Source: §I · p. 1, §VII · p. 7

    The reviewed literature is limited to selected papers published from January 2023 through June 2024.

    Source: §I · p. 1

Abstract

from arXiv · show

This survey offers a comprehensive overview of recent advancements in Large Language Model (LLM) serving systems, focusing on research since the year 2023. We specifically examine system-level enhancements that improve performance and efficiency without altering the core LLM decoding mechanisms. By selecting and reviewing high-quality papers from prestigious ML and system venues, we highlight key innovations and practical considerations for deploying and scaling LLMs in real-world production environments. This survey serves as a valuable resource for LLM practitioners seeking to stay abreast of the latest developments in this rapidly evolving field.

I. INTRODUCTION

LLM serving faces substantial compute and memory demands, while rapid growth in research makes it difficult for practitioners to track deployment-relevant advances. This survey reviews recent system-level work that leaves core decoding mechanisms unchanged and organizes it into four areas.

I. INTRODUCTION

  • High compute and memory demands can strain even high-performance GPU servers, as models and processed sequences grow.The survey is motivated by increasing deployment demand and the difficulty practitioners face in tracking the rapidly expanding literature.Source: §I · p. 1
  • The review focuses on selected high-quality LLM serving papers published from January 2023 through June 2024, excluding work that modifies decoding algorithms.Selection prioritizes prestigious ML and systems venues and impactful arXiv work from established research groups.Source: §I · p. 1
  • Unlike broader prior surveys [4], [5], [6], this survey emphasizes system-level solutions that do not alter core decoding mechanisms.The authors note that many papers in earlier surveys modify decoding algorithms that can affect model accuracy, while this survey includes substantial research published after those surveys.Source: §I · p. 1
  • The survey organizes LLM serving research into KV-cache and memory management, computation optimization, cloud deployment, and emerging fields such as RAG and MoE.It identifies retrieved-document input overhead for RAG and expert communication and load balancing for MoE as challenges.Source: §I · pp. 1–2

II. BACKGROUND

Transformer-based LLMs use multi-head attention followed by a positionwise feed-forward network. The attention heads process different representational spaces, and the FFN further refines their output.

A. Overview of Transformer-based LLM Architecture

  • Each transformer block applies query, key, and value weight matrices to encoded input X, then computes attention for each head.Multiple heads are concatenated and linearly projected into the final attention result.Source: A. Overview of Transformer-based LLM… · p. 2
  • Multi-head attention focuses on different sequence parts in different representational spaces, then feeds normalized output to a positionwise FFN.The FFN consists of two linear transformations with a ReLU activation and can be applied separately at each position to refine attention information.Source: A. Overview of Transformer-based LLM… · p. 2
  • The FFN output has the same dimension as input X.The architecture combines multi-head attention and feed-forward network components.Source: A. Overview of Transformer-based LLM… · p. 2

B. Overview of LLM Inference

LLM inference has a prefill phase followed by autoregressive decoding. Prefill processes the prompt and caches key-value pairs; decoding generates subsequent tokens while updating that cache.

B. Overview of LLM Inference

  • During prefill, the encoded prompt passes through transformer layers, a token is generated, and key-value pairs are cached for the prompt tokens.The cache avoids recomputing key-value vectors for all previous tokens during subsequent generation.Source: B. Overview of LLM Inference · p. 2
  • During decoding, the model predicts the next token autoregressively, appends it to the prompt, and updates the KV cache.The KV cache grows linearly with the number of generated tokens.Source: B. Overview of LLM Inference · p. 2
  • The inference process begins with a forward pass over the encoded input sequence and stores the resulting KV cache before generating tokens.Generation continues until an EOS token or the maximum output length is reached.Source: B. Overview of LLM Inference · p. 2

III. MEMORY MANAGEMENT AND CACHING

The KV cache grows substantially with generated tokens, motivating efficient management to reduce memory footprint and access overhead and support larger batches and longer contexts.

III. MEMORY MANAGEMENT AND CACHING

  • The KV cache grows substantially with generated tokens while model parameters remain constant and intermediate activations stay relatively small.Efficient KV-cache management is pursued to reduce memory footprint and access overhead and enable larger batches and longer context processing.Source: §III · p. 2

A. Efficient Management of KV Cache

KV-cache work addresses dynamic memory growth through non-contiguous allocation, reuse, distribution, and compression. Long-context systems also distribute or selectively manage cache state to handle sequences that exceed local memory.

A. Efficient Management of KV Cache

  • PagedAttention manages dynamically growing KV caches as non-contiguous blocks, reducing pre-allocation waste and fragmentation and becoming common in serving frameworks.Its dynamic-block design addresses request lengths and generation lifetimes that are unknown in advance.Source: A. Efficient Management of KV Cache · pp. 2–3
  • PagedAttention [10] requires attention-kernel rewrites and adds memory-manager complexity, redundancy, and performance overhead; vAttention [13] retains contiguous virtual memory and uses demand paging.vAttention overlaps allocation with computation, pre-allocates memory, and defers reclamation to hide allocation latency.Source: A. Efficient Management of KV Cache · p. 3
  • Prompt Cache reuses attention states from predefined prompt modules, while AttentionStore stores inactive-session KV caches in CPU memory or disk and prefetches them.AttentionStore overlaps cache loading with computation and uses prefetching and eviction policies.Source: A. Efficient Management of KV Cache · p. 3

B. Support for Long-Context Applications

  • Ring Attention [16] overlaps distributed attention and feedforward computation with KV-cache communication, extending context length by the device count times.Infinite-LLM [17] divides the cache into rBlocks across GPUs and CPUs; MemServe [18] manages cluster-wide KV-cache reuse through MemPool and a global scheduler.Source: B. Support for Long-Context Applications · p. 3
  • InfiniGen [19] speculates which KV-cache entries matter and prefetches essential entries, while LoongServe [20] uses Elastic Sequence Parallelism to adapt resources across requests and inference phases.The described approaches reduce data-transfer overhead, KV-cache migration overhead, and fragmentation when serving long sequences.Source: B. Support for Long-Context Applications · p. 3

C. Compression of KV Cache

KV-cache compression methods reduce memory use through quantization, error approximation, or sharing similar states. Separately, autoregressive token generation motivates system-level scheduling of inference computation.

C. Compression of KV Cache

  • FlexGen compresses weights and KV cache to 4 bits with fine-grained groupwise quantization, while KIVI [22] quantizes keys per-channel and values per-token.KIVI applies asymmetric quantization based on the KV-cache element distribution to minimize quantization error.Source: C. Compression of KV Cache · p. 3
  • Gear [23] compresses most KV-cache entries by magnitude-based quantization and approximates quantization error with a low-rank matrix; MiniCache [24] merges similar adjacent-layer states.MiniCache retains distinct states considered crucial for model performance to prevent information loss.Source: C. Compression of KV Cache · p. 3

IV. COMPUTATION TASK SCHEDULING

  • Because autoregressive token generation is sequential, LLMs generate one token at a time per request, motivating system solutions for scheduling inference computation tasks.The passage contrasts inference’s lower GPU resource efficiency with training workloads designed for massively parallel execution.Source: §IV · p. 3

A. Request Batching

Serving systems address variable request lengths through batching and scheduling strategies, separate prefill from decoding, and select parallelism and hardware configurations for inference.

A. Request Batching

  • Response Length Perception and Sequence Scheduling batches requests with similar predicted output lengths, while S3 predicts sequence lengths and retrains its predictor after mispredictions.The passage notes that prediction-based batching depends strongly on predictor quality.Source: A. Request Batching · pp. 3–4
  • Continuous batching admits new requests as soon as others finish, avoiding reliance on generation-length prediction; it is incorporated into TGI, vLLM, and TensorRT-LLM.DeepSpeed-FastGen [28] uses SplitFuse to divide long prompts into chunks and combine short prompts across iterations; Sarathi-Serve [29] schedules prefill chunks alongside ongoing decoding without stalls.Source: A. Request Batching · p. 4

B. Disaggregated Inference

  • TetriInfer [30] separates prefill and decode instances and uses predicted resource needs in two-level scheduling to limit decode hotspots and contention.The separation addresses interference between batch-like prefill jobs and latency-critical decode tasks.Source: B. Disaggregated Inference · p. 4
  • Splitwise [31] assigns prefill and decoding to separate machines for specialized hardware; DistServe [32] places stages and tunes their parallelism according to cross-node bandwidth.DistServe optimizes per-GPU goodput in high-bandwidth clusters and favors same-node placement for instances of the same stage when bandwidth is limited.Source: B. Disaggregated Inference · p. 4

C. Model Parallelism

  • Pope et al. model inference efficiency to select multidimensional partitioning for TPU v4 slices, while HeteGen distributes computation across CPUs and GPUs with asynchronous overlap.HeteGen uses overlap to mitigate CPU–GPU I/O bottlenecks.Source: C. Model Parallelism · p. 4
  • ExeGPT selects batch size and tensor-parallelism degree to maximize throughput under a latency limit; Helix [35] models heterogeneous GPU and network partitioning as max-flow.ExeGPT uses input- and output-length distributions for resource allocation; Helix represents GPU and network heterogeneity as edge capacities in a directed, weighted graph.Source: C. Model Parallelism · p. 4

V. LLMS IN THE CLOUD

Cloud deployment offers scalable infrastructure but raises cost and utilization challenges. The surveyed systems address deployment costs and interruptions through resource selection, recovery, and runtime scheduling.

V. LLMS IN THE CLOUD

  • Cloud platforms provide scalable LLM deployment without expensive hardware investment, but infrastructure costs and efficient utilization remain obstacles for providers.The passage frames cloud resource flexibility as a way to adjust resources with demand and minimize downtime.Source: §V · p. 4

A. Cloud Deployment Cost

  • SpotServe adapts to changing spot-instance availability, limits migration costs after interruptions, and commits inference progress at token level for recovery.Spot instances are discounted spare capacity but can be preempted when other instances need that capacity.Source: A. Cloud Deployment Cost · p. 4
  • ServerlessLLM [37] addresses cold-start latency with a new checkpoint format and loading system, live migration, and locality-aware server allocation.Serverless cold starts arise when an inactive service instance is shut down and must be reinitialized for a new request.Source: A. Cloud Deployment Cost · pp. 4–5
  • Mélange [38] chooses cost-efficient heterogeneous GPU allocations based on request size, request rate, and service-level objective; Llumnix [39] reschedules requests across model instances.Llumnix uses live migration of requests and in-memory state, with scheduling for load balancing, defragmentation, prioritization, and auto-scaling; the passage reports cost savings with similar tail latency.Source: A. Cloud Deployment Cost · p. 5

B. Cloud Efficiency

Cloud-efficiency work addresses power management, co-serving inference with fine-tuning, and token-streaming experience through resource allocation and scheduling.

B. Cloud Efficiency

  • POLCA dynamically applies GPU frequency locking and power capping to manage inference-cluster power.PerLLM [41] optimizes edge-cloud scheduling and resource allocation subject to processing-time, bandwidth, and computing-power constraints.Source: B. Cloud Efficiency · p. 5
  • FlexLLM [42] makes co-serving inference and fine-tuning feasible by breaking fine-tuning into token-level computations that minimize memory use and inference latency.The system targets workloads that are co-located in the cloud and have different resource constraints.Source: B. Cloud Efficiency · p. 5
  • Andes [43] allocates GPU resources among requests with a token-level preemptive scheduler to optimize Quality of Experience for text streaming.Its QoE metric compares actual token delivery timelines with expected timelines based on expected time to first token and token delivery speed; Andes improves average QoE and handles higher request rates while maintaining similar token-generation throughput.Source: B. Cloud Efficiency · p. 5

A. Retrieval Augmented Generation

The survey describes RAG approaches for managing retrieved context and MoE systems for communication, expert placement, and inference efficiency.

A. Retrieval Augmented Generation

  • RAG adds external information to LLM generation through retrieval followed by generation, with retrieved contexts concatenated to the query or supplied through cross-attention.The section notes that retrieved documents increase input length and can make RAG computationally expensive.Source: A. Retrieval Augmented Generation · p. 5
  • Sparse RAG [45] parallel-encodes retrieved documents and selectively decodes by attending to relevant caches, while RAGCache [46] and CacheBlend [47] reuse or selectively recompute KV-cache states.RAGCache organizes cached knowledge states in a tree for sharing across queries; CacheBlend recomputes a small portion of the KV cache based on preceding input text.Source: A. Retrieval Augmented Generation · p. 5

B. Mixture-of-Experts Inference

  • MoE models route inputs to specialized experts through a gating network and sparsely activate experts rather than processing the entire model for every inference.The section identifies communication and load balancing as challenges for MoE inference.Source: B. Mixture-of-Experts Inference · p. 6
  • Lina dynamically schedules resources to balance distributed MoE all-to-all transfer size and bandwidth, while ExFlow places experts by inter-layer affinity to reduce cross-GPU routing latency and improve throughput.Source: B. Mixture-of-Experts Inference · p. 6
  • SiDA-MoE predicts token-level expert activations to load active experts onto GPUs and offload inactive ones, while MoE-Infinity [51] traces sequence-level activations to predict and prefetch experts.Both approaches exploit sparse expert activation; MoE-Infinity also leverages temporal locality.Source: B. Mixture-of-Experts Inference · p. 6
  • Fiddler [52] keeps frequently used non-expert layers and selected experts on GPUs while placing remaining experts in CPU memory; other techniques address dynamic gating, expert buffering, and load balancing.Dynamic gating varies tokens processed per expert; buffering places hot experts in GPU memory and less active ones in CPU memory to reduce static GPU allocation.Source: B. Mixture-of-Experts Inference · p. 6

C. Miscellaneous Fields

The survey covers fairness and sustainability, inference-pipeline and frugal-inference techniques, and concludes that system-level solutions support LLM performance and efficiency.

C. Miscellaneous Fields

  • Virtual Token Counter [54] tracks service by client using input- and output-token processing costs to prioritize clients receiving the least service, while Sprout [55] uses generation directives to balance sustainability and generation quality.Source: C. Miscellaneous Fields · p. 6
  • FlashDecoding++ [56], Parrot [57], and FlashAttention-3 [58] optimize inference kernels, multi-request workflows, and attention, respectively.The passage describes double buffering for flat GEMM, data-flow analysis of request correlations, and warp specialization with asynchronous block-wise operations; FlashAttention-3 reports speedups on Hopper GPUs over its predecessor and reduced FP8 numerical errors.Source: C. Miscellaneous Fields · p. 6
  • FrugalGPT [59], SpecInfer, and RouteLLM [61] describe ways to reduce inference cost through prompt caching or model cascading, speculative predictions, and dynamic choice between stronger and weaker LLMs.Source: C. Miscellaneous Fields · p. 6

VII. CONCLUSION

  • The survey concludes that system-level solutions are important for enhancing LLM-serving performance and efficiency and for deploying and scaling LLMs.Source: §VII · p. 7
Loading 2407.12391v1…