Source-linked AI summary
LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, Xin Jin
TL;DR
Long-context LLM serving has highly variable resource demands across requests and processing phases, while static parallelism limits resource utilization. LoongServe introduces elastic sequence parallelism with elastic scaling and scheduling mechanisms to adapt parallelism dynamically. Across diverse real-world datasets, it improves throughput by up to 3.85× over chunked prefill and 5.81× over prefill-decoding disaggregation.
Problem
Long-context serving is difficult because request lengths and prefill-versus-decoding phases create large, changing demands that static strategies cannot efficiently match.
Method
LoongServe uses elastic sequence parallelism to adjust degree of parallelism dynamically, supported by scaling mechanisms and a scalable scheduler.
Results
3.85× higher throughput than chunked prefill and 5.81× higher than prefill-decoding disaggregation were achieved on real-world datasets.
Takeaways & Limitations
LoongServe improves prefill and decoding performance simultaneously while serving long-context LLMs under dynamic workloads.
Takeaways & Limitations
Reactive KV-cache migration can require per-instance unused memory of O(blsh/d), causing out-of-memory errors despite sufficient aggregate free capacity.
Abstract
from arXiv · showhide
The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance between different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the maximum throughput by up to 3.85$\times$ compared to the chunked prefill and 5.81$\times$ compared to the prefill-decoding disaggregation.
1 Introduction
Long-context LLM serving faces highly variable resource demands across requests and across prefill and decoding phases, which static parallelism and grouping cannot match efficiently. LoongServe addresses this with elastic sequence parallelism and mechanisms designed to improve throughput while limiting communication and memory overhead.
- Motivation: Longer context windows increase both KV-cache memory consumption and attention computation, making long-context serving resource-intensive.A 1M-token request can require 488GB for KV cache alone, while attention complexity is quadratic in input length.
- Motivation: Static parallelism is inefficient because request lengths and phase-specific resource demands vary substantially.Prefill and decoding have different demands, and those demands can change dynamically at iteration granularity.
- Limitations of Existing Systems: Static grouping further mismatches changing demand, migrates KV caches during phase transitions, and fragments GPU memory across isolated groups.These limitations prevent shared use of available memory for long sequences.
- Elastic Sequence Parallelism: ESP dynamically selects each request’s degree of parallelism per iteration, using more GPUs for prefill and fewer for lightweight decoding.This adapts resource allocation to phase-specific demands while releasing resources for other requests.
- LoongServe: LoongServe combines ESP with elastic scaling mechanisms, communication-aware decoding, and a scalable scheduler for iteration-level decisions.Its design targets scaling overhead, KV-cache movement, batching, placement, and elastic resource allocation.
- Evaluation: 3.85× throughput improvement over chunked prefill and 5.81× over prefill-decoding disaggregation were achieved on real-world datasets.The experiments also report simultaneous improvement in prefill and decoding performance.
2 Background and Motivation
LLM serving workloads vary sharply across request lengths and between prefill and decoding, exposing limitations in static parallelism, phase isolation, and locality-constrained KV-cache placement. ESP addresses these challenges by dynamically adjusting sequence parallelism and distributing KV caches across instances.
- LLM inference: The prefill phase builds the KV cache from all input tokens, while decoding generates subsequent tokens using cached intermediate states.
- Existing limitations: Static grouping mismatches iteration-level resource demand, migrates KV caches during phase transitions, and fragments GPU memory across isolated groups.
- Workload dynamics: 105.97 times slower processing 100K input tokens than 1K tokens on 8 GPUs demonstrates the steep length sensitivity of prefill computation.
- Static parallelism limits: Prefill benefits from higher DoP, whereas decoding may gain little because additional communication overhead offsets computation improvements.
- Elastic sequence parallelism: ESP extends sequence parallelism to decoding, dynamically distributes input tokens, adjusts DoP without repartitioning model parameters, and enables cross-instance memory use.
- Challenges: KV-cache redistribution makes elasticity costly for long contexts, requiring migration mechanisms that avoid frequent communication overhead.
3 LoongServe Overview
LoongServe implements ESP with elastic instances coordinated by a global manager, dynamically adapting grouping, dispatch, DoP, and KV-cache placement to changing request demands. Its lifecycle supports real-time scaling decisions across request iterations.
- System architecture: LoongServe organizes elastic instances into disjoint ESP groups that process batches with different configurable degrees of parallelism.
- Global coordination: The global manager uses scaling information and runtime state to dynamically adjust batches, grouping, dispatching, DoP, and KV-cache placement.
- Request lifecycle: The dispatcher assigns new requests, while the elasticity controller updates instance configurations to form the ESP groups specified by the scaling plan.
4 LoongServe Elastic Instances
LoongServe organizes GPUs as elastic instances and dynamically changes parallel-group size across prefill and decoding demands. Its proactive scale-down and multi-master scale-up mechanisms avoid key-value-cache migration while addressing memory fragmentation and computation bottlenecks.
- Elastic instances: Elastic instances replicate model weights, while the global manager assigns them to parallel groups with batch-specific degrees of parallelism before each iteration.The number of instances in a group determines its DoP.
- Elastic scale-down: After prefill, LoongServe can scale down decoding batches because decoding requires substantially less computation, optionally releasing resources for prefill batches.Scaling decisions account for changing decoding computation and KV-cache capacity.
- Elastic scale-down: Reactive scale-down migrates KV tensors after prefill, with overhead increasing linearly with sequence length and requiring substantial unused memory in every instance.For a 1M-token request, KV tensors can exceed 488GB; uneven allocation can instead create computational imbalance.
- Elastic scale-down: Proactive migration retains selected KV tensors in the future smaller group during prefill, reusing sequence-parallel communication to achieve zero-overhead scale-down.It supports token-level allocation based on instance memory without computational load imbalance and reuses existing buffer space.
- Elastic scale-up: During decoding scale-up, multi-master distributed decoding lets multiple instances compute local layers and store corresponding KV tensors locally, avoiding migration of existing caches.Adding instances requires joining the parallel group and executing, while multi-master coordination improves computation distribution and overlaps query exchange with local attention.
5 LoongServe Global Manager
LoongServe’s global manager converts the dynamic scheduling space into four polynomial-time steps: dispatching, elastic instance allocation, batching, and elastic scaling. These steps jointly select requests, resources, batches, and scaling actions while accounting for GPU computing, GPU memory, interference, migration cost, and latency.
- Scheduling overview: The global manager decouples scheduling into dispatching, elastic instance allocation, batching, and elastic scaling plan generation, then combines their outputs.Each subproblem is solved in polynomial time to meet real-time scheduling constraints.
- Dispatching: Dispatching selects pending requests in first-come-first-served order while checking GPU computing and key-value-cache memory constraints.Requests are rejected when insufficient cache capacity could trigger eviction and recomputation.
- Dispatching: The manager stops adding requests when a prefill batch crosses the profiled memory-bound-to-compute-bound tipping point.Before the tipping point, additional requests improve GPU efficiency; afterward, they mainly extend execution time.
- Elastic instance allocation: Elastic instance allocation repeatedly adds the instance with fewest used key-value-cache slots while latency gain exceeds migration cost.Migration cost depends on the existing cache volume and average bandwidth to target instances.
- Batching: Batching assigns requests with different sequence lengths to elastic instances and optimizes their degree-of-parallelism allocation to minimize prefill latency.The dynamic program uses token counts, unused cache slots, and interval input-latency sums.
- Elastic scaling: LoongServe scales up when decoding becomes compute-bound or GPU memory is insufficient, using a profiled batch-size threshold for compute pressure.The multi-master decoding mechanism is used when it can reduce memory fragmentation or address the scaling condition.
6 Implementation
LoongServe combines Python-based global management with C++ batching logic and Ray/NCCL communication across elastic instances. Its implementation manages token-granular key-value caches, supports ESP-aware attention kernels, and provides profiling infrastructure for analytical models.
- System implementation: LoongServe is implemented in approximately 15K lines using C++, CUDA, Python, and Triton, reusing components from vLLM and LightLLM.The code is publicly available.
- System implementation: The global manager is mainly implemented in Python, while core batching logic uses C++ and concurrent batches are assigned to Python coroutines.This separates orchestration from performance-sensitive looped functions.
- Communication: Ray carries communication with elastic instances, while NCCL broadcasts information within tensor-parallel instances to reduce serialization overhead.ESP metadata is cached and RPC parameters are designed to limit added serialization cost.
- GPU kernels and memory: Each elastic instance manages a token-granular key-value-cache pool with PagedAttention and uses tuned StripedAttention and custom Flash-Decoding implementations.The optimizations support MHA, MQA, and GQA with the same accuracy as the original implementations.
- Communication: Inter-instance communication uses NCCL with dedicated CUDA streams, separate tensor- and sequence-parallel communicators, and grouped point-to-point operations.These mechanisms support multiple dynamic parallel groups at iteration level.
- Profiling: LoongServe stores profiling results in SQLite so analytical models can be trained by selecting scenario-specific measurements.The profiling tools generate results under different scenarios.
7 Evaluation
LoongServe is evaluated on real-world workloads using throughput and normalized latency metrics across single- and multi-node settings. It outperforms baselines through elastic parallelism, while ablations show benefits from dynamic scheduling, elastic scale-up, and low-overhead scaling.
- Overall performance: LoongServe improves total throughput and input throughput by up to 3.85× and 3.37×, respectively, compared with other systems.Its output latency is also significantly better than other baselines because elastic instances protect decoding from prefill interference.
- Multi-node performance: LoongServe achieves the best performance across all metrics in a 16-GPU multi-node setting by choosing an appropriate DoP for each request.The experiment extends ESP to 8 while retaining the same model-parallelism strategy used in the single-node evaluation.
- Elastic sequence parallelism: ESP improves P90 goodput by 2.33×, 1.98×, and 1.53× under different sequence-length distributions.Traditional tensor parallelism, static hybrid parallelism, and replication do not adequately handle dynamic inference workloads.
- Elastic scale-up: Elastic scale-up raises ShareGPT P90 goodput by 2.87× over disabling elastic scale-up.The global manager triggers an average of 7.12 elastic scale-up operations per 10 seconds at an arrival rate of 25 requests per second.
- Scaling overhead: Scaling down adds less than 2% overhead, while scaling up remains below 10% overhead for small batches and can halve per-iteration latency for large batches.LoongServe dynamically selects the better option according to workload conditions.
- Analytical model: LoongServe’s analytical model deviates by less than 10% across batch sizes, sequence lengths, and parallelism strategies.The evaluation finds the model reliable for guiding the global manager.
8 Related Work
LoongServe differs from prior LLM serving, sequence-parallel training, long-context accuracy-efficiency, and elastic-training work by targeting elastic serving under dynamic request and phase demands. Its design combines elastic scaling with ESP while addressing serving-specific cache, decoding, batching, and latency constraints.
- LLM serving systems: Prior serving systems use chunking or phase disaggregation, but they can retain interference, locality constraints, or static resource allocation limitations.LoongServe instead introduces elastic scaling and ESP without locality constraints.
- Sequence parallelism: Sequence-parallel training methods generally use fixed DoP and focus on training, whereas LoongServe applies elastic sequence parallelism to serving.Its serving setting includes dynamic request phases and stricter latency constraints.
- Long-context LLMs: Long-context methods that alter attention or prune key-value caches trade efficiency for accuracy, while LoongServe preserves the original LLM accuracy.The paper also states compatibility with MQA, GQA, and MoE.
- Elastic training: Unlike elastic DNN training, LoongServe proposes a serving-oriented parallelism strategy and accounts for decoding, key-value tensor management, request batching, and strict latency constraints.The comparison highlights serving factors not present in the cited training-oriented work.
9 Conclusion
The paper proposes ESP and builds LoongServe to serve long-context LLMs under dynamic workloads. Across diverse real-world datasets, it reports simultaneous performance improvements for prefill and decoding over existing solutions.
- Conclusion: LoongServe combines ESP, elastic scaling mechanisms, and a scalable scheduling algorithm for dynamic long-context LLM workloads.The paper presents these components as its core system contribution.
- Conclusion: Evaluation across diverse real-world datasets shows significant performance improvements in both the prefill and decoding phases.The conclusion reports simultaneous gains over existing solutions.