Source-linked AI summary
Llumnix: Dynamic Scheduling for Large Language Model Serving
Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, Wei Lin
TL;DR
LLM serving must handle requests with heterogeneous latency and resource requirements whose execution and memory demands are unpredictable, limiting existing scheduling approaches. Llumnix responds with runtime rescheduling across model instances, implemented through live migration and a unified dynamic scheduling policy. It improves tail-latency metrics, accelerates high-priority requests, and reduces cost in the reported evaluations.
Problem
Heterogeneous requests and unpredictable execution make efficient LLM serving difficult, while existing systems inadequately address cross-instance scheduling and multitenant goals.
Method
Llumnix uses runtime rescheduling across model instances, with live migration of requests and GPU memory states plus a distributed policy based on unified virtual usage.
Results
15× P99 first-token latency improvement and 2× P99 per-token generation latency improvement are reported against INFaaS, with 1.5× faster high-priority requests and 36% cost savings at similar tail latencies.
Takeaways & Limitations
Llumnix demonstrates that runtime request migration can support load balancing, isolation, prioritization, and resource de-fragmentation in multi-instance LLM serving.
Abstract
from arXiv · showhide
Inference serving for large language models (LLMs) is the key to unleashing their potential in people's daily lives. However, efficient LLM serving remains challenging today because the requests are inherently heterogeneous and unpredictable in terms of resource and latency requirements, as a result of the diverse applications and the dynamic execution nature of LLMs. Existing systems are fundamentally limited in handling these characteristics and cause problems such as severe queuing delays, poor tail latencies, and SLO violations. We introduce Llumnix, an LLM serving system that reacts to such heterogeneous and unpredictable requests by runtime rescheduling across multiple model instances. Similar to context switching across CPU cores in modern operating systems, Llumnix reschedules requests to improve load balancing and isolation, mitigate resource fragmentation, and differentiate request priorities and SLOs. Llumnix implements the rescheduling with an efficient and scalable live migration mechanism for requests and their in-memory states, and exploits it in a dynamic scheduling policy that unifies the multiple rescheduling scenarios elegantly. Our evaluations show that Llumnix improves tail latencies by an order of magnitude, accelerates high-priority requests by up to 1.5x, and delivers up to 36% cost savings while achieving similar tail latencies, compared against state-of-the-art LLM serving systems. Llumnix is publicly available at https://github.com/AlibabaPAI/llumnix.
1 Introduction
LLM serving faces heterogeneous, unpredictable requests that expose limitations in existing scheduling systems. Llumnix addresses these challenges through runtime request rescheduling, live migration, and unified distributed scheduling.
- Challenges: Heterogeneous request lengths, latency requirements, and unpredictable execution demands make LLM serving a dynamic multitenant scheduling problem.Output lengths and GPU memory demands can grow unpredictably during iterative generation.
- Challenges: Existing systems primarily optimize throughput within individual instances, while cross-instance scheduling receives comparatively little attention.This leaves important multitenant goals such as isolation and priority handling insufficiently addressed.
- Llumnix: Llumnix reschedules requests across model instances to balance load, reduce fragmentation and queuing, prioritize urgent requests, and support faster instance scaling.The design treats rescheduling similarly to context switching across CPU cores.
- Evaluation: 15× higher P99 first-token latency improvement and 2× higher P99 per-token generation latency improvement are achieved against INFaaS, alongside 1.5× faster high-priority requests and 36% cost savings at similar tail latency.The system is implemented as a scheduling layer over inference engines and evaluated on a 16-GPU cluster.
- Llumnix: Llumnix implements rescheduling with an efficient migration mechanism for requests and their GPU memory states.The migration coordinates computation and memory transfer to provide near-zero downtime that remains constant with sequence length.
- Llumnix: A distributed scheduling architecture and virtual-usage policy unify multiple rescheduling goals through continuous, scalable load balancing.Virtual GPU-memory usage rules feed a simple load-balancing policy across instances.
2 Background
LLM applications are diverse and task-agnostic, producing requests with different sequence lengths and latency requirements. Autoregressive generation and dynamic KV-cache memory demands motivate continuous batching and dynamic memory allocation.
- Application diversity: Task-agnostic LLMs support diverse applications including chatbots, search, summarization, coding, assistants, and agents.Context-specific prompts allow the same model to serve different tasks.
- Application diversity: Different applications create requests with varied input and output lengths, expected latencies, and urgency levels.Summarization needs long inputs, writing needs long outputs, and interactive applications generally expect shorter latency.
- Autoregressive generation: Autoregressive inference generates one token per iteration while repeatedly using the input and previously generated tokens.The number of generated tokens is not known beforehand, making execution duration dynamic.
- Memory management: The KV cache stores intermediate key and value tensors in GPU memory so they can be reused when generating later tokens.These states participate in the generation of all following tokens.
- Batching and memory management: Continuous batching lets requests join or leave a running batch immediately, while dynamic KV-cache allocation avoids reserving memory for maximum sequence lengths.These techniques accommodate varying sequence lengths and dynamic arrivals while supporting larger batches and higher throughput.
3 Motivation
LLM serving experiences preemptions, performance interference, and memory fragmentation because request lengths and memory demands vary dynamically. These effects create severe tail-latency and queuing challenges even under moderate load.
- Performance interference among requests: Decode-step latency varies with sequence length and batch size for both LLaMA-7B and LLaMA-30B.Figure 4 evaluates one decode step across different sequence lengths and batch sizes.
- Unpredictable memory demands and preemptions: P99 per-token decode latency was 3.8× P50, with preemption loss accounting for 70% of the P99 request.The P99 request incurred 50 seconds of total preemption loss after being preempted twice.
- Unpredictable memory demands and preemptions: A P99 request experienced 50 seconds of preemption loss, producing severe service stalls and user-experience degradation.The request was preempted twice; the loss included extra queuing and recomputation of previous KV cache.
- Memory fragmentation: Memory fragmentation can leave free memory unable to satisfy head-of-line requests across four LLaMA-7B instances.Figure 5 compares total free memory with the demands of head-of-line queuing requests.
4 Llumnix Design
Llumnix dynamically reschedules LLM requests across instances using live migration, distributed scheduling, and virtual usage. Its policy targets latency, load adaptability, request priorities, and cost efficiency.
- Overview: Llumnix reschedules requests across model instances to react to workload dynamics and support load balancing, defragmentation, and priority differentiation.The design treats runtime rescheduling analogously to context switching across CPU cores.
- Overview: Naïve KV-cache recomputation or copying can cause computation stalls and downtime exceeding 50× decoding cost.The large, sequence-length-dependent KV cache limits scheduling flexibility under growing context lengths.
- 4.2 Live Migration of LLM Requests: Llumnix pipelines KV-cache copying with decoding because the cache is append-only, reducing request downtime to the copying period for one iteration.The total copy duration depends on sequence length, but downtime remains near-zero and constant to sequence length.
- 4.3 Distributed Scheduling Architecture: Llumnix combines a cluster-level global scheduler with distributed instance-level llumlets for continuous rescheduling at scale.The global scheduler pairs source and destination instances using reported loads, while llumlets select and migrate requests.
- 4.4.1 Goals and Definitions: Auto-scaling keeps cluster load appropriate for cost efficiency and the benefits of rescheduling.The policy is designed to improve prefill and decode latencies by reducing queuing delays, preemptions, and interference.
- 4.4.1 Goals and Definitions: Llumnix supports scheduling and execution priorities so higher-priority requests receive earlier scheduling and lower instance load.The implementation demonstrates high and normal classes while allowing generalization to more priorities.
- 4.4.3 Policies: The virtual usage abstraction unifies different rescheduling goals into a simple instance-level load metric.Rules assign request virtual usages, and a load-balancing policy uses them to guide dispatch and migration.
- 4.4.3 Policies: Instance freeness is F = (M −∑V)/B, so Llumnix dispatches each request to the instance with the highest freeness.M is total memory, V is request virtual usage, and B is batch size; negative values mark overloaded instances.
5 Implementation
Llumnix is implemented as a standalone, extensible library integrated with backend inference engines. Its components use distributed actors, an OpenAI-style frontend, and KV-cache transfer mechanisms, with fault-tolerance behavior for failures.
- Implementation: Llumnix comprises 3,300 lines of Python and provides its own components plus an interface for backend integration.The architecture is non-intrusive and extensible; the current backend is vLLM.
- Implementation: Llumnix instantiates backend instances and components as Ray actors and exposes an OpenAI-style API through request frontend actors.Generated tokens are forwarded through the frontend even when requests migrate between backend instances.
- Implementation: KV-cache transfer during migration uses the Gloo collective communication library with Send/Recv primitives.This mechanism supports copying KV-cache state between instances.
- Fault tolerance: When the global scheduler fails, Llumnix falls back to scheduler-bypassing dispatch while disabling migration.When an instance or colocated llumlet fails, requests running on it are aborted.
6 Evaluation
Llumnix is evaluated on realistic workloads using a 16-GPU cluster, where it improves migration efficiency, serving latency, priority handling, cost efficiency, and scheduling scalability.
- Serving performance: 15× P99 and 7.7× mean prefill-latency improvements are achieved over INFaaS through de-fragmentation on 16 LLaMA-7B instances.Llumnix also improves P99 decode latency by up to 2× by reducing preemptions.
- Migration efficiency: Near-zero migration downtime remains roughly constant with sequence length, while baseline downtime reaches 111× Llumnix’s.Recomputing an 8k LLaMA-30B sequence takes 3.5 seconds, equivalent to about 54 decode steps.
- Migration efficiency: Migration changes per-step decode performance by at most 1% and occurs during only roughly 10% of each instance’s serving time.The measured migration overhead is therefore even smaller in effective operation.
- Serving performance: Llumnix outperforms INFaaS++ across real and generated traces, including up to 5.5× lower P99 prefill latency and 1.3× lower P99 decode latency.Across generated traces, improvements reach 14.8× for P99 prefill latency.
- Priority support: 1.2×–1.5× mean high-priority latency gains persist as workload burstiness increases, while normal-request mean latency rises by at most 4.5%.High-priority prefill and decode latencies also improve, with P99 gains reaching 10× and 2.2× respectively.
- Cost efficiency: 36% cost savings are achieved while preserving similar P99 prefill latency through migration-assisted queue reduction and more efficient auto-scaling.The similar P99 prefill latency is roughly 5 seconds.
- Scalability: At high request rates, Llumnix avoids the baseline’s scheduling stalls of up to 40 ms per iteration and its resulting 1.7× slowdown.The baseline stalls arise from centralized scheduler communication and synchronization.
7 Related Work
Related work spans optimized LLM inference, cost-efficient and preemptive serving, model-variant scheduling, and migration techniques for dynamic workloads.
- LLM inference: Prior LLM-serving systems optimize inference kernels, cost efficiency, completion times, or preemptive time slicing.Examples include FasterTransformer, TurboTransformer, LightSeq, FlashAttention, SpotServe, and FastServe.
- Migration: The append-only KV cache enables migration and opens policy space for priority, isolation, memory efficiency, and auto-scaling.The paper identifies global-local scheduling interplay as future work.
- Request scheduling: INFaaS schedules across model types and variants to account for application-specific performance and accuracy requirements.Potential LLM variants include fine-tuned models and models with different sizes or precisions.
- Migration: Llumnix uses runtime request migration to address unpredictable autoregressive execution while reconciling packing and spreading goals.This extends migration ideas previously applied to deep-learning training jobs.
8 Conclusion
Llumnix advances a vision of serving LLMs as Unix by applying operating-system abstractions to their universal, multitenant, and dynamic workload characteristics.
- 8 Conclusion: Llumnix draws on operating-system wisdom to introduce abstractions such as isolation and priorities for LLM serving.The vision is motivated by shared universality, multi-tenancy, and dynamism between LLMs and modern operating systems.
Abstract
The artifact provides the code and scripts needed to reproduce the paper’s experiments and evaluation results.
- Abstract: The artifact includes source code and experiment scripts for reproducing the evaluation results.
Scope
The artifact reproduces Llumnix’s migration-efficiency, serving-performance, priority-support, and auto-scaling experiments.
- Figure 10 evaluates migration efficiency.
- Figure 11 evaluates serving performance.
- Figure 13 evaluates support for priorities.
- Figures 14 and 15 evaluate auto-scaling.
Hosting
The artifact is publicly available in the osdi24ae branch of Llumnix’s GitHub repository, which differs from the official release’s main branch.
- The artifact is publicly available at the AlibabaPAI/llumnix GitHub repository on the osdi24ae branch.This is not the same branch as Llumnix’s official release; their differences are described later.
Requirements
Reproducing the artifact’s results requires a specific GPU setup, while production or further research should use the official release because the artifact is a research prototype with fewer capabilities.
- Reproduction requires four GPU machines, each with four 24 GB A10 GPUs.The artifact runs on GPU machines, with dependencies mostly matching vLLM.
- The artifact is a research prototype used for the paper’s experiments.
- The artifact is directly based on vLLM, whereas the official release is a standalone Python library.The standalone design makes the official release more extensible and non-intrusive to backend inference engines.
- The artifact is not fault-tolerant, whereas the official release provides fault tolerance for each component.
- The official release supports or plans newer vLLM versions, more models, distributed request frontends, and improved scheduling policies.
- The artifact reproduces the paper’s experiments, but the official release is recommended for production or further research.