Source-linked AI summary
ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, Luo Mai
TL;DR
LLM serverless inference must meet low-latency demands despite large checkpoints, cold starts, and unpredictable interactive workloads. ServerlessLLM uses local multi-tier storage, loading-optimized checkpoints, live migration, and locality-aware scheduling; evaluations report 10 to 200 times latency improvements across real-world workloads. Its current sequential I/O loading policy does not yet provide concurrent loading with a fairness guarantee.
Problem
LLM serverless inference faces high cold-start latency because large checkpoints require costly downloads and loading, while interactive inference has unpredictable duration.
Method
ServerlessLLM combines loading-optimized multi-tier checkpoint storage, token-only live migration with destination KV-cache recomputation, and locality-aware startup scheduling.
Results
10 to 200 times improvement in latency was achieved for OPT inferences across GSM8K and ShareGPT workloads against KServe, Ray Serve, and a locally caching Ray Serve variant.
Takeaways & Limitations
ServerlessLLM demonstrates that local checkpoint storage, efficient migration, and locality-aware scheduling can jointly support lower-latency serverless LLM inference.
Takeaways & Limitations
ServerlessLLM currently uses sequential model loading on the I/O path; concurrent loading with a fairness guarantee remains future work.
Abstract
from arXiv · showhide
This paper presents ServerlessLLM, a distributed system designed to support low-latency serverless inference for Large Language Models (LLMs). By harnessing the substantial near-GPU storage and memory capacities of inference servers, ServerlessLLM achieves effective local checkpoint storage, minimizing the need for remote checkpoint downloads and ensuring efficient checkpoint loading. The design of ServerlessLLM features three core contributions: (i) \emph{fast multi-tier checkpoint loading}, featuring a new loading-optimized checkpoint format and a multi-tier loading system, fully utilizing the bandwidth of complex storage hierarchies on GPU servers; (ii) \emph{efficient live migration of LLM inference}, which enables newly initiated inferences to capitalize on local checkpoint storage while ensuring minimal user interruption; and (iii) \emph{startup-time-optimized model scheduling}, which assesses the locality statuses of checkpoints on each server and schedules the model onto servers that minimize the time to start the inference. Comprehensive evaluations, including microbenchmarks and real-world scenarios, demonstrate that ServerlessLLM dramatically outperforms state-of-the-art serverless systems, reducing latency by 10 - 200X across various LLM inference workloads.
1 Introduction
ServerlessLLM targets low-latency LLM serverless inference by exploiting local GPU-server storage and memory to reduce checkpoint-loading overhead. It combines fast multi-tier loading, live inference migration, and startup-time-aware scheduling, with evaluations showing substantial latency reductions.
- Motivation: LLM serverless inference faces strict response-time requirements, high GPU consumption, and unpredictable latency from iterative token generation.Interactive requests may require multiple GPUs for seconds to minutes, while output length varies across requests.
- Motivation: Cold starts are especially costly because LLM checkpoints span gigabytes to terabytes and require complex tensor deserialization, allocation, and shape parsing.Remote downloads and tensor-processing steps compound startup delays.
- Design: ServerlessLLM exploits underused GPU-server storage hierarchies to store checkpoints locally and reduce remote checkpoint downloads.The design addresses local storage across multiple tiers rather than relying primarily on remote repositories.
- Design: Its fast multi-tier loader combines a loading-optimized checkpoint format with a storage system that uses local capacity and bandwidth across tiers.The format supports sequential chunk reads and efficient tensor addressing, while the subsystem uses chunk pooling, memory-copy-efficient paths, and staged loading.
- Design: Live migration transfers tokens instead of the large KV-cache and recomputes the cache at the destination to support locality-driven inference efficiently.This design reduces migration network traffic while allowing new inferences to use local checkpoints.
- Design: Startup-time-optimized scheduling estimates checkpoint-loading and inference-migration costs, then selects servers that minimize model startup latency.The scheduler accounts for checkpoint locality across GPU servers.
- Evaluation: 3.6 - 8.2X faster checkpoint loading and 10 to 200 times lower inference latency were achieved against existing loaders and serverless baselines.The evaluation covered large models, LoRA adaptors, and real-world workloads modeled on the Azure Trace.
2 Background and Motivation
Serverless inference reduces deployment costs and supports dynamic workloads, but LLM cold starts create severe latency because models are large and expensive to download and load. Existing remedies—including warmed GPUs, host-memory caching, and additional storage servers—remain constrained for LLM-scale checkpoints.
- Why serverless inference: Serverless inference lets providers multiplex models while charging users for inference duration and processed data.It is particularly suited to applications with dynamic or unpredictable demand.
- System background: A controller routes requests to running instances or activates inference processes on unallocated GPUs through a model-loading scheduler.The request router and scheduler coordinate request placement and model startup.
- System background: LLM inference streams autoregressively generated tokens and uses a KV-cache, with performance measured by first-token and per-token latency.The generated sequence length makes total inference time nondeterministic.
- Cold-start challenge: Over 40% of Azure-traced functions have cold-start rates exceeding 25%, while approximately 25% exceed 60% within a 5-minute keep-alive interval.The paper relates these cold starts to very high LLM initialization latency.
- Cold-start challenge: LLM checkpoints can exceed hundreds of gigabytes, making remote downloads costly and prolonging cold starts.Examples include Grok-1 checkpoints over 600 GB, DBRX at 250GB, and Mixtral-8x22B at about 280GB.
- Existing solutions: Existing approaches warm GPUs, cache checkpoints in host memory, or add storage servers, but each has limitations for large LLMs.Host-memory caching is inadequate for checkpoints exceeding hundreds of gigabytes, and optimized local-server downloads can still exceed 20 seconds.
3 Exploiting In-Server Multi-Tier Storage
ServerlessLLM leverages underused memory and storage in GPU servers as a local multi-tier checkpoint hierarchy. Its design must also handle locality-aware placement, live migration, and startup-time estimation for interactive, unpredictable LLM workloads.
- Design rationale: ServerlessLLM addresses high model download and loading times with a cost-effective, scalable, and long-term viable design.The approach reuses storage resources already present in inference servers.
- Design rationale: An 8-GPU server can provide up to 4 TBs of main memory, 64 TBs of NVMe SSDs, and 192 TBs of SATA SSDs, much of which remains underused.These capacities motivate local checkpoint storage.
- Design rationale: GPU-server links provide substantial aggregate bandwidth between memory, GPUs, and SSD tiers for local checkpoint movement.The paper cites 512 GB/s between host memory and GPUs and around 60 GB/s from NVMe SSDs for an 8-GPU server.
- Design rationale: Using unused in-server storage makes local model loading cost-effective and scalable as inference servers are added.The approach is intended to reduce latency by loading models from local tiers.
- Design concerns: Local checkpoint loading alone is insufficient because request scheduling must account for checkpoint locality and unpredictable, interactive inference durations.These properties motivate live migration rather than relying on inaccurate duration predictions or preemption.
- Design overview: The controller estimates startup times from cluster-wide checkpoint locality and selects resources to minimize model startup latency.The overview combines multi-tier loading, live migration, and locality-aware scheduling.
4 Fast Multi-Tier Checkpoint Loading
ServerlessLLM converts checkpoints into a loading-optimized format and uses a multi-tier subsystem to exploit GPU-server bandwidth and capacity. It separates loading from inference initialization and coordinates chunk management, direct data paths, and tier-specific pipelines.
- Objectives: The loading subsystem targets full utilization of multi-tier local-storage bandwidth, predictable performance, and framework-generic checkpoint support.These objectives reflect serverless inference’s repeated-load workload.
- Loading-optimized checkpoints: Because inference checkpoints are uploaded once but loaded repeatedly, ServerlessLLM converts them into a loading-optimized format.This reverses the usual training and debugging pattern of frequent writes and infrequent loads.
- Loading-optimized checkpoints: The format groups tensors by GPU into sequential binary chunks and uses an index mapping tensor names to GPU id, offset, and size.These structures support large sequential reads and direct tensor restoration.
- Loading architecture: A model manager loads checkpoint data while the inference process initializes tensor pointers, allowing loading and initialization to be pre-scheduled and overlapped.Synchronization ensures the model is fully initialized before inference begins.
- Multi-tier subsystem: Chunk-based management uses parallel PCIe links and application-specific allocation and eviction controls for concurrent, fine-grained data handling.The memory pool manages cached or evicted chunks beyond simple caching.
- Multi-tier subsystem: The data path uses direct file access to reduce copying and improve consistency relative to memory-mapped files.The design prioritizes predictable loading performance.
- Multi-tier subsystem: A multi-tier pipeline provides dedicated access methods for local, remote, and in-memory storage interfaces and supports adding new tiers.I/O threads pass chunk indices between successive storage tiers.
5 Efficient Live Migration of LLM Inference
ServerlessLLM uses live migration to exploit checkpoint locality while avoiding the queuing delays, underutilization, and interruption of simpler policies. It transfers compact inference state and recomputes the KV-cache at the destination through a multi-round process.
- Design objectives: Live migration is designed to minimize resource overhead and user interruption while supporting locality-driven inference.The design addresses both minimal migrated state and rapid destination synchronization.
- Motivation: Live migration improves latency for both models by combining checkpoint locality with continued inference operation.The approach outperforms availability-, locality-, and preemption-driven policies in the example.
- Migration design: The source migrates tokens instead of the large KV-cache, reducing network traffic while the destination recomputes the KV-cache.Migrated tokens are typically 10-100s KB, whereas the KV-cache is typically 1-10s GB.
- Migration design: Destination recomputation is usually an order of magnitude faster than generating equivalent new tokens, enabling rapid multi-round convergence.Each round narrows the token gap between source and destination until migration can complete.
- Migration process: The migration process loads the model at the destination, transfers intermediate tokens, recomputes the KV-cache, and reroutes inference after the source stops.The scheduler then unloads the source model and can start loading another model.
6 Startup-Time-Optimized Model Scheduling
ServerlessLLM schedules models by estimating loading and migration times across servers and storage tiers. It combines queue-aware bandwidth estimates with migration-aware server selection, while sequential loading remains a fairness limitation.
- Scheduler design: The scheduler estimates loading and migration times to select the server with the lowest estimated model startup time.It evaluates server and GPU-slot assignments and can defer tasks when no suitable GPU is available.
- System operation: The scheduler maintains per-server task queues and reliable server-status updates to support estimation continuity and failure recovery.Asynchronous operations allow the scheduler to handle thousands of loading tasks per second on a standard server.
- Loading-time estimation: Loading time is estimated as q + n/b, combining queueing time, model size, and available transfer bandwidth.Bandwidth is tracked across network, SSD, and DRAM, while q accumulates from earlier queued models.
- Loading-time estimation: The scheduler uses sequential per-server loading and the slowest bandwidth in multi-tier pipelines to reduce contention and capture the bottleneck.It also updates bandwidth estimates using observed loading latency.
- Migration-time estimation: Migration-time estimation focuses on model resuming time using input tokens, generated output tokens, and model-specific parameters.The scheduler estimates output tokens from inference duration and average token-generation time.
- Practical concerns: Concurrent loading with a fairness guarantee remains future work because the current I/O path uses sequential model loading.The system treats models with equal importance and aims to prevent migrations from affecting latency.
7 Evaluation
Across microbenchmarks and end-to-end workloads, ServerlessLLM substantially reduces checkpoint-loading and inference startup latency, while locality-aware scheduling and live migration remain effective under contention until GPU resources become limiting.
- Checkpoint loading: 3.6–8.2X faster checkpoint loading than Safetensors and PyTorch across evaluated LLMs, with 4.4X faster LoRA-adapter loading.For OPT-2.7B, ServerlessLLM is 3.6X faster than Safetensors; for LLaMA-2-70B, it is 4.7X faster than Safetensors and 8.2X faster than PyTorch.
- Checkpoint loading: ServerlessLLM saturates diverse storage-device bandwidths and benefits particularly from faster RAID0-NVMe storage.Its direct-I/O and chunk-based parallel loading design is better suited to newer, faster storage than PyTorch and Safetensors.
- Checkpoint loading: 1.2X–2.3X throughput gains come from bulk reading, direct I/O, multithreading, pinned memory, and pipelining.The measured improvements are 1.2X for bulk reading, 2.1X for direct I/O, 2.3X for multithreading, 1.4X for pinned memory, and 1.5X for pipelining.
- Model scheduling: At RPS 1.4 on GSM8K, ServerlessLLM beats Shepherd* and the random serverless scheduler by 1.27X and 1.95X on P99 latency.The comparison involves 53 migrations for ServerlessLLM and 9 preemptions for Shepherd*, with preemptions producing longer latency than migrations.
- Model scheduling: For larger models, locality-aware schedulers perform better, while ServerlessLLM’s effectiveness becomes constrained when GPU resources are exhausted.With ShareGPT, latency reaches 89.9 seconds for OPT-30B when all GPUs are occupied and migration cannot free additional resources.
- End-to-end serving: 0.8 seconds versus 12.1 seconds for Ray Serve and 8.2 seconds for Ray Serve with Cache when starting OPT-6.7B on GSM8K.Even with a 100 Gbps network, Ray Serve remains 4.7 times slower; for OPT-30B, ServerlessLLM starts in 7.5 seconds versus 213 seconds for Ray Serve.
- End-to-end serving: 89% of OPT-30B requests meet a 300-second timeout with ServerlessLLM, compared with 26% for Ray Serve with Cache.The result reflects the substantial user-facing impact of lower model-start latency.
- End-to-end serving: 212 times better latency than Ray Serve and Ray Serve with Cache for ShareGPT across RPS 0.2–1.1, before GPU limits raise latency at RPS 1.4.With GSM8K, ServerlessLLM maintains approximately one-second latency as RPS increases, while both Ray Serve variants rise after RPS exceeds 0.5.
8 Related Work
Prior serverless research targets batching, scheduling, resource efficiency, and generic cold-start reduction, while LLM serving research improves inference efficiency through batching, parallelism, cache management, and storage offloading. These approaches generally do not address the full challenge of low-latency LLM serverless inference.
- Serverless inference systems: Serverless inference research addresses batching, scheduling, and resource efficiency through academic and industry systems.
- Serverless cold-start optimizations: Generic cold-start methods accelerate containers or virtual machines but typically do not load large external model states.
- Comparison: ServerlessLLM differs by targeting LLM-specific cold-start latency rather than relying solely on techniques developed for containers, virtual machines, or general model serving.
- LLM serving systems: LLM serving systems improve utilization, throughput, cache management, or parameter capacity through continuous batching, model parallelism, PagedAttention, phase separation, and storage offloading.
9 Conclusion
ServerlessLLM is presented as a first step toward low-latency serverless computing for LLMs, using loading-optimized checkpoints, live migration, and locality-aware scheduling. The authors identify fairness and checkpoint placement as future extensions and position the system as a platform for further research.
- Core design: ServerlessLLM combines loading-optimized checkpoints, live migration, and checkpoint-locality-aware cluster scheduling for low-latency LLM inference.
- Future work: The authors identify fairness across the cluster and smart checkpoint placement as open directions for future work.
- Research platform: The system is positioned as an initial platform for testing new research ideas in serverless computing for LLMs.