SpotServe: Serving Generative Large Language Models on Preemptible Instances

Xupeng Miao; Chunan Shi; Jiangfei Duan; Xiaoli Xi; Dahua Lin; Bin Cui; Zhihao Jia

Source-linked AI summary

SpotServe: Serving Generative Large Language Models on Preemptible Instances

Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, Zhihao Jia

arXiv:2311.15566v1Nov 27, 2023cs.DCcs.CLcs.LG

TL;DR

Serving generative LLMs cheaply is difficult because their computational and memory demands are high, while preemptible GPUs can be reclaimed at any time. SpotServe addresses this with adaptive parallelization, communication-efficient migration, and token-level inference recovery, achieving lower tail latency and cost than the stated baselines.

Problem
Generative LLMs are expensive to serve because of their high computational and memory requirements, motivating cheaper preemptible GPU infrastructure.
Method
SpotServe combines adaptive parallelization, bipartite-matching-based migration, and stateful token-level recovery for distributed LLM serving on preemptible instances.
Results
2.4–9.1× lower P99 tail latency and up to 54% lower monetary cost are reported compared with existing serving systems and on-demand instances, respectively.
Takeaways & Limitations
SpotServe demonstrates that distributed generative LLM serving can use preemptible instances while preserving close average inference latency and reducing cost.
Takeaways & Limitations
The approach relies strongly on cloud-provided grace periods and primarily targets homogeneous single-type GPU instances.

Abstract

from arXiv · show

The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPUs at a much cheaper price than regular instances but may be preempted by the cloud at any time. Serving LLMs on preemptible instances requires addressing challenges induced by frequent instance preemptions and the necessity of migrating instances to handle these preemptions. This paper presents SpotServe, the first distributed LLM serving system on preemptible instances. Several key techniques in SpotServe realize fast and reliable serving of generative LLMs on cheap preemptible instances. First, SpotServe dynamically adapts the LLM parallelization configuration for dynamic instance availability and fluctuating workload, while balancing the trade-off among the overall throughput, inference latency and monetary costs. Second, to minimize the cost of migrating instances for dynamic reparallelization, the task of migrating instances is formulated as a bipartite graph matching problem, which uses the Kuhn-Munkres algorithm to identify an optimal migration plan that minimizes communications. Finally, to take advantage of the grace period offered by modern clouds, we introduce stateful inference recovery, a new inference mechanism that commits inference progress at a much finer granularity and allows SpotServe to cheaply resume inference upon preemption. We evaluate on real spot instance preemption traces and various popular LLMs and show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances.

1 Introduction

SpotServe addresses the cost and reliability challenges of serving generative LLMs on preemptible GPUs by adapting parallelism, minimizing migration overhead, and recovering inference progress after preemption.

GPT-3 requires more than 16 NVIDIA A100-40GB GPUs and costs over $66 per hour for a single inference pipeline.
SpotServe is the first distributed generative LLM serving system designed for spot instances, combining data, tensor, and pipeline parallelism.
Dynamic reparallelization: SpotServe dynamically adapts parallelization to changing spot-instance availability and workload while balancing throughput and latency.
Instance migration: The migration planner formulates device mapping as bipartite graph matching and uses the Kuhn-Munkres algorithm to minimize communication during reparallelization.
Grace-period utilization: Stateful inference recovery commits progress at token granularity and migrates committed key/value caches so inference can resume after preemption.
Evaluation: 2.4–9.1× lower P99 tail latency and up to 54% lower monetary cost are reported against existing serving systems and on-demand instances, respectively.

2 Background and Related Work

Generative LLM inference uses autoregressive decoding and KV caching, while distributed serving combines pipeline and tensor parallelism. These mechanisms create substantial sensitivity to preemption because one lost instance can disrupt an entire inference pipeline and discard cached progress.

Generative LLM inference: Autoregressive decoding processes the input initially, then repeatedly generates one output token using the input and previously generated tokens.
Generative LLM inference: KV caching avoids recomputing preceding tokens during attention, but its memory consumption grows with the output sequence length.
Distributed inference: Pipeline parallelism partitions consecutive Transformer layers into dependent stages that overlap execution and communicate across stages.
Distributed inference: Tensor model parallelism shards each operator and its tensors across devices, requiring collective communication such as All-Reduce.
Preemption challenges: A single preempted GPU can hang other instances in its pipeline, potentially breaking multiple pipelines hosted across the same instance.
Preemption challenges: When a GPU is preempted, existing systems may lose the request’s KV cache and restart inference after reinitializing the pipeline.

3 SpotServe Design

SpotServe coordinates adaptive LLM parallelization, context-preserving migration, and interruption recovery to maintain serving under changing spot-instance availability. Its design balances latency, throughput, cost, and migration memory constraints while exploiting cloud grace periods.

Interruption Recovery: SpotServe uses the cloud grace period to commit inference progress at token granularity and resume interrupted decoding with less lost work.The design addresses pipeline interruption and reinitialization overhead while supporting dynamic configuration transitions.
System Overview: SpotServe’s architecture combines a request manager, meta-context manager, instance manager, and GPU-resident inference engines with context daemons and interruption arrangers.The controller proposes configurations, while device mapping, migration planning, and interruption handling materialize transitions across instances.
Parallelization Controller: The adaptive optimizer selects parallel configurations using available instances, request arrival rate, throughput, latency, and monetary cost.It minimizes end-to-end latency subject to throughput requirements and prefers fewer instances when configurations have similar latency.
Device Mapper: SpotServe maps available GPUs to new pipeline-stage-shard positions with Kuhn-Munkres matching to minimize context-migration data transmission.The mapping reuses model parameters and KV cache instead of reinitializing all devices from scratch.
Migration Planner: Progressive migration prioritizes front pipeline layers so serving can resume before later layers finish transferring, while preserving all-layer cache context for interruption fault tolerance.This can overlap front-stage serving with subsequent migration and reduce migration overhead toward a single stage’s transfer cost.
Migration Planner: The migration planner tracks per-instance buffer memory, skips transfers exceeding U_max, and orders remaining layers through a min-max optimization.This prevents migration buffers from inflating peak memory and forcing higher-latency parallel configurations.

4 Stateful Inference Recovery

SpotServe recovers interrupted LLM inference by committing progress at token-level granularity and migrating cached request state, while arranging decoding work to fit available grace periods. It also includes mechanisms for consecutive interruptions and unexpected preemptions, though overlapping interruptions and underestimated migration costs remain limitations.

Recovery mechanism: Stateful inference recovery commits progress at the token level, allowing interrupted requests to resume without recomputing previously generated output tokens.A context daemon maintains each request’s cache context so another inference pipeline can continue from the saved state.
Just-in-time arrangement: SpotServe uses just-in-time arrangement to choose how many decoding iterations to run before an interruption, based on remaining grace time and migration cost.For preemption, the system maximizes feasible iterations; for acquisition, it minimizes iterations until the remaining time reaches the initialization threshold.
Just-in-time arrangement: The arrangement must ensure context migration does not increase request latency, and rerouting can be preferable when too little time remains to generate useful tokens.This trade-off is especially relevant when arrival requests are sparse and migration overhead would dominate.
Interruption fault-tolerance: Previous recovery arrangements handle only single interruptions, so consecutive compact interruptions or underestimated migration costs may leave insufficient time for decoding or context migration.Network vibration is given as an example of an unforeseen factor that can increase migration cost.
Interruption fault-tolerance: If an instance is preempted earlier than expected, SpotServe discards the cache context and migrates only model context; if all replicas are lost, it reloads weights locally or from remote storage.This fallback supports recovery when unexpected failures eliminate every replica of a model-context piece.

5 Implementation

SpotServe’s inference server combines C++ and Python components with FasterTransformer-based inference and TCP-delivered JSON migration plans.

Implementation: The SpotServe server comprises 5.6K lines of C++ and 2.2K lines of Python, including resident processes for request, instance, and meta-context management.Migration plans are serialized as JSON and sent to running instances over TCP.
Implementation: SpotServe builds its inference engine on FasterTransformer, using CUDA, cuBLAS, and C++.

6 Evaluation

SpotServe is evaluated against rerouting and reparallelization across real and synthesized traces, stable and fluctuating workloads, multiple LLMs, and cost settings. It consistently improves latency while reducing monetary cost, with its components contributing through adaptive parallelization, migration planning, and context recovery.

Experiment Setup: The evaluation uses OPT-6.7B, GPT-20B, and LLaMA-30B on replayed AWS spot-instance traces with stable and fluctuating request workloads.The real trace contains two representative 20-minute segments, and each replay instance has four NVIDIA Tesla T4 GPUs.
Stable Workload: SpotServe defeats Rerouting and Reparallelization on all latency metrics across four stable-workload traces.On LLaMA-30B, P99 latency improves 1.34–2.43× over Reparallelization and 2.14–9.13× over Rerouting.
Stable Workload: SpotServe avoids overload after preemption by selecting alternative parallel configurations, whereas Rerouting may drop a pipeline and accumulate requests.For GPT-20B, SpotServe changes to (D=2, P=3, M=4) after an instance loss, while Rerouting degenerates to (D=1, P=2, M=8).
Monetary Cost: 54% lower monetary cost is achieved with SpotServe while average latency rises less than 18% and P99 latency rises 90% relative to the cost comparison baseline.On-demand instances cost 3.9 USD/h versus 1.9 USD/h for spot instances in the GPT-20B comparison.
Ablation Study: Removing all optimizations increases tail latency by 1.61× on trace A_S and 3.41× on trace B_S.The ablation attributes improvements to the parallelization controller, migration planner, interruption arranger, and device mapper.
Fluctuating Workload: 2.94× and 1.73× lower P99 tail latency are achieved than Reparallelization and Rerouting, respectively, under fluctuating workloads.The fluctuating-workload experiment mixes on-demand instances with spot instances and uses rescaled MAF traces.

7 Related Work

Prior ML serving systems use spot instances mainly for small models, while general DNN serving improves utilization through scheduling and batching. These approaches do not directly address distributed LLM serving requirements, and serverless systems face GPU and communication constraints.

DNN Inference Systems: General DNN serving systems improve utilization through batching, scheduling, temporal multiplexing, or model selection.Examples include Clipper, Clockwork, Nexus, INFaaS, and Shepherd.
ML Serving over Spot Instance: Spot-instance ML serving systems such as Cocktail and MArk primarily target small models and address preemption through redundancy, rerouting, or resource provisioning.These approaches exploit spot instances for cost-effectiveness but are designed around models that fit on one spot instance or use data parallelism.
Serverless Computing and ML Serving: Serverless functions are poorly suited to distributed LLM inference because they have limited resources, difficult GPU provisioning, and no direct inter-function communication.Distributed LLM inference requires GPUs and communication among participating components.

8 Limitations and Future Work

The paper identifies reliance on cloud grace periods, focus on homogeneous single-type GPUs, latency-centered optimization, and limited parallelization exploration as boundaries for SpotServe and opportunities for future work.

Limitations: SpotServe’s proactive handling of instance availability changes strongly relies on the grace period offered by cloud providers.The paper proposes combining SpotServe with inference-workload or instance-availability prediction as a future direction.
Future Work: The current approach mainly targets single-type GPU instances, leaving heterogeneous spot instances and multi-cloud deployments for future exploration.These settings introduce additional context-migration challenges.
Future Work: SpotServe currently optimizes inference latency, while strict SLOs, high throughput, and emerging model variants could require other objectives or a larger configuration space.The paper specifically mentions mixture-of-experts models as a future target.

9 Conclusion

SpotServe serves generative LLMs across preemptible instances by adapting parallelization, migrating context efficiently, and recovering inference state during grace periods. Across real traces and LLM scales, it reduces tail latency and monetary cost relative to existing approaches and on-demand instances.

Conclusion: SpotServe dynamically adapts parallelization, performs efficient context migration, and uses stateful inference recovery for LLM serving on preemptible instances.Its configuration optimization considers throughput, latency, and monetary cost.
Conclusion: 2.4–9.1× lower P99 tail latency and 54% lower monetary cost are reported relative to existing approaches and on-demand instances, respectively.The evaluation uses real traces and popular LLMs at various scales.