Source-linked AI summary
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, Ion Stoica
TL;DR
Serving large models across bursty, multi-model workloads makes it difficult to meet latency constraints while using accelerator resources efficiently. AlpaServe automatically explores model-parallel and placement strategies, and production evaluations report higher request rates, lower latency deadlines, or greater burst tolerance than prior systems. Its future extensions include parameter-efficient adapted models, models with dependencies, and autoregressive models.
Problem
Large models’ memory and compute demands, combined with bursty multi-model requests, make serving costly, while model parallelism’s effects in latency-sensitive serving remain underexplored.
Source: §1 · p. 1Method
AlpaServe uses cluster resources, model sets, and periodic workload profiles to choose model partitions and placements and schedule requests for SLO attainment.
Source: §1 · p. 2Results
On production workloads, AlpaServe can increase request processing rate 10×, achieve 2.5× lower latency deadlines, or tolerate 6× burstier traffic than prior state-of-the-art systems, optimizing one metric at a time.
Source: §1 · p. 2Takeaways & Limitations
Model parallelism can be useful beyond models that exceed one GPU’s memory or required SLO, and AlpaServe provides techniques to navigate its trade-offs.
Source: §8 · p. 14The authors identify parameter-efficient adapted models, models with dependencies, and autoregressive models as future extensions.
Source: §8 · p. 14
Abstract
from arXiv · showhide
Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.
1 Introduction
Serving large models is costly under bursty, multi-model workloads, and the effects of model parallelism in latency-sensitive serving are underexplored. AlpaServe navigates the trade-offs between parallelism overhead and statistical multiplexing, with production results showing substantial gains over prior systems.
1 Introduction
- Large models can exceed a single device’s memory, while bursty demand and multiple model variants make provisioning for peak loads costly and leave resources underused.Common workload traces show demand spikes up to 50× the average.Source: §1 · p. 1
- Even when models fit on one GPU, model-parallel overhead can be outweighed by co-location’s statistical multiplexing benefit for bursty requests.In the two-model example, average latency falls from 2.5y to 1.85y despite 10% model-parallel overhead.Source: Figure 1 · p. 1, §1 · p. 2
- Model-parallel placement has overheads that may outweigh its benefits on less-bursty workloads, and its effectiveness also depends on memory capacity, arrival patterns, SLOs, and parallelism style.The paper frames these factors as a multi-dimensional, multi-objective design space.Source: §1 · p. 2
- AlpaServe uses cluster resources, models, and periodic workload profiles to place and partition models and schedule requests for SLO attainment.Its design analyzes parallelization trade-offs and includes simulator-guided co-location and cluster group-partition algorithms.Source: §1 · p. 2
- On production workloads, AlpaServe can increase request processing rate 10×, achieve 2.5× lower latency deadlines, or tolerate 6× burstier traffic than prior state-of-the-art systems, optimizing one metric at a time.The evaluation used a 64-GPU cluster.Source: §1 · p. 2
- AlpaServe’s contributions include analyzing model-parallel trade-offs, developing placement algorithms, and evaluating the system with synthetic and production workloads.Source: §1 · p. 2
2 Background
Model serving must balance latency SLOs against accelerator costs under unpredictable request bursts, while often supporting multiple related model instances. Model parallelism divides execution across devices, with intra-operator and inter-operator approaches offering distinct benefits and costs.
2 Background
- Serving systems face aggressive latency SLOs and ongoing accelerator costs; unpredictable bursts make peak provisioning costly and leave resources underused.Dynamic scaling can take multiple seconds because a large model must be swapped into accelerator memory.Source: §2 · p. 2
- Large-model serving commonly uses multiple instances of the same or similar architectures, including fine-tuned variants that may share some parameters or none.The paper excludes shared parameters because it targets general settings where full-weight tuning remains a major use case.Source: §2 · pp. 2–3
2.1 Model Parallelism in Model Serving
- Intra-operator parallelism splits an operator across devices, potentially reducing single-request latency and increasing available memory, at the cost of communication overhead.Source: §2.1 · p. 3
- Inter-operator parallelism assigns model operators to pipeline stages on different devices; it typically increases single-request execution time and is traditionally used to exceed one GPU’s memory limit.Source: §2.1 · p. 3
3 Motivation and Tradeoff Analysis
Model parallelism can trade execution overhead for statistical multiplexing across models, and its value depends on memory capacity, request patterns, and latency objectives. The analysis combines serving experiments with queueing theory to characterize that tradeoff.
3 Motivation and Tradeoff Analysis
- Partitioning models across devices can fit more models on a device, enabling statistical multiplexing for bursty requests.The paper examines this benefit alongside the overheads introduced by model parallelism.Source: §3 · p. 3
3.1 Case Study: A Two-model Example
- With equal-rate Poisson arrivals, model-parallel placement cuts mean latency from 0.70s to 0.55s (1.3×); burstier arrivals raise speedup to 1.9×.With an 80/20 request split, model-parallel placement reduces mean latency by 6.6×.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 When is Model Parallelism Beneficial
- When device memory is limited, model parallelism can improve serving performance; its benefit diminishes as memory lets replication fit more models per GPU.When a GPU can hold all models, the analysis finds no gain from model parallelism.Source: §3.2 · p. 4
- At low arrival rates and higher request CV, model parallelism performs better; near cluster saturation, replication can win as multiplexing benefits diminish and overhead matters.The comparison varies arrival rate and CV, with serving results shown in Figures 5 and 6.Source: §3.2 · pp. 4–5, Figure 5 · p. 4, Figure 6 · p. 4
- For tight SLOs below 10× model latency, model parallelism can improve SLO attainment; at looser SLOs its attainment plateaus while replication continues to improve.The paper notes that real-world SLO requirements are often less than 5× model execution latency.Source: §3.2 · p. 5, Figure 7 · p. 5
3.3 Overhead of Model Parallelism
- Inter-op parallelism mainly incurs uneven-stage overhead, while intra-op parallelism incurs higher communication overhead; intra-op reduces single-input latency, whereas inter-op achieves higher throughput.Both approaches split model weights across GPUs, keeping total memory usage constant as GPU count increases.Source: §3.3 · pp. 5–6, Figure 8 · p. 5, Figure 9 · p. 6
3.4 Queueing Theory Analysis
- Under deterministic service and Poisson arrivals, equal request shares make model-parallel waiting half of simple-placement waiting; skew raises simple-placement latency, while burstier arrivals permit higher overhead.The queueing analysis treats pipeline communication and uneven stages as separate overhead factors whose allowable maxima vary with total utilization.Source: §3.4 · pp. 6–7, Figure 10 · p. 6
4 Method
AlpaServe addresses inference parallelization and placement challenges with algorithms that optimize model configurations and cluster placements for SLO attainment. Its runtime centrally dispatches requests across model-serving groups.
4 Method
- AlpaServe uses a centralized controller to dispatch requests to groups, each of which hosts model replicas on a shared model-parallel runtime.The system targets efficient inference strategies and placements that maximize SLO attainment.Source: §4 · p. 7
4.1 Automatic Parallelization for Inference
- The inference compiler minimizes the maximum pipeline-stage latency and accelerates profiling by measuring individual layers and summing their latencies across candidate stages.Its intra-op pass excludes data parallelism, leaving model replication to placement.Source: §4.1 · p. 7
4.2 Placement Algorithm
- A two-level placement search uses simulator-guided greedy model selection and enumerates group partitions and parallel configurations to maximize SLO attainment.The placement problem is combinatorial, and its SLO objective lacks a simple analytical formula.Source: §4.2 · pp. 7–8
4.3 Runtime Scheduling
- At runtime, a centralized controller sends each request to the group with the shortest queue; groups use first-come-first-serve queues and reject requests predicted to miss SLO.Batching is disabled except in §6.5 to isolate model-parallelism benefits.Source: §4.3 · p. 9
- First-come-first-serve scheduling can cause convoy effects for models with different execution times, while model swapping is omitted and all models are assumed to remain on GPUs.The design also does not focus on fault tolerance; a single GPU failure can disable a group, and the centralized controller is a single point of failure.Source: §4.3 · p. 9
5 Implementation
AlpaServe is implemented as a real system and simulator, extending Alpa's auto-parallelization for inference. Its discrete-event simulator models cluster requests and executions.
5 Implementation
- The roughly 4,000-line Python implementation extends Alpa's auto-parallelization for inference and uses a centralized controller to dispatch requests to group runtimes.A separate simulator is also implemented.Source: §5 · p. 9
- The continuous-time, discrete-event simulator maintains a global clock and simulates requests and model executions across the cluster.Discrete-event simulation makes it orders of magnitude faster than real experiments.Source: §5 · p. 9
- A 24-hour trace takes less than one hour to simulate, and the simulator's fidelity is reported as very high based on predictable DNN execution verified in §6.1.The simulator's speed follows from modeling discrete events rather than running real experiments.Source: §5 · p. 9
6 Evaluation
AlpaServe is evaluated across model sizes, workload conditions, and serving strategies using production traces, synthetic workloads, simulation, and testbed runs. The results examine overall SLO performance, robustness, batching, and the proposed parallelization and placement algorithms.
6 Evaluation
- AlpaServe can save up to 2.3× devices, handle 10× higher rates, 6× more burstiness, or 2.5× more stringent SLOs while meeting latency SLOs for over 99% of requests.These are alternative optimizations, not simultaneous gains.Source: §6 · p. 9
6.1 Experiment Setup
- Across different SLO scales, simulator-reported SLO attainment differs from real testbed runs by less than 2% in all cases, supporting simulator accuracy; the simulator is used for most experiments.The testbed has 8 nodes and 64 GPUs.Source: §6.1 · p. 10
6.2 End-to-end Results with Real Workloads
- AlpaServe uses fewer devices to reach 99% SLO attainment and remains effective on bursty traffic, even against dynamically adjusted Clockwork++.For skewed, rapidly changing MAF2 traffic, replication methods devote many GPUs to hot-model replicas that may go idle between bursts; AlpaServe needs fewer replicas for peak traffic. Figure 12 presents SLO attainment across settings.Source: §6.2 · p. 11, Figure 12 · p. 11
6.3 Serving Very Large Models
- On very large models, AlpaServe’s traffic-balanced groups with (4, 8) inter-/intra-op parallelism outperform enumerated parallelism strategies and expose statistical multiplexing opportunities.Figure 13 varies rate, CV, and SLO scale.Source: §6.3 · p. 12, Figure 13 · p. 12
6.4 Robustness to Changing Traffic Patterns
- With changed traffic patterns, AlpaServe maintains performance and outperforms Clockwork++ using static placement, while selective replication’s performance drops significantly.Figure 14 compares systems when their assumed and actual traffic differ.Source: §6.4 · p. 12, Figure 14 · p. 13
6.5 Benefits of Dynamic Batching
- When SLOs are tight, batching can violate them; with looser SLOs it improves attainment, but stage imbalance and pipeline bubbles limit AlpaServe’s gains relative to Clockwork++.A batch size of 2 with sequence length 2048 already saturates the GPU, so larger maximum batches bring no improvement.Source: §6.5 · p. 13, Figure 15 · p. 13
6.6 Ablation Study
- For 8 pipeline stages, auto-parallelization cuts overhead by 32.9% for Transformer 1.3B and 46.7% for Transformer 2.6B.Group partitioning raises the rate supported above 99% SLO attainment by 1.5× versus greedy placement alone, and raises supported burstiness at 99% SLO attainment by 1.3×; round robin never reaches 99%. Figures 16 and 17 show the respective comparisons.Source: §6.6 · pp. 13–14, Figure 16 · p. 13, Figure 17 · p. 14
7 Related Work
Related work spans general serving systems, SLO-aware distributed serving, large-model inference optimizations, training parallelism, and resource multiplexing. AlpaServe’s distinction is its empirical study of parallelization and multiplexing for serving multiple large models.
Model serving systems
- General production serving systems such as TensorFlow Serving and Triton lack automatic placement or latency constraints, while other systems target single models or narrower model classes.AlpaServe targets a broader set of models and features.Source: §7 · p. 14
SLO-aware distributed serving
- Many SLO-aware systems ignore placement interactions among co-located models; weight loading and preemption can also overwhelm practical SLOs for large models.Clockwork focuses on predictability, while Shepherd uses preemption to correct scheduling decisions.Source: §7 · p. 14
- Nexus examines model placement but retains replication, leaving out the parallelization strategies explored by AlpaServe.Source: §7 · p. 14
Inference optimizations for large models
- Large-model inference optimizations—including quantization, distillation, offloading, operator parallelism, and kernel optimization—are complementary to AlpaServe, but their gains are partial.The paper notes that large-model serving challenges continue to escalate despite these efforts.Source: §7 · p. 14
Model parallelism for training
- AlpaServe is largely orthogonal to model-parallel training research, though it builds on some of that work’s parallelization techniques.Serving has constraints and opportunities that training workloads do not share.Source: §7 · p. 14
Resource allocation and multiplexing
- Prior multiplexing work spans swapping, preemption, interleaving, and space-sharing; AlpaServe contributes an empirical analysis of these ideas for serving multiple large models.Source: §7 · p. 14
8 Conclusion and Future Work
AlpaServe integrates model parallelism into multi-model serving and provides techniques to navigate its tradeoffs. The paper identifies adapted models, models with dependencies, and autoregressive models as future extensions.
8 Conclusion and Future Work
- AlpaServe is a system for serving multiple large deep-learning models.Source: §8 · p. 14
- AlpaServe integrates model parallelism into multi-model serving, demonstrates its usefulness in additional scenarios, and provides techniques to navigate its tradeoff space.The paper also quantifies the tradeoffs between model-parallelism overheads and their benefits.Source: §8 · p. 14
- Future extensions include serving parameter-efficient adapted models such as LoRA, models with dependencies, and autoregressive models.Source: §8 · p. 14