PaperlayerView exact revision on arXiv

Source-linked AI summary

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang

arXiv:2401.09670v3cs.DC

TL;DR

Existing LLM serving colocates prefill and decoding, causing interference and coupling resource and parallelism choices despite distinct TTFT and TPOT requirements. DistServe disaggregates the phases across GPUs and independently co-optimizes their allocation and parallelism. Across varied models, applications, and latency constraints, it serves up to 7.4× more requests or supports 12.6× tighter SLOs than state-of-the-art solutions.

  • Problem

    Colocated prefill and decoding interfere and force shared resource and parallelism settings, making it difficult to meet both TTFT and TPOT SLOs without over-provisioning.

  • Method

    DistServe assigns prefill and decoding to separate GPU instances, independently co-optimizes their GPU allocation and parallelism, and places them to manage intermediate-state communication.

  • Results

    7.4× more requests or 12.6× tighter SLOs are achieved than with state-of-the-art solutions under varied models, applications, and latency constraints.

  • Takeaways & Limitations

    Disaggregating prefill and decoding improves per-GPU goodput and can reduce cost per LLM query while maintaining SLO attainment.

  • Takeaways & Limitations

    In resource-constrained deployments with only a few or a single GPU, DistServe has limited design space and may struggle or fail to improve serving performance.

Abstract

from arXiv · show

DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests. We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase. In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both. DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application's TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy tailored for each phase. DistServe also places the two phases according to the serving cluster's bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU. Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 7.4x more requests or 12.6x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests.

1 Introduction

LLM serving must satisfy separate TTFT and TPOT requirements while controlling GPU cost, but colocated prefill and decoding interfere and couple their resource plans. DistServe separates the phases, independently optimizes them, and improves goodput under latency constraints.

  • Motivation: LLM queries use prefill to generate the first token and decoding to generate subsequent tokens, with TTFT and TPOT measuring their respective latency requirements.Different applications prioritize these metrics differently: chatbots favor low TTFT, while document summarization emphasizes low TPOT.
  • Motivation: Existing systems colocate and batch both phases to maximize throughput, but must over-provision resources to satisfy stringent latency requirements.This approach optimizes tokens generated per second across users and requests rather than independently meeting both phase-specific SLOs.
  • Problems: Prefill-decoding interference delays decoding with long prefill steps and increases TTFT when decoding steps share the batch.Scheduling the phases separately still leaves them competing for GPU resources and causing queuing delays.
  • Problems: Colocation couples resource allocation and parallelism, preventing phase-specific strategies and often forcing configurations tailored to the more demanding TTFT or TPOT requirement.The resulting coupling can require resource over-provisioning to meet both SLOs.
  • DistServe: DistServe assigns prefill and decoding to separate GPUs, eliminating their interference and enabling independent scaling with tailored resource allocation and parallelism.It also co-optimizes these choices for per-GPU goodput and evaluates the system across LLMs and realistic application workloads.

2 Background and Motivation

LLM serving combines compute-intensive prefill with iterative decoding, whose different workloads and latency goals make shared batching and parallelism difficult to optimize. DistServe addresses these problems by separating the phases and assigning each independently optimized resources.

  • LLM serving background: Prefill processes all prompt tokens concurrently, whereas decoding processes one generated token per step and reuses stored KV caches.Prefill tends to be compute-bound for substantial prompts, while both phases share model weights and memory.
  • Existing systems: Continuous batching improves GPU utilization and aggregate throughput by combining new-request prefill with ongoing-request decoding, but it trades off TTFT against TPOT.Chunked prefill with piggybacking alleviates but does not eliminate this interference.
  • Prefill-decoding interference: Adding one prefill job to a decoding batch significantly slows both processes, with decoding delayed by the longer prefill and prefill slowed by concurrent decoding.The slowdown intensifies for longer prefills and when the GPU is already at capacity.
  • Scheduling: Sequentially scheduling unbatched phases does not remove interference because waiting jobs incur queuing delays, while decode-only batches can underutilize the GPU.Prioritizing either phase adversely affects the other phase’s latency.
  • Resource and parallelism coupling: Prefill and decoding require different resource and parallelism choices, but colocation forces them to share settings and can cause resource over-provisioning.Prefill benefits from more intra-op parallelism for TTFT, whereas decoding’s optimal configuration depends on batch size.
  • Opportunity: Disaggregation gives prefill and decoding separate instances, allowing independent optimization for TTFT and TPOT and flexible instance ratios to improve GPU utilization.A prefill instance transfers intermediate results, mainly KV caches, to a decoding instance; multiple prefill instances may feed one decoding instance.

3 Tradeoff Analysis

Disaggregation separates prefill and decoding so each phase can be analyzed, scheduled, and scaled according to its latency requirements. The analysis shows that batching and parallelism choices depend on workload, arrival rate, SLO, and hardware characteristics.

  • Overview: Disaggregation lets prefill and decoding be scaled and scheduled independently according to their latency requirements.This expands the design space for per-GPU goodput optimization.
  • Prefill batching: A 13B LLM’s prefill phase can become compute-bound at a single sequence of 512 tokens, so larger batches may increase processing time without improving efficiency.Requests should be batched only when input length remains below the profiled threshold Lm.
  • Prefill parallelism: At lower arrival rates, intra-op parallelism is more efficient, whereas inter-op parallelism becomes advantageous as queuing delay grows with the arrival rate.This comparison follows the execution-time versus queuing-delay components of average TTFT.
  • Prefill parallelism: Stricter TTFT SLOs favor intra-op parallelism, but lower speedup coefficient K reduces its efficacy.K depends on input length, model architecture, communication bandwidth, and placement.
  • Decoding batching: Decoding benefits from larger batches because a single decoding job is bandwidth-bound, and disaggregation allows multiple prefill instances to feed one decoding instance without sacrificing TPOT.Model parallelism and KV-cache memory management can further increase decoding batch size.
  • Practical considerations: Non-uniform prompt lengths create pipeline bubbles for inter-op prefill parallelism, so workload-aware search and scheduling are needed to reduce deviations from uniform-length queueing analysis.The paper identifies automatic navigation of this configuration space as challenging.

4 Method

DistServe searches for a placement that maximizes per-GPU goodput while satisfying workload, latency, SLO, capacity, and network constraints. Its method combines phase-specific configuration search, replication, node-aware placement, and runtime scheduling.

  • 4 Method: DistServe determines phase parallelism, instance counts, and physical placement to maximize per-GPU goodput under model, workload, latency, and SLO requirements.The output is called a placement.
  • 4.1 High Node-Affinity Placement: For high-bandwidth clusters, the placement algorithm separately optimizes prefill and decoding configurations, then replicates them to match traffic rate.With Infiniband, KV-cache transmission overhead across nodes is negligible.
  • 4.1 High Node-Affinity Placement: The simulator estimates goodput and SLO attainment across feasible parallel configurations using workload-aware execution-time models based on FLOPs and memory accesses.Its estimates align with real profiling results.
  • 4.1 High Node-Affinity Placement: 36: Algorithm 1 has complexity O(NM^2), and solving time is under 1.3 minutes in the largest evaluated setting.N is the node limit per instance and M is the typical number of GPUs per node.
  • 4.2 Low Node-Affinity Placement: For limited cross-node bandwidth, DistServe constrains placement by co-optimizing model parallelism and keeping corresponding prefill and decoding segments on the same node.This forces KV-cache transfer between corresponding layers to use NVLINK.
  • 4.3 Online scheduling: The runtime uses centralized FCFS dispatch, sending requests to the shortest prefill queue and then to the least-loaded decoding instance.Scheduling enhancements balance pipeline execution and use pull-based KV-cache transmission to handle workload burstiness.
  • 4.3 Online scheduling: DistServe leaves preemption and fault tolerance for future work, and dependencies between phases can allow a decoding fault to propagate across mapped prefill instances.The current FCFS policy can also produce a convoy effect in prefill.

5 Implementation

DistServe provides a distributed serving system with placement, frontend, orchestration, and parallel execution components. Its implementation supports workload evaluation and manages prefill/decoding instances, KV-cache transmission, and results delivery.

  • System implementation: DistServe comprises placement, frontend, orchestration, and parallel execution components for end-to-end LLM serving.The first three components use 6.5K lines of Python, while the parallel execution engine uses 8.1K lines of code.
  • System implementation: The placement module selects deployment decisions for a model and cluster using the system’s algorithm and simulator.
  • System implementation: The frontend exposes an OpenAI API-compatible interface with configurable sampling parameters such as maximum output length and temperature.
  • System implementation: The orchestration layer manages prefill and decoding instances, KV-cache transmission, request dispatching, and result delivery.It uses NCCL for cross-node GPU communication and asynchronous CudaMemcpy for intra-node communication.

6 Evaluation

DistServe is evaluated across LLM sizes, applications, latency requirements, and real cluster settings against established serving baselines. It consistently improves supported request rates or SLO stringency while maintaining latency requirements for over 90% of requests.

  • Overall evaluation: 7.4× higher rates or 12.6× more stringent SLOs are achieved while meeting latency requirements for over 90% of requests.The evaluation spans models from 13B to 175B and chatbot, code-completion, and summarization applications.
  • Experimental setup: DistServe is tested on a four-node, 32-GPU cluster of NVIDIA A100-80GB GPUs with 25Gbps cross-node bandwidth.Most experiments use a low node-affinity placement algorithm because of limited cross-node bandwidth.
  • Experimental setup: The evaluation uses OPT models with FP16 precision and three application workloads sampled from ShareGPT, HumanEval, and LongBench.Request arrivals are generated with Poisson distributions, and SLOs are set empirically from application service targets.
  • Chatbot: On ShareGPT, DistServe sustains 2.0×–4.6× higher request rates than vLLM and achieves 1.8×–3.2× more stringent SLOs.Compared with DeepSpeed-MII, it achieves 1.7×–1.8× more stringent SLOs; chunked-prefill mitigates but does not eliminate interference.
  • Code completion: 5.7× higher request rates and 1.4× more stringent SLOs are achieved than vLLM for code completion on OPT-66B.Relative to DeepSpeed-MII, DistServe achieves 1.6× higher request rates and 1.4× more stringent SLOs.
  • Summarization: 4.3× higher request rates and 12.6× more stringent SLOs are achieved than vLLM for summarization on OPT-66B.Relative to DeepSpeed-MII, DistServe achieves 1.8× higher request rates and 2.6× more stringent SLOs.
  • Latency breakdown: KV-cache transmission accounts for less than 0.1% of total latency, with over 95% of requests experiencing transmission delays below 30ms.Bandwidth-aware placement keeps corresponding prefill and decoding stages on one machine to use intra-node NVLINK bandwidth.
  • Simulator validation: Simulator SLO-attainment error is below 2% across tested rates when compared with real testbed runs.

7 Discussion

DistServe is optimized for latency-sensitive, goodput-oriented LLM serving, but its benefits depend on the workload and available resources. It may be less suitable when throughput dominates or when deployments cannot support flexible disaggregation and parallelism.

  • DistServe targets goodput optimization under latency requirements rather than a one-size-fits-all LLM serving objective.The paper frames different service scenarios as having different optimization targets and resource limits.
  • Throughput-optimized scenarios: In throughput-optimized offline applications, DistServe’s effectiveness may be compromised because latency requirements are lower.Chunked-prefill with piggyback may maintain higher GPU utilization by filling batches to the compute-bound threshold.
  • Resource-constrained scenarios: In resource-constrained deployments with only a few or a single GPU, DistServe has limited ability to adjust resource allocation and parallelism strategies.The paper notes that simpler non-disaggregated architectural choices may be preferable in such settings.
  • Long-context scenarios: In long-context serving, KV-cache transmission grows linearly with prompt length while prefill computation grows quadratically, reducing transmission’s relative duration.Longer contexts also increase the computational disparity between prefill and decoding, so disaggregation remains promising.

8 Related Work

Related work spans general LLM inference serving, goodput optimization, resource disaggregation, and model parallelism. DistServe combines these directions for goodput optimization in autoregressive LLM inference.

  • Inference serving: LLM serving systems improve throughput, KV-cache management, or prefill scheduling through techniques such as continuous batching, paged attention, and chunked prefill.The surveyed systems include production serving platforms and Transformer-specific serving systems.
  • Goodput-optimized systems: Goodput-oriented systems address scheduling, heterogeneous resource matching, latency-aware preemption, or model-parallel execution, but prior approaches have narrower targets.DistServe is presented as the first work to optimize goodput for autoregressive LLM inference.
  • Resource disaggregation: Resource disaggregation separates hardware resources into independently managed pools, whereas DistServe applies the concept to prefill and decoding computation.The provided related-work passage states that DistServe shares this disaggregation concept.
  • Model parallelism for training: DistServe is orthogonal to training model-parallelism research and can integrate new parallelism optimizations into its placement-search algorithm.Inference-serving workloads have characteristics that differ from training settings.

9 Conclusion

DistServe disaggregates prefill and decoding to maximize per-GPU goodput while satisfying SLO attainment goals. The paper reports lower cost per query and positions disaggregation as a promising strategy for latency-sensitive LLM services.

  • DistServe disaggregates prefill and decoding computation into a new LLM serving architecture.
  • Up to 7.4× lower cost per LLM query is achieved while guaranteeing satisfaction of SLOs through per-GPU goodput optimization.Per-GPU goodput is defined as the maximum request rate served under the SLO attainment goal for each provisioned GPU.
  • DistServe’s findings support prefill-decoding disaggregation as a promising strategy for improving performance and service quality guarantees as latency becomes more important.

A.2 Prefill Phase Latency Modeling

The prefill latency model separates compute-heavy matrix multiplications, memory-bound FlashAttention, and calibrated overheads. It uses workload characteristics and profiling to estimate total prefill latency.

  • The model treats four prefill matrix multiplications as compute-bound and estimates their latency from total FLOPs.Their arithmetic intensity is O(t), exceeding the A100 compute-bound threshold when t reaches several hundred.
  • FlashAttention prefill attention is modeled as memory-bound because its arithmetic intensity is 10.677 for b = 16 or 21.333 for b = 32 on an A100 GPU.
  • The overall prefill phase latency is obtained by combining the modeled operation costs into a phase-level latency model.
  • C3 captures runtime and system-noise overheads, while profiling and interpolation determine C1, C2, and C3.

A.3 Decoding Phase Latency Modeling

DistServe models decoding latency by separating memory-bound GEMM and attention costs, then sums them into a phase-level estimate. Profiling and interpolation provide the model coefficients.

  • Decoding latency modeling begins by focusing on the GEMMs executed during the decoding phase.
  • Because batch size is constrained by GPU memory and latency requirements, the decoding GEMMs are modeled as memory-bound.Their total memory traffic is 8Bh + 4h2 + 2hm + 2Bm.
  • Decoding attention is modeled as memory-bound, with 3sl memory reads and writes and 2sl FLOPs per attention head and request.
  • The latency of the decoding phase is obtained by summing the modeled component latencies.
  • Profiling and interpolation determine C4 and C5, while fixed GEMM costs allow overhead to be absorbed into C4.

B DistServe Placements in End-to-end Experiments

DistServe selects tensor-parallelism and pipeline-parallelism configurations separately for prefill and decoding instances in its end-to-end experiments. These choices are documented in Table 3.

  • The reported configurations include tensor parallelism and pipeline parallelism choices for both prefill and decoding instances.
  • DistServe’s placement results therefore specify phase-specific TP and PP configurations rather than one shared configuration.
  • Table 3 reports the parallelism strategies selected by DistServe for the end-to-end experiments.

C End-to-end Results under 99% SLO attainment

Under a 99% SLO attainment goal, DistServe maintains substantial rate and SLO advantages over vLLM and DeepSpeed-MII across the evaluated end-to-end workloads. The comparisons cover chatbot, code completion, and summarization applications.

  • 3×–8× higher rate and 1.24×–6.67× more stringent SLO are achieved by DistServe versus vLLM under the 99% attainment goal.
  • 1.32×–8× higher rate and 1.20×–1.58× more stringent SLO are achieved by DistServe versus DeepSpeed-MII under the same goal.
Loading 2401.09670v3…