Source-linked AI summary
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, Yuxiong He
TL;DR
LLM serving systems must balance throughput and latency, particularly when long prompt processing can preempt generation and threaten service-level agreements. DeepSpeed-FastGen addresses this with Dynamic SplitFuse, which composes prompt and generation tokens in fixed-sized forward passes. Across evaluated models and hardware, it reports higher throughput and lower latency than vLLM, while its roadmap identifies further performance improvements.
Problem
Long prompt processing can preempt generation and threaten service-level agreements in existing LLM serving systems.
Method
DeepSpeed-FastGen uses Dynamic SplitFuse to compose prompt and generation tokens in fixed-sized forward passes, together with DeepSpeed-MII and DeepSpeed-Inference.
Results
Up to 2.3x higher effective throughput, 2x lower average latency, and 3.7x lower token-level tail latency are reported compared with vLLM.
Takeaways & Limitations
The evaluated system provides higher throughput and lower latency across the reported models and hardware configurations.
Takeaways & Limitations
The roadmap includes further performance improvements.
Abstract
from arXiv · showhide
The deployment and scaling of large language models (LLMs) have become critical as they permeate various applications, demanding high-throughput and low-latency serving systems. Existing frameworks struggle to balance these requirements, especially for workloads with long prompts. This paper introduces DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and generation composition strategy, to deliver up to 2.3x higher effective throughput, 2x lower latency on average, and up to 3.7x lower (token-level) tail latency, compared to state-of-the-art systems like vLLM. We leverage a synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced implementation supports a range of models and offers both non-persistent and persistent deployment options, catering to diverse user scenarios from interactive sessions to long-running applications. We present a detailed benchmarking methodology, analyze the performance through latency-throughput curves, and investigate scalability via load balancing. Our evaluations demonstrate substantial improvements in throughput and latency across various models and hardware configurations. We discuss our roadmap for future enhancements, including broader model support and new hardware backends. The DeepSpeed-FastGen code is readily available for community engagement and contribution.
1 Introduction
DeepSpeed-FastGen addresses inconsistent quality of service in LLM serving, especially for long prompts, by combining DeepSpeed-MII, DeepSpeed-Inference, and Dynamic SplitFuse. It reports higher effective throughput and lower latency than systems such as vLLM.
- Long prompts challenge consistent quality of service because prompt processing can preempt generation and risk breaking service-level agreements.
- DeepSpeed-FastGen combines DeepSpeed-MII and DeepSpeed-Inference with Dynamic SplitFuse for LLM serving.
- Up to 2.3x higher effective throughput, 2x lower average latency, and 3.7x lower token-level tail latency are reported versus systems such as vLLM.
2 Existing LLM Serving Techniques in Literature
Existing LLM serving techniques improve memory efficiency and batch utilization through blocked KV caches and continuous batching, but long prompts can still stall generation. Dynamic SplitFuse is proposed as a prompt-and-generation composition strategy to address this limitation.
- 2.1 Blocked KV Caching: Blocked KV caches use fixed-sized pages to reduce memory fragmentation and increase sequence concurrency.
- 2.2 Continuous Batching: Dynamic batching can improve GPU utilization but may require padding inputs or stalling while constructing larger batches.
- 2.2 Continuous Batching: Continuous batching lets requests join or leave at each forward pass, eliminating padding and improving overall throughput.
- 2.2 Continuous Batching: Current systems either preempt generation for prompt processing or add prompts to running batches, and both approaches can stall generation for long prompts.
- 2.2 Continuous Batching: Dynamic SplitFuse is proposed as a novel strategy for composing prompts and generation.
3 Dynamic SplitFuse: A Novel Prompt and Generation Composition Strategy
DeepSpeed-FastGen designs Dynamic SplitFuse around forward-pass token counts: it decomposes and composes prompt tokens with generation tokens to maintain efficient, consistent batch sizes. The strategy targets responsiveness, throughput, and latency consistency by scheduling work near the throughput-saturating region.
- 3 Dynamic SplitFuse: A Novel Prompt and Generation Composition Strategy: DeepSpeed-FastGen combines continuous batching and non-contiguous KV caches with dynamic prompt and generation decomposition to improve occupancy and throughput.
- 3.1.1 What factors impact the forward pass of a single LLM?: Forward-pass token count is the key scheduling signal because sequence batch composition has negligible performance impact by comparison.
- 3.1.2 How does a model’s throughput respond to changing the number of tokens in the forward pass?: Throughput increases toward a saturation region as forward-pass tokens grow, then remains near-constant; efficient execution keeps passes in that region.
- 3.1.3 How should a pool of tokens be scheduled across multiple forward passes?: For a fixed pool of tokens, equal partitioning across forward passes maximizes throughput under the observed concave token-throughput curve.
- 3.2 Dynamic SplitFuse: Dynamic SplitFuse composes fixed-sized batches containing both generation and prompt tokens, unlike strategies that separate or fully combine these phases.
- 3.2 Dynamic SplitFuse: Long prompts are split across multiple forward passes, with generation performed only on the final pass, while short prompts are decomposed or combined to fill the target token budget.
- 3.2 Dynamic SplitFuse: The resulting consistent forward sizes improve responsiveness, maintain high-throughput operation, and reduce latency variance.
- 3.2 Dynamic SplitFuse: Dynamic SplitFuse consumes prompt tokens while sustaining generation and system utilization, providing lower latency and higher-throughput streaming generation.
4 Performance Evaluation
DeepSpeed-FastGen evaluates LLM serving with throughput, latency, and chat-oriented SLA metrics across multiple models and hardware configurations. The evaluation reports state-of-the-art performance using blocked KV caches and Dynamic SplitFuse continuous batching.
- DeepSpeed-FastGen achieves up to 2.3x higher effective throughput, 2x lower average latency, and up to 3.7x lower token-level tail latency than vLLM.The evaluation covers multiple models and hardware configurations.
- Throughput is measured in requests per second, while latency measures each request’s responsiveness using 512 requests and 1–32 concurrent clients.
- The chat scenario measures prompt submission, first-token return, and subsequent token streaming as distinct user-facing stages.
- The proposed SLA framework considers both slow first-token delivery and pauses during streamed generation.
- Effective throughput counts requests satisfying prompt and generation SLAs, using normalized prompt latency and EMA generation-rate thresholds.The generation thresholds are 2, 4, or 6 tokens/sec.
- The evaluation compares vLLM and DeepSpeed-FastGen on Llama-2 7B, 13B, and 70B using NVIDIA A100, H100, and A6000 GPUs.
4.2 Throughput-Latency Analysis
Throughput-latency analysis shows DeepSpeed-FastGen outperforming vLLM on Llama 2 models, offering either higher throughput at equal latency or lower latency at equal throughput.
- 2x higher throughput is achieved at identical 9-second latency on Llama 2 70B using 4 A100x80GB GPUs.DeepSpeed-FastGen reaches 1.36 rps versus vLLM’s 0.67 rps.
- 50% latency reduction is achieved at the same 1.2 rps throughput, with latency decreasing from 14 seconds to 7 seconds.
- The same throughput-latency trends also hold for Llama 2 13B.
4.3 Effective Throughput Analysis
Effective throughput analysis combines first-token latency with generation rate and evaluates performance as client load increases. DeepSpeed-FastGen reaches higher effective throughput than vLLM, while excessive load eventually causes SLA failures.
- 2.3x higher effective throughput is achieved by DeepSpeed-FastGen than vLLM when both first-token latency and generation rate are considered.
- The effective-throughput curves use Llama 2 70B on four A100-80GB GPUs with prompt and generation lengths averaging 2600 and 60, respectively.
- Effective throughput initially rises with client count but saturates or decreases as latency increases near system capacity and more requests miss the SLA.
4.4 Significant Tail Latency Reduction for Token Generation
DeepSpeed-FastGen reduces high-percentile generation latency relative to vLLM while maintaining similar median latency. The difference is associated with vLLM preempting generation to process new prompts.
- 3.7 times lower P95 generation latency is achieved by DeepSpeed-FastGen, while both systems show similar P50 latency.
- vLLM’s P90 and P95 latencies are significantly higher because it preempts ongoing generation to process new prompts.
- DeepSpeed-FastGen processes prompts and generation for previous requests concurrently, producing more consistent generation latency.
4.5 Scalability using Load Balancing
DeepSpeed-FastGen scales across replicas through replica-level load balancing, achieving nearly perfect scalability up to 16 replicas.
- Scalability using Load Balancing: 23.7 queries/sec with 16 replicas represents a linear 16x increase over the 1.46 queries/sec single-replica throughput.The deployment used 4 A100 GPUs per replica and 8 nodes for 16 replicas.
4.6 Other Hardware Platforms
Additional evaluations on H100 and A6000 hardware reproduce the performance trends observed on A100, alongside measurements under varied benchmark workloads.
- Other Hardware Platforms: The same performance trends observed on A100 were also found on A6000 and H100 hardware.The paper reports additional benchmarking results for both platforms.
- Other Hardware Platforms: Per-token latency on Llama 2 70B was measured across 4 A100-80GB GPUs with 16 clients and 2,600-token average prompts.Generation lengths averaged 128 tokens, with 30% variance applied to prompt and generation lengths.
- Other Hardware Platforms: Scalability measurements used prompt and generation lengths averaging 2,600 and 60 tokens, respectively, with 30% variance.The corresponding figure evaluates scalability using the load-balancing feature.
- Other Hardware Platforms: Throughput-latency curves and effective throughput were evaluated for Llama 2 70B on 8 H100 GPUs and Llama 2 7B on an A6000.Both experiments used 2,600-token average prompts and 60-token average generations with 30% variance.
5 DeepSpeed-FastGen: Implementation and Usage
DeepSpeed-FastGen combines DeepSpeed-MII and DeepSpeed-Inference into a serving system with multiple deployment modes, supported model families, and portable optimized kernels.
- Implementation: DeepSpeed-FastGen combines DeepSpeed-MII and DeepSpeed-Inference to provide APIs, scheduling infrastructure, optimized kernels, and model-construction tools.Dynamic SplitFuse is used to schedule batches within the system.
- Model Support: The alpha release supports HuggingFace model families through backend APIs supplying model weights and tokenizers.Additional model architectures were planned for future releases.
- Deployment Options: Users can choose between a non-persistent pipeline for temporary interactive sessions and a persistent serving deployment for long-running applications.The non-persistent option lasts only for the duration of the Python script, while the persistent option uses a lightweight GRPC server.
- Deployment Options: Persistent deployments support concurrent clients through DeepSpeed-MII’s built-in load balancer and can be terminated when no longer needed.The deployment and client interfaces are designed to require only a few lines of code.
- Implementation: DeepSpeed-Kernels distributes pre-compiled custom kernels to reduce compile times and support NVIDIA GPUs with compute capabilities 8.0+, CUDA 11.6+, and Ubuntu 20+.The library is intended to be portable across supported environments and is installed as a dependency.
6 Release: Try Out DeepSpeed-FastGen
The release invites users to try DeepSpeed-FastGen through its public ecosystem and outlines a roadmap for continued development and community contribution.
- Release: Users are directed to the DeepSpeed-MII GitHub landing page to get started with the alpha release.DeepSpeed-FastGen is presented as part of the broader DeepSpeed ecosystem.
- Community: The project provides websites, tutorials, documentation, and social channels for ongoing information about DeepSpeed.These resources include English, Japanese, and Chinese communication channels.
- Roadmap: The roadmap includes performance improvements, new hardware backends through partner collaboration, and released performance benchmarks.The project planned to engage with the community through GitHub issues and pull requests.
- Community: The release encourages users to star the DeepSpeed and DeepSpeedMII GitHub repositories.