PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He

arXiv:2207.00032v1cs.LGcs.DCcs.PF

TL;DR

Transformer inference must handle diverse architectures, scales, latency goals, and hardware while overcoming low-batch inefficiency and multi-device communication overhead. DeepSpeed Inference combines optimized single- and multi-GPU methods with heterogeneous CPU/NVMe memory, achieving up to 7.3× lower latency and trillion-parameter inference at unprecedented scale.

  • Problem

    Efficient transformer inference is difficult because latency-critical workloads require high memory-bandwidth and compute utilization at small batch sizes while large models require communication-efficient multi-device parallelism.

  • Method

    DeepSpeed Inference combines optimized single-GPU kernels, dense multi-GPU parallelism, sparse MoE scaling, and CPU/NVMe memory offloading for models exceeding aggregate GPU memory.

  • Results

    Up to 7.3× lower latency is achieved for large MoE inference, while DeepSpeed-MoE scales inference to trillion-parameter models across hundreds of GPUs.

  • Takeaways & Limitations

    The system broadens access to large-model inference by supporting efficient execution with as few as a single GPU or at unprecedented multi-GPU scale.

Abstract

from arXiv · show

The past several years have witnessed the success of transformer-based models, and their scale and application scenarios continue to grow aggressively. The current landscape of transformer models is increasingly diverse: the model size varies drastically with the largest being of hundred-billion parameters; the model characteristics differ due to the sparsity introduced by the Mixture-of-Experts; the target application scenarios can be latency-critical or throughput-oriented; the deployment hardware could be single- or multi-GPU systems with different types of memory and storage, etc. With such increasing diversity and the fast-evolving pace of transformer models, designing a highly performant and efficient inference system is extremely challenging. In this paper, we present DeepSpeed Inference, a comprehensive system solution for transformer model inference to address the above-mentioned challenges. DeepSpeed Inference consists of (1) a multi-GPU inference solution to minimize latency while maximizing the throughput of both dense and sparse transformer models when they fit in aggregate GPU memory, and (2) a heterogeneous inference solution that leverages CPU and NVMe memory in addition to the GPU memory and compute to enable high inference throughput with large models which do not fit in aggregate GPU memory. DeepSpeed Inference reduces latency by up to 7.3X over the state-of-the-art for latency-oriented scenarios and increases throughput by over 1.5x for throughput-oriented scenarios. Moreover, it enables trillion parameter scale inference under real-time latency constraints by leveraging hundreds of GPUs, an unprecedented scale for inference. It can inference 25x larger models than with GPU-only solutions, while delivering a high throughput of 84 TFLOPS (over $50\%$ of A6000 peak).

I. INTRODUCTION · 1) DeepSpeed Transformer:

Transformer inference must simultaneously address stringent latency and throughput demands, architectural diversity, and models that exceed single-node GPU memory. DeepSpeed Inference addresses these challenges through a GPU-only DeepSpeed Transformer system that supports dense and sparse models from one GPU to hundreds of GPUs.

  • I. INTRODUCTION: DeepSpeed Inference addresses transformer inference challenges spanning latency, throughput, architectural diversity, and limited GPU resources.The paper identifies models ranging from over trillion parameters, dense and sparse Mixture-of-Experts architectures, varied workload targets, and deployment constraints.
  • I. INTRODUCTION: Small-batch latency is bounded by loading model parameters, requiring near-peak memory bandwidth and low kernel-launch and data-movement overhead.Training-oriented GeMM and kernel implementations optimized for large batches are suboptimal for latency-critical inference.
  • I. INTRODUCTION: Large models require aggregate memory bandwidth across devices, with parallelism that minimizes communication while adapting to transformer architectures and hardware.Tensor and pipeline parallelism apply to dense transformers, expert parallelism to sparse transformers, and mixed architectures require combined strategies.
  • I. INTRODUCTION: Throughput-oriented inference must overlap computation with model-weight reads while maintaining high memory and compute utilization at small batch sizes.Inference also differs from training because generative transformers have token dependencies and must retain previously generated states.
  • I. INTRODUCTION: Models with tens or hundreds of billions of parameters can exceed single-GPU or single-node aggregate memory, making inference feasibility a central constraint.MT-NLG 530B requires about 1TB of GPU memory and over three DGX-2 nodes with more than two dozen A100 40GB GPUs.
  • 1) DeepSpeed Transformer:: DeepSpeed Transformer is a GPU-only solution designed to minimize latency and maximize throughput for dense and sparse models, scaling from one GPU to hundreds.It supports inference of multitrillion-parameter models through state-of-the-art latency and throughput across model sizes.
  • 1) DeepSpeed Transformer:: The DeepSpeed Transformer architecture combines single-GPU kernels, many-GPU dense layers, and massive-GPU sparse layers into compatible system components.The layers target low-batch memory bandwidth and high-batch throughput, dense scaling through tensor-slicing and inference-optimized pipeline parallelism, and sparse MoE scaling to hundreds of GPUs.

2) ZeRO-Inference: · II. BACKGROUND AND RELATED WORK

DeepSpeed Inference combines optimized kernels, multi-GPU parallelism, and heterogeneous GPU–CPU–NVMe execution to support efficient inference across dense and sparse transformers from billions to trillions of parameters. The paper positions these techniques as adaptations of prior memory- and performance-scaling methods for inference, including resource-constrained deployments.

  • 2) ZeRO-Inference:: ZeRO-Inference enables inference of models with hundreds of billions of parameters on one or multiple GPUs when CPU or NVMe memory can store the parameters.It targets applications that are less latency sensitive but resource constrained and uses heterogeneous GPU+CPU+NVMe memory.
  • 2) ZeRO-Inference:: DeepSpeed Inference combines single-GPU kernels, dense and sparse multi-GPU systems, and ZeRO-Inference to address diverse transformer inference scales and resource constraints.Its contributions target latency, throughput, trillion-parameter sparse models, and inference using CPU, NVMe, and GPU resources.
  • 2) ZeRO-Inference:: DeepSpeed Inference supports dense or sparse models, small or large batches, billions to trillions of parameters, and deployments from single GPUs to hundreds of GPUs.The system is presented as a versatile solution spanning latency- and throughput-oriented inference scenarios.
  • II. BACKGROUND AND RELATED WORK: Transformer models have scaled from hundreds of millions to hundreds of billions of parameters, including GPT-3 175B, Gopher 280B, and MT-NLG 530B [1],, [9].This growth motivates inference systems that address increasingly large dense models.
  • II. BACKGROUND AND RELATED WORK: Mixture-of-Experts introduces sparsity by replacing transformer feed-forward blocks with conditional computation, motivating distinct sparse-model inference techniques [10],.The passage contrasts sparse MoE architectures with typical transformer blocks containing self-attention followed by position-wise feed-forward sublayers.
  • II. BACKGROUND AND RELATED WORK: Tensor parallelism reduces local compute granularity and adds per-layer allreduce communication, while pipeline parallelism scales across nodes more efficiently but can impose splitting and micro-batching restrictions [16]–.These trade-offs help explain why training-oriented parallelism requires inference-specific optimization.
  • II. BACKGROUND AND RELATED WORK: Prior approaches also partition model states with ZeRO, combine data, tensor, and pipeline parallelism for trillion-parameter models, and distribute experts with expert parallelism –.DeepSpeed Inference leverages these strategies while adding optimizations for inference performance.
  • II. BACKGROUND AND RELATED WORK: Existing transformer kernels, DNN runtimes, compilers, and compression methods accelerate operators or single-GPU models, whereas this work complements them for larger-scale inference –.The cited platforms include TVM, ONNXRuntime, and TensorRT; compression can be combined with the proposed system.

III. INFERENCE-OPTIMIZED TRANSFORMER KERNELS · A. Inference Challenges on Different Batch Sizes

DeepSpeed Inference-optimized transformer kernels target high-performance inference across small and large batch sizes. Small batches are constrained by memory-bandwidth and invocation overheads, while large batches face compute-utilization losses from launches and memory transfers; Deep-Fusion and a custom GeMM kernel address these bottlenecks.

  • III. INFERENCE-OPTIMIZED TRANSFORMER KERNELS: The section develops transformer-kernel designs and optimizations for high-performance inference across both small and large batch sizes.
  • A. Inference Challenges on Different Batch Sizes: Small-batch inference is limited by memory-bandwidth utilization when reading model weights.
  • A. Inference Challenges on Different Batch Sizes: Limited work in small-batch transformer-layer kernels makes kernel-invocation overhead a major performance challenge.
  • A. Inference Challenges on Different Batch Sizes: Small-batch kernels also incur global-memory data movement because each invocation writes data that subsequent GPU computation reads.
  • A. Inference Challenges on Different Batch Sizes: Large-batch inference is limited by compute utilization, with kernel launches and GPU-core-to-global-memory transfers constraining overall utilization beyond GeMM operations.
  • A. Inference Challenges on Different Batch Sizes: Deep-Fusion reduces kernel-invocation and data-movement overheads by fusing multiple kernels beyond element-wise operations.
  • A. Inference Challenges on Different Batch Sizes: A custom GeMM kernel improves memory-bandwidth utilization for relatively small batches and supports fusion through Deep-Fusion.

B. Deep-Fusion … D. Putting It Together

Deep-Fusion enables transformer operators with cross-thread-block dependencies to be fused by tiling dependency-free dimensions and retaining dependent dimensions within the same thread-block. The system combines this approach with a bandwidth-oriented small-batch GeMM, transformer-layer fusion, multiple data types, and CUDA-Graph support.

  • B. Deep-Fusion: Deep-Fusion tiles dependency-free computation dimensions across thread-blocks while processing dependency-containing dimensions within one thread-block, avoiding global synchronization.This extends fusion beyond element-wise operators to transformer computations involving layout transformations, reductions, and GeMMs.
  • B. Deep-Fusion: At tile granularity, Deep-Fusion fuses reductions, transpositions, and GeMMs when each second-operator tile depends on exactly one first-operator output tile.This permits all layer-norm micro-operations, including multiple reductions, to execute in one fused kernel when tiled without cross-tile dependencies.
  • C. SBI-GeMM: Custom GeMM for Small Batch Size: The custom SBI-GeMM is designed for Deep-Fusion and maximum memory-bandwidth utilization through tiling, cooperative-group reduction, and data-layout transformation.These three components target the constraints of small-batch transformer inference.
  • 1) Tiling Strategies:: The tiling strategy uses output-dimension tiling for single-kernel reduction within tiles, adding input-dimension tiling and a second kernel when small outputs lack sufficient parallelism.The second configuration enables reduction across tiles for small models.
  • 2) Cooperative-Group Reduction:: A shared-memory data-layout transpose replaces binary-tree reduction and its repeated warp-level synchronizations for combining partial results within a thread-block.The passage identifies the usual shared-memory reduction as a performance bottleneck and motivates the cooperative-group alternative.
  • 3) Leveraging Full Cache-line:: Transposing the weight matrix during initialization makes M rows per column contiguous, improving cache-line utilization without reading multiple output-dimension elements per thread.The design addresses underutilized 128-byte L1 cache lines for FP16 or INT8 accesses while preserving parallel tiles.
  • D. Putting It Together: For small-batch inference, transformer-layer fusion covers QKV GeMM with input layer-norm, transposition with attention, postattention layer-norm with intermediate GeMM, and bias with residual addition.For large batches, the same fusion strategy is used while GeMM operations rely on CUBLAS and remain unfused.
  • D. Putting It Together: CUDA-Graph support records the first forward-computation kernel trace to reduce CPU-side launch overhead after kernel execution becomes the latency bottleneck.This targets small to moderate models with small batch sizes.

IV. INFERENCE-ADAPTED DENSE TRANSFORMER MODELS ON MANY-GPU SYSTEMS · A. Aggregate Memory Bandwidth via Tensor Parallelism · B. Aggregate Memory via Pipeline Parallelism — Challenges

This section combines tensor and pipeline parallelism to reduce latency and expand memory capacity for dense-transformer inference across many GPUs. It adapts parallel execution to inference-specific communication, autoregressive dependencies, pipeline bubbles, and activation-cache memory demands.

  • IV. INFERENCE-ADAPTED DENSE TRANSFORMER MODELS ON MANY-GPU SYSTEMS: Model parallelism targets lower latency through aggregate GPU memory bandwidth and larger model capacity through aggregate GPU memory across devices and nodes.These techniques build on the single-transformer kernels, but inference introduces challenges distinct from training.
  • B. Aggregate Memory via Pipeline Parallelism — Challenges: Hybrid scheduling assigns different micro-batch counts to pipeline stages, reducing both prompt-processing and token-generation latency.Fig. 3 contrasts schedules where small micro-batch counts favor token-generation latency while larger counts favor prompt processing.
  • A. Aggregate Memory Bandwidth via Tensor Parallelism: Tensor parallelism partitions transformer operators across GPUs, adding communication operations while automatically scaling dense models across multiple devices.It uses single-GPU kernels for per-GPU memory-bandwidth utilization and NCCL all-reduce collectives for communication.
  • A. Aggregate Memory Bandwidth via Tensor Parallelism: Tensor parallelism achieves strong aggregate memory-bandwidth utilization within a node but scales inefficiently beyond one node because of communication overhead.Its cross-GPU communication relies on NCCL all-reduce collectives.
  • B. Aggregate Memory via Pipeline Parallelism — Challenges: Pipeline parallelism fits models exceeding one node’s memory capacity and scales across nodes more efficiently than tensor parallelism because it has lower communication overhead.Unlike tensor parallelism, it does not increase aggregate memory bandwidth because each micro-batch traverses the full model depth sequentially.
  • B. Aggregate Memory via Pipeline Parallelism — Challenges: Autoregressive generation creates token dependencies that make batch-oriented training pipelines unsuitable for inference and induce frequent pipeline bubbles.The dependency structure and resulting bubbles are illustrated in Fig. 2 for generating tokens across four pipeline stages.
  • B. Aggregate Memory via Pipeline Parallelism — Challenges: Key-value activation caches avoid recomputation during autoregressive generation, but their memory grows with concurrently generated sequences and can limit inference performance.This makes memory capacity a performance constraint for large transformer models.

C. Inference Optimized Pipeline Parallelism … 3) Communication Optimization:

DeepSpeed Inference optimizes pipeline-parallel inference through scheduling, activation-memory reduction, and communication-aware offloading. Its design adapts microbatching to prompt processing and token generation while coordinating CPU transfers across GPUs to reduce contention.

  • C. Inference Optimized Pipeline Parallelism: The pipeline-parallel design combines scheduling, memory footprint reduction, and communication optimization to address inference-specific challenges.
  • 1) Hiding data dependencies and hybrid scheduling:: Using at least P micro-batches keeps all pipeline stages utilized, while additional micro-batches increase latency and memory costs.The sequence is divided into micro-batches that progress through pipeline stages as new tokens are produced.
  • 1) Hiding data dependencies and hybrid scheduling:: Hybrid scheduling uses more micro-batches for prompt processing to reduce pipeline bubbles and fewer during token generation to reduce execution time.Prompt processing is compute-saturating, whereas token generation is memory-bandwidth-bound and its microbatch execution time changes little with microbatch size.
  • 2) Offloading Activations to CPU Memory:: Cached key-value activations are reused predictably, enabling inactive tensors to be offloaded from GPU to CPU memory when allocated activation memory exceeds a threshold.The saved GPU memory supports larger batch sizes and better system utilization.
  • 3) Communication Optimization:: CPU offloading can stall transformer kernels over low-bandwidth PCIe, so the system overlaps communication with computation and uses architecture-aware transfer scheduling.
  • 3) Communication Optimization:: On systems where two GPUs share a PCIe link, odd-numbered GPUs offload odd-numbered layers while even-numbered GPUs offload even-numbered layers.This scheduling prevents PCIe contention and lets each GPU fully leverage available PCIe bandwidth during offloading.

V. MASSIVE SCALE SPARSE MODEL INFERENCE … C. Highly Optimized Computation Kernels for MoE

DeepSpeed Inference extends its dense-model system to massive-scale MoE inference through coordinated tensor, expert, and parallelism strategies plus optimized sparse computation kernels. These techniques reduce MoE kernel latency by over 6× and improve communication scaling to hundreds of GPUs.

  • V. MASSIVE SCALE SPARSE MODEL INFERENCE: The MoE inference system addresses sparse-model challenges with three main components built on the dense inference techniques.Sparse models require higher aggregate memory bandwidth, while expert parallelism introduces communication that becomes inefficient at hundreds of devices.
  • A. Orchestration of Tensor, Data, &Expert Parallelism for MoE: Tensor parallelism splits non-expert and expert parameters across GPUs, while expert parallelism extends scaling beyond the few GPUs limited by tensor-parallel communication overhead.The combined approach leverages aggregate GPU memory bandwidth and scales expert parameters to hundreds of GPUs.
  • B. PCC: Parallelism Coordinated Communication for MoE: PCC coordinates local transformations, intra- and inter-model-parallel all-to-all, allgather, and a final local transformation to accelerate communication.Despite four steps, the PCC sequence is faster than the baseline all-to-all shown in Fig. 5.
  • B. PCC: Parallelism Coordinated Communication for MoE: Expert parallelism alone scales poorly because all-to-all latency increases linearly with the number of devices, motivating communication optimization with tensor-slicing.When expert-parallel and tensor-sliced operators are combined, communication overhead is reduced from O(p) to O(p/L) + O(L).
  • B. PCC: Parallelism Coordinated Communication for MoE: On 128 GPUs, 8-way tensor-slicing reduces all-to-all latency overhead from (128C1 + C2) to (16C1 + C2).The reduction comes from combining tensor-slicing with 128-way expert parallelism.
  • C. Highly Optimized Computation Kernels for MoE: MoE kernels are slowed by sparse tensor representations, repeated kernel launches, and gating operations for token masks, top-k expert selection, cumulative sums, and sparse matrix multiplication.The optimized design replaces one-hot mappings with compact tables, scans mappings in parallel, and replaces sparse-einsum scattering.
  • C. Highly Optimized Computation Kernels for MoE: Over 6× lower MoE kernel-related latency results from replacing sparse einsums with data-layout transformations, shared-memory use, and kernel fusion.These optimizations reduce operation complexity from S × E × M × ce to S × M × ce.

VI. DEMOCRATIZATION OF LARGE MODEL INFERENCE. · A. ZeRO-Inference Design · B. Performance Optimizations

ZeRO-Inference democratizes large-model inference by using heterogeneous memory and as few as one GPU, while streaming model layers to support large batches. Its optimizations overlap transfers with computation and exploit aggregate multi-GPU PCIe bandwidth to improve efficiency.

  • VI. DEMOCRATIZATION OF LARGE MODEL INFERENCE.: ZeRO-Inference enables large-model inference with as few as a single GPU, broadening access beyond deployments requiring dozens of GPUs.It targets non-latency-sensitive applications and achieves high performance by leveraging heterogeneous memory.
  • A. ZeRO-Inference Design: ZeRO-Inference uses GPU memory, DRAM, and NVMe to fit massive models, adapting ZeRO-Infinity’s offloading techniques to inference.Terabytes of aggregate heterogeneous memory can accommodate hundreds of billion-parameter models even when GPU resources are limited.
  • A. ZeRO-Inference Design: Pinning model weights in GPU memory can avoid fetch latency, but it permits only small batch sizes, reducing inference efficiency.The design must apportion GPU memory among weights, inputs, and intermediate results.
  • A. ZeRO-Inference Design: ZeRO-Inference pins weights in DRAM or NVMe and streams each layer into GPU memory, allowing large inference batches despite PCIe transfer latency.Limiting GPU-resident weights to one or a few layers leaves more GPU memory available for batching.
  • B. Performance Optimizations: ZeRO-Inference mitigates weight-fetching overhead through prefetching, multi-GPU PCIe utilization, and additional I/O optimizations targeting near-peak NVMe bandwidth.The additional techniques include asynchronous bulk I/O, parallel request scheduling, memory pinning, and avoiding data copies; prior work introduced them.
  • B. Performance Optimizations: Prefetching configurable numbers of layers overlaps weight transfers with current-layer computation, improving throughput at the cost of additional GPU memory.The prefetch depth is configurable.
  • B. Performance Optimizations: In multi-GPU inference, each GPU fetches a layer partition and aggregates it over the faster GPU-GPU interconnect, using aggregate PCIe bandwidth to reduce transfer time.This optimization targets layer-transfer time in multi-GPU scenarios.

VII. PERFORMANCE EVALUATION … B. Evaluation of DeepSpeed Inference for Latency Sensitive Workloads

DeepSpeed Inference is evaluated across latency-sensitive and throughput-oriented workloads using dense and sparse GPT-style models at unprecedented scale. The evaluation compares specialized baselines on latency, token throughput, and compute throughput across GPU, CPU, and NVMe-equipped systems.

  • VII. PERFORMANCE EVALUATION: Up to 1.9× and 7.3× lower latency is achieved than state-of-the-art baselines for dense models with hundreds of billions and sparse models with trillions of parameters.The evaluation also reports up to 1.5× higher throughput for massive-model inference.
  • 1) Baseline:: Dense-model experiments use NVIDIA’s FasterTransformer, while sparse-model experiments use a distributed PyTorch implementation supporting tensor and expert parallelism [37].These systems provide the dense and sparse comparison baselines, respectively.
  • 2) Metrics:: The evaluation measures end-to-end generation latency, token throughput in tokens per second, and compute throughput in TFLOPS per GPU.Latency is measured for a batch of input prompts.
  • 3) Workloads:: The study evaluates GPT-style decoder models across varied hidden dimensions, layer counts, attention heads, and parameter counts based on GPT-3 and public variants.Sparse MoE configurations additionally vary expert degree, covering models from 52B to 2 trillion parameters.
  • 4) Testbeds:: Experiments run on up to 256 NVIDIA Ampere A100 40GB GPUs across 32 DGX systems, plus A6000 and V100 servers with DRAM and NVMe storage.The A6000 workstation has 2×48GB GPUs, 256GB DRAM, and 2TB NVMe; the DGX2 has 16×32GB V100 GPUs, 1500GB DRAM, and 30TB NVMe.
  • B. Evaluation of DeepSpeed Inference for Latency Sensitive Workloads: DeepSpeed Inference supports fast inference for dense models over 530B parameters and sparse models exceeding 2 trillion parameters at unprecedented scale.The latency-sensitive evaluation includes sparse models scaled to hundreds of GPUs.
  • B. Evaluation of DeepSpeed Inference for Latency Sensitive Workloads: DeepSpeed Transformer is compared with FasterTransformer [31] across models and batch sizes, while DeepSpeed-MoE is compared with a baseline on 256 GPUs.The figures report latency and throughput comparisons, with per-GPU throughput and latency improvement speedups for DeepSpeed-MoE.

1) Dense Model Evaluation: · 2) Sparse Model Evaluation: · C. Throughput Oriented Massive Model Inference

DeepSpeed Inference improves dense, sparse, and throughput-oriented massive-model inference through optimized kernels, fusion, parallelism, memory optimization, and communication minimization. It achieves up to 7.3× lower latency for MoE models and 1.51× higher throughput for GPT-3 175B than FasterTransformer configurations described in the supplied results.

  • 1) Dense Model Evaluation:: DeepSpeed Inference evaluates dense models up to 175B parameters with up to 16-way tensor parallelism, comparing DeepSpeed-FP16 and DeepSpeed-INT8 against FasterTransformer FP16 under identical tensor-parallel strategies.This setup attributes latency differences to kernel implementation differences rather than tensor-parallel configuration.
  • 1) Dense Model Evaluation:: Up to 1.55× speedup is achieved by DeepSpeed-FP16 over the baseline at small batch sizes on single- and multi-GPU configurations.The gains primarily result from deep-fusion and custom GeMMs, especially for smaller models affected by kernel-launch overhead and inefficient small, skinny GeMM execution.
  • 1) Dense Model Evaluation:: Up to 1.93× latency reduction is achieved by DeepSpeed-INT8 over the FasterTransformer FP16 baseline at larger batch sizes.DeepSpeed-FP16 reaches up to 1.57× improvement in the same setting, with deep-fusion reducing non-GeMM data-movement overhead.
  • 1) Dense Model Evaluation:: FasterTransformer’s INT8 support was limited to encoder-only Transformer models at the time of comparison, excluding decoders used by large-scale models such as GPT-3.This limitation constrains direct INT8 baseline comparisons for decoder-based models.
  • 1) Dense Model Evaluation:: Up to 7.3× latency reduction is achieved by DeepSpeed-MoE over the baseline for single-token generation on 100B–2T MoE models using up to 256 GPUs.The baseline and DeepSpeed-MoE use identical data, tensor, and expert parallelism configurations, isolating the impact of DeepSpeed Inference optimizations.
  • C. Throughput Oriented Massive Model Inference:: Throughput-oriented inference targets applications such as offline query rewriting in web-scale search and recommendation systems, where token generation is less latency-focused.For the 530B model, FasterTransformer could not run with combined tensor and pipeline parallelism without crashing.
  • C. Throughput Oriented Massive Model Inference: 1.51× throughput improvement is achieved over the best FasterTransformer configuration for GPT-3 175B on two nodes with 2 × 8 A100 GPUs.The improvement comes from an improved pipeline-parallelism schedule and larger batch sizes enabled by memory optimization and communication minimization.

D. Democratizing Larger Model Inference with ZeRO-Inference … 3) Prompt latency improvement with hybrid scheduling:

ZeRO-Inference democratizes massive-model inference by enabling 530B-parameter models on a single A6000 while delivering high throughput through CPU/NVMe offloading and scalable multi-GPU execution. DeepSpeed Inference further improves performance through optimized kernels, scheduling, memory, communication, and hybrid parallelism.

  • 1) Model Scale:: 530B-parameter inference on a single A6000 is 25× larger than the largest GPU-only model and 10× larger than the CPU-only solution.This enables data scientists to test massive models without large GPU clusters or their associated costs.
  • 2) Inference Throughput:: 84 TFLOPS reaches 54% of the 158.4-TFLOPS theoretical peak for offline inference with very large batch sizes.For models fitting in CPU memory, throughput is over 25× higher than CPU-only; for models fitting in one GPU, it is over 50% better than GPU-only.
  • 2) Inference Throughput:: CPU or NVMe parameter offloading leaves GPU memory for activations, enabling larger batch sizes that improve throughput.The throughput benefit of larger batch sizes is shown in Fig. 9(a).
  • 3) Scalability:: Additional GPUs provide near-perfect linear throughput scaling by aggregating PCIe bandwidth across GPUs.The scaling behavior is shown in Fig. 9(c) and uses the aggregate PCIe bandwidth described in Sec. VI-B.
  • 1) Dense GPU kernel performance breakdown:: Deep-fusion reduces latency by lowering kernel-launch and data-movement overheads, while custom GeMM further helps small batches through higher memory-bandwidth utilization.The comparison is against the PyTorch baseline in Fig. 10(a).
  • 2) Throughput breakdown for massive model GPU-Inference:: DeepSpeed Inference combines dense optimized kernels, optimized scheduling, memory optimizations, and communication optimizations to increase massive-model GPU inference throughput.These optimizations increase batch size and reduce PCIe data-movement overheads.
  • 3) Prompt latency improvement with hybrid scheduling:: Hybrid scheduling achieves 1.18× and 3.06× prompt-processing speed-ups over FasterTransformer for GPT-3 175B with PP+MP and MP-only configurations, respectively.The experiment used two nodes with eight A100 GPUs each, enabled pipeline and tensor parallelism, and used batch size 24 because larger batches sharply increased latency.

4) Memory bandwidth scalability for sparse MoE models: … VIII. CONCLUSION

DeepSpeed Inference improves sparse-MoE memory-bandwidth scalability, heterogeneous-inference throughput, and encoder latency through system-wide optimizations. Its broader contribution is efficient inference across diverse transformer architectures, model scales, GPU counts, and latency requirements.

  • 4) Memory bandwidth scalability for sparse MoE models:: DeepSpeed Inference achieves higher per-GPU memory bandwidth and better scalability than PyTorch for a 52B MoE model across 8 to 128 GPUs, improving latency and throughput.The result combines MoE kernels with all-to-all optimizations.
  • 5) Impact of pre-fetching on ZeRO-Inference throughput:: Prefetching improves ZeRO-Inference throughput at small batch sizes, but its benefit diminishes at larger batches as higher arithmetic intensity hides CPU/NVMe-to-GPU communication overhead.This result is shown for a single V100 GPU.
  • 6) Comparison with E.T.:: 1.7x and 1.4x faster than E.T. on DistilBERT and BERT encoders, respectively, DeepSpeed Inference benefits from broader operator fusion and higher memory-bandwidth utilization.DeepFusion fuses more operators, reducing kernel invocation overhead; the system also supports encoder, decoder, and sparsely gated MoE models at larger scales.
  • 6) Comparison with E.T.:: DeepSpeed Inference delivers throughput across batch sizes and model scales, including GPT-NeoX-20B and GPT-50B evaluated on A6000 and DGX2 V100 systems.The GPT-50B comparison uses up to 16 GPUs and reaches 67 TFLOPS, or 53% of peak, on the DGX2 V100.
  • 5) Impact of pre-fetching on ZeRO-Inference throughput:: Deep-Fusion and optimized GeMM improve GPT2 throughput, while pipeline-parallelism optimizations improve throughput for a 530B model.The figure also reports prefetching effects for ZeRO-Inference on a single V100 GPU.
  • 6) Comparison with E.T.:: DeepSpeed Inference is evaluated against FasterTransformer using prompt-processing latency and TFLOPS under hybrid scheduling.The supplied passage identifies the comparison but does not provide the plotted values.
  • VIII. CONCLUSION: The system enables efficient transformer inference at unprecedented model sizes, GPU counts, and performance levels as architectures evolve and latency requirements become more stringent.Its system-wide innovations target speedy, efficient, and economical inference across increasingly diverse transformer models and applications.
Loading 2207.00032v1…