PaperlayerView exact revision on arXiv

Source-linked AI summary

TurboTransformers: An Efficient GPU Serving System For Transformer Models

Jiarui Fang, Yang Yu, Chengduo Zhao, Jie Zhou

arXiv:2010.05680v4cs.DCcs.AIcs.LG

TL;DR

Transformer serving must handle high computation costs and variable-length inputs while meeting online latency and throughput requirements. TurboTransformers combines a specialized runtime with a serving framework and introduces GPU kernel, memory allocation, and batch scheduling techniques. The paper reports state-of-the-art transformer serving performance, smaller memory footprint, and improved throughput, subject to stated scheduling and memory-management assumptions.

  • Problem

    Efficient GPU serving is difficult because transformer models demand substantial computation and variable-length inputs complicate memory management and batching.

  • Method

    TurboTransformers combines a lightweight runtime and serving framework with optimized batch-reduction kernels, sequence-length-aware memory allocation, and dynamic-programming-based batch scheduling.

  • Results

    TurboTransformers reports state-of-the-art speed, smaller memory footprint, and improved throughput compared with existing transformer-serving runtimes.

  • Takeaways & Limitations

    The system supports efficient GPU transformer serving and can be integrated into PyTorch code with a few lines of Python.

  • Takeaways & Limitations

    The batch-scheduling method assumes a request-scheduling strategy can meet the server’s SLO; multi-server deployments require an upper-level load balancer to avoid overload.

Abstract

from arXiv · show

The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years. Unlike the Recurrent Neural Network (RNN) models, Transformers can process on dimensions of sequence lengths in parallel, therefore leading to better accuracy on long sequences. However, efficient deployments of them for online services in data centers equipped with GPUs are not easy. First, more computation introduced by transformer structures makes it more challenging to meet the latency and throughput constraints of serving. Second, NLP tasks take in sentences of variable length. The variability of input dimensions brings a severe problem to efficient memory management and serving optimization. This paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework to solve the above challenges. Three innovative features make it stand out from other similar works. An efficient parallel algorithm is proposed for GPU-based batch reduction operations, like Softmax and LayerNorm, major hot spots besides BLAS routines. A memory allocation algorithm, which better balances the memory footprint and allocation/free efficiency, is designed for variable-length input situations. A serving framework equipped with a new batch scheduler using dynamic programming achieves the optimal throughput on variable-length requests. The system can achieve the state-of-the-art transformer model serving performance on GPU platforms and can be seamlessly integrated into your PyTorch code with a few lines of code.

1 Introduction

Transformer services are difficult to deploy efficiently because they combine high computation demands with variable-length inputs that complicate memory management and batching. TurboTransformers addresses these challenges through a specialized runtime, memory allocator, batch scheduler, and GPU kernel optimizations.

  • Motivation: Transformer inference requires substantially more computation than representative CNN models, increasing the difficulty of meeting serving latency and throughput constraints.A 40-word base BERT inference requires 6.9 Gflops, while a 20-word Chinese-English Seq2seq translation requires over 20 Gflops.
  • Motivation: Variable-length inputs force intermediate tensor dimensions to change and prevent pre-optimization of memory space for known lengths.Unlike RNNs, transformers cannot split variable-length inputs into sequential fixed-length inputs to simplify allocation.
  • Motivation: Zero-padding short requests can offset the GPU efficiency gains of batching variable-length requests.Serving frameworks must often pad requests to the longest sequence before processing them together, introducing extra computation.
  • TurboTransformers: TurboTransformers combines a lightweight runtime with a serving framework that supports variable-length inputs, fused kernels, and input-dependent memory optimization.The runtime rewrites the computation graph, fuses non-GEMM kernels, provides CUDA implementations, and performs lightweight memory optimization before inference.
  • Results: TurboTransformers reports state-of-the-art speed, smaller memory footprint, easy integration, and improved service throughput compared with existing runtimes.Its runtime can provide end-to-end speedup with a few lines of Python code, while the serving framework improves throughput through variable-length-aware batching.
  • Contributions: The system introduces GPU batch-reduction kernels, sequence-length-aware memory allocation, and dynamic-programming-based batch scheduling.These innovations target Softmax and LayerNorm efficiency, memory reuse for variable dimensions, and optimal-throughput batching.

2 Backgrounds

Transformers use self-attention to process sequence positions in parallel, but this creates variable-length tensors that complicate batching and serving. Background material describes the architecture, serving requirements, and limitations of existing inference systems.

  • 2.1 Transformer Models: Self-attention lets transformers attend to different input positions and process variable-length sequences through stacks of attention layers.The architecture may use encoder and decoder components, although encoder-only models such as BERT are also common.
  • 2.1 Transformer Models: Multi-head attention transforms query, key, and value tensors through linear layers, scaled dot-product attention, concatenation, and a final linear layer.Scaled dot-product attention applies Softmax to obtain weights over values.
  • 2.1 Transformer Models: Sequence-length parallelism makes the dimensions of Q, K, and V unpredictable during serving and complicates batching.When requests are processed together, shorter sequences are filled with zeros to match the longest sequence.
  • 2.2 Serving Systems: Serving repeatedly performs model inference on online inputs and must satisfy real-time, low-latency requirements.Unlike training, serving uses repeated forward passes without backward computation.
  • 2.2 Serving Systems: Existing inference runtimes often target fixed-length workloads and use preprocessing or zero-padding to handle variable-length transformer requests.The cited systems include onnxruntime, TensorFlow XLA, TVM, and TensorRT; onnxruntime supports dynamic axes after version 1.3.
  • 2.2 Serving Systems: Serving frameworks use batching to improve GPU utilization, but static batching may require zero-padding when requests do not fill a batch.Batching is identified as the paper’s main serving-framework focus.

3 Design Overview

TurboTransformers consists of an inference runtime and a serving framework that receive network requests, run transformer inference, and return results to users.

  • Design Overview: TurboTransformers has two components: an inference runtime and a serving framework.The serving framework exposes gRPC/HTTP endpoints and wraps the runtime as a service.
  • Design Overview: The system accepts network requests, processes them with transformer models, and responds with the resulting outputs.The runtime and serving framework are described separately in the paper’s subsequent sections.

4 Inference Runtime

TurboTransformers’ inference runtime targets transformer serving bottlenecks through kernel fusion, GPU batch-reduction optimization, and variable-length-aware memory management.

  • Kernel Fusion: 61.8% of PyTorch BERT inference time is spent on GEMM kernels and 38.2% on non-GEMM kernels at batch size 20 and sequence length 128.At batch size 1 and sequence length 40, the GPU is idle 80.64% of the time, highlighting launch-overhead problems for small workloads.
  • Kernel Fusion: Kernel fusion combines non-GEMM operators between GEMM kernels, reducing memory accesses, improving cache locality, and lowering kernel-launch overhead.The runtime reorganizes the transformer computation graph into a more compact graph and implements fused non-GEMM kernels with CUDA.
  • GPU-based Batch Reduction: TurboTransformers treats Softmax and LayerNorm as batch-reduction operations that reduce multiple 1D arrays in parallel.These operators calculate reductions such as summation, maximum, mean, and variance, making them important attention-layer hotspots when poorly optimized.
  • GPU-based Batch Reduction: Combining X independent reductions reduces synchronization cost, merges boundary processing, and removes instruction dependencies.The proposed warpAllReduceSum_XElem routine addresses synchronization overhead, warp divergence, and inefficient instruction issuing in classical GPU reductions.
  • GPU-based Batch Reduction: TurboTransformers’ LayerNorm optimization uses an equivalent variance formula so x and x^2 can be reduced simultaneously.The warpAllReduceSum_2Elem routine reduces synchronization requirements while improving instruction execution efficiency.
  • Memory Management: TurboTransformers uses chunk-based caching and computation-graph-aware reuse to balance allocation efficiency and memory footprint for variable-length tensors.Tensor lifetimes and sequence lengths determine offsets, allowing tensors with non-overlapping lifetimes to share memory; the allocator uses 2MB default chunks and a 1.2 size scale in the implementation.

5 Serving Framework

TurboTransformers’ serving framework combines caching with batching, then schedules variable-length requests to balance padding overhead against batching gains. Its dynamic-programming scheduler optimizes throughput under latency constraints, subject to an SLO-feasibility premise.

  • Serving Framework: Batching packages requests arriving over a time period to improve GPU utilization, with especially significant speedups for short sequences.The framework first receives requests through a message queue and can return cached results before inference.
  • Serving Framework: Variable-length batching must balance zero-padding overhead against the performance benefits of larger batches.Padding all requests to the longest sequence can make indiscriminate batching less efficient than serving without batching.
  • Serving Framework: A five-request example achieves optimal throughput by packing three batches, improving response throughput by 35%.The request lengths are 17, 18, 52, 63, and 77; batching all five together is less efficient than no batching.
  • Serving Framework: The sequence-length-aware scheduler uses dynamic programming in O(n^2) time to maximize response throughput from measured inference costs.It sorts requests by sequence length and uses cached costs indexed by sequence length and batch size.
  • Serving Framework: The scheduler assumes a request-assignment strategy exists that can satisfy the server’s latency SLO.A load balancer can help ensure that requests assigned to each server do not overload it.

6 Experimental Results

Experiments show that TurboTransformers improves runtime speed and memory usage while its variable-length-aware scheduler substantially increases serving throughput. The benefits are strongest for short or highly variable requests, where conventional batching incurs zero-padding overhead.

  • Runtime performance: TurboTransformers outperformed PyTorch on average for BERT, Albert, DistilBert, and Decoder variable-length inference.Average speedups were 1.25x, 1.17x, 1.13x, and 1.16x, respectively.
  • Runtime performance: 1.01x average BERT speedup over onnxruntime and 1.03x for DistilBert show similar runtime performance between TurboTransformers and onnxruntime.The reported ranges were 0.88x-1.05x for BERT and 0.83x-1.36x for DistilBert.
  • Memory optimization: 663 MB peak GPU memory for Turbo was lower than PyTorch’s 1307 MB and onnxruntime’s 1653 MB.The comparison reflects peak device memory usage measured during runtime execution.
  • Memory optimization: 1.8% average offset-scheduling overhead supports the model-aware allocator’s variable-length memory reuse strategy.The allocator reduces tensor-address computation for repeated structures by reusing addresses across equivalent structures.
  • Serving framework: 402 resp/sec was achieved by Turbo-DP-Batch, compared with 99 resp/sec for PyTorch-Nobatch and 323 resp/sec for Turbo-Naive-Batch.The dynamic-programming scheduler reached 4.06x the PyTorch-Nobatch throughput under the reported workload.
  • Serving framework: 144 resp/sec for Turbo-TC-DP-Batch exceeded Turbo-TC-NoBatch’s 120 resp/sec and avoided the 98 resp/sec throughput of naive batching.The workload used sequence lengths from 5 to 500, where zero-padding reduced naive-batching efficiency.

7 Conclusion

TurboTransformers addresses transformer serving’s computation and variable-length-input challenges through innovations in computation, memory allocation, and request batching. Its runtime improves speed and memory footprint, while its scheduler increases throughput for variable-length requests.

  • Conclusion: TurboTransformers targets transformer serving’s computation pressure and variable-length input problems in GPU datacenters.The conclusion frames these as the two critical deployment problems addressed by the system.
  • Conclusion: The system combines parallel batch reduction, sequence-length-aware memory allocation, and sequence-length-aware batch scheduling.These innovations operate at the computing, memory, and serving levels.
  • Conclusion: The runtime is faster than PyTorch, similar in speed to onnxruntime, and uses a smaller memory footprint in variable-length request tests.The conclusion also reports comparable speed with TensorFlow-XLA, TensorRT, and FasterTransformers in fixed-length tests.
  • Conclusion: The serving framework achieves higher throughput than conventional batching for variable-length requests.The conclusion attributes this result to the proposed batch scheduler.

A.1 Abstract

The artifact provides TurboTransformers runtime code, instructions for reproducing critical results, and scripts for running the paper’s experiments.

  • Artifact: The artifact contains runtime code, reproduction instructions, and experiment-running scripts for TurboTransformers.It specifically covers the runtime component rather than the full serving framework.

A.2.1 Check-list (artifact meta information).

The artifact includes the proposed runtime algorithms, randomly generated benchmark inputs, build instructions, and an open-source delivery channel.

  • Algorithms: The artifact includes parallel batch-reduction algorithms and the model-aware allocator.These correspond to the runtime algorithms proposed in the paper.
  • Datasets and compilation: Benchmark scripts use randomly generated inputs, with compilation instructions for g++ 7.5.0, nvcc 10.2, and Docker 19.03.8.The listed runtime environment uses CentOS 7 with CUDA 10.2.
  • Delivery: The artifact is available as open source under the BSD license through the TurboTransformers GitHub repository.The artifact branch is identified as ppopp21_artifact_centos.

A.3 Installation

Installation uses Docker and NVIDIA Docker to build and run a GPU-enabled container, then builds the artifact inside it. Benchmark scripts compare Turbo Runtime with PyTorch and reproduce results for BERT and ALBERT.

  • Installation: Build a Docker image with the provided Dockerfile and build script.Use `bash tools/build_docker_gpu.sh $PWD`.
  • Installation: Run the image as a GPU-enabled container with NVIDIA Docker and the specified workspace mount.The command uses `--gpus all`, host networking, automatic removal, interactive mode, and names the container `turbo_dev:latest`.
  • Installation: Build the artifact inside the container, including on CentOS 7 using the provided command.The instructions specify building inside the container and provide a CentOS 7 build procedure.
  • Benchmarking: Run the benchmark-directory scripts to compare Turbo Runtime with PyTorch and reproduce BERT and ALBERT results.The variable-length benchmark reproduces Figure 9 results, while the fixed-length benchmark reproduces Figure 14 results.
Loading 2010.05680v4…