PaperlayerView exact revision on arXiv

Source-linked AI summary

Full Stack Optimization of Transformer Inference: a Survey

Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, Amir Gholami

arXiv:2302.14017v1cs.CLcs.LG

TL;DR

The paper addresses the challenge of deploying increasingly compute- and bandwidth-intensive Transformer models in latency-sensitive settings. It surveys full-stack inference optimizations and evaluates them through a Gemmini case study, achieving 88.7× EDP improvement without a noticeable performance drop versus a naive implementation. The findings also identify scope constraints, including limited NAS search spaces and CNN-oriented accelerator designs that handle Transformer operations poorly.

  • Problem

    Transformer inference is becoming increasingly compute- and bandwidth-intensive, while the workload characteristics and design principles needed for efficient deployment remain insufficiently understood.

  • Method

    The paper analyzes and surveys Transformer inference across hardware and software layers, then applies the surveyed optimization methods in a Gemmini case study.

  • Results

    88.7× EDP improvement was achieved without a noticeable performance drop compared with a naive implementation lacking full-stack considerations.

  • Takeaways & Limitations

    Full-stack co-design and co-optimization can substantially improve Transformer inference, including through on-chip nonlinear-operation support and memory-hierarchy rebalancing.

  • Takeaways & Limitations

    Supernet-based weight sharing can constrain NAS search spaces, potentially limiting discovery of unique or innovative Transformer architectures.

Abstract

from arXiv · show

Recent advances in state-of-the-art DNN architecture design have been moving toward Transformer models. These models achieve superior accuracy across a wide range of applications. This trend has been consistent over the past several years since Transformer models were originally introduced. However, the amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate, and this has made their deployment in latency-sensitive applications challenging. As such, there has been an increased focus on making Transformer models more efficient, with methods that range from changing the architecture design, all the way to developing dedicated domain-specific accelerators. In this work, we survey different approaches for efficient Transformer inference, including: (i) analysis and profiling of the bottlenecks in existing Transformer architectures and their similarities and differences with previous convolutional models; (ii) implications of Transformer architecture on hardware, including the impact of non-linear operations such as Layer Normalization, Softmax, and GELU, as well as linear operations, on hardware design; (iii) approaches for optimizing a fixed Transformer architecture; (iv) challenges in finding the right mapping and scheduling of operations for Transformer models; and (v) approaches for optimizing Transformer models by adapting the architecture using neural architecture search. Finally, we perform a case study by applying the surveyed optimizations on Gemmini, the open-source, full-stack DNN accelerator generator, and we show how each of these approaches can yield improvements, compared to previous benchmark results on Gemmini. Among other things, we find that a full-stack co-design approach with the aforementioned methods can result in up to 88.7x speedup with a minimal performance degradation for Transformer inference.

1 INTRODUCTION

Transformer models offer strong accuracy but create growing compute, memory, bandwidth, and latency challenges, motivating efficient inference across the hardware–software stack. The paper surveys these challenges and optimization strategies, then applies them to Gemmini, finding substantial gains from full-stack co-design.

  • Transformer workloads are increasingly difficult to deploy on latency-sensitive, resource-constrained devices because their compute and bandwidth demands are growing.Edge devices also face tight energy and memory constraints.
  • The paper analyzes Transformer runtime characteristics and bottlenecks while surveying hardware, fixed-architecture optimization, operation mapping, scheduling, and architecture adaptation.The survey spans multiple levels of end-to-end inference, including nonlinear operations and neural architecture search.
  • The Gemmini case study applies the surveyed methodologies to a full-stack DNN accelerator generator originally developed for deep learning workloads.The stated goal is to characterize factors across the hardware and software stack for Transformer inference.
  • Gemmini designs optimized for CNN workloads are not well suited to Transformer inference because floating-point nonlinear, quantization, and dequantization operations can dominate performance.The paper identifies these operations as a primary bottleneck rather than assuming linear operations are always dominant.
  • Transformer accelerators may benefit from larger accumulators and smaller scratchpads than CNN-oriented designs, yielding a 36% latency improvement over a CNN-optimized baseline.The cited comparison concerns an accelerator architecture changed according to Transformer-specific observations.

2 TRANSFORMER MODEL ARCHITECTURE AND PERFORMANCE BOTTLENECKS

Transformer inference combines projection-heavy MHA and FFN computation with sequence-dependent act-to-act attention and nonlinear operations. As sequence length grows, quadratic attention costs, lower arithmetic intensity, and memory movement increasingly dominate performance, especially for decoder-only models.

  • Architecture: Transformer blocks contain MHA and FFN modules, with MHA using projections and act-to-act matmuls while FFN uses two linear layers.MHA includes query, key, value, and output projections plus query × key and attention-score × value matmuls.
  • Workload scaling: Act-to-act matmuls scale quadratically with sequence length, whereas MHA and FFN projection costs scale linearly.The quadratic terms arise in attention computations such as query × key and attention score × value.
  • Arithmetic intensity: Arithmetic intensity in encoder-only models initially increases with sequence length but decreases for longer sequences as low-intensity MHA act-to-act matmuls become dominant.The FFN module has higher arithmetic intensity than the MHA module, helping explain the decline at long sequence lengths.
  • Model comparison: Decoder-only GPT-2 inference has lower arithmetic intensity and longer latency than encoder-only BERT despite similar model configurations and FLOPs.Matrix-vector operations limit data reuse because weights cannot be shared across generated tokens, making decoder inference memory-bandwidth-bound.
  • Workload scaling: At longer sequence lengths, act-to-act matmuls consume more computation and, together with Softmax, dominate memory operations while having lower arithmetic intensity than projection layers.Softmax contributes substantial memory traffic because l × l attention matrices must be written or loaded per head.
  • CNN comparison: Transformer nonlinear operations consume relatively few FLOPs but can account for substantial memory operations, unlike several CNN nonlinearities that can be fused or folded into preceding layers.ResNet50 ReLU and BatchNorm can be fused or folded, whereas Transformer nonlinear operations present greater inference-efficiency challenges.

3 HARDWARE DESIGN

This section describes hardware architectures and analytical methods for efficient Transformer inference, then shows how adapting a CNN-oriented accelerator addresses Transformer-specific bottlenecks. The Gemmini case study combines memory, arithmetic, and non-linear-operation changes to substantially improve BERT performance.

  • Accelerator Organization: Typical DNN accelerators combine off-chip DRAM, on-chip buffers, processing elements, and local memories to exploit data reuse.Reducing memory traffic is critical because local-buffer and DRAM reads can cost roughly 6× and 200× a MAC, respectively.
  • Analytical Modeling: Analytical modeling estimates runtime behavior and bottlenecks when direct hardware profiling is difficult, guiding accelerator design decisions.The section uses latency breakdowns and non-ideal arithmetic intensity to assess Transformer execution on target hardware.
  • Transformer Bottlenecks: Transformer inference exposes low arithmetic intensity and non-linear operations that CNN-oriented accelerators handle inefficiently.Large output activations, tiling, and CPU execution of LayerNorm, Softmax, and GELU reduce utilization and increase transfer overhead.
  • Hardware-Software Co-Design: 39.6× overall end-to-end BERT inference improvement followed hardware changes that increased accumulator capacity, added integer BERT support, and integrated normalizer and activation units.These changes reduced non-matmul bottlenecks and enabled GELU fusion with preceding matrix multiplications.
  • Hardware-Software Co-Design: I-BERT implementations of LayerNorm, Softmax, and GELU increase accelerator area by 5–15% and add 8% to total latency while avoiding costly CPU offloading.The result addresses the practical latency impact of non-linear operations without relying solely on ideal FLOP, area, or power analysis.

4 MODEL OPTIMIZATION

Model optimization improves Transformer inference by reducing precision, computation, memory traffic, and arithmetic cost, while requiring accuracy-aware handling of quantization and hardware constraints.

  • 4.1 Quantization: 4× smaller model size results when model weights are quantized from FP32 to INT8, reducing off-chip storage and bandwidth.Quantized activations can also reduce intermediate storage and allow memory hierarchies to store more parameters or use smaller buffers.
  • 4.1 Quantization: 39.6× lower end-to-end inference latency is achieved by integer-only quantization on Gemmini.This setting conducts all computations using integer arithmetic.
  • 4.1 Quantization: Aggressive precision reduction can significantly degrade accuracy, motivating mixed-precision methods that assign higher precision to more quantization-sensitive layers.Q-BERT uses Hessian information, while HAT uses reinforcement learning to select layer precisions.
  • 4.1 Quantization: Activation outliers challenge uniform quantization of pre-trained Transformers because the expanded value range can map more values to the same quantized value.Non-uniform quantization and higher precision for outlier-containing activations are described as possible remedies.
  • 4.1 Quantization: Quantization compresses parameters and activations by representing them with lower-bit precision, reducing memory consumption and potentially improving inference efficiency.It can reduce off-chip storage and bandwidth, memory traffic, and the size, latency, and energy consumption of arithmetic units.
  • 4.2 Sparsity: Unstructured pruning can require compressed memory formats and specialized computation units, otherwise decompression and recompression add overhead.Commodity DNN accelerators may therefore fail to efficiently exploit unstructured sparsity patterns.

5 MAPPING TRANSFORMERS TO HARDWARE

Transformer mapping converts model operations into hardware schedules, but the many valid choices for memory movement, parallelism, loop ordering, and tiling make performant mapping difficult.

  • 5.1 What are Mappings?: A mapping or schedule is a sequence of hardware instructions that executes operations and moves data on a specific target architecture.For Gemmini, this can include dense matrix multiplications and transfers between off-chip DRAM and local SRAMs.
  • 5.1 What are Mappings?: Different valid mappings can produce different latency, bandwidth, and energy consumption, so mapping frameworks seek Pareto-optimal schedules.The objective depends on the desired performance metrics and target hardware.
  • 5.1 What are Mappings?: Mapping core Transformer operators is challenging because their mapspaces are large, yet good choices can substantially improve overall execution speed.The mapspace contains alternative valid decisions whose performance differs across latency, bandwidth, and energy objectives.
  • 5.1 What are Mappings?: Transformer and CNN mappings assign loop levels to memory locations, spatial or temporal execution, and tiled or untiled traversal.The examples contrast six nested loops for convolution with three nested loops for matrix multiplication.

5.2 What Are the Key Mapping Decisions?

Transformer scheduling requires graph-level and operator-level decisions, including fusion, sparsification, tiling, dataflow, and parallelization, across a combinatorially large decision space.

  • 5.2 What Are the Key Mapping Decisions?: Mapping first transforms the graph into tensor operations, then schedules each operation into hardware instructions.Graph transformation may fuse operations, sparsify tensors, and select quantization strategies.
  • 5.2.1 Graph-level: Layer fusion reduces interlayer communication by keeping intermediate results on chip, but Transformer fusion can increase intralayer communication and may provide limited latency improvement.Its effectiveness depends on the target hardware platform.
  • 5.2.2 Operation-level: Operation-level scheduling chooses tile dimensions, dataflow and loop ordering, and which axes execute spatially or serially.These choices determine memory-hierarchy placement, stationary tensors, and parallel execution.
  • 5.2.3 Mapspace: Mapping choices can affect performance by up to several orders of magnitude, making cost minimization over the mapspace the mapper’s central goal.Typical objectives include energy, energy-delay product, and latency.
  • 5.2.3 Mapspace: Optimal mappings can differ substantially across hardware architectures, requiring mapping decisions for each neural-network and hardware pair.This increases the difficulty of hardware–software codesign.

5.3 Finding Performant Mappings

Performant mapping methods manage large scheduling spaces through subspace restriction, graph-level optimization, brute-force or feedback-driven search, and constrained optimization.

  • 5.3 Finding Performant Mappings: Mapping algorithms reduce search cost by exploring only a subspace of the full mapspace rather than making every required scheduling decision.This is a response to the size of the mapping search space.
  • 5.3 Finding Performant Mappings: Graph-level schedulers optimize operation fusion, resource allocation, graph partitioning, and graph rewriting, with fusion improving data reuse across layers.Transformer-specific fusion techniques have also been proposed.
  • 5.3 Finding Performant Mappings: Operation-level mappers commonly use brute-force search, feedback-based search, or constrained optimization to select tiling, dataflow, and spatio-temporal mappings.These categories differ in how they explore or formulate scheduling decisions.
  • 5.3 Finding Performant Mappings: Brute-force methods combine sampling or exhaustive exploration with heuristics and lightweight performance models to compare valid mappings.Their cost is reduced by pruning the mapspace before evaluating candidates.
  • 5.3 Finding Performant Mappings: Feedback-driven methods can learn scheduling spaces or cost models, but their evaluation and learning costs are significant.They typically target existing hardware or analytical models where large-scale measurement is feasible.
  • 5.3 Finding Performant Mappings: Constrained-optimization methods formulate scheduling as numerical optimization with variables, constraints, and objectives, including mixed-integer programming and polyhedral transformations.These methods provide an alternative to exhaustive and learning-based search.
  • 5.3 Finding Performant Mappings: Mapping strategies developed for CNNs can also target Transformers because both rely on tensor-algebra operations, while Transformer-specific decisions remain important.Operator fusion encodes a large space of choices about overlapping layer execution.

5.4 Performance Modeling of Mappings

Performance models help mappers compare Transformer mappings without executing hardware, but they trade evaluation cost against fidelity. Code-generation and scheduling tools then implement selected mappings while accounting for hardware-specific instructions and edge cases.

  • Performance modeling: Performance models provide mapping feedback without hardware execution and differ in fidelity, runtime cost, workload scope, and algorithm compatibility.They serve as performance proxies for mapping optimization.
  • Performance modeling: Analytical and polynomial models estimate Transformer mapping performance from iteration-space bounds and statically analyzable data-access patterns.Polynomial models can also serve directly as optimization objectives.
  • Performance modeling: Data-driven machine-learning models fit mapping-performance data iteratively, but typically require substantial data for accurate predictions.After training, they can integrate with ML-based mappers.
  • Performance modeling: Cycle-exact models and FPGA emulation provide higher-fidelity estimates by reflecting real hardware implementations, though they require more implementation effort.Lower-fidelity models may miss hardware implementation differences and produce suboptimal mappings.
  • Implementation: Code generation and user-schedulable languages translate mapping decisions into executable implementations while handling memory movement, instruction generation, and edge cases.These tools can also support hardware-aware neural architecture search through accurate performance evaluation.

5.5 Transformer vs CNN Mapping

Transformer matmul mapping is as difficult to optimize as CNN convolution mapping, despite fewer loop levels, while LayerNorm and Softmax introduce additional scheduling constraints. The best fusion and overlap choices depend on hardware resources and sequence length.

  • 5.5.2 Nonlinear-operation scheduling: LayerNorm and Softmax add scheduling complexity beyond Transformer matmuls, requiring deeper analysis of mapping and operation interactions.The section explicitly characterizes Transformer mapspaces and examines scheduling complexity caused by these nonlinear operations.
  • 5.5.1 Mapspace characterization: 100K random valid mappings were evaluated for BERT and ResNet50 to compare their legal mapping spaces and estimated latency and energy.The study used Timeloop and targeted Gemmini’s systolic hardware architecture.
  • 5.5.2 Nonlinear-operation scheduling: Fusion can be counterproductive on memory-constrained edge devices because normalization needs long vectors resident on-chip and may force awkward matmul tile shapes.These constraints can undermine the expected benefits of fusing matmuls with normalization operations.
  • 5.5.2 Nonlinear-operation scheduling: 78% of total MHA cycles can come from Softmax, making latency hiding through overlap especially important when scheduling query × key and Softmax.The query × key matmul remains relatively unchanged because its inner dimension is small for BERT-Base.
  • 5.5.2 Nonlinear-operation scheduling: 27% higher total latency can result from fusion-optimized scheduling, showing that benefits depend on accumulator SRAM size and sequence length.For BERT-Base, overlapping query × key with Softmax is favorable, whereas chaining FFN W2 with LayerNorm is disadvantageous.

6 ADAPTING TRANSFORMER ARCHITECTURE WITH NAS

Neural architecture search adapts Transformer designs to hardware constraints by exploring architectures through defined search, sampling, and evaluation procedures. In the Gemmini case study, supernet-based search and evolutionary optimization found Pareto-efficient models with substantial efficiency gains at small perplexity costs.

  • 6.1 Neural architecture search: NAS searches architecture spaces through three components: a search space, a search method, and an evaluation method.The search space defines valid architectures, the search method explores them, and evaluation measures candidate performance.
  • 6.2 Hardware-aware NAS: Hardware-aware NAS combines accuracy with latency or related hardware metrics, using lookup tables or prediction models when direct measurement is slow.Direct hardware measurement is accurate but slow and poorly parallelizable.
  • 6.3 Transformer-specific NAS: Supernet-based NAS can limit architectural diversity because its constraints restrict the range of architectures that can be discovered.The paper identifies balancing search flexibility and efficiency as an open need.
  • 6.3 Transformer-specific NAS: Transformer NAS commonly relies on weight sharing and evolutionary search because Transformer training is computationally expensive.Existing methods often tune hidden dimensions, depth, and head count rather than discovering more novel Transformer variants.
  • 6.4 NAS case study: 1.4× lower latency and 1.6× lower energy were achieved with 0.1 point perplexity degradation, increasing to 2.4× and 4.4× respectively at 1 point degradation.The reported trade-offs came from a single co-design run on a specific hardware platform.
  • 6.4 NAS case study: 2.2× lower EDP was achieved with 0.1 point perplexity degradation, while 10.6× lower EDP was achieved with 1 point degradation versus the baseline.These results came from the WikiText-2 case study using optimized Gemmini hardware.

7 CONCLUSION

The paper surveys Transformer inference bottlenecks and full-stack optimization opportunities, then evaluates co-design techniques across hardware and software. Its case studies report substantial efficiency gains while preserving performance or perplexity under stated trade-offs.

  • Scope: The study analyzes Transformer runtime characteristics and surveys hardware, fixed-architecture, mapping, scheduling, and architecture-search optimizations.The survey spans accelerator design, pruning and quantization, operation mapping, scheduling, and automated NAS.
  • Key findings: Nonlinear operations can strongly affect performance despite small FLOPs counts because Softmax and LayerNorm require runtime statistics, unlike inference-fusable CNN BatchNorm.The finding motivates hardware designs that explicitly account for Transformer nonlinear operations.
  • Key findings: Transformer hardware design differs from CNN design: increasing Gemmini accumulator sizes to enable higher output reuse yielded significant performance improvement.The paper cautions that accelerator choices effective for CNNs may not transfer directly to Transformers.
  • Key findings: Matmul scheduling has three loops rather than convolution scheduling’s six, yet similar decision-point counts and performance variation can make it equally challenging.The apparent structural simplicity of matmul scheduling does not eliminate mapping difficulty.
  • Key findings: Fusing LayerNorm with preceding matmuls constrains mapping, particularly tile sizes, so fusion requires case-by-case evaluation rather than being assumed beneficial.The paper identifies operation fusion as a design trade-off in Transformer mappings.
  • Case-study outcomes: 88.7× EDP improvement was achieved without a noticeable performance drop versus a naive implementation lacking full-stack considerations.Additional case studies reported 39.6× latency reduction from on-chip nonlinear operations and memory-hierarchy rebalancing, plus NAS-driven EDP reductions of 2.24× or 10.56× under different perplexity trade-offs.

A.1 Decoder Model Architecture

The Transformer decoder’s computation is organized around MHA and FFN modules and is primarily composed of matrix-vector operations for each decoder iteration. Decoder inference also reuses cached keys and values from earlier token-generation iterations.

  • Decoder model architecture: Decoder computations are primarily matrix-vector operations organized within the MHA and FFN modules.This contrasts with the predominance of matrix-matrix operations commonly associated with encoder processing.
  • Decoder model architecture: Decoder inference caches keys and values from previous token-generation iterations.The cache is one of the main computational differences between Transformer decoder and encoder execution.
  • Decoder model architecture: The decoder computation map covers one decoder block and the lth iteration.The figure presents the computation performed during a single iterative decoding step.

A.2 High-Level Overview of CNN Architecture

CNN architectures combine convolutions with nonlinear activation, normalization, residual, and pooling operations. Their convolutional layers apply kernels across inputs to generate output channels, while pooling supports downsampling.

  • CNN architecture: ResNet-50 uses residual additions, stride-2 operations in first blocks of each type, and optional 1×1 projections to match residual dimensions.The architecture diagram omits ReLU, BatchNorm, and Softmax layers for simplicity.
  • Convolution: A 2D convolution slides kernels across an input matrix, with each kernel producing a separate output channel.Stride specifies the spacing between successive filter applications; stride 2 applies the kernel to every second input region.
  • Supporting operations: CNNs use ReLU, BatchNorm, and pooling alongside convolutions.ReLU computes max(0,x), while BatchNorm normalizes per channel using training-time statistics and can be fused with a preceding operation during inference.
  • Supporting operations: Average and max pooling downsample inputs using fixed filters that respectively aggregate equal-valued elements or select the maximum element.Pooling applies its filter element-wise to the input.

A.3 Additional Profiling Results

The profiling tables report per-layer FLOPs, memory operations, and arithmetic intensity for BERT and GPT-2 across sequence lengths 128, 512, and 4096. They highlight greater memory traffic and lower arithmetic intensity in GPT-2 than in the BERT encoder.

  • Profiling setup: Tables 10 and 11 profile per-layer FLOPs, MOPs, and arithmetic intensity at sequence lengths 128, 512, and 4096.Table 10 covers a hypothetical four-head BERT encoder, while Table 11 covers GPT-2.
  • BERT profiling: The hypothetical four-head BERT encoder has FLOPs per operation similar to the 12-head BERT-Base encoder, but lower activation-to-activation matmul MOPs.The comparison is reported across the listed sequence lengths.
  • GPT-2 profiling: GPT-2 has FLOPs similar to the BERT-Base encoder but substantially more MOPs, resulting in lower arithmetic intensity.The profiling comparison is reported for sequence lengths 128, 512, and 4096.

A.4 Additional ResNet-50 Workload Analysis

Table 12 analyzes convolutional layers in ResNet-50 using FLOPs, memory operations, and arithmetic intensity.

  • Table 12 reports FLOPs for different convolutional layers in ResNet-50.
  • Table 12 reports memory operations for different convolutional layers in ResNet-50.
  • Table 12 reports arithmetic intensity for different convolutional layers in ResNet-50.

A.5 Additional Analytical Modeling Results

The analytical results model Transformer latency and computation breakdowns under explicit hardware and operation-fusion assumptions. They show that FLOP counts, nonlinear operations, and fusion choices can substantially affect runtime comparisons.

  • The model analyzes BERT-Base, BERT-Large, and GPT-2 latency breakdowns under square tiling and no operation fusion.Each operation is assumed to read inputs from external memory and flush outputs.
  • ResNet-50 runs 1.28 times faster than BERT-Base at sequence length 128, despite having 3.07 times fewer FLOPs.The comparison demonstrates that FLOP differences do not necessarily predict runtime latency.
  • 32.4% of ResNet-50 runtime comes from nonlinear operations without fusion, although convolutions account for 99.3% of FLOPs.
  • 1.32 times speedup is obtained by fusing BatchNorm and ReLU with prior convolutional layers in ResNet-50.For Transformers, fusing LayerNorm or Softmax may require tiling changes that increase runtime latency.

A.6 Acronyms and Abbreviations

Table 13 provides the full names of acronyms and abbreviations used throughout the paper.

  • Table 13 summarizes acronyms and abbreviations used throughout the paper.
  • The table supplies the full names corresponding to the paper’s acronyms and abbreviations.
  • Table 13 functions as a reference for terminology used across the paper.
Loading 2302.14017v1…