PaperlayerView exact revision on arXiv

Source-linked AI summary

A Practical Survey on Faster and Lighter Transformers

Quentin Fournier, Gaétan Marceau Caron, Daniel Aloise

arXiv:2103.14636v2cs.LG

TL;DR

Transformers achieve strong sequence-modeling performance through attention but incur quadratic computational and memory costs, creating a need for more efficient designs. This survey reviews general efficiency techniques and lower-complexity Transformer variants, explaining their assumptions, strengths, and limitations. It recommends practical strategies for selecting and training efficient models.

  • Problem

    Transformer efficiency methods offer different trade-offs, but limited understanding and inconsistent evaluation make it difficult to determine which approach fits a task.

  • Method

    The survey examines general methods and lower-complexity Transformer variants, discussing their assumptions, strengths, limitations, and practical use.

  • Results

    The survey advises mixed-precision and gradient checkpointing, recommends pre-trained models when possible, and otherwise suggests a small vanilla Transformer for identifying task dependencies.

  • Takeaways & Limitations

    Efficient Transformers may improve performance, expand applications, increase equity between researchers, and potentially reduce environmental impact.

  • Takeaways & Limitations

    Practical efficiency is not determined by asymptotic complexity alone because hidden constants and hardware limitations can make theoretically efficient methods slower.

Abstract

from arXiv · show

Recurrent neural networks are effective models to process sequences. However, they are unable to learn long-term dependencies because of their inherent sequential nature. As a solution, Vaswani et al. introduced the Transformer, a model solely based on the attention mechanism that is able to relate any two positions of the input sequence, hence modelling arbitrary long dependencies. The Transformer has improved the state-of-the-art across numerous sequence modelling tasks. However, its effectiveness comes at the expense of a quadratic computational and memory complexity with respect to the sequence length, hindering its adoption. Fortunately, the deep learning community has always been interested in improving the models' efficiency, leading to a plethora of solutions such as parameter sharing, pruning, mixed-precision, and knowledge distillation. Recently, researchers have directly addressed the Transformer's limitation by designing lower-complexity alternatives such as the Longformer, Reformer, Linformer, and Performer. However, due to the wide range of solutions, it has become challenging for researchers and practitioners to determine which methods to apply in practice in order to meet the desired trade-off between capacity, computation, and memory. This survey addresses this issue by investigating popular approaches to make Transformers faster and lighter and by providing a comprehensive explanation of the methods' strengths, limitations, and underlying assumptions.

1 INTRODUCTION

RNNs process variable-length sequences but struggle with fixed-size representations, long-term dependencies, and longer sequences. The Transformer addresses these limitations with attention, while creating substantial computational and memory costs that motivate efficiency methods.

  • RNNs iteratively construct hidden representations and outputs for variable-length input sequences using shared parameters.
  • Sequence-to-sequence models use an encoder’s fixed-size context as the decoder’s initial state to support input-output sequences of different lengths.
  • Inter-attention replaces the single-context bottleneck with a weighted sum of encoder hidden representations.
  • LSTMs have an estimated relative effective context length of approximately 400 words, while distant positions are remembered only vaguely.
  • The Transformer relates arbitrary input positions through attention, improving results across language, vision, speech, and biological sequence tasks.
  • The Transformer’s quadratic computational and memory complexity restricts training, fine-tuning, and usable sequence lengths.

2 TRANSFORMER

The Transformer uses scaled dot-product self-attention within stacked encoder and decoder layers. Full attention compares every pair of sequence positions, producing quadratic cost that limits affordability and scalability.

  • Attention combines values according to compatibility scores between queries and keys, with QK^T computing all pairwise dot products.
  • Scaled dot-product attention divides scores by a factor related to d before applying Softmax to obtain attention weights.
  • Multi-head attention projects queries, keys, and values into h distinct subspaces before combining the resulting heads.
  • Each encoder layer contains self-attention and a position-wise feed-forward network, with residual connections and LayerNorm around the sub-layers.
  • Decoder layers add masked self-attention and cross-attention, while decoder depth may differ from encoder depth.
  • 2.4 Complexity: Full attention requires n^2 computations and memory because every output position can attend to every input position.
  • 2.4 Complexity: Quadratic complexity limits affordable experimentation and prevents applying Transformers to long sequences such as books, videos, and DNA.

3 GENERAL APPROACHES

General approaches reduce Transformer computation, memory, or parameter costs through techniques such as mixed-precision, gradient checkpointing, micro-batching, parameter sharing, pruning, distillation, and architecture search. These methods expose trade-offs between memory, computation, implementation complexity, and model capacity.

  • General methods: Gradient checkpointing stores activations for only selected layers and recomputes others during backpropagation, trading memory use for additional computation.An implementation reports a 10× memory reduction with a 20% increase in computation time.
  • Trade-offs: Memory-saving methods can exchange computation for memory, which is important when memory limits prevent using a model altogether.Gradient checkpointing and reversible layers recompute activations during backpropagation rather than retaining all of them.
  • General methods: Parameter sharing reduces storage by reusing identical parameters, while pruning removes low-saliency weights but can require training a large model first.Unstructured pruning can produce sparse models that are not optimized for modern GPUs and TPUs.
  • General methods: Knowledge distillation trains a smaller student to reproduce a larger teacher’s outputs or internal behavior, with the teacher discarded at inference time.For a fixed parameter budget, distilled networks usually outperform models trained directly on the task.
  • General methods: Mixed-precision stores and computes weights, activations, and gradients with fewer bits to accelerate training and reduce memory consumption.Modern GPUs and TPUs perform at least twice as many half-precision operations as single-precision operations; some methods quantize weights and activations to 8-bit integers.
  • General methods: Micro-batching distributes model layers across accelerators and divides mini-batches into smaller units to reduce communication-related waiting.GPipe enables larger models across accelerators; a 48-layer Transformer trained across 8 TPUs achieved 4.8 times higher throughput with 32 micro-batches.
  • Trade-offs: Micro-batching and mixture-of-experts allow large models to be trained on many weaker accelerators, but both require complex implementations.The methods offer an alternative to relying on powerful and expensive accelerators.
  • General methods: Neural architecture search can design faster Transformers, but handcrafted architectures may achieve similar goals with much lower search cost.One search made miniBERT 1.7× faster with a performance drop smaller than 0.3%, while Lite Transformer required about 14,000× less GPU time than Evolved Transformer in mobile NLP.

4 SPECIALIZED APPROACHES

Specialized Transformer methods reduce quadratic attention costs by exploiting sparse connectivity or factorizing the attention computation. Their benefits depend on the sparsity structure, approximation assumptions, sequence length, and task setting.

  • Sparse attention: Sparse attention reduces complexity by allowing each position to attend only to a subset of positions, using fixed, learned, clustered, or locality-sensitive patterns.The motivation is that Softmax attention concentrates weight on a few positions, while masked values contribute zero after Softmax.
  • Fixed and random sparse patterns: Star-Transformer reduces complexity to linear by restricting attention to adjacent positions while using a single global token to preserve long-term dependencies.The global token, or shared relay node, can attend to every position.
  • Fixed and random sparse patterns: The Sparse Transformer reduces complexity to O(n√n) with strided and fixed attention patterns.The supplied figure caption illustrates strided and fixed connectivity with stride 3.
  • Fixed and random sparse patterns: Cascade attention uses exponentially growing sliding windows, yielding O(n·b·m^l) complexity but approaching full-attention complexity in deep networks.Its window size grows with the number of layers, making the approach better suited to shallow networks.
  • Learned sparse patterns: SparseBERT reaches 91.2% sparsity with an average GLUE score of 80.9%, only 3% below full BERT, using learned task-specific masks with structural constraints.The constraints activate the first and last mask rows and columns and share parameters along diagonals.
  • Practical limitations: Sparse attention does not always produce practical efficiency: unstructured sparsity lacks efficient implementations, while Adaptively Sparse Transformer is 25% slower than the original Transformer.The latter still computes every query-key score, preventing its sparsity from reducing memory and computation.

5 SHORTCOMINGS

The survey identifies limited understanding of self-attention, inconsistent evaluation, and gaps between theoretical and practical efficiency as key shortcomings in efficient Transformer research.

  • Self-attention remains insufficiently understood, including why it works, what it learns, and whether it is interpretable.
  • Different tasks can favor different efficient architectures, as Synthesizer performs well on NLP tasks but vanilla Transformer outperforms it on Long-Range Arena.
  • Long-Range Arena provides five challenging tasks spanning text, images, and mathematical expressions for evaluating long-term dependencies.
  • Comparisons are affected by model size, hyperparameters, implementation, hardware, and general methods used for memory reduction.
  • O(nlogn) asymptotic complexity does not guarantee practical speed, because hidden constants can make Reformer slower than vanilla Transformer on small sequences.

6 BROADER IMPACT OF EFFICIENT TRANSFORMER

Efficient Transformers may broaden access to deep learning, enable applications involving very long sequences, and reduce resource and environmental pressures.

  • Computational resources are expensive and unevenly distributed, limiting who can train massive state-of-the-art models.
  • CUDA-kernel dependencies can make Sparse Transformer and Longformer difficult to implement on TPUs, preventing direct efficiency reporting.
  • Lower-complexity Transformers enable applications on extremely long sequences, including genomics, biology, and minute-long musical composition.
  • Training a Transformer with neural architecture search was estimated to generate up to 284,000 kg of CO2.

7 FUTURE RESEARCH DIRECTIONS

The survey highlights adaptive, hardware-aware sparsity and efficiency-compatible generalization methods as promising directions, while noting that no specialized approach works universally.

  • 6.1 Efficiency and Affordability: No specialized approach has yet improved Transformer efficiency across every task, dataset, and hardware setting.
  • 6.1 Efficiency and Affordability: Sparse attention can reduce computation and memory in proportion to masked positions, but practical gains depend on hardware support.
  • 6.1 Efficiency and Affordability: Promising sparse patterns should be learned from data, adapt to content, be structured for hardware, and include global tokens.
  • 6.2 Generalization Performance: Scaling model size improves reported performance but is resource-expensive and conflicts with affordability.
  • 6.2 Generalization Performance: Combining sparse patterns with conditional computing and independent mechanisms is proposed as a way to address complex tasks without large-scale resources.

8 CONCLUSION

The survey reviews general and lower-complexity techniques for reducing Transformer costs, then offers practical recommendations and identifies broader potential impacts.

  • Transformers achieve state-of-the-art performance in many NLP tasks but incur quadratic memory and computational complexity.
  • The survey advises mixed-precision and gradient checkpointing because of their simplicity and overall benefits.
  • When general techniques are insufficient, the survey reviews lower-complexity Transformer variants and their assumptions and shortcomings.
  • The survey recommends pretrained models when possible, or a small vanilla Transformer with mixed-precision and gradient checkpointing for selecting suitable models.
  • Affordable Transformers may improve performance, expand applications, increase research equity, and potentially reduce environmental impact.

A INTRODUCTION TO MACHINE LEARNING

AI has excelled at rule-based challenges, while tasks humans solve instinctively have proved more difficult because they resist formal description.

  • Early AI research rapidly solved problems that could be described as explicit rules.
  • Chess exemplifies a complex task that AI solved brilliantly despite its difficulty for humans.
  • Tasks humans perform instinctively proved harder for AI because they are not easily expressed formally.

B PRACTICAL GUIDELINES - GENERAL METHODS

The survey presents general efficiency methods before specialized lower-complexity Transformers, with practical recommendations tied to the bottleneck and training or inference phase.

  • General approaches apply to both the original Transformer and lower-complexity alternatives.
  • The guidelines help practitioners choose methods according to the bottleneck and whether it occurs during optimization or inference.
  • The survey emphasizes making Transformers more efficient and affordable, highlighting substantial performance losses and significant drawbacks.
  • Unless otherwise specified, the discussed methods are available in PyTorch and TensorFlow.

B.1 Optimization

Optimization is especially resource-intensive because of iterative training, quadratic attention, and stored intermediates, so the survey emphasizes methods that reduce computation or memory.

  • Optimization is resource-intensive because training is iterative, attention has quadratic complexity, and forward-pass intermediates must remain in memory.
  • Pre-training initializes weights in a favorable region and can help models converge faster.
  • Pre-trained models are mainly available for conventional data and tasks, so other settings require principled initialization and sample-efficient objectives.
  • Gradient checkpointing is recommended first for memory bottlenecks because its memory–computation trade-off is highly adjustable.
  • Gradient checkpointing can interfere with PyTorch Distributed and Data Parallel APIs, causing instability with multiple GPUs.
  • Reversible layers decouple model depth from activation memory by recomputing intermediates, but can accumulate numerical errors and require manual implementation.
  • Parameter sharing reduces memory straightforwardly but lowers model capacity, with the trade-off controlled by how many parameters are shared.
  • Mixture-of-experts and micro-batching may let memory-limited GPUs train Transformers, but both require substantial implementation effort and incur communication costs.

B.2 Inference

For deployment, the survey discusses methods that reduce model size and computation under parameter constraints, while noting implementation considerations.

  • Neural architecture search can identify a model within a deployment parameter budget after training large models during development.
  • Neural architecture search is not part of standard libraries at the time of the survey.
  • Structured pruning and distillation reduce memory and computation with fine-grained control, while reported Transformer performance does not significantly degrade.

B.3 Optimization and Inference

Automatic mixed-precision is presented as a broadly compatible way to reduce memory use and accelerate Transformer computation. Eight-bit quantization is mainly intended for inference and has more limited hardware support.

  • Automatic mixed-precision: Mixed-precision reduces memory use and accelerates computation on modern GPUs.It is compatible with virtually every neural network and can be combined with other efficiency methods.
  • Automatic mixed-precision: Mixed-precision is simple to implement, requiring only a few lines of code in PyTorch and TensorFlow.
  • Quantization: 8-bit quantization primarily targets inference and is less readily available than 16-bit mixed-precision.PyTorch lacked quantized GPU operators when the survey was written, while TensorFlow warned of hardware-dependent deviations.

C PRACTICAL GUIDELINES - SPECIALIZED METHODS

Specialized Transformer alternatives offer different memory, computation, accuracy, and implementation trade-offs. The survey recommends hardware-aware experimentation because no simple universal guideline applies across tasks and settings.

  • Memory efficiency: At least 56% and 88% lower memory use is reported for Linformer, Performer, and Linear Transformer at 1,000 and 4,000 tokens, respectively.These models are presented as better suited to memory-limited environments, consistent with their linear complexity.
  • Computation efficiency: Linformer, Performer, Sinkhorn Transformer, and Linear Transformer are significantly faster than the original Transformer at 4,000 tokens on TPU V3.They perform on par with the original Transformer at 1,000 tokens, but the survey cautions that speedups vary across hardware and implementations.
  • Implementation constraints: The Linformer requires fixed-sized inputs because its projection matrices have dimensions k×n.Sequences therefore must be padded to the largest dataset sequence length.
  • Sparse attention: Sparse models require structured sparsity and carefully selected masks to deliver practical improvements without harming necessary dependencies.The survey notes that unstructured sparsity may be slower than dense computation and recommends inspecting attention activation patterns before selecting a sparse model.
  • Evaluation scope: No modification evaluated across SuperGLUE, XSum, and WebQ was able to improve performance universally.
  • Practical recommendations: The survey finds no simple universal guidelines and recommends using a small vanilla Transformer with mixed-precision and gradient checkpointing as a baseline when the setting is nonstandard.For standard tasks, it recommends consulting existing comparisons and experimenting with already pre-trained models.

E SUMMARY OF THE SPECIALIZED APPROACHES

Table 4 summarizes specialized methods alongside the models associated with them. It serves as a compact reference for organizing these alternatives.

  • Specialized methods: Table 4 summarizes specialized methods.
  • Associated models: Table 4 lists the models associated with the specialized methods.
  • Summary reference: The table provides a summary view of specialized method–model relationships.
Loading 2103.14636v2…