PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Transformers

Tianyang Lin, Yuxin Wang, Xiangyang Liu, Xipeng Qiu

arXiv:2106.04554v2cs.LGcs.AIcs.CL

TL;DR

The survey addresses the missing systematic and comprehensive review of diverse Transformer variants and the challenges motivating their development. It proposes a taxonomy and reviews X-formers by architectural modification, pre-training, and applications, while outlining future research directions.

  • Problem

    A systematic and comprehensive literature review of diverse Transformer variants is missing, amid challenges including long-sequence computation and memory complexity.

  • Method

    The survey proposes a taxonomy of X-formers and reviews them through architectural modification, pre-training, and applications, focusing mainly on general architecture variants.

  • Results

    The survey provides a comprehensive overview of X-formers, whose improvements address efficiency, generalization, structural priors, lightweight architecture, pre-training, and applications.

  • Takeaways & Limitations

    The review organizes existing Transformer improvements and identifies further opportunities for improving efficiency and global interaction mechanisms beyond full attention.

  • Takeaways & Limitations

    X-formers still face challenges, including efficiency and generalization, and further improvements remain an open direction.

Abstract

from arXiv · show

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

1 INTRODUCTION

Transformers have become widely adopted across AI fields, prompting many variants that improve the vanilla architecture in different ways. This survey organizes X-formers by architectural modification, pre-training, and applications.

  • Transformers are widely used in natural language processing, computer vision, speech processing, chemistry, and life sciences.
  • Transformer variants, called X-formers, have been proposed to improve the vanilla Transformer from different perspectives.
  • Model Efficiency: Long-sequence processing is inefficient mainly because self-attention has high computation and memory complexity.
  • Model Generalization: Limited structural assumptions make Transformers difficult to train on small datasets, motivating structural bias, regularization, and pre-training.
  • Survey Scope: The survey reviews X-formers comprehensively and proposes a taxonomy based on architecture modification, pre-training, and applications.

2 BACKGROUND

The vanilla Transformer is a sequence-to-sequence encoder-decoder built from attention, feed-forward, residual, normalization, and positional components. Its flexible self-attention supports parallel long-range modeling, but its quadratic sequence-length costs create a bottleneck for long inputs.

  • Vanilla Transformer: The vanilla Transformer uses encoder and decoder stacks of identical blocks, with decoder cross-attention and masked self-attention preventing access to future positions.
  • Attention Modules: Multi-head attention projects queries, keys, and values through multiple learned projections, concatenates the head outputs, and projects them back.
  • Attention Types: Self-attention sets Q = K = V, while cross-attention uses decoder queries and encoder-derived keys and values.
  • FFN and Normalization: Position-wise FFNs operate separately and identically at each position, while residual connections and Layer Normalization support deeper Transformer models.
  • Inductive Bias: Transformers lack recurrence and convolution, so positional representations are needed to model token ordering.
  • Architectural Forms: A Transformer can be used as an encoder-decoder, encoder-only, or decoder-only architecture for sequence-to-sequence, classification, labeling, or generation tasks.
  • Model Analysis: For long sequences, self-attention becomes the bottleneck because computation requires storing a T × T attention distribution matrix.
  • Self-Attention Properties: Self-attention has constant sequential operations and maximum path length, enabling parallelization and long-range dependency modeling.

3 TAXONOMY OF TRANSFORMERS

The survey categorizes Transformer variants from three perspectives—architecture modification, pre-training, and applications—and focuses primarily on architecture modifications.

  • Transformer variants are organized by architecture modification, pre-training methods, and applications.
  • The survey focuses on architecture modifications, especially attention-related variants and other module- and architecture-level changes.
  • Pre-training methods and applications are reviewed briefly, alongside representative models in the taxonomy.

4 ATTENTION

The survey presents attention variants that address self-attention’s long-sequence complexity and limited structural bias through sparse, linearized, compressed, prior-based, and multi-head approaches.

  • Motivation: Self-attention becomes a bottleneck for long sequences because its complexity is O(T^2 · D) and it requires storing a T × T attention matrix.
  • Overview: Transformer attention variants address complexity and structural-prior challenges through sparse attention, linearized attention, query or memory reduction, attention priors, and improved multi-head mechanisms.
  • Sparse Attention: Position-based sparse attention restricts query-key connections using predefined patterns that can be decomposed into atomic patterns and composed into larger structures.
  • Sparse Attention: Atomic sparse patterns include global, band, dilated, random, and block-local attention, each imposing a different connection structure.
  • Sparse Attention: Compound sparse models combine atomic patterns, such as band-plus-global attention in Star-Transformer and Longformer, with additional dilation or random attention in some models.
  • Sparse Attention: Sparse-attention extensions are adapted to data types through factorized patterns for periodic or non-periodic data and specialized structures for text and vision.

4.2 Linearized Attention

Linearized attention replaces or approximates the unnormalized attention matrix with feature-map products, enabling reversed-order computation and linear sequence-length complexity.

  • Linearized Attention: Standard attention has quadratic sequence-length complexity, while disentangling attention into Q′K′⊤ enables computation in reversed order with O(T) complexity.
  • Linearized Attention: Linearized attention approximates exp(QK⊤) with ϕ(Q)ϕ(K)⊤ and computes ϕ(Q)(ϕ(K)⊤V) to avoid explicitly forming the full attention matrix.
  • Linearized Attention: The formulation aggregates feature-mapped key-value outer products into a memory matrix and retrieves values using feature-mapped queries.
  • Feature Maps: Linear Transformer uses ϕ_i(x) = elu(x_i)+1 and is empirically reported to perform on par with the standard Transformer.
  • Feature Maps: Performer uses random feature maps, while RFA and related work also approximate the order-1 arc-cosine kernel with ReLU feature maps.
  • Feature Maps: Trigonometric random features may produce unstable behavior because they do not guarantee non-negative attention scores; positive random features address this issue.
  • Memory Updates: Gating and write-and-remove updates modify memory aggregation by decaying older associations or selectively replacing them.

4.3 Query Prototyping and Memory Compression

Query prototyping and memory compression reduce attention complexity by decreasing the number of queries or key-value pairs processed by attention.

  • Query Prototyping: Query prototyping computes attention distributions for representative query prototypes and transfers or approximates them for other query positions.
  • Query Prototyping: Clustered Attention groups queries and lets members share the distribution computed from their cluster centroid.
  • Query Prototyping: Informer selects the top-u queries using an explicit sparsity measurement and assigns the remaining queries discrete uniform distributions.
  • Memory Compression: Memory compression reduces the number of keys and values before attention, providing an alternative route to lower complexity.
  • Memory Compression: Memory Compressed Attention uses strided convolution to reduce keys and values by a factor of kernel size k, enabling longer sequences under fixed computation resources.
  • Memory Compression: External trainable global nodes summarize inputs into compressed memory, reducing self-attention complexity to linear in sequence length.
  • Memory Compression: Linformer projects keys and values from length n to n_k, reducing complexity to linear but requiring an assumed input sequence length and excluding autoregressive attention.

4.4 Low-rank Self-Attention

Low-rank self-attention variants exploit the attention matrix’s low-rank structure either through explicit parameterization or approximation. These approaches aim to encode long-range interactions while reducing computational complexity.

  • Low-rank structure: The self-attention matrix is often low-rank, enabling explicit low-rank parameterization or replacement by a low-rank approximation.Its rank is reported to be far lower than the input sequence length.
  • Explicit parameterization: Limiting D_k can impose a low-rank inductive bias and support a separate band-attention component.This design targets short-input over-parameterization while preserving long-range non-local interactions.
  • Approximation methods: Kernel-inspired methods approximate attention using random feature maps, as in Performer’s approximation of a Gaussian kernel matrix.Performer decomposes the attention distribution matrix into C_Q G C_K and approximates G with random Fourier features.
  • Approximation methods: Nyström-based methods select m landmark nodes from the inputs and use landmark queries and keys to approximate attention.Landmarks can be selected using down-sampling methods such as strided average pooling.
  • Approximation methods: CSALR adds an identity matrix when inversion may fail, whereas Nyströmformer uses the Moore-Penrose pseudoinverse for singular matrices.Both modifications address cases where the matrix inverse required by the approximation does not exist.

4.5 Attention with Prior

Attention with prior supplements or replaces input-generated attention distributions using externally specified or previous-layer information. X-formers implement this idea through locality priors, cross-layer reuse, task-conditioned adapters, and input-independent distributions.

  • General formulation: Attention with prior combines or replaces input-generated attention distributions with distributions from other sources.The fused scores are then used for attention computation.
  • Locality priors: Gaussian priors encode locality by multiplying generated attention with a position-based density, equivalently adding a Gaussian bias to attention scores.Higher bias values indicate higher prior probability for attending between positions.
  • Locality priors: Gaussian-prior variants predict or assume central positions, with standard deviation controlled by a hyperparameter or predicted from inputs.Gaussian Transformer instead assumes each query’s central position equals its sequence position and uses scalar parameters controlling deviation and central-position weighting.
  • Cross-layer priors: Previous-layer attention can serve as a prior because adjacent Transformer layers often have similar attention distributions.A function transforms prior scores before combining them with current-layer scores using learned weights.
  • Cross-layer priors: Predictive Attention Transformer convolves previous scores before convex combination, while Realformer adds them directly as an attention-map residual connection.Both report improvements over baseline models, and Realformer remains stronger with substantially lower pre-training budgets.
  • Cross-layer priors: Lazyformer shares attention maps across adjacent layers, computing them once and reusing them to reduce computation while remaining effective.Its formulation alternates which adjacent layer supplies the shared map.
  • Task-conditioned priors: CAMTL uses a task-conditioned trainable block-diagonal prior as an adapter for parameter-efficient multi-task inductive knowledge transfer.The prior is added to attention scores in upper layers of pre-trained Transformers.
  • Input-independent priors: Input-independent attention variants use uniform, Gaussian, learnable, or query-conditioned scores without requiring pairwise input interactions.Average Attention Network retains parallel training with cumulative averages, while Synthesizer variants achieve competitive performance with vanilla Transformer.

4.6 Improved Multi-Head Mechanism

Improved multi-head mechanisms address insufficient head diversity, restricted attention spans, and limited aggregation by adding interaction, regularization, routing, or sharing mechanisms. These variants also use span restrictions to encode locality and improve efficiency.

  • Head diversity and interaction: Vanilla multi-head attention lacks explicit guarantees that heads learn distinct behaviors or mechanisms for interaction.This motivates methods that guide head diversity and communication.
  • Head diversity and interaction: Disagreement regularization encourages head diversity by separating input subspaces, output representations, and attended positions.The method uses cosine-distance objectives and element-wise attention-matrix products.
  • Head diversity and interaction: Talking-head Attention projects scores between head spaces around softmax to enable learnable information exchange among heads.The mechanism maps from h_k heads to h heads, then to h_v heads for value aggregation.
  • Restricted spans: Restricted attention spans impose locality and can scale to long sequences without additional memory or computational cost when implemented appropriately.Masks multiply attention values by a distance-dependent function before renormalization.
  • Restricted spans: Adaptive-span models learn attention ranges, outperform baselines in character-level language modeling, and use significantly fewer FLOPS.Lower layers generally learn smaller spans, while higher layers learn larger spans, indicating hierarchical feature composition.
  • Restricted spans: Multi-Scale Transformer assigns fixed maximum spans by layer and head, with larger scales generally used in higher layers.The span design follows linguistic intuition and empirical observations from BERT.
  • Head aggregation: Routing-based variants replace simple summation by iteratively transforming head outputs into input and output capsules.The resulting capsules are concatenated as the final multi-head output.
  • Efficient multi-head variants: Multi-query attention shares key-value pairs across heads, reducing decoding memory bandwidth and improving decoding speed with minor quality degradation.It uses one key projection and one value projection for all attention heads.

5 OTHER MODULE-LEVEL MODIFICATIONS

This section reviews modifications to positional representations, normalization, and feed-forward layers that address Transformer limitations or improve efficiency and training. The surveyed variants alter positional encoding, layer-normalization placement or replacement, and FFN capacity or computation.

  • 5.1 Position Representations: Transformer position representations range from absolute sinusoidal or learned embeddings to relative, hybrid, and content-position formulations.Absolute encodings are added to token embeddings, while relative methods represent pairwise positional relationships and hybrid methods combine absolute and relative information.
  • 5.1 Position Representations without Explicit Encoding: Masked self-attention is not permutation equivariant, and language-modeling results report that removing explicit position encodings can improve performance.This property allows decoder-only models to sense positional information through the causal attention mask.
  • 5.2 Placement of Layer Normalization: Pre-LN Transformers avoid the large output-layer initialization gradients associated with unstable post-LN training, allowing the warm-up stage to be removed.Post-LN often performs better after convergence, while residual-dependency control can improve its convergence and performance relative to pre-LN.
  • 5.2 Substitutes of Layer Normalization: Normalization substitutes include a parameter-efficient spherical projection and ReZero, which improves dynamic isometry and accelerates convergence.The spherical projection uses a learnable scalar radius and is effective on machine-translation datasets, especially in low-resource settings.
  • 5.3 Feed-Forward Layer: FFN modifications increase capacity through sparsely gated experts, reduce computation by routing each token to one expert, or remove decoder FFNs with slight or no performance loss.Switch Transformer reports faster pre-training with similar FLOPS, while decoder-FFN removal improves training and inference speed.

6 ARCHITECTURE-LEVEL VARIANTS

Architecture-level X-formers modify Transformer beyond individual modules to improve efficiency, information flow, or input-dependent computation. Representative designs restructure layers, connect representations across depth, adapt computation per input, or process long sequences recurrently.

  • Architecture-level variants: Architecture-level variants modify Transformer beyond individual modules, targeting lightweight computation, deeper-model optimization, adaptive computation, or long-sequence processing.The survey distinguishes higher-level architectural changes from module-level modifications.
  • Lightweight architectures: Lite Transformer uses parallel long-range attention and local depth-wise convolution with linear layers to reduce model size and computation.Its two branches separately capture long-range contexts and local dependencies.
  • Lightweight architectures: Funnel Transformer progressively pools sequence representations and later upsamples them, reducing FLOPs and memory relative to the vanilla encoder.The same resources can therefore support a deeper or wider model.
  • Information flow: Transparent Attention connects each decoder cross-attention module to encoder representations from all layers, shortening paths to the error signal.The design is intended to ease optimization in deeper encoder-decoder Transformers.
  • Adaptive computation: Adaptive computation methods allocate processing dynamically: hard examples receive feature refinement, while easy examples can use reduced computation.CCT uses gating and an auxiliary loss to align practical computation with a computation budget.
  • Recurrent and hierarchical Transformers: Recurrent Transformers cache prior-segment information, while Transformer-XL reuses cached representations to extend context by L × N_mem.Compressive Transformer additionally stores compressed older activations in a second memory level.
  • Alternative architectures: Sandwich Transformer improves perplexity on multiple language-modeling benchmarks without increasing parameters, memory, or training time by placing attention lower and FFN modules higher.MAN instead adds token-, distance-, and head-conditioned dynamic masking, consistently outperforming its baseline in translation and summarization.

7 PRE-TRAINED TRANSFORMERS

Transformer pre-training addresses limited-data generalization by learning representations from large unlabeled corpora before downstream fine-tuning. The survey covers encoder-only, decoder-only, and encoder-decoder designs, along with architectural variants used inside pre-trained models.

  • Pre-training motivation: Pre-training on large corpora learns universal language representations that can benefit downstream tasks and enables fine-tuning instead of training from scratch.The surveyed objectives are self-supervised, such as predicting masked words from context.
  • Pre-trained architectures: Encoder-only models such as BERT support language understanding using masked language modeling and next sentence prediction, while RoBERTa removes the latter objective.The survey notes that RoBERTa removes NSP because it was found to hurt downstream performance.
  • Pre-trained architectures: Encoder-decoder models such as BART and T5 support both language understanding and generation through denoising-style pre-training and task-specific text prefixes.BART extends BERT’s denoising objective to an encoder-decoder architecture.
  • Architectural variants in PTMs: Transformer architecture variants also appear in pre-trained models, including BigBird’s sparse attention, GPT-3’s alternating dense and locally banded sparse attention, and Switch Transformer’s mixture-of-experts FFNs.These examples connect architectural modifications with pre-training configurations.

8 APPLICATIONS OF TRANSFORMER

Transformer began as a machine-translation model but has expanded across language, vision, audio, and multimodal applications. The survey presents representative tasks in each domain and links broad adoption to the architecture’s flexibility and language-model pre-training.

  • Overview: Transformer was originally designed for machine translation but is now widely used in computer vision and audio processing because of its flexible architecture.The survey frames this expansion as extending beyond NLP.
  • Natural Language Processing: NLP applications include machine translation, language modeling, and named entity recognition, with large-scale text pre-training identified as a major reason for wide adoption.The survey lists both general Transformer and variant-based NLP work.
  • Computer Vision: Vision applications include image classification, object detection, image generation, and video processing.The survey points readers to dedicated reviews for further coverage of visual Transformers.
  • Audio Applications: Audio applications include speech recognition, speech synthesis, speech enhancement, and music generation.
  • Multimodal Applications: Multimodal applications include visual question answering, visual commonsense reasoning, caption generation, speech-to-text translation, and text-to-image generation.

9 CONCLUSION AND FUTURE DIRECTIONS

The survey organizes X-formers into a taxonomy spanning efficiency, generalization, and applications, while identifying unresolved theoretical, interaction-mechanism, and multimodal challenges. It is intended as a reference for understanding current progress and improving Transformers across applications.

  • Conclusion: The survey provides a comprehensive overview and new taxonomy of X-formers, covering structural priors, lightweight architectures, pre-training, and related improvements.
  • Future directions: Despite X-formers’ success across tasks, the survey states that challenges remain beyond current concerns such as efficiency and generalization.
  • Theoretical Analysis: The theoretical reason for Transformer’s capacity and flexibility relative to CNNs and RNNs remains unclear, motivating theoretical analysis.
  • Global Interaction Mechanisms: Because full attention calculates interactions for all nodes despite being unnecessary for many, future work may seek more efficient global interaction mechanisms beyond attention.The survey mentions dynamic routing and memory-enhanced models as alternatives.
  • Multimodal Frameworks: A unified multimodal framework could capture connections among text, image, video, and audio, but its intra-modal and cross-modal design remains an open issue.
  • Conclusion: The survey aims to serve as a hands-on reference for understanding Transformer research and further improving Transformers for varied applications.
Loading 2106.04554v2…