PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

arXiv:1706.03762v7cs.CLcs.LG

TL;DR

Sequence transduction models rely on sequential recurrent or convolutional computation, limiting parallelization during training. This paper introduces the attention-only Transformer, which achieves state-of-the-art translation quality on both WMT 2014 tasks while training significantly faster.

  • Problem

    Sequential computation in recurrent sequence models limits parallelization within training examples, especially for longer sequences.

  • Method

    The Transformer replaces recurrent layers with multi-headed self-attention to compute sequence representations and draw global dependencies.

  • Results

    28.4 BLEU on WMT 2014 English-to-German established a new state of the art, while the Transformer also achieved state-of-the-art results on English-to-French.

  • Takeaways & Limitations

    For translation, the Transformer can be trained significantly faster than recurrent or convolutional architectures while achieving state-of-the-art quality.

Abstract

from arXiv · show

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

1 Introduction

The Transformer replaces recurrent sequence modeling with attention alone, addressing the limited parallelization of recurrent computation while drawing global input-output dependencies. It achieves greater parallelization and new state-of-the-art translation quality after as little as twelve hours on eight P100 GPUs.

  • 1 Introduction: Recurrent and gated architectures had been established as state-of-the-art approaches for sequence modeling and transduction, including language modeling and machine translation [7] [2] [5].Subsequent work continued improving recurrent language models and encoder-decoder architectures [38] [24] [15].
  • 1 Introduction: Recurrent models factor computation across sequence positions, making each hidden state depend on the previous state and preventing parallelization within training examples.This limitation becomes more important for longer sequences because memory constraints restrict batching across examples.
  • 1 Introduction: The Transformer eschews recurrence and relies entirely on attention to draw global dependencies between input and output.Attention mechanisms model dependencies regardless of their distance in input or output sequences [2, 19].
  • 1 Introduction: The Transformer enables significantly more parallelization and reaches new state-of-the-art translation quality after as little as twelve hours on eight P100 GPUs.Its design replaces the recurrent component rather than combining attention with a recurrent network.

2 Background

Prior approaches reduced sequential computation with convolutional or recurrent attention mechanisms, while self-attention had already succeeded across several language tasks. The Transformer is presented as the first transduction model using only self-attention, without sequence-aligned recurrence or convolution.

  • Convolutional models: Extended Neural GPU, ByteNet [18], and ConvS2S [9] use convolutions to compute all input and output hidden representations in parallel while reducing sequential computation.Relating arbitrary positions requires operations growing linearly with distance for ConvS2S and logarithmically for ByteNet.
  • Self-attention: Self-attention relates positions within one sequence to compute its representation and had succeeded in reading comprehension, summarization, textual entailment, and sentence representation learning [4] [28] [22].
  • Related attention models: End-to-end memory networks use recurrent attention instead of sequence-aligned recurrence and perform well on simple-language question answering and language modeling.
  • Novelty: The Transformer is described as the first transduction model relying entirely on self-attention, without sequence-aligned RNNs or convolution.The paper motivates self-attention and its advantages over models such as [17] [18] and [9].

3 Model Architecture · 3.1 Encoder and Decoder Stacks · 3.2 Attention

The Transformer replaces recurrent or convolutional sequence transduction with stacked self-attention and point-wise fully connected layers in encoder and decoder stacks. Its attention mechanisms use scaled dot products, multiple parallel heads, and masking to support autoregressive decoding.

  • 3 Model Architecture: The Transformer uses stacked self-attention and point-wise fully connected layers for both encoder and decoder, following an encoder-decoder architecture [5] [2].The encoder maps input symbols to continuous representations, while the autoregressive decoder generates output symbols sequentially [10].
  • 3.1 Encoder and Decoder Stacks: The encoder contains N = 6 identical layers, each combining multi-head self-attention with a position-wise feed-forward network, residual connections, and layer normalization [11] [1].Each sub-layer uses LayerNorm(x + Sublayer(x)).
  • 3.1 Encoder and Decoder Stacks: The decoder contains N = 6 layers with encoder-decoder attention, residual connections, layer normalization, and masked self-attention that prevents access to subsequent positions.The masking preserves autoregressive generation by blocking illegal future-position connections.
  • 3.2 Attention: Scaled Dot-Product Attention computes query-key dot products, divides them by √dk, applies softmax weights, and forms weighted sums of values.Scaling counteracts large dot products that can push softmax into regions with extremely small gradients for large dk.
  • 3.2.2 Multi-Head Attention: Multi-head attention projects queries, keys, and values into multiple learned lower-dimensional subspaces, processes them in parallel, concatenates outputs, and projects them again.This lets the model jointly attend to different representation subspaces and positions, avoiding the averaging limitation of a single head.
  • 3.2.2 Multi-Head Attention: The model uses h = 8 attention heads with dk = dv = dmodel/h = 64, keeping total computational cost similar to full-dimensional single-head attention.The reduced dimensionality of each head offsets the cost of running heads in parallel.
  • 3.2.3 Applications of Attention in our Model: Attention is applied as encoder-decoder attention over all input positions, encoder self-attention over the previous layer, and decoder self-attention over positions up to the current one.Decoder self-attention masks illegal connections by setting their softmax inputs to −∞, preserving left-to-right information flow.

3.3 Position-wise Feed-Forward Networks

Each encoder and decoder layer adds a position-wise feed-forward network alongside attention. It applies two linear transformations with an intervening ReLU, using shared parameters across positions but distinct parameters across layers.

  • 3.3 Position-wise Feed-Forward Networks: Each encoder and decoder layer contains a position-wise feed-forward network applied separately and identically at every position.The network consists of two linear transformations with a ReLU activation between them.
  • 3.3 Position-wise Feed-Forward Networks: The feed-forward network uses the same linear-transformation parameters across positions but different parameters from layer to layer.It can also be described as two convolutions with kernel size 1.
  • 3.3 Position-wise Feed-Forward Networks: The model uses d_model = 512 for input and output representations and d_ff = 2048 for the feed-forward network’s inner layer.

3.4 Embeddings and Softmax

The Transformer uses learned token embeddings and a learned linear transformation with softmax to produce next-token probabilities. It shares one weight matrix across both embedding layers and the pre-softmax transformation, scaling embedding weights by √dmodel.

  • 3.4 Embeddings and Softmax: Learned embeddings map input and output tokens to dmodel-dimensional vectors, while a learned linear transformation and softmax produce predicted next-token probabilities.These components follow the standard sequence transduction setup.
  • 3.4 Embeddings and Softmax: The model shares one weight matrix across its two embedding layers and the pre-softmax linear transformation, following [30].
  • 3.4 Embeddings and Softmax: Embedding-layer weights are multiplied by √dmodel before use.

3.5 Positional Encoding

Because the Transformer lacks recurrence and convolution, it adds positional encodings to encoder and decoder input embeddings to represent token order. The paper uses sinusoidal encodings, whose relative-position structure may support attention and extrapolation beyond training lengths.

  • 3.5 Positional Encoding: Sinusoidal positional encodings add order information to encoder and decoder input embeddings, using the same dimension as embeddings so they can be summed.The encodings are necessary because the model contains neither recurrence nor convolution.
  • 3.5 Positional Encoding: Each encoding dimension is a sinusoid with wavelengths geometrically progressing from 2π to 10000 · 2π.The design was hypothesized to help the model learn attention by relative positions because PEpos+k can be represented linearly from PEpos for fixed k.
  • 3.5 Positional Encoding: Learned positional embeddings [9] and sinusoidal encodings produced nearly identical results, but the sinusoidal version was selected for possible length extrapolation.The comparison is reported in Table 3 row (E).

4 Why Self-Attention

The section motivates self-attention by comparing computational complexity, parallelization, and path length for long-range dependencies against recurrent and convolutional layers. Self-attention offers constant sequential depth, favorable complexity for typical sentence lengths, and potentially interpretable attention patterns.

  • The comparison evaluates layers by per-layer computational complexity, parallelizable computation, and the path length required to learn long-range dependencies.Shorter paths make long-range dependencies easier to learn.
  • Self-attention connects all input-output positions with a constant number of sequential operations, whereas recurrent layers require O(n) sequential operations.
  • Self-attention is faster than recurrent layers when sequence length n is smaller than representation dimensionality d, as commonly occurs with word-piece and byte-pair sentence representations.For very long sequences, restricting attention to a neighborhood of size r can improve computational performance but increases maximum path length.
  • Convolutional layers require O(n/k) contiguous-kernel or O(logk(n)) dilated-convolution layers to connect all positions, and are generally more expensive than recurrent layers by a factor of k.Separable convolutions decrease convolutional complexity.
  • Attention distributions may improve interpretability because individual heads learn different tasks and often exhibit syntactic or semantic behavior.The section notes that these patterns are illustrated and discussed in the appendix.

5 Training

The Transformer was trained on WMT 2014 translation datasets using Adam, a warmup-and-decay learning-rate schedule, and regularization. Training used 8 NVIDIA P100 GPUs, with configurations ranging from 12 hours to 3.5 days, while achieving better BLEU scores than prior state-of-the-art models at lower training cost.

  • Training used about 4.5 million English-German sentence pairs with a shared 37,000-token byte-pair vocabulary and 36 million English-French sentences with a 32,000-token word-piece vocabulary.Sentence pairs were batched by approximate sequence length.
  • On one machine with 8 NVIDIA P100 GPUs, base models trained for 100,000 steps in 12 hours, while big models trained for 300,000 steps over 3.5 days.Base-model steps took about 0.4 seconds; big-model steps took 1.0 second.
  • Optimization used Adam with β1 = 0.9, β2 = 0.98, and ϵ = 10^-9, increasing the learning rate for 4,000 warmup steps before inverse-square-root decay.
  • The Transformer achieves better BLEU scores than previous state-of-the-art models on WMT 2014 English-to-German and English-to-French tests at a fraction of the training cost.
  • Regularization included residual and embedding-position dropout at Pdrop = 0.1 in the base model, plus label smoothing with ϵls = 0.1 that improved accuracy and BLEU despite worsening perplexity.

6 Results

The Transformer achieves state-of-the-art translation results with substantially lower training costs, while its component studies identify important architectural and regularization choices. It also generalizes well to English constituency parsing, outperforming prior systems except the Recurrent Neural Network Grammar.

  • Translation results: 28.4 BLEU on WMT 2014 English-to-German exceeds the best previously reported models, including ensembles, by more than 2.0 BLEU.Training took 3.5 days on 8 P100 GPUs.
  • Translation results: 41.0 BLEU on WMT 2014 English-to-French outperforms all previously published single models at less than one-quarter of the previous state-of-the-art training cost.The English-to-French big model used dropout rate Pdrop = 0.1.
  • Ablation studies: On English-to-German development data, too few or too many attention heads reduce quality, smaller key dimensions hurt, and larger models and dropout improve performance.Single-head attention is 0.9 BLEU worse than the best setting; learned positional embeddings perform nearly identically to sinusoidal encodings [9].
  • English constituency parsing: On English constituency parsing, the Transformer outperforms all previously reported models except the Recurrent Neural Network Grammar [8].It also beats the Berkeley-Parser when trained only on the 40K-sentence WSJ training set.

7 Conclusion

The Transformer is a sequence transduction model based entirely on multi-headed self-attention, replacing recurrent encoder-decoder layers. It trains faster than recurrent or convolutional architectures and achieves state-of-the-art results on both WMT 2014 translation tasks.

  • The model is the first sequence transduction architecture based entirely on attention, replacing recurrent layers with multi-headed self-attention.
  • The Transformer achieves state-of-the-art results on WMT 2014 English-to-German and English-to-French while outperforming all previously reported ensembles on English-to-German.It can also be trained significantly faster than architectures based on recurrent or convolutional layers.
  • Future work will extend attention-based models beyond text, develop restricted attention for large inputs and outputs, and make generation less sequential.The proposed modalities include images, audio, and video.

Attention Visualizations

Attention visualizations show that encoder self-attention heads learn distinct behaviors related to long-distance dependencies, anaphora resolution, and sentence structure.

  • Attention Visualizations: Encoder self-attention heads in layer 5 follow long-distance dependencies, with many attending to the dependency linking ‘making’ and ‘more difficult’.The visualization shows attention for the word ‘making’, with different colors representing different heads.
  • Attention Visualizations: Two layer-5 attention heads appear involved in anaphora resolution, producing especially sharp attention for the word ‘its’.The figure presents full attentions for head 5 and isolated attentions from ‘its’ for heads 5 and 6.
  • Attention Visualizations: Many attention heads exhibit behavior related to sentence structure, and different heads clearly learn different tasks.The examples come from two encoder self-attention heads in layer 5 of 6.
Loading 1706.03762v7…