Attention Is All You Need

1 Introduction

• Recurrent models’ sequential hidden-state computation prevents parallelization within training examples, becoming a critical limitation for longer sequences.Memory constraints limit batching across examples. Factorization tricks [21] and conditional computation [32] improve efficiency, but neither removes the fundamental sequential constraint.Source: §1 · p. 2
• Attention models dependencies regardless of sequence distance [2, 19], but is usually combined with recurrent networks, with only a few exceptions.This motivates an architecture that uses attention without recurrence.Source: §1 · p. 2
• Twelve hours on eight P100 GPUs can suffice for the attention-only Transformer to reach new state-of-the-art translation quality while enabling significantly more parallelization.The architecture eschews recurrence and relies entirely on attention to capture global dependencies between input and output.Source: §1 · p. 2

• A constant number of operations connects arbitrary positions in the Transformer, versus linear growth in ConvS2S [9] and logarithmic growth in ByteNet [18] as distance increases.Extended Neural GPU, ByteNet [18] and ConvS2S [9] use convolutions to compute hidden representations in parallel across input and output positions. Distance-dependent operation counts make distant dependencies harder to learn.Source: §2 · p. 2
• Multi-Head Attention counteracts the reduced effective resolution caused by averaging attention-weighted positions in the Transformer.Source: §2 · p. 2
• Self-attention, also called intra-attention, relates different positions within one sequence to compute its representation.Prior successes include reading comprehension, abstractive summarization, textual entailment and task-independent sentence representations [4] [28] [22]. End-to-end memory networks instead use recurrent attention rather than sequence-aligned recurrence, performing well on simple-language question answering and language modeling [34].Source: §2 · p. 2
• To the authors’ knowledge, the Transformer is the first transduction model computing input and output representations entirely through self-attention, without sequence-aligned RNNs or convolution.Source: §2 · p. 2

3 Model Architecture

• The Transformer uses stacked self-attention and point-wise fully connected layers within an encoder-decoder architecture that generates output symbols autoregressively [10].The encoder maps input symbol representations to continuous representations; the decoder consumes these representations and previously generated symbols to produce successive outputs.Source: §3 · pp. 2–3

3.1 Encoder and Decoder Stacks

• 6 identical encoder layers each combine multi-head self-attention with a position-wise feed-forward network.Residual connections [11] wrap each sub-layer, followed by layer normalization [1], yielding LayerNorm(x + Sublayer(x)). All sub-layers and embedding layers produce outputs of dimension dmodel = 512.Source: §3.1 · p. 3
• 6 identical decoder layers add encoder-output attention to the encoder layer structure, using residual connections followed by layer normalization.Masked self-attention and output embeddings offset by one position ensure that predictions at position i depend only on known outputs at positions less than i.Source: §3.1 · p. 3

3.2 Attention

• Attention maps a query and key-value pairs to a weighted sum of values, with weights determined by query–key compatibility.The query, keys, values, and output are all vectors.Source: §3.2 · pp. 3–4

3.2.1 Scaled Dot-Product Attention

• Scaled dot-product attention divides query–key dot products by √dk before softmax weighting of values, counteracting suspected small-gradient effects at large dk.Queries and keys have dimension dk, while values have dimension dv. The authors suspect large dot products push softmax into regions with extremely small gradients.Source: §3.2.1 · p. 4
• Dot-product attention is faster and more space-efficient than additive attention [2] in practice because it uses optimized matrix multiplication.Both have similar theoretical complexity; additive attention [2] computes compatibility with a single-hidden-layer feed-forward network. They perform similarly for small dk, but additive attention outperforms unscaled dot-product attention at larger dk [3].Source: §3.2.1 · p. 4

3.2.2 Multi-Head Attention

• Multi-head attention jointly attends to different representation subspaces and positions, avoiding the averaging limitation of a single head.Different learned projections produce parallel attention outputs, which are concatenated and projected again. With h = 8 heads and dk = dv = dmodel/h = 64, reduced head dimensions keep computational cost similar to full-dimensional single-head attention.Source: §3.2.2 · pp. 4–5

3.2.3 Applications of Attention in our Model

• Encoder self-attention spans all input positions, encoder-decoder attention queries encoder outputs, and decoder self-attention masks future positions.Encoder-decoder queries come from the previous decoder layer, while keys and values come from encoder outputs; encoder self-attention takes all three from the previous encoder layer. Decoder masking sets illegal pre-softmax connections to −∞ to preserve autoregressive generation.Source: §3.2.3 · p. 5

3.3 Position-wise Feed-Forward Networks

• Each encoder and decoder layer supplements attention with a fully connected feed-forward network comprising two linear transformations separated by ReLU.The network is applied to each position separately and identically.Source: §3.3 · p. 5
• Linear transformations share parameters across positions but use different parameters across layers, equivalently forming two convolutions with kernel size 1.Source: §3.3 · p. 5
• 512 is the input and output dimensionality dmodel; 2048 is the inner-layer dimensionality dff.Source: §3.3 · p. 5

• The two embedding layers and the pre-softmax linear transformation share the same weight matrix, similar to [30].Source: §3.4 · p. 5
• Learned embeddings map input and output tokens to dmodel-dimensional vectors, with embedding weights multiplied by dmodel^0.5.Source: §3.4 · p. 5
• A learned linear transformation followed by softmax converts decoder outputs into predicted next-token probabilities.Source: §3.4 · p. 5

3.5 Positional Encoding

• Positional encodings supply sequence-order information by being added to input embeddings at the bottoms of both encoder and decoder stacks.The encodings and embeddings share dimension dmodel, allowing them to be summed.Source: §3.5 · p. 6
• Sine and cosine functions encode positions using wavelengths that form a geometric progression from 2π to 10000 · 2π.For any fixed offset k, PE_pos+k is a linear function of PE_pos; the authors hypothesized this would facilitate learning to attend by relative positions.Source: §3.5 · p. 6
• Learned positional embeddings [9] and sinusoidal encodings produced nearly identical results.The authors chose sinusoids because they might allow extrapolation to sequence lengths longer than those encountered during training.Source: §3.5 · p. 6

• The comparison evaluates self-attention, recurrence, and convolution by per-layer computational complexity, minimum sequential operations, and maximum dependency-path length.Shorter paths for forward and backward signals make long-range dependencies easier to learn.Source: §4 · p. 6
• O(n) sequential operations are required by recurrence, versus a constant number for self-attention; self-attention is computationally faster when sequence length n < representation dimensionality d.The n < d condition commonly holds for sentence representations used in machine translation.Source: §4 · pp. 6–7
• O(n/r) maximum path length would result from restricting self-attention to neighborhoods of size r, a proposed computational improvement for very long sequences.This restricted-attention approach is left for future investigation.Source: §4 · p. 7
• O(n/k) contiguous convolutional layers or O(logk(n)) dilated convolutional layers [18] are needed to connect all positions, lengthening dependency paths.Convolutions generally cost k times as much as recurrence; separable convolutions [6] reduce complexity to O(k · n · d + n · d^2). At k = n, this matches self-attention plus a point-wise feed-forward layer.Source: §4 · p. 7
• Individual attention heads learn different tasks, and many appear to reflect syntactic and semantic structure, suggesting potential interpretability benefits.The authors inspect attention distributions and present examples in the appendix.Source: §4 · p. 7

5.1 Training Data and Batching

• About 4.5 million English-German sentence pairs and 36M English-French sentences from WMT 2014 supplied training data, batched by approximate sequence length.English-German used byte-pair encoding [3] with a shared source-target vocabulary of about 37000 tokens; English-French used a 32000 word-piece vocabulary [38]. Each batch contained approximately 25000 source tokens and 25000 target tokens.Source: §5.1 · p. 7

5.2 Hardware and Schedule

• 8 NVIDIA P100 GPUs trained base models for 100,000 steps (12 hours) and big models for 300,000 steps (3.5 days) on one machine.Each training step took about 0.4 seconds for base models and 1.0 seconds for big models.Source: §5.2 · p. 7

5.3 Optimizer

• Adam [20] used a learning rate that increased linearly for 4000 warmup steps, then decreased proportionally to the inverse square root of the step number.Optimizer parameters were β1 = 0.9, β2 = 0.98, and ϵ = 10^-9.Source: §5.3 · p. 7

5.4 Regularization

• Dropout and label smoothing [36] regularized training, with smoothing improving accuracy and BLEU despite hurting perplexity.Dropout was applied to sub-layer outputs before residual addition and normalization, and to embedding-plus-positional-encoding sums in both stacks; the base-model rate was 0.1. Label smoothing used a value of 0.1.Source: §5.4 · p. 8

6.1 Machine Translation

• 28.4 BLEU on WMT 2014 English-to-German exceeds the best previously reported models, including ensembles, by more than 2.0 BLEU.Training the big model took 3.5 days on 8 P100 GPUs. The base model also surpassed all previously published models and ensembles at a fraction of competitive models’ training costs.Source: §6.1 · p. 8
• 41.0 BLEU on WMT 2014 English-to-French outperforms all previously published single models at less than 1/4 the previous state-of-the-art model’s training cost.The big English-to-French model used dropout Pdrop = 0.1 instead of 0.3.Source: §6.1 · p. 8

6.2 Model Variations

• 0.9 BLEU separates single-head attention from the best setting at constant computation, while using too many heads also reduces quality.These architecture variations were evaluated on the English-to-German development set, newstest2013, without checkpoint averaging.Source: Table 3 · p. 9, §6.2 · p. 9
• Smaller attention keys hurt quality, larger models perform better, and dropout helps prevent overfitting; learned positional embeddings [9] perform nearly identically to sinusoidal encodings.The authors suggest that determining compatibility may benefit from a function more sophisticated than a dot product.Source: §6.2 · p. 9

6.3 English Constituency Parsing

• 40K WSJ training sentences suffice to outperform Berkeley-Parser; overall parsing results surpass previously reported models except Recurrent Neural Network Grammar [8].A four-layer Transformer with dmodel = 1024 was evaluated using WSJ-only training and a semi-supervised setting with approximately 17M sentences [37]. Only a few experiments selected dropout, learning rates, and beam size; other parameters remained unchanged from the English-to-German base model.Source: §6.3 · pp. 9–10, Table 4 · p. 10

7 Conclusion

• The Transformer is the first sequence transduction model based entirely on attention, replacing recurrent encoder-decoder layers with multi-headed self-attention.Source: §7 · p. 10
• The Transformer achieves state-of-the-art results on both WMT 2014 English-to-German and English-to-French translation, outperforming all previously reported ensembles on English-to-German.Training is significantly faster than for architectures based on recurrent or convolutional layers.Source: §7 · p. 10
• Future work targets non-text input and output modalities, local restricted attention for large inputs and outputs, and less sequential generation.Proposed modalities include images, audio, and video; local, restricted attention would aim to handle these efficiently.Source: §7 · p. 10

Attention Visualizations

• Many attention heads in encoder layer 5 of 6 track the long-distance dependency connecting ‘making’ with ‘more difficult’.The visualization displays attention only from the word ‘making’, with different colors identifying different heads.Source: Figure 3 · p. 13
• Two attention heads in encoder layer 5 of 6 appear involved in anaphora resolution, with sharply focused attention from ‘its’.The visualization shows full attention for head 5 and isolates attention from ‘its’ for heads 5 and 6.Source: Figure 4 · p. 14
• Different attention heads exhibit distinct learned behaviors, including sentence-structure-related patterns and apparent anaphora resolution.Two sentence-structure examples come from different encoder self-attention heads in layer 5 of 6.Source: Figure 4 · p. 14, Figure 5 · p. 15