Source-linked AI summary
Self-Attention with Relative Position Representations
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
TL;DR
The Transformer needs explicit position information because its attention-only structure is otherwise invariant to sequence ordering. This paper extends self-attention with efficiently computed relative position representations and shows improved machine translation performance, while framing the mechanism as relation-aware attention. The authors also identify open questions about whether all relative-position components are necessary for other tasks and about efficient extensions to labeled graphs.
Problem
The Transformer lacks explicit relative or absolute position information in its structure and therefore requires position representations to use sequence order.
Method
The paper extends Transformer self-attention with efficiently implemented relative position representations and casts the result as relation-aware attention over pairwise input relations.
Results
The approach improved translation over the baseline by 0.3 and 1.3 BLEU for English-to-German base and big configurations, and by 0.5 and 0.3 BLEU for English-to-French base and big configurations.
Takeaways & Limitations
Relative position representations improve machine translation performance, while combining them with sinusoidal position encodings produced no additional benefit in these experiments.
Takeaways & Limitations
Whether using relative position representations only for compatibility is sufficient on other tasks remains unresolved.
Abstract
from arXiv · showhide
Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.
1 Introduction
The Transformer lacks built-in sequence-order information because it uses neither recurrence nor convolution. This work adds relative position representations directly to self-attention and reports improved translation quality.
- The Transformer is invariant to sequence ordering unless explicit position information is incorporated.
- Relative position representations provide an alternative to absolute position encodings in the Transformer’s self-attention mechanism.
- The proposed approach improves translation quality on two machine translation tasks even when replacing absolute position encodings.
- The approach extends self-attention to model arbitrary relations between pairs of input elements, including labeled, directed graph relations.
2 Background
The Transformer uses sinusoidal absolute position encodings because its attention-based architecture does not otherwise represent sequence order. Its self-attention computes weighted combinations of transformed inputs using compatibility-based attention weights.
- The Transformer adds sinusoidal position encodings to encoder and decoder inputs before the first layer.
- Sinusoidal encodings were hypothesized to help generalize to sequence lengths unseen during training by supporting attention based on relative position.
- Each attention head maps an input sequence of n elements to an output sequence of the same length.
- Each output element is a weighted sum of linearly transformed input elements, with weights computed using softmax.
- Attention weights are based on a compatibility function comparing pairs of input elements, using scaled dot product for efficient computation.
3 Proposed Architecture
The proposed architecture makes self-attention relation-aware by incorporating pairwise edge representations, including clipped relative distances for sequences. It also provides an implementation that limits the added memory cost and incurs a modest speed reduction.
- 3.1 Relation-aware Self-Attention: The extension models pairwise relationships between input elements as a labeled, directed, fully connected graph.
- 3.1 Relation-aware Self-Attention: Distinct edge representations are used in the output and compatibility computations, and they may be shared across attention heads.
- 3.1 Relation-aware Self-Attention: Relative edge representations are incorporated both into the sublayer output and into compatibility between input elements.
- 3.2 Relative Position Representations: For sequences, relative distances are clipped at maximum absolute value k, yielding 2k + 1 unique edge labels.
- 3.3 Efficient Implementation: Sharing relative position representations across heads reduces storage from O(hn^2da) to O(n^2da), while total self-attention space becomes O(bhndz + n^2da).
- 3.3 Efficient Implementation: The efficient implementation preserves the same model and batch sizes on P100 GPUs while decreasing steps per second by 7%.
4 Experiments
The experiments compare relative-position self-attention with a sinusoidal-position Transformer baseline across WMT translation tasks and model variations. Relative-position representations improve translation quality, while clipping distance and representation ablations probe the method’s behavior.
- Experimental Setup: The evaluation uses WMT 2014 English-to-German and English-to-French translation datasets, with approximately 4.5M and 36M sentence pairs, respectively.
- Experimental Setup: The comparison isolates relative position representations by evaluating them against a baseline Transformer with sinusoidal position encodings.
- Machine Translation: 0.3 and 1.3 BLEU improvements are obtained over the baseline for English-to-German base and big configurations, respectively.
- Machine Translation: 0.5 and 0.3 BLEU improvements are obtained over the baseline for English-to-French base and big configurations, respectively.
- Machine Translation: Adding sinusoidal position encodings to relative position representations provides no observed translation-quality benefit.
- Model Variations: For clipping distance k ≥ 2, BLEU scores show little variation, while multiple encoder layers may propagate precise relative-position information beyond the clipping distance.
- Model Variations: Whether using relative position representations only for determining element compatibility is sufficient remains unresolved for other tasks.
5 Conclusions
The paper extends Transformer self-attention to incorporate relative position information for sequences. This improves machine-translation performance and provides a formulation that can extend to arbitrary directed, labeled graph inputs.
- Conclusions: The proposed self-attention extension incorporates relative position information into sequence modeling and improves machine-translation performance.
- Conclusions: Future work targets arbitrary directed, labeled graph inputs and nonlinear combinations of input and edge representations, with efficient implementation remaining important.