PaperlayerView exact revision on arXiv

Source-linked AI summary

Improve Transformer Models with Better Relative Position Embeddings

Zhiheng Huang, Davis Liang, Peng Xu, Bing Xiang

arXiv:2009.13658v1cs.CL

TL;DR

The paper addresses the limited use of position information in Transformer models, especially the relevance of relative positions. It proposes relative position embeddings with stronger query–key–position interactions and finds improved SQuAD1.1 accuracy, inductive robustness, and efficient adoption for large models.

  • Problem

    Existing position-embedding methods do not fully utilize position information, while absolute positions can be arbitrary when text is processed in independent chunks.

  • Method

    The paper reviews absolute and relative position embeddings and proposes methods that increase interactions among query, key, and relative position embeddings in self-attention.

  • Results

    The best method, a generalization of absolute position embedding, improves accuracy on SQuAD1.1, is empirically robust for longer sequences, and boosts large-model performance with a small computation budget.

  • Takeaways & Limitations

    The proposed relative embedding can serve as an effective and efficient drop-in replacement for improving large Transformer models.

  • Takeaways & Limitations

    Method 3 has a significantly higher training memory footprint, fitting only 2 sequences per GPU versus 20 for another method.

Abstract

from arXiv · show

Transformer architectures rely on explicit position encodings in order to preserve a notion of word order. In this paper, we argue that existing work does not fully utilize position information. For example, the initial proposal of a sinusoid embedding is fixed and not learnable. In this paper, we first review absolute position embeddings and existing methods for relative position embeddings. We then propose new techniques that encourage increased interaction between query, key and relative position embeddings in the self-attention mechanism. Our most promising approach is a generalization of the absolute position embedding, improving results on SQuAD1.1 compared to previous position embeddings approaches. In addition, we address the inductive property of whether a position embedding can be robust enough to handle long sequences. We demonstrate empirically that our relative position embedding method is reasonably generalized and robust from the inductive perspective. Finally, we show that our proposed method can be adopted as a near drop-in replacement for improving the accuracy of large models with a small computational budget.

1 Introduction

The paper argues that existing position embeddings underuse relative position information and introduces methods to increase interactions among query, key, and relative position embeddings. It evaluates their accuracy, inductive robustness on longer sequences, and computational efficiency.

  • Absolute positions can be arbitrary when training uses independent text chunks, making the relative distance j − i between tokens more relevant.
  • The proposed embeddings increase interactions among query, key, and relative position embeddings in self-attention.
  • The proposed embeddings outperform Shaw’s widely used relative position embedding on SQuAD1.1.
  • Ablation studies examine how the clipping value k affects accuracy and show robustness to the inductive challenge of handling longer sequences.
  • The novel position embedding improves BERT-large performance with only a few fine-tuning epochs and a small computation budget.

2 Related Work

Prior work developed sinusoidal, learned, and relative position representations for transformer attention. Relative representations improved translation quality and connect this paper to broader efforts to incorporate location information into attention.

  • The original Transformer uses sine and cosine functions at different frequencies to encode absolute positions.
  • Sinusoidal encodings allow a fixed offset to be represented as a linear function of the original position encoding, while BERT uses learnable position embeddings.
  • Shaw et al. introduced an efficient relative-position mechanism in transformer self-attention and reported significant translation-quality improvements on two machine translation tasks.
  • The paper notes that the hypothesis about sinusoidal encodings was not rigorously verified in experiments.
  • Location-aware attention methods have also been proposed for sequence-to-sequence automatic speech recognition.

3 Position Embeddings

This section reviews absolute and relative position embeddings, introduces a range of increasingly complex relative-position methods, and analyzes their complexity.

  • The paper reviews absolute position embeddings from BERT and relative position embeddings from Shaw et al. and Dai et al.
  • It proposes multiple relative position embeddings ranging from simpler to more complex formulations.
  • The paper analyzes the complexity of each position embedding method.

3.1 Self-Attention review

The Transformer encoder processes sequences through stacked self-attention and point-wise fully connected layers. Each attention head computes weighted representations using softmax-normalized, scaled dot-product attention.

  • BERT uses a Transformer encoder, whose layers contain self-attention and point-wise fully connected components.
  • Each self-attention sublayer uses multiple heads whose outputs are concatenated.
  • Attention weights αij are computed with a softmax function.
  • The attention score eij is a scaled dot product after linear transformations.
  • The parameter matrices WQ, WK, and WV are unique to each layer and attention head.

3.2 Absolute position embedding in BERT

BERT forms each first-layer input by combining token, segment, and learnable absolute position embeddings. This approach requires a preset maximum sequence length and does not have the inductive property, although it is effective when inference length is bounded.

  • BERT sums token, segment, and absolute position embeddings to form each input to the first transformer layer.The embeddings are denoted ti, si, and wi, respectively.
  • Absolute position embeddings model position directly, associating each token with its absolute location rather than a relative distance.
  • The maximum sequence length n must be fixed before training, so this approach lacks the inductive property.It remains effective when inference already enforces the same maximum length.
  • Figure 2 depicts the absolute position embedding used in the first transformer layer.

3.3 Shaw’s relative position embedding

Shaw’s method represents relative positions as edge representations that modify attention between token pairs. A clipping value limits the distance resolution to 2k + 1 distinct relative-position labels.

  • Shaw’s method adds edge representations aij to model how much token ti attends to token tj based on their distance.
  • The clipped distance value k limits the maximum relative position considered and yields 2k + 1 unique edge labels.
  • The authors assume precise relative-position information is unnecessary beyond a certain distance.
  • Figure 3 illustrates the edge representations aij with k = 3.

3.4 XLNet’s relative position embedding

XLNet uses relative sinusoid encodings in attention together with learnable query-bias terms. In the authors’ implementation, the bias terms caused training instability, while removing them produced slightly lower accuracy than Shaw’s method.

  • XLNet revises the attention score to combine content-based attention with location-based attention from relative position encodings.
  • The relative position encoding uses a learnable matrix WR applied to sinusoid vectors Rij between locations i and j.
  • XLNet introduces trainable query-bias terms u and v in addition to the sinusoid formulation.
  • The query-bias terms caused training instability, and removing them left accuracy slightly worse than Shaw’s method.

3.5 Proposed position embeddings

The paper proposes four relative position embedding variants that increase interactions among query, key, and position representations, varying whether distances are signed and whether embeddings are scalar or vector. The methods culminate in a formulation that models all pairwise interactions and generalizes absolute position embeddings.

  • The four variants vary relative-position sign encoding and whether embeddings are scalars or vectors.
  • Method 1 uses a scalar embedding based only on absolute distance |j −i| and introduces multiplicative interaction in attention.It does not distinguish whether one token lies before or after the other.
  • Method 2 uses scalar embeddings that distinguish the sign of distance, assigning different attention weights to previous and future tokens.
  • Method 3 replaces scalar distances with vectors and explicitly multiplies query, key, and relative-position vectors elementwise before summing.The relative position embedding acts as a gate on query–key content similarity.
  • Method 4 models query–key, query–position, and key–position interactions, while its first term generalizes absolute position embeddings.The formulation shares relative-position embeddings across two factors and is presented as enabling more reliable estimation than Shaw’s method.

3.6 Complexity Analysis

The complexity analysis compares parameter counts and runtime storage across position embedding methods. Although the methods add few parameters relative to BERT, vector method 3 has a substantially higher training memory footprint.

  • Shaw’s method has runtime storage complexity O(mhn^2d) because its relative-position parameters are reused in attention computation.
  • Shaw, method 3, and method 4 introduce mh(2n −1)d parameters, while all position methods add only a small number relative to BERT’s 108M parameters.At maximum, the reported example gives 147K additional parameters.
  • Method 3 requires substantially more training memory than the other methods, fitting 2 sequences per GPU versus 20 for the others.The paper attributes this difference to inefficient GPU implementation of summing elementwise vector products compared with matrix multiplication.
  • Shaw’s method and proposed methods 1, 2, and 4 have training and inference speeds similar to the absolute position baseline.

4 Experiments

Experiments compare absolute, Shaw relative, and four proposed position embeddings on GLUE and SQuAD, then test clipping distance, longer fine-tuning sequences, large BERT models, and attention patterns. The proposed methods show their clearest benefits on SQuAD, while GLUE results are largely similar across embeddings.

  • SQuAD evaluation: 90.53 F1 is achieved by method 4 on SQuAD1.1, slightly exceeding method 3 at 90.50 and Shaw’s relative embedding at 89.37.The absolute-position baseline reaches 88.59 F1, while method 1 reaches 87.96 and method 2 reaches 88.86.
  • GLUE evaluation: GLUE results are generally similar across absolute, Shaw, and proposed embeddings, except method 3 reaches 82.86 F1 on MRPC.The paper suggests position embeddings affect complex question answering more than the evaluated GLUE tasks.
  • Clipping distance: Accuracy on SQuAD development remains similar with k ≥32, suggesting relative distances beyond 32 provide only marginal information during training.The SQuAD sequences average 130 tokens in training and 133 in development.
  • Longer sequences: Fine-tuning with maximum sequence length 576 produces the highest reported F1 score, 90.71%, after pre-training method 4 with k = 256.The gain mainly comes from the small percentage of SQuAD sequences containing more than 512 tokens.
  • Large BERT models: Method 4 raises BERT-large SQuAD F1 from 93.15 to 93.55 with negligible increases in parameters and inference latency.Allowing maximum fine-tuning lengths of 576, 640, and 704 produces no additional gain in this large-model setting.
  • Visualization: Method 4’s first-layer attention concentrates on nearby tokens, with nearly zero attention to distant tokens and usually near-zero attention to the token itself.The visualization supports why a small clipping distance can be sufficient and relates self-attention patterns to masked language modeling.

5 Conclusion

The paper proposes relative position embeddings that increase interaction among query, key, and position embeddings, with a best method that generalizes absolute position embeddings. The method improves SQuAD1.1 accuracy, shows inductive robustness, and boosts large-model performance with limited computation.

  • The proposed methods increase interactions among query, key, and relative position embeddings in self-attention.
  • The best method generalizes absolute position embeddings and achieves higher SQuAD1.1 accuracy than absolute and previous relative position embeddings.
  • The relative embedding method is reasonably generalized and robust from the inductive perspective.
  • The proposed technique can serve as an efficient drop-in replacement that boosts large-model performance with a small computational budget.
Loading 2009.13658v1…