PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Positional Encoding in Language Pre-training

Guolin Ke, Di He, Tie-Yan Liu

arXiv:2006.15595v4cs.CLcs.LG

TL;DR

Standard positional encoding can mix heterogeneous word and position information and treat [CLS] like an ordinary position. TUPE separates these correlations, specializes [CLS], and achieves stronger GLUE performance, including while using 30% of baseline pre-training computational costs.

  • Problem

    Existing positional encodings mix heterogeneous word and position information, while treating [CLS] like ordinary positions may hinder its sentence-level role.

  • Method

    TUPE removes input-layer absolute position addition, computes word and positional correlations separately with distinct projections, and resets positional correlations involving [CLS].

  • Results

    TUPE consistently improves performance across GLUE tasks and can outperform baselines using only 30% of pre-training computational costs.

  • Takeaways & Limitations

    Separating positional and word correlations and untying [CLS] provides a more effective positional-encoding design for language pre-training.

Abstract

from arXiv · show

In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources. It may bring unnecessary randomness in the attention and further limit the expressiveness of the model. Second, we question whether treating the position of the symbol \texttt{[CLS]} the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method called \textbf{T}ransformer with \textbf{U}ntied \textbf{P}ositional \textbf{E}ncoding (TUPE). In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together. This design removes the mixed and noisy correlations over heterogeneous embeddings and offers more expressiveness by using different projection matrices. Furthermore, TUPE unties the \texttt{[CLS]} symbol from other positions, making it easier to capture information from all positions. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method. Codes and models are released at https://github.com/guolinke/TUPE.

1 INTRODUCTION

The paper identifies problems in standard positional encoding and proposes TUPE, which separates word and positional correlations and treats [CLS] differently. Experiments on GLUE show consistent performance improvements and more efficient pre-training.

  • Background: Positional encoding is essential because Transformer components are otherwise invariant to sequence order.Absolute encoding adds a position embedding to each word embedding, while relative encoding incorporates distance-related bias inside self-attention.
  • Problems with Existing Encodings: Adding heterogeneous word and positional embeddings creates mixed correlations, including word-to-position and position-to-word interactions.The paper argues these interactions may be weak and introduce unnecessary randomness into attention.
  • Problems with Existing Encodings: Treating [CLS] like an ordinary word position can bias attention toward early words and hinder whole-sentence understanding.[CLS] summarizes information from all positions for downstream sentence-level tasks.
  • TUPE: TUPE computes word contextual and positional correlations separately with distinct parameters, then combines them in self-attention.It also assigns [CLS] a specialized positional correlation to support global sentence representations.
  • Results: Extensive GLUE experiments and ablations show that TUPE consistently improves performance and can substantially reduce pre-training time.The method is presented as an efficient implementation validated across GLUE tasks.

2 PRELIMINARY

This section introduces self-attention and explains how Transformer models encode sequence position. It contrasts absolute and relative positional encoding, while noting an empirical weakness in some relative-position modifications.

  • Attention Module: Self-attention computes representations by querying key-value pairs and can use multiple heads to attend across representation subspaces.The formulation uses query, key, and value projections, followed by concatenation and output projection across heads.
  • Attention Module: Without positional information, self-attention is permutation-invariant and cannot directly use word order.Positional encoding is therefore needed for sequence-structured language.
  • Positional Encoding: Absolute positional encoding assigns each position a learnable vector and adds it to the word embedding before processing.This lets the Transformer distinguish words occurring at different positions.
  • Positional Encoding: Relative positional encoding models the distance between positions through learnable position-dependent terms in self-attention.Shaw et al. use learnable relative-position embeddings, while T5 simplifies this to shared scalar biases.
  • Positional Encoding: A cited modification that adds relative-position information to values did not improve performance much in experiments.This result concerns the referenced prior positional-encoding approach.
  • Positional Encoding: BERT visualizations show relatively uniform word-to-position and position-to-word correlation matrices.The figure compares word-to-word, word-to-position, position-to-word, and position-to-position correlations.

3 TRANSFORMER WITH UNTIED POSITIONAL ENCODING

TUPE separates word and positional correlations in self-attention and unties [CLS] from ordinary positions, addressing limitations of absolute positional encoding.

  • 3.1 UNTIE THE CORRELATIONS BETWEEN POSITIONS AND WORDS: Absolute positional encoding mixes word and positional information through shared projections, producing word-to-position and position-to-word correlations despite their heterogeneous meanings.The expansion contains four terms: word-to-word, word-to-position, position-to-word, and position-to-position correlations.
  • 3.1 UNTIE THE CORRELATIONS BETWEEN POSITIONS AND WORDS: TUPE computes word and positional relationships separately with different projection matrices, removes the two cross-correlation terms, and adds the resulting correlations in self-attention.The design can also incorporate relative positional encoding by adding complementary positional information.
  • 3.2 UNTIE THE [CLS] SYMBOL FROM POSITIONS: Treating [CLS] like ordinary words may bias some attention heads toward local words, although [CLS] is intended to summarize the whole input for sentence-level prediction.Regular words often have strong local dependencies, whereas [CLS] is used as the sentence representation downstream.
  • 3.2 UNTIE THE [CLS] SYMBOL FROM POSITIONS: TUPE resets positional correlations involving [CLS] using separate learnable parameters for attention from [CLS] and attention to [CLS].The position-only correlation remains unchanged for pairs unrelated to [CLS].
  • 3.3 IMPLEMENTATION DETAILS AND DISCUSSIONS: TUPE combines untied word-position correlations with untied [CLS] handling in TUPE-A, and adds relative positional encoding in TUPE-R.The implementation shares positional projection matrices across layers, while head-specific projections and reset parameters remain distinct.
  • 3.3 IMPLEMENTATION DETAILS AND DISCUSSIONS: Absolute and relative positional encodings can capture complementary information because their correlation matrices form different expressive subspaces.The paper analyzes relative encodings as Toeplitz matrices and absolute positional correlations as projected position matrices.

4 EXPERIMENT

Experiments compare TUPE variants with BERT baselines under BERT-Base and evaluate overall performance, efficiency, learned positional patterns, and individual modifications. TUPE variants improve GLUE performance, converge faster, and benefit from both untied positional-word correlations and untied [CLS].

  • Experimental Setup: Experiments use BERT-Base with 12 Transformer layers, hidden size 768, and 12 attention heads, comparing TUPE-A and TUPE-R against corresponding BERT-A and BERT-R baselines.Pre-training uses English Wikipedia and BookCorpus, with GLUE as the downstream evaluation benchmark.
  • Overall Comparison: 1.38 GLUE average points separate TUPE-R from BERT-R, while TUPE-R exceeds TUPE-A by 0.57 points.TUPE-R is consistently better on almost all tasks, especially MNLI-m/mm, CoLA, and MRPC.
  • Efficiency: 30% pre-training steps suffice for TUPE-A and TUPE-R to achieve better GLUE average scores than the baselines.The variants also achieve smaller validation loss during pre-training, with similar improvements reported for BERT-Large and ELECTRA settings.
  • Positional Correlations: TUPE-A learns five positional attention patterns across 12 heads: global, local, broad, previous-position, and next-position attention.These patterns can be extracted from random initialization, and local dependencies support untying [CLS].
  • Ablation Study: TUPE-A consistently outperforms TUPE-Atie-cls, especially on low-resource tasks such as CoLA and RTE.The ablation isolates the contribution of untying [CLS] from other positions.
  • Ablation Study: TUPE-Atie-cls is 0.17 points better than BERT-Ad and more computationally efficient, indicating benefits from separate projection matrices without word-position correlations.BERT-Ad additionally computes the middle two correlation terms in all layers.
  • Ablation Study: Untying [CLS] helps low-resource tasks, while separating positional and word correlations helps high-resource tasks; combining both yields consistent GLUE improvements.The reported high-resource examples include MNLI-m and MNLI-mm.
  • Positional Correlations: Figure 6 visualizes learned positional correlations by TUPE-A.The visualization supports inspection of the positional attention patterns learned by the model.

5 RELATED WORK

Related work develops relative positional encoding for Transformer attention, with later methods re-parameterizing or simplifying these representations and extending their integration.

  • Relative Positional Encoding: Shaw et al. introduced relative positional encoding for Transformers, and subsequent methods re-parameterized, simplified, or extended that approach.Transformer-XL integrates relative positions through self-attention re-parameterization, T5 uses scalar relative positions, and He et al. adds position-to-position terms.

6 CONCLUSION

The paper concludes that TUPE changes positional encoding by separating word and positional correlations and by treating [CLS] independently. These changes improve GLUE performance and can reduce pre-training computation.

  • Conclusion: TUPE separates positional and word correlations with different projection matrices and unties [CLS] from sequence positions.TUPE removes absolute positional embeddings from Transformer inputs, computes correlations separately in self-attention, and resets correlations involving [CLS].
  • Conclusion: TUPE achieves better GLUE performance and can outperform baselines using only 30% of pre-training computational costs.The conclusion attributes this efficiency to a better inductive bias over positional information.

A PROOF OF PROPOSITION 1

The proof constructs a circulant extension of a Toeplitz matrix, factorizes it spectrally, and recovers the original matrix through block multiplication.

  • Matrix Construction: A 2n × 2n circulant matrix is constructed from the 2n − 1 values defining the Toeplitz matrix B.The construction additionally defines b_-n = b_n = b_0 for reference.
  • Circulant Structure: The proof verifies the extended matrix is circulant using shifted-index equalities and boundary wraparound conditions.The relevant identities establish that neighboring rows and columns shift consistently.
  • Spectral Factorization: The circulant matrix is factorized as B̂ = QDQ*, where Q is a 2n × 2n Vandermonde matrix and D is diagonal.The factorization uses a theorem for circulant matrices and the explicitly defined entries of Q.
  • Recovery of B: Because B is the top-left n × n submatrix of B̂, block matrix multiplication yields B = GDG*.The matrix G is defined within the theorem referenced by the proof.

B EXPERIMENTAL DETAILS

The experiments pre-train and fine-tune models using BERT- and GLUE-based settings, with specified data processing, optimization, evaluation, and positional-correlation rescaling procedures.

  • Pre-training: Pre-training uses English Wikipedia and BookCorpus, concatenated into a roughly 16GB corpus and processed through normalization, tokenization, and BPE.The pipeline also includes document cleaning and sequence preparation procedures.
  • Pre-training: Models are trained with masked language modeling for 1000k steps, batch size 256, maximum sequence length 512, and a 0.15 masking probability.Masked tokens are replaced by [MASK] 80% of the time, random words 10%, and left unchanged 10%.
  • Fine-tuning: Fine-tuning evaluates nine GLUE tasks, reporting Matthews correlation for CoLA, Pearson correlation for STS-B, and accuracy for the remaining tasks.Learning rates are searched separately for each downstream task, while the optimizer and other hyperparameters follow pre-training.
  • Normalization and rescaling: TUPE rescales separate word and positional correlation terms with 1/√d factors to maintain comparable scales after summation.Layer normalization is applied to positional embeddings whenever they are used.

C MORE RESULTS IN THE BERT-LARGE/ELECTRA-BASE SETTINGS

Additional experiments apply TUPE-R to BERT-Large and to both components of ELECTRA-Base, where it outperforms the corresponding baseline methods.

  • Additional settings: TUPE-R improves over the corresponding baselines in both BERT-Large and ELECTRA-Base experiments.The method is integrated into a 24-layer BERT-Large model and into both the generator and discriminator of ELECTRA-Base.

D FAILED ATTEMPTS

The authors report failed positional-encoding variants whose training or performance was unsatisfactory.

  • Failed attempts: Replacing the parametric positional-correlation form with a non-parametric form makes training converge much more slowly than the baselines.The authors also tried parameterizing relative-position bias with projected relative embeddings, but the improvement was only marginal.
Loading 2006.15595v4…