PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding

Yu-An Wang, Yun-Nung Chen

arXiv:2010.04903v1cs.CLcs.AI

TL;DR

The paper addresses the limited analysis of what position embeddings in pre-trained Transformers learn. It analyzes embeddings and their task effects, finding different positional information in encoders and autoregressive decoders, with position use varying across tasks, architectures, and objectives.

  • Problem

    Most pre-trained Transformer work used learned position embeddings without analysis, although different pre-training objectives may learn different position information.

    Source: §1 · p. 1

  • Method

    The paper analyzes BERT, RoBERTa, GPT-2, and sinusoidal position encodings, and uses empirical experiments to examine what positional information they capture.

    Source: §4 · p. 3, §6 · p. 9

  • Results

    Encoders learn local position information described as effective only in masked language modeling, while autoregressive decoders learn absolute positions; position use may vary across tasks, architectures, and objectives.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The study is intended to inform positional-encoding choices or position-model design for target NLP tasks.

    Source: §6 · p. 9

    The encoder-local-position account is an inference tested through skip-position experiments, which assume nearby skips lose local information while leaving absolute position order largely unchanged.

    Source: §4.3.1 · p. 5, §5.2 · p. 7

Abstract

from arXiv · show

In recent years, pre-trained Transformers have dominated the majority of NLP benchmark tasks. Many variants of pre-trained Transformers have kept breaking out, and most focus on designing different pre-training objectives or variants of self-attention. Embedding the position information in the self-attention mechanism is also an indispensable factor in Transformers however is often discussed at will. Therefore, this paper carries out an empirical study on position embeddings of mainstream pre-trained Transformers, which mainly focuses on two questions: 1) Do position embeddings really learn the meaning of positions? 2) How do these different learned position embeddings affect Transformers for NLP tasks? This paper focuses on providing a new insight of pre-trained position embeddings through feature-level analysis and empirical experiments on most of iconic NLP tasks. It is believed that our experimental results can guide the future work to choose the suitable positional encoding function for specific tasks given the application property.

1 Introduction

The paper studies what position embeddings in pre-trained Transformers learn and how they affect NLP tasks. It analyzes embeddings across models and tasks, connecting performance with task properties to inform positional-encoding choices.

1 Introduction

  • Transformers use feature-level positional encoding to represent sequence position, avoiding the time dependency that makes RNNs difficult to parallelize efficiently.The introduction contrasts this with RNNs, whose hidden states represent ordered-sequence information and are updated through backpropagation through time.Source: §1 · p. 1
  • Most pre-trained Transformer work used learned position embeddings without analysis, although different pre-training objectives may learn different position information.The paper presents its work as among the first complete analyses of what learned embeddings capture across pre-trained models.Source: §1 · pp. 1–2
  • The authors analyze learned position embeddings in BERT, RoBERTa, and GPT-2 across text classification, language modeling, and machine translation.They also analyze and explain the embeddings’ meaning and influence from different aspects.Source: §1 · pp. 1–2
  • The study connects positional-embedding performance with task properties to guide choices for target NLP tasks.It examines performance across multiple NLP tasks and analyzes what the embeddings capture.Source: §1 · p. 2

2 Related Work

Related work traces positional encoding from learned embeddings in convolutional seq2seq and sinusoidal encodings in Transformers to relative and other learned representations.

2 Related Work

  • Convolutional seq2seq introduced position embeddings for position-insensitive models, while Transformers proposed a predefined sinusoidal positional encoding (Gehring et al., 2017; Vaswani et al., 2017).The passage describes convolutional seq2seq as a CNN-based encoder-decoder architecture.Source: §2 · p. 2
  • Shaw et al. (2018) proposed relative position representations at the attention level, and Dai et al. (2019) used adaptive relative embeddings with segment-level recurrence.The relative representations address attention values being the same across input positions.Source: §2 · p. 2
  • Wang et al. (2019) extended positional embeddings to complex values and proposed a learnable encoding function beyond simple position-to-embedding mapping.Source: §2 · p. 2

3 Transformer

The section introduces Transformers as position-insensitive self-attention models that require positional information in their inputs. It outlines attention and the encoder-decoder components used in the experiments.

3 Transformer

  • Because Transformer self-attention is position-insensitive, positional information is added to input representations.The architecture is an encoder-decoder sequence-to-sequence model.Source: §3 · p. 2
  • Each token representation adds its word embedding to a positional encoding, which may be learned or predefined.Source: §3 · p. 2
  • Scaled dot-product attention uses query, key, and value matrices; multi-head self-attention concatenates its heads and projects the result into the hidden state.The passage identifies W as a linear projection and Q, K, and V as the query, key, and value matrices.Source: §3 · p. 2
  • Encoder layers stack self-attention and position-wise feed-forward networks, while decoder layers add encoder attention and mask subsequent tokens for autoregressive generation.Source: §3 · pp. 2–3

4 Position Embedding Analysis

The paper analyzes learned position embeddings in Transformer encoders and a decoder alongside sinusoidal encoding. It frames the analysis around what positional information the embeddings capture, including whether they represent absolute positions.

4 Position Embedding Analysis

  • The analysis compares BERT and RoBERTa encoder embeddings, GPT-2 decoder embeddings, and the sinusoidal function.Source: §4 · p. 3

4.1 Do Embeddings Learn the Meaning of Positions?

  • The analysis asks whether learned embedding spaces can represent absolute word positions.It treats position embeddings as mappings from position space P to embedding space X.Source: §4.1 · p. 3

2. Are P and X isomorphic?

The section tests whether learned position embeddings preserve absolute positions and relations, then examines what encoder embeddings capture and how pre-training choices may shape them. Regression, similarity, and eigenvalue analyses distinguish sinusoid and GPT-2 from BERT and RoBERTa, while a simplified RoBERTa experiment relates a missing position-128 gap to a possible effect of batch size.

4.1.1 Absolute Position Regression

  • Sinusoid embeddings reconstruct absolute positions perfectly and GPT-2 has only a small error, whereas BERT and RoBERTa embeddings—especially BERT—show poor absolute-position recovery.Linear regression is evaluated by mean absolute error; nonlinear SVM and MLP models overfit and test worse, suggesting the position information can be modeled linearly.Source: §4.1.1 · p. 3

4.1.2 Relative Position Regression

  • Encoder embeddings capture less information about relative position relations, especially BERT; sinusoid performs better than encoders but has a higher error rate than GPT-2.Because direct distance regression was too difficult, the analysis simplifies the task to classifying the order of pairs of positions.Source: §4.1.2 · p. 4

4.2 What do Transformer Encoders Capture about Positions?

  • The weak absolute- and relative-position results for BERT and RoBERTa leave open what position information Transformer encoders capture instead.The section turns from testing position recovery to investigating the information present in encoder embeddings.Source: §4.2 · p. 4

4.2.1 Position-Wise Cosine Similarity

  • Sinusoid and GPT-2 show periodic similarity patterns, while BERT has nearby-position similarity without clear long-range patterns and a gap at position 128; RoBERTa is similar to BERT with limited nearby patterns.The authors connect BERT’s gap to its two-stage training lengths, 128 then 512, and say first-stage position information does not fully generalize; lighter figure regions indicate higher similarity.Source: §4.2.1 · p. 4, Figure 1 · p. 4

4.2.2 Informativeness of Position Embeddings

  • BERT requires many top eigenvalues to account for a high proportion of total eigenvalues, and RoBERTa requires more than GPT-2 and sinusoid, suggesting encoder embeddings capture more than position alone.The paper proposes that this richer information may be useful in Transformer encoders and says experiments will investigate that assumption.Source: §4.2.2 · pp. 4–5, Figure 2 · p. 4

4.3 What Make the Differences?

  • The section frames the encoder–decoder embedding differences as a question of what makes their learned position embeddings different.It transitions from characterizing the embeddings to examining possible sources of the differences.Source: §4.3 · p. 5

4.3.1 Pre-Training Objectives

  • The paper identifies pre-training objectives as one main reason for encoder–decoder differences: encoders use masked language modeling, while decoders use autoregressive language modeling.The authors infer that bidirectional encoder prediction with sufficient context can rely on nearby tokens, so encoder embeddings may encode local rather than precise absolute positions.Source: §4.3.1 · p. 5

4.3.2 Differences Between BERT and RoBERTa

  • A simplified RoBERTa trained with batch size 256 and shorter blocks resembles BERT’s similarity pattern but lacks its position-128 gap, leading the authors to infer that large-batch pre-training may make encoder position embeddings more robust and clearer.The experiment is simplified because computing resources constrained large-batch experiments; the paper presents the batch-size explanation as an inference.Source: §4.3.2 · pp. 5–6

5 Performance Effect

Position-embedding effects vary across NLP tasks and input lengths. GPT-2 embeddings are favored for longer inputs, while skipping positions particularly harms masked-language-model encoders.

5.1 Text Classification

  • For classification lengths below one standard deviation, GPT-2 and sinusoid remain slightly more accurate; for longer sentences, the encoder–GPT-2 gap is significant.For extremely long sentences, BERT and random initialization perform much worse, though the authors caution that this range may be biased and results may not be robust.Source: Figure 4 · p. 6, §5.1 · pp. 6–7
  • BERT and RoBERTa underperform GPT-2 and sinusoid on most classification datasets; TREC and MPQA show little positional effect, while absolute positions may matter more for longer inputs.The authors attribute the small TREC and MPQA differences to their shorter average lengths.Source: §5.1 · p. 6, Table 3 · p. 7

5.2 Language Modeling

  • The skip-position attack significantly harms masked language models but does not affect the autoregressive model; on Wikitext-103, BERT is especially impaired.The authors infer that encoder embeddings capture nearby positions, with BERT capturing even less position information than RoBERTa.Source: Table 4 · p. 7, §5.2 · pp. 7–8

5.3 Machine Translation

  • In translation, position embeddings are more effective in the encoder than the decoder; BERT embeddings may slightly harm decoder performance.The authors relate the difference to the decoder’s inability to capture source-language order.Source: §5.3 · p. 8
  • Using different pre-trained embeddings in the translation encoder and decoder performs worse, which the authors suggest may reflect divergence between embeddings from different models.The study also concludes that mixing different position embeddings in one model is not suitable.Source: §5.3 · pp. 8–9
  • On the full translation set, the three pre-trained embeddings perform similarly, but GPT-2 is much better than BERT and RoBERTa on longer sentences.The study concludes that GPT-2 embeddings outperform others when modeling longer sentences.Source: Table 6 · p. 8, §5.3 · p. 9

6 Conclusion

The conclusion distinguishes encoder and decoder position learning and emphasizes that task and training setup affect how position information is used. It suggests choosing positional encodings in light of target-task properties.

6 Conclusion

  • Transformer encoders learn local position information that can be effective in masked language modeling, whereas autoregressive decoders learn absolute positions.The paper reports that its empirical experiments validate this hypothesis.Source: §6 · p. 9
  • Different NLP tasks, model architectures, and training objectives may use position information in different ways.Source: §6 · p. 9
  • The study argues that positional encodings or other position-modeling methods should be chosen with the target task’s properties in mind.Source: §6 · p. 9

A.1 Datasets

The paper states that pages linked in the appendix provide downloads for the datasets used in its three task areas.

A.1 Datasets

  • The appendix provides a linked page for the text-classification datasets.Source: §A.1 · p. 10
  • The appendix provides a linked page for the language-modeling datasets.Source: §A.1 · p. 10
  • The appendix provides a linked page for the machine-translation datasets.Source: §A.1 · p. 10

Machine Translation

The machine-translation experiments use mostly default settings and report their hardware and training-time requirements.

Machine Translation

  • For the position-embedding comparison, the authors use mostly default Transformer hyperparameters and do not explore many alternatives.Source: Machine Translation · p. 10

A.3 Running Time

  • Experiments run on one GTX 2080 TI GPU.Source: §A.3 · p. 10
  • Wikitext-103 language modeling takes about 10 hours; all other training runs finish within 2 hours.Source: §A.3 · p. 10

Related papers

Loading 2010.04903v1…