Source-linked AI summary
Transformer Language Models without Positional Encodings Still Learn Positional Information
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, Omer Levy
TL;DR
The paper asks whether causal transformer language models truly require explicit positional encodings. It compares NoPos models across settings and probes their internal representations, finding robust performance and implicit absolute-position information, while attributing this behavior to causal attention.
Problem
Transformer attention is position-agnostic, creating a question about whether causal language models require explicit positional information to model token order.
Method
The paper compares NoPos models with position-aware baselines across datasets, model sizes, and sequence lengths, then probes representations and tests masked language models.
Results
NoPos models are competitive across settings, encode approximate absolute positions internally, and fail to extend to transformer encoders trained with the MLM objective.
Takeaways & Limitations
Causal language models can derive positional awareness from causal attention in addition to explicit positioning mechanisms.
Takeaways & Limitations
The experiments cover models from 125M to 1.3B parameters, and NoPos remains slightly worse in perplexity than positional-encoding methods.
Abstract
from arXiv · showhide
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.
1 Introduction
Although transformer attention is position-agnostic, causal language models without explicit positional encodings remain competitive and learn implicit absolute positions. The evidence suggests causal masking enables this positional awareness, unlike bidirectional masked-language-model encoders.
- Empirical findings: NoPos language models remain competitive with position-aware models across datasets, model sizes, and input sequence lengths.The comparison includes sinusoidal embeddings, learned embeddings, and ALiBi.
- Implicit position awareness: Probes predict token positions from NoPos representations with mean absolute distances similar to those of learned absolute-position models.The probing classifiers evaluate representations across network layers.
- Proposed mechanism: Causal attention may let models infer absolute position by counting the predecessors each token can attend to.The causal mask limits attention in one direction and preserves autoregressive language modeling.
- Scope: Without positional information, bidirectional masked language models fail to converge, indicating that NoPos success is specific to causal language models.Masked language models use order-invariant attention because they lack a causal mask.
2 Positional Encodings
Transformer self-attention and feed-forward layers are order-invariant, so positional information is commonly injected explicitly. The paper considers learned and sinusoidal absolute embeddings alongside ALiBi relative attention biases.
- Absolute encodings: Absolute positional embeddings add one position-specific vector to each input token embedding before the first transformer layer.Learned embeddings train vectors for absolute positions, while sinusoidal embeddings compute them from fixed sine and cosine functions.
- Absolute encodings: Learned positional embeddings are trainable representations of absolute positions used in masked and autoregressive language models.The paper lists GPT-3 among large autoregressive models using learned positional embeddings.
- Absolute encodings: Sinusoidal embeddings use constant vectors computed by non-parametric sine and cosine functions of each token’s absolute position.Different frequencies assign sinusoidal structure across encoding dimensions.
- Relative encodings: ALiBi injects relative distance information by adding negative biases to attention scores.The biases grow linearly with the distance between token pairs.
3 Experiment Setup
The experiments compare NoPos language models with several positional-encoding baselines in WikiText-103 and a large-scale Pile setting. They vary model size and sequence length to test consistency across configurations.
- Experimental design: Validation perplexity is compared between NoPos models and models using learned, sinusoidal, or ALiBi positional encodings.The study evaluates models trained from scratch in canonical WikiText-103 and large-scale Pile settings.
- Datasets and models: WikiText-103 uses a 247M-parameter adaptive embedding transformer with 512-token input sequences.The model has 16 layers, 1024 model dimensions, 4096 feed-forward dimensions, and 8 attention heads.
- Datasets and models: The Pile experiment uses filtered shards, GPT-2 tokenization, a 50K-token vocabulary, 2000 validation documents, and 15M training documents.The validation set contains 2.6M tokens, while the training documents contain 21B tokens.
- Datasets and models: The Pile baseline uses a 1.3B-parameter GPT-3 XL architecture with a default sequence length of 1024 tokens.It contains 24 transformer layers, 2048 model dimensions, 8192 feed-forward dimensions, and 32 attention heads.
- Scaling experiments: Scaling experiments vary model size from 125M to 1.3B parameters and sequence length across 256, 512, 1024, and 2048 tokens.The sequence-length study uses the 1.3B-parameter XL model.
- Additional analysis: An additional analysis compares performance across different parts of the input sequence.Detailed results are reported in Appendix A.
4 Results
NoPos language models approach position-aware baselines across datasets, model sizes, and sequence lengths, supporting the feasibility of training without explicit positional encoding.
- 0.55 perplexity on WikiText-103 and 0.05 on the Pile separate NoPos models from learned positional embeddings.
- NoPos performance remains robust across corpus, model size, and sequence length, although ALiBi becomes more beneficial on longer sequences.
- The authors publicly release their trained 1.3B-parameter models for future research and analysis.
- Smaller models benefit from sinusoidal and ALiBi encodings, but these gaps diminish as model size increases.
- Performance differences between NoPos, learned, and sinusoidal encodings remain consistently small as sequence length varies.
5 Analysis
Probing shows that NoPos models acquire positional information within the network, and disrupting token order substantially worsens their token-level loss.
- A probe predicts each token’s absolute position from hidden representations at different transformer layers.
- Within four layers, NoPos becomes position-aware and by the middle layer predicts absolute positions about as well as learned embeddings.
- NoPos begins near a random baseline, appears more position-informative than ALiBi, and sheds positional information in final layers.
- Probe predictions are more accurate near the sequence beginning and become fuzzier as position increases.
6 Conjecture
The authors conjecture that causal attention lets models infer absolute position from the number of preceding attendable tokens, whereas bidirectional models lack this cue.
- Causal attention may encode each token’s absolute position through the number of preceding tokens it can attend to.
- Figure 3 plots mean probe predictions with 95%-confidence intervals and overlays predictions from one randomly selected sequence.
- Masked language models without positional encodings fail to converge, while position-aware baselines reach very low perplexities.
7 Related Work
Prior work has explored implicit positional information, while masked-language-model results show that removing positional encodings can prevent convergence in bidirectional settings.
- The paper notes relatively little prior work investigating models’ ability to infer positions implicitly.
- In RoBERTa-large masked language models processing 128-token sequences, NoPos training fails while position-aware models converge to very low perplexities.
- Earlier speech-recognition transformers without positional encoding outperformed models using sinusoidal embeddings.
- A focused language-modeling experiment found NoPos comparable to other position-embedding methods, but it used a small-scale setting.
8 Conclusion
Transformer language models can learn positional information without explicit positional encodings, with robustness across language-modeling settings. This phenomenon does not extend to transformer encoders trained with masked language modeling, and may arise from causal attention.
- NoPos transformer language models learn positional information without explicit positional encodings.
- The phenomenon is robust across different language-modeling settings, and token positions can be approximated from internal representations.
- The positional-information effect does not extend to transformer encoders trained with the masked language-modeling objective.
- The authors conjecture that one-directional causal attention implicitly provides positional information.
9 Limitations
The study evaluates language models from 125M to 1.3B parameters, leaving behavior at substantially larger scales uncertain. NoPos models are consistently slightly worse in perplexity, suggesting positional encoding retains an inductive-bias benefit.
- The experiments cover language models ranging from 125M to 1.3B parameters, not the much larger models currently available.
- The authors state that results for models more than one hundred times larger than 1.3B parameters could be unexpected.
- NoPos perplexity is always slightly worse than that of the compared positional methods, despite very small margins.
- The authors suggest that positional encoding provides an important inductive bias.
A NoPos Performance Across Different Segments of the Input
The segment analysis evaluates whether NoPos performance varies across sequence regions. On the Pile with 1.3B-parameter models and 1024-token inputs, NoPos performs similarly or slightly worse than the learned-position baseline throughout the sequence.
- The analysis splits each input sequence into eight consecutive segments and computes perplexity separately for each segment.
- On the Pile, NoPos and Learned 1.3B models are evaluated with 1024-token inputs using the standard validation set.
- NoPos performs similarly or slightly worse than the learned absolute-position baseline across all input segments.
B Word Order Analysis
The word-order experiment tests whether NoPos next-token predictions depend on the ordering of preceding tokens. Shuffling prefix tokens causes a massive token-level-loss degradation, showing that predictions are not order-invariant.
- The experiment shuffles tokens in the prefix while computing loss for a selected next-token prediction.
- The NoPos model is tested on WikiText-103 with 512-token inputs and randomly sampled target positions from 5 to 512.
- 100 different inputs are evaluated, and the results show that next-word predictions are not order-invariant.
- Shuffling input tokens for causal language modeling causes a massive degradation in token-level loss.