PaperlayerView exact revision on arXiv

Source-linked AI summary

The Impact of Positional Encoding on Length Generalization in Transformers

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, Siva Reddy

arXiv:2305.19466v2cs.CLcs.AIcs.LG

TL;DR

Length generalization remains a challenge for Transformers, and the impact of positional encoding schemes on downstream extrapolation is unclear. This study compares five schemes across reasoning and mathematical tasks, finding that NoPE outperforms explicit positional encodings without additional attention computation.

  • Problem

    The impact of positional encoding schemes on Transformer length generalization to longer downstream sequences remains unclear despite its importance.

  • Method

    The authors systematically evaluate APE, T5’s Relative PE, ALiBi, Rotary, and NoPE on decoder-only Transformers across reasoning and mathematical tasks.

  • Results

    NoPE outperforms all explicit positional encoding schemes on downstream length generalization without computing additional attention terms.

  • Takeaways & Limitations

    Explicit positional encodings are not essential for decoder-only Transformers to generalize well to longer sequences, while scratchpad effectiveness depends on task and format.

  • Takeaways & Limitations

    The study could not assess how large-scale pretraining affects different positional encodings because comparable pretrained models and sufficient compute were unavailable.

Abstract

from arXiv · show

Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.

1 Introduction

Length generalization remains a major challenge for Transformer language models, and positional encoding is a central factor in this problem. This study compares five positional encoding schemes and finds that NoPE outperforms explicit alternatives on downstream length-generalization tasks.

  • Length generalization is the ability to generalize from smaller training context sizes to larger ones and remains difficult even for larger Transformers.
  • NoPE outperforms all explicit positional encoding schemes without computing additional attention terms.
  • Scratchpad methods are architecture-independent, but whether they make positional encoding choice irrelevant for length generalization remains an open question.
  • The study evaluates APE, T5’s Relative PE, ALiBi, Rotary, and NoPE in decoder-only Transformers across reasoning and mathematical tasks.
  • ALiBi, Rotary, and APE are ill-suited for downstream length generalization and are outperformed by T5’s Relative PE.

2 Background: Positional Encoding in Transformers

Transformers use positional encoding to represent word order, either through absolute positions or relative token distances. The section reviews APE, T5’s relative bias, Rotary, and ALiBi, and notes that decoder-only models can model sequences without explicit positions.

  • 2 Background: Positional Encoding in Transformers: Transformers use positional encoding to encode word order, with absolute schemes representing positions directly and relative schemes using distances between tokens.Transformers are parallel architectures, unlike sequential models such as RNNs.
  • 2 Background: Positional Encoding in Transformers: APE adds position vectors to word embeddings, using either periodic sine-and-cosine functions or learned embeddings trained with model parameters.Learned APE is used in GPT3 and OPT.
  • 2 Background: Positional Encoding in Transformers: T5’s relative bias maps token distance to a learned scalar added to query-key dot products, sharing parameters for distances beyond a threshold.This thresholding enables generalization to unseen distances.
  • 2 Background: Positional Encoding in Transformers: Rotary rotates query and key representations according to absolute positions, making attention depend only on relative token distance.Rotary is used in PaLM and LLaMA.
  • 2 Background: Positional Encoding in Transformers: ALiBi subtracts a distance-dependent linear bias from attention scores, creating a preference for recent tokens.Its mechanism is similar to T5’s relative bias but uses subtraction and linear distance growth.
  • 2 Background: Positional Encoding in Transformers: Decoder-only Transformers with causal attention can model sequences without explicit position information, unlike encoder-only Transformers that become bag-of-words models.Whether decoder-only models encode position implicitly or generalize to unseen lengths remains unclear here.

3 Model Evaluation

The study evaluates decoder-only Transformers on synthetic sequence-to-sequence tasks for length generalization, comparing four explicit positional encoding schemes with removing positional encoding. Models are trained from scratch for each task, and performance is assessed across seen and unseen sequence lengths.

  • Length Generalization Setup: Models train on examples up to finite lengths and are tested at both seen and unseen lengths for algorithmic sequence-to-sequence tasks.The input is a task instance and the output is its solution.
  • Architecture: The experiments compare Absolute Position Embedding (APE), ALiBi, Rotary, T5’s Relative Bias, and removing positional encoding in a decoder-only Transformer.Each positional encoding variant is trained from scratch on the corresponding task data with an autoregressive language modeling objective.
  • Tasks: The evaluation covers primitive, mathematical, and reasoning synthetic tasks, including Copying, Reversing, Addition, Polynomial Evaluation, Sorting, Summation, Parity, and LEGO.These task categories have been widely used to investigate length generalization.
  • Tasks: Results are reported over ten tasks with three seeds for each dataset–positional-encoding pair.This setup provides repeated evaluations across task and encoding combinations.

4 What Is The Effect of Positional Encoding?

Positional encoding methods fit training-length examples well but diverge substantially when extrapolating beyond seen lengths. T5’s Relative Bias generally outperforms explicit alternatives, while NoPE matches or exceeds it without added attention computation.

  • Length extrapolation: Models achieve perfect or near-perfect accuracy on training-distribution lengths, but positional-encoding differences emerge on lengths larger than those seen during training.Figure 3 evaluates generalization across six datasets and shows that I.I.D.-length performance is largely saturated.
  • Explicit positional encodings: T5’s Relative Bias outperforms other explicit positional encodings in most extrapolation scenarios.ALiBi is intermediate, whereas APE and Rotary show poor generalization.
  • Explicit positional encodings: Rotary generalizes more like APE than other relative schemes, while ALiBi underperforms T5’s Relative Bias in most cases.This agrees with prior findings reporting no significant improvement from ALiBi.
  • No positional encoding: NoPE performs on par with or better than T5’s Relative Bias while adding no computational overhead in the attention mechanism.Because NoPE computes no additional attention term, it can affect runtime and memory footprint.

5 How Does NoPE Represent Positions?

NoPE can recover absolute positions in its first layer and implement relative positional encoding in later layers. The paper then examines which positional mechanism SGD learns in practice by comparing NoPE attention patterns with other positional encoding schemes.

  • Theoretical analysis: NoPE’s first layer can recover absolute positions and write them into the next hidden state.Theorem 1 states that suitable attention and feed-forward parameters compute positions [1, . . . , T + 1].
  • Theoretical analysis: SGD can potentially learn to recover absolute positions in NoPE Transformers.The theorem establishes representational possibility rather than specifying that SGD always learns this mechanism.
  • Theoretical analysis: If absolute positional information persists after the first layer, subsequent layers can implement relative positional encoding.For l ≥2, attention dot products can depend on content and relative distance through functions fcnt and frel.
  • Theoretical analysis: The theoretical results suggest that SGD can choose between relative and absolute encoding in NoPE Transformers.The mechanism learned by SGD in practice remains unclear, motivating the empirical analysis.
  • Empirical analysis: The empirical analysis compares NoPE attention patterns with models using different positional encodings after training.The study feeds identical inputs to both models, computes minimum cross-head Jensen–Shannon divergence at each layer, and uses different NoPE seeds as a baseline.

6 Does Scratchpad Render The Choice of Positional Encoding Irrelevant?

Scratchpad does not make positional encoding irrelevant: its benefits are task- and format-dependent, while positional encoding remains crucial for length generalization. Attention patterns further show that NoPE resembles T5’s Relative PE, unlike ALiBi’s short-range bias.

  • Scratchpad format: The scratchpad format has a non-trivial impact on model performance, making its representation of intermediate computations an important design choice.The evaluated components are <input>, <computation>, <output>, <variable_update>, and <remaining_input>.
  • Scratchpad effectiveness: Scratchpad is beneficial solely for the addition task across all positional encodings and formats.The experiments systematically enable or disable five scratchpad components to compare formats.
  • Scratchpad format: The commonly used scratchpad format contains redundant information, including repetition of the remaining input that attention can access directly.This motivates examining which scratchpad components different positional encodings rely on.
  • Attention patterns: NoPE and T5’s Relative PE exhibit similar bimodal attention distributions spanning short-range and long-range tokens.The normalized attention distance distinguishes nearby scratchpad-step tokens from distant input tokens.
  • Attention patterns: ALiBi strongly favors short-range attention because of its recency bias.Values of normalized distance close to 0 indicate attention to nearby tokens, whereas values close to 1 indicate distant tokens.

7 Discussion

The discussion argues that downstream length generalization distinguishes positional encodings more clearly than I.I.D. evaluation, with NoPE outperforming explicit alternatives. Larger-scale results reinforce this concern, while language-modeling perplexity may not reliably predict downstream extrapolation.

  • Positional-encoding choice: I.I.D. evaluation shows similar performance across positional encodings, making the optimal architectural choice difficult before costly pretraining.This contrasts with downstream length-generalization evaluation, which reveals clearer differences among positional-encoding approaches.
  • Downstream length generalization: Downstream length generalization shows that NoPE outperforms explicit positional encodings, while commonly used methods lag behind T5’s Relative PE.The paper uses downstream length generalization as a measure of positional-encoding expressivity.
  • Architectural implications: The empirical and theoretical findings suggest that removing positional encoding is a promising modification to decoder-only Transformers.The discussion links explicit positional encodings’ disadvantages in length extrapolation to broader evidence that they challenge Transformer performance.
  • Scaling up to 1B models: At 1B parameters, all tested variants have similar I.I.D. perplexity, but Rotary fails at length generalization as its perplexity explodes.The variants were ALiBi, Rotary, and NoPE, trained with 1024-token contexts on a subset of StarCoder.
  • Perplexity vs. downstream performance: Language-modeling perplexity may misrepresent downstream length generalization because short-range dependencies and positional-encoding recency bias can distort extrapolation behavior.The discussion specifically identifies ALiBi’s recency bias as a potential source of unrealistic performance estimates.

8 Related Work

Prior work has documented length-generalization failures in conventional Transformers with absolute positional encoding, while also showing that Transformers without explicit positional encoding can perform comparably across multiple domains. This paper extends those observations by studying operation without explicit position information in length generalization.

  • Length Generalization Failure In Transformers: Prior studies reported length-generalization failures of conventional Transformers using APE on specific datasets.The passage situates this failure within a longstanding research topic in neural sequence models.
  • Positional Encoding: Positional encoding is needed to represent input order because self-attention is order-invariant and otherwise risks becoming a bag-of-words model.The passage describes absolute and relative positional encoding as established approaches for addressing this limitation.
  • Transformers Without Positional Encoding: Transformers without positional encoding have performed on par with explicit-PE models in machine translation, language modelling, vision, and speech.The cited work spans several application domains, motivating further study of Transformers without explicit positional encoding.

9 Conclusion

The study finds that NoPE is effective for length generalization in decoder-only Transformers, whereas widely used explicit positional encodings are unsuitable. It also shows that NoPE can learn positional information implicitly and that scratchpad effectiveness varies.

  • Conclusion: NoPE is effective for length generalization, while widely used explicit positional encodings are not suited to this setting.This conclusion follows an extensive empirical study across downstream mathematical and reasoning tasks.
  • Conclusion: NoPE can implicitly learn both absolute and relative positions, using relative positions in practice.The paper combines a theoretical proof with an observation about the positional patterns learned in practice.
  • Conclusion: Scratchpad effectiveness is not uniform across length-generalization settings.The supplied passage states that the effectiveness of scratchpad is variable, but does not provide the remainder of the finding.

Limitations … B.2 Positional Encoding

The paper limits its positional-encoding study by omitting large-scale pretraining comparisons, then defines decoder-only Transformer notation and formulates several positional-encoding schemes through their attention dot products.

  • Limitations: The study could not assess how large-scale pretraining affects positional encodings because comparable publicly available models were unavailable and compute was limited.The authors leave this comparison for future work.
  • A Number of instances decreases rapidly as sequence length grows: Instruction-finetuning datasets often contain mostly short sequences, with median instruction lengths substantially below their maximum lengths.The passage identifies FLAN (CoT subset) and Super Natural Instructions as examples.
  • B Background: The background section establishes notation for the decoder-only Transformer model and is referenced by Appendices C.1 and C.2.The model is denoted fθ and maps input sequence x to output sequence y.
  • B.1 Preliminaries: Decoder-only Transformers use causal attention, so each position’s attention weights depend only on previous positions.The model processes x = [x0, x1, . . . , xT ] and produces y = [y0, y1, . . . , yT ].
  • B.1 Preliminaries: Each Transformer layer combines multi-head self-attention with a feed-forward sub-layer and computes hidden states column-wise from the preceding layer.The formulation defines hidden states H(l), attention outputs, feed-forward transformations, and layer normalization.
  • B.2 Positional Encoding: Positional-encoding methods are formulated primarily by specifying how the query–key dot product is computed.The paper uses this perspective to introduce APE, T5’s Relative PE, ALiBi, Rotary, and NoPE.
  • B.2 Positional Encoding: APE combines word embeddings with vectors assigned to absolute positions, whereas T5’s Relative PE adds a learned bucketed bias based on relative distance.T5 uses B = 32 buckets and D = 128 maximum distance in its original implementation.
  • B.2 Positional Encoding: ALiBi applies a distance-dependent linear bias, Rotary rotates query and key representations by absolute positions, and NoPE adds no explicit positional encoding.Rotary’s resulting attention dot product depends solely on relative distance, while NoPE leaves the dot product otherwise unmodified.

C Proofs … D Experimental Details

The proofs show that NoPE can recover absolute positions in its first layer and use them to implement relative positional encoding in later layers. These constructions rely on causal attention, a <bos> anchor, and preserving positional information in the hidden state.

  • C Proofs: NoPE can implicitly learn both absolute and relative positional encodings.The proofs establish these capabilities theoretically.
  • C.1 Absolute Positional Encoding in NoPE: The first layer can recover absolute positions [1, . . . , T + 1] in the hidden state.Theorem 1 asserts that suitable attention and feedforward parameters compute and write these positions.
  • C.1 Absolute Positional Encoding in NoPE: A constructed attention head writes absolute position information to the third hidden-state dimension.A sufficiently wide ReLU feedforward sub-layer can recover [1, 2, . . . , T +1] from that dimension.
  • C.1 Absolute Positional Encoding in NoPE: Causal attention and a <bos> token enable NoPE to anchor absolute position information.The causal mask restricts attention to positions ≤t, while <bos> provides the anchor.
  • C.2 Relative Positional Encoding in NoPE: If absolute positions are preserved after the first layer, subsequent attention layers can implement relative positional encoding.Theorem 2 states that for l ≥2, query-key dot products can combine content and relative-distance functions.
  • C.2 Relative Positional Encoding in NoPE: The constructed query-key dot product depends on relative token distance, while the first-layer MLP can support more complex schemes.The proof uses linear spacing, and relative dependence assumes other terms are prevented from canceling it.

D.1 Tasks … F.2 Dataset Selection

The appendix details the evaluation tasks, shared experimental setup, compute budget, full-results references, and a 1.3B-parameter code-pretrained model configuration. It uses standardized datasets and training settings across positional encoding schemes while documenting task-specific generation and evaluation procedures.

  • D.1 Tasks: Each evaluation task uses 100K training examples, 10K test examples, and 15% of training data for validation.The same sampling protocol is stated for each task.
  • D.1 Tasks: The task suite covers arithmetic, polynomial evaluation, sorting, summation, parity, computation graphs, copying, and reversal, with correctness as the evaluation criterion where specified.Sorting includes Single Token and Multi Digit variants; copy and reverse include multiple variants.
  • D.2 Hyperparameters: All models and positional encoding schemes use the same hyperparameters, and tested learning-rate and weight-decay alternatives showed no significant differences.The tested learning rates were 0.00001, 0.00003, and 0.00005, while WeightDecay values were 0, 0.05, and 0.1.
  • D.3 Compute: Training uses single-GPU runs on NVIDIA V100, RTX8000, and A100 hardware, taking 6 to 15 hours per seed and totaling about 870 individual runs.The total includes datasets, positional encoding schemes, scratchpad experiments, and three seeds.
  • F Pretraining at 1.3B Scale: The 1.3B-scale preliminary study pretrains models with varied positional encodings on identical code data and consistent parameters, reflecting the importance of element positions in code semantics.The study uses code-based pretraining for this reason.
  • F.2 Dataset Selection: Pretraining uses 30M StarCoder documents comprising 40% Python, 25% Java, 25% JavaScript, 5% GitHub issues, and 5% GitHub commits, yielding 30B tokens for one epoch.The StarCoder tokenizer has a vocabulary size of 49,152.

F.3 Generalization Evaluation

The evaluation measures perplexity as context size changes, including beyond the 1024-token training context, across source-code documents grouped by length. All models benefit from larger contexts up to 1024, while Rotary fails to generalize out of distribution.

  • Evaluation setup: Perplexity is evaluated on the final token of 1500 validation documents, each representing a single source code file, across varying context sizes.The evaluation focuses especially on context sizes exceeding the training context size.
  • Evaluation setup: Documents are grouped by length to calculate average perplexity, distinguishing document length from the length of underlying dependencies.Longer documents do not necessarily imply longer dependencies.
  • Generalization results: All models benefit from larger contexts up to the 1024-token training size, with no significant differences in the I.I.D. case across length buckets.The I.I.D. case is defined as context length ≤1024.
  • Generalization results: Rotary fails to generalize in the O.O.D. case because its perplexity explodes across all length buckets.The O.O.D. case is defined as context length >1024.
Loading 2305.19466v2…