Source-linked AI summary
Distance generalization in transformers: why bother with positional encoding?
Daniel Henrik Nevermann, Claudius Gros
TL;DR
The paper asks how transformers generalize when source–recall distances change while context length stays fixed. It studies full and selective delay-copy tasks across positional encodings, distance diversity, and transfer-learning settings, finding that NoPE often generalizes best, diversity helps with diminishing relative returns, and transfer can help or hinder.
Problem
Distance generalization tests whether transformers handle unseen token dependencies within a fixed context, complementing length generalization’s focus on unseen positions.
Method
The study evaluates full and selective synthetic delay-copy tasks while varying positional encoding, training-distance diversity, and transfer-learning conditions.
Results
NoPE often generalizes best, broader distance diversity improves absolute performance with diminishing relative returns, and transfer learning can either improve or hinder distance generalization.
Takeaways & Limitations
Distance generalization provides a complementary way to investigate transformer generalization across positions and to assess the effects of positional encoding, data diversity, and transfer.
Takeaways & Limitations
The data do not rule out a minimum training-data diversity required for generalization, and the authors leave that question for future work.
Abstract
from arXiv · showhide
Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.
1 Introduction
The paper studies distance generalization: changing source–recall delays while keeping context length fixed, complementing length generalization and separating unseen positions from unseen token dependencies. It investigates positional encoding, training-distance diversity, and transfer learning using synthetic delay-copy tasks.
- Motivation: Distance generalization keeps context length fixed while changing the inter-token delay between source and recall at inference.This contrasts with length generalization, which evaluates longer sequences and contexts than those seen during training.
- Approach: The study uses full and selective delay-copy tasks derived from a task-switching framework to assess unseen delays.The selective task copies only part of the source sequence, whereas the full task copies all designated tokens.
- Findings: NoPE produces the best distance-generalization capabilities, outperforming explicit positional encodings such as RoPE and ALiBi.The result is described as paradoxical and agrees with observations from length-generalization studies.
- Findings: Greater training-distance diversity improves absolute generalization but yields strongly diminishing returns when performance is evaluated relatively.The paper also examines whether auxiliary-task transfer helps distance generalization, finding benefits in some cases but not consistently.
- Significance: The paper argues that distance generalization should be investigated to understand how transformers generalize across positions.Its contribution is to examine positional encoding, data diversity, and positive or negative transfer effects together.
2 Related literature
Related work has mainly studied length generalization and methods for improving it, while this paper connects positional encoding, data diversity, and transfer learning to broader questions of transformer generalization.
- Length generalization: Length generalization evaluates whether transformers extrapolate from short training sequences to longer test sequences, often using synthetic copying, reverse-copying, or arithmetic tasks.Prior approaches include specialized positional encodings, training protocols, scratchpads, and knowledge transfer.
- Positional encoding: RoPE is widely used but has poor length generalization, whereas ALiBi improves length extrapolation and NoPE has also been preferred.These findings motivate testing positional encoding schemes for distance generalization.
- Transfer learning: Transfer learning research shows that knowledge from related tasks can be reused, including for improving a main task’s length generalization.Earlier work trains a main task alongside an auxiliary task at longer context lengths.
- Data diversity: Prior studies identify training-data diversity as an important determinant of generalization and link insufficient diversity to non-generalizing positional shortcuts.Increasing diversity has also been associated with the formation of induction heads.
3 Experimental setup
The experiments use fixed-length, task-switching sequences containing random, full delay-copy, and selective delay-copy tasks, and evaluate teacher-forced next-token accuracy across trained and unseen delays.
- Tasks: The random task inserts unpredictable tokens that are excluded from the loss, discouraging shortcut learning in copy tasks.Random tokens force models to copy rather than rely on hidden patterns.
- Tasks: The full delay-copy task recalls m preceding tokens after a sampled distance dC, while the selective task recalls only even-numbered tokens in their original order.The full and selective tasks can use different distance ranges when combined.
- Data construction: Sequences use a fixed context length T = 256 and switch among synthetic tasks, with datasets DRC, DRS, and DRCS combining random, full-copy, and selective-copy tasks.The task-switching setup lets the experiments vary which delay-copy tasks appear together.
- Task switching: Task durations are sampled from a Binomial distribution with n = ℓmax and p = µ/ℓmax, using µ = 6 and ℓmax = 9.The experiments typically use Nbase = 128 integer data tokens.
- Model and evaluation: Models are conventional causal decoder-only transformers with 8 layers, 8 attention heads, and model dimension 512, tested with ALiBi, RoPE, or NoPE.Training uses next-token prediction with teacher forcing and cross-entropy optimization, while evaluation measures full-sequence teacher-forced accuracy rather than free-form generation.
4 Results
Distance generalization is strongly shaped by positional encoding, training-distance diversity, and task transfer. NoPE generally performs best, broader distance coverage improves absolute extrapolation with diminishing relative returns, and transfer effects vary by encoding and task separation.
- Positional encoding: NoPE outperforms ALiBi and RoPE on distance generalization, while RoPE performs worst despite all schemes achieving high in-distribution accuracy.The experiments use basic and selective delay-copy tasks with training delays sampled from 15 to 25 and m = 10 copied tokens.
- Training data diversity: Increasing training-distance diversity raises out-of-distribution performance but lowers the relative-performance ratio, indicating sub-linear gains with diversity.The conclusion is attributed to models reaching an accuracy floor before the evaluation range ends.
- Training data diversity: The data does not rule out a minimum distance diversity threshold for developing generalization, so this possibility remains unresolved.The authors leave detailed investigation of this question for future work.
- Basic transfer learning: With contiguous but disjoint task-distance ranges, RoPE shows mostly negative transfer, whereas ALiBi and NoPE show positive transfer at moderate but negative transfer at larger delays.The auxiliary task primarily acts as a distractor for RoPE, contrasting with mostly positive transfer reported for length generalization.
- Complexity of transfer learning: With enlarged task separation, NoPE and ALiBi show constructive transfer from selective to basic copy at large distances but destructive effects around the basic-copy training region.The authors suggest, without establishing it, that selective copy’s greater structural complexity may favor transfer toward basic copy.
- Complexity of transfer learning: RoPE exhibits generally weak transfer but avoids the destructive effects within original training regions observed for NoPE and ALiBi.Thus, weaker transfer for RoPE coincides with less interference with bare task training.
5 Discussion
Distance generalization isolates unseen token dependencies within a fixed context, revealing effects of positional encoding, data diversity, and transfer learning. The discussion also identifies experimental-scope concerns and directions for further study.
- 5 Discussion: Distance generalization evaluates unseen token dependencies at fixed context length, separating them from failures on unseen positions in length generalization.This distinction motivates testing models on changed inter-token distances while keeping the context window fixed.
- 5 Discussion: RoPE falls behind ALiBi and NoPE, although explicit positional encoding remains necessary for small models.The authors note that RoPE is not encountering untrained angular regimes at test time in this setting.
- 5 Discussion: Increasing training-distance diversity raises absolute out-of-distribution performance but lowers relative performance, indicating diminishing returns.The reported interpretation relies on the relative metric P and saturation of in-distribution accuracy before the evaluation range ends.
- 5 Discussion: Joint transfer-learning datasets reduce task-specific examples, but saturated in-distribution accuracy and distance-specific degradation argue against exposure deficit alone.The authors present these observations as a response to concerns about the experimental setup.
- 5 Discussion: The study selects the best of five runs because runs that fail to learn copying are uninformative for distance generalization.The authors report that differences between runs are minor.
A.1 Model Architecture
The study uses a standard causal decoder-only Transformer and compares RoPE, ALiBi, and NoPE positional configurations.
- A.1 Model Architecture: The model has 8 Transformer layers, hidden size 512, 8 attention heads, head dimension 64, ReLU MLPs, and dropout rate 0.1.Input and output embeddings are untied.
- A.1 Model Architecture: The positional-encoding comparison includes RoPE, ALiBi, and no positional embedding (NoPE).
A.2 Training Procedure
The training procedure uses AdamW with a warm-up schedule, fixed batch size and duration, and autoregressive next-token prediction.
- A.2 Training Procedure: Training uses AdamW, learning rate 3 × 10^-4, 3,000-step linear warm-up, weight decay 0.05, batch size 64, and 40,000 updates.The learning rate remains constant after warm-up.
- A.2 Training Procedure: The objective is standard autoregressive next-token prediction, excluding the first few sequence tokens from loss and accuracy computation.This exclusion avoids early-sequence perplexity effects.
A.3 Dataset
The dataset is a synthetic multi-task sequence collection built with task switching across random, delay-copy, and selective-delay-copy tasks.
- A.3 Dataset: Training uses three recurrently switched tasks: random (R), delay copy (C), and selective delay copy (S).The delay-copy tasks are the synthetic tasks used to study distance generalization.
- A.3 Dataset: Dataset configuration is summarized in Table 3.
B Additional Results
Figure 6 supplements the main results with multiple training runs across positional encodings and both delay copy tasks. Solid lines identify the best-performing runs using cumulative accuracy over the plotted distance range.
- Figure 6 compares positional encodings across the delay copy and selective delay copy tasks.The tasks are evaluated with delays sampled from dC/S ∼unif(15, 25).
- The figure reports multiple training runs for each task and positional encoding.
- Solid lines show the best-performing runs, measured by cumulative accuracy over the full plotted distance range.
Variance of training runs.
The results use a best-run policy, while additional experiments show that distance generalization depends strongly on model size. NoPE can generalize well, but explicit encodings are more stable for smaller models.
- Main-text results generally select the best of five equal-parameter training runs using cumulative accuracy.Figure 6 provides complementary runs, while other main-text results use the same best-model policy.
- Larger models generally generalize better across distances, while small ALiBi models can break down substantially faster.The study varies head count and embedding dimension while keeping eight layers fixed.
- NoPE often achieves strong distance generalization but breaks down in small models.Small NoPE models may struggle even to learn the delay-copy mechanism in-distribution.
- Explicit positional encodings are more stable than NoPE for smaller models.