PaperlayerView exact revision on arXiv

Source-linked AI summary

RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings

Jarod Lévy, Mathurin Videau, Jad Yehya, Jean-Rémi King, Stéphane d'Ascoli, Thomas Moreau

arXiv:2609.34556v2cs.CLcs.LG

TL;DR

RoPE’s distance-based attention decay can favor nearby tokens, even where recurring motifs make proximity a questionable proxy for relevance. DaRoPE retains RoPE’s fast bands and replaces its slow-band positions with bounded, context-derived coordinates. Across the reported evaluations, it leads on non-text tasks while matching HoPE in language modeling and supporting extrapolation without inference-time rescaling.

  • Problem

    RoPE’s attention can decay with relative distance and favor nearby interactions, although recurring motifs in music, genomes, and neural rhythms make proximity a questionable proxy for relevance.

    Source: §1 · pp. 1–2

  • Method

    DaRoPE preserves index-based RoPE on fast bands and replaces slow-band positions with bounded, per-head coordinates predicted from contextual token representations.

    Source: §2.3 · p. 4

  • Results

    DaRoPE leads on non-textual tasks, matches HoPE on language modeling, and extrapolates without inference-time rescaling.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The authors recommend DaRoPE as the first positional encoding to evaluate when no domain-specific evidence favors another.

    Source: §1 · p. 2

Abstract

from arXiv · show

Transformers process tokens without any inherent notion of order, making positional encoding a fundamental requirement rather than an architectural refinement. Rotary Position Embedding (RoPE) has become the default positional encoding in modern language models, yet it is heavily biased toward nearby tokens. Existing alternatives have been evaluated under different settings, leaving the literature fragmented and without a clear replacement. We bring structure to this landscape by examining a specific weakness of RoPE: its slow frequency bands, whose wavelengths exceed the training context and expose models to unseen angles during extrapolation. We therefore introduce Data aware RoPE (DaRoPE), which preserves standard RoPE on the fast bands but replaces absolute position on the slow bands with bounded coordinates learned from contextual representations. Therefore, the slow-band geometry depends on the data rather than only on positional distance. We compare representative encodings under matched conditions across synthetic tasks, symbolic music, genomics, neural signals, and language models spanning 124M to 50B parameters. Across these experiments, DaRoPE leads on non-text benchmarks, mitigates recency bias, while remaining best or on par in language modeling and length extrapolation. Moreover, the learned coordinates also make the mechanism interpretable, revealing how attention layers leverage contextual information beyond token distance. Together, these results support DaRoPE as the best overall default among the evaluated methods, when there is no domain-specific reasons to prefer another.

1 INTRODUCTION

RoPE’s distance-based attention decay can favor nearby tokens, while evidence comparing positional encodings remains fragmented across settings. The paper introduces DaRoPE and evaluates it under controlled conditions, recommending it as a practical default when domain-specific evidence does not favor another encoding.

1 INTRODUCTION

  • RoPE can favor nearby interactions through attention decay with relative distance, while slow bands have wavelengths longer than the training context.This can make distance a poor proxy for relevance when useful evidence occurs thousands of tokens earlier.Source: §1 · pp. 1–2
  • Recurring motifs in music and genomes, and quasi-periodic neural rhythms, make these domains useful tests of whether proximity is an appropriate relevance prior.Existing positional conventions differ across domains, including relative attention in music and absolute embeddings in early genomic models.Source: §1 · p. 2
  • Comparisons among alternative positional encodings remain fragmented because studies use different model sizes, datasets, context lengths, and metrics.Metrics such as perplexity, retrieval, and few-shot accuracy can rank methods differently.Source: §1 · p. 2
  • DaRoPE leaves RoPE’s fast bands unchanged and replaces absolute position on slow bands with a bounded coordinate learned from contextual representations.The method adds few parameters, preserves linear overhead and FlashAttention compatibility, and requires no inference modification.Source: §1 · p. 2
  • The evaluation fixes architecture, data, and training budget while comparing encodings across domains and scales, including validation in a 50B-parameter mixture-of-experts model.The experiments cover synthetic capabilities, symbolic music, genomics, neural time series, and language modeling.Source: §1 · p. 2
  • DaRoPE is recommended as the first positional encoding to evaluate when there is no domain-specific evidence favoring another.The recommendation combines RoPE-like efficiency and language-modeling quality with strong non-text performance and resistance to recency and retrieval interference.Source: §1 · p. 2

2 FROM ROPE’S SLOW BANDS TO DAROPE

RoPE’s slow bands encounter angles beyond the training context, and existing alternatives address them through rescaling, removing positional information, or content-dependent schemes. DaRoPE keeps RoPE’s fast-band position signal and uses bounded, context-derived coordinates on slow bands.

2.1 ROPE AND ITS FAST AND SLOW BANDS

  • Slow bands have wavelengths longer than the training context and see only part of a rotation during training, so extrapolation encounters unseen angles.The fast–slow boundary depends on the rotary base; at L=4096 and d=128, increasing the base from 10k to 500k raises the slow-band count from 18 to 32 of 64.Source: §2.1 · p. 3

2.2 EXISTING APPROACHES AND THEIR TRADE-OFFS

  • Existing remedies rescale frequencies, remove slow-band distance information, or make position content-dependent, with trade-offs in target-length requirements or computational cost.YaRN requires the target length and can shift in-domain perplexity; HoPE zeros slow-band rotation, while CoPE materializes the full L × L attention matrix and is inefficient.Source: §2.2 · p. 3

2.3 DAROPE: DATA-AWARE ROPE

  • DaRoPE retains index-based rotations on fast bands and replaces the token index on slow bands with a bounded, per-head coordinate predicted from contextual representations.The coordinate combines a learned projection of the hidden state with a learned positional prior, allowing context to reorder tokens while retaining a head-specific positional signal.Source: §2.3 · p. 4
  • Similar-content tokens can receive nearby slow-band coordinates and interact as neighbors regardless of their separation, while fast bands continue to encode true position.The coordinate stays in its trained range at any evaluation length, so extrapolation requires no target-length-dependent modification; the method retains O(n) cost and FlashAttention compatibility.Source: §2.3 · p. 4
  • The schematic contrasts RoPE’s token-order coordinates, HoPE’s removed slow-band rotation, and DaRoPE’s nearby coordinates for “cat” and “feline.”Ablations report weaker performance when the positional prior is removed or its strength is fixed, per-band, or token-dependent.Source: §2.3 · p. 4

3 LANGUAGE MODELING FROM 124M TO 50B

Across language-modeling experiments, HoPE and DaRoPE extrapolate beyond the training window without target-length-dependent rescaling, while DaRoPE’s long-context advantage is concentrated in tasks requiring distributed evidence. At 50B scale, DaRoPE remains close to HoPE in-context and improves on selected long-context tasks at RoPE-like deployment cost.

Evaluation scope

  • The language-modeling comparison includes RoPE-10k/500k, NoPE, HoPE, PoPE, and DaRoPE; CoPE appears only on toy tasks because it does not scale to larger settings.The 50B evaluation compares one trained model per method.Source: §3 · p. 5, Table 3 · p. 6

In-domain language modeling

  • At 1B over positions 2–4k, trained methods retaining positional signals cluster at 17.2–17.5 perplexity; NoPE reaches 21.0 and inference-rescaled YaRN reaches 19.1.The comparison uses matched decoder-only Transformers at 124M, 350M, and 1B parameters with a 4096-token training context.Source: §3 · p. 5

124M–1B extrapolation

  • At 1B over positions 8–16k, HoPE and DaRoPE stay near 19 perplexity, while PoPE reaches 42.9 and RoPE-500k exceeds 450; both match YaRN without inference-time rescaling.NoPE and the other RoPE baselines diverge further beyond the training window.Source: §3 · p. 5

50B in-context evaluation

  • At 50B, DaRoPE and HoPE average 54.7 and 54.3 across 15 in-context benchmarks, with DaRoPE leading seven tasks and HoPE eight.The comparison spans commonsense reasoning, knowledge, math, and code.Source: §3 · p. 6, Table 2 · p. 6

50B evaluation

  • Across 15 in-context benchmarks, DaRoPE averages 54.7 versus HoPE’s 54.3; on long-context tasks, its advantage concentrates in distributed evidence, including a 1.6-point LongBench lead.On RepoBench, DaRoPE improves every metric (+5.0 edit similarity, +3.1 exact match, −0.32 answer NLL); CrossCodeEval is tied.Source: §3 · p. 6

Efficiency

  • At context 4096, DaRoPE reaches 59.8 ktok/s, versus 61.7 for RoPE, 43.4 for PoPE, and 4.3 for CoPE.PoPE is 1.4× and CoPE 13.8× slower than DaRoPE before context grows further.Source: §3 · p. 6

4 DAROPE RESISTS RECENCY AND INTERFERENCE

DaRoPE stays strong when a correct answer competes with recent decoys or many bindings. Its advantage persists through late layers, limiting drift toward recent context and preserving retrieval through prediction.

Task design

  • The probes test retrieving an early codeword over a plausible recent decoy and selecting a queried value among randomly ordered bindings.Difficulty increases through digression length and number of competing values; the reported outcome is correct-answer selection.Source: §4 · pp. 6–7

Retrieval stress tests

  • DaRoPE improves difficulty-grid average accuracy over the strongest competitor by 5.3 percentage points on return-from-digression and 9.6 points on key–value recall.At the hardest settings, it reaches 88.1% after a 2048-token digression versus 77.6% for RoPE-500k, and 30.5% exact selection with 256 bindings versus at most 7.25% for any other method.Source: §4 · p. 7, Figure 2 · p. 7

Layerwise retrieval

  • RoPE-10k’s correct-answer margin reaches +2.0 at layer 10 but ends at −3.9, whereas DaRoPE strengthens the answer through the final layers and ends at +6.5.Randomly reassigning DaRoPE’s learned coordinates across tokens erases most of both retrieval gains.Source: §4 · p. 7
  • At K=256, DaRoPE’s final key–value margin is +4.71 versus HoPE’s +2.37 on the diagnostic subset.Both models retain a positive signal through depth.Source: §4 · p. 7

Interpretation

  • The probes isolate recovery of distant content without being overwritten by the latest context, and the paper reports that this advantage also survives in language modeling and non-text results.Figure 2 tracks accuracy, correct-answer log-probability margins, and attention ratios through model depth.Source: §4 · p. 7, Figure 2 · p. 7

5 BEYOND LANGUAGE

Across four non-text datasets, DaRoPE improves over HoPE on every dataset and leads three, while retaining RoPE-like cost. On synthetic tasks, several methods form the strongest aggregate group, with different task-level strengths.

5 BEYOND LANGUAGE

  • DaRoPE improves over HoPE on all four datasets: 0.4547 versus 0.4567 on JSB, 1.4180 versus 1.4289 on MAESTRO, 4.3153 versus 4.3164 on HRG, and 4.2235 versus 4.2255 on Sleep-EDF.It leads JSB, MAESTRO, and Sleep-EDF and has the best average rank across the four datasets.Source: §5 · p. 8
  • Musical phrases, genomic motifs, and EEG rhythms recur at variable distances; DaRoPE’s consistent gain over HoPE is reported as evidence that its learned coordinate transfers beyond text.The evaluation uses matched architecture, data order, and token budget within each domain, with three training seeds.Source: §5 · p. 8
  • DaRoPE leads three of four non-text datasets, improves over HoPE on all four, and retains RoPE cost.PoPE has the lowest HRG NLL (4.3127), but each PoPE run averaged 38.9 training hours versus 11.8 for DaRoPE on eight V100s.Source: §5 · p. 8

5.1 TOY TASKS

  • Synthetic-task results use chance-normalized in-domain token accuracy averaged across tasks, with three-seed training of four-layer decoders on length-256 sequences.The suite covers Chomsky-hierarchy language classes and five additional tasks probing relative representations and data-aware position.Source: §5.1 · pp. 8–9
  • Across 15 synthetic tasks, CoPE, HoPE, DaRoPE, and RoPE form the strongest aggregate group, while PoPE and NoPE trail overall.PoPE is strongest on Chomsky-hierarchy tasks but weaker on retrieval and positional probes; NoPE especially struggles when answers require position.Source: §5.1 · p. 9

6 RELATED WORK

Prior positional encodings derive geometry from token indices or offsets, while data-aware methods incorporate content in different ways. DaRoPE focuses this design choice on RoPE’s slow bands, retaining fast-band order while learning an interpretable reordering.

6 RELATED WORK

  • Absolute and relative encodings, RoPE, distance-dependent biases, and distance-dependent scaling all derive positional geometry from token indices or offsets.RoPE rotates queries and keys according to token index; ALiBi, KERPLE, and FIRE add distance-dependent biases, while xPos adds distance-dependent scaling.Source: §6 · p. 9
  • Length-extension methods broaden training positions, remap coordinates or frequencies, or target periodic extension, while also depending on rotary base, numerical precision, and frequency allocation.The paper frames its question as which bands should encode token index at all.Source: §6 · p. 9
  • NoPE and related methods remove or reduce positional geometry, while content-aware alternatives adapt distances, biases, transformations, or rotations.DaRoPE targets only slow bands, making them an interpretable learned reordering while fast-band RoPE preserves token order at essentially RoPE cost.Source: §6 · p. 9

7 CONCLUSION

DaRoPE replaces absolute position on RoPE’s slow bands with bounded, per-head content coordinates while preserving fast-band RoPE. The paper reports broad performance strengths and identifies other content axes, band allocations, and long-context fine-tuning as open directions.

7 CONCLUSION

  • DaRoPE preserves exact fast-band RoPE and replaces slow-band absolute position with bounded, per-head content coordinates.Across evaluations, it matches HoPE on language modeling, retains in-domain quality, extrapolates without inference-time rescaling, runs at near-RoPE cost, leads on non-text tasks, and reduces recency bias and retrieval interference.Source: §7 · p. 9
  • Other content axes, band allocations, and long-context fine-tuning remain open.Source: §7 · p. 9
  • Generative AI tools assisted implementation, debugging, data reformatting, figures, code, and manuscript editing; the authors reviewed outputs and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A DENSE LANGUAGE-MODEL DETAILS

A.1 DENSE-MODEL ARCHITECTURE AND EVALUATION

Across dense scales, positional encodings that retain a positional signal remain within a few tenths of perplexity, with DaRoPE matching the strongest baseline; full NoPE regresses. The appendix also specifies matched evaluation protocols, DaRoPE’s slow-band split, and the formulation-selection procedure.

A.1 DENSE-MODEL ARCHITECTURE AND EVALUATION

  • The three dense scales use Lingua decoder blocks, 4096-token contexts, and within-scale comparisons, with different training corpora at 124M/350M versus 1B.Training uses AdamW and bf16 across 8×A100-80GB GPUs; the rotary base is 500k except for RoPE-10k and PoPE.Source: §A.1 · p. 17
  • In-domain perplexity is measured at training length on held-out pretraining-distribution tokens, with every encoding scoring the identical token stream.Source: §A.1 · p. 17
  • The length-behavior evaluation scores 256 PG-19 documents through 16,384 positions in one forward pass, without fine-tuning or positional interpolation.YaRN is the exception, applying NTK-by-parts frequency rescaling at inference with extension factor 4.Source: §A.1 · p. 17

A.2 DAROPE SPECIFICS

  • The learned content projection and positional-prior scalar add under 0.1% of model parameters at every scale.Their initialization values were selected by a small 124M hyperparameter search and reused at 350M and 1B.Source: §A.2 · p. 17
  • The wavelength criterion θ_j < 2π/L defines the fast/slow split, placing it at the midpoint for 4096-token language models but shifting it for shorter music and genomic contexts.Source: §A.2 · pp. 17–18
  • DaRoPE’s formulation ablation compares five variants at 124M using one seed, selecting among in-domain-tied variants by mean PG-19 NLL over positions 8–16k.The canonical per-head scalar is reported as the simplest formulation with the strongest selected extrapolation result; Table 5 is selection evidence, not an untouched-test evaluation.Source: §A.2 · p. 18, Table 5 · p. 18

A.3 IN-DOMAIN VALIDATION

  • Every scheme retaining a positional signal lands within a few tenths of a perplexity point at each dense scale, with DaRoPE matching the strongest baseline; only full NoPE regresses.Table 6 reports in-domain validation perplexity at the training length on each model’s held-out pretraining distribution; values are comparable within, not across, columns.Source: §A.3 · p. 18, Table 6 · p. 18

B 50B RESULTS AND ARCHITECTURE

At 50B, HoPE and DaRoPE remain close on PG-19 and controlled long-context tasks, with task-level results supporting a tie across RULER and BABILong. The appendix describes matched evaluation and non-text training procedures, alongside data corrections that affect comparison with earlier HRG results.

B 50B RESULTS AND ARCHITECTURE

  • HoPE and DaRoPE are compared on 15 in-context benchmarks, PG-19 loss to 16k, and controlled RULER and BABILong tasks, with one trained model per method.The results compare matched checkpoints rather than variability across retraining.Source: B 50B RESULTS AND ARCHITECTURE · p. 18
  • 2.5325 versus 2.5390 is the full-16k mean NLL for HoPE versus DaRoPE, although DaRoPE is marginally lower over the first 8k and HoPE over 8–16k.Figure 5 reports per-position validation NLL for models trained at context length 4096 and evaluated up to 16,384 tokens; their curves nearly overlap within and beyond training.Source: B 50B RESULTS AND ARCHITECTURE · p. 19, Figure 5 · p. 19
  • Across RULER and BABILong, median advantages reverse across context lengths and metrics, and broad interquartile overlap supports a tie between the methods.Figure 6 shows task-level median accuracy and CE gain, with shaded interquartile ranges and the 4k training-context mark.Source: B 50B RESULTS AND ARCHITECTURE · p. 19, Figure 6 · p. 19
  • LongBench excludes news-related data and evaluates only its English subset because the model was trained on an English-only corpus.Source: B 50B RESULTS AND ARCHITECTURE · p. 19

C NON-TEXTUAL SEQUENCE MODELING DETAILS

  • Music and genomic models share architecture, data order, optimizer, and token budget across methods, while Sleep-EDF uses the same sanitized training protocol.Source: C NON-TEXTUAL SEQUENCE MODELING DETAILS · p. 20
  • Non-text experiments select architecture and shared hyperparameters using RoPE-10k validation loss, then fix these across methods and give DaRoPE a small validation-only sweep.The canonical formulation is used except for MAESTRO, which uses 16 slow bands instead of 17.Source: C NON-TEXTUAL SEQUENCE MODELING DETAILS · pp. 20–21
  • Corrected reproduction replaces an upstream finite-data loader with a true infinite training loader and deterministic full-split evaluation for every method.Rebuilt HRG records remove duplicated test-sequence overlap, making its stricter test NLL higher and not directly comparable with the originally reported value.Source: C NON-TEXTUAL SEQUENCE MODELING DETAILS · p. 21

C.1 DATASET CONFIGURATIONS

The non-text evaluations use domain-specific datasets and model configurations, while paired testing compares DaRoPE with four baselines within each domain.

C.1 DATASET CONFIGURATIONS

  • JSB Chorales and MAESTRO use full-dimension rotation and 2048-token contexts; their models contain 3.57M and 85.3M parameters, respectively.The music models use nanoGPT-style decoders without bias.Source: §C.1 · p. 21
  • The genome and EEG evaluations use 306.25M- and 4.79M-parameter models, with held-out NLL and deterministic-window evaluation, respectively.The genome test set has 8,399 sequences; Sleep-EDF evaluation uses 57,782 test sequences and three training seeds.Source: §C.1 · p. 21

C.2 TEST SCORING AND PAIRED SIGNIFICANCE

  • Paired per-sequence NLL differences are tested with two-sided paired t-tests and Holm correction across four baselines within each domain.Each test example is first averaged across three seeds; Figure 3 marks significant tests when DaRoPE has lower NLL.Source: §C.2 · p. 22

D RECENCY AND CONTENT COORDINATES

The section describes synthetic tests of recency resistance and key–value retrieval, then examines attention and layerwise diagnostics. Shuffling DaRoPE’s learned coordinates reduces its final-layer margins on both tasks.

D RECENCY AND CONTENT COORDINATES

  • Return-from-digression tests whether the model selects a distant correct answer over a recent decoy, while key–value recall tests retrieval when binding distance is uninformative.The latter increases the number of bindings K; exact selection requires the correct value to rank first.Source: D RECENCY AND CONTENT COORDINATES · p. 22
  • Layerwise diagnostics track answer decodability with a logit-lens margin and compare attention to the correct value against the decoy or competing values.Margins use the final normalization and output head at each layer; attention ratios average across all 16 heads, without selecting a layer or head.Source: D RECENCY AND CONTENT COORDINATES · pp. 22–23
  • Without explicit positional signals, NoPE’s early layers attend similarly to correct and competing values; a strong preference emerges only after several layers.Figure 7 compares attention ratios at D=2048 and K=256, including DaRoPE with coordinates shuffled across token positions.Source: Figure 7 · p. 22, D RECENCY AND CONTENT COORDINATES · p. 23
  • Shuffling DaRoPE’s coordinates reduces the final-layer margin by 8.79±0.77 nats on return-from-digression and 3.76±0.47 on key–value recall.Native coordinates are favored on 96% and 93% of prompts, respectively; the intervals are 95% confidence intervals.Source: D RECENCY AND CONTENT COORDINATES · p. 23

E.1 TASK DESCRIPTIONS

  • The 15 synthetic tasks are scored on the supervised output span; per-token accuracy and exact match coincide for single-symbol answers but differ for sequences.Figure 8 reports mean and standard deviation of per-task token accuracy in-domain and at 2× and 8× training length.Source: §E.1 · pp. 23–24, Figure 8 · p. 23

E.2 CHOMSKY-HIERARCHY BENCHMARK (15 TASKS)

  • Ten Chomsky-hierarchy tasks are learnable by a four-layer Transformer and carry the main results; three remain at chance and two are excluded.Binary multiplication and compute sqrt reach zero exact-match at depth four with a 256-token context for every encoding.Source: §E.2 · p. 24, Table 9 · p. 24

E.3 RETRIEVAL PROBES (2 TASKS)

The retrieval probes are kept separate from the benchmark because their results differ notably. The positional probes isolate positional demands by holding other task features constant.

E.3 RETRIEVAL PROBES (2 TASKS)

  • NoPE’s large win occurs only on the two retrieval tasks, and pooling them with the benchmark inflates its average.The retrieval table describes these probes as pure content matching, where answers never depend on absolute position.Source: §E.3 · p. 24, Table 10 · p. 25

E.4 POSITIONAL PROBES (3 TASKS)

  • The three positional probes share rendering, alphabets, and answer format, differing only in how much positional information the answer requires.The assoc recall control differs from dup key recall only in having a unique query key, isolating the positional requirement from capacity.Source: §E.4 · p. 25, Table 11 · p. 25

Related papers

Loading 2609.34556v2…