PaperlayerView exact revision on arXiv

Source-linked AI summary

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Yufeng Du, Phillip Harris, Minyang Tian, Eliu A Huerta, Srikanth Ronanki, Subendhu Rongali, Aram Galstyan, Hao Peng

arXiv:2605.15514v1cs.CLcs.AIcs.LG

TL;DR

Long-context Transformer models often struggle despite advertised context lengths, raising whether RoPE has intrinsic limitations. This paper theoretically analyzes RoPE attention and empirically finds that it loses reliable distinctions between positions and tokens as contexts grow.

  • Problem

    The paper asks whether recurring long-context failures reflect engineering choices or intrinsic RoPE limitations, a distinction important for guiding future progress.

  • Method

    The authors model RoPE attention scores as normal random variables and analyze position and token identification as context length increases.

  • Results

    RoPE intrinsically fails to distinguish positions and token identities in long inputs, while increasing its base trades off these objectives rather than preserving both.

  • Takeaways & Limitations

    Reliable context extension requires addressing both position and token identification, motivating fundamentally new positional mechanisms for long-context language models.

  • Takeaways & Limitations

    The real-model experiments do not theoretically analyze multi-head, multi-layer attention or directly prove that observed failures are caused by RoPE confusions.

Abstract

from arXiv · show

We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We prove that as context length increases, RoPE-based attention becomes unpredictable and loses two properties that are central to its effectiveness. First, it loses its locality bias: RoPE is no more likely to favor nearer positions than substantially farther ones. Second, it loses consistency in token relevance: a key vector that receives a higher attention score than an alternative at one position may receive a lower score at another. In both cases, the probability of failure approaches 0.5, no better than random guessing. We further prove that the attention score can remain unchanged when a key token is moved to a different position, or even replaced by a different token, indicating a failure to distinguish positions or tokens. Adjusting the RoPE base trades off distinguishing positions against distinguishing tokens but cannot preserve both at the same time. Increasing the RoPE base hyperparameter, a common practice in today's long-context models, helps distinguish different tokens, but inevitably sacrifices the ability to distinguish positions. Our empirical analysis shows that multi-head, multi-layer architectures are insufficient to overcome these limitations. Our findings suggest that fundamentally new mechanisms for encoding position and token order may be needed in future Transformer long-context language models.

1 Introduction

The paper argues that RoPE has intrinsic long-context limitations: as context length grows, its attention effects become unpredictable and undermine position and token distinction. Theory and experiments show these failures persist in real multihead, multilayer language models.

  • Theoretical contribution: RoPE’s theoretical limitations depend only on context length, not specific context content, and emerge under mild assumptions as contexts grow.The analysis studies single-head attention and abstracts away from context content.
  • Position failures: As context length increases, position inversion approaches probability 0.5, so RoPE becomes no better than random chance at favoring nearer over farther positions.This failure effectively removes RoPE’s locality inductive bias.
  • Position failures: The analysis identifies position aliasing, where moving a key token can leave its attention score unchanged for a fixed query and key.This provides a concrete mode in which RoPE fails to distinguish positions.
  • Analytical method: The key analytical insight is modeling the unnormalized attention score as a normal random variable.The paper derives its theoretical results from this probabilistic treatment.
  • Empirical validation: On Llama 3.1-8B, 75K position pairs showed position aliasing across only 8K tokens, while around 150 positions exhibited token aliasing.These empirical patterns appeared regardless of positional proximity and confirmed the theoretical conclusions about position and token inversion.
  • Empirical validation: Experiments on six models from 7B to over 100B confirmed that RoPE’s failures persist in real multihead, multilayer language models.The tested task required identifying the value at the k-th list position, targeting position distinction rather than token identity.

2 Demystifying RoPE

This section frames RoPE attention around position and token identification and analyzes its un-normalized attention score as a function of relative distance. Its key perspective treats the RoPE product as approximately normally distributed, with context length governing frequency-driven decay and oscillation.

  • Objectives: Attention must support position identification for word-order dependencies and token identification for distinguishing contextually salient tokens.Failures in position identification can impair counting and reasoning, while failures in token identification affect contextual salience detection.
  • RoPE product: The RoPE product is the un-normalized query–key dot product after applying RoPE, analyzed through relative distance and context length.The section uses this quantity to study how RoPE supports position and token identification and how attention behaves as context length increases.
  • Frequency components: High-frequency components oscillate, whereas low-frequency components rotate slowly and exhibit a decay effect as relative distance grows.The analysis separates components at λ(M) = Θ(h log_B M), with the threshold determined by the context limit M.
  • Frequency components: High-frequency oscillation helps distinguish close positions, while low-frequency decay distinguishes distant position pairs and supports locality bias.Low-frequency components also stabilize token identification through their slower rotation.
  • Probabilistic characterization: The RoPE product is approximately normal, with its mean governed by low-frequency terms and its variance by high-frequency terms.As context length grows, the mean decreases through decay while the variance increases through oscillation; decay is preferred but not guaranteed.

3 RoPE Fails to Distinguish Positions in Long Contexts

As context length grows, RoPE can reverse its locality bias, making substantially farther positions receive higher attention than nearer ones with probability approaching 0.5. RoPE also develops position aliasing, where distinct distances share attention scores, becoming nearly inevitable and causing attention invariance even at short contexts.

  • Position inversion: Position inversion reverses RoPE’s locality bias when a substantially farther key receives a higher attention score than the same key at a nearer position.The analysis considers pairs from opposite halves of the context, with m1 < M/2 ≤ m2.
  • Position inversion: As context length and RoPE base grow, position-inversion probability approaches 0.5, making attention nearly arbitrary between nearby and farther positions.This unpredictability can prevent the model from identifying a reliable positional pattern.
  • Position aliasing: Position aliasing becomes exponentially likely as context length M increases, while the total number of aliasing pairs grows with both M and RoPE base B.Aliasing means that changing the query-key distance does not change the attention score, so different positions become indistinguishable.
  • Position aliasing: Under 8K context and BF16 precision, almost every distance participates in an aliasing pair, with more than 75k pairs and 1,491 attention-invariance cases.Position aliasing can make swapping keys at aliasing positions leave the attention output unchanged, even at short context lengths.

4 RoPE Fails to Distinguish Tokens in Long Contexts

In long contexts, RoPE becomes unreliable at distinguishing tokens: it can reverse the relevance ordering between keys and assign identical attention scores to different tokens. These failures become more likely with context length, while increasing the RoPE base mitigates but does not eliminate them.

  • Token inversion: Token inversion reverses the relevance ordering between two keys, with its probability lower bound approaching 1/2 as context length M approaches Θ(B).The lower bound decreases with the RoPE base B.
  • Token aliasing: Token aliasing occurs when replacing one key with another leaves the attention score unchanged, so RoPE fails to distinguish different tokens at that position.The number of aliasing positions increases with M and decreases with B.
  • Token aliasing: Up to 5% of positions exhibit token aliasing for h = 64 under BF16, corresponding to 1.6K aliasing positions in 32K tokens.Increasing the RoPE base can mitigate token aliasing, but it remains almost always present in long inputs.
  • Overall implication: As context length increases, token inversion and token aliasing become more likely; increasing RoPE base mitigates both but does not fully resolve them.The RoPE base trades these token failures against worsening position inversion and position aliasing.

5 How Do Multilayer, Multihead Transformer LLMs Fare?

An empirical evaluation of six open RoPE-based long-context LLMs finds that multilayer, multihead architectures do not overcome RoPE’s position-confusion limitation. All models begin near-perfect on indexing but quickly approach random-guessing performance at several-thousand-token contexts.

  • Evaluation: Six open RoPE-based long-context LLMs of different sizes were evaluated in a controlled comparison.Closed-source models were excluded because their architectures and positional-embedding choices are unknown.
  • Evaluation: The indexing task required models to return arr[i] from Python lists containing only integers 0, 1, 2, and 3.Each list length used 100 samples, and the four possible values made the task multiple-choice.
  • Results: All models started near-perfect but quickly dropped close to random guessing as context length increased.The evaluation therefore exposed position confusion rather than merely general language-model failure.
  • Results: Serious position confusion appeared with contexts as short as several thousand tokens, despite the models’ multilayer, multihead designs.The paper attributes this cost to optimizing long-context models for token identification.

6 Conclusion and Discussion

RoPE intrinsically fails to distinguish positions and token identities in long inputs, and changing its base only trades off these objectives. The same bottleneck persists in practical multi-head, multi-layer models, motivating mechanisms that preserve both distinctions.

  • RoPE intrinsically fails to distinguish both position and token identity in long inputs.
  • The RoPE base selection trades off distinguishing positions against distinguishing token identities rather than achieving both objectives.
  • The same bottleneck persists in empirical multi-head, multi-layer models.
  • A robust positional mechanism should maintain the ability to distinguish positions and tokens, which is necessary for effectively extending context length.
  • Recent alternatives explore improved context management and recursive or agentic language-model paradigms without directly resolving RoPE’s intrinsic limit.

Limitations

The analysis uses simplifying assumptions and approximations, does not theoretically explain multi-layer multi-head failures, and does not fully analyze specific RoPE scaling variants. A Llama-3.1 case study nevertheless suggests scaling does not fundamentally resolve position–token confusion.

  • Ambiguity of the frequency threshold: The frequency threshold λ(M) = Θ(h logB M) is only rough because threshold frequencies with n = Θ(h logB M) require separate treatment.The normal approximation is slightly skewed relative to the real distribution.
  • Assumption of regular rotary amplitudes: The analysis assumes relatively uniform RoPE-product amplitudes, although real attention heads can contain dimensions with significantly larger amplitudes.Dominating dimensions heuristically reduce the effective number of dimensions by suppressing other components.
  • Real models: The paper neither theoretically analyzes multi-layer multi-head attention nor categorizes real-model failure modes, because failures can arise from mixed factors.Its final experiment demonstrates that real models still face position or token confusion and must trade position confusion for better token identification.
  • RoPE scaling: The paper does not provide detailed analysis of specific RoPE scaling variants, while a Llama-3.1 case study suggests scaling does not fundamentally resolve the problem.Llama-3.1 uses RoPE scaling, but the case study indicates that scaling does not resolve the underlying confusion, if it helps at all.

A Rotary Positional Embedding … C The Failure Modes

RoPE encodes relative position through query-key rotations, combining high-frequency oscillations that distinguish nearby positions with low-frequency components that support decay and token relevance. As context length grows, these frequency components lose their intended behavior, while the RoPE product can be approximated statistically under stated assumptions and extended to many scaling variants.

  • A Rotary Positional Embedding: RoPE applies the same multiplication-based positional embedding to query and key vectors, making attention depend on their normalized inner product.The RoPE product is designed to depend only on relative distance, i − j.
  • A Rotary Positional Embedding: High-frequency components create oscillations that distinguish close position pairs across distance m.These rapid rotations generate variation in RoPE products for nearby positions.
  • A Rotary Positional Embedding: Low-frequency components produce an interval of decay that supports recency bias and identification of distant positions.When n ≫ λ(M), the cosine term decreases with m over m ∈ [0, M).
  • A Rotary Positional Embedding: For a given RoPE base B, the natural context-length limit is Θ(B), because the lowest-frequency term eventually oscillates and makes positional embeddings ambiguous.The stated upper bounds are M < 2πB for uniqueness and roughly πB when recency bias must remain decreasing.
  • A Rotary Positional Embedding: RoPE decay is not universal: it occurs only over an interval and may not begin initially, depending on collective low-frequency rotations and their phases.Negligible low-frequency amplitudes effectively shrink the base wavelength, shortening both the natural context limit and decay interval.
  • B RoPE Product Can Be Seen as a Normal Variable: Under a uniform-distance assumption and sufficiently large intervals, the high-frequency RoPE product behaves heuristically like a normal variable.The approximation uses pseudo-independence, negligible covariances, and Lindeberg’s central limit theorem when no frequency term dominates.
  • B RoPE Product Can Be Seen as a Normal Variable: The normal approximation is empirically usually adequate when λ(M) > 20; for h = 64, this corresponds to a stated lower bound on M.The supplied passage truncates before giving the numerical lower bound on M.
  • B.1 RoPE Scaling: The analysis extends to most RoPE scaling variants, including NTK scaling, by analyzing frequency independence and modifying λ(M) so that Mω_λ(M) = Θ(1).The treatment assumes angular frequencies are polynomial functions of θ^n and addresses dependent terms through negligible covariances.

C.1 Position Inversion

This section defines position inversion as a farther key receiving a higher RoPE attention score than a nearer key. It shows that inversion becomes increasingly likely with context length and RoPE base, approaching random-guessing probability under long-context conditions.

  • Definition: Position inversion occurs when a uniformly sampled farther position m2 receives a higher score than a nearer position m1.The positions satisfy m1 ∈ [0, M/2) and m2 ∈ [M/2, M).
  • Probability estimate: The inversion probability is estimated by modeling the difference between RoPE products as the difference of independent normal variables.The resulting expression uses Φ, the cumulative distribution function of the standard normal distribution.
  • Asymptotic behavior: As log M or log B increases, the lower bound on position-inversion probability increases.The estimate assumes the sequence {a_n} is sufficiently regular and depends on the actual q, k values for more accurate calculation.
  • Asymptotic behavior: As M → Θ(B), the probability of position inversion approaches 1/2.When the relevant means become equal for sufficiently large M, the proof obtains Pr = 0.5.
  • RoPE-base tradeoff: Increasing B raises inversion probability and decreases the maximum context length compatible with a threshold constraint Pr < α.The section states that this relationship is illustrated in Table 3 for h = 64 and threshold probability 0.3.

C.2 Position Aliasing … D.1.1 Aliasing Probs for FP16 in Case Study

The paper formalizes position and token aliasing and token inversion as RoPE failure modes, derives datatype-sensitive resolution limits, and evaluates them in a Llama3.1-8B case study using FP16. The analysis shows that finite numerical precision can make distinct positions or tokens produce indistinguishable attention behavior.

  • C.2 Position Aliasing: Position aliasing occurs when distinct distances m1 and m2 yield identical numerical RoPE products under a given datatype.This definition captures indistinguishable positions through equal computed attention scores.
  • C.2 Position Aliasing: 5.6‰ at 32k context length is the FP16 positional-aliasing probability for a random pair with RoPE base 10,000.Increasing the base to 100,000 raises this probability to 6.5‰; across O(M^2) position pairs, this implies 3.5 million aliased pairs.
  • C.3 Token Inversion: Token inversion occurs when one key outranks another at distance 0 but reverses its ordering at some positive distance m.The analysis compares a key token with a hypothetical prime token that attains the maximum RoPE product at m = 0.
  • C.4 Token Aliasing: Token aliasing occurs when two distinct key vectors produce identical numerical RoPE products for a query at some position m.The probability analysis models the difference between the two RoPE products as a normal variable under fixed-head magnitude assumptions.
  • D.1 Case Study: The case study uses Head 0, Layer 0 of Llama3.1-8B, while the method is stated to apply to any head and layer in RoPE-based decoder transformers.For each failure mode, the study computes S(m) across the context range using repeated key tokens and the selected head’s final attention-score row.
  • D.1.1 Aliasing Probs for FP16 in Case Study: FP16 uses 10 explicit fraction bits, producing aliasing probabilities distinct from BF16 for the case study.Position-aliasing results are reported in Fig. 10.

D.2 The Indexing Task

The indexing task evaluates whether models can retrieve arr[i] from randomly generated integer lists across increasing context lengths. It also reports accuracy variability across repeated lists and query sessions, alongside position- and token-aliasing analyses.

  • Task setup: Lists contain integers 0–3 and have powers-of-two lengths from 4 to 4096, with roughly 3 input tokens per element.The model must answer the value at a specified index arr[i].
  • Aliasing analysis: Token-aliasing probability for keys “cat” and “dog” with query “pet” converges close to 0.006 in Llama3.1-8B Layer 0 Head 0.Figure 11 reports the distribution and probability of token aliasing.
  • Evaluation: Each list is tested through 10 independent query sessions with random indices, and results report mean accuracy plus mean standard deviation across lists.The study uses 10 randomly generated lists and averages input tokens and accuracy across sessions.
  • Evaluation: The evaluation prompt presents the array and requests the indexed value directly in the form arr[{key}] =.Models are instructed not to think and to answer concisely.
  • Models: The study evaluates smaller models below 10B and larger models above 100B, including Llama-3.1-8B-Instruct, DeepSeek-V3.1, Kimi-K2.5, and gpt-oss-120b.The smaller-model group also includes Mistral-7B-Instruct-v0.3 and Qwen3-8B.

E RoPE in Real Models · F Related Works · A War of Increasing RoPE Base

The paper argues that real-model redundancy offers limited protection against RoPE failures, while related work motivates increasing the RoPE base to extend context but frames a trade-off between positional and token-wise accuracy. As context grows, oscillation and decay increasingly impair the distinction of distant positions and tokens.

  • E RoPE in Real Models: RoPE failures may persist in real models because multiple heads and layers provide only limited, preliminary protection.The discussion is explicitly open-ended and intuitive rather than evidence-based.
  • E RoPE in Real Models: Specialized, sparsely activated attention heads limit redundancy when retrieving information across different dependency ranges.Prior work reports that heads handle different text-dependency ranges and that only a small number are active simultaneously.
  • E RoPE in Real Models: Residue connections limit layer redundancy because errors introduced early propagate through later layers in series.Parallel residue connections may reduce accumulated error, but final-layer errors cannot be calibrated before producing an output token.
  • E RoPE in Real Models: As context increases, oscillation reduces positional uniqueness while decay compresses token-wise differences, impairing distant text representation.Faraway positions and token identities become increasingly likely to be poorly distinguished, causing weak or erroneous contributions to context-aware representations.
  • F Related Works: Long-context language models increasingly support context windows well beyond 1 million tokens, motivating improvements in data, training, and deployment.The related-work discussion identifies these areas alongside architectural strategies for delivering and utilizing long-context models.
  • A War of Increasing RoPE Base: RoPE’s decaying attention effect models weaker long-distance dependencies but is widely believed to limit long-context performance.The decay is purposefully designed for natural language, yet prior work identifies it as at least a partial limitation.
  • A War of Increasing RoPE Base: Increasing the RoPE base slows decay, while partially removing RoPE or restricting it to selected dimensions is analyzed as an infinitely large base.These approaches are presented as common strategies for extending context length.
  • A War of Increasing RoPE Base: Prior analyses characterize RoPE products through distance-dependent oscillation and decay, while the long-context dilemma frames uncertainty as a choice between positional and token-wise accuracy.Signal-processing analysis introduces uncertainty involving numeric precision and the Nyquist Limit.
Loading 2605.15514v1…