PaperlayerView exact revision on arXiv

Source-linked AI summary

DoPE: Denoising Rotary Position Embedding

Jing Xiong, Liyang Fan, Hui Shen, Zunhai Su, Min Yang, Lingpeng Kong, Ngai Wong

arXiv:2511.09146v2cs.CL

TL;DR

The paper investigates how RoPE interacts with implicit positional bias to produce massive activations and attention sinks. It analyzes low-frequency alignment and introduces training-free, entropy-based selective denoising; in the reported 24k setting, DoPE-by-Gaussian improves on the baseline.

  • Problem

    How explicit RoPE interacts with causal attention’s implicit positional bias to shape massive activations remains poorly understood.

    Source: §1 · p. 1

  • Method

    DoPE uses truncated matrix entropy to identify heads for selective positional-encoding suppression and reparameterizes their attention maps with an isotropic Gaussian.

    Source: §1 · p. 1

  • Results

    At 24k tokens, DOPE-by-Gaussian improves from the 75.417 baseline to 84.354.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The paper treats instability as head-specific, intervening only where attention maps become noise-dominated rather than globally modifying or removing RoPE.

    Source: §5 · p. 8

    Head selection adds computation, entropy-based separation may not hold across all models, layers, or domains, and evaluation is limited to long-context inference tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Positional encoding is essential for large language models (LLMs) to represent sequence order, yet recent studies show that Rotary Position Embedding (RoPE) can induce massive activation. We investigate the source of these instabilities via a spectral analysis of RoPE, and show that its low-frequency components concentrate structured energy, producing low-rank, over-aligned attention patterns. We theoretically reveal that this low-frequency alignment manifests as activation noise, degrading stability during long-context extrapolation. To mitigate this effect, we introduce Denoising Rotary Position Embedding (DoPE), a training-free method that identifies and suppresses noisy attention heads using truncated matrix entropy, then reparameterizes their attention maps with an isotropic Gaussian distribution. Across a range of settings, DoPE improves length extrapolation performance without fine-tuning, increases robustness to perturbations, and boosts both needle-in-a-haystack and many-shot in-context learning tasks. These results suggest that selective positional encoding is key to robust extrapolation. Our project page is Project: https://The-physical-picture-of-LLMs.github.io

1 Introduction

RoPE is widely used for relative-position encoding, but how it interacts with causal attention’s implicit positional bias to shape massive activations remains poorly understood. The paper analyzes that interaction and proposes selectively denoising RoPE heads using truncated matrix entropy.

1 Introduction

  • The paper frames the open question as how explicit RoPE interacts with implicit positional bias and shapes massive activations.Prior work links implicit positional encoding to massive activations and the attention-sink phenomenon, but the interaction with RoPE remains poorly understood.Source: §1 · p. 1
  • Alternatives that learn attention-map feature maps or remove positional encoding altogether have challenged the necessity of explicit positional encoding.The paper investigates how RoPE injects massive activations across heads and introduces structured attention-map noise associated with attention sinks.Source: §1 · p. 1
  • DoPE is a training-free scheme that selectively suppresses positional encoding and theoretically explains how it shapes massive activation and attention sinks.The approach treats attention maps as noisy feature maps, using truncated matrix entropy to identify heads and an isotropic Gaussian to reparameterize selected maps.Source: §1 · pp. 1–2
  • RoPE’s low-frequency alignment is linked to long-range dependency heads, while intrinsically low-rank extrapolative heads benefit from preserving positional encoding.The paper presents these as distinct head behaviors motivating selective, rather than uniform, positional encoding.Source: §1 · p. 2

2 Related Work

The related work covers RoPE-based length extrapolation and approaches that rely on causal attention without explicit positional encodings.

2.1 Length Extrapolation with RoPE

  • RoPE is widely adopted and often extrapolates well, but performance can degrade when sequences exceed training length.Inference-time extensions include interpolation and NTK-based adjustments to positional scaling or the frequency spectrum, alongside continuous formulations.Source: §2.1 · p. 2

2.2 Length Extrapolation without Explicit Positional Encoding

  • NoPE argues that the causal mask supplies relative-position cues, though prior work also questions whether implicit cues suffice for robust length generalization.Other studies propose that positional information can emerge through embedding similarity.Source: §2.2 · p. 2

3 Denoising Rotary Position Embedding

The method connects low-frequency RoPE alignment with spectral amplification and attention sinks, then selectively denoises heads using truncated matrix entropy. It leaves other heads on standard extrapolation and offers both frequency-level and head-level positional-encoding interventions.

3.1 Preliminary

  • RoPE rotates queries and keys so attention logits depend on relative position, whereas NoPE removes explicit positional encoding but requires training from scratch.The section introduces both as background for a hybrid approach.Source: §3.1 · p. 3

3.2 Spectral Amplification of RoPE Bands

  • Under a narrow-cone condition, low-frequency RoPE bands align directions, amplify band-wise Gram structure, and reinforce stable directions into massive activations.The paper connects this spectral amplification to low-frequency contributions to attention sinks.Source: §3.2 · pp. 3–4

3.3 Denoising via Truncated Matrix Entropy

  • Heads with low truncated matrix entropy are selected for denoising, while other heads remain extrapolative and follow standard dynamic-NTK extrapolation.A quantile threshold determines selection; only the low-entropy heads undergo denoising.Source: §3.3 · p. 4
  • For selected heads, DoPE-by-parts removes corresponding low-frequency components from query and key matrices, while DoPE-by-all masks head-level positional encoding.The parts variant uses a phase threshold approximating the low-frequency region under the cone condition.Source: §3.3 · p. 4
  • DoPE-by-Gaussian fully masks positional encodings in denoised heads and reparameterizes their attention maps with an isotropic Gaussian distribution.The Gaussian noise terms for queries and keys are specified as samples from N(0, σ^2I).Source: §3.3 · p. 4

4 Experiment

DoPE is evaluated on needle retrieval and many-shot in-context learning, including noisy and extended-context settings. The experiments examine how entropy-based head selection relates to attention sinks and low-rank structure.

4.1 Experimental Setup

  • The experiments cover needle retrieval at 24K and 64K under original and noisy conditions, and MATH-based MICL at 8K and 16K.The noisy retrieval setups insert attention-sink tokens near the needle; MICL includes both standard and needle-insertion variants.Source: §4.1 · p. 5, §4.3 · p. 7
  • Head selection varies the number of selected heads, entropy-computation stage and representation, sorting direction, and calibration sequence length.The reported tables include MICL extrapolation from 4K to 16K and 64K extrapolation using heads selected at several sequence lengths.Source: §4.1 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6

4.2 Main results

  • At 24K, DOPE-by-Gaussian improved performance from 75.417 to 84.354, while introducing attention-sink tokens sharply degraded performance.Source: §4.2 · p. 6
  • Truncated matrix entropy typically outperformed vanilla matrix entropy; at 64K, r = 1 yielded the best results, while cross-length calibration generally underperformed same-length calibration.The authors report that r = 1 can be regarded as equivalent to the spectral norm and that sparser settings sharpen the singular-value distribution.Source: §4.2 · pp. 6–7

4.3 Many-shot In-Context Learning

  • MICL reasoning improved at appropriate lengths but dropped markedly at 16K, and inserting test exemplars caused substantial accuracy drops at 24K and 64K.Dual Chunk Attention and Positional Interpolation achieved accuracies close to zero; sparse patterns estimated from synthetic tasks transferred to more complex reasoning tasks.Source: §4.3 · p. 7

4.4 Matrix Entropy Meets Attention Sink

  • Figure 2 links truncated-matrix-entropy selection to attention sinks: low-entropy heads often showed recency bias, while remaining high-entropy heads attended to the needle.With vanilla matrix entropy, high-entropy heads corresponded to severe sinks, and low-entropy heads still failed to attend to the needle position.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 RoPE Induces Low-rankness

  • Entropy-selected heads showed low-rank structure, with similarity mass concentrated in a few dimensions; the truncated-entropy-selected head also showed periodicity that vanilla entropy did not reliably distinguish.The analysis visualized token-query cosine similarity against eigenvectors spanning 128 dimensions.Source: §4.5 · p. 8

5 Conclusion

The conclusion frames RoPE’s low-frequency components as structured amplifiers that promote over-aligned, low-rank attention and undermine stability as context grows. DoPE responds with head-specific intervention, and the reported perturbation gains point to controlling positional information across heads rather than choosing one encoding scheme.

5 Conclusion

  • RoPE’s low-frequency components can concentrate energy into over-aligned, low-rank attention patterns that undermine stability as context grows.The conclusion describes these components as structured amplifiers.Source: §5 · p. 8
  • DoPE treats instability as head-specific, intervening only where attention maps become noise-dominated rather than modifying or removing RoPE globally.Source: §5 · p. 8
  • DoPE’s gains under perturbation suggest that long-context robustness is less about one positional encoding scheme and more about controlling positional information across heads.Source: §5 · p. 8

Limitations

DoPE has computation overhead from head selection, relies on entropy criteria to distinguish noise-dominated heads, and is evaluated only on long-context inference tasks.

Limitations

  • Head selection adds computation to DoPE.Source: Limitations · p. 9
  • Entropy-based criteria may not reliably separate noise-dominated heads across all models, layers, or domains.Source: Limitations · p. 9
  • The evaluation focuses on long-context inference tasks, leaving generalization to broader settings unvalidated.Source: Limitations · p. 9

A.1 Proofs

Appendix A.1 gives an entry-level lower bound for a rectangular matrix in terms of its largest singular value and states its square-matrix specialization.

A.1 Proofs

  • Lemma A.1 is an entry-level lower bound for a rectangular matrix Σ with largest singular value σ1(Σ).Source: §A.1 · p. 12
  • The indices i and j denote the rows and columns of Σ, respectively.Source: §A.1 · p. 12
  • The proof invokes the Frobenius/spectral norm relation to establish the rectangular entry-level bound.Source: §A.1 · p. 12
  • For a square matrix, the maximum absolute entry is at least the largest singular value divided by N.Here N denotes sequence length when the matrix is formed over N tokens.Source: §A.1 · p. 12

A. Massive Activation in Band-wise Representations

The section links coherent rotations in low-frequency RoPE bands to norm amplification, under a cone condition on projected keys. It frames this result using band-wise Gram matrices and sequence length N.

A. Massive Activation in Band-wise Representations

B. Attention Sink Amplification in Band-wise Attention Logits

Under query and key cone constraints, coherent multiplication can amplify band-wise attention logits and concentrate attention mass on a few entries, producing an attention sink. The lower bound on the largest entry remains Ω(1) as sequence length grows.

B. Attention Sink Amplification in Band-wise Attention Logits

  • The leading singular-value bound scales with √dh N, query and key norms, their cone-alignment factors, and cos ψ.ψ is the angle between the dominant singular directions of the query and key representations.Source: B. Attention Sink Amplification in… · p. 13
  • The maximum absolute attention entry has a lower bound proportional to the query and key norms and alignment factors, remaining Ω(1) as sequence length grows.The entry-level bound follows from Lemma A.1.Source: B. Attention Sink Amplification in… · p. 13
  • When both query and key bands satisfy the cone constraint, coherent multiplication yields an O(N) scaled singular value and concentrates attention mass on a few entries—the attention-sink pattern.The passage identifies this concentration as the attention sink.Source: B. Attention Sink Amplification in… · p. 13

C. Truncated Matrix Entropy and Spectral Amplification

Truncated matrix entropy measures energy concentration among an attention head’s leading spectral components. Under spectral amplification, this entropy decreases, making low-entropy heads a quantitative criterion for identifying potentially noisy heads in DoPE.

C. Truncated Matrix Entropy and Spectral Amplification

  • Truncated matrix entropy measures the information-weighted energy concentration of an attention head’s top-r singular components.It is defined for a head-level Gram matrix with ordered positive eigenvalues.Source: C. Truncated Matrix Entropy and… · p. 13
  • Truncated entropy decreases monotonically with stronger spectral amplification, including a larger dominant eigenvalue or smaller δ.The proof considers an entropy-maximizing distribution of the remaining trace mass across the other eigenvalues.Source: C. Truncated Matrix Entropy and… · p. 13
  • Heads with low truncated entropy correspond to spectral amplification and potential attention sinks, motivating their use as DoPE’s noisy-head criterion.The discussion links this pattern to a dominant spectral direction and suppressed remaining eigenvalues.Source: C. Truncated Matrix Entropy and… · p. 13

A.2 Experimental Setup

The experiments evaluate RoPE-based extrapolation and training-free context-extension methods on needle-in-a-haystack and many-shot in-context learning tasks. They vary task lengths, insertion depths, and DoPE entropy settings.

A.2 Experimental Setup

  • The setup uses Qwen2.5-Math-7B and LLaMA-3-8B-Instruct, applying RoPE-based extrapolation such as Dynamic-NTK beyond pretraining limits.The passage also describes the models as decoder-only transformers using RoPE.Source: §A.2 · p. 13
  • Experiments use greedy decoding; head selection is global across all attention heads, with five A100 GPUs used for inference.The setup gives 1,024 total heads for LLaMA-3-8B and 784 for Qwen2.5-Math-7B.Source: §A.2 · pp. 13–14
  • NIH tests needle retrieval at 10 evenly spaced depths over 24K, 64K, and 128K contexts; MICL evaluates 100 MATH problems at four depths, yielding 400 configurations.MICL uses a 16K target context; the listed task depths include the beginning, one-third, two-thirds, and end.Source: §A.2 · p. 14
  • DoPE uses Gaussian noise with σ = 1.0 and evaluates top-k entropy for k ∈ {1, 4, 8, 16, 32} alongside full matrix entropy.The Gaussian sampling uses fixed seed 42.Source: §A.2 · p. 14
  • Compared training-free extensions include Dynamic-NTK frequency adjustment, Dual Chunk Attention’s local chunks and cross-chunk routing, and positional interpolation.Positional interpolation rescales input positions into the training range before RoPE.Source: §A.2 · p. 14
Loading 2511.09146v2…