PaperlayerView exact revision on arXiv

Source-linked AI summary

Round and Round We Go! What makes Rotary Positional Encodings useful?

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković

arXiv:2410.06205v3cs.CLcs.LG

TL;DR

RoPE is widely used in LLMs, but its usefulness is often attributed to distance-related attention decay despite limited understanding of how it is used. The paper combines theoretical analysis and experiments on trained models to investigate RoPE’s frequency-specific roles. It reports that high frequencies support positional attention, low frequencies are associated with semantic attention, and removing some low frequencies can maintain or improve performance in Gemma 2B experiments.

  • Problem

    Why RoPE is useful remains poorly understood, and its commonly cited distance-decay rationale relies on query-key conditions that are uncommon in practice.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    The authors analyze pretrained Gemma 7B empirically and theoretically, examine frequency use, and test p-RoPE, which removes a fraction of the lowest frequencies.

    Source: §1 · p. 2, §6.1 · p. 9

  • Results

    Gemma 7B uses high-frequency bands for positional attention and low-frequency bands for semantic attention; truncating low frequencies maintains performance and improves it on 2-billion-parameter models.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The authors present a more nuanced account of RoPE and suggest that removing low-frequency rotations could improve semantic attention and its generalization to longer contexts.

    Source: §7 · p. 10

    The authors could not validate the proposed long-context benefit at larger context lengths, noting that their 8k context was not very large and that wavelength-increase gains were typically observed at 32k.

    Source: §6.1 · p. 10

Abstract

from arXiv · show

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

1 INTRODUCTION

RoPE is widely used in LLMs, but why it helps remains poorly understood. The paper examines its frequency-specific use, arguing that attention decay is not the core explanation and identifying distinct roles for high and low frequencies.

1 INTRODUCTION

  • Gemma 7B uses RoPE’s highest frequencies to construct robust positional attention heads, a behavior the authors also construct mathematically.The construction makes a key align with a query at a chosen relative position, yielding lower attention weights for other positions.Source: §1 · p. 2, Figure 1 · p. 2
  • Gemma 7B largely prefers RoPE’s lowest frequencies, while the first and last layers show the most use of high frequencies.The authors conjecture that low-frequency bands serve as semantic information channels.Source: §1 · p. 2
  • The paper challenges the common claim that RoPE helps decay attention coefficients with relative distance, noting many situations where decay does not occur and Gemma 7B attention heads sometimes exploit this.Source: §1 · p. 1
  • The authors propose p-RoPE, which removes some lowest frequencies, and report maintained performance and improvements on 2-billion-parameter models.They suggest this work may help explain why increasing RoPE’s maximum wavelength helps with long context.Source: §1 · p. 2

2 BACKGROUND

The background defines causal attention and NoPE, describes RoPE’s frequency-specific rotations, and reviews competing explanations and modifications for positional information and long-context generalization.

2 BACKGROUND

  • Causal attention assigns zero weight to future tokens and normalizes attention over prior tokens; NoPE uses a dot product without positional information.The cited work argues that causal masking can in principle recover absolute positions, though the authors note its proof relies on universal approximation.Source: §2 · p. 3

2.1 ROTARY POSITIONAL ENCODINGS (ROPE)

  • RoPE rotates 2D query and key chunks at distinct frequencies, from 1 radian per token at the fastest component to approximately 1/θ at the slowest.The base wavelength θ is 10,000 by default, with cited work exploring 500,000.Source: §2.1 · p. 3

2.2 RELATED WORKS

  • Figure 2 contrasts activation decay for all-ones queries and keys with its absence for queries and keys containing IID Gaussian entries.This motivates the paper’s challenge to decay-based explanations of RoPE.Source: Figure 2 · p. 4
  • Increasing θ is commonly justified as slowing RoPE-induced attention decay, while the authors question this explanation for modifications such as a base wavelength of 500,000.The cited motivation is more robust learning over larger contexts.Source: §2.2 · p. 4
  • Related work reports NoPE’s out-of-distribution strengths and randomized RoPE positions’ long-context benefits; the authors argue RoPE and NoPE have complementary strengths and weaknesses.They also frame their analysis as a mechanistic perspective on positional encodings.Source: §2.2 · p. 4

3 DOES ROPE DECAY ACTIVATIONS WITH DISTANCE?

The section challenges the claim that RoPE generally makes attention activations decay with relative distance. Its propositions and synthetic experiments show that decay depends on query-key conditions and can be absent.

3 DOES ROPE DECAY ACTIVATIONS WITH DISTANCE?

  • RoPE can make the attention weight largest at any chosen relative distance, so distance alone does not ensure decay.The paper gives a construction for any query and selected distance, and links this mechanism to Gemma 7B’s positional attention heads.Source: §3 · pp. 4–5
  • For independently sampled standard Gaussian queries and keys, expected activation is zero and independent of relative distance.This contradicts a general expectation of distance-driven activation decay.Source: §3 · p. 5
  • In a synthetic experiment, activation decay appears for all-ones queries and keys but not for random Gaussian vectors.The authors note that the all-ones setup assumes aligned queries and keys, which they consider an unrealistic oversimplification.Source: §3 · pp. 4–5
  • The authors conclude that RoPE produces distance-related decay only under specific conditions, which Gemma 7B exploits to create specific attention patterns.They connect the absent decay cases to the positional-attention mechanism discussed in the following section.Source: §3 · p. 5

4 HOW ARE DIFFERENT FREQUENCIES USED?

Gemma 7B concentrates query and key usage in RoPE’s lowest frequencies, with high-frequency use standing out in the first and last layers and in specific heads. The section also cautions that high attention scores are not settled evidence of meaningful preference.

4 HOW ARE DIFFERENT FREQUENCIES USED?

  • The highest RoPE frequencies rotate rapidly and are sensitive to small token rearrangements, making their use in LLMs a question this section investigates.The authors describe their contributions to the dot product as behaving similarly to random noise.Source: §4 · p. 5
  • Gemma’s query and key usage favors low frequencies; the corresponding distribution does not occur in value vectors, which are not rotated.The authors suggest learning suppresses medium-to-high-frequency chunks in queries and keys, reducing their impact on the dot product.Source: §4 · pp. 5–6
  • Whether high attention scores imply a meaningful preference remains under debate.The authors flag this as a qualification when interpreting their attention-based findings.Source: §4 · p. 5
  • Across layers, query and key chunk norms are higher at lower RoPE frequencies, with high-frequency use especially visible in the first and last layers.The mean is over 16 heads per layer; chunk norms serve as an upper bound on each frequency’s effect on the activation dot product.Source: §4 · p. 5, Figure 3 · p. 6
  • In the first layer, Heads 5 and 8 stand out for high-frequency use, especially in keys, and correspond to positional heads.The frequency usage is sparse, with high-norm bands; the authors suggest these bands may support sparse query-key semantic matching.Source: §4 · p. 6, Figure 4 · p. 6

5 HIGH FREQUENCIES: POSITIONAL ATTENTION

The section shows that RoPE can construct sharp positional attention patterns that NoPE cannot, and finds that Gemma’s positional heads rely on high frequencies. It connects this behavior to a construction in which rapid rotations distinguish relative positions.

5 HIGH FREQUENCIES: POSITIONAL ATTENTION

  • Gemma exhibits purely positional heads, including a diagonal head at the last layer and a previous-token head at the first layer.These heads attend based on relative position without regard to semantics; the reason for the diagonal pattern is outside the work’s scope.Source: §5 · p. 7, Figure 5 · p. 7
  • The authors leave the usefulness of diagonal attention unresolved and suggest it may be a training bug requiring further analysis.They note that a purely diagonal pattern can behave like a residual connection.Source: §5 · p. 7
  • The diagonal head mostly relies on the highest frequencies, and high-frequency usage consistently helps identify positional heads in Gemma.The first-layer Heads 5 and 8 correspond to a diagonal head and a previous-token head, respectively.Source: §5 · p. 7, Figure 6 · p. 7
  • The RoPE construction sets queries and keys equal for diagonal attention and inverse-rotates them by one unit for off-diagonal attention.The highest frequencies misalign query and key rotations fastest as relative distance changes, making their dot product position-sensitive.Source: §5 · p. 8
  • RoPE can learn diagonal or off-diagonal positional attention patterns, whereas NoPE cannot learn these patterns.The paper summarizes this contrast as a mechanism for constructing positional attention patterns with high frequencies.Source: §5 · p. 8

6 LOW FREQUENCIES: SEMANTIC ATTENTION

Gemma 7B uses low RoPE frequencies for semantic attention, while high frequencies can help identify positional patterns. The section shows why low-frequency channels may become unreliable at long contexts and evaluates truncating them with p-RoPE.

6 LOW FREQUENCIES: SEMANTIC ATTENTION

  • Low-frequency rotations minimally affect dot products, making them useful for semantic attention, but even these frequencies eventually misalign over long relative distances.The section argues that increasing RoPE’s base wavelength may be useful as contexts grow.Source: §6 · p. 8
  • In Figure 7, a Gemma 7B head uses high frequencies to detect an apostrophe in the previous token and low frequencies to attend to BOS otherwise.The example combines semantic detection with a positional condition.Source: §6 · pp. 8–9, Figure 7 · p. 9
  • With one RoPE frequency, sufficiently long sequences prevent a head from assigning attention arbitrarily close to 1 to any target token.The authors attribute this limitation to irrational rotations becoming dense; their argument uses the two-dimensional, single-frequency case.Source: §6 · p. 9

6.1 TRUNCATING THE LOWEST FREQUENCIES OF ROPE

  • p-RoPE truncates the lowest RoPE frequencies, with p denoting the fraction kept; p=0 is NoPE and p=1 is RoPE.The authors investigate whether removing the least robust semantic channels can preserve performance while providing distance-agnostic semantic channels.Source: §6.1 · p. 9
  • On Gemma 2B validation, truncating the lowest frequencies maintains or improves performance; 0.25-RoPE harms it but remains better than NoPE.The authors also report p-RoPE outperforming a 500k wavelength, 0.75-RoPEreversed, and RoPEpartial; Table 2 gives the validation perplexity comparison.Source: §6.1 · p. 10, Table 2 · p. 10
  • The ablation uses an 8k context, and the authors lacked resources to test larger contexts, leaving long-context generalization unvalidated.They expect that removing a small percentage of the lowest frequencies may help, but frame this as a future direction.Source: §6.1 · p. 10

7 CONCLUSION

The conclusion challenges distance-decay as a sufficient account of RoPE and describes how Gemma uses frequency bands for positional and semantic attention. The appendix proofs include constructions showing that RoPE can target arbitrary relative distances and that Gaussian queries and keys need not produce distance decay.

A.1 DOES ROPE DECAY WITH DISTANCE?

  • The conclusion also states that attention need not decay with distance when queries and keys are Gaussian random vectors.The appendix notes that a standard multivariate Gaussian is invariant to rotations.Source: §7 · p. 10, §A.1 · p. 14

7 CONCLUSION

  • High-frequency bands support circuitry for positional attention in Gemma 7B, while NoPE cannot produce similar positional attention.The authors identify a construction in Gemma 7B resembling their theoretical construction.Source: §7 · p. 10
  • Low frequencies support semantic attention, and removing their rotation may improve semantic attention and its generalization to longer contexts.The authors report verifying the modification through a Gemma 2B ablation.Source: §7 · p. 10

A PROOFS

  • The appendix supplies omitted proofs and further discussion of the paper’s theoretical statements and constructions.It identifies sections containing proofs for the main text’s statements.Source: A PROOFS · p. 13

A.1 DOES ROPE DECAY WITH DISTANCE?

  • For any non-zero query and any relative distance, a RoPE key can be constructed so that attention is largest at that distance.The proof constructs the key by rotating the query in a way that makes the target distance maximize the activation.Source: §A.1 · p. 13

A.2 POSITIONAL ATTENTION PATTERNS.

RoPE can construct diagonal or off-diagonal attention patterns that an isolated NoPE head cannot, while single-frequency RoPE can lose near-exclusive target attention on sufficiently long sequences.

A.2 POSITIONAL ATTENTION PATTERNS.

  • RoPE can learn diagonal or off-diagonal attention patterns, whereas an isolated NoPE head cannot; Gemma’s diagonal construction is robust with chunk norms below 10.The authors note that softmax can make the construction robust without very large norms.Source: §A.2 · pp. 14, 16

A.3 SEMANTIC ATTENTION.

  • With d=2 and one RoPE frequency, sufficiently long sequences defeat near-exclusive target attention; token swaps can make a non-target activation win.The proof uses the requirement that a softmax activation must be the largest to receive an attention coefficient above 1/2.Source: §A.3 · pp. 16–17

B.1 ADDITIONAL RELATED WORKS

  • The authors identify an apostrophe head and describe their focus as explaining how RoPE implements attention heads, not only identifying their functions.They say, to their knowledge, the apostrophe head is the first of its kind identified.Source: §B.1 · p. 17

B.2 OTHER POSITIONAL ENCODINGS

  • The authors suspect their frequency findings may transfer to sinusoidal APE, which adds positional signals rather than applying RoPE’s linear map.They leave further analysis of this difference for future work.Source: §B.2 · pp. 17–18
  • Unlike RoPE, Alibi does decay attention coefficients with distance; analyzing Alibi similarly is proposed as future work.Source: §B.2 · p. 18

B.3 RANDOMISED POSITIONAL ENCODINGS

  • The authors hypothesize that randomized positions push models toward the lowest RoPE frequencies by encouraging soft invariance to relative distance, but leave testing this hypothesis to future work.Source: §B.3 · p. 18
  • Increasing the position-sampling bound L speeds decay for all-ones vectors but does not alter activation patterns for Gaussian-sampled vectors.The authors attribute the increased decay in the all-ones case to larger average relative distances.Source: §B.3 · p. 18, Figure 8 · p. 18

B.4 ADDITIONAL RESULTS ON SYNTHETIC ACTIVATION DECAY

A fixed Gaussian query and key pair shows no clear RoPE-induced decay.

B.4 ADDITIONAL RESULTS ON SYNTHETIC ACTIVATION DECAY

  • Fixed Gaussian random query and key vectors show no clear decay, supporting the finding that RoPE decay is not general to Gaussian vectors.Source: §B.4 · p. 19, Figure 9 · p. 19

B.5 LIMITATIONS OF OUR WORK

The authors note that their mechanistic account may be incomplete and that p-RoPE’s evaluation remains preliminary.

B.5 LIMITATIONS OF OUR WORK

  • The frequency account may not cover all RoPE use: middle-band norms are small but nonzero and could still matter.The authors also call for larger-scale experiments on low-frequency removal, particularly for large contexts, while noting their greater resource cost.Source: §B.5 · p. 19
  • p-RoPE was evaluated by training 2B-parameter Gemma models from scratch and measuring validation perplexity, which need not predict downstream performance.The authors leave more careful evaluation for future work.Source: §B.5 · p. 19

C LLAMA3.1 8B

Llama 3.1 8B shows frequency-use patterns similar to those reported for Gemma, including slower-frequency query/key activity and a high-frequency diagonal head.

C LLAMA3.1 8B

  • In Llama 3.1 8B, queries and keys show higher activation at slower RoPE frequencies, with lower frequencies preferred; values lack a comparable norm distribution, consistent with RoPE not being applied to them.Source: C LLAMA3.1 8B · pp. 19–20, Figure 10 · p. 20
  • A highlighted head in Llama 3.1 8B’s final layer uses high frequencies prominently and is a diagonal attention head.Source: C LLAMA3.1 8B · p. 20

D SUPPLEMENTARY p-ROPE EXPERIMENTAL DETAILS AND RESULTS

The supplement compares p-RoPE with RoPEpartial and reports that p-RoPE performs better. It also details Gemma 2B training experiments and additional Gemma 7B analyses.

D SUPPLEMENTARY p-ROPE EXPERIMENTAL DETAILS AND RESULTS

E SUPPLEMENTARY GEMMA 7B RESULTS

  • The Gemma 7B supplement provides evidence about learned attention constructions and plots queries, keys, and attention patterns for heads studied in the main text.It presents construction evidence in E.1 and the head plots in E.2.Source: E SUPPLEMENTARY GEMMA 7B RESULTS · p. 22

E.1 CONSTRUCTIONS LEARNT BY GEMMA 7B

The supplement examines how Gemma 7B heads use RoPE frequencies to form attention patterns. It reports low-frequency semantic behavior and query-key alignment in diagonal and previous-token heads.

E.1 CONSTRUCTIONS LEARNT BY GEMMA 7B

  • Diagonal heads have query-key activations close to the Cauchy-Schwarz upper bound, while previous-token activations also approach it but less closely.The authors attribute the larger gap for the previous-token head to its need to model a more complicated block-diagonal rotation.Source: §E.1 · p. 22, Table 3 · p. 23, Table 4 · p. 23
  • The apostrophe head defaults to attending to BOS, but an apostrophe token can dominate; the low-frequency channel is stable at Gemma 7B's context length but is expected to break on sufficiently long sequences.The channel contributes positive activations for BOS and negative activations for non-BOS tokens.Source: §E.1 · p. 24
  • The apostrophe head's low-frequency semantic channel distinguishes BOS from non-BOS tokens, while apostrophe tokens can override its BOS default.At the lowest band described, the maximum rotation over an 8k context is about 1.64 radians.Source: §E.1 · p. 24

E.2 ANALYSIS OF PATTERNS IN DIFFERENT ATTENTION HEADS

  • The query, key, and value norm plots show frequency patterns in queries and keys but no clear corresponding pattern in values, where RoPE is not applied.The figure caption presents this as support for attributing the query and key patterns to RoPE.Source: Figure 13 · p. 25
  • The plotted first- and last-layer diagonal heads implement residual connections; the other plots show previous-token and apostrophe attention patterns.The previous-token head attends to the immediately preceding token, and the apostrophe head directs tokens after an apostrophe to it.Source: Figure 14 · p. 26, Figure 15 · p. 27, Figure 16 · p. 28
  • A semantic head's queries show high-norm bands at low frequencies, and query-frequency usage is very similar across the illustrated input domains.The domains shown include Italian, Chinese, Python code, and arithmetic additions.Source: Figure 18 · p. 30, Figure 19 · p. 31
Loading 2410.06205v3…