PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs

Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Zhaoxiang Liu, Shiguo Lian, Ziwei He, Xipeng Qiu

arXiv:2512.07525v1cs.CL

TL;DR

Standard RoPE discards the imaginary component of its complex attention score, leaving a gap in how that information contributes to long-context modeling. RoPE++ restores the component as a parallel group of attention heads. The authors report stronger average performance than vanilla RoPE and other position embeddings, with larger gains in long-context settings.

  • Problem

    Standard RoPE discards the imaginary part of its complex attention score, and few works examine this information loss or the positional role of the imaginary component.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    RoPE++ reintroduces the discarded imaginary component as a group of attention heads alongside real attention while retaining the absolute–relative position-embedding format.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    At 376M and 776M model sizes, RoPE++EH and RoPE++EC outperform vanilla RoPE and other position embeddings on average, with larger gains in long-context scenarios.

    Source: §6 · p. 9

  • Takeaways & Limitations

    RoPE++ maintains more stable performance as context length grows and achieves the best performance in 64k context-length extrapolation consistently.

    Source: §4.3 · p. 7

    Real and imaginary attention must share Wq, so configurations with only imaginary attention or unequal imaginary-versus-real head allocations are not possible.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.

1 INTRODUCTION

The introduction identifies standard RoPE’s real-only attention as an information-loss limitation and proposes RoPE++ to restore the imaginary component for long-context modeling.

1 INTRODUCTION

  • RoPE encodes query and key positions so their dot product injects relative distance into attention scores.The paper describes this as combining absolute and relative position information.Source: §1 · p. 1
  • Standard RoPE discards the imaginary part of its complex attention score, causing irreversible information loss.RoPE preserves the direct equivalence between complex multiplication and vector rotation by retaining only the real part.Source: §1 · p. 1
  • Imaginary attention preferentially captures long-context information, while real attention exhibits stronger semantic locality.The paper proposes computing the imaginary component in parallel with real attention heads.Source: §1 · p. 2, Figure 1 · p. 2
  • RoPE++EH matches the head count while halving KV cache, whereas RoPE++EC matches cache size while doubling attention heads.On long-context benchmarks, EH achieves results comparable to vanilla RoPE with half the cache, while EC significantly outperforms it at the same cache cost.Source: §1 · p. 2
  • Both RoPE++ configurations outperform vanilla RoPE and other position embeddings on average across short- and long-context benchmarks.RoPE++EH uses equal head count with half the KV cache; RoPE++EC uses equal cache size with doubled attention heads.Source: §1 · p. 2

2 RELATED WORK

Prior RoPE research has largely refined scaling, interpolation, data awareness, or feature partitioning, while fewer works revisit its intrinsic computation and discarded imaginary information.

2 RELATED WORK

  • Many RoPE extensions target length extrapolation through rotary-base scaling, index interpolation or compression, and sparse attention.Other lines extend RoPE to cross-modal inputs or encode contextual cues through parametric schemes.Source: §2 · p. 3
  • Few works analyze RoPE’s intrinsic computation, and the imaginary information discarded in its rotation format remains overlooked.Although prior work incorporated full complex computation into attention or neural networks, the positional-embedding role of the imaginary component remained unexplored.Source: §2 · p. 3

3 METHODOLOGY

RoPE++ adds the discarded negative-imaginary component as a parallel attention group while retaining RoPE’s positional-embedding form. The section examines its long-distance attention pattern, resource configurations, and positional-range effects.

3.1 IMAGINARY EXTENSION OF ROPE

  • RoPE++ recovers the negative-imaginary attention component, which retains relative-position information and can be computed by rotating q_t by −π/2 before applying standard positional embedding.The resulting component preserves RoPE’s ability to express position as either relative or absolute embeddings.Source: §3.1 · pp. 3–4

3.2 CAPTURE LONGER DEPENDENCY

  • Imaginary attention retains semantic aggregation while assigning more weight to distant positions, supporting retrieval of long-context information.Its characteristic curve declines slowly beyond a certain distance, unlike the real component’s stronger semantic locality.Source: §3.2 · p. 4

3.3 CACHE AND PARAMETRIC EFFICIENCY

  • RoPE++EC doubles attention heads without adding KV cache, whereas RoPE++EH keeps head count fixed and halves QKV parameters and KV cache.RoPE++EH raises throughput in long-context scenarios; both configurations share Wq between real and imaginary attention.Source: Figure 2 · p. 5, §3.3 · p. 5

3.4 IMPACT ON LENGTH EXTRAPOLATION

  • RoPE++ exposes positional dimensions to positive and negative values; its combined heads reach the full cosine and sine range after half a sinusoidal period, versus a full period for RoPE.The authors report that perplexity grows more slowly beyond the maximum supported context length.Source: §3.4 · p. 6

4 EXPERIMENT

Across short- and long-context evaluations, RoPE++ achieves the best average performance, with especially stable results as context grows. RoPE++EH retains this edge with half the KV cache and QKV parameters.

4.1 SETUP

  • The 376M and 776M models are pre-trained for 50B tokens, then compared with position-embedding baselines trained at 4k and further trained at 32k context.The models use the DCLM-Baseline-1.0 corpus; continuous long-context pre-training uses 10B tokens.Source: §4.1 · p. 6

4.2 SHORT-CONTEXT EVALUATION

  • RoPE++EC and RoPE++EH achieve the best average scores on short-context tasks, and RoPE++EH uses half the KV-cache and QKV parameters of standard RoPE.RoPE++ retains its edge after further long-context pre-training.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 LONG-CONTEXT EVALUATION

  • RoPE++ achieves the best average performance on RULER and BABILong, with stable results and consistently best performance at 64k extrapolation.RoPE++EH is comparable to vanilla RoPE with half the KV cache and QKV parameters; RoPE++EC delivers gains at the same cache size.Source: §4.3 · pp. 6–7, Table 2 · p. 7

5 DISCUSSION

RoPE++EH reduces memory cost and speeds decoding, with a widening margin at longer contexts. Imaginary attention focuses more globally than real attention, and RoPE++ retains its advantage when combined with other long-context techniques.

5.1 ROPE++ AS CACHE OPTIMIZATION

  • RoPE++EH consistently reduces memory cost and speeds decoding at 376M and 776M, with the efficiency margin widening as context length increases.The evaluation spans 2k to 32k contexts and measures memory cost and time per output token on one NVIDIA H200 GPU.Source: §5.1 · pp. 7–8, Figure 4 · p. 8

5.2 ATTENTION PATTERN OF ROPE++

  • At σ = 1.0, real-noised RoPE++ exceeds imaginary-noised RoPE++ by 5 points at 376M and 8 points at 776M on RULER-4k.Imaginary heads show stronger global focus, while real heads focus more on local context.Source: §5.2 · pp. 8–9, Figure 5 · p. 8

5.3 COMBINATION WITH OTHER LONG-CONTEXT TECHNIQUES

  • Across 376M and 776M models, RoPE++ achieves the highest average scores on RULER, BABILong, and short-context tasks when combined with Linear PI or YaRN.The combination experiments use 32k further pre-training, with Linear PI coefficient s = 8 and YaRN coefficient s = 32.Source: §5.3 · p. 9, Table 3 · p. 9

6 CONCLUSION

RoPE++ reintroduces imaginary attention as a new group of heads while preserving the unified absolute–relative position-embedding format. At 376M and 776M, both configurations outperform vanilla RoPE on average, with larger gains in long-context scenarios.

6 CONCLUSION

  • At 376M and 776M, RoPE++EH and RoPE++EC outperform vanilla RoPE and other position embeddings on average across short-context tasks, with larger gains in long-context scenarios.Further analysis finds imaginary attention more dominant than real attention in long-context modeling.Source: §6 · p. 9
  • RoPE++ reintroduces discarded imaginary attention as a new group of heads while retaining the unified absolute–relative position-embedding format.RoPE++EH uses equal head count and halved cache; RoPE++EC uses equal cache and doubled heads.Source: §6 · p. 9
  • The paper’s mathematical analysis identifies imaginary attention as having potential for modeling long-context dependencies.Source: §6 · p. 9

ETHICAL STATEMENT

The authors report that the study processes no sensitive personal data, involves no human subjects, and targets no ethically risky applications. They state that experiments and analyses follow recognized guidelines.

ETHICAL STATEMENT

REPRODUCIBILITY STATEMENT

The paper releases RoPE++ resources and reviews how RoPE encodes positions and how its real and imaginary attention components behave. Its preliminary analysis derives characteristic curves and is explicitly expectation-based.

REPRODUCIBILITY STATEMENT

  • RoPE++, its trained checkpoints, and complete training and evaluation code are publicly released.The authors describe these resources as a reference for future long-context LLM work.Source: REPRODUCIBILITY STATEMENT · p. 10

A USE OF LARGE LANGUAGE MODELS

B PRELIMINARY: KEY PROPERTIES OF ROPE

  • RoPE rotates 2-D query and key feature pairs, combining absolute position rotations into relative-position terms in the attention matrix.Source: B PRELIMINARY: KEY PROPERTIES OF ROPE · p. 15
  • The real and imaginary attention characteristic curves derive from cosine and sine averages, respectively; the analysis is expectation-based rather than case-study-level.The real curve is positive and decaying; the authors also consider dropping rotary angles for low-frequency dimensions in both components.Source: B PRELIMINARY: KEY PROPERTIES OF ROPE · p. 16

C MORE EXPERIMENT

Experiments extend validation to a 1.5B model and examine training-token scale. RoPE++ continues to outperform RoPE across the tested model scales, while results indicate little additional gain beyond 50B tokens at this scale.

C MORE EXPERIMENT

C.1 VALIDATION ON LARGER SCALE

  • RoPE++ outperforms RoPE on short- and long-context tasks in 1.5B models trained on 50B tokens and further trained for 5B tokens at 32k context.The authors report validation across 376M, 776M, and 1.5B model scales.Source: §C.1 · p. 17, Table 5 · p. 17, Table 6 · p. 17
  • At this scale, 50B tokens are sufficient for convergence, and training beyond 50B tokens shows no significant further gain.Plateaued training loss, validation loss, and average short-context scores support the convergence judgment.Source: §C.1 · p. 17, Table 7 · p. 17, Table 8 · p. 18
  • RoPE++ and RoPE have almost overlapping loss curves without training-stability issues, although RoPE may lead early before RoPE++ ultimately surpasses it in average scores.Source: §C.1 · p. 17

C.2 MORE DETAILED RESULTS

Detailed experiments report short-context task results and throughput and storage comparisons for further-trained models. The accompanying discussion weighs RoPE++'s cache and head trade-offs against redundancy, conflict, and computation costs.

C.2 MORE DETAILED RESULTS

  • RoPEEC keeps cache size fixed but increases computation and Wo size; the authors judge this acceptable because long-context inference is primarily IO-bound and dominated by KV-cache memory cost.They plan further computation optimizations for RoPE++.Source: §C.2 · p. 18
  • The paper reports short-context task results for 776M and 376M models further trained with 5B tokens at 32k using YaRN and Linear PI, plus throughput and storage comparisons at 4k and 32k.Source: §C.2 · p. 18, Table 10 · p. 19, Table 11 · p. 19

D.1 POTENTIAL REDUNDANCY AND CONFLICT

  • RoPE++EH outperforms RoPE while using half the cache, and its real and imaginary attentions show distinct functional patterns.The authors interpret this as lower redundancy between the two components than among standard heads.Source: §D.1 · p. 18
  • RoPE++EC outperforms RoPE at identical cache size, indicating that the benefits of its potential attention conflict outweigh possible drawbacks.Source: §D.1 · p. 19

D.2 PERPLEXITY CURVE OF ROPE++

RoPE++ does not extend the stable context upper bound of short-context-trained RoPE, but beyond that point its perplexity rises more gradually. The paper also notes that the method requires training from scratch and does not provide plug-and-play length extrapolation.

D.2 PERPLEXITY CURVE OF ROPE++

  • RoPE++ cannot directly extrapolate beyond its maximum supported context length; perplexity then rises, though more gradually because query and key dimensions saw full position-value ranges during training.The described training exposure includes both positive and negative positions.Source: §D.2 · p. 19
  • Beyond the shared stable context upper bound, RoPE++ perplexity increases more slowly than RoPE's, with or without fixed-NTK interpolation at λ = 4.Figure 6 compares short-context-trained RoPE and RoPE++ at 376M and 776M model sizes.Source: §D.2 · p. 19, Figure 6 · p. 20
  • RoPE++ requires training from scratch and does not provide plug-and-play length extrapolation, falling behind FoPE and PaTH on that capability.Source: LIMITATION · p. 20
Loading 2512.07525v1…