PaperlayerView exact revision on arXiv

Source-linked AI summary

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs

Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang

arXiv:2602.05258v1cs.CLcs.AIcs.LG

TL;DR

Long-context RoPE adaptations address OOD extrapolation and semantic attention as separate objectives, despite sharing a low-frequency failure mode. CoPE softly clips those components and consistently improves performance across diverse tasks and context lengths, reaching nearly 2× RoPE performance at up to 256k tokens.

  • Problem

    Existing long-context RoPE adaptations treat OOD mitigation and semantic modeling as distinct objectives despite their shared low-frequency failure mode.

  • Method

    CoPE is a plug-and-play RoPE modification that softly attenuates low-frequency components with a cosine-decay taper.

  • Results

    Nearly 2× RoPE performance at context lengths up to 256k tokens, with consistent improvements across diverse tasks and context lengths.

  • Takeaways & Limitations

    CoPE provides a simple, scalable drop-in replacement for RoPE that improves long-context capability across diverse tasks and lengths.

  • Takeaways & Limitations

    Synthetic recall tasks provide limited insight into real-world long-context performance because they may saturate early or fail to distinguish models meaningfully.

Abstract

from arXiv · show

Rotary Positional Embedding (RoPE) is a key component of context scaling in Large Language Models (LLMs). While various methods have been proposed to adapt RoPE to longer contexts, their guiding principles generally fall into two categories: (1) out-of-distribution (OOD) mitigation, which scales RoPE frequencies to accommodate unseen positions, and (2) Semantic Modeling, which posits that the attention scores computed with RoPE should always prioritize semantically similar tokens. In this work, we unify these seemingly distinct objectives through a minimalist intervention, namely CoPE: soft clipping lowfrequency components of RoPE. CoPE not only eliminates OOD outliers and refines semantic signals, but also prevents spectral leakage caused by hard clipping. Extensive experiments demonstrate that simply applying our soft clipping strategy to RoPE yields significant performance gains that scale up to 256k context length, validating our theoretical analysis and establishing CoPE as a new state-of-the-art for length generalization. Our code, data, and models are available at https://github.com/hrlics/CoPE.

1. Introduction

The introduction unifies OOD mitigation and semantic modeling as consequences of low-frequency RoPE behavior during extrapolation, motivating CoPE’s soft clipping intervention. Experiments present CoPE as a scalable drop-in RoPE replacement that improves performance across tasks and context lengths up to 256k.

  • Long-context scaling commonly modifies RoPE frequencies during a post-pretraining long-context training stage.
  • Existing RoPE adaptations generally pursue either OOD mitigation or semantic modeling, with low-frequency components causing severe OOD issues at unseen positions.
  • Low-frequency RoPE components jointly govern extrapolation OOD behavior and long-context semantic-attention stability, unifying the two adaptation objectives.
  • CoPE softly attenuates low-frequency RoPE components to eliminate OOD outliers, refine semantic signals, and prevent spectral leakage from hard clipping.
  • CoPE consistently improves diverse tasks and context lengths up to 256k as a drop-in RoPE replacement with other configurations unchanged.Experiments follow a Qwen3-style recipe using ABF for long-context training and YaRN for test-time extrapolation.

2. Preliminaries

RoPE is introduced as the dominant positional encoding in modern LLMs, representing token positions through frequency-specific rotations applied to query and key dimensions. Its attention score uses the relative distance between token positions.

  • Rotary Position Embedding: RoPE has become the dominant positional encoding choice in modern LLMs.Transformer models use positional encodings to incorporate sequential information.
  • Rotary Position Embedding: RoPE partitions query and key dimensions into d/2 chunks and assigns each chunk a unique rotation frequency.The query and key vectors are represented in dimension chunks before rotation.
  • Rotary Position Embedding: Each chunk uses θ_i = b^−2i/d, with i ∈ {0, 1, . . . , d/2 − 1} and typically b = 10,000.The base frequency b is predefined and is typically set to 10,000.
  • Rotary Position Embedding: A block-diagonal rotation matrix R_n ∈ R^d×d applies position-dependent rotations using cosine and sine terms for the assigned frequencies.The matrix contains rotation blocks with angles nθ_i.
  • Rotary Position Embedding: The resulting attention score between q_n and k_m is defined using their relative distance m − n.Here, m − n denotes the relative distance between the query and key positions.

3. Analysis

The analysis identifies low-frequency RoPE components as the shared source of OOD extrapolation and long-term semantic-attention decay. It motivates treating these two problems through their common behavior beyond the pre-training regime.

  • RoPE frequency behavior: Low-frequency components have periods exceeding the pre-training context window, causing OOD extrapolation, while high-frequency components remain well-behaved.For Llama-3-8B, the 35-th chunk’s period already exceeds the 8192-token pre-training window, leaving the last 29 of 64 chunks without a complete pre-training period.
  • Critical dimension in extrapolation: For Llama-3-8B with Lpre = 8192, d = 128, and b = 500, 000, the critical dimension is 70, corresponding to the 35-th rotation chunk.The critical dimension marks the boundary before which dimensions perceive complete periodic patterns during pre-training.
  • OOD mitigation methods: OOD mitigation methods interpolate low-frequency components while minimally perturbing high frequencies, differing primarily in their per-frequency scaling factors.PI, NTK, YaRN, and LongRoPE represent alternative choices of frequency scaling, with PI uniformly scaling all frequencies and potentially reducing local positional resolution.
  • Semantic modeling: RoPE’s long-term semantic-attention decay worsens with relative distance and is primarily governed by low-frequency components.The semantic preference for similar tokens over random tokens diminishes as distance increases, revealing low frequencies as an unreliable semantic channel.
  • Unifying insight: OOD extrapolation and long-term semantic decay are two manifestations of the same underlying issue: suboptimal low-frequency behavior beyond the pre-training regime.Low frequencies both exceed the pre-training context through their long periods and carry semantic information that degrades over long relative distances.

4. CoPE: Clipped Rotary Position Embedding

CoPE softly clips RoPE’s low-frequency components with a smooth taper, addressing OOD outliers and semantic signals while avoiding the spectral leakage and ringing caused by hard clipping. It is implemented as a drop-in RoPE replacement compatible with optimized inference kernels and standard inference speeds.

  • Method: CoPE softly clips RoPE’s low-frequency components to eliminate OOD outliers, refine semantic signals, and avoid severe spectral leakage from hard clipping.The method is designed to scale favorably as the context window increases.
  • Hard-clipping limitation: Hard clipping creates a sharp spectral cutoff that causes spectral leakage and long-range oscillatory ringing in the attention signal.The ringing artifacts are identified as the Gibbs phenomenon.
  • Hard-clipping limitation: Theorem 4.1 attributes hard-clipping error to a sinc kernel with slow O(1/τ) decay, whose Gibbs oscillations disrupt attention decay and induce spurious long-range correlations.The theorem analyzes a hard high-pass filter at cutoff θc applied to the continuous attention score A(τ).
  • Soft clipping: CoPE replaces the binary frequency mask with scalar weights w_j ∈[0, 1] using a cosine-decay taper to minimize spectral discontinuity.The taper smoothly suppresses low-frequency components instead of abruptly removing them.
  • Implementation: CoPE modifies only RoPE frequency initialization, providing a drop-in replacement compatible with FlashAttention and standard inference speeds without changing model architecture.The implementation is designed for seamless integration into modern LLM frameworks.

5. Experiment

CoPE consistently improves long-context performance over RoPE and hard clipping, with gains increasing through 256k context while preserving short-context capabilities. Synthetic benchmarks often fail to distinguish real-world performance, whereas HELMET reveals CoPE’s advantages across diverse tasks and context lengths.

  • Evaluation Benchmarks: HELMET evaluates synthetic recall, RAG, ICL, long-document QA, and summarization, alongside standard short-context benchmarks including MMLU, GPQA, and GSM8K.The evaluation combines diverse long-context tasks with benchmarks for broad reasoning and knowledge capabilities.
  • Performance on HELMET: At 64k, CoPE improves average performance over RoPE by 10.84%, while at 256k it achieves approximately 2× RoPE’s performance.CoPE outperforms RoPE and HardClip across nearly all tasks and context lengths, without compromising in-distribution performance.
  • Scalable Performance Gain of CoPE: Average performance gains rise from 4.54% at 8-16k to 10.39% at 32k–64k and 58.61% at 128k–256k.The scaling trend reflects suppression of unstable low-frequency behavior that becomes more pronounced during long-context extrapolation.
  • Synthetic Benchmark Limitations: Synthetic benchmarks are poor proxies for real-world long-context performance because they saturate early, poorly distinguish methods, or remain insensitive to context length.RULER-NIAH and RULER-MK reach near-perfect accuracy from 8k-64k, InfiniteBench KV is nearly identical across methods at 8k-32k, and InfiniteBench Math Find stays near ∼35% through 256k.
  • Short Context Evaluation: CoPE preserves performance and yields slight gains on all standard short-context benchmarks, indicating that soft clipping does not trade off general capabilities.The results suggest clipping suppresses suboptimal low-frequency behavior rather than erasing semantic information.
  • Clipping Onset Ablation: All CoPE clipping-onset variants outperform vanilla RoPE, while the default configuration clipping ∼75% of low frequencies performs best.The ablation indicates that preserving some stable low frequencies is preferable to overly aggressive suppression.

6. Related Work

Prior RoPE extensions address context scaling through two broad principles: mitigating OOD issues or preserving semantic attention. CoPE unifies these perspectives by targeting the suboptimal behavior of low-frequency RoPE components during extrapolation.

  • Related work overview: RoPE modifications for context extension generally follow two principles: OOD mitigation and semantic modeling.RoPE is widely adopted in modern LLMs and is closely tied to length generalization.
  • RoPE OOD Mitigation: OOD-mitigation methods such as PI, NTK, YaRN, and LongRoPE scale RoPE frequencies to map extended contexts back to the training range.Their differences primarily concern per-frequency scaling factors.
  • RoPE OOD Mitigation: These scaling methods interpolate low frequencies while minimizing changes to high frequencies that completed multiple cycles during pre-training.The methods differ mainly in their choice of per-frequency scaling factors.
  • RoPE Semantic Modeling: Semantic-modeling work identifies long-term decay of semantic attention, where preference for semantically similar tokens over random tokens weakens with relative distance.This decay occurs in addition to RoPE’s general activation decay.
  • Unified perspective: CoPE argues that OOD extrapolation and unreliable long-distance semantic discrimination share the same cause: suboptimal low-frequency RoPE components.Low-frequency components both drive OOD extrapolation and act as a semantic channel whose discriminative power decays with distance.

7. Conclusion

The paper unifies long-context RoPE adaptations as OOD mitigation or semantic modeling, tracing both to suboptimal low-frequency behavior during extrapolation. It introduces CoPE as a plug-and-play RoPE enhancement that softly clips low-frequency components.

  • Existing long-context RoPE adaptations fall into two paradigms: OOD mitigation and semantic modeling.
  • Both paradigms originate from suboptimal behavior of low-frequency components in the extrapolation regime.
  • CoPE is introduced as a plug-and-play enhancement for RoPE that softly clips low-frequency components.

A. Proofs · A.1. Long-term Decay of Semantic Attention

The appendix provides proofs for the paper’s theoretical claims, including a derivation of Theorem 3.2 on RoPE’s long-term decay of semantic attention. Under the theorem’s assumptions, the semantic-attention advantage generally decreases with relative distance despite oscillatory behavior.

  • A. Proofs: The proof appendix develops detailed derivations for the theoretical statements presented in the paper.
  • A.1. Long-term Decay of Semantic Attention: Theorem 3.2 states that RoPE induces long-term decay in semantic attention as relative distance increases.The theorem concerns the ability to attend more to semantically similar tokens than to random ones.
  • A.1. Long-term Decay of Semantic Attention: The derivation for Equation 3.2 analyzes a query q ∈ R^d and key k ∈ R^d separated by distance ∆t.
  • A.1. Long-term Decay of Semantic Attention: The theorem assumes q and k have i.i.d. components with standard deviation σ.
  • A.1. Long-term Decay of Semantic Attention: A similar key is modeled as k′ = q + ϵ, where ϵ is a zero-mean perturbation.
  • A.1. Long-term Decay of Semantic Attention: The mean parameter µ denotes the mean of the i.i.d. components in q and k.
  • A.1. Long-term Decay of Semantic Attention: The cosine sum Σ_i=0^(d/2−1) cos(∆tθ_i) is oscillatory rather than monotonic in ∆t, but exhibits general decay as ∆t increases.This behavior is shown in Figure 3.

A.2. Spectral Leakage from Hard Clipping · B. Further Experimental Details

Hard high-pass clipping adds a spectral-leakage error to continuous attention scores, with a sinc kernel whose slow decay causes Gibbs oscillations and spurious long-range correlations. The paper also provides further experimental details covering benchmarks, additional results, and a case study.

  • A.2. Spectral Leakage from Hard Clipping: Hard high-pass filtering at cutoff θc transforms the continuous attention score into ˜A(τ) = A(τ) + E(τ).This result is stated as Theorem 4.1.
  • A.2. Spectral Leakage from Hard Clipping: The sinc kernel’s slow O(1/τ) decay introduces Gibbs oscillations that disrupt A(τ)’s general decay and induce spurious long-range correlations.The paper also illustrates this negative effect in Figure 4.
  • A.2. Spectral Leakage from Hard Clipping: The derivation defines hard clipping as an ideal high-pass filter, represented as the complement of an ideal low-pass rectangular window.The frequency-domain formulation uses Hhigh(ω) and Hlow(ω).
  • A.2. Spectral Leakage from Hard Clipping: By the Convolution Theorem, frequency-domain filtering becomes time-domain convolution with the inverse Fourier transform of the low-pass window.The filtered spectrum is formed by element-wise multiplication before applying F−1.
  • A.2. Spectral Leakage from Hard Clipping: The rectangular low-pass window has a normalized sinc impulse response, which decays asymptotically as O(1/τ).Substituting this kernel identifies the hard-clipping error term E(τ) = ˜A(τ) − A(τ).
  • B. Further Experimental Details: Further experimental details cover benchmark descriptions, additional results, and a case study.This section supplements the paper’s experimental presentation.

B.1. Benchmark Description

The study evaluates long-context LLMs using HELMET, RULER, and Infinite Bench. These benchmarks span real-world and synthetic tasks, while differing in task coverage, context length, and discriminative power.

  • Benchmark suite: The experiments use HELMET, RULER, and Infinite Bench as complementary long-context benchmarks.Together, they provide coverage of comprehensive real-world evaluation, synthetic testing, and extremely long-context understanding.
  • HELMET: HELMET evaluates long-context LLMs on real-world tasks spanning summarization, long-document QA, many-shot ICL, synthetic recall, RAG, citation generation, and passage re-ranking.The benchmark is presented as improving upon purely synthetic or more limited real-world evaluations.
  • RULER: RULER is a purely synthetic benchmark that expands needle-in-a-haystack evaluation into 13 tasks with varied needle types and quantities.The paper notes that synthetic tasks can saturate within the training range or provide limited signals for real-world performance.
  • Infinite Bench: Infinite Bench targets extremely long-context understanding with synthetic and real-world tasks averaging ∼200k tokens across novel understanding, code execution, and mathematical calculation.Some tasks have limited discriminative power among methods or are insensitive to context length.

B.2. Additional Results · B.3. Case Study

CoPE shows limited separation from RoPE on most RULER tasks but delivers substantial gains at 256k context and consistent improvements on real-world HELMET tasks. Long-document QA examples further show CoPE producing more coherent, detail-preserving answers than RoPE.

  • B.2. Additional Results: CoPE improves RULER performance by up to +18.0 under extremely long 256k contexts.Most RULER tasks otherwise provide limited discrimination between methods.
  • B.2. Additional Results: Most RULER tasks exhibit limited performance differences between CoPE and RoPE.The exception is the extremely long-context 256k setting, where CoPE gains substantially.
  • B.2. Additional Results: CoPE consistently yields significant gains on real-world HELMET tasks.The cited tasks include retrieval-augmented generation, in-context learning, and long-form summarization.
  • B.3. Case Study: In long-context settings, RoPE often produces repetitive, less informative responses that miss fine-grained details.These observations come from the long-document QA examples reported in Table 6.
  • B.3. Case Study: CoPE produces more coherent answers than RoPE in long-document QA examples.The examples indicate that CoPE better preserves answer detail under long-context conditions.
  • B.3. Case Study: CoPE preserves more fine-grained detail than RoPE in the reported long-document QA cases.This contrasts with RoPE’s tendency to miss fine-grained details under long-context settings.
Loading 2602.05258v1…