PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws of RoPE-based Extrapolation

Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, Dahua Lin

arXiv:2310.05209v2cs.CLcs.AI

TL;DR

RoPE-based LLMs lose performance when context exceeds training length, and prior work has focused on particular bases and tuning lengths rather than their relationship to extrapolation. This paper develops a periodic framework for that relationship and explains the issue through a critical dimension. It reports nearly 1M-token context on LLaMA2 7B and 13B with 16K tuning length using a smaller base.

  • Problem

    RoPE-based LLM performance drops when context exceeds training length, and the relationships among rotary base, tuning length, and extrapolation performance remain insufficiently understood.

    Source: §1 · pp. 1–2

  • Method

    The paper develops a unified periodic framework relating rotary base and tuning context to extrapolation, including a critical dimension that explains the extrapolation issue.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 7

  • Results

    A base of 500 achieves almost 1M-token context with 16K tuning length, while base 1,000,000 extends LLaMA2 7B and 13B beyond 100K with 4K tuning length.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper provides strategies for extrapolating to limited or unpredictable contexts and insights into base modifications during fine-tuning or inference.

    Source: §5 · p. 9

    The paper leaves integration of smaller-base performance into pre-training and discovery of an optimal base–training-length pair for future work.

    Source: LIMITATION · p. 25

Abstract

from arXiv · show

The extrapolation capability of Large Language Models (LLMs) based on Rotary Position Embedding is currently a topic of considerable interest. The mainstream approach to addressing extrapolation with LLMs involves modifying RoPE by replacing 10000, the rotary base of $\theta_n={10000}^{-2n/d}$ in the original RoPE, with a larger value and providing longer fine-tuning text. In this work, we first observe that fine-tuning a RoPE-based LLM with either a smaller or larger base in pre-training context length could significantly enhance its extrapolation performance. After that, we propose \textbf{\textit{Scaling Laws of RoPE-based Extrapolation}}, a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well as tuning context length. In this process, we also explain the origin of the RoPE-based extrapolation issue by \textbf{\textit{critical dimension for extrapolation}}. Besides these observations and analyses, we achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.

1 INTRODUCTION

RoPE-based LLM performance drops when context exceeds training length, limiting long-text modeling and summarization. The paper studies how rotary base and tuning length relate to extrapolation, then presents a periodic framework and strategies for extending context.

1 INTRODUCTION

  • RoPE performance drops beyond training context length, constraining long-text modeling and summarization.The introduction identifies a gap in understanding how base value, tuning length, and extrapolation performance relate.Source: §1 · pp. 1–2
  • Fine-tuning with either smaller or larger bases can improve extrapolation, and smaller bases can match or surpass larger ones after longer-context tuning.The paper reports that a base of 1,000,000 can also reach 100K extrapolation with original training length.Source: §1 · p. 2
  • The Scaling Laws of RoPE-based Extrapolation provide a unified periodic framework for relating base and tuning context to extrapolation, and identify a critical dimension underlying the issue.The paper distinguishes these mathematical relations from model-size scaling laws.Source: §1 · p. 2
  • Fine-tuning with base 500 achieves almost 1M-token context with 16K tuning length, while base 1,000,000 extends LLaMA2 7B and 13B beyond 100K with 4K tuning length.The paper also reports that 10000 is the worst fine-tuning base for extrapolation.Source: §1 · p. 2

2 OBSERVATION

The observations show that changing the rotary base in either direction can improve extrapolation, but the behavior differs between larger and smaller bases. Larger bases have an extrapolation limit, whereas smaller bases show a gentler decline without an obvious breaking point.

2.1 LARGER BASES PROMISE BETTER EXTRAPOLATION

  • Larger bases extend LLaMA2 beyond training length and can surpass Dynamic NTK within their extrapolation upper bound, but degrade faster beyond it.Bases above 600000 reach beyond 100K with only 4K tuning length.Source: §2.1 · p. 3

2.2 SMALLER BASES ALSO PROMISE BETTER EXTRAPOLATION

  • At base 500, the extrapolation curve is smooth, performs strongly beyond 48K, and surpasses Dynamic NTK; smaller-base gains are not uniform.The decline in perplexity with growing context is gentler for smaller bases, which have no distinct upper bound.Source: §2.2 · p. 3

2 OBSERVATION

  • Base 10000 yields the worst extrapolation; increasing or decreasing it improves performance, with a clear upper bound for larger bases and no obvious breaking point for smaller bases.Figure 4’s heat map shows a continuous boundary for larger bases and a transition phase for smaller bases.Source: §2.2 · p. 3, Figure 4 · p. 4

3 EXPLANATION

The section explains how changing RoPE’s base alters the periods learned during tuning, and develops scaling laws around the critical dimension to account for extrapolation behavior. It reports that base 500 is particularly advantageous for unpredictable-length extrapolation, while larger bases have a theory-predicted upper bound.

3.1 SCALING LAW FOR SMALLER BASES

  • For LLaMA2’s 4096-token training length, bases 2608, 1304, and 652 correspond to cosine or sine inputs spanning π/2, π, and 2π, respectively.The authors associate these thresholds with progressively stronger extrapolation improvement as the base decreases.Source: §3.1 · p. 4

3.2 CRITICAL DIMENSION FOR EXTRAPOLATION

  • For LLaMA2, the critical dimension is 92: the first 92 dimensions receive complete positional information during pre-training, while the last 36 are insufficiently trained.Beyond the critical dimension, positions outside training length become out-of-distribution and the associated attention scores deviate from their expected distribution.Source: §3.2 · pp. 5–6

3.3 SCALING LAW FOR LARGER BASES

  • Fine-tuning with a larger base improves extrapolation, but the period of the critical dimension sets an upper bound that rises with the base.Empirical supported context lengths align with Theorem 2’s predicted bounds; Figure 7 also reports validation at 32K context and 70B model size.Source: §3.3 · p. 7, Figure 7 · p. 7

3.4 FURTHER VALIDATION FOR EXTRAPOLATION

  • At 1M context, LLaMA2 7B with base 500 has a stated advantage over base 1000000, especially with log-scaled attention.The authors also report that bases 500 and 1000000 outperform Linear PI and NTK in Figure 1.Source: §3.4 · p. 8, Table 1 · p. 8

4 RELATED WORK

Related work addresses long-sequence modeling through approaches to reduce attention computation, parallelize sequence processing, and extend RoPE-based models beyond training length. The cited RoPE approaches include changing rotary angles or bases, while prior methods can still show performance collapse beyond an extrapolation bound.

4 RELATED WORK

  • Efficient-attention methods reduce the quadratic computation through recalculation, sparse attention, low-rank approximation, or kernel approximations; FlashAttention reduces HBM reads and writes.The passage describes FlashAttention as improving compute-resource utilization through iterative self-attention.Source: §4 · pp. 8–9
  • Figure 8 compares attention-score behavior and extrapolation performance for LLaMA 7B evaluated or fine-tuned with original base 10000.Its rows correspond to the same comparisons described for Figure 6.Source: Figure 8 · p. 8
  • Sequence-parallel methods segment long sequences across devices, with approaches that overlap communication, balance loads, or use all-to-all communication for attention.The passage also notes that some sequence-parallel approaches have compatibility or communication-overhead trade-offs.Source: §4 · p. 9
  • RoPE extension methods include linear angle reduction, Dynamic NTK, larger bases, truncation, and scaling functions, but the cited methods can collapse beyond an unpredictable extrapolation bound.The passage gives examples ranging from 16K to 128K context extensions, using varied tuning lengths and methods.Source: §4 · p. 9

5 CONCLUSION

The conclusion presents a periodic framework for RoPE extrapolation and identifies the critical dimension as a key factor. The appendix describes RoPE’s use of rotary angles to encode position through query and key vectors, with relative position conveyed by sine and cosine.

5 CONCLUSION

  • The paper’s conclusion identifies the critical dimension and periodic perspective as central to explaining and improving RoPE-based extrapolation.It also describes an extended scaling law for longer-context tuning and notes the theory’s relevance to inference-time context extension.Source: §5 · p. 9

A.1 ROPE FROM SEQUENCE DOMAIN

  • RoPE splits query and key features into pairs, then applies angle-parameterized rotary vectors to encode position.The relative position t − s is acquired through cosine and sine when attention is calculated.Source: §A.1 · pp. 15–16
  • Although RoPE can theoretically convey relative information at any context length, it can fail to extrapolate practically; rotary angles are a focus of extrapolation methods.The vanilla design defines the angles using a base of 10000.Source: §A.1 · p. 16

A.2 ROPE FROM FREQUENCY DOMAIN

The frequency-domain view treats RoPE’s rotations as an inverse Fourier transform from frequency to time. Higher feature dimensions correspond to lower frequencies and longer periods, which can leave their positional waves incompletely represented during training.

A.2 ROPE FROM FREQUENCY DOMAIN

  • RoPE’s rotation can be viewed from the frequency domain as an inverse Fourier transform into the time domain.The passage frames this as a perspective on the rotation operation.Source: §A.2 · p. 16
  • Higher dimensions correspond to lower-frequency features with longer periods, reflecting longer contextual semantic relationships.The passage relates this correspondence to the rotary angle θn.Source: §A.2 · p. 16
  • When lower-frequency waves do not complete a period within training context, the model may learn them inadequately, making the number of well-trained dimensions important.The passage connects this count to the critical dimension discussed in Section 3.2.Source: §A.2 · p. 16

B.1 EXPERIMENT SETUP

The experiments fine-tune pretrained LLaMA2 models with varied RoPE bases and context lengths, then evaluate extrapolation and compare with other strategies. The setup keeps the model architecture unchanged apart from position embedding.

B.1 EXPERIMENT SETUP

  • Experiments use pretrained LLaMA2 7B and 13B models, fine-tuned with 32 A100 GPUs using ZeRO3.The setup also specifies AdamW, learning rate 2 × 10^-5, no warmup, and zero weight decay.Source: §B.1 · p. 16
  • For different-base RoPE tuning, the context is 4K, matching the training length, and evaluation extends to 100K.Models are tuned for 1K steps on Pile data and evaluated on a Books3 validation subset.Source: §B.1 · p. 17
  • The comparisons include Linear Position Interpolation and NTK, with Linear PI tuned at 8K context.For 16K-context fine-tuning, the global batch size is 32.Source: §B.1 · p. 17
  • Except for position embedding, the LLaMA2 architecture is unchanged.Table 2 describes Scaling RoPE as achieving the maximum context window and extension ratio among the listed strategies.Source: §B.1 · p. 17, Table 2 · p. 17

B.2 SHORT CONTEXT VALIDATION

Short-context evaluations test whether base adjustment preserves LLM performance after fine-tuning. The reported Scaling-RoPE results are comparable with pretrained or directly fine-tuned LLaMA2.

B.2 SHORT CONTEXT VALIDATION

  • Table 3 reports short-context results for 7B and 13B models, with original LLaMA2 also serving as the Dynamic NTK result.The final SuperGLUE score averages its subtasks.Source: Table 3 · p. 18
  • Scaling-RoPE results are comparable with pretrained or directly fine-tuned LLaMA2 on short-context tasks.The evaluated suite includes 0-shot Lambada and 5-shot MMLU, among other benchmark tasks.Source: §B.2 · pp. 17–18
  • Table 4 continues the short-context results and reports MMLU's final score as the average across subtasks.Source: Table 4 · p. 19

B.3 LONG CONTEXT VALIDATION

Long-context validation evaluates Scaling RoPE on real-world tasks at 16K context. The authors report that tuning at the original 4K pretraining context can still help on these tasks.

B.3 LONG CONTEXT VALIDATION

  • 4K-context fine-tuning still has benefits on the listed L-Eval tasks evaluated within 16K context.The suite includes GSM8K, TOEFL-QA, TopicRetrieval, QuALITY, and Coursera.Source: §B.3 · pp. 19–20
  • Table 5 presents long-context validation results for LLaMA2 7B and 13B.Source: Table 5 · p. 20
  • With 16K-context fine-tuning, extrapolation first weakens and then strengthens as the base increases from 500 to 1,000,000.Figure 9 plots perplexity on the Books3 validation data.Source: Figure 9 · p. 21

C EXTENSION

The extended scaling law relates extrapolation to the RoPE base and tuning length relative to pretraining length. It distinguishes base-dependent regimes and explains observed improvements through periodic behavior across dimensions.

C EXTENSION

  • When tuning length is at least the pretraining length, adjusting the RoPE base improves extrapolation, with a critical base determined by both lengths.The theorem extends to arbitrary bases and fine-tuning context lengths.Source: C EXTENSION · p. 21
  • At or below the critical base, the upper bound remains the tuning length and the critical dimension is updated.If the base is sufficiently small relative to β1, β2, and β3, extrapolation beyond the tuning length improves further.Source: C EXTENSION · p. 21
  • The extended law combines and generalizes prior results; for 16K-tuned LLaMA2 7B and 13B, maximum context reaches or exceeds 16K regardless of base.When tuning length exceeds pretraining length, the models can accommodate a broader context window.Source: C EXTENSION · p. 21
  • For bases above the critical base, the extrapolation upper bound depends on the base and critical dimension; for LLaMA2 tuned at 16K, the critical base is 71,738.Bases above 71,738, including 80,000, 120,000, and 1,000,000, have upper bounds beyond 16K, with larger bases corresponding to longer contexts.Source: C EXTENSION · pp. 21–22
  • At 16K tuning length, bases below 71,738 produce increasingly stable performance curves as the base decreases; base 500 performs similarly to 1,000,000, reaching at least 100K context.The authors attribute the improvement at base 10,000 to sine and cosine inputs traversing π/2 within 16K.Source: C EXTENSION · p. 22
  • The periodic account links base-dependent extrapolation to attention-score fluctuations and the dimensions whose positional information changes during tuning.At base 120,000, the critical dimension returns to 92; at 1,000,000, the first 92 dimensions have further-extended periods.Source: C EXTENSION · pp. 21–22

D DISCUSSION

Discussion compares inference-time corrections across RoPE base regimes: log-scaled attention is more useful at smaller bases, whereas Dynamic NTK is more useful at larger bases when scaled with T_extra. A text-continuation case study reports coherent output after a 32K prompt for bases 500 and 1,000,000.

D DISCUSSION

  • Log-scaled attention uses T_extra as its logarithm base and applies no correction within that extrapolation upper bound.The method multiplies the attention matrix by the logarithm of inference length, with a lower correction limit of 1.Source: D DISCUSSION · p. 23
  • The xPos method is used as a soft sliding window at inference and complements log-scaled attention, with T_extra replacing T_train in its denominator.The paper makes no other substantial modification to the method.Source: D DISCUSSION · p. 23
  • Dynamic NTK replaces vanilla RoPE's base 10,000 with the fine-tuning base and uses T_extra in place of T_train.These are the two stated modifications to Dynamic NTK.Source: D DISCUSSION · p. 23
  • Cutting the last 36 query/key dimensions improves log-scaled attention for fine-tuned LLaMA2, linking extrapolation failure to fluctuations beyond the critical dimension.Log-scaled attention alone does little for the unmodified pretrained and fine-tuned models in this comparison.Source: D DISCUSSION · p. 24
  • At base 500, log-scaled RoPE's flat perplexity curve indicates extrapolation to 100K, while Dynamic NTK shows a clear side effect.As base decreases, the log-scaled attention improvement becomes more dominant and Dynamic NTK's effect shrinks.Source: D DISCUSSION · p. 24
  • For bases 40,000–600,000, Dynamic NTK yields more remarkable extrapolation improvement; base 1,000,000 is omitted because it already reaches 100K.The discussion tests inference-phase extrapolation beyond 100K for bases above 10,000 and below 1,000,000.Source: D DISCUSSION · pp. 24–25
  • 400,000- and 600,000-base Dynamic NTK gains can be limited or destroyed when scaling uses T_train rather than T_extra.The discussion recommends replacing T_train with the extrapolation upper bound for larger bases.Source: D DISCUSSION · p. 25

E TEXT CONTINUATION

  • After a 32K Books3 prompt, LLaMA2 7B continuations remain coherent for bases 500 and 1,000,000; base 500 improves with longer tuning or log-scaled attention.The passage describes the outputs as grammatically accurate and logically consistent as well.Source: E TEXT CONTINUATION · p. 25

LIMITATION

The paper leaves several questions open: bounds on base adjustment, pretraining integration of smaller bases, and an optimal pairing of base with pretraining context length.

LIMITATION

  • The upper and lower bounds for adjusting the rotary base remain unresolved.The paper identifies these bounds as a question for future work.Source: LIMITATION · p. 25
  • Integrating smaller-base RoPE in the pretraining stage requires additional effort.The passage identifies this as an open research need.Source: LIMITATION · p. 25
  • An optimal pairing of rotary base and pretraining context length remains to be discovered.Both are described as key to extrapolation.Source: LIMITATION · p. 25
Loading 2310.05209v2…