PaperlayerView exact revision on arXiv

Source-linked AI summary

Extending Context Window of Large Language Models via Positional Interpolation

Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian

arXiv:2306.15595v2cs.CLcs.AIcs.LG

TL;DR

Pretrained LLMs have fixed context limits, and directly adapting them to substantially longer contexts can be inefficient or unstable. Position Interpolation down-scales position indices into the original range, enabling up to 32768-token contexts with minimal fine-tuning. The resulting models perform long-context tasks effectively while preserving original-context quality relatively well.

  • Problem

    Pretrained LLMs have fixed context windows, while extending them by direct long-context fine-tuning can adapt slowly and RoPE-based models have limited extrapolation applicability.

  • Method

    Position Interpolation down-scales position indices from the extended range into the original pretrained range before computing RoPE.

  • Results

    Position Interpolation extends LLaMA context windows to 32768 with 1000 fine-tuning steps, supports long-context modeling and summarization, and largely preserves quality within 2048 tokens.

  • Takeaways & Limitations

    The extended models retain the original architecture and can reuse most pre-existing optimization and infrastructure for long- and short-input applications.

  • Takeaways & Limitations

    Position Interpolation can cause minor perplexity degradation within the original context window, and regularizing query/key products to mitigate extrapolation remains future work.

Abstract

from arXiv · show

We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.

1 INTRODUCTION

Position Interpolation extends RoPE-based LLM context windows by down-scaling position indices into the pretrained range, avoiding direct extrapolation. Across LLaMA models, it reaches 32768-token contexts with brief fine-tuning, supports long-context tasks, and largely preserves short-context quality.

  • Existing LLaMA models are limited to fewer than 2048 input tokens, although applications such as long conversations and document summarization often require longer contexts.
  • Fine-tuning an existing Transformer directly on longer contexts required more than 10000 batches while increasing the effective window only from 2048 to 2560.
  • Position Interpolation down-scales position indices so the maximum matches the original pretrained range, interpolating neighboring positional encodings instead of extrapolating beyond it.
  • ~600× smaller attention-score upper bounds make interpolation theoretically more stable than extrapolation in the LLaMA 7B setting.
  • 32768-token context windows were achieved from an initial 2048 using only 1000 fine-tuning steps on the Pile, with good quality across 7B–65B LLaMA models.
  • Extended models gain perplexity on greatly enlarged contexts and achieve competitive long-document summarization performance.
  • Within the original 2048-token limit, extended models show relatively well-preserved quality, with only minor degradation on several standard benchmarks.
  • The work attributes weaknesses in language-modeling extrapolation to direct positional-encoding extrapolation and reports that interpolation can largely mitigate them.

2 METHOD

RoPE-based models can behave catastrophically when attention scores are extrapolated beyond the training context, so Position Interpolation rescales positions into the original range and supports stable extension with minimal architectural change.

  • 2.1 BACKGROUND: ROTARY POSITION EMBEDDING (ROPE): RoPE attention depends on relative position, but direct extension beyond the trained context can produce extremely high perplexity and catastrophic failures.The model may fail to answer questions even when relevant evidence lies within its original effective distance range.
  • 2.2 DIRECT EXTRAPOLATION: Fitting RoPE attention scores within [0, 2048] can still yield values above 8000 outside that range, because the trigonometric basis can approximate arbitrary functions.The coefficients depend on query and key vectors, allowing small in-range values but much larger out-of-range values.
  • 2.3 PROPOSED APPROACH: POSITION INTERPOLATION (PI): Interpolation between neighboring integer positions remains smooth and well-behaved, unlike extrapolation beyond the pretrained range.The interpolation bound is derived from known behavior at integer grid points and bounds deviation from linear interpolation.
  • 2.3 PROPOSED APPROACH: POSITION INTERPOLATION (PI): Position Interpolation down-scales positions from [0, L′) to [0, L) before applying RoPE, aligning extended relative distances with the pretrained range.This transformation reduces the maximum relative distance from L′ to L without changing the model architecture or adding weights.
  • 2.3 PROPOSED APPROACH: POSITION INTERPOLATION (PI): 600× smaller is the interpolation bound than the extrapolation bound, making interpolated attention scores substantially more stable.For c = 10000, the interpolation bound is approximately d max_j |h_j|, while the comparison bound is much larger.
  • 2.3 PROPOSED APPROACH: POSITION INTERPOLATION (PI): Position Interpolation can be fine-tuned on an extended context using a pretraining corpus, starting from the pretrained model and adapting only its context range.The authors report that the fine-tuning result is not sensitive to the choice of examples.

3 EXPERIMENTS

The experiments evaluate Position Interpolation across language modeling, passkey retrieval, original-window benchmarks, and summarization, comparing it with direct fine-tuning. Position Interpolation extends effective context windows substantially with short fine-tuning while preserving much original-window quality, though some degradation remains.

  • 3 EXPERIMENTS: Position Interpolation extends LLaMA models from 2048 to context windows up to 32768 without changing the model architecture.The experiments cover 7B, 13B, 33B, and 65B models, using Position Interpolation or direct fine-tuning.
  • 3.2 LONG SEQUENCE LANGUAGE MODELING: Increasing context from 2048 to 16384 reduces perplexity by 0.28 and 0.5 for LLaMA 7B, 0.27 and 0.48 for 13B, and 0.14 and 0.42 for 33B across the two datasets.For LLaMA 65B, extending to 8192 reduces perplexity by 0.12 and 0.3.
  • 3.2 LONG SEQUENCE LANGUAGE MODELING: Position Interpolation models consistently benefit from longer context, including continued gains through 32768 tokens for LLaMA 7B and 13B on PG19.Direct fine-tuning instead shows regression up to +0.48 or only minor improvement up to -0.12 at longer windows.
  • 3.2 LONG SEQUENCE LANGUAGE MODELING: Position Interpolation can slightly worsen perplexity at the original 2048-token context, with Proof-pile degradation ranging from 0.01 to 0.05 across extended models.The paper attributes this expected trade-off to compressing original position encodings into a narrower region.
  • 3.2 LONG SEQUENCE LANGUAGE MODELING: 1000 fine-tuning steps with Position Interpolation produce steadily improved perplexity on PG19 for models extended to 8192 and 16384 tokens.At 200 steps, the extended models surpass the original model’s 2048-context perplexity; performance improves further by 1000 steps.
  • 3.3 MEASURING EFFECTIVE CONTEXT WINDOW SIZE THROUGH PASSKEY RETRIEVAL: After 200 steps, Position Interpolation reaches the target effective context window, whereas direct fine-tuning increases effective length only from 2048 to 2560 after more than 10000 steps.This result holds across 7B and 33B models and target windows up to 32768.
  • 3.4 BENCHMARKS ON ORIGINAL CONTEXT WINDOW SIZE: Models extended to 8192 show benchmark degradation of up to 2% on tasks designed for the original 2048-token window, while longer extensions regress more.The extended models nevertheless remain within reasonable ranges for most tasks.
  • 3.5 LONG DOCUMENT SUMMARIZATION: Position Interpolation achieves competitive R1 scores for long-document summarization with a 16384-token context window and minimal hyperparameter tuning.The result is compared with two baselines from the SCROLLS leaderboard.

4 RELATED WORK

Related work extends Transformers through retrieval, recurrence, memory, attention approximation, and positional techniques, while Position Interpolation targets context extension in existing LLMs.

  • Retrieval-augmented LLMs add fetched documents to the input context, and Position Interpolation is complementary because its longer context can include more documents.
  • Recurrent and memory Transformers handle very long sequences through recurrence or memory, but may restrict attention to lossy compressed past inputs.
  • Approximated multi-head attention methods reduce memory and computational complexity through approximation or sparsification, and Position Interpolation remains compatible with most of them.
  • Length-extrapolation methods train on short sequences and infer on longer ones, but have not enabled extrapolation for many existing pretrained models such as LLaMA or OPT.
  • Vision Transformer interpolation linearly interpolates learned position embeddings for higher resolution, whereas this work applies interpolation to extend LLM context windows.
  • Together with prior Vision Transformer results, the authors view these findings as empirical evidence that Transformers can handle sequences significantly longer than those used during training.

5 CONCLUSIONS

Position Interpolation substantially enlarges LLaMA context windows with minimal fine-tuning while preserving performance on both extended and original-length inputs.

  • Position Interpolation extends LLaMA context windows significantly using minimal fine-tuning, while supporting varied tasks on extended contexts and retaining original abilities relatively well.
  • Extended models can reuse most pre-existing infrastructure and optimization, making Position Interpolation attractive for practical applications.
  • The authors conjecture that Position Interpolation may generalize to other positional encodings and enable extensions for additional LLM types.

A PROOF

The proof bounds interpolation error by comparing the attention score at an interpolated position with linear interpolation between neighboring integer grid points.

  • The interpolation bound treats attention score a(s) between grid points s1 and s2, whose endpoint behavior is established by LLM pre-training.
  • Linear interpolation provides the reference value formed from the attention scores at neighboring grid points a(s1) and a(s2).
  • Taylor expansion is used to express the difference between the attention score and its linear interpolation through second-order derivative terms.
  • The proof bounds the second-order derivative using the magnitude of complex-valued components and the inequality |Re(x)| ≤ |x|.
  • For interpolation between adjacent integer positions, (s − s1)(s2 − s) ≤ 1/4, which controls the interpolation error term.

B VISUALIZATION OF QUANTITIES IN EXTRAPOLATION BOUND

The extrapolation-bound visualization examines B(s)/d as positional difference s grows and shows that the bound often remains large despite decaying with s.

  • The extrapolation bound contains B(s), defined as a sum of complex exponentials over half the positional dimensions.
  • Using the LLaMA-7B setting c = 10000 and d = 128, B(s)/d is almost always larger than 1 and often much larger than 1.
  • Figure 5 plots B(s)/d against positional difference s and shows that the bound decays as s increases, while remaining at least 1 numerically.

C.1 CODE FOR FIG. 2

The Figure 2 code constructs sinusoidal positional features, solves for coefficients, and plots attention-score behavior for extrapolation and interpolation.

  • Feature construction: The code computes RoPE frequencies from d = 4096 and theta = 10000, then builds sinusoidal and cosine positional features.The frequency basis uses even dimensions and concatenates sine and cosine components.
  • Coefficient fitting: A regularized linear solve estimates coefficients using XᵀX and Xᵀy.The solve adds eps times the identity matrix before inversion.
  • Attention-score plots: The interpolation plot evaluates positions from 25 to 75 in increments of 0.125 and marks integer positions while labeling the interpolation effect.Vertical guides are drawn for each integer from 25 through 74.
  • Attention-score plots: The first plot compares attention score a(s) against positional difference s within the original range, overlaying sampled observations.The plot uses x2[:L] and y2[:L], with scatter points from x and y.
  • Attention-score plots: The extrapolation plot extends positional differences to 2L and marks the original boundary at L.The effect of extrapolation is labeled directly in the figure.

C.2 CODE FOR FIG. 5

The Figure 5 code evaluates a normalized quantity B(s)/d across positional differences s and compares it with a horizontal reference at 1.0.

  • Figure 5 visualization: The plot displays B(s)/d as a function of positional difference s.The plotted values are computed by summing mags across dimensions and dividing by d.
  • Figure 5 visualization: A horizontal line at 1.0 provides the reference level for interpreting the normalized curve.The reference is drawn across the plot with a dashed line.
Loading 2306.15595v2…