PaperlayerView exact revision on arXiv

Source-linked AI summary

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

Han Tian, Luxuan Chen, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Jinman Zhao, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Linghe Kong, Dawei Yin

arXiv:2605.14589v2cs.CL

TL;DR

Long-context adaptation usually requires costly target-length training, raising memory and computation demands. EndPrompt instead uses an intact short context plus a terminal prompt with distant positional indices, achieving strong benchmark performance from sparse positional supervision. The method extends LLaMA-family models from 8K to 64K while avoiding full-length training sequences.

  • Problem

    Reliable context extension requires models to preserve local coherence and interactions between distant tokens, but target-length training is costly because attention computation scales quadratically.

  • Method

    EndPrompt retains an intact short context and appends a terminal prompt assigned positional indices near the target context length to provide local and long-range positional supervision.

  • Results

    76.03 average RULER score surpasses LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23), while EndPrompt achieves the highest average on LongBench.

  • Takeaways & Limitations

    Long-context generalization can emerge from structured sparse positional supervision without full-length training sequences.

  • Takeaways & Limitations

    The current formulation relies on an explicit terminal segment, with future work proposed for streaming, hierarchical, or multimodal architectures.

Abstract

from arXiv · show

Extending the context window of large language models typically requires training on sequences at the target length, incurring quadratic memory and computational costs that make long-context adaptation expensive and difficult to reproduce. We propose EndPrompt, a method that achieves effective context extension using only short training sequences. The core insight is that exposing a model to long-range relative positional distances does not require constructing full-length inputs: we preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length. This two-segment construction introduces both local and long-range relative distances within a short physical sequence while maintaining the semantic continuity of the training text--a property absent in chunk-based simulation approaches that split contiguous context. We provide a theoretical analysis grounded in Rotary Position Embedding and the Bernstein inequality, showing that position interpolation induces a rigorous smoothness constraint over the attention function, with shared Transformer parameters further suppressing unstable extrapolation to unobserved intermediate distances. Applied to LLaMA-family models extending the context window from 8K to 64K, EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23) while requiring substantially less computation. These results demonstrate that long-context generalization can be induced from sparse positional supervision, challenging the prevailing assumption that dense long-sequence training is necessary for reliable context-window extension. The code is available at https://github.com/clx1415926/EndPrompt.

1 Introduction

EndPrompt extends context windows using short sequences by combining intact local context with a terminal anchor carrying distant positional indices. The method is theoretically motivated and empirically outperforms representative baselines on RULER and LongBench without full-length training sequences.

  • 76.03 average RULER score surpasses LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23).Experiments extend LLaMA-family models from 8K to 64K and report the highest average score on LongBench.
  • EndPrompt appends a terminal prompt to an intact short context, using positional index manipulation to simulate long-range positional supervision.The first segment retains local indices, while the terminal segment receives indices near the target context length.
  • Preserving the original context as an undivided segment maintains semantic continuity while the end prompt creates stable long-distance relationships for next-token prediction.
  • Position interpolation and shared Transformer parameters support smooth generalization across unobserved intermediate relative distances.The analysis frames sparse positional supervision through Rotary Position Embedding and position interpolation.
  • EndPrompt achieves competitive or superior performance on RULER and LongBench while avoiding full-length training sequences.The approach addresses the cost of target-length training, whose attention computation scales quadratically.

2 Preliminary

The preliminary analysis describes RoPE attention as dependent on relative positional distance and explains how position interpolation rescales indices to smooth attention variation. This smoothness bias supports stable attention across distances not observed during training.

  • RoPE attention scores depend on assigned relative distance through sinusoidal phase rotations applied to query and key components.For a head, the relative distance is d = p_m − p_n.
  • Position interpolation rescales positional indices by a target factor s > 1, reducing the effective angular frequencies.The rescaling maps p to p/s and changes θ_j to θ_j/s.
  • Lower effective frequency bounds the attention function’s rate of change and curvature across the distance dimension.These bounds follow from the finite trigonometric-polynomial form of the RoPE attention score.
  • Position interpolation provides a smoothness bias rather than guaranteeing perfect reconstruction of unseen distances.The method combines this smoothness with targeted long-distance supervision to stabilize unobserved-distance attention scores.

3 Method

EndPrompt creates sparse long-distance supervision within short training sequences by retaining the original context and appending a terminal segment with positions near the target boundary. Its training objective combines local prediction with terminal-token losses, constraining shared Transformer parameters and regularizing attention across unobserved distances.

  • 3.1 Overview: EndPrompt decouples physical token order from positional indices, retaining the intact context while placing an appended terminal segment near the target context boundary.This creates local and long-range relative distances without splitting contiguous text.
  • 3.2 Positional Index Manipulation: The short context and end prompt occupy separate positional intervals, so attention uses assigned relative distance rather than physical token distance.The resulting observed distances include local intervals and a long-range interval between the segments.
  • 3.2 Positional Index Manipulation: Sparse training supervises local and selected long distances instead of explicitly covering every distance in the gap region.RoPE and position interpolation constrain behavior over unobserved intermediate distances.
  • 3.3 End Prompt as the Terminal Segment: The end prompt is a structural terminal cue whose placement near the assigned context boundary matters more than any unique prompt string.Stable terminal cues can induce long-distance interactions without conflicting semantics.
  • 3.4 Training Objective: Prompt-token losses receive smaller but nonzero weights, preserving terminal-token supervision while reducing excessive reliance on prompt-token prediction.Terminal states can attend to the original context across large assigned positional distances.
  • 3.4 Training Objective: Local and terminal long-distance losses constrain shared Transformer parameters, reducing the feasible parameter region and implicitly regularizing the attention function.The long-distance constraints remove parameter configurations that fail to generalize to long distances.

4 Experiments

Experiments evaluate EndPrompt across long-context benchmarks, ablations, model families, context lengths, data volumes, prompt variants, and short-text recovery. EndPrompt achieves the strongest reported overall performance while retaining robustness across configurations and downstream settings.

  • Main results: 76.03 is ET’s average RULER score, exceeding LongLoRA (72.95) and LCEG (72.24).
  • Main results: 38.30 is ET’s average LongBench score, exceeding full-length fine-tuning (35.63).
  • Main results: ET remains robust on complex retrieval, outperforming LongLoRA on Vt (82.00 vs. 65.70) and Fwe (83.53 vs. 58.17).In multi-needle retrieval, ET scores 81.67 on Niah_MV and 82.06 on Niah_MQ.
  • Main results: 66.48 is ET’s Code Completion score, exceeding LCEG (46.86) and LongLoRA (45.86).ET also scores 68.04 in Few-Shot Learning and leads Single-Doc QA, Multi-Doc QA, and Summarization.
  • Ablations: 72.82 is the RULER average at 128K, while LongBench ranges from 36.45 at 32K to 35.68 at 128K.The reported results describe minor degradation at extreme lengths rather than catastrophic forgetting.
  • Ablations: LongBench changes from 38.30 with EP_1 to 37.95 with EP_3, while RULER ranges from 76.45 with EP_2 to 74.63 with EP_3.These results indicate stable performance across end-prompt configurations.

5 Conclusion

EndPrompt extends LLaMA-family models from 8K to 64K without full-length sequence training, combining terminal anchoring with sparse positional supervision. It achieves strong RULER and LongBench results while suggesting a practical alternative to dense long-sequence adaptation.

  • Conclusion: EndPrompt extends LLaMA-family models from 8K to 64K context without training on full-length sequences.The method preserves an intact short context and appends a terminal prompt near the target context boundary.
  • Conclusion: 76.03 average RULER score exceeds the reported LCEG, LongLoRA, and full-length fine-tuning baselines.The conclusion reports strong performance on both RULER and LongBench.
  • Conclusion: RoPE, position interpolation, and shared Transformer parameters explain how sparse positional supervision stabilizes attention across unobserved distances.The analysis attributes this behavior to shared spectral structure, suppressed high-frequency variation, and coupling between local and terminal supervision.
  • Conclusion: The current formulation relies on an explicit terminal segment, motivating future combinations with streaming, hierarchical, or multimodal architectures.The stated practical direction is to design informative positional supervision while keeping physical training sequences short.

A.1 Experiment details

The experiment fine-tunes Meta-Llama-3-8B with Position Interpolation using 8K training sequences and resource-conscious full-parameter training.

  • A.1 Experiment details: Meta-Llama-3-8B serves as the base model, with Position Interpolation extending its context window.The training corpus has a maximum sequence length of 8K.
  • A.1 Experiment details: 8K is the maximum sequence length used during data preprocessing.
  • A.1 Experiment details: Full-parameter fine-tuning runs for one epoch on a single node with 8 NVIDIA A800 GPUs.BF16, DeepSpeed ZeRO Stage-3, FlashAttention, and gradient checkpointing are used to reduce memory and improve efficiency.

A.2 Evaluation Tasks

The evaluation combines LongBench’s multi-task long-context assessment with RULER’s fine-grained analysis of effective context length.

  • A.2 Evaluation Tasks: LongBench evaluates single-document and multi-document question answering, document summarization, few-shot learning, synthetic tasks, and code completion.
  • A.2 Evaluation Tasks: RULER provides a fine-grained analysis of the model’s effective context length.
  • A.2 Evaluation Tasks: The evaluation uses diverse datasets across multiple long-context evaluation frameworks.Together, the frameworks cover reasoning, summarization, few-shot learning, synthetic tasks, code completion, and context-length analysis.

B Supplementary Experiment

The robustness analysis evaluates three alternative terminal prompts, ranging from an explicit phrase to a native termination token and a minimal string.

  • B Supplementary Experiment: EP_1 uses the explicit phrase “This is the end of text, please pay attention here”.
  • B Supplementary Experiment: EP_2 uses the native LLaMA-3 termination token <|eot_id|>.
  • B Supplementary Experiment: EP_3 uses the minimal terminal prompt “End.”.

B.2 Training Efficacy

ET improves training efficiency at 32K and 64K context lengths by reducing memory use while accelerating computation relative to baseline methods. At 64K, it uses 36.52 GB and achieves up to 1.77× acceleration over the listed baselines.

  • Efficiency comparison: ET strictly dominates baseline methodologies in memory footprint and time consumption at 32K and 64K context lengths.The comparison evaluates both memory usage and temporal cost.
  • Memory footprint: 36.52 GB is ET's memory requirement at 64K, 52% below the 76.00 GB required by full-length fine-tuning.
  • Temporal cost: 1.41×, 1.69×, and 1.77× are ET's acceleration ratios over full-length fine-tuning, LongLoRA, and LCEG, respectively.
  • Efficiency comparison: The combined reduction in memory utilization and increase in computational speed supports ET's training efficiency.

B.3 Complete Table1

The supplied material identifies RULER result tables at 4K, 8K, 16K, and 32K, alongside background on positional scaling, efficient adaptation, and long-context evaluation benchmarks.

  • RULER results: Tables 6–9 report RULER synthetic benchmark results at 4K, 8K, 16K, and 32K.
  • Position-based context extension: Position Interpolation rescales positional indices to extend context beyond the pretraining limit, while NTK-aware scaling and YaRN modify rotary frequencies.
  • Efficient long-context adaptation: LongLoRA uses shifted sparse attention for efficient fine-tuning, whereas PoSE simulates long distances by manipulating positional indices across disjoint text chunks.
  • Long-context evaluation: LongBench and L-Eval evaluate diverse long-context tasks, ZeroSCROLLS targets zero-shot long-text summarization, and RULER provides configurable synthetic evaluation.
Loading 2605.14589v2…