Source-linked AI summary
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, Armaghan Eshaghi
TL;DR
Long input sequences challenge LLMs because computational and memory demands rise, alongside difficulties in preserving context. This survey organizes methods for extending sequence length across model-development stages. It concludes that these methods improve long-sequence processing capability, while noting continuing trade-offs and research challenges.
Problem
Long sequences can exceed LLM training contexts, while processing them raises computational and memory demands and challenges contextual coherence.
Source: §1 · p. 1Method
Results
The survey concludes that reviewed techniques improve LLM capability to process long sequences efficiently.
Source: §7 · p. 7Takeaways & Limitations
Computational cost, interpretability, external-knowledge integration, and trade-offs among complexity, speed, and accuracy remain research concerns.
Source: §7 · p. 7The survey identifies continuing challenges in computational cost, interpretability, and integrating external knowledge, alongside trade-offs among complexity, speed, and accuracy.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational requirements. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.
1 Introduction
LLMs must handle sequences longer than their training contexts, but long inputs raise computational, memory, coherence, and evaluation challenges. This survey reviews long-context methods and how they apply across model-development stages.
1 Introduction
- Long inputs can sharply increase computation and memory demands while making long-range coherence and evaluation more challenging.The survey notes that memory demands can exceed advanced GPU capacity.Source: §1 · p. 1
- The survey reviews architectural changes, including positional-encoding and attention modifications, alongside model-compression techniques.These methods aim to process longer sequences without exponentially increasing computational and memory demands.Source: §1 · p. 1
- Its focus is strategies for managing longer LLM input sequences, distinct from broader LLM surveys and reviews of efficient Transformers.Figure 1 groups the literature into length extrapolation, attention approximation, attention-free Transformers, model compression, and hardware-aware Transformers, with methods spanning training, fine-tuning, and inference.Source: §1 · p. 2, Figure 1 · p. 2
2 Length Extrapolation
Length-extrapolation methods modify positional information or manage long inputs in pieces to support sequences beyond fixed context windows. The section covers extrapolation, interpolation, segmentation and sliding, and prompt compression.
Section overview
- Positional extrapolation and interpolation modify positional embeddings, while segmentation and sliding divide or move context windows and prompt compression condenses inputs.The section presents these as approaches for supporting longer sequences.Source: §2 · p. 2
Positional Extrapolation and Interpolation
- ALiBi biases query-key scores by token distance and reports stronger extrapolation to unseen lengths than other positional-encoding schemes.xPOS extends causal RoPE with a dimension-wise exponential decay factor; CLEX generalizes positional-encoding scaling with ordinary differential equations.Source: §2 · pp. 2–3
- Linear positional interpolation scales position indices to the pretraining context limit and is reported as more stable and less fine-tuning-intensive than direct extrapolation.YaRN unevenly interpolates RoPE frequencies while preserving high-frequency components to retain positional information.Source: §2 · p. 3
Context Window Segmentation and Sliding
- Segmentation methods process long inputs in chunks; StreamingLLM combines window context with the first token to generalize finite-window models to infinite sequence lengths without fine-tuning.Structured prompting encodes demonstration examples individually, while PCW restricts attention to separate chunks.Source: §2 · p. 3
Prompt Compression
- Prompt compression removes redundant or extraneous input while retaining important information; LLMLingua reports compression up to 20 times with minimal performance decline.LongLLMLingua reports up to 21.4% improvement in retrieval-augmented generation while using one quarter of the tokens; MemGPT autonomously manages context through function calls and a memory hierarchy.Source: §2 · p. 3
3 Attention Approximation
Attention-approximation methods seek to reduce the quadratic computation and memory costs of self-attention while retaining its effectiveness. The section surveys low-rank, sparse-pattern, and softmax-free approaches.
Section overview
- Vanilla self-attention has O(n^2) computation and memory complexity, motivating low-rank, sparse, and softmax-related approximations.These approaches aim to approximate attention while managing long sequences more efficiently.Source: §3 · p. 3
Low-rank Decomposition
- Low-rank methods shrink attention computations by projecting or decomposing Q, K, and V; Autoformer instead uses FFT-based autocorrelation for time-series decomposition.Deeptensor uses a DNN to learn a tensor-decomposition regularizer for non-Gaussian tensor distributions.Source: §3 · pp. 3–4
Sparse Pattern
- Sparse attention limits each token to selected tokens; LogSparse guarantees pairwise information exchange with O(n(Log n)^2) memory, while LongNet reports O(n) computation.LongNet uses mixed dilation rates for local and global dependencies; Routing Transformer uses dynamic key-value clustering and has O(n^1.5) computation.Source: §3 · p. 4
Softmax-free Attention
- Softmax-free methods replace softmax with alternatives such as cosine, Gaussian-kernel, L1-norm, ReLU, or kernelized attention.Performer models attention with positive orthogonal random features and linear time and space complexity; RetNet supports parallel training and sequential inference.Source: §3 · p. 4
4 Attention-free Transformers
Attention-free Transformers provide token-dependency information without conventional attention and maintain sub-quadratic memory complexity. The section surveys state-space models and position-dependent approaches.
State Space Model
- SSMs compute output sequences from input sequences through linear projections of hidden states and can be analytically unrolled as convolutions.The convolution can process full sequences rapidly and in parallel, but its kernel's computational expense constrained SSM applications until S4.Source: §4 · pp. 4–5
- H3 combines two SSMs for local attention and global recall, while Hyena uses long convolutions and data-controlled gating and reports comparable perplexity at reduced computational cost.Mamba augments SSMs with MLPs and reorganizes gating to refine reasoning capabilities.Source: §4 · p. 5
Position-dependent Attention
- AFT uses a fixed, learnable position-based matrix W with K and V but no Q; RWKV uses a smaller vector with linear time decay.RWKV can use a Transformer form for parallel training and an RNN form for inference without sequence-length limits.Source: §4 · p. 5
5 Model Compression
The survey reviews compression strategies that reduce model size, computation, or memory use to support longer inputs. It covers attention designs, quantization, and pruning, with trade-offs in speed, memory, and accuracy.
5 Model Compression
- Compression methods target model size, computation, or memory efficiency to help LLMs accommodate longer input sequences.The survey groups approaches around removing redundant weights, lowering computation precision, and improving memory efficiency or data storage.Source: §5 · p. 5
- Quantization-aware training adapts the network to quantization during training, while LLM-QAT uses data-free knowledge distillation to address the cost of training on the full dataset.LLM-QAT uses data generated by the model and is described as applicable independently of the original training dataset.Source: §5 · p. 5
- Post-training quantization reduces weight and activation precision after training to lower memory and computation needs, but can degrade performance.It offers an efficiency route without extensive training; the survey distinguishes weight-only from weight-activation quantization.Source: §5 · p. 5
- PTQ examples include LLM.int8(), which halves inference memory while matching full-precision performance, and OPTQ, which uses 3- or 4-bit weights with negligible accuracy degradation.AWQ retains weights associated with large activations in full precision, while OWQ assigns higher precision to weights associated with activation outliers.Source: §5 · pp. 5–6
- Structured pruning removes larger structures such as neurons, layers, or matrix rows and columns; Sheared LLaMA can outperform same-size models trained from scratch.Its two-stage approach first prunes toward a target shape, then modifies model structure during training based on losses across domains.Source: §5 · p. 6
- Unstructured pruning removes individual parameters but can hinder efficiency gains because irregular structures and memory access require specialized deployment support.SparseGPT compresses models by as much as 60% with almost no performance impact, while Wanda selects weights using their values and input activations.Source: §5 · p. 6
- Multi-query attention reuses the same key-value tensors across query heads, reducing decoding memory bandwidth and enabling longer contexts and faster decoding.Pope et al. [2023] report up to 32× larger context lengths with MQA; grouped-query attention shares a subset of KV tensors to trade off speed and performance degradation.Source: §5 · p. 6
6 Hardware-aware Transformers
The section surveys hardware-aware methods for making long-sequence processing more efficient across GPU memory, devices, and inference-time memory management. These methods address distinct resource bottlenecks rather than relying on one technique.
6 Hardware-aware Transformers
- FlashAttention uses IO-aware, tiled attention without materializing the full softmax matrix, while BPT fuses feedforward and attention operations and handles sequences up to four times longer than FlashAttention.FlashAttention recomputes attention during the backward pass using blocks in SRAM rather than storing attention results in HBM.Source: §6 · p. 6
- For SSMs, FlashConv uses FFT decomposition and sequential processing of sequence portions, while Mamba combines kernel fusion, parallel scan, and recomputation for accelerator memory efficiency.These methods address convolution-dominated training computation in SSMs.Source: §6 · pp. 6–7
- Ring Attention distributes block-wise attention across GPUs while overlapping key-value communication with computation, enabling sequences several times longer than BPT.LongNet distributes multiple attentions with distinct dilated sparsity patterns across devices.Source: §6 · p. 7
- Autoregressive generation is memory-bound, and growing KV caches with unpredictable lifetimes make efficient batching and memory use difficult.PagedAttention segments KV caches into blocks to mitigate internal and external fragmentation; Flash-Decoding parallelizes over the key/value sequence and achieves up to 8× faster generation for very long sequences.Source: §6 · p. 7
- FlashDecoding++ removes synchronization for partial softmax computations, while LLM-in-Flash loads model parameters from flash memory into DRAM on demand.LLM-in-Flash could load a model up to twice the available DRAM size, potentially supporting longer sequences, though it is not specifically designed to increase sequence length.Source: §6 · p. 7
7 Conclusion and Future Directions
The survey reviews methods for extending LLM sequence length across architectural changes and training, fine-tuning, and inference. It identifies persistent challenges and proposes directions for improving long-sequence processing and evaluation.
7 Conclusion and Future Directions
- The survey synthesizes approaches including positional-encoding and attention-mechanism changes, as well as methods applicable during training, fine-tuning, and inference.These methods aim to improve long-sequence handling without significantly increasing computational cost.Source: §7 · p. 7
- Computational cost, interpretability, and external-knowledge integration remain challenges, alongside trade-offs among model complexity, processing speed, and accuracy.The survey suggests more efficient architectures, effective knowledge integration and retrieval, and training methods such as curriculum learning.Source: §7 · p. 7
- Future evaluation should include comprehensive benchmarks and datasets that specifically challenge long-context processing capabilities.The survey also points to gradually increasing sequence lengths during training as one direction for investigation.Source: §7 · pp. 7–8