A Comprehensive Survey on Long Context Language Modeling

• Recent LCLMs extend context windows from thousands to millions of tokens, enabling single-pass long-document processing and test-time exploration, reflection, backtracking, and summarization.Source: §1 · p. 5
• The survey addresses three research questions on obtaining effective LCLMs, efficient training and deployment, and comprehensive evaluation and analysis, extending coverage to applications.Source: §1 · p. 5
• For effective LCLMs, it reviews pre- and post-training data strategies, position embeddings and architectural designs, and workflows including prompt compression, memory, RAG, and agents.Source: §1 · p. 5
• For efficient training and inference, the survey examines I/O, GPU memory-access, and communication-computation optimizations alongside five efficient inference strategies.The inference strategies include quantization, memory management, prefilling-decoding disaggregation, GPU-CPU parallel inference, and speculative decoding.Source: §1 · pp. 5–6
• The evaluation and analysis sections cover long-context comprehension and long-form generation benchmarks, external performance measures, and internal model-structure analyses.External analyses include effective context length, perplexity, and lost-in-the-middle; internal analyses include position embeddings, attention heads, and MLP layers.Source: §1 · pp. 6–7
• The survey covers applications in agents, RAG, code, and multimodal tasks, and proposes five future directions while aiming to inform researchers and practitioners.The directions include long chain-of-thought reasoning, effective context extension, efficient architectures and infrastructure, robust evaluation, and mechanistic interpretability.Source: §1 · p. 7
• Unlike existing surveys that often focus on specific topics, this survey provides an in-depth overview spanning the broader LCLM landscape and its research questions.Source: §1 · p. 7

2. Data

• The data section presents long-context data strategies for both pre-training and post-training, with pre-training covering filtering, mixture, and synthesis.Figure 3 illustrates the LCLM training pipeline.Source: §2 · p. 7, Figure 3 · p. 7

2.1.1. Data Filtering

• Standard pre-training data filtering uses heuristic rules, deduplication, embedding-based semantic deduplication, and diversity- or difficulty-oriented selection [430, 472, 651] [274] [3] [357] [521].Examples include removing short entries, MinHashLSH, dataset-graph message passing, and clustering.Source: §2.1.1 · p. 8
• Long-context filtering evaluates text coherence, cohesion, and complexity [333], long-range dependency characteristics [61], or attention patterns [576].The dependency criteria are strength, distance, and specificity; LongAttn uses self-attention to quantify long-range dependencies for data selection.Source: §2.1.1 · p. 8

2.1.2. Data Mixture

• Long-context mixture strategies studied include cutting documents at 4K or 128K, global or per-source upsampling, and upsampling Arxiv, books, or GitHub.Earlier approaches often chose domain weights heuristically or tuned them on downstream tasks; Data Mixing Laws and RegMix use scaling laws to seek efficient mixtures.Source: §2.1.2 · p. 8
• Continual pre-training on a small amount of long-context data improves 7B retrieval, while scaling context calls for oversampling lengthy sequences and preserving domain diversity.ProLong finds that combining code repositories and long books with high-quality short-context sources improves extended-input performance while preserving short-context proficiency.Source: §2.1.2 · pp. 8–9
• GrowLength progressively expands input size during pre-training to optimize computational resources and improve efficiency.ProLong mixes code repositories and long books with high-quality short-context sources to support extended inputs while retaining short-context proficiency.Source: §2.1.2 · p. 9

2.1.3. Data Synthesis

• Long-context data synthesis combines semantically related texts or non-adjacent sentences, packs similar retrieved documents, or groups diverse documents through shared queries [171] [278] [481] [135].ICP [464] uses a traveling-salesman algorithm to address redundancy, while another method [519] shuffles document chunks into knotted structures for models to untangle and search.Source: §2.1.3 · p. 9

2.2.1. Data Filtering

• Earlier post-training filters selected influential short-form SFT samples, while GATEAU [470] targets long-range dependencies using Homologous Models’ Guidance and Contextual Awareness Measurement.Prior work used proprietary-model feedback [57, 60, 330, 576] or open-source-LLM metrics [44, 146, 288, 592], but overlooked long-context alignment.Source: §2.2.1 · p. 9

2.2.2. Data Synthesis

• Ziya-Reader [180] creates multi-document QA focused on different context positions, while Xiong et al. [599] prompt a model to generate QA pairs from a selected pre-training chunk.These tasks address “lost in the middle” [324] or derive question-answer pairs from selected source text.Source: §2.2.2 · p. 9
• An et al. [13] use information-intensive QA covering details from one segment and reasoning across multiple segments; Xu et al. [605] assemble related paragraphs and insert a ground-truth summary to create long-context instructions.Source: §2.2.2 · p. 9
• MIMG [74] improves long-context instruction-data quality with verification and single-hop generation agents, question sampling, and a multi-hop question-merger agent.Source: §2.2.2 · p. 10
• Long-context preference optimization addresses a gap left by methods focused on short contexts, including DPO [429] and its efficiency or effectiveness variants.DPO avoids a separate reward model and trains rejection of misaligned responses and acceptance of preferred ones; variants include SimPO [365], ORPO [190], TPO [445], 2D-DPO [294], and Constitutional DPO [551].Source: §2.2.2 · p. 10
• LongReward [654] scores responses across helpfulness, logicality, faithfulness, and completeness; LOGO [503] synthesizes preference and dispreference data, while LongDPO [412] uses Monte Carlo Tree Search for stepwise preferences and step-level DPO.Source: §2.2.2 · p. 10

2.3. Training Data

• The survey summarizes long-context datasets for both pre-training and post-training in Table 2.Source: §2.3 · p. 10, Table 2 · p. 10

3.1.1. Position Embedding Types

• Position embeddings range from absolute and relative encodings to content-aware designs, including RoPE’s relative-distance behavior and CoPE or DAPE’s context-dependent positions.Absolute approaches include functional, learned, and no-position variants; relative approaches include T5-style biases and ALiBi’s recency bias.Source: §3.1.1 · pp. 12–14

3.1.2. Extrapolation Methods of Position Embeddings

• Position extrapolation either maps longer inputs into supported positions, often without tuning, or extends the supported range, often through training.Mapping methods reorganize or interpolate positions, while range-extension methods use hierarchical encodings or simulate long-position training with shorter inputs, as in PoSE and its follow-ups.Source: §3.1.2 · pp. 14–16

3.2.1. Transformer-Based Architecture

• Transformer self-attention has O(n^2) time complexity, motivating sparse-attention methods that reduce computation and KV-cache memory during training or inference.Training-based approaches sparsify heads or context windows; training-free approaches include static windows and dynamic token retention, with layer- and head-specific cache allocation also explored.Source: §3.2.1 · pp. 16–19
• Other Transformer modifications structure attention hierarchically, combine it with recurrent memory for long-range dependencies, or compress cached keys and values to reduce inference memory.Examples include sentence-to-document attention, segment-level recurrence, and Multi-head Latent Attention’s compression of keys and values into a latent vector.Source: §3.2.1 · pp. 19–20

3.2.2. Linear-Complexity Architecture

• Linear-complexity approaches include State Space Models, linear-attention improvements, and test-time training; SSMs model evolving internal states and outputs from inputs.For discrete NLP inputs, LSSL introduces recurrent and convolutional representations of continuous SSMs, while Mamba discretizes SSMs using zero-order hold.Source: §3.2.2 · pp. 20–21
• Mamba’s hardware-aware parallel-scan algorithm achieves 5x higher throughput than the Transformer and scales linearly with sequence length.Mamba addresses the limitations of time-invariant state-space models by making their parameters input-dependent.Source: §3.2.2 · p. 21
• Linear attention methods use kernelized attention or chunk-wise recurrence to reduce memory use or achieve linear sequence complexity while supporting training parallelism and low-cost inference.The linear transformer uses kernel-based self-attention and associative matrix products [245], while RetNet’s chunk-wise recurrent mode combines linear complexity with parallel training and low inference cost [494].Source: §3.2.2 · p. 22

3.2.3. Hybrid Architecture

• Mamba and Mamba-2 perform well in language modeling but underperform Transformers on long-context tasks; adding standard Transformer layers can overcome this limitation.The comparison is reported by Waleffe et al. [530] and Park et al. [402]; Chen et al. [67] and Waleffe et al. [530] find that adding Transformer layers helps.Source: §3.2.3 · p. 23
• At 256K context, Jamba 1.5’s 7:1 Mamba-to-Transformer ratio requires 9GB of KV cache versus 80-88GB for similarly sized Transformer models.Layer-wise hybrids mix linear-complexity and standard attention across layers; experiments also found this approach outperformed pure Mamba and pure attention models of similar size on standard benchmarks.Source: §3.2.3 · p. 24
• Hybrid attention designs use approximately 6:1 or 7:1 ratios of linear-complexity to full attention, a balance reported across independent research efforts.Gemma uses a 1:1 full-attention-to-sliding-window-attention ratio, while Character.ai research found approximately 6:1 optimal [49].Source: §3.2.3 · p. 24
• Prefilling-decoding hybrids use linear-complexity attention for prefilling and hybrid attention for decoding, with YOCO reducing KV-cache requirements by roughly the number of model layers.GoldFinch’s TokenCat generates a compressed key cache in linear time and space, with cache size savings 756-2550 times smaller than traditional Transformer caches.Source: §3.2.3 · pp. 24–25
• Head-wise hybrids assign different mechanisms within one layer: Hymba combines Transformer attention and state-space heads, while Samba combines Mamba with sliding-window attention.Hymba’s attention heads provide high-resolution recall and its SSM heads summarize context; Samba combines local attention with global attention for long-distance dependencies.Source: §3.2.3 · p. 25

4.1. Prompt Compression

• Prompt compression reduces long inputs while retaining essential information, typically through plug-in modules that leave the original LLM parameters unchanged.It addresses longer prompts’ costs to inference speed, memory, and API usage, and divides into hard compression using words or subwords and soft compression using embeddings.Source: §4.1 · p. 26

4.1.1. Hard Prompt Compression

• LLMLingua achieves up to 20x prompt compression, while LLMLingua-2 reports 3x-6x speed improvements over LLMLingua while preserving crucial information.Hard compression either selects relevant tokens or rewrites prompts for brevity. Methods include token- and sentence-selection schemes, RL-based compression, and rewriting; LongLLMLingua adds question-aware compression and document reordering. [221] [224] [397]Source: §4.1.1 · pp. 26–27

4.1.2. Soft Prompt Compression

• Soft compression replaces text prompts with continuous embeddings; LLM-fixed methods preserve the original parameters, whereas gist-token methods typically update them.LLM-fixed approaches train additional modules, with Kuratov et al. [261] demonstrating near-lossless ∼1500× compression as an upper-bound reference. Gist methods condense context into reusable tokens, but typically modify model parameters. [378]Source: §4.1.2 · pp. 27–28

4.2. Memory-Based Methods

• Memory-based methods externalize long-context storage as continuous, textual, or parametric memory, easing context-window limits but introducing retrieval, consistency, and update challenges.Examples include LongMem’s retrieval of cached key-value pairs, Generative Agents’ recency, importance, and relevance scoring, and DSI’s storage of document-query mappings in model weights. [553] [401] [507]Source: §4.2 · pp. 29–30

4.3. RAG-Based Methods

• RAG reduces long contexts through chunking, retrieval, and generation, but chunking must balance size, contextual integrity, efficiency, and data format.Dense retrievers are mostly used for semantic matching; generation can concatenate retrieved passages, use soft features, adjust decoding with nearest neighbors, or integrate information through trained cross-attention. [211] [247] [34]Source: §4.3 · pp. 30–32

4.4. Agent-Based Methods

• Agent-based methods use single-agent or multi-agent architectures to manage long texts through memory, planning, reflection, and collaboration.For generation, Re3 [618] iteratively plans, drafts, rewrites, and edits stories, while RecurrentGPT [698] combines short-term summaries with retrieval of relevant prior paragraphs. Multi-agent methods divide processing among specialized agents, such as workers coordinated by a manager. [674] [681]Source: §4.4 · pp. 32–33

5. Infrastructure

• Infrastructure methods for LCLM training and inference differ from those typically used for general LLMs, with most emphasizing efficiency and others combining engineering and algorithmic approaches to improve long-context performance.The section’s Table 5 compares infrastructure optimizations using ✓ for positive impact, ✗ for negative impact, and – for no impact or not involved.Source: §5 · p. 34, Table 5 · p. 34

5.1. Training of LCLMs

• Long sequences make GPU memory limits, inefficient computation and memory access, and communication bottlenecks central challenges for training efficiency.The survey identifies parallel computation and communication–computation overlap as key strategies for addressing these constraints and maximizing hardware utilization.Source: §5.1 · pp. 34–35

5.1.1. I/O Optimization

• Long-context data pipelines must handle large, variable-length batches while balancing data packing, I/O throughput, and CPU, memory, and bandwidth limits.I/O threads and pinned memory can help but require case-specific tuning; packing concatenates samples and may alter data order and distribution, while pre-fetching, caching, and 3FS can overlap or reduce I/O latency. Progressive length extension is used by recent pretrained models and has improved performance on long-text benchmarks.Source: §5.1.1 · p. 35

5.1.2. Optimizations on GPU Constraints and Memory Access

• Mixed precision and quantization reduce long-context training memory demands, but require managing overflow and precision-related performance loss.FP16 uses half the storage of FP32 but can overflow; BF16 increases exponent bits and is standard alongside FP16, while FP32 is retained for precision-sensitive operations and communication reductions. FP8 may offer a 2x performance boost over BF16 for less precision-sensitive matrix multiplications; quantization also targets weights, optimizer states, and activations, with scaling and outlier suppression used to limit error.Source: §5.1.2 · pp. 35–36
• FlashAttention uses block-wise processing and optimized softmax to reduce memory access, while later versions refine scheduling and overlap computation with data loading.FlashAttention-v3 adds FP8 acceleration, asynchronous loading and computation, overlapping softmax and GEMM, and block-wise quantization; sparse variants are also discussed.Source: §5.1.2 · p. 36
• Sharding and context parallelism reduce per-device memory pressure, while Ulysses Parallelism combines layer and context sharding for extremely long windows.Sequence Parallelism distributes layers across devices but requires communication; Context Parallelism divides the context among devices, and Ulysses uses an interleaved strategy to limit communication overhead.Source: §5.1.2 · p. 37

5.1.3. Optimizations on Communication-Computation Overlapping

• Communication–computation overlap improves utilization in distributed long-context training, where communication can lag computation and high per-sample memory use limits local batches.Gradient accumulation simulates large-batch training through mini-batches and can overlap forward computation with backward communication; ZeRO-2 may require communication after each mini-batch, limiting complete overlap.Source: §5.1.3 · p. 37

5.2.1. Quantization

• Quantization reduces decoding data transfers by compressing the KV cache or both model weights and activations; KV-only approaches may require specialized kernels without native mixed-precision support.KV-cache methods can quantize keys and values differently, filter outliers, or adjust channel sizes [192, 339, 686].Source: §5.2.1 · pp. 38–39

5.2.2. Memory Management

• Virtual-memory techniques reduce KV-cache fragmentation, while prefix-sharing schedulers deduplicate caches across batches to improve memory use and computational efficiency.PagedAttention [264] places corresponding token caches across layers and heads in the same page; ChunkAttention [630], MemServe [195], and SGLang [691] support cache sharing.Source: §5.2.2 · p. 39

5.2.3. Prefilling-Decoding Disaggregated Architecture

• Prefilling-decoding disaggregation assigns compute-intensive prefill and bandwidth-sensitive decoding to dedicated server pools, improving TTFT and TPOT through stage-specific resource allocation.DistServe [694] optimizes allocation and scheduling, Splitwise [403] studies cluster allocation, and Mooncake [418] uses prediction-based early rejection; CacheGen [337] reduces cache-transfer time, while LoongServe [571] adapts sequence parallelism.Source: §5.2.3 · p. 39

5.2.4. GPU-CPU Parallel Inference

• GPU-CPU parallel inference offloads KV caches to relieve GPU memory pressure and overlaps CPU work with PCIe transfers to reduce transfer bottlenecks.FlexGen [462] and PipeSwitch [22] overlap computation with loading the next layer’s cache, while FastDecode [178] computes attention scores on the CPU.Source: §5.2.4 · p. 40

5.2.5. Speculative Decoding

• Speculative decoding drafts multiple tokens and verifies them together, reducing model-parameter transfers; variants use self-drafting, sparse attention, or specialized draft architectures [276].Self-speculative methods use layer skipping [119, 191, 657], while Medusa [42], Eagle [302], and Mamba-based draft models [84] modify the draft architecture.Source: §5.2.5 · p. 40

6. Evaluation

• Long-context modeling is evaluated as long-context comprehension of inputs and long-form generation of outputs.These are the two aspects used to divide the capacity of long-context modeling.Source: §6 · p. 41
• For comprehension, the section covers evaluation paradigms, recent benchmarks, and how to design better benchmarks.Source: §6 · p. 41
• For long-form generation, the section reviews its definition, representative benchmarks, data sources, evaluation methods, challenges, and trends.Source: §6 · p. 41

6.1.1. Evaluation Paradigm

• Long-context comprehension is organized into five levels: language modeling, retrieval, aggregation, reasoning, and real-world adaptation, with practical tasks combining core capabilities.Real-world tasks include question answering, summarization, document retrieval and reranking, retrieval-augmented generation, in-context learning, and coding.Source: §6.1.1 · pp. 41, 43
• A falling cumulative NLL with token position or falling sliding-window PPL as window size grows indicates that a model uses longer context to predict tokens more accurately.The survey describes both cumulative average NLL and sliding-window PPL as ways to assess long-context language modeling.Source: §6.1.1 · p. 41
• Retrieval tests exact or semantic extraction; aggregation tests statistical or semantic synthesis; reasoning tests parallel or iterative inference across distributed information.Synthetic evaluations include needle retrieval, numerical operations and synthesis tasks, and multi-needle reasoning benchmarks such as BABILong and NeedleBench.Source: §6.1.1 · pp. 42–43
• Real-world evaluations span question answering, summarization, document ranking, RAG, many-shot ICL, and repository-level coding, where current models still face performance limitations.ICL studies report degradation beyond some context lengths, example-order effects, and recency bias; repository-level code remains challenging, including on SWE-Bench.Source: §6.1.1 · pp. 43–45

6.1.2. Evaluation Benchmarks

• Synthetic benchmarks commonly use NIAH variants to test retrieval robustness, mostly rely on automatic metrics, and enable controlled comparisons of language, position, and patterns.A few use LLM or hybrid evaluation for longer outputs such as summaries or mathematical reasoning.Source: §6.1.2 · p. 45
• Real-world benchmarks most often assess question answering, then summarization; some use multiple-choice formats, and many include synthetic tasks alongside human-annotated ones.Medical, financial, and code-related domains particularly demand strong long-context processing.Source: §6.1.2 · p. 45

6.1.3. Discussion

• Good benchmarks should match model context lengths, test fundamental capabilities under dispersed and semantically oriented information, and assess downstream tasks such as QA, summarization, RAG, and ICL.Synthetic-task strength alone does not guarantee downstream competence; multiple-choice formulations also enable direct accuracy measurement.Source: §6.1.3 · p. 48

6.2.1. Evaluation Benchmark

• Long-form generation requires a prompt that explicitly or implicitly calls for a lengthy response, with “long” defined relative to task type.The section gives over 1,000 words for writing and over 500 words for question answering as typical thresholds.Source: §6.2.1 · pp. 48–49
• Long-form QA benchmarks span open-ended, ambiguous, scientific, expert-written, and retrieval-augmented questions, while evaluation also addresses factuality and fairness.Min et al. [372] assess the proportion of supported atomic facts; Jeung et al. [218] cover 14 topics and 10 demographic axes.Source: §6.2.1 · p. 49
• Long-form summarization benchmarks test both comprehension and condensation of lengthy inputs and the generation of expanded summaries.MultiNews [121] covers multi-document news, AQUAMUSE [257] mines summaries from Natural Questions and Common Crawl, and LCFO [91] includes 252 long documents across domains.Source: §6.2.1 · p. 49
• Instruction-following benchmarks assess straightforward and constrained compliance, structured problem-solving, personalized generation, and long creative outputs.Examples include DoLoMiTes [359], LongLaMP [258], LongWriter-6K [21], Suri [410], and the LongGen benchmark introduced with Self-Lengthen [424].Source: §6.2.1 · p. 51
• Mixed benchmarks broaden evaluation across tasks: Facts Grounding [215] covers question answering, summarization, and rewriting, while HelloBench [425] spans five tasks and 38 subcategories.HelloBench contains 647 samples sourced from real-world scenarios and publicly available datasets.Source: §6.2.1 · p. 51

6.2.2. Data Source

• Benchmark data comes from web, user, synthetic, crowdsourced, and publicly available sources, each offering different benefits and limitations.Web data can be diverse but low quality; user data reflects practical scenarios but varies in quality; synthetic data can ease automatic evaluation but may lack real-world alignment; existing datasets risk leakage; crowdsourcing can be high quality yet impractical.Source: §6.2.2 · pp. 51–52, §6.2.4 · p. 53

6.2.3. Evaluation Paradigm

• Evaluation combines automatic metrics, human assessment, and LLM-as-a-Judge, since automatic metrics may correlate poorly with human judgments.Automatic measures include semantic similarity, repetition or diversity, accuracy, and task-specific metrics; LLM judges can score proxy-question correctness, compare responses, or apply checklists with human-correlated weights.Source: §6.2.3 · pp. 52–53

6.2.4. Discussion

• The section proposes user-sourced data with detailed post-processing as a promising direction for realistic, high-quality evaluation.Direct user interaction data captures real issues, but filtering low-quality samples and human involvement in post-processing are essential; fewer samples should be acceptable if quality is ensured.Source: §6.2.4 · pp. 53–54

7. Analysis

• The analysis examines LCLM performance externally and model structure internally.Figure 9 illustrates this analysis.Source: §7 · p. 54, Figure 9 · p. 54

7.1.1. The False Promise of Support Context Length

• RULER [194] finds that most models’ effective context length rarely exceeds half their claimed length; performance also drops when key information lies in the middle [324].The lost-in-the-middle pattern is U-shaped, with stronger performance when information appears at the beginning or end; An et al. [13] and He et al. [179] also report declining tracking as distance increases.Source: §7.1.1 · pp. 54–55, Table 9 · p. 55

7.1.2. Relevance of Long Context Perplexity and Real-World Performance

• Long-context perplexity often failed to predict comprehension [198, 10, 489], but GovReport perplexity correlated with downstream results [345], as did LongPPL [125].The GovReport study fine-tuned LLaMA2-7B using multiple context-extension methods and assessed Needle-in-a-Haystack, LongBench, and RULER; LongPPL computes perplexity only on context-sensitive token distributions.Source: §7.1.2 · p. 55

7.1.3. RAG Meets Long Context LLMs

• With ample computational resources, LCLMs deliver superior average performance to RAG, despite not being specifically trained for these tasks [269, 303].RAG retrieves key information before generation, whereas LCLMs process entire corpora as input and implicitly identify relevant information.Source: §7.1.3 · pp. 55–56
• To balance performance and cost, Self-Route [303] routes queries between RAG and LCLMs, while other work improves retrieval units [228] and mitigates hard negatives [230].The hard-negative approaches include training-free and training-based methods.Source: §7.1.3 · p. 56

7.2.1. Positional Embedding

• Position interpolation [63] linearly rescales positions into the pretrained range, while NTK-aware scaling [32] and YaRN preserve high-frequency embeddings before transitioning to scaled low frequencies.The survey notes that injecting high-frequency embeddings improves performance and convergence [499].Source: §7.2.1 · pp. 56–57
• Scaling down the rotation base gains effectiveness only up to the critical base, while scaling up shifts the phase-change point beyond which perplexity explodes [332].The critical base is defined by the largest wavelength smaller than the pretrained context length.Source: §7.2.1 · p. 57
• Scaling down rotation bases yields only superficial long-context capability; effective context length relates polynomially to base size, and larger bases slow attention decay with distance [363].The authors derive a lower bound on the rotation base for an expected context length.Source: §7.2.1 · pp. 57–58

7.2.2. Attention and MLP Analysis

• Retrieval heads extract information from long contexts, and ablating them degrades performance; refinements identify echo and induction heads [500] and retrieval-reasoning heads [134].Echo heads attend to identical tokens at the current location, while induction heads attend to the token after an identical token.Source: §7.2.2 · p. 58
• As sequences grow, softmax attention scores become more uniform [173], while attention sinks concentrate scores on the first token and removing sink tokens from KV cache impairs performance [595].Proposed mitigations include a trainable sink token or alternatives to softmax such as SoftMax-One [370].Source: §7.2.2 · p. 58
• Certain MLP neurons correlate strongly with token position and can act as n-gram detectors; relative-position-sensitive neurons may also exist [529].The cited analysis examines absolute positional embeddings.Source: §7.2.2 · p. 58

7.2.3. Layer Interaction

• Alternating full attention with linear-complexity mechanisms across layers improves extrapolation, while alternating NoPE and RoPE combines retrieval strengths with RoPE’s local-information recency bias [612].Examples of linear-complexity mechanisms include window attention [612] and lightning attention [373].Source: §7.2.3 · p. 58

8.1. Application in Agent

• Long-context comprehension supports agents that must reason over extended interaction histories, observations, and plans [587, 626, 700].Applications include GUI agents [197, 321, 597, 696], software-engineering agents [229, 556, 589, 617, 673], gaming [498], travel planning [50], machine-learning problem solving [46, 201], and agent optimization [701, 711]. Platforms such as Dify [509] and Coze [508], and general agents such as OpenAI Computer-Using Agent [516] and Manus [515], support tasks from orchestrating apps to web browsing and coding in virtual PCs.Source: §8.1 · pp. 59–60

8.2. Application in RAG

• Long-context technology strengthens RAG by enabling larger text chunks, more relevant retrieval, and more complete answers, while supporting complex queries and multi-hop reasoning [230, 638, 707].LongRAG [227] combines a long retriever and long reader for document question answering; longer retrieved fragments can improve answer relevance and completeness [230, 463]. The section also describes “Reasoning with Attributions” as using a larger context window for multi-hop reasoning and reducing reliance on external tools.Source: §8.2 · p. 60

8.3. Application in Chatbot

• Long-context dialogue systems retain conversational history to improve fluency, contextual coherence, and personalized or style-consistent interactions [268, 693, 553, 555].Persistent-memory platforms include ChatGPT [392] and Pi [209], while Character AI [49] and Talkie [6] support style-consistent dialogue; maintaining context across conversations is valuable for tutoring, healthcare monitoring, and counseling [235].Source: §8.3 · p. 60

8.4. Application in Code

• Long-context models improve repository-level coding by processing extensive codebase context, enabling more scalable code completion, understanding, and generation [225, 468].Earlier systems used retrieval or repository-aware prompts, including RepoCoder [650] and RLPG [469]; models such as StarCoder2 [341], Qwen2.5-Coder [207], and Granite Code Models [374] use long-context comprehension. GitHub Copilot [151] and Anysphere Cursor [15] apply it to completion, documentation, and predictive debugging.Source: §8.4 · pp. 60–61

8.5. Application in Traditional NLP Tasks

• Long context improves traditional NLP tasks by enabling whole-document summarization, longer-text semantic retrieval, and translation of lengthy documents [155, 552, 646, 391, 168, 58, 186, 546].Longformer [27] and LongT5 [170] support document summarization; text-embedding-3-large [391], jina-embeddings-v2 [168], and BGE-M3 [58] accommodate longer retrieval inputs. Extended context can also help translate polysemous words and make whole-book translation more feasible [349].Source: §8.5 · pp. 61–62

8.6. Application in Multimodal Tasks

• Multimodal long-context methods extend MLLM context through long multimodal training data, positional modeling, and visual-token or video-frame reduction.Training strategies include progressively longer multi-image and video contexts [633, 115, 296, 176, 280, 523, 554, 609, 660, 684], preference optimization [291], and using long-context LLMs as foundations [216]. Methods extend RoPE dimensions [143, 485, 548] or compress visual tokens with modules [548, 73, 9, 287, 628, 709]; training-free approaches select relevant visual tokens [175, 329, 524, 620] or keyframes [561, 623, 307, 176, 637, 128].Source: §8.6 · pp. 62–63
• Multimodal long-context benchmarks test question answering over multiple images or long videos, including many-shot learning, retrieval, and document comprehension.MMLongBench [560] covers multimodal needle-in-a-haystack, long-document VQA, visual RAG, and many-shot ICL; long-video benchmarks use diverse videos and challenging tasks [47, 148, 695, 712] or event-focused questions [115, 383, 575].Source: §8.6 · p. 63

8.7. Application in Specific Domains

• Long-context models are applied to news summarization [139], legal document interpretation [243], healthcare synthesis [123], financial analysis [360, 388, 432], and biology [187, 454].These applications use long-context processing to synthesize information or extract insights from extensive source material, including patient records, reports, molecular structures, and genomic sequences.Source: §8.7 · p. 63

9.1. Long Context Modeling for o1-like Long Reasoning

• o1-like models’ extended-CoT test-time scaling supports trial-and-error reasoning [59, 169, 206, 380, 419, 513, 675, 687], but redundant CoTs and degradation in very long chains motivate better evaluation, generation, and efficiency.Long-context models can help evaluate long reasoning processes, improve the reliability of generated chains, and adapt techniques such as KV-cache and prompt compression to reasoning workloads.Source: §9.1 · pp. 63–64

9.2. Further Extending Context Window and Improving Modeling Capabilities

• Future work should both extend context windows and improve effective modeling within supported limits, while pursuing performance-oriented architectures and domain-specific optimizations.The survey notes a gap between claimed and effective context lengths and points to enhanced positional encodings and hybrid attention as architectural directions.Source: §9.2 · pp. 64–66
• Long-context training should develop rewards for extensive inputs and better preference and data recipes, while leveraging strong LCLMs to filter training examples and assist both training stages.Directions include human-annotated or synthesized preferences, filtering for long-range dependencies, queries requiring dispersed evidence, transferable RL tasks, and domain and sequence-length distributions that balance cost and performance.Source: §9.2 · pp. 64–65
• Long-form generation needs stronger output planning and evaluation, since automated metrics such as ROUGE are unreliable for lengthy texts and manual assessment is costly.The survey identifies narrative creation, repository-scale code generation, and long CoT as promising applications, and argues that generation and automated evaluation should advance together.Source: §9.2 · p. 65

9.3. Efficient Architecture Design, Training, and Deployment of LCLMs

• Efficient LCLMs need memory-efficient KV-cache architectures, refined training partitioning and recomputation, fused and reordered inference, broader quantization, RL-specific framework optimization, and decoding hardware with greater HBM capacity and bandwidth.Selective 8-bit quantization has shown promise; decoding remains bandwidth-bound, motivating hardware designed specifically for that phase.Source: §9.3 · p. 66

9.4. More Reliable Evaluation Framework for Long Context Language Modeling

• Evaluation should target authentic usage and specialized domains, use efficient annotation for long inputs, and decompose long-form assessment into coarse-to-fine aspects that combine LLM judges with human or proxy evaluation.The survey notes that end-to-end evaluation of long responses remains challenging and calls for further accuracy improvements to systems such as ProxyQA [497], HelloBench [425], and LongFact [563].Source: §9.4 · p. 67

9.5. Towards Mechanistical Interpretability for Long Context Modeling

• Mechanistic interpretability can investigate positional information, causes of long-context biases and extrapolation failures, and model components whose modification could address these issues.The survey highlights retrieval heads [580], positional-embedding analysis by Voita et al. [529], positional bias discussed by Liu et al. [324], and digit-alignment failures reported by Zhou et al. [703].Source: §9.5 · pp. 67–68

Conclusion

• It surveys long-context data strategies, Transformer architectures and position embeddings, and workflow methods such as prompt compression and agent-based approaches.Data strategies cover pre-training and post-training; the workflow methods aim to enhance long-context abilities.Source: §10 · p. 68
• It covers evaluation benchmarks for understanding and generation, long-context training and inference infrastructure, and interpretability analysis of LCLMs.Source: §10 · p. 68
• It summarizes applications including agents, RAG, and code, discusses remaining challenges, and aims to guide developers and support foundation-model growth.Source: §10 · p. 68

Project Leaders

• Jiaheng Liu and Dawei Zhu organized the whole project.Liu is affiliated with Nanjing University, the Institute of Automation, Chinese Academy of Sciences, and M-A-P; Zhu is affiliated with Peking University.Source: Project Leaders · p. 69

Core Contributors (Alphabet Order)

• Core contributors covered architecture, applications, data, evaluation, workflow design, analysis, and infrastructure.Zhiqi Bai, Ge Zhang, and Zekun Wang contributed to architecture; Yancheng He to applications; Huanxuan Liao to data; Haoran Que and Ge Zhang to evaluation; Zekun Wang to workflow design; Chenchen Zhang to analysis; and Jiebin Zhang and Yuanxing Zhang to infrastructure.Source: Core Contributors (Alphabet Order) · p. 69

Contributors (Alphabet Order)

• Additional contributors worked on architecture, applications, data, workflow design, analysis, and future directions.Architecture contributors were Zhuo Chen, Yong Shan, and Ruijie Zhu; application contributors were Hangyu Guo, Shilong Li, and Yifan Song; Ziqiang Liu handled workflow design, Jiayi Tian data, Zhejian Zhou analysis, and Wenhao Wu future directions.Source: Contributors (Alphabet Order) · p. 69

Sponsor Committee (Alphabet Order)

• The sponsor committee included 13 members from universities, research institutes, companies, and independent research.Members were Junlan Feng, Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, Jian Yang, Wei Ye, Bo Zheng, and Wangchunshu Zhou.Source: Sponsor Committee (Alphabet Order) · p. 69

Corresponding Authors

• The corresponding authors were Jiaheng Liu, Dawei Zhu, Wenhao Huang, Sujian Li, and Zhaoxiang Zhang.Their listed affiliations include Nanjing University, Peking University, ByteDance, M-A-P, and the Institute of Automation, Chinese Academy of Sciences.Source: Corresponding Authors · p. 70