QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization

1 Introduction

• Long-context processing remains costly because of quadratic sequence computation, while LLMs can struggle to prioritize critical information in lengthy inputs.RAG relies on coarse chunk-level embeddings, while sparse attention can require substantial data construction, computationally intensive training, and specialized infrastructure.Source: §1 · pp. 1–2
• Dynamic context optimization replaces RAG’s coarse chunk retrieval with precise token-level selection and works as a plug-and-play component compatible with downstream LLMs without SA-style retraining.It compresses contexts into query-tailored segments at different granularities.Source: §1 · p. 2, Figure 2 · p. 3
• QWENLONG-CPRS reports context compression rates of 72.6 to 290.5 times and superiority over direct prompting, RAG, and sparse attention across four long-context benchmarks.The evaluated context lengths range from 4K to 2M tokens, with higher inference efficiency also reported.Source: §1 · p. 2
• QWENLONG-CPRS combines token-level critical scoring, a hybrid attention mechanism, and a language-modeling-head token critic, alongside a framework for training-data construction.Its hybrid attention combines bidirectional language modeling for context location with causal language modeling for language representation.Source: §1 · pp. 2–3
• The framework is presented as a context-optimization augmentation with performance gains and reduced inference overhead across LLM scales and context lengths.Evaluations across four benchmarks support the reported efficacy and versatility.Source: §1 · p. 3

2.1 Dynamic Context Optimization

• Context optimization seeks the shortest subset that preserves maximally informative content for high-quality responses, with a natural-language prompt controlling granularity and use.The selected context is formalized as Xs = Fϕ(P, q, Xl), where F is token selection.Source: §2.1 · pp. 3–4

2.2 Model Architecture

• System prompts, user queries, and long contexts jointly guide QWENLONG-CPRS to adaptively extract informative context.The prompt specifies optimized-context properties, such as granularity and its relationship to the query.Source: §2.2 · p. 4
• Causal lower layers preserve language modeling, while bidirectional upper layers use forward and backward signals to improve boundary detection.The design targets global awareness of long-range contextual dependencies while retaining linguistic knowledge.Source: §2.2 · p. 5
• A language-modeling head predicts token semantic categories while a secondary head scores sequence-label boundaries, combining both decisions under prompt conditioning.The framework uses the base model’s vocabulary and parameter inheritance to preserve linguistic knowledge.Source: §2.2 · p. 5
• Window-parallel inference splits long contexts into non-overlapping windows and processes each alongside the prompt and query.With fixed window size and parallelism, the stated prefill complexity is lower than direct prompting, and memory overhead per parallel unit remains fixed as context length scales.Source: §2.2 · p. 5

2.3 Training Data Construction

• Training uses multi-granularity data for context grounding and query-aware data for query-context interaction and response alignment.The query-aware set includes supporting-fact extraction and synthesized query-answer examples with answer-consistency verification.Source: §2.3 · pp. 5–6
• The training data covers keyword, sentence, and paragraph compression, including annotated documents and extractive summaries.The resulting multi-domain, multilingual corpus contains 126K samples and 1.2B tokens.Source: §2.3 · pp. 5–6

3.1 Implementation Details

• QWENLONG-CPRS is initialized from Qwen2-7b-Base, with causal layers 1–21 and bidirectional reasoning layers 22–28, then fine-tuned for three epochs.Training uses 8192-token windows, global batch size 256, learning rate 1e-5, Zero-3 with optimizer-state offloading, and random gradient masking on 50% of non-critical positions.Source: §3.1 · p. 6

3.2 Benchmarks

• Five long-context benchmarks cover synthetic retrieval and tracking, multilingual and multi-task contexts, long-document QA and summarization, complex reasoning, and needle retrieval.The set comprises Ruler-128K, InfiniteBench, LongBench V1, LongBench V2, and Needle-in-a-Haystack; InfiniteBench spans 122K–2M words, while needle tests span 32K–1M tokens.Source: §3.2 · pp. 6–7

3.3 Compared Baselines

• Comparisons include proprietary and open-source LLMs, RAG retrieving from 600-token chunks up to 16K tokens, and sparse-attention approaches.Sparse-attention comparisons include Minference, InfiniteRetrieval, MOBA, and NSA.Source: §3.3 · p. 7

4.1 Overall Results

• LLaMA3.1-8b-Instruct, Qwen2.5-7b-Instruct, and Qwen2.5-32b-Instruct gain 39.72, 55.79, and 19.26 on Ruler-128K, respectively, with gains across all evaluated LLMs.InfiniteBench gains are 13.30, 21.95, and 18.83 for those models, respectively; LongBench gains average +2.8 on V1 and +3.0 on V2.Source: §4.1 · p. 8
• Average gains reach 38.20 on Ruler-128K, 18.02 on InfiniteBench, and 10.5 on LongBench V2-Long; NIAH accuracy is perfect across depths and 32K–1M contexts.LongBench V1’s shorter 2K–18K contexts show minor improvements.Source: §4.1 · pp. 8–9, Table 3 · p. 8, Table 2 · p. 8
• QWENLONG-CPRS improves over RAG by 18.52% and 25.20% in multi-hop and high-similarity scenarios, and reports stronger comparisons than sparse-attention methods.System-prompt granularity variants show consistent performance gains; the NSA comparison reports relative improvements of +5.14 versus +2.62.Source: §4.1 · p. 9, Table 4 · p. 9, Table 5 · p. 9

4.2 QWENLONG-CPRS with Stronger LLMs

• 77.5 on Ruler-128K and 62.9 on InfiniteBench are the best reported scores among the stronger LLMs augmented with QWENLONG-CPRS.Average gains are 54.9, 49.0, 21.7, and 15.7 for LLMs with 32K, 64K, 128K, and 1M context windows, respectively.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Context Optimization Efficiency

• QWENLONG-CPRS uses 99.59%, 92.66%, and 99.66% fewer tokens than RAG’s peak requirements on three tasks, while exceeding RAG’s maximum scores by 17.27% on NIAH-sub and 4.03% on QA.RAG performance peaks at 16K retrieved tokens, then fluctuates or declines as retrieval grows on some tasks.Source: §4.3 · p. 11, Figure 6 · p. 11

4.4 Latency Analysis

• At 128K tokens, QWENLONG-CPRS gives 3.47× acceleration over direct prompting, with linear rather than quadratic latency growth.Minference is slower than direct prompting below 96K tokens (10.42s vs 8.35s at 64K); RAG has constant-time latency but accuracy costs.Source: Figure 7 · p. 11, §4.4 · p. 12

4.5 Prompt-Agnostic Integration with Foundation Models

• Standard and customized prompting have statistically comparable performance for Qwen2.5-7B-Instruct (∆=+0.20) and Qwen2.5-32B-Instruct (∆=-0.29).The authors report stable outputs and compatibility with existing LLM interfaces.Source: §4.5 · p. 12, Table 6 · p. 12

4.6 Case Study

• The case studies show query-relevant sentence compression for downstream inference and standalone contractual-element extraction without LLM integration.The examples include multi-value Needle-in-a-Haystack and English multi-hop QA cases, plus contract-element extraction.Source: §4.6 · p. 12

5 Conclusion and Future Works

• QWENLONG-CPRS enables controlled context compression through natural language guidance, bidirectional reasoning, token critics with language modeling heads, and window-parallel inference.Source: §5 · p. 12
• Across five long-context benchmarks and 10 mainstream LLMs, QWENLONG-CPRS improves performance and efficiency, with 97.3% relative compression versus RAG and 7.3% accuracy improvements.Linear latency scaling yields 3.47× acceleration over direct prompting at 128K-token inputs.Source: §5 · pp. 12–13
• Future work targets computational efficiency through KV-cache and kernel optimization, global context awareness, and adaptation to additional applications.Source: §5 · p. 13

• Figure 8 presents a multi-value Needle-in-a-Haystack test case.Source: Figure 8 · p. 17
• Figure 9 presents an English multi-hop QA case.Source: Figure 9 · p. 18
• Figure 10 presents a contract-element extraction case.Source: Figure 10 · p. 19