Source-linked AI summary
Qwen2.5-1M Technical Report
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, Zipeng Zhang
TL;DR
Qwen2.5-1M addresses the limited context length and deployment cost of long-context language models. It combines long-context pre-training and post-training with length extrapolation, sparse attention, and inference-engine optimizations. The resulting models improve long-context performance while supporting efficient 1M-token processing, although performance drops can occur in some DCA-integrated sparse-attention cases.
Problem
Limited context length restricts the amount of text LLMs can process at once, limiting complex tasks that require extensive information processing.
Method
The report develops Qwen2.5-1M through long-context training and an inference framework combining extrapolation, sparse attention, sparsity refinement, and engine optimizations.
Results
3 to 7 times faster prefill is achieved in 1M-context scenarios, while DCA enables over 80% Passkey Retrieval accuracy up to 1 million tokens.
Takeaways & Limitations
The open-source models and inference optimizations make long-context processing more practical for deployment.
Takeaways & Limitations
Integrating sparse attention with DCA causes performance drops in some length-extrapolation cases.
Abstract
from arXiv · showhide
We introduce Qwen2.5-1M, a series of models that extend the context length to 1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series have significantly enhanced long-context capabilities through long-context pre-training and post-training. Key techniques such as long data synthesis, progressive pre-training, and multi-stage supervised fine-tuning are employed to effectively enhance long-context performance while reducing training costs. To promote the use of long-context models among a broader user base, we present and open-source our inference framework. This framework includes a length extrapolation method that can expand the model context lengths by at least four times, or even more, without additional training. To reduce inference costs, we implement a sparse attention method along with chunked prefill optimization for deployment scenarios and a sparsity refinement method to improve precision. Additionally, we detail our optimizations in the inference engine, including kernel optimization, pipeline parallelism, and scheduling optimization, which significantly enhance overall inference performance. By leveraging our inference framework, the Qwen2.5-1M models achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million tokens of context. This framework provides an efficient and powerful solution for developing applications that require long-context processing using open-source models. The Qwen2.5-1M series currently includes the open-source models Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly improved in long-context tasks without compromising performance in short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model significantly outperforms GPT-4o-mini in long-context tasks and supports contexts eight times longer.
1 Introduction
Qwen2.5-1M extends Qwen’s context length to 1 million tokens while targeting stronger long-context capabilities and practical deployment. The report combines long-context training strategies with an optimized inference framework for efficient processing.
- 1 million tokens is the target context length for the Qwen2.5-1M series, extending prior Qwen context windows that reached 128k tokens.
- The series includes open-source 7B and 14B instruction-tuned models plus the API-accessible Qwen2.5-Turbo model.
- Efficient Long-Context Training: Long-context training uses synthetic long-range data, progressive length expansion, agent-generated instructions, multi-stage SFT, and RL.
- Efficient Inference and Deployment: The inference framework combines training-free length extrapolation, sparse attention, sparsity refinement, and engine-level optimizations.
- Efficient Inference and Deployment: 3 to 7 times faster prefill is reported for 1M-context scenarios using the inference framework.
2 Architecture
Qwen2.5-1M retains the Qwen2.5 Transformer architecture and provides dense open-source models alongside an MoE model for API access.
- The series supports context lengths up to 1M tokens and includes Qwen2.5-7B-1M, Qwen2.5-14B-1M, and Qwen2.5-Turbo.
- The models retain Qwen2.5’s Transformer-based architecture for inference compatibility.
- The architecture uses GQA, SwiGLU, RoPE, QKV bias, and RMSNorm with pre-normalization.
3 Pre-training
Qwen2.5-1M improves long-context pre-training efficiency by combining diverse natural data with synthetic long-range tasks and progressive context expansion. RULER evaluations show that longer-sequence training improves performance at corresponding and shorter context lengths.
- Long-context pre-training is computationally expensive, motivating improvements in data efficiency and training strategy.
- Natural and Synthetic Data: Natural data from Common Crawl, arXiv, books, and code repositories is combined with synthetic data targeting long-range dependencies.
- Natural and Synthetic Data: Synthetic tasks include Fill in the Middle, keyword- and position-based retrieval, and paragraph reordering.
- Natural and Synthetic Data: Synthetic long-range tasks improve long-range information capture, data efficiency, and computational cost.
- Training Strategy: Five progressive stages expand context lengths from 4096 to 262,144 tokens while mixing current-maximum-length and shorter sequences.
- Training Strategy: Progressively longer training sequences consistently improve RULER comprehension at corresponding lengths and significantly improve performance on 128K samples.
4 Post-Training
Post-training is designed to improve long-context performance without reducing short-task capability. It uses synthetic long-context instructions, staged supervised fine-tuning, and offline reinforcement learning, while DCA supports position remapping for long contexts.
- Post-training targets stronger long-context performance while preserving short-task performance.
- Synthesizing Long Instruction Data: Synthetic long-context question-answer pairs are generated from long documents to address costly and unreliable human annotation.
- Two-stage Supervised Fine-tuning: Two-stage SFT first trains on short instructions and then introduces long-context data to balance short- and long-task performance.
- Reinforcement Learning: Offline RL uses short samples up to 8,192 tokens and is reported to generalize effectively to long-context tasks.
- Length Extrapolation: DCA remaps relative positions to smaller values, avoiding large positions absent from training.
5 Inference and Deployment
Qwen2.5-1M addresses long-context inference challenges through length extrapolation, sparse attention, chunked prefill, and system-level optimizations. These methods extend supported context lengths, reduce memory and computation costs, and improve retrieval and inference speed.
- 5.1 Length Extrapolation: Length extrapolation supports inference at contexts four times or more longer than training, using DCA to remap relative positions into smaller distances.DCA divides sequences into chunks and uses intra-, inter-, and successive-chunk attention patterns to preserve manageable positional distances.
- 5.1 Length Extrapolation: Over 80% accuracy is achieved on Passkey Retrieval at up to 1 million tokens for models trained only on sequences up to 32K tokens.The result applies to both Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct with DCA.
- 5.1 Length Extrapolation: Training on sequences up to 256K tokens substantially improves extrapolation to longer contexts compared with the 128K versions.The improvement is reported across the evaluated long-context setting described for Figure 3.
- 5.2 Efficient Inference with Sparse Attention: At 1 million tokens, conventional attention can consume over 90% of forward-pass time, motivating sparse attention for practical deployment.The framework combines MInference-based sparse attention with chunked prefill, length extrapolation, and sparsity refinement.
- 5.2 Efficient Inference with Sparse Attention: Refined critical-token selection recovers most retrieval performance while maintaining about 4 times prefilling speedup.The refinement addresses accuracy degradation observed with original MInference beyond 400K tokens.
6 Evaluation
Qwen2.5-1M models substantially improve long-context performance over 128k counterparts while preserving similar short-context performance. Their sparse-attention and optimized inference framework also accelerates 1M-token processing.
- Long Context Benchmarks: Qwen2.5-1M models significantly outperform their 128k counterparts on most long-context tasks, especially sequences exceeding 64k tokens.On RULER, all Qwen2.5-1M models surpass GPT-4, while Qwen2.5-14B-Instruct-1M reaches 92.2 accuracy on 128k sequences.
- Long Context Benchmarks: 92.2 accuracy is achieved by Qwen2.5-14B-Instruct-1M on 128k sequences, the first Qwen2.5 model to exceed 90 points.
- Long Context Benchmarks: Qwen2.5-Turbo performs between the 7B and 14B instruction-tuned models on long-context benchmarks while providing faster inference and lower costs.
- Short Context Benchmarks: Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M maintain short-context performance similar to their 128k versions.Qwen2.5-14B-Instruct-1M and Qwen2.5-Turbo also achieve similar short-task performance to GPT-4o-mini while supporting eight times longer contexts.
- Speed Comparison: 3.2 to 6.7 times speedup is achieved for 1M-token contexts across model sizes and devices using sparse attention and optimized inference engines.On H20, Qwen2.5-14B-Instruct-1M drops from 12.2 minutes with full attention to 109 seconds, while Qwen2.5-Turbo drops from 4.9 minutes to 68 seconds.
7 Conclusion
The report introduces Qwen2.5-1M models and training techniques that improve long-context effectiveness at lower costs. It also presents open-sourced inference optimizations, while acknowledging that long-context models still have substantial room for improvement.
- Qwen2.5-1M includes Qwen2.5-7B-Instruct-1M, Qwen2.5-14B-Instruct-1M, and the API-accessible Qwen2.5-Turbo.
- Long-context pre-training, post-training, data synthesis, and progressive training improve training effectiveness while reducing costs.
- Training-free length extrapolation, sparse attention, and inference-engine enhancements improve efficiency and reduce operational costs for long-sequence models.
- Several inference optimizations are open-sourced to support progress in long-context model deployment.
- Long-context models still have significant potential for improvement, particularly for strong performance across short- and long-context tasks.