Source-linked AI summary
MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
MiniCPM Team, Wenhao An, Yingfa Chen, Yewei Fang, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Li, Biyuan Lin, Chuan Liu, Hezi Liu, Siyuan Liu, Hongya Lyu, Yinxu Pan, Shixin Ren, Xingyu Shen, Zhou Su, Haojun Sun, Yangang Sun, Zhen Leng Thai, Xin Tian, Rui Wang, Xiaorong Wang, Yudong Wang, Bo Wu, Xiaoyue Xu, Dong Xu, Shuaikang Xue, Jiawei Yang, Bowen Zhang, Jinqian Zhang, Letian Zhang, Shengnan Zhang, Xinyu Zhang, Xinyuan Zhang, Zhu Zhang, Hengyu Zhao, Jiacheng Zhao, Zhi Zheng, Jie Zhou, Zihan Zhou, Shuo Wang, Chaojun Xiao, Xu Han, Zhiyuan Liu, Maosong Sun
TL;DR
Ultra-long-context LLMs face full-attention compute and memory bottlenecks, while sparse and linear alternatives trade performance against efficiency. MiniCPM-SALA combines InfLLM-V2 and Lightning Attention in a hybrid architecture with HyPE and continual Transformer-to-hybrid training. It retains general capabilities comparable to full-attention models while improving efficiency, reaching up to 3.5× speed at 256K tokens and supporting 1M-token contexts on a single GPU.
Problem
Full-attention Transformers face severe computational and KV-Cache memory bottlenecks for ultra-long contexts, while sparse and linear methods have distinct efficiency and performance limitations.
Method
MiniCPM-SALA combines InfLLM-V2 sparse attention and Lightning Attention linear attention, uses HyPE, and converts a pretrained Transformer through continual training.
Results
MiniCPM-SALA retains general capabilities comparable to full-attention models while improving efficiency, achieving up to 3.5× inference speed at 256K tokens and supporting 1M-token contexts on a single GPU.
Takeaways & Limitations
MiniCPM-SALA provides a scalable long-context architecture that reduces training costs by approximately 75% and enables context processing beyond the memory limits of standard 8B full-attention models.
Abstract
from arXiv · showhide
The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer architecture. While existing sparse and linear attention mechanisms attempt to mitigate these issues, they typically involve a trade-off between memory efficiency and model performance. This paper introduces MiniCPM-SALA, a 9B-parameter hybrid architecture that integrates the high-fidelity long-context modeling of sparse attention (InfLLM-V2) with the global efficiency of linear attention (Lightning Attention). By employing a layer selection algorithm to integrate these mechanisms in a 1:3 ratio and utilizing a hybrid positional encoding (HyPE), the model maintains efficiency and performance for long-context tasks. Furthermore, we introduce a cost-effective continual training framework that transforms pre-trained Transformer-based models into hybrid models, which reduces training costs by approximately 75% compared to training from scratch. Extensive experiments show that MiniCPM-SALA maintains general capabilities comparable to full-attention models while offering improved efficiency. On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens and supports context lengths of up to 1M tokens, a scale where traditional full-attention 8B models fail because of memory constraints.
1 Introduction
MiniCPM-SALA addresses the computational and memory bottlenecks of full attention in ultra-long contexts with a sparse-linear hybrid design. It combines sparse attention’s long-context fidelity with linear attention’s efficiency, while using continual training and HyPE to preserve capability and reduce development cost.
- Motivation: Full attention scales as O(N^2), while KV-Cache requirements for millions of tokens can reach dozens or hundreds of gigabytes.
- Approach: MiniCPM-SALA combines sparse and linear attention to mitigate inference overhead and memory consumption while addressing precision limitations in pure linear architectures.
- Approach: 25% InfLLM-V2 and 75% Lightning Attention balance sparse attention’s local detail focus with linear attention’s O(N) broad-context efficiency.
- Training: Approximately 25% of the training budget is required relative to training a comparable model from scratch through Transformer-to-hybrid conversion of pretrained weights.
- Results: HyPE harmonizes performance across short and long contexts, while general knowledge, mathematics, and coding capabilities remain comparable to modern full-attention models.
- Results: Up to 3.5× inference speed is achieved at 256K tokens on an NVIDIA A6000D, while 1M-token inference succeeds where Qwen3-8B fails from out-of-memory errors.
2 Model Development
MiniCPM-SALA is developed by interleaving sparse and linear attention layers and converting a pretrained Transformer through continual training. Its design combines complementary attention mechanisms with positional and stability improvements across a staged training pipeline.
- Model Architecture: MiniCPM-SALA interleaves sparse and linear attention layers while retaining the FFN block after each attention block.
- Model Architecture: A 1:3 mixing ratio assigns 25% of layers to sparse attention and 75% to linear attention, combining constant sequence-length complexity with long-range dependency modeling.
- Training Strategy: Continual training converts a pretrained Transformer into a hybrid model by recycling weights and representations instead of training from scratch.
- Attention Modules: InfLLM-V2 adds no parameters and can switch between dense and sparse modes, supporting stable initialization from dense weights during conversion.
- Architectural Improvements: HyPE applies RoPE to linear attention layers and removes it from sparse attention layers to balance positional awareness with long-range information retention.
- Training Pipeline: The pipeline uses HALO conversion followed by continual stable-training, short-decay training, long-decay context extension, and supervised fine-tuning.
- Training Pipeline: Approximately 2T tokens are used for Transformer-to-hybrid training, about 25% of the 8T tokens required to train MiniCPM-4.0 from scratch.
3 Experiments
Experiments evaluate MiniCPM-SALA’s standard, long-context, ultra-long-context, and inference-efficiency performance against comparable open-source models. Results show competitive general capabilities, strong long-context performance, substantial latency advantages, and successful million-token inference where Qwen3-8B encounters memory failures.
- Standard Evaluation: MiniCPM-SALA achieves an average standard-evaluation score of 76.53, remaining competitive among similarly sized open-source models.
- Standard Evaluation: MiniCPM-SALA scores 95.12 on HumanEval, 89.11 on MBPP, 83.75 on AIME24, and 78.33 on AIME25.
- Long-Context Evaluation: MiniCPM-SALA scores 89.37 on RULER at 128K context and 23.86 on NoLiMa at 128K, with a 38.97 overall long-context average.
- Ultra-Long Context: MiniCPM-SALA maintains a score of 81.6 at 2048K tokens despite training at 520K, and scores 86.3 versus Qwen3-Next-80B-A3B-Instruct’s 80.3 at 1000K.
- Inference Speed: At 256K tokens on an NVIDIA A6000D, MiniCPM-SALA reduces TTFT from Qwen3-8B’s 180.8s to 51.6s.
- Inference Speed: MiniCPM-SALA processes 1024K tokens on A6000D and 5090 GPUs, while Qwen3-8B encounters OOM failures at tested long-context lengths.
4 Conclusion
The paper presents MiniCPM-SALA as a sparse-linear hybrid architecture for ultra-long-context modeling. Its Transformer-to-hybrid training paradigm retains general capabilities while reducing training costs and enabling faster, million-token inference on single GPUs.
- MiniCPM-SALA combines sparse and linear attention to address computational and memory bottlenecks in ultra-long-context modeling.
- The Transformer-to-hybrid training paradigm reduces training costs by approximately 75% while retaining the general capabilities of full-attention models.
- MiniCPM-SALA achieves substantial inference speedups and enables 1M-token context processing on single GPUs such as the NVIDIA A6000D.
5 Contributions and Acknowledgments
This section records authorship, project coordination, and affiliations for MiniCPM-SALA. It also directs readers to prior work for architecture details.
- Acknowledgments: Architecture details are referenced to Chen et al. (2026) and Zhao et al. (2025).
- Acknowledgments: MiniCPM-SALA reflects the collective efforts of all team members.
- Contributors and Affiliations: Contributors are listed in ordered-by-last-name form, with some marked as affiliated with XCORE SIGMA.
- Contributors and Affiliations: The remaining contributors are affiliated with OpenBMB, while project design and coordination are attributed to a separate listed group.