Source-linked AI summary
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Zhu, Jian Sun, Jiaqi Zhuang, Jiaren Cai, Jiayuan Song, Jin Zhu, Jingyang Li, Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng, Ke Yang, Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Li, Lin Zheng, Linge Du, Lingyu Yang, Lunbin Zeng, Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin, Nan Hu, Nongyu Di, Peng Gao, Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng, Shaoxiang Chen, Shaoyu Chen, Shengmin Shi, Shitong Weng, Shuchang Guan, Shuqi Yu, Sichen Li, Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma, Yang Wang, Yiqi Shi, Yiran Zhong, Yonghong Duan, Yongxiang Fu, Yongyi Hu, Yu Gao, Yuanxiang Fan, Yufeng Yang, Yuhao Li, Yulin Hu, Yunan Huang, Yunji Li, Yunzhi Xu, Yuxin Mao, Yuxuan Shi, Yuze Wenren, Zehan Li, Zelin Li, Zhanxu Tian, Zhengmao Zhu, Zhenhua Fan, Zhenzhen Wu, Zhichao Xu, Zhihang Yu, Zhiheng Lyu, Zhuo Jiang, Zibo Gao, Zijia Wu, Zijian Song, Zijun Sun
TL;DR
MiniMax-M1 targets the computational difficulty of extending reasoning and processing long contexts in traditional transformers. It combines hybrid MoE and Lightning Attention with CISPO and large-scale RL, achieving strong open-weight performance, especially on software engineering, tool use, and long-context tasks.
Problem
Traditional softmax attention has quadratic computational complexity, limiting efficient extension of reasoning processes.
Method
MiniMax-M1 combines hybrid MoE and Lightning Attention with CISPO and large-scale RL on diverse reasoning and real-world software-engineering environments.
Results
MiniMax-M1 is comparable or superior to strong open-weight models overall, with particular advantages in complex software engineering, tool use, and long-context tasks.
Takeaways & Limitations
M1 supports up to 1M-token inputs and 80K-token generation, providing a foundation for complex scenarios requiring long context and extended reasoning.
Abstract
from arXiv · showhide
We introduce MiniMax-M1, the world's first open-weight, large-scale hybrid-attention reasoning model. MiniMax-M1 is powered by a hybrid Mixture-of-Experts (MoE) architecture combined with a lightning attention mechanism. The model is developed based on our previous MiniMax-Text-01 model, which contains a total of 456 billion parameters with 45.9 billion parameters activated per token. The M1 model natively supports a context length of 1 million tokens, 8x the context size of DeepSeek R1. Furthermore, the lightning attention mechanism in MiniMax-M1 enables efficient scaling of test-time compute. These properties make M1 particularly suitable for complex tasks that require processing long inputs and thinking extensively. MiniMax-M1 is trained using large-scale reinforcement learning (RL) on diverse problems including sandbox-based, real-world software engineering environments. In addition to M1's inherent efficiency advantage for RL training, we propose CISPO, a novel RL algorithm to further enhance RL efficiency. CISPO clips importance sampling weights rather than token updates, outperforming other competitive RL variants. Combining hybrid-attention and CISPO enables MiniMax-M1's full RL training on 512 H800 GPUs to complete in only three weeks, with a rental cost of just $534,700. We release two versions of MiniMax-M1 models with 40K and 80K thinking budgets respectively, where the 40K model represents an intermediate phase of the 80K training. Experiments on standard benchmarks show that our models are comparable or superior to strong open-weight models such as the original DeepSeek-R1 and Qwen3-235B, with particular strengths in complex software engineering, tool utilization, and long-context tasks. We publicly release MiniMax-M1 at https://github.com/MiniMax-AI/MiniMax-M1.
1. Introduction
MiniMax-M1 addresses the quadratic cost of extending reasoning in traditional transformers with hybrid MoE and Lightning Attention. It combines efficient long-context reasoning, CISPO-based RL scaling, and diverse task environments, achieving strong results on complex software engineering, tool-use, and long-context tasks.
- Traditional softmax attention has quadratic computational complexity, making continuously extending reasoning challenging.
- MiniMax-M1 combines a hybrid MoE architecture with Lightning Attention, alternating one softmax-attention transformer block after every seven lightning-attention transnormer blocks.The model has 456 billion total parameters, 45.9 billion activated parameters, and 32 experts.
- At 64K tokens, M1 uses less than 50% of DeepSeek R1’s FLOPs, while at 100K tokens it uses approximately 25%.The architecture is designed to scale reasoning lengths to hundreds of thousands of tokens.
- MiniMax-M1 supports up to 1 million input tokens and is designed for complex tasks requiring long inputs and extended thinking.The paper positions these capabilities as exceeding other open-weight reasoning models.
- CISPO clips importance-sampling weights rather than token updates, and achieves a 2x speedup compared with DAPO in a controlled Qwen2.5-32B study.The full RL run completed in 3 weeks on 512 H800 GPUs at an approximate rental cost of $0.53M USD.
- MiniMax-M1 uses diverse RL problems, including logical reasoning, mathematical reasoning, competitive programming, and sandboxed software engineering environments.
2. Preparation for Scalable RL: Continual Pretraining and SFT
The preparation pipeline strengthens MiniMax-Text-01 through continued pretraining and supervised fine-tuning before reinforcement learning. It uses additional data, adjusted optimization, and long-CoT examples to improve reasoning and provide a stable RL starting point.
- The preparation stage continually trains MiniMax-Text-01, then applies cold-start SFT to inject reasoning patterns before RL.
- 7.5T additional tokens are used with optimized data quality and mixture to enhance reasoning and long-context capabilities.
- Training data emphasizes high-recall extraction of mathematical and code-related content, natural QA pairs, and semantic deduplication without synthetic data.
- The training recipe reduces the MoE auxiliary-loss coefficient and uses 2.5T tokens at learning rate 8e-5 followed by 5T tokens decaying to 8e-6.
- Aggressive length extension can trigger gradient explosions because earlier and later lightning-attention layers have different decay rates.
- SFT uses high-quality long-CoT examples across math, coding, STEM, writing, QA, and multi-turn chat, with math and coding comprising around 60% of the data.
3. Efficient RL Scaling: Algorithms and Lightning Attention
The section presents CISPO and hybrid-attention techniques for making reinforcement-learning scaling more efficient and stable. It addresses token clipping, precision mismatch, and other training obstacles encountered with MiniMax-M1.
- Efficient RL scaling: 3 weeks on 512 H800 GPUs completes MiniMax-M1’s full RL training cycle, costing approximately $0.53M USD.The efficient framework combines hybrid-attention scaling with algorithmic and engineering solutions.
- CISPO: Low-probability reflective tokens can stabilize entropy and support scalable RL, but PPO/GRPO clipping may drop these tokens and degrade long-chain-of-thought learning.Increasing DAPO’s upper clipping bound was less effective in the reported setup with 16 off-policy updates per generation batch.
- CISPO: 50% of training steps lets CISPO achieve performance comparable to DAPO on AIME 2024, while outperforming GRPO and DAPO at equal training steps.The comparison uses Qwen2.5-32B-base in a zero-RL training setting.
- CISPO: CISPO clips importance-sampling weights rather than token updates, preserving gradient contributions from all tokens during long responses.It uses GRPO’s group-relative advantage and token-level loss; without weight clipping, it reduces to standard policy gradient.
- Lightning Attention challenges: Hybrid attention offers more efficient inference and thereby supports RL scaling with increasingly longer generated responses, but introduces architecture-specific stability challenges.The section also addresses optimizer sensitivity and pathological repetition during RL training.
- Lightning Attention challenges: FP32 precision in the LM output head raises training–inference probability correlation from approximately 0.9x to 0.99x, enabling reward increase.The mismatch between training and inference kernels otherwise prevented reward growth during RL.
4. Scaling Reinforcement Learning with Diverse Data
MiniMax-M1’s RL pipeline combines rule-verifiable and reward-model-verified tasks through a curriculum, while addressing reward-model bias and heterogeneous task demands. The data includes mathematical, logical, programming, software-engineering, factual, and open-ended problems.
- Data and reward design: The RL pipeline integrates rule-verifiable environments and general tasks requiring reward-model feedback within a designed curriculum.Rule-based correctness rewards are used for deterministic tasks, while reward models verify broader tasks.
- Verifiable reasoning and coding: The training data spans competition mathematics, 41 synthesized logical-reasoning tasks, competitive programming, and real-world software-engineering environments.Software-engineering environments execute code and use test outcomes as reward signals.
- General-domain tasks: The general RL dataset contains 25K complex samples, including verifiable answers difficult to check with rules and open-ended tasks without ground-truth answers.These categories require different verification strategies, including generative reward models and reference answers.
- Reward-model bias: Generative reward models were found to prefer longer outputs regardless of reasoning quality, risking verbosity-driven reward hacking during RL.Offline diversification, adversarial examples, and architectural refinement often failed to prevent this bias during training.
- Online mitigation and curriculum: The pipeline continuously monitors output-length bias and recalibrates reward models when length-seeking behavior appears.Training also uses a curriculum and dynamic weighting that gradually mixes general-domain tasks with reasoning-intensive tasks.
5. Extending RL Scaling to Longer Thinking
MiniMax-M1 extends RL training from 40K to 80K output tokens using staged window expansion and data filtering. The scaling process required stability measures because longer outputs could trigger pattern collapse and degraded text quality.
- Longer-thinking training: The 40K-token RL model was extended to 80K tokens, producing MiniMax-M1-80k because the hybrid architecture supports near-linear longer-sequence scaling.The initial 40K model serves as the basis for the longer-thinking version.
- Data preparation: The 80K training data is filtered with the 40K model to remove easy samples and emphasize difficult mathematical and coding problems.Synthetic reasoning data is also downsampled after destabilizing long-context RL training.
- Staged window expansion: Training expands the output window through 40K, 48K, 56K, 64K, 72K, and 80K stages to maintain stability.Transitions depend on empirical indicators including generated-sequence perplexity and output-length statistics.
- Training stability: Pattern collapse emerged late in each length window, with later generated text becoming incoherent or garbled as perplexity increased.The issue was identified as a critical training-instability problem during output-length scaling.
6. Evaluations
MiniMax-M1 is evaluated across mathematics, coding, software engineering, reasoning, long context, tool use, factuality, and general capabilities. Results show competitive broad performance, with particular strengths in software engineering, long-context understanding, tool use, and scaled reasoning.
- Core benchmarks: 86.0% on AIME 2024 places MiniMax-M1-80k second among open-weight models, behind DeepSeek-R1-0528.The model is comparable to Seed-Thinking-v1.5 across multiple mathematical benchmarks.
- Software engineering: 55.6% and 56.0% on SWE-bench Verified are achieved by MiniMax-M1-40k and MiniMax-M1-80k, respectively.These scores are slightly below DeepSeek-R1-0528’s 57.6% and above other open-weight models.
- Long context: MiniMax-M1 models significantly outperform other open-weight models on long-context understanding using their 1M-token context window.They rank second globally on the reported comparisons, behind Gemini-2.5 Pro.
- Agentic tool use: MiniMax-M1-40k surpasses all open-weight models and Gemini-2.5 Pro in TAU-bench agentic tool use.The evaluation uses dynamic conversations requiring API-tool use under domain-specific policies.
- Effect of RL scaling: AIME 2024 accuracy rises from 68% to 80% during RL training while average AIME and LiveCodeBench responses exceed 20,000 tokens.The evaluation tracks accuracy and response length across RL steps for AIME 2024, AIME 2025, and LiveCodeBench v5.
7. Conclusion and Future work
MiniMax-M1 combines lightning attention with large-scale RL and CISPO to support long inputs and extended reasoning. Its reported strengths are validated on software-engineering, agentic tool-use, and long-context benchmarks, while the authors identify future applications in real-world workflows and scientific research.
- Conclusion: MiniMax-M1 supports inputs up to 1M tokens and generation lengths of 80K tokens through its lightning-attention design.The model is released as an open-weight large-scale reasoning model.
- Conclusion: Combining lightning attention with CISPO completes full RL training in three weeks on 512 H800 GPUs.The conclusion attributes the training efficiency to architectural advantages and the RL algorithm.
- Conclusion: The model’s long-context and extended-reasoning capabilities are empirically validated on software engineering, agentic tool use, and long-context benchmarks.Across comprehensive evaluations, MiniMax-M1 ranks among the strongest open-weight models alongside DeepSeek-R1 and Qwen3-235B.
- Future work: The authors foresee efficient architectures supporting company-workflow automation and scientific research as test-time compute scales.They emphasize real-world agents interacting with environments, tools, computers, or other agents.
A. Contributors
This section lists the report’s contributors in alphabetical order.
- The report states that its contributors are listed in alphabetical order.
- The contributor list begins with Aili Chen, Aonian Li, and Bangwei Gong.
- The listed contributors continue through names beginning with B, C, and D.