MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

• M1 combines a 456-billion-parameter, 32-expert MoE with Lightning Attention and one softmax-attention block per seven transnormer blocks.It uses 45.9 billion activations; versus DeepSeek R1, it consumes less than 50% of FLOPs at 64K generation tokens and approximately 25% at 100K.Source: §1 · p. 2
• M1 supports a 1-million-token context—eight times DeepSeek R1—and targets tasks requiring long inputs and extended thinking.The authors state that its attention design theoretically enables reasoning lengths of hundreds of thousands of tokens.Source: §1 · p. 2
• M1 training continues pretraining on 7.5T reasoning-intensive tokens, applies SFT to establish chain-of-thought patterns, then uses RL across diverse problems and environments.RL data includes mathematics, programming, 41 SynLogic logical-reasoning tasks, SWE-bench-derived software-engineering sandboxes with execution-based rewards, and unverifiable domains such as QA and creative writing using generative reward models.Source: §1 · pp. 2–3
• CISPO clips importance-sampling weights, uses all tokens for gradients, and achieves a 2x speedup over DAPO in a controlled Qwen2.5-32B study.The approach abandons the trust-region constraint; the hybrid architecture also required solutions to a training–inference precision mismatch that otherwise prevented reward growth. The full RL run took three weeks on 512 H800 GPUs, at approximately $0.53M rental cost.Source: §1 · p. 3

2.1. Continual Pre-Training: Foundation for RL Scaling

• Continual pretraining adds 7.5T tokens to MiniMax-Text-01 to strengthen reasoning and long-context capabilities while maintaining data diversity.The training uses optimized data quality and mixture.Source: §2.1 · p. 4
• Pretraining emphasizes natural, deduplicated QA data, improves math and code extraction, avoids synthetic data, and raises STEM, code, book, and reasoning data to 70%.The authors say this improves complex-task ability without compromising other general capabilities.Source: §2.1 · p. 4
• The recipe reduces MoE auxiliary-loss weight and adjusts parallelism for larger microbatches, then trains for 2.5T tokens at 8e-5 before decaying over 5T tokens to 8e-6.The auxiliary-loss change is intended to mitigate its detrimental effects on overall performance.Source: §2.1 · p. 4
• To avoid gradient explosions from overly aggressive context extension, training smoothly expands the context window in four stages from 32K to 1M tokens.The authors attribute the optimization challenge to earlier layers lagging behind later layers, whose lightning-attention decay rates differ.Source: §2.1 · p. 4

2.2. Supervised Fine-Tuning: Focused Alignment for Efficient RL

• SFT uses high-quality, long-CoT examples across math, coding, STEM, writing, QA, and multi-turn chat to teach reflection-based reasoning and support more efficient, stable RL.Math and coding account for around 60% of the SFT data.Source: §2.2 · p. 4

3. Efficient RL Scaling: Algorithms and Lightning Attention

• The complete M1 RL training cycle takes three weeks on 512 H800 GPUs, at an approximate rental cost of $0.53M USD.The section attributes this efficiency to hybrid-attention advantages and targeted solutions to challenges encountered during RL training.Source: §3 · p. 5

3.1. Efficient RL Scaling with CISPO

• Rare, low-probability reflective tokens often have high importance weights and are clipped from later off-policy updates; DAPO’s higher upper bound was less effective with 16 update rounds.These reasoning tokens can act as forks in reasoning paths.Source: §3.1 · pp. 5–6
• CISPO outperforms GRPO and DAPO at the same training steps and matches DAPO’s AIME 2024 performance with 50% of the steps.The comparison trains Qwen2.5-32B-base in a zero-RL setting on a mathematical reasoning dataset.Source: Figure 2 · p. 6, §3.1 · p. 7
• CISPO clips importance-sampling weights rather than token updates, preserving gradient contributions from all tokens, especially in long responses, while reducing variance and stabilizing RL.The method adopts group-relative advantages and token-level loss; its gradient is slightly biased by weight clipping.Source: §3.1 · pp. 6–7

3.2. Efficient RL Scaling with Lightning Attention – Challenges and Recipes

• Using FP32 for the LM output head raises training–inference probability correlation from approximately 0.9x to 0.99x, with stable correlation enabling reward growth.The mismatch arose from precision differences between training and inference kernels and prevented reward growth.Source: §3.2 · pp. 7–8, Figure 3 · p. 8
• For stability, MiniMax-M1 uses AdamW with β1 = 0.9, β2 = 0.95, eps=1e-15 and truncates generation after 3,000 consecutive tokens each exceed 0.99 probability.The truncation rule targets repetitive loops that threatened model stability and improved generation throughput.Source: §3.2 · p. 8

4.1. Reasoning-Intensive Tasks with Rule-based Verification

• Rule-based tasks use final correctness plus format reward; the curated data include nearly 50K mathematical samples and approximately 53K logical reasoning samples.Mathematical samples are filtered for quality and difficulty; logical data are synthesized with task-specific generators and verifiers.Source: §4.1 · p. 9
• Competitive programming contributes 30K filtered samples, while software engineering uses several thousand GitHub-based tasks in an executable sandbox rewarded by test outcomes.The software tasks cover bug localization, code repair, and test-case synthesis.Source: §4.1 · pp. 9–10

4.2. General Domain Tasks with Model-based Feedbacks

• General-domain RL adds 25K complex samples, including objective-answer tasks that are difficult to check with rules and open-ended tasks without ground-truth answers.Reward models provide feedback for tasks that cannot be easily verified by rules.Source: §4.2.1 · p. 10, §4.2 · p. 10

4.2.1. Data and Reward Models

• For tasks with ground-truth answers, GenRM uses a five-grade reward scale and is assessed on a human-annotated benchmark and by comparing Best-of-N responses with pass@N metrics.These evaluations guide optimization of GenRM data distribution and prompt design.Source: §4.2.1 · p. 10
• For open-ended tasks, the team pairs prompts with quality-reviewed references and scores responses by pairwise comparisons from -1 to 1; constraint-following also receives rule-based rewards.The preference process uses blind judgments and bias-reduction methods, then a Swiss Round scoring system selects reference answers.Source: §4.2.1 · pp. 10–11

4.2.2. Addressing Bias of Generative Reward Models for Long CoT

• Offline GenRM bias mitigations failed to prevent length bias during RL; the team monitors length-seeking, recalibrates models, and applies reward shaping, value clipping, and normalization.The monitoring targets output-length increases without gains in task success or reasoning depth.Source: §4.2.2 · p. 11

4.3. Curriculum of Incorporating Diverse Data

• Training starts with rule-rewarded reasoning tasks, then gradually mixes general tasks using dynamic weighting to retain verifiable skills while broadening generalization.The curriculum is intended to prevent catastrophic forgetting while supporting flexible responses to general queries.Source: §4.3 · p. 11

Extending RL to 80K

• The model extends RL generation from 40K to 80K tokens, producing MiniMax-M1-80k.The authors cite the hybrid architecture’s near-linear scaling for longer sequences as enabling this extension.Source: §5 · p. 11

Data

• For 80K RL data, the 40K model filters out easily solved samples, favors difficult math and coding problems, and downsampling synthetic reasoning data avoids destabilizing training.Synthetic reasoning outputs were often repetitive and homogeneous, and continued exposure harmed overall performance.Source: §5 · p. 12

Length Scaling Strategy

• RL output limits progress from 40K through 48K, 56K, 64K, and 72K to 80K tokens.The next window begins when generated-sequence perplexity converges and the 99th percentile of output lengths approaches the current context limit, supporting training stability.Source: §5 · p. 12

Addressing Training Instability During Scaling

• At later stages of each length window, pattern collapse degraded generated sequences and coincided with increased perplexity.The authors attribute this to negative samples lengthening faster than positive samples and accumulating disproportionate negative gradients in later sequence segments. They use early stopping for repetitive patterns, combined sample-level loss with token-level normalization, and lower gradient clipping and 𝜖𝐼𝑆 thresholds.Source: §5 · p. 12

6.1. Core Benchmarks

• MiniMax-M1-80k scores 86.0% on AIME 2024, second among open-weight models; it matches Qwen3-235B on LiveCodeBench and outperforms it on FullStackBench.On reasoning and knowledge benchmarks, it trails DeepSeek-R1-0528 but is competitive with other leading open-weight models.Source: Table 2 · p. 13, §6.1 · p. 14
• On SimpleQA, MiniMax-M1 models underperform DeepSeek-R1 but outperform all other open-weight models and Seed-Thinking-v1.5; on MultiChallenge, both perform comparably to DeepSeek-R1-0528 and Claude 4 Optus.Source: §6.1 · p. 14
• MiniMax-M1-40k and 80k score 55.6% and 56.0% on SWE-bench Verified; M1 leads open-weight models in long-context understanding, and 40k surpasses Gemini-2.5 Pro on TAU-bench.MiniMax-M1-80k consistently outperforms MiniMax-M1-40k across most benchmarks.Source: §6.1 · p. 14

6.2. Effect of RL Scaling

• During RL training, AIME 2024 accuracy rises from 68% to 80%, while average response lengths on AIME and LiveCodeBench exceed 20,000 tokens.The reported visualizations show accuracy gains correlated with increased response length.Source: §6.2 · p. 14, Figure 4 · p. 15

Conclusion

• MiniMax-M1 supports inputs up to 1M tokens and generations up to 80K tokens, with strong results on software engineering, agentic tool use, and long-context benchmarks.The authors report that it ranks among the world’s best open-weight models alongside DeepSeek-R1 and Qwen3-235B.Source: §7 · p. 15
• Three weeks on 512 H800 GPUs completed MiniMax-M1’s full RL training, combining lightning attention’s efficiency with the CISPO algorithm.The work presents CISPO as a novel RL algorithm to accelerate training.Source: §7 · p. 15

Future work

• The authors envision MiniMax-M1 supporting agents that reason across dozens to hundreds of turns while integrating long-context information for workflow automation (Xu et al., 2025) and scientific research (OpenAI, 2025; Si et al., 2024).They plan to continue evolving the model toward these real-world applications as test-time compute scales.Source: §7 · p. 15