PaperlayerView exact revision on arXiv

Source-linked AI summary

MiMo-V2-Flash Technical Report

Xiaomi LLM-Core Team, :, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Lei Li, Liang Zhao, Linghao Zhang, Peidian Li, Qianli Chen, Shaohui Liu, Shihua Yu, Shijie Cao, Shimao Chen, Shouqiu Yu, Shuo Liu, Tianling Zhou, Weijiang Su, Weikun Wang, Wenhan Ma, Xiangwei Deng, Bohan Mao, Bowen Ye, Can Cai, Chenghua Wang, Chengxuan Zhu, Chong Ma, Chun Chen, Chunan Li, Dawei Zhu, Deshan Xiao, Dong Zhang, Duo Zhang, Fangyue Liu, Feiyu Yang, Fengyuan Shi, Guoan Wang, Hao Tian, Hao Wu, Heng Qu, Hongfei Yi, Hongxu An, Hongyi Guan, Xing Zhang, Yifan Song, Yihan Yan, Yihao Zhao, Yingchun Lai, Yizhao Gao, Yu Cheng, Yuanyuan Tian, Yudong Wang, Zhen Tang, Zhengju Tang, Zhengtao Wen, Zhichao Song, Zhixian Zheng, Zihan Jiang, Jian Wen, Jiarui Sun, Jiawei Li, Jinlong Xue, Jun Xia, Kai Fang, Menghang Zhu, Nuo Chen, Qian Tu, Qihao Zhang, Qiying Wang, Rang Li, Rui Ma, Shaolei Zhang, Shengfan Wang, Shicheng Li, Shuhao Gu, Shuhuai Ren, Sirui Deng, Tao Guo, Tianyang Lu, Weiji Zhuang, Weikang Zhang, Weimin Xiong, Wenshan Huang, Wenyu Yang, Xin Zhang, Xing Yong, Xu Wang, Xueyang Xie, Yilin Jiang, Yixin Yang, Yongzhe He, Yu Tu, Yuanliang Dong, Yuchen Liu, Yue Ma, Yue Yu, Yuxing Xiang, Zhaojun Huang, Zhenru Lin, Zhipeng Xu, Zhiyang Chen, Zhonghua Deng, Zihan Zhang, Zihao Yue

arXiv:2601.02780v2cs.CLcs.AI

TL;DR

MiMo-V2-Flash addresses the need for long-context modeling that is both fast and strong for reasoning and agentic workflows. It combines hybrid Sliding Window Attention, Multi-Token Prediction, and Multi-Teacher On-Policy Distillation, achieving strong benchmark performance and long-context retrieval while remaining fast at inference. The paper reports that it rivals larger open-weight models, though a gap remains to the strongest closed-weight models.

  • Problem

    Long-context modeling for reasoning and autonomous agentic workflows must become simultaneously faster and stronger.

  • Method

    MiMo-V2-Flash combines hybrid Sliding Window Attention, lightweight Multi-Token Prediction, and Multi-Teacher On-Policy Distillation for efficient reasoning, agentic performance, and inference.

  • Results

    73.4% on SWE-Bench Verified and nearly 100% long-context retrieval success from 32K to 256K; it also achieves performance comparable to Kimi-K2-Thinking and DeepSeek-V3.2-Thinking on most reasoning benchmarks.

  • Takeaways & Limitations

    MiMo-V2-Flash rivals larger open-weight models while delivering fast inference, and its model and three-layer MTP weights are available for open research.

  • Takeaways & Limitations

    A clear gap remains to the strongest closed-weight models, while architectural exploration and analysis of design trade-offs remain limited.

Abstract

from arXiv · show

We present MiMo-V2-Flash, a Mixture-of-Experts (MoE) model with 309B total parameters and 15B active parameters, designed for fast, strong reasoning and agentic capabilities. MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention, with a 128-token sliding window under a 5:1 hybrid ratio. The model is pre-trained on 27 trillion tokens with Multi-Token Prediction (MTP), employing a native 32k context length and subsequently extended to 256k. To efficiently scale post-training compute, MiMo-V2-Flash introduces a novel Multi-Teacher On-Policy Distillation (MOPD) paradigm. In this framework, domain-specialized teachers (e.g., trained via large-scale reinforcement learning) provide dense and token-level reward, enabling the student model to perfectly master teacher expertise. MiMo-V2-Flash rivals top-tier open-weight models such as DeepSeek-V3.2 and Kimi-K2, despite using only 1/2 and 1/3 of their total parameters, respectively. During inference, by repurposing MTP as a draft model for speculative decoding, MiMo-V2-Flash achieves up to 3.6 acceptance length and 2.6x decoding speedup with three MTP layers. We open-source both the model weights and the three-layer MTP weights to foster open research and community collaboration.

1 Introduction

MiMo-V2-Flash targets fast, strong reasoning and agentic performance through an efficient MoE design, hybrid attention, MTP, and MOPD post-training. It reports strong long-context and software-engineering results while competing with larger open-weight models.

  • MiMo-V2-Flash is a 309B-parameter MoE with 15B parameters activated per token, designed for efficient reasoning and agentic performance.
  • A 128-token sliding window and 5:1 local-to-global attention ratio reduce long-context KV-cache storage and attention computation by nearly 6×.The architecture interleaves local sliding-window and global attention.
  • The model is trained on 27T tokens with FP8 mixed precision, starting at 32K context and extending to 256K.
  • MiMo-V2-Flash achieves nearly 100% long-context retrieval success from 32K to 256K and shows minimal degradation on GSM-Infinite from 16K to 128K.
  • MOPD scales post-training by combining specialized teacher models' dense token-level rewards with verifiable outcome-based rewards.The process includes general SFT, specialized RL/SFT for domain teachers, and student distillation.
  • The model reaches 73.4% on SWE-Bench Verified and 71.7% on SWE-Bench Multilingual while surpassing larger full-attention models on V2 and MRCR.

2 MiMo-V2-Flash Model Architecture

MiMo-V2-Flash combines MoE with hybrid local/global attention and lightweight MTP to target efficient long-context modeling, reasoning, and decoding. Experiments report that 128-token hybrid SWA can retain or improve performance across general, long-context, and complex-reasoning evaluations.

  • Overall Architecture: MiMo-V2-Flash stacks hybrid blocks that interleave five SWA blocks with one GA block, using a 128-token sliding window.The architecture includes 39 SWA layers and 9 GA layers overall.
  • Overall Architecture: The model is a sparse MoE with 256 experts per layer and 8 experts activated per token.Both SWA and GA blocks use the sparse MoE FFN, without shared experts.
  • MTP Integration: MTP uses a lightweight dense-FFN SWA block with 0.33B parameters per block, supporting speculative decoding without a large additional inference cost.During post-training, multiple MTP heads are replicated and jointly trained for multi-step prediction.
  • Architecture Experiments: 128-token hybrid SWA with attention-sink bias consistently recovers or improves general-benchmark performance relative to all-GA attention.Without sink bias, the 128-token hybrid variant shows noticeable degradation.
  • Architecture Experiments: 128-token hybrid SWA surpasses all-GA across challenging reasoning benchmarks and outperforms the 512-token hybrid variant.The authors hypothesize that smaller windows improve regularization and separate local from long-range information processing.

3 Pre-Training

MiMo-V2-Flash is pre-trained on diverse, long-range data through staged context scaling from 32K to 256K tokens. Its hybrid attention and training design target efficient long-context reasoning while maintaining competitive benchmark performance.

  • Data: 27T tokens form the pre-training corpus, combining public web content, books, academic papers, code, mathematics, and broader STEM materials.The data pipeline emphasizes long-form documents and repository-level code, pull requests, issues, and commit histories.
  • Training stages: Three stages progress from 32K-context general pre-training, through code-heavy mid-training with approximately 5% synthetic reasoning data, to 256K context extension.The stages span 0–22T, 22–26T, and 26–27T tokens, respectively.
  • Evaluation: Nearly 100% long-context retrieval success is reported from 32K to 256K, while GSM-Infinite shows minimal degradation from 16K to 128K.The base model also consistently outperforms peers on reasoning-focused tasks and surpasses Kimi-K2-Base on SWE-Bench using less than one-third the parameters.

4 Post-Training

Post-training combines supervised instruction tuning, domain-specialized reinforcement learning, and Multi-Teacher On-Policy Distillation. The framework integrates complementary teacher capabilities while addressing capability imbalance and learning inefficiency.

  • Motivation: MOPD addresses capability imbalance and learning inefficiency in post-training pipelines that combine specialized models.Capability imbalance includes the “see-saw” effect, where improving one skill can regress others.
  • Framework: The three-stage pipeline consists of general SFT, domain-specialized teacher training, and on-policy distillation into the student model.Teachers are trained for agentic tasks such as search, coding, and tool use, and non-agentic tasks such as mathematics, reasoning, and safety alignment.
  • Framework: MOPD uses the student’s evolving on-policy distribution and token-level teacher supervision instead of parameter merging or static expert-generated datasets.Training-inference importance sampling discards tokens with large discrepancies, and MOPD advantages can be combined with ORM advantages.
  • Advantages: Dense teacher-logit rewards provide stable credit assignment and rapid convergence while reducing exposure bias and distribution mismatch associated with off-policy training.The framework is described as preserving the strongest teacher’s peak performance across domains.
  • Advantages: MOPD supports flexible teacher integration and iterative teacher-student co-evolution, including specialized RL teachers, SFT teachers, and the student itself.Distilled students can re-enter specialized RL to produce stronger teachers for subsequent generations.
  • SFT: Stable num-zeros is treated as a critical MoE SFT metric because increasing values signal deteriorating expert balance, while decreasing values suggest overfitting.Its stability depends critically on the expert bias update rate and AdamW 𝜖 parameter.
  • Agentic RL: Scaling agentic RL across approximately 120K environments significantly improves SWE-Bench-Verified and SWE-Bench-Multilingual performance over the SFT base model.The training uses on-policy rollouts and updates across diverse agentic environments.

4.4 Technical Formulation of MOPD

MOPD formalizes multi-teacher integration as on-policy reinforcement learning. The student receives token-level reverse-KL supervision from domain teachers, combined by default with outcome-model advantages.

  • Objective: MOPD casts multi-teacher distillation as an on-policy reinforcement learning objective integrating specialized teacher capabilities into a unified student.The student samples from its own evolving policy rather than relying on static offline expert data.
  • Objective: The reverse-KL loss compares the target student policy with the domain-specialized teacher policy on student-sampled tokens.The formulation distinguishes the training policy, sampling policy, and teacher policy, and uses a stop-gradient operator.
  • Advantage construction: The final MOPD advantages combine teacher-derived advantages with advantages from Outcome Reward Models, including GRPO.This combines dense token-level supervision with outcome-based reward signals.
  • Evaluation: On AIME 2025 and LiveCodeBench, MOPD matches or exceeds the strongest teacher in most domains while combining specialized capabilities.The comparison is presented against traditional post-training approaches.

4.5 Evaluations

MiMo-V2-Flash is evaluated across reasoning, coding, long-context, search, and tool-use tasks, achieving competitive or leading results across these settings. The evaluation also examines the infrastructure supporting its RL and agent training.

  • Post-Training Comparisons: The post-training evaluation compares RL with ORM, MOPD without outcome rewards, and MOPD on math and code tasks.
  • Evaluation Results: MiMo-V2-Flash performs comparably to Kimi-K2-Thinking and DeepSeek-V3.2-Thinking on most reasoning benchmarks while retaining competitive general writing capabilities.
  • Software Engineering: 73.4% on SWE-Bench Verified outperforms all open-source competitors and approaches GPT-5-High.
  • Search and Tool Use: 45.4 on BrowseComp increases to 58.3 with the Appendix C context-management method, while τ2-Bench scores are 95.3, 79.5, and 66.0 for Telecom, Retail, and Airline.
  • RL Infrastructure: The RL infrastructure uses SGLang for inference, Megatron-LM for training, and FP8 for both training and inference.
  • RL Infrastructure: Rollout Routing Replay preserves routed experts between rollout and training, while the Data Scheduler and Tool Manager address distributed-training utilization and agent-tool resource contention.

5 MTP Speedup

This section studies how MTP acceptance length relates to predictive uncertainty and measures its decoding-speed benefits. Lower-entropy contexts permit longer accepted sequences, while speedup depends on acceptance length and hardware conditions.

  • MTP Acceptance Length: Lower-entropy contexts reach approximately 3.6 accepted tokens, whereas higher-uncertainty tasks have shorter acceptance lengths.
  • MTP Acceptance Length: The fitted relationship between next-token cross-entropy and acceptance length achieves R^2 = 0.995.
  • Decoding Speedup: Table 10 evaluates three-layer MTP across per-node batch sizes and acceptance lengths using 16K inputs and 1K outputs.
  • Decoding Speedup: MTP consistently outperforms decoding without MTP, and its speedup scales linearly with acceptance length.
  • Decoding Speedup: Speedup varies with batch size because computation, I/O demands, and kernel efficiency differ across settings.

6 Conclusion, Limitation, and Future Work

The conclusion attributes MiMo-V2-Flash’s reasoning, agentic, and inference-speed strengths to hybrid SWA, lightweight MTP, and MOPD. It identifies remaining gaps in closed-model performance and architectural analysis.

  • Conclusion: MiMo-V2-Flash combines hybrid Sliding Window Attention, lightweight Multi-Token Prediction, and MOPD post-training for strong reasoning, agentic capabilities, and fast inference.
  • Conclusion: The model rivals larger open-weight models such as DeepSeek-V3.2 and Kimi-K2.
  • Limitations and Future Work: A clear performance gap remains relative to the strongest closed-weight models.
  • Limitations and Future Work: The architectural exploration remains preliminary, with limited analysis of design trade-offs.
  • Limitations and Future Work: Future work will study more robust agentic-oriented architectures and scale iterative teacher-student co-evolution in MOPD.

A Contributions and Acknowledgments

The paper acknowledges contributors and teams that supported the work. It also explains that authors within each role are listed alphabetically by first name.

  • Acknowledgments: The acknowledgments thank contributors from the Xiaomi Data Platform, CloudML, NGK, MiChat, Mify, MiKS, and LLM-Plus teams.
  • Acknowledgments: The paper also expresses gratitude to contributors whose names are not explicitly listed.
  • Author Listing: Authors within each role are listed alphabetically by their first name.

Core Contributors

The paper lists a large group of contributors, including the named individuals in the supplied contributor passages.

  • Contributors include Bangjun Xiao, Bingquan Xia, Bo Yang, and Bofei Gao.
  • The contributor list also includes Gang Wang, Gang Xie, Hailin Zhang, and Hanglong Lv.
  • Additional listed contributors include Wenhan Ma, Xiangwei Deng, Xing Zhang, and Yifan Song.

B Reward Hacking of SWE-Bench

The authors identify a flaw in official SWE-Bench images that can enable reward hacking during reinforcement learning. They update the evaluation images and verify that self-built training images do not exhibit this behavior.

  • The official SWE-Bench images fail to properly delete ground-truth commits.This bug can let models peek at future commits during reinforcement learning.
  • Reward hacking can inflate evaluation when models obtain rewards by peeking at future commits.The paper illustrates this behavior with keyword-based counting of git-hacking attempts in rollout trajectories.
  • The authors update to the newest SWE-Bench image and repeatedly confirm no reward hacking in self-built training images.

C Context Management

The approach manages context by refining the conditioning context rather than only optimizing model parameters. It combines file-based context augmentation with aggressive memory compression for context consolidation.

  • Context management strategically refines the conditioning context C in P(y| C, θ).The method complements parameter optimization by changing the information conditioning the model.
  • Context augmentation: Tools, documents, and databases are exposed uniformly as files for retrieval through Bash commands.This leverages the model’s native code-generation capabilities for context augmentation.
  • Context consolidation: Aggressive memory compression addresses the “Lost in the Middle” phenomenon during context consolidation.
Loading 2601.02780v2…