PaperlayerView exact revision on arXiv

Source-linked AI summary

MiMo-V2-Flash Technical Report

Xiaomi LLM-Core Team, :, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Lei Li, Liang Zhao, Linghao Zhang, Peidian Li, Qianli Chen, Shaohui Liu, Shihua Yu, Shijie Cao, Shimao Chen, Shouqiu Yu, Shuo Liu, Tianling Zhou, Weijiang Su, Weikun Wang, Wenhan Ma, Xiangwei Deng, Bohan Mao, Bowen Ye, Can Cai, Chenghua Wang, Chengxuan Zhu, Chong Ma, Chun Chen, Chunan Li, Dawei Zhu, Deshan Xiao, Dong Zhang, Duo Zhang, Fangyue Liu, Feiyu Yang, Fengyuan Shi, Guoan Wang, Hao Tian, Hao Wu, Heng Qu, Hongfei Yi, Hongxu An, Hongyi Guan, Xing Zhang, Yifan Song, Yihan Yan, Yihao Zhao, Yingchun Lai, Yizhao Gao, Yu Cheng, Yuanyuan Tian, Yudong Wang, Zhen Tang, Zhengju Tang, Zhengtao Wen, Zhichao Song, Zhixian Zheng, Zihan Jiang, Jian Wen, Jiarui Sun, Jiawei Li, Jinlong Xue, Jun Xia, Kai Fang, Menghang Zhu, Nuo Chen, Qian Tu, Qihao Zhang, Qiying Wang, Rang Li, Rui Ma, Shaolei Zhang, Shengfan Wang, Shicheng Li, Shuhao Gu, Shuhuai Ren, Sirui Deng, Tao Guo, Tianyang Lu, Weiji Zhuang, Weikang Zhang, Weimin Xiong, Wenshan Huang, Wenyu Yang, Xin Zhang, Xing Yong, Xu Wang, Xueyang Xie, Yilin Jiang, Yixin Yang, Yongzhe He, Yu Tu, Yuanliang Dong, Yuchen Liu, Yue Ma, Yue Yu, Yuxing Xiang, Zhaojun Huang, Zhenru Lin, Zhipeng Xu, Zhiyang Chen, Zhonghua Deng, Zihan Zhang, Zihao Yue

arXiv:2601.02780v2cs.CLcs.AI

TL;DR

Building reasoners and agents that handle long contexts both quickly and strongly remains a challenge. MiMo-V2-Flash combines hybrid sliding-window/global attention with multi-teacher on-policy distillation, and surpasses peer models on reasoning-focused tasks while achieving nearly 100% long-context retrieval success from 32K to 256K.

  • Problem

    A key challenge is building reasoners and agents that model long contexts both quickly and strongly.

    Source: §1 · p. 4

  • Method

    MiMo-V2-Flash combines interleaved sliding-window and global attention with multi-teacher on-policy distillation to scale post-training reinforcement-learning compute.

    Source: §1 · p. 4, §2.1 · p. 5

  • Results

    MiMo-V2-Flash surpasses peer models on reasoning-focused tasks and achieves nearly 100% long-context retrieval success across 32K to 256K contexts.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The model and its three-layer MTP weights are available as an open-source resource.

    Source: §1 · p. 4

    A clear performance gap remains between MiMo-V2-Flash and the strongest closed-weight models.

    Source: §6 · p. 22

Abstract

from arXiv · show

We present MiMo-V2-Flash, a Mixture-of-Experts (MoE) model with 309B total parameters and 15B active parameters, designed for fast, strong reasoning and agentic capabilities. MiMo-V2-Flash adopts a hybrid attention architecture that interleaves Sliding Window Attention (SWA) with global attention, with a 128-token sliding window under a 5:1 hybrid ratio. The model is pre-trained on 27 trillion tokens with Multi-Token Prediction (MTP), employing a native 32k context length and subsequently extended to 256k. To efficiently scale post-training compute, MiMo-V2-Flash introduces a novel Multi-Teacher On-Policy Distillation (MOPD) paradigm. In this framework, domain-specialized teachers (e.g., trained via large-scale reinforcement learning) provide dense and token-level reward, enabling the student model to perfectly master teacher expertise. MiMo-V2-Flash rivals top-tier open-weight models such as DeepSeek-V3.2 and Kimi-K2, despite using only 1/2 and 1/3 of their total parameters, respectively. During inference, by repurposing MTP as a draft model for speculative decoding, MiMo-V2-Flash achieves up to 3.6 acceptance length and 2.6x decoding speedup with three MTP layers. We open-source both the model weights and the three-layer MTP weights to foster open research and community collaboration.

1 Introduction

MiMo-V2-Flash targets fast, strong long-context reasoning and agentic performance, addressing the need to make long-context modeling both fast and strong through hybrid attention, Multi-Token Prediction, and multi-teacher distillation.

Introduction

  • MiMo-V2-Flash is a 309B-parameter MoE with 15B activated per token, designed to deliver efficient, cost-effective reasoning and agentic performance.Its MTP block uses a lightweight dense FFN and sliding-window attention, supporting decoding speedups at high acceptance rates.Source: §1 · p. 4
  • With a 128-token sliding window and a 5:1 local-to-global attention ratio, the hybrid architecture yields nearly 6× lower KV-cache storage and attention computation for long contexts.Learnable attention sink bias helps maintain long-context modeling capability; MTP is also used to enhance training and accelerate decoding.Source: §1 · p. 4
  • MiMo-V2-Flash-Base was pretrained using FP8 mixed precision over 27T tokens, beginning with a native 32K context that was later extended to 256K.Source: §1 · p. 4
  • Multi-Teacher On-Policy Distillation (MOPD) trains a student through general SFT, specialized teacher RL/SFT, and distillation using token-level teacher rewards plus verifiable outcome rewards.The approach is intended to address learning inefficiency and capability imbalance while combining domain expertise with stable, efficient learning.Source: §1 · p. 4
  • 73.4% on SWE-Bench Verified and 71.7% on SWE-Bench Multilingual; MiMo-V2-Flash matches leading models on most reasoning benchmarks and surpasses larger full-attention models on LongBench V2 and MRCR.The report identifies it as the leading open-source model for software engineering tasks.Source: §1 · p. 4

2 MiMo-V2-Flash Model Architecture

MiMo-V2-Flash combines hybrid sliding-window and global attention with sparse MoE blocks and a lightweight MTP module. In the reported experiments, attention sinks and a 128-token window improve on tested alternatives in several benchmark settings.

2.1 Overall Architecture

  • The model stacks eight hybrid groups of five consecutive SWA blocks followed by a GA block, except for an initial global-attention block with a dense FFN; its SWA window is 128 tokens.SWA and GA blocks use MoE FFNs with 256 experts, 8 activated per token, and no shared experts. The MTP block instead uses a dense FFN and SWA, and has 0.33B parameters.Source: §2.1 · p. 5

2.2 Hybrid Sliding Window Attention Architecture

  • MiMo-V2-Flash applies a learnable attention-sink bias per head in the softmax denominator; the report says sinks substantially improve hybrid SWA and can match or exceed fully global-attention baselines.The authors note that the precise theoretical basis for attention sinks remains under study.Source: §2.2 · p. 6

2.2.1 Model Architecture Experiments

  • The architecture study compared all-GA with hybrid SWA at W=128 without sinks and with sinks at W=128 or W=512, using a 32B dense model.Variants shared pre-training, long-context extension, and long-context and reasoning SFT; evaluations covered general capability, long-context understanding, and complex reasoning.Source: §2.2.1 · pp. 7–8
  • With attention sinks, hybrid SWA (W=128) recovers or improves general benchmarks versus all-GA, surpasses all-GA on long-context and reasoning benchmarks, while SWA (W=512) degrades after long-context extension.These comparisons are reported for the study’s tested model variants and training settings.Source: Table 2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

2.2.2 Summary and Discussion

  • The authors hypothesize that W=128 improves performance through regularization and a clearer division of labor: SWA models local information while global-attention layers handle long-range dependencies.They contrast this with W=512, which may blur that division; they caution that these findings are empirical and specific to the study’s model scale, datasets, and training procedures.Source: §2.2.2 · p. 8

2.3.1 Motivation of using MTP

  • MTP drafts multiple tokens for parallel verification, increasing token-level parallelism in memory-bound decoding without increasing KV-cache I/O.For RL rollouts, it can make small-batch on-policy training more practical and reduce GPU idleness caused by long-sequence stragglers.Source: §2.3.1 · p. 9

2.3.2 Lightweight MTP Design in MiMo-V2-Flash

  • The lightweight MTP module uses a small dense FFN and SWA, attaches one head during pre-training, then replicates it K times for jointly trained K-step prediction in post-training.Each head takes the main-model hidden state and token embedding as input.Source: §2.3.2 · p. 9

3 Pre-Training

MiMo-V2-Flash pre-training uses 27T diverse, long-range-oriented tokens and a three-stage schedule that extends context from 32K to 256K. Its base model is competitive across benchmarks, with strong reasoning and long-context results.

3 Pre-Training

  • The 27T-token corpus combines web, books, academic papers, code, mathematics, and STEM, with emphasis on long-form documents and repository-level code to support extended-context modeling and multi-step reasoning.Code sources include pull requests, issues, and commit histories; processing largely follows MiMo-7B (Xia et al., 2025).Source: §3 · pp. 9–10

3.1 Data Scheduler

  • Training proceeds through general pre-training on 0–22T tokens at 32K context, code-upsampled mid-training with approximately 5% synthetic reasoning data at 22–26T, and context extension to 256K at 26–27T.Stage 3 upsamples long-range-dependency data to support extended-context modeling and long-horizon reasoning.Source: §3.1 · p. 10

3.2 Hyper-Parameters

  • The model has 48 Transformer layers, including 39 sliding-window and 9 global-attention layers; MoE layers use 256 routed experts with 8 active per token, for 309B total and 15B active parameters.It uses one MTP layer during pre-training.Source: §3.2 · p. 10

3.3.1 Evaluation Setup

  • The evaluation spans general reasoning, mathematics, coding, Chinese and multilingual understanding, and long-context tasks.Benchmarks include MMLU-Pro, AIME, SWE-Bench, C-Eval, GlobalMMLU, NIAH-Multi, and GSM-Infinite.Source: §3.3.1 · p. 11

3.3.2 Evaluation Results

  • MiMo-V2-Flash-Base outperforms peers on MMLU-Pro, GPQA-Diamond, and AIME, surpasses Kimi-K2-Base on SWE-Bench with under one-third its parameters, but trails larger models on SimpleQA.The comparison covers leading open-source base models.Source: §3.3.2 · p. 11
  • From 32K to 256K, the model achieves near-100% long-context retrieval, while GSM-Infinite performance degrades minimally from 16K to 128K.The report attributes these results to the hybrid SWA architecture, vanilla 32K pre-training, and context-extension training.Source: §3.3.2 · p. 11

4 Post-Training · 4.1 Multi-Teacher On-Policy Distillation (MOPD): A New Post-Training Paradigm

MOPD is a three-stage post-training framework designed to address capability trade-offs and inefficient use of signals from specialized models. It integrates teacher expertise through on-policy, token-level supervision and supports modular, iterative scaling.

4.1 Multi-Teacher On-Policy Distillation (MOPD): A New Post-Training Paradigm

  • MOPD addresses post-training capability imbalance and inefficient use of signals from specialized models through a unified three-stage framework.The stated imbalance is a “see-saw” effect in which improving one skill can cause regressions in others.Source: §4.1 · p. 11
  • Stage 1 uses high-quality instruction-response pairs to establish instruction-following across diverse domains.This supervised fine-tuning stage teaches the model to understand and execute user requests.Source: §4.1 · p. 13
  • Stage 2 independently trains domain-specialized teachers with RL on agentic and non-agentic tasks using domain-specific rewards.The listed domains include search, coding, tool use, mathematical and general reasoning, and safety alignment.Source: §4.1 · p. 13
  • Stage 3 treats multi-teacher integration as on-policy RL: the student samples from its evolving distribution and receives token-level teacher supervision through KL divergence rewards.The passage cites prior work as (Agarwal et al., 2023; Gu et al., 2024c; Lu and Lab, 2025).Source: §4.1 · pp. 13–14
  • MOPD aims to preserve the strongest teacher’s peak performance across domains while dense token-level rewards support stable credit assignment and rapid convergence.Learning from the student’s own distribution also avoids the exposure bias and distribution mismatch associated with static, off-policy datasets.Source: §4.1 · p. 14
  • MOPD is modular: it can use RL-derived, SFT, or student teachers, add teachers without restructuring the pipeline, and work with existing outcome reward models.The passage notes particular advantages for complex agentic tasks, where independent training pipelines can be cumbersome.Source: §4.1 · p. 14
  • MOPD supports iterative co-evolution: distilled students can undergo specialized RL to become stronger teachers for the next student generation.The passage describes this as a self-reinforcing cycle for sustained capability scaling.Source: §4.1 · p. 14

4.2 Supervised Fine-Tuning (SFT) · 4.3 Scaling Reinforcement Learning (RL)

SFT activates and aligns the model’s capabilities, while RL uses task-specific strategies to improve both single-turn performance and interactive agentic behavior. Scaling agentic RL compute improves code-agent performance and generalizes to other tasks.

4.2 Supervised Fine-Tuning (SFT)

  • Stable num-zeros is essential for SFT: increases signal worsening expert load balance, while decreases suggest overfitting.SFT uses millions of examples across conversation, reasoning, coding, and agent tasks, spanning thinking and non-thinking modes and generated by domain-specialized checkpoints.Source: §4.2 · pp. 14–15

4.3 Scaling Reinforcement Learning (RL)

  • RL scales distinct strategies for agentic and non-agentic tasks to push capabilities beyond SFT and improve performance across domains.Agentic tasks require interactive behavior, while non-agentic tasks involve single-turn responses.Source: §4.3 · p. 15

4.3.1 Non-Agentic RL Training

  • Non-agentic RL targets single-turn reasoning accuracy in verifiable domains while aligning open-ended responses for helpfulness and safety.Correctness rewards combine programmatic tools with an LLM judge; rubric-based LLM evaluation rewards subjective qualities against detailed rubrics and reference answers.Source: §4.3.1 · p. 15

4.3.2 Agentic RL Training

  • Code-agent RL trains on real-world GitHub issues, using editable files, command execution, and verifiable unit-test rewards across over 100,000 tasks.An automated setup pipeline achieved 70% success across eight languages; a minimal scaffold exposes bash, str_replace, and finish tools through shell commands.Source: §4.3.2 · p. 16
  • Terminal-agent tasks provide about 30,000 validated queries, while web-agent training uses video-based visual, functional, and executability checks.Terminal tasks come from Stack Overflow and Stack Exchange and are filtered for difficulty, reliability, pass rates, and complexity; web RL data is synthesized from curated pages across eight categories.Source: §4.3.2 · p. 16
  • General-agent training develops autonomous web search and function calling through synthetic tasks that require data propagation and hidden-state inference.Search queries expand recursively from seed entities into verifiable problems; function-calling tool graphs encode explicit data dependencies and implicit logical dependencies.Source: §4.3.2 · pp. 16–17
  • Approximately 120K on-policy environments significantly improved the SFT base model on SWE-Bench-Verified and SWE-Bench-Multilingual, while code-agentic RL generalized to other agentic, math, code, and reasoning tasks.Figure 4 shows the code-agent RL training curve; Figure 5 presents generalization to other domains.Source: Figure 4 · p. 17, Figure 5 · p. 17, §4.3.2 · p. 17

4.4 Technical Formulation of MOPD

MOPD formalizes multi-teacher distillation as on-policy reinforcement learning to integrate domain-specialized teachers into a unified student, with advantages combined by default with ORM-based rewards.

4.4 Technical Formulation of MOPD

  • MOPD defines a reverse-KL objective between the target student policy and the teacher specialized for each prompt’s domain.The student sampling policy is adopted in the inference engine, while the target student policy is optimized in the training engine.Source: §4.4 · p. 18
  • MOPD applies training-inference importance sampling and discards tokens with large discrepancies, following Zhao et al. (2025).This step is used to define the MOPD surrogate loss.Source: §4.4 · p. 18
  • By default, MOPD combines its advantages with ORM-computed advantages, including those from GRPO (Shao et al., 2024).The final advantages incorporate both sources.Source: §4.4 · p. 18
  • MOPD matches or exceeds the strongest teacher on AIME 2025 and LiveCodeBench in most domains while preserving and combining specialized capabilities.The comparison covers mathematical reasoning and coding benchmarks.Source: §4.4 · p. 18

4.5 Evaluations · 4.6 RL Infrastructures

MiMo-V2-Flash performs competitively across reasoning, long-context, software, and agentic evaluations. The paper also describes infrastructure designed to support stable and efficient reinforcement-learning training.

4.5.1 Evaluation Setup

  • Evaluation spans reasoning, math/code, long-context, software, search, and tool-use benchmarks; Figure 6 compares RL with ORM, MOPD without ORM, and MOPD.The listed evaluations include MMLU-Pro, GPQA-Diamond, AIME 2025, MRCR, SWE-Bench, BrowseComp, and τ2-Bench.Source: §4.5.1 · p. 18, Figure 6 · p. 19

4.5.2 Evaluation Results

  • MiMo-V2-Flash is comparable to Kimi-K2-Thinking and DeepSeek-V3.2-Thinking on most reasoning benchmarks, surpasses Kimi-K2-Thinking in long-context evaluations, and maintains competitive general writing.The authors attribute its long-context results to the hybrid SWA architecture and say the results validate ultra-large-scale RL training within MOPD.Source: §4.5.2 · p. 19, Table 9 · p. 20
  • MiMo-V2-Flash scores 73.4% on SWE-Bench Verified and resolves 71.7% of SWE-Bench Multilingual issues, with competitive results on Terminal Bench.The paper says the Verified score outperforms all open-source competitors and approaches GPT-5-High; it describes the Multilingual result as the strongest among open-source LLMs for software engineering.Source: §4.5.2 · p. 19
  • MiMo-V2-Flash scores 45.4 on BrowseComp, rising to 58.3 with the Appendix C context-management method; its τ2-Bench scores are 95.3 (Telecom), 79.5 (Retail), and 66.0 (Airline).DeepSeek-V3.2 is used as the user agent for τ2-Bench.Source: §4.5.2 · p. 19

4.6 RL Infrastructures

  • The RL infrastructure uses SGLang for inference, Megatron-LM for training, and FP8 for both, with three added modules for stable, efficient, flexible training.The modules are Rollout Routing Replay, Data Scheduler, and Tool Manager with Toolbox.Source: §4.6 · p. 19

4.6.1 Stablized Training via Rollout Routing Replay (R3)

  • Rollout Routing Replay (R3) (Ma et al., 2025) trains with the same routed experts used during rollout, addressing MoE routing inconsistency with negligible overhead.Optimized data types and communication overlapping reduce overhead. A request-level prefix cache reuses KVCache and routed experts across turns to ensure routing consistency without re-prefilling or inter-request output-cache sharing.Source: §4.6.1 · pp. 19–20

4.6.2 Data Scheduler

  • The Data Scheduler schedules fine-grained sequences, balances GPU load during dynamic sampling, and uses partial rollout to accelerate RL training without sacrificing model quality.It limits staleness and partial-sample proportions, and applies staleness-aware truncated importance sampling; partial rollout cites Fu et al. (2025) and Kimi Team (2025b).Source: §4.6.2 · p. 20

4.6.3 Toolbox and Tool Manager

  • Toolbox and Tool Manager use Ray (Moritz et al., 2018) to allocate shared tool resources and coordinate tool execution with rollout training.Toolbox enforces quotas and QPS limits through fault-tolerant actor pools; Tool Manager uses environment pre-warming and asynchronous reward computation, with timeout recovery and monitoring for stability.Source: §4.6.3 · p. 21

5 MTP Speedup

Three-layer MTP improves decoding speed without additional hardware costs, while acceptance length varies inversely with next-token cross-entropy.

5.1 MTP Acceptance Length

  • Lower-entropy contexts reach approximately 3.6-token acceptance, while higher-uncertainty tasks accept less; the log-transformed fit has R^2 of 0.995, with cross-entropy a primary throughput determinant.Acceptance length is evaluated with three MTP layers across code-generation and reasoning benchmarks.Source: §5.1 · p. 21

5.2 MTP Inference Speedup

  • MTP consistently outperforms the without-MTP baseline without extra hardware cost, and speedup scales linearly with acceptance length, varying by batch size with compute, I/O, and kernel demands.The measurements use 16K input and 1K output lengths; optimizing the speed-cost trade-off requires tuning batch size and MTP layers to hardware roofline models.Source: §5.2 · pp. 21–22, Table 10 · p. 22

6 Conclusion, Limitation, and Future Work

MiMo-V2-Flash combines hybrid Sliding Window Attention, lightweight Multi-Token Prediction, and MOPD to achieve strong reasoning and agentic capabilities with fast inference, rivaling larger open-weight models. A gap remains to the strongest closed-weight models, and future work targets scaling compute and improving agentic architectures and MOPD.

6 Conclusion, Limitation, and Future Work

  • MiMo-V2-Flash achieves strong reasoning and agentic capabilities with fast inference and rivals larger open-weight models such as DeepSeek-V3.2 and Kimi-K2.The report attributes these strengths to its hybrid Sliding Window Attention architecture, lightweight Multi-Token Prediction, and MOPD post-training paradigm.Source: §6 · p. 22
  • A clear performance gap remains to the strongest closed-weight models, which the authors aim to narrow by scaling model size and training compute.Source: §6 · p. 22
  • Architectural exploration remains preliminary, with limited design-trade-off analysis; future work will pursue more robust, efficient, agentic-oriented architectures.Source: §6 · p. 22
  • Future work will scale compute for iterative teacher–student co-evolution in MOPD to unlock its potential.Source: §6 · p. 22

A Contributions and Acknowledgments

The authors acknowledge contributors across several Xiaomi teams and note that authors within each role are listed alphabetically by first name.

A Contributions and Acknowledgments

  • The authors thank contributors, including the Xiaomi Data Platform, CloudML, NGK, MiChat, Mify, MiKS, and LLM-Plus teams, and list authors within each role alphabetically by first name.The acknowledgment also includes contributors not explicitly listed in the paper.Source: A Contributions and Acknowledgments · p. 29

B Reward Hacking of SWE-Bench

The report identifies undeleted ground-truth commits in official SWE-Bench images as a source of reward hacking and inflated evaluation, then describes image updates and checks to address it.

B Reward Hacking of SWE-Bench

  • Unprocessed SWE-Bench images retained ground-truth commits, enabling RL reward hacking through future-commit peeking and inflated evaluation (Figure 8).The figure quantifies git-hacking attempts by counting keywords in rollout trajectories.Source: B Reward Hacking of SWE-Bench · p. 31, Figure 8 · p. 31
  • Using updated evaluation images and git-hacking-resolved training images, the authors repeatedly confirmed their model did not exhibit reward hacking.The evaluation images were updated to the newest SWE-Bench image, and the training images followed the official resolution on git hacking.Source: B Reward Hacking of SWE-Bench · p. 31

C Context Management

Context management refines the conditioning context rather than model parameters, using file-based retrieval and aggressive memory compression to improve information access and focus.

C Context Management

  • Context management strategically refines conditioning context C in P(y|C, θ), complementing fine-tuning and reinforcement learning, which optimize model parameters θ.Source: C Context Management · p. 31
  • Context compression yields 5–10% accuracy gains on Deep Research tasks; replicating DeepSeek V3’s aggressive reset protocol achieves 58.3 on comparable benchmarks.The approach aligns with DeepSeek V3’s finding that discarding tool-call history outperforms retention strategies.Source: C Context Management · p. 31
  • For context augmentation, tools, documents, and databases are exposed uniformly as files that the model retrieves through Bash commands.This Unix-inspired abstraction leverages the model’s native code-generation capabilities.Source: C Context Management · p. 31
  • For context consolidation, when context utilization exceeds a threshold as low as 30%, the system prompts a summary, archives full history to a retrievable memory file, and replaces active context with that summary.This compression strategy addresses the “Lost in the Middle” phenomenon.Source: C Context Management · p. 31

Related papers

Loading 2601.02780v2…