Source-linked AI summary
MiMo-Embodied: X-Embodied Foundation Model Technical Report
Xiaoshuai Hao, Lei Zhou, Zhijian Huang, Zhiwen Hou, Yingbo Tang, Lingfeng Zhang, Guang Li, Zheng Lu, Shuhuai Ren, Xianhui Meng, Yuchen Zhang, Jing Wu, Jinghui Lu, Chenxu Dang, Jiayi Guan, Jianhua Wu, Zhiyi Hou, Hanbing Li, Shumeng Xia, Mingliang Zhou, Yinan Zheng, Zihao Yue, Shuhao Gu, Hao Tian, Yuannan Shen, Jianwei Cui, Wen Zhang, Shaoqing Xu, Bing Wang, Haiyang Sun, Zeyu Zhu, Yuncheng Jiang, Zibin Guo, Chuhong Gong, Chaofan Zhang, Wenbo Ding, Kun Ma, Guang Chen, Rui Cai, Diyun Xiang, Heng Qu, Fuli Luo, Hangjun Ye, Long Chen
TL;DR
Existing embodied VLMs specialize in separate autonomous-driving or embodied-AI capabilities, motivating a unified model for both domains. MiMo-Embodied integrates these areas through a cross-embodied VLM and achieves superior performance across 29 benchmarks, with reported gains from multi-stage training.
Problem
Existing embodied VLMs emphasize separate capabilities in autonomous driving or embodied AI rather than integrating both domains in one model.
Method
MiMo-Embodied is a unified vision-language model integrating autonomous driving and embodied AI, trained with multi-stage learning and reinforcement learning.
Results
MiMo-Embodied outperforms open-source, closed-source, general, and specialized baselines across 17 embodied-AI and 12 autonomous-driving benchmarks.
Takeaways & Limitations
The reported results indicate that multi-stage training can improve embodied and autonomous-driving capabilities together without sacrificing single-task performance.
Abstract
from arXiv · showhide
We open-source MiMo-Embodied, the first cross-embodied foundation model to successfully integrate and achieve state-of-the-art performance in both Autonomous Driving and Embodied AI. MiMo-Embodied sets new records across 17 embodied AI benchmarks in Task Planning, Affordance Prediction and Spatial Understanding, while also excelling in 12 autonomous driving benchmarks across Environmental Perception, Status Prediction, and Driving Planning. Across these tasks, MiMo-Embodied significantly outperforms existing open-source, closed-source, and specialized baselines. Our results indicate that through multi-stage learning, curated data construction, and CoT/RL fine-tuning, these two domains exhibit strong positive transfer and mutually reinforce one another. We provide a detailed analysis of our model design and training methodologies to facilitate further research. Code and models are available at https://github.com/XiaomiMiMo/MiMo-Embodied.
1 Introduction
MiMo-Embodied unifies autonomous driving and embodied AI in one open-source VLM, using curated multimodal data and progressive training to support both domains. It achieves leading results across 17 embodied-AI and 12 autonomous-driving benchmarks.
- Motivation and contribution: MiMo-Embodied is a unified VLM that merges autonomous driving and embodied AI tasks in a single model.It evaluates environmental perception, status prediction, and driving planning alongside embodied affordance prediction, task planning, and spatial understanding.
- Training approach: The model is trained with a diverse dataset spanning general visual-language understanding, embodied tasks, and autonomous-driving scenarios.The training pipeline further applies progressive learning and reinforcement-learning fine-tuning.
- Evaluation scope: 17 embodied-AI benchmarks and 12 autonomous-driving benchmarks show broad coverage across planning, perception, prediction, and spatial reasoning.
- Results: MiMo-Embodied outperforms open-source, closed-source, general-purpose, and specialized baselines across the evaluated embodied and driving tasks.
2 Architecture
MiMo-Embodied combines visual encoding, modality projection, and language reasoning to process single images, multiple images, and videos. Its architecture aligns visual representations with an LLM for multimodal understanding and response generation.
- Architecture: The architecture has three components: a Vision Transformer, a projector, and an LLM for visual encoding, alignment, and textual reasoning.
- Visual Input Processing: The Vision Transformer encodes single images, multiple images, and videos into visual tokens for subsequent processing.Self-attention extracts salient features and relationships from the visual inputs.
- Visual Input Processing: The projector uses an MLP to map high-dimensional visual tokens into a latent space compatible with the LLM.
- Language Reasoning: The LLM interprets projected visual data and generates coherent, contextually relevant responses.
3 Training Dataset
MiMo-Embodied is trained on general multimodal data plus specialized embodied-AI and autonomous-driving datasets. These sources provide supervision for visual understanding, affordances, planning, spatial reasoning, and driving-focused perception.
- Dataset composition: The training corpus combines general visual-language data with specialized embodied-AI and autonomous-driving datasets.The three dataset components target foundational understanding, embodied affordance/planning/spatial capabilities, and driving perception/prediction/planning.
- General Dataset: General data covers visual grounding, document and chart comprehension, video understanding, and multimodal reasoning.
- Embodied AI Dataset: Embodied-AI data targets affordance prediction, high-level task planning, and spatial understanding.
- Affordance Prediction: RoboAfford supports object- and scene-level interaction reasoning, including functional-part localization and placement-region identification.
- Task Planning: Planning data integrates Cosmos-Reason1, EgoPlan-IT, and RoboVQA to cover physical grounding, egocentric next-action prediction, and long-horizon reasoning.
- Spatial Understanding: Spatial datasets provide 3D question answering, grounding, bounding-box localization, and reasoning over object relationships and scene layouts.
4 Training Strategy
MiMo-Embodied uses four progressive training stages that build embodied reasoning, driving specialization, chain-of-thought transparency, and reinforcement-learning refinement. The stages are designed to accumulate capabilities across both domains.
- Training Strategy: A progressive four-stage strategy develops embodied and driving capabilities before applying chain-of-thought and reinforcement-learning refinement.
- Stage 1: Embodied AI Supervised Fine-tuning: Stage 1 combines general data with embodied-AI supervision to establish vision-language understanding, affordance prediction, planning, and spatial reasoning.
- Stage 2: Autonomous Driving Fine-tuning: Stage 2 specializes the model for autonomous driving using samples covering multi-view spatial reasoning, temporal consistency, and traffic-scenario analysis.
- Stage 3: Chain-of-Thought Fine-tuning: Stage 3 uses generated chain-of-thought samples to make analysis, candidate evaluation, and final decisions explicit across embodied and driving tasks.
- Stage 4: Reinforcement Learning Fine-tuning: Stage 4 applies GRPO-based reinforcement learning to improve precision, reliability, correctness, and response quality on curated multimodal tasks.
5 Evaluation
MiMo-Embodied is evaluated across embodied AI and autonomous driving benchmarks, including affordance prediction, task planning, spatial understanding, perception, prediction, and planning. It also demonstrates practical capabilities in robotic manipulation and complex driving scenarios.
- Embodied AI Benchmarks: MiMo-Embodied achieves state-of-the-art performance across all affordance-prediction benchmarks, with large margins on VABench-Point, Part-Afford, and RoboAfford-Eval.The results support fine-grained affordance reasoning for interpreting commands during physical interaction.
- Embodied AI Benchmarks: MiMo-Embodied leads RoboVQA and performs competitively on EgoPlan2, demonstrating causal inference and long-horizon action-sequence reasoning.Task planning translates abstract instructions into executable action sequences.
- Embodied AI Benchmarks: MiMo-Embodied achieves state-of-the-art results on CV-Bench and leads RoboSpatial, RefSpatial-Bench, and relation-CRPE, while remaining competitive on other spatial benchmarks.The results cover spatial relationships, language grounding, object references, and contextual question answering.
- Autonomous Driving Benchmarks: MiMo-Embodied demonstrates strong autonomous-driving perception, including state-of-the-art panoramic semantic understanding and robustness in challenging local-perception scenarios.It also captures individual behavioral intentions and models multi-agent interactions on MME-RealWorld and DriveLM.
- Autonomous Driving Benchmarks: MiMo-Embodied achieves outstanding performance across planning-oriented driving benchmarks and produces coherent decisions for turning, U-turning, car-following, and lane-change overtaking.Trajectory planning gains are especially pronounced for turns, obstacle nudges, and lane changes under the open-loop L2 evaluation.
- Qualitative Evaluation: In manipulation tasks, MiMo-Embodied combines affordance understanding with spatial and relational reasoning to localize grasp and placement points.Examples include identifying handles, counting objects, interpreting “between,” and selecting objects among multiple household scenes.
- Qualitative Evaluation: MiMo-Embodied consistently outperforms the Qwen2.5-VL baseline in 3-second trajectory prediction, especially for interactive maneuvers requiring anticipation of other road users.The setup uses five front-view frames sampled at 2 Hz and evaluates open-loop L2 distance.
- Overall Evaluation: 62.4% average embodied-task performance and 63.3% best autonomous-driving performance demonstrate synergistic improvement from multi-stage training.These correspond to improvements of 4% and 8.1%, respectively, over Embodied+AD.
6 Conclusion and Next Steps
MiMo-Embodied is presented as a cross-embodied vision-language model for autonomous driving and embodied AI, with future work extending it toward vision-language-action models.
- MiMo-Embodied achieves state-of-the-art performance across autonomous driving and embodied AI tasks.
- The report describes MiMo-Embodied as the first open-source VLM integrating autonomous driving and embodied AI.
- Future work will explore embodied AI and autonomous driving VLA models for interaction, task execution, and responses to dynamic scenarios.
7 Contributions and Acknowledgments
The acknowledgments section lists the report’s contributors.
- The report lists Xiaoshuai Hao, Lei Zhou, and Zhijian Huang among its contributors.
- The listed contributors also include Zhiwen Hou, Yingbo Tang, and Lingfeng Zhang.
- Additional listed contributors are Guang Li, Zheng Lu, Shuhuai Ren, Fuli Luo, Hangjun Ye, and Long Chen.
A.1 General Visual Understanding
After multi-stage embodied and autonomous-driving fine-tuning, MiMo-Embodied retains general visual understanding while improving several benchmarks.
- Table 8 compares MiMo-Embodied with general visual understanding models, marking the best and second-best listed results.
- MiMo-Embodied preserves its base model’s general capabilities after embodied and autonomous-driving fine-tuning.
- 9.71 and 9.88 points are gained on MMMU-Prostandard and MMMU-Provision, respectively.
- MiMo-Embodied reaches 81.10% on Mantis and 84.20% on AI2D for visual question answering.
- MiMo-Embodied improves over MiMo-VL by 4.15 points on PixmoCount and 2.03 points on CountBench.
A.2.1 Spatial Understanding
The supplied examples cover embodied spatial understanding, affordance prediction, planning, and autonomous-driving perception, prediction, and planning.
- Spatial Understanding: Eight examples illustrate embodied spatial understanding tasks.
- Affordance Prediction: Ten examples illustrate embodied affordance prediction tasks involving object points and free-space locations.
- Planning and Autonomous Driving: Four examples illustrate embodied planning tasks, while autonomous-driving examples cover scene perception, prediction ability, and planning.