PaperlayerView exact revision on arXiv

Source-linked AI summary

MiMo-Embodied: X-Embodied Foundation Model Technical Report

Xiaoshuai Hao, Lei Zhou, Zhijian Huang, Zhiwen Hou, Yingbo Tang, Lingfeng Zhang, Guang Li, Zheng Lu, Shuhuai Ren, Xianhui Meng, Yuchen Zhang, Jing Wu, Jinghui Lu, Chenxu Dang, Jiayi Guan, Jianhua Wu, Zhiyi Hou, Hanbing Li, Shumeng Xia, Mingliang Zhou, Yinan Zheng, Zihao Yue, Shuhao Gu, Hao Tian, Yuannan Shen, Jianwei Cui, Wen Zhang, Shaoqing Xu, Bing Wang, Haiyang Sun, Zeyu Zhu, Yuncheng Jiang, Zibin Guo, Chuhong Gong, Chaofan Zhang, Wenbo Ding, Kun Ma, Guang Chen, Rui Cai, Diyun Xiang, Heng Qu, Fuli Luo, Hangjun Ye, Long Chen

arXiv:2511.16518v2cs.ROcs.CLcs.CV

TL;DR

Specialized embodied VLMs focus on individual capabilities or driving-specific aspects, while a unified model spanning embodied AI and autonomous driving remains a research gap. MiMo-Embodied brings both domains into one VLM and reports leading results across their benchmarks, outperforming general and specialized models.

  • Problem

    The cited embodied and driving VLMs focus on individual capabilities or domain-specific driving aspects, motivating a model that spans both domains.

    Source: §1 · p. 3

  • Method

    MiMo-Embodied merges autonomous driving and embodied AI tasks into a single VLM.

    Source: §1 · p. 3

  • Results

    MiMo-Embodied delivers leading performance across 17 embodied AI benchmarks and 12 autonomous driving benchmarks, outperforming general and specialized models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    In the reported ablation, multi-stage training reached 62.4% average performance on embodied tasks and 63.3% best performance on autonomous driving tasks without sacrificing single-task performance.

    Source: §5.3 · p. 23

Abstract

from arXiv · show

We open-source MiMo-Embodied, the first cross-embodied foundation model to successfully integrate and achieve state-of-the-art performance in both Autonomous Driving and Embodied AI. MiMo-Embodied sets new records across 17 embodied AI benchmarks in Task Planning, Affordance Prediction and Spatial Understanding, while also excelling in 12 autonomous driving benchmarks across Environmental Perception, Status Prediction, and Driving Planning. Across these tasks, MiMo-Embodied significantly outperforms existing open-source, closed-source, and specialized baselines. Our results indicate that through multi-stage learning, curated data construction, and CoT/RL fine-tuning, these two domains exhibit strong positive transfer and mutually reinforce one another. We provide a detailed analysis of our model design and training methodologies to facilitate further research. Code and models are available at https://github.com/XiaomiMiMo/MiMo-Embodied.

1 Introduction

MiMo-Embodied unifies autonomous driving and embodied AI in one VLM and evaluates capabilities across both domains. The introduction reports strong performance across 17 embodied-AI and 12 autonomous-driving benchmarks.

1 Introduction

  • Specialized embodied VLMs focus on narrow indoor or outdoor scenarios, leaving a domain gap and lacking both a unified model and comprehensive cross-embodiment evaluation.The introduction argues that this fragmentation hinders generalization of spatial understanding and reasoning across diverse settings.Source: §1 · p. 3
  • MiMo-Embodied is presented as the first open-source VLM to unify autonomous driving and embodied AI, with comprehensive cross-embodiment benchmarks covering three core capabilities in each domain.The evaluated capabilities are environmental perception, status prediction, and driving planning for autonomous driving, and affordance prediction, task planning, and spatial understanding for embodied AI.Source: §1 · p. 3, Figure 2 · p. 4
  • MiMo-Embodied uses a diverse general, embodied, and driving dataset with four training stages: general and embodied learning, mixed-domain driving learning, Chain-of-Thought fine-tuning, and GRPO reinforcement learning.The stages build affordance understanding, task planning, and spatial reasoning, then integrate driving knowledge before refining complex reasoning and task-specific performance.Source: §1 · p. 3
  • 17 embodied-AI benchmarks and 12 autonomous-driving benchmarks demonstrate MiMo-Embodied’s strength and leading performance, outperforming open-source and closed-source general VLMs as well as specialized models.The benchmarks span affordance prediction, task planning, and spatial understanding in embodied AI, and environmental perception, status prediction, and driving planning in autonomous driving.Source: §1 · p. 3, Figure 2 · p. 4

2 Architecture

MiMo-Embodied combines a ViT, an MLP projector, and an LLM to process visual inputs and support multimodal reasoning. Its ViT, projector, and LLM are initialized from MiMo-VL [59].

2.1 Core Components of MiMo-Embodied

  • MiMo-Embodied combines a ViT, an MLP projector, and an LLM, with the components initialized from their respective MiMo-VL pre-trained weights [59].The projector maps visual tokens into the LLM’s input space.Source: §2.1 · p. 4, Figure 3 · p. 5

2.2 Visual Input Processing

  • The ViT encodes single images, multiple images, and videos using self-attention, extracting visual features and formatting tokens for subsequent processing.This encoding represents visual context for interpretation by the LLM.Source: §2.2 · p. 4, Figure 3 · p. 5

2.3 Latent Space Projection and Output Generation

  • The MLP projector maps high-dimensional visual tokens into a latent space compatible with the LLM, which interprets the projected data and generates responses.The projection is intended to preserve essential characteristics of the visual representation.Source: §2.3 · p. 5

3 Training Dataset

MiMo-Embodied’s training data combines a general multimodal corpus with specialized embodied-AI and autonomous-driving datasets, curated to support foundational capabilities and real-world tasks.

3 Training Dataset

  • The corpus combines general multimodal data with specialized embodied-AI and autonomous-driving datasets to build foundational capabilities aligned with real-world applications.Figure 4 groups the training data into general, embodied-AI, and autonomous-driving components.Source: §3 · p. 5, Figure 4 · p. 6

3.1 General Dataset

  • MiMo-VL [59] contributes high-resolution images, video, long text and documents, and synthetic reasoning data spanning visual grounding, document and chart comprehension, video understanding, and multimodal reasoning.These categories support object localization, structured-content extraction, temporal reasoning, and complex logical inference.Source: §3.1 · pp. 5–6

3.2.2 High-level Task Planning

  • Planning data combine cross-embodiment long-video understanding and reasoning, egocentric multiple-choice next-action prediction, and long-horizon activity question answering.Cosmos-Reason1 uses structured video captions and long-chain reasoning; EgoPlan-IT provides historical task video and candidate actions, while RoboVQA adds ten QA types.Source: §3.2.2 · p. 7

3.2.3 Spatial Understanding

  • Spatial data combine 3D question answering and camera-coordinate box grounding with egocentric spatial and navigation QA, RGB-D referring, and object-localization tasks.The supervision covers 3D relationships, scene layouts, route planning, spatial constraints, and free-space identification for placement.Source: §3.2.3 · pp. 7–8

3.3.1 Environmental Perception

  • Driving perception data span scene-level context and traffic elements, fine-grained object attributes and behavior, and localization of decision-critical obstacles, signs, and pedestrians.These datasets support both holistic scenario understanding and targeted analysis of localized objects and their driving impact.Source: §3.3 · p. 8, §3.3.1 · pp. 8–9

3.3.2 Status Prediction

  • Status-prediction data support forecasting traffic-element motion and interactions from sequential observations, including intent prediction in real-world driving scenarios.The supervision is intended to help anticipate behavior such as turns, pedestrian crossings, and lane changes.Source: §3.3.2 · p. 9

3.3.3 Driving Planning

  • Driving-planning supervision covers safety-checked ego meta-actions and goal alignment, plus interpretable decision reasoning grounded in environmental factors, traffic rules, and safety priorities.The described data also pair driving events or video with actions and justifications, and include knowledge for defensive driving.Source: §3.3.3 · p. 10

4 Training Strategy

MiMo-Embodied uses progressive four-stage training, building on MiMo-VL with embodied-AI and driving supervision before CoT reasoning and reinforcement learning to develop multimodal capabilities across both domains.

4 Training Strategy

  • The four-stage strategy builds on previously acquired capabilities, adding specialized embodied-AI and driving supervision before CoT fine-tuning and reinforcement learning.The training configuration for each stage is summarized in Table 1.Source: §4 · p. 10, Table 1 · p. 11

4.1 Stage 1: Embodied AI Supervised Fine-tuning

  • Stage 1 combines general vision-language data with curated embodied datasets to establish grounded instruction understanding, affordance prediction, task planning, and spatial reasoning.Training spans fine-grained object-part localization through scene-level spatial relationships. Here, “MiMo-VL” denotes the 7B-SFT-2508 checkpoint, rather than the earlier 7B-SFT-2505 checkpoint reported under that name.Source: §4.1 · pp. 10–11

4.2 Stage 2: Autonomous Driving Supervised Fine-tuning

  • Stage 2 specializes the model for driving through multi-camera, temporal, and traffic-scenario reasoning across environment comprehension, status prediction, and driving planning.Supervision emphasizes safety-critical perception, including hazardous-object identification, traffic-participant intent prediction, and complex road geometry.Source: §4.2 · p. 11

4.3 Stage 3: Chain-of-Thought Reasoning Supervised Fine-tuning

  • Stage 3 uses generated chain-of-thought samples to train sequential analysis, candidate evaluation, and justified decisions for embodied and driving problems.Examples cover affordances and spatial constraints in embodied tasks, and risk assessment, trajectory evaluation, and behavior justification in driving, supporting interpretable multi-step reasoning.Source: §4.3 · p. 11

4.4 Stage 4: Reinforcement Learning Fine-Tuning

  • Stage 4 applies GRPO [19] to curated multimodal tasks, targeting precision and reliability in visual reasoning, spatial grounding, and complex instruction following.Rewards use exact-match answers for multiple-choice reasoning, IoU for grounding or pointing, and strict template checks for format compliance; training targets corner cases and earlier failure modes.Source: §4.4 · p. 12

5 Evaluation

This section evaluates MiMo-Embodied through benchmark comparisons and downstream navigation, manipulation, and trajectory-planning tests, alongside ablations of cross-domain and multi-stage training. It examines performance across embodied AI and autonomous driving, including reported synergistic gains without sacrificing single-task performance.

5.1.1 Embodied AI Benchmarks

  • MiMo-Embodied achieves SOTA across affordance benchmarks, leads RoboVQA and several spatial benchmarks, and remains competitive on EgoPlan2, EmbSpatial, SAT, and MetaVQA.The affordance results are strongest on VABench-Point, Part-Afford, and RoboAfford-Eval; spatial results lead on CV-Bench, RoboSpatial, RefSpatial-Bench, and CRPE-relation.Source: §5.1.1 · pp. 12–13, Table 2 · p. 13, Table 3 · p. 14

5.1.2 Autonomous Driving Benchmarks

  • In autonomous-driving benchmarks, MiMo-Embodied shows state-of-the-art panoramic semantic understanding and robust local perception, strong prediction, and outstanding planning results.It captures individual behavioral intentions on MME-RealWorld and multi-agent interactions on DriveLM; the planning results cover accurate decisions and interpretable reasoning aligned with traffic logic.Source: §5.1.2 · pp. 15–16, Table 4 · p. 15, Table 5 · p. 16

5.2.1 MiMo-Embodied for Embodied Navigation & Manipulation

  • In household navigation, MiMo-Embodied localizes target objects precisely, including multiple plant pots, while GPT-4o and RoboBrain-2.0 produce less precise or incomplete points.The four scenarios require decomposing high-level instructions into map-region navigation and egocentric object localization.Source: §5.2.1 · pp. 17–18, Figure 5 · p. 17, Figure 6 · p. 18
  • In manipulation tasks, MiMo-Embodied integrates affordance prediction with spatial reasoning for grasping and placement, including ordinal counting, relational placement, and height-based selection.Across the compared visualizations, it identifies functionally appropriate points where GPT-4o and RoboBrain-2.0 often misalign, while Qwen2.5-VL struggles with spatial affordance.Source: §5.2.1 · p. 19, Figure 8 · p. 20

5.2.2 MiMo-Embodied for Autonomous Driving

  • On NAVSIM, MiMo-Embodied consistently outperforms comparable competing models; its qualitative examples include intersection turns, U-turns, car-following, and lane-change overtaking.All models use a single front-view image and the same number of training epochs; MiMo-Embodied uses 3D convolutions to reduce LLM tokens while preserving spatial context, with DiffGRPO applied after imitation learning.Source: §5.2.2 · pp. 20–21, Table 6 · p. 20, Figure 9 · p. 21
  • On proprietary trajectory planning, MiMo-Embodied reduces error versus Qwen2.5-VL across evaluated categories, with the largest gains in turns, obstacle nudges, and lane changes.The open-loop evaluation predicts a 3-second trajectory from five front-view frames sampled at 2 Hz, using L2 distance over the horizon.Source: §5.2.2 · p. 22, Figure 10 · p. 22

5.3 Ablation Study

  • Ablations indicate that single-domain training limits cross-domain generalization and joint training slightly reduces driving performance, whereas multi-stage training mitigates interference through progressive course learning.The comparison controls for approximately the same number of loss tokens in the joint-training setting.Source: §5.3 · p. 22, Table 7 · p. 23
  • 62.4% average performance on embodied tasks, a 4% improvement compared to Embodied+AD, was achieved by the multi-stage training strategy.Source: §5.3 · p. 23
  • 63.3% best performance on autonomous driving tasks, an 8.1% improvement compared to Embodied+AD, was achieved by the multi-stage training strategy.Source: §5.3 · p. 23
  • The results demonstrate synergistic improvement in embodied intelligence and autonomous driving capabilities without sacrificing single-task performance.The report presents this multi-stage training strategy as an effective paradigm for building a unified embodied foundation model.Source: §5.3 · p. 23

6 Conclusion and Next Steps

MiMo-Embodied achieves state-of-the-art performance across embodied AI and autonomous driving, outperforming general and specialized VLMs on 29 benchmarks. The authors plan to extend the model toward VLA systems and multimodal learning.

6 Conclusion and Next Steps

  • Across 29 benchmarks, MiMo-Embodied achieves superior performance in embodied AI and autonomous driving, outperforming open- and closed-source general VLMs and single-domain specialized VLMs.The report presents it as the first open-source VLM integrating these two domains.Source: §6 · p. 23
  • Future work will explore embodied AI and autonomous driving VLA models, multimodal learning with 3D point clouds, long-term planning, and human-robot interaction.The planned VLA models aim to support natural-language task execution and responses to real-time traffic conditions.Source: §6 · p. 23

Appendix · A Appendix · A.1 General Visual Understanding

After multi-stage embodied and autonomous-driving fine-tuning, MiMo-Embodied preserves its base model’s general visual proficiency and improves on several benchmarks. The evaluation covers reasoning, visual question answering, visual search, and counting tasks.

A.1 General Visual Understanding

  • MiMo-Embodied preserves MiMo-VL’s general capabilities after specialized fine-tuning while improving on some general visual benchmarks.The report concludes that training for embodied AI and autonomous driving preserves and amplifies general vision-language proficiency.Source: §A.1 · p. 29, Table 8 · p. 29
  • MiMo-Embodied gains 9.71 points on MMMU-Prostandard and 9.88 points on MMMU-Provision over its base model.Both are absolute gains on complex reasoning tasks.Source: §A.1 · p. 29, Table 8 · p. 29
  • MiMo-Embodied scores 81.10% on Mantis and 84.20% on AI2D, with competitive visual question answering results.The evaluation metrics for all benchmarks are accuracy.Source: §A.1 · p. 29, Table 8 · p. 29
  • MiMo-Embodied achieves the best performance on the V* benchmark.The comparison includes its base model and top-tier proprietary and open-source models.Source: §A.1 · p. 29, Table 8 · p. 29
  • MiMo-Embodied improves over MiMo-VL by 4.15 points on PixmoCount and 2.03 points on CountBench.These gains are reported for general counting benchmarks.Source: §A.1 · p. 29, Table 8 · p. 29
  • The report suggests gains may stem from better fine-grained object recognition, structural understanding, dynamic relationship modeling, and 3D spatial reasoning.It links the first two mechanisms to reasoning and counting gains, and the latter two to spatial benchmarks.Source: §A.1 · p. 29

A.2 Embodied Visualization Examples · A.2.1 Spatial Understanding

The visualization section presents eight examples of embodied spatial understanding, shown in Figures 11–18.

A.2.1 Spatial Understanding

A.2.2 Affordance Prediction … A.3.3 Planning Task

The appendix illustrates embodied affordance prediction and planning alongside autonomous-driving scene perception, prediction, and planning. The examples include prompts about locating points in free space and on vehicle features.

A.2.2 Affordance Prediction

  • The prompts request points in free space, including locations beside the blue, between the yellow cup and cardboard, and in front of a bottle.Source: §A.2.2 · p. 38
  • The cited Xiaomi-car prompts ask for several points on the logo (specified as on the left in the image), the license plate, the wheels, and the door.Source: §A.2.2 · p. 45

A.2.3 Planning Task

A.3.1 Scene perception

A.3.2 Prediction ability

A.3.3 Planning Task

Related papers

Loading 2511.16518v2…