PaperlayerView exact revision on arXiv

Source-linked AI summary

Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Execution

Rui Cai, Jun Guo, Xinze He, Piaopiao Jin, Jie Li, Bingxuan Lin, Futeng Liu, Wei Liu, Fei Ma, Kun Ma, Feng Qiu, Heng Qu, Yifei Su, Qiao Sun, Dong Wang, Donghao Wang, Yunhong Wang, Rujie Wu, Diyun Xiang, Yu Yang, Hangjun Ye, Yuan Zhang, Quanyun Zhou

arXiv:2602.12684v2cs.ROcs.LG

TL;DR

Large VLA models face inference latency that challenges smooth action chaining. Xiaomi-Robotics-0 combines cross-embodiment pre-training with asynchronous-execution training and deployment strategies, achieving state-of-the-art performance across simulation benchmarks and strong real-robot throughput.

  • Problem

    Large VLA models incur inference latency that challenges smooth chaining of actions across consecutive inference steps.

    Source: §1 · p. 1

  • Method

    The model combines cross-embodiment robot-trajectory and vision-language pre-training with asynchronous-execution post-training and overlapping action execution during inference.

    Source: §1 · pp. 1–2, §2.3 · p. 6

  • Results

    Xiaomi-Robotics-0 achieves state-of-the-art performance across three simulation benchmarks and reaches 1.2 pcs/min on Towel Folding, outperforming three comparison methods.

    Source: §1 · p. 2, §3.2.3 · p. 10

  • Takeaways & Limitations

    The approach supports fast execution and robust performance in challenging deformable-object manipulation.

    Source: §3.2.3 · p. 10

    Action-prefix conditioning can let policies copy prior actions instead of attending to visual and language inputs, making them less reactive and degrading performance.

    Source: §2.2.2 · p. 5

Abstract

from arXiv · show

In this report, we introduce Xiaomi-Robotics-0, an advanced vision-language-action (VLA) model optimized for high performance and fast and smooth real-time execution. The key to our method lies in a carefully designed training recipe and deployment strategy. Xiaomi-Robotics-0 is first pre-trained on large-scale cross-embodiment robot trajectories and vision-language data, endowing it with broad and generalizable action-generation capabilities while avoiding catastrophic forgetting of the visual-semantic knowledge of the underlying pre-trained VLM. During post-training, we propose several techniques for training the VLA model for asynchronous execution to address the inference latency during real-robot rollouts. During deployment, we carefully align the timesteps of consecutive predicted action chunks to ensure continuous and seamless real-time rollouts. We evaluate Xiaomi-Robotics-0 extensively in simulation benchmarks and on two challenging real-robot tasks that require precise and dexterous bimanual manipulation. Results show that our method achieves state-of-the-art performance across all simulation benchmarks. Moreover, Xiaomi-Robotics-0 can roll out fast and smoothly on real robots using a consumer-grade GPU, achieving high success rates and throughput on both real-robot tasks. To facilitate future research, code and model checkpoints are open-sourced at https://xiaomi-robotics-0.github.io

1 Introduction

VLA models can suffer inference latency that disrupts smooth action chaining, while Xiaomi-Robotics-0 combines cross-embodiment pre-training and asynchronous execution techniques for fast, smooth real-robot rollouts [4].

1 Introduction

  • Pre-training on cross-embodiment robot trajectories and vision-language data supports generalizable action generation while maintaining VLM capabilities; the pre-trained model matches its underlying VLM [1] on several vision-language and embodied-reasoning benchmarks.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The model combines a pre-trained VLM [1] for vision and language inputs with a diffusion transformer [48] that generates actions via flow-matching [34] [39].The report releases pre-trained and post-trained checkpoints and inference code to support future research.Source: §1 · pp. 1–2
  • 98.7% average success rate on LIBERO; SimplerEnv results were 85.5% visual matching and 74.7% visual aggregation on Google Robot, and 79.2% on WidowX.The model achieves state-of-the-art performance across three widely used simulation benchmarks.Source: Figure 1 · p. 2, §1 · p. 2
  • On Lego Disassembly and Towel Folding, Xiaomi-Robotics-0 achieves high success rates and outperforms state-of-the-art methods [5] [19] in throughput, enabling smooth real-time execution.Both tasks require precise and dexterous bimanual manipulation.Source: Figure 1 · p. 2, §1 · p. 2
  • For asynchronous execution, action-prefix conditioning [5] is paired with a Λ-shape attention mask [16] [20] [71] to discourage prefix imitation, and deployment aligns consecutive chunks’ timesteps for continuous rollouts.Prefix-only shortcuts can reduce visual and language responsiveness; the attention mask encourages action generation to attend to those conditions.Source: §1 · p. 2
  • 4.54 to 4.75 on ABC→D and 4.67 to 4.80 on ABCD→D: Xiaomi-Robotics-0 improves average length of completing 5 tasks in a row on CALVIN.Source: §1 · p. 2

2 Xiaomi-Robotics-0

Xiaomi-Robotics-0 maps images, language instructions, and robot state to bimanual action chunks. Its training combines robot trajectories with vision-language data, and its post-training and deployment methods support asynchronous execution with continuous action-chunk transitions.

2 Xiaomi-Robotics-0

  • The end-to-end VLA takes observation images, a language instruction, and robot proprioceptive state as input and outputs action chunks for a bimanual robot.The action chunk is cited as [75].Source: §2 · p. 3

2.1 Data

  • Training uses open-source and in-house robot trajectories totaling about 200M timesteps, alongside more than 80M vision-language samples across four tasks.In-house data include 338 hours of Lego Disassembly and 400 hours of Towel Folding trajectories; vision-language tasks include visual grounding, VQA, image captioning, and embodied reasoning and planning.Source: §2.1 · p. 3

2.2 Model & Training

  • The 4.7B-parameter model combines pretrained Qwen3-VL-4B-Instruct with a diffusion transformer that generates action chunks via flow-matching.The VLM processes images and the user instruction; the DiT conditions on its KV cache and the robot proprioceptive state.Source: §2.2 · p. 3

2.2.1 Pre-training

  • Pre-training first co-trains the VLM on robot actions and vision-language data, then freezes it while a DiT learns action generation from robot trajectories.For action prediction, the VLM scores N action-chunk candidates and updates only the candidate with the lowest L1 distance; vision-language and robot data are sampled at a 1:6 ratio.Source: §2.2.1 · p. 4, Figure 3 · p. 4

2.2.2 Post-training

  • Asynchronous post-training prefixes previously committed actions to noisy DiT tokens for chunk consistency, but this can encourage copying the prefix and reduce visual-language reactivity.The method adds a +10 RoPE positional-index offset to distinguish noisy action tokens from clean prefix tokens.Source: §2.2.2 · p. 5, Figure 4 · p. 5

2.3 Deployment

  • During deployment, the robot executes the current chunk while the next is inferred, with the action prefix covering inference latency to keep actions available throughout.The newly generated chunk starts at step Δtinf, and Δtc is set at least as large as Δtinf. On an RTX 4090, inference takes 80 ms; inputs are synchronized to a 30 Hz timeline.Source: §2.3 · p. 6

3 Experiments

Experiments report strong results across simulation benchmarks and real-robot tasks, and assess whether pre-training preserves the model’s vision-language capabilities.

3.1 Simulation Benchmarks

  • 98.7% LIBERO average success, CALVIN averages of 4.80 (ABCD→D) and 4.75 (ABC→D) tasks, and SimplerEnv averages of 85.5%, 74.7%, and 79.2% surpassed baselines.The SimplerEnv averages correspond to Google Robot Visual Matching, Google Robot Variant Aggregation, and WidowX, respectively; the CALVIN figures are average tasks completed in a row of five.Source: Table 1 · p. 7, §3.1 · p. 7, Table 2 · p. 8

3.2.1 Evaluation Details

  • Real-robot evaluation used a bimanual two-arm setup to test Lego disassembly and sorting, and long-horizon folding of deformable towels.Lego evaluation covered large assemblies of 5, 10, or 20 bricks and multi-assemblies, measuring sorting success rate and throughput; towel trials used six towels, two 30-minute rollouts per method, and counted attempts exceeding two minutes as failures.Source: §3.2.1 · pp. 8–9

3.2.3 Results

  • Xiaomi-Robotics-0 achieved the highest Lego Disassembly throughput despite comparable success rates, and folded 1.2 pcs/min versus 1 pcs/min for each comparator.On Lego Disassembly, synchronous methods had slightly higher success rates, while Xiaomi-Robotics-0 outperformed the Training RTC asynchronous variant in throughput. Training RTC could enter repetitive failure loops during towel flinging; Xiaomi-Robotics-0 avoided these failures.Source: §3.2.3 · p. 10, Figure 6

3.2.2 Implementation Details

  • The real-robot comparison included π0.5, Xiaomi-Robotics-0, its synchronous variant, and a Training RTC asynchronous variant.Training used 40k pre-training steps and post-training for 40k steps on Lego Disassembly and 80k on Towel Folding; the action chunk length was T = 30, corresponding to one second of actions.Source: §3.2.2 · p. 9

3.3 Preservation of Vision-Language Capabilities

  • The pre-trained model’s vision-language capabilities were evaluated on general vision-language benchmarks and ERQA, with comparisons to VLA models trained with and without vision-language data.The benchmark suite covered tasks from general question answering to hallucination detection, while ERQA assessed embodied reasoning.Source: §3.3 · p. 10

Model ERQA SEED POPE AI2D MMBench MME MMMU TextVQA SciQA ChartQA

Xiaomi-Robotics-0 outperforms the compared VLA baselines on nearly all general vision-language and embodied reasoning benchmarks while largely retaining its underlying VLM’s capabilities.

Model ERQA SEED POPE AI2D MMBench MME MMMU TextVQA SciQA ChartQA

  • Xiaomi-Robotics-0 outperforms all compared VLA baselines on all but one benchmark, while trailing its underlying pre-trained VLM slightly on most general vision-language benchmarks.The results include strong performance on POPE object-hallucination evaluations and OCR-related tasks such as AI2D.Source: Table 3 · p. 10, Model ERQA SEED POPE AI2D MMBench MME… · p. 10
  • 40.8 vs. 40.0: Xiaomi-Robotics-0 slightly surpasses Qwen3-VL-4B-Instruct on ERQA.The authors hypothesize that vision-language data derived from robot trajectories strengthens visual perception on robot-centric inputs.Source: Model ERQA SEED POPE AI2D MMBench MME… · p. 10
  • Without explicit vision-language supervision, robot-trajectory-only training fails to retain general-purpose vision-language knowledge, causing severe catastrophic forgetting.Xiaomi-Robotics-0 without vision-language data scores zero across all tasks, while π0 achieves near-zero performance on most vision-language tasks.Source: Model ERQA SEED POPE AI2D MMBench MME… · pp. 10–11

4 Related Work

VLA models use cross-embodiment robot data for broad task and out-of-distribution generalization, while their training must balance precise action generation with preserving pre-trained visual-semantic knowledge. For real-robot execution, asynchronous methods address inference latency, but conditioning on committed actions during training can reduce reactivity.

Related Work

  • VLA models leverage large-scale, diverse robot data to solve broad task ranges and generalize to novel environments, instructions, and objects [7] [12] [23] [66] [10] [19].They are typically built on pre-trained vision-language models that capture broad visual-semantic knowledge.Source: §4 · p. 11
  • Discretized action tokens enable next-token prediction [24] [50] but can reduce control precision, while flow matching [3] [10] [19] [45] [52] and diffusion [36] [37] [38] [68] model trajectory distributions.Prior approaches preserve VLM knowledge by jointly training on vision-language data and robot trajectories [19] [26], or detaching the flow-matching objective from the VLM backbone [13].Source: §4 · p. 11
  • Xiaomi-Robotics-0 combines a pre-trained VLM [1] with a diffusion transformer [48] for flow-matching action generation, jointly trains on robot and vision-language data, then freezes the VLM while training the DiT.The two-stage pre-training strategy is intended to avoid catastrophic forgetting of vision-language knowledge.Source: §4 · p. 11
  • Large VLA models can incur non-negligible inference latency during real-robot rollouts [4]; synchronous execution [3] causes pauses, while asynchronous execution lets the robot continue during inference [3] [19] [62].Inference acceleration has also been explored to achieve real-time performance [43].Source: §4 · p. 11
  • RTC [4] inpaints future actions around a frozen prefix, while training RTC [5] conditions on prefixed actions but can exploit temporal correlations, resulting in less reactive behavior.Xiaomi-Robotics-0 proposes practical techniques to address this issue and achieve high throughput on precise, dexterous manipulation tasks.Source: §4 · p. 11

5 Conclusions

Xiaomi-Robotics-0 combines large-scale pre-training with asynchronous-execution post-training for real-time robot control. It achieves state-of-the-art performance across simulation benchmarks and runs fast and smoothly on real robots with a consumer-grade GPU.

5 Conclusions

  • The proposed method achieves state-of-the-art performance across all simulation benchmarks and fast, smooth execution with high success rates and strong throughput on two real-robot tasks using a consumer-grade GPU.The real-robot tasks require precise and dexterous bimanual manipulation.Source: §5 · p. 11
  • Large-scale robot-trajectory and vision-language pre-training supports strong action generation while preventing catastrophic forgetting of the underlying pre-trained VLM’s visual-semantic knowledge.The model is designed for high performance and smooth real-time execution.Source: §5 · p. 11
  • Practical post-training techniques for asynchronous execution enable continuous and reactive real-time execution on real robots.Source: §5 · p. 11
  • Future work will explore larger, more diverse robot datasets and further improve robustness and generalization in real-world tasks.Source: §5 · p. 11

Appendix · A Case Study for Preservation of Vision-Language Capabilities

The appendix presents qualitative case-study results on Xiaomi-Robotics-0’s vision-language capabilities, including comparisons with baseline methods and examples of remaining limitations.

A Case Study for Preservation of Vision-Language Capabilities

B Detailed results on SimplerEnv

This section presents SimplerEnv evaluation results for WidowX and Google Robot.

B Detailed results on SimplerEnv

  • Table 4 covers Xiaomi-Robotics-0’s results on the WidowX evaluations of SimplerEnv.The supplied passage gives the table’s scope but no result values.Source: Table 4 · p. 22
  • Table 5 covers Xiaomi-Robotics-0’s results on the Google Robot evaluations of SimplerEnv.The supplied passage gives the table’s scope but no result values.Source: Table 5 · p. 22

C VLM Benchmark Details

The model’s vision-language capabilities are evaluated on 10 diverse benchmarks spanning comprehensive capabilities, hallucination, reasoning, and fine-grained perception.

C VLM Benchmark Details

  • Ten diverse benchmarks cover comprehensive capabilities, hallucination, reasoning, and fine-grained perception in evaluating the model’s vision-language capabilities.The passage identifies the benchmark set’s scope but does not provide individual results.Source: Table 6 · p. 23

Related papers

Loading 2602.12684v2…