Source-linked AI summary
Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model
Xinghang Li, Jun Guo, Qiwei Li, Long Qian, Hang Lai, Yueze Wang, Hongyu Yan, Jiahang Cao, Xi Chen, Jingen Qu, Jiaxi Song, Nan Sun, Hanye Zhao, Futeng Liu, Wanli Peng, Heyun Wang, Yunhong Wang, Caoyu Xia, Jack Zhao, Diyun Xiang, Hangjun Ye, Heng Qu, Huaping Liu, Jason Li
TL;DR
Embodied generation requires geometric, multi-view, temporal, and robot-compatibility constraints that conventional foundation generators do not reliably satisfy. Xiaomi-Robotics-U0 unifies general and embodied generation in one autoregressive model, achieving state-of-the-art performance across embodied generation tasks and improving downstream policy robustness with generated data.
Problem
Embodied generation requires consistent geometry, interaction dynamics, and robot-compatible observations beyond visual realism, while robot-only adaptation can weaken foundation models’ broader visual capabilities.
Method
Xiaomi-Robotics-U0 continually trains a world foundation model on general and embodied data under one autoregressive objective spanning image, editing, scene, transfer, and video generation.
Results
Xiaomi-Robotics-U0 achieves state-of-the-art performance across single-step and sequential embodied generation tasks while improving downstream robot-policy robustness with generated data.
Takeaways & Limitations
World foundation models can serve as both embodied world models and scalable data engines for embodied intelligence.
Takeaways & Limitations
Embodied transfer relies on depth estimation, while separate scene and video generation can accumulate errors and the 32K context window limits long-duration video modeling.
Abstract
from arXiv · showhide
Recent foundation image and video generation models offer strong generalization and controllability, but their direct application to embodied scenarios is limited by requirements for multi-view consistency, geometric coherence, and robot embodiment constraints. Existing methods typically adapt foundation models with limited robot data, often sacrificing visual knowledge acquired during large-scale pre-training. We present Xiaomi-Robotics-U0, a 38-billion-parameter multimodal autoregressive model for unified embodied synthesis. It treats embodied generation as an extension of foundation image and video generation and jointly optimizes text-to-image generation, image editing, embodied scene generation, embodied transfer, and embodied video generation. This unified framework preserves the generalization of the pre-trained world foundation model while adapting it to embodied settings. Xiaomi-Robotics-U0 is the first model to support high-quality multi-view scene generation across multiple robot embodiments and to introduce structured, controllable embodied transfer for fine-grained editing while preserving multi-view consistency and interaction dynamics. It achieves state-of-the-art results on single-step and sequential generation tasks, outperforming GPT-Image-2.0 in human evaluations of embodied scene generation and transfer, ranking first on World Arena for embodied video generation, and improving the out-of-distribution success rate of pi_0.5 from 36.9% to 63.2% on challenging real-world manipulation tasks. These results show that foundation world models can serve both as embodied world models and scalable data engines for embodied intelligence. Code and checkpoints are available at https://robotics.xiaomi.com/xiaomi-robotics-u0.html.
1 Introduction
Xiaomi-Robotics-U0 unifies foundation image/video generation with embodied synthesis to address multi-view, geometric, physical, and robot-control constraints. It extends embodied generation from static scenes to sequential world modeling and scalable trajectory generation while achieving state-of-the-art results across embodied tasks.
- Motivation: Embodied scenarios require multi-view consistency, geometric and physical coherence, robot embodiment constraints, and temporally consistent interaction dynamics.Generated observations must remain compatible with robot kinematics, camera calibration, and downstream manipulation policies.
- Motivation: Directly applying foundation image or video models often produces inconsistent geometry, implausible robot states, and poor compatibility with robot control.These failures arise because embodied observations must satisfy physical and control constraints rather than merely appear visually realistic.
- Sequential world modeling: The model extends embodied generation to sequential world modeling by learning interleaved embodied sequences and manipulation videos across multiple temporal resolutions.This captures long-horizon task progression and fine-grained interaction dynamics.
- Sequential world modeling: Generated embodied scenes can be rolled out into temporally coherent manipulation videos, providing synthetic trajectories for downstream policy learning.This turns embodied generation into a scalable trajectory generation engine for agentic embodied data generation.
- Results: Xiaomi-Robotics-U0 achieves state-of-the-art performance across single-step and sequential embodied generation tasks, including multi-view scene generation, controllable embodied transfer, and zero-shot embodied video generation.The framework is described as the first unified embodied synthesis model supporting these capabilities within a single autoregressive framework and outperforms GPT-Image-2.0 in human evaluations.
- Unified framework: Xiaomi-Robotics-U0 reformulates embodied synthesis as an extension of foundation image and video generation within a unified continual training paradigm.The framework jointly integrates foundation image generation, image editing, embodied scene generation, embodied transfer, and embodied video generation.
2 Xiaomi-Robotics-U0 · 2.1 Problem Formulation
Xiaomi-Robotics-U0 formulates embodied generation and prediction as a unified multimodal autoregressive modeling problem. It represents diverse single-step and sequential tasks as token sequences optimized with next-token prediction, enabling knowledge sharing across embodied capabilities.
- 2.1 Problem Formulation: The model learns the joint distribution of subsequent multimodal sequences containing text, image, and robot-related control tokens.This formulation uses standard next-token prediction over unified multimodal contexts.
- 2.1 Problem Formulation: Training samples are organized into complementary single-step and sequential groups with different temporal structures.This division covers one-shot input-output mappings as well as long-horizon interaction and future evolution.
- 2.1 Problem Formulation: Single-step formats include Text-to-Image, Any-to-Image editing and generation, and embodied scene generation from robot embodiment and scene descriptions.Any-to-Image uses one to three reference images together with text, while embodied scene generation produces initial multi-view robot observations.
- 2.1 Problem Formulation: Sequential formats include image-text interleaved subtask-subgoal sequences and embodied manipulation videos collected at multiple frame rates.These formats capture sparse task-level transitions and dense interaction dynamics over long-horizon tasks.
- 2.1 Problem Formulation: All tasks are represented as unified multimodal token sequences and optimized under the same autoregressive objective.The shared objective supports knowledge transfer across image generation, scene understanding, embodied reasoning, and future prediction.
- 2.1 Problem Formulation: The model is trained end-to-end at scale with a unified next-token prediction objective and includes an efficient inference extension.The overview identifies both the training objective and the inference-side extension.
2.2 Model Architecture
Xiaomi-Robotics-U0 builds on EMU3.5 and represents modalities in a shared autoregressive framework for unified embodied generation. An anti-diagonal decoding head, optimized KV-cache management, and continuous batching improve image-generation efficiency.
- Base model and tokenization: The model initializes from open-sourced EMU3.5, built on the Qwen-3-32B decoder-only Transformer for image generation and image-text interleaved modeling.Images are tokenized with the IBQ Tokenizer using a spatial compression ratio of 16 × 16, while the Qwen vocabulary is extended with the IBQ codebook.
- Unified formulation: All generation tasks are formulated as a single autoregressive sequence-modeling problem instead of using task-specific prediction heads.The unified formulation supports image generation, embodied scene synthesis, scene transfer, sequential subtask prediction, and video generation while enabling multi-task co-training.
- Inference acceleration: 82.9× faster image generation at 1024×1024 resolution is achieved through anti-diagonal visual-token decoding, optimized KV-cache management, and vLLM continuous batching.An additional vertical prediction head generates multiple visual tokens simultaneously, following an inference-accelerating adaptation inspired by FlashAR.
2.3 Dataset
The dataset combines 9.5 million single-step samples and 2.6 million video clips across diverse visual and embodied domains. Unified annotations, task-specific organization, and domain-aware balancing provide geometric, semantic, and trajectory-level supervision for controllable embodied generation.
- Data Collection: 9.5 million samples for single-step generation correspond to 56.4 billion tokens, while 2.6 million sequential-generation video clips correspond to 49.6 billion tokens.
- Unified Annotation Pipeline: A unified annotation pipeline uses Qwen3-VL-235B to generate scene, embodied-structure, geometric, and trajectory-level supervision across heterogeneous data.Structured embodied annotations partition scenes into workspace, task objects, irrelevant objects, lighting, and background, while inverse depth maps provide geometric conditioning.
- Task-Oriented Data Organization: Annotated data is organized into embodied transfer, scene generation, and video generation formats with multi-view RGB, depth, initial-frame, and optional robot-mask supervision.Video samples use 15–25 RGB frames and can append robot mask images rendered from robot actions and URDF as action guidance.
- Data Balancing: Sample-level reweighting by domain and task type, combined with fine-grained filtering and grouping, reduces redundancy while prioritizing long-tail objects, skills, and interaction phases.Invalid embodied episodes are removed, and novel visuals, layouts, manipulation skills, and temporal stages receive preference during sampling.
2.4 Training
The model is adapted through single-step and sequential continued training to preserve foundation-model visual capabilities while learning embodied generation, transfer, reasoning, and interaction dynamics. Training combines four complementary one-step datasets with interleaved task sequences and multi-frame-rate manipulation videos.
- Training setups: Training uses single-step and sequential continued-training setups to adapt the pre-trained image generator into a unified embodied world model while preserving visual generation.The two setups target one-step generation and long-horizon embodied reasoning and interaction modeling, respectively.
- Single-step training: Single-step training jointly uses T2I, X2I, Embodied Scene Generation, and Embodied Transfer datasets.T2I and X2I reinforce visual generation and editing, while the embodied datasets teach plausible robot observations and cross-scene multi-view generation preserving interaction states and camera conditions.
- Sequential training: Interleaved language-described subtasks, subgoals, and post-execution multi-view observations train task decomposition and future embodied-observation prediction.This format further aligns sequence modeling with the base model’s inherited capabilities.
- Sequential training: Manipulation videos use multi-frame-rate sequences at 1, 3, and 5 FPS to improve dynamic interaction modeling.Sparse sequences encourage long-horizon planning and future prediction, whereas denser sequences capture fine-grained manipulation dynamics.
2.5 Inference
FlashAR+ accelerates Xiaomi-Robotics-U0 inference by applying parallel autoregressive decoding only to target image regions while preserving multi-source conditioning. Integrated vLLM further reduces latency without sacrificing embodied-task generation quality.
- FlashAR+ inference: FlashAR+ preserves multi-source conditioning and reduces serial decoding by parallelizing only the final target image region.Conditional inputs remain unchanged, while target-image tokens are generated in parallel anti-diagonal groups.
- FlashAR+ inference: Target tokens are generated in increasing r + c steps, with same-step tokens mutually invisible to prevent information leakage.Each target token attends to the full conditional prefix and only earlier anti-diagonal steps within the target image.
- Efficiency: 5.44 seconds per image with FlashAR+ and vLLM delivers an additional 3.04× speedup over FlashAR+ while preserving high generation quality on embodied tasks.The vLLM result uses max_num_seq=28.
- Efficiency: 450.77 seconds per image with standard AR falls to 16.56 seconds with FlashAR+, achieving a 27.22× speedup on 1024 × 1024 T2I generation.The comparison uses a single H20 GPU.
3 Experiments
Section 3 evaluates Xiaomi-Robotics-U0 across embodied transfer, multi-view scene and video generation, policy augmentation, and retained general image capabilities. The results emphasize geometric and physical consistency, improved robustness under distribution shifts, and competitive image generation and editing after embodied post-training.
- Overview: Xiaomi-Robotics-U0 unifies embodied transfer, scene generation, and video generation while retaining general image editing and generation capabilities.The embodied transfer method enhances robot trajectory diversity while preserving geometric consistency.
- Metrics: Generation quality is assessed against reference views using SI-RMSE, δ1, and AbsRel for depth consistency.These metrics measure how faithfully generated images respect the 3D geometry imposed by the input depth.
- Embodied Scene Generation: Xiaomi-Robotics-U0 generates coherent multi-view scenes with consistent object identity, appearance, arrangement, and spatial geometry across views.Compared with GPT-Image-2, it provides stronger cross-view geometric consistency while maintaining comparable instruction-following performance.
- Embodied Scene Generation: The model produces photorealistic, geometrically consistent multi-view initial scenes for diverse prompts and multiple robot arm embodiments.It extends single-view text-to-image synthesis to multi-view robotic observation generation by learning relations between arm configurations and camera projection.
- Policy Evaluation: Xiaomi-Robotics-U0-Aug consistently outperforms Original under interference conditions, while remaining comparable in the base group.Under heavily corrupted observations, the policies can still fail through grasping, orientation, localization, or object-identification errors.
- Embodied Video Generation: Embodied video generation is evaluated across 15 metrics spanning visual quality, motion quality, content consistency, physics adherence, 3D accuracy, and controllability.Examples show accurate contact modeling and material-dependent physical behavior during robot-object interactions.
- General Image Capabilities: After embodied post-training, Xiaomi-Robotics-U0 preserves strong general text-to-image and image-editing capabilities without catastrophic forgetting.Performance drops are concentrated in highly complex cases requiring multi-constraint composition, precise spatial grounding, or symbolic binding.
4 Related Work
Related work spans foundation generative models, embodied world models, controllable editing, multi-view synthesis, and video-based robot learning. Xiaomi-Robotics-U0 unifies these directions while addressing limitations in robot-centric data scale, embodiment-aware consistency, and independently modeled generation tasks.
- Embodied World Models: Embodied world models adapt foundation models to manipulation trajectories and robot videos for future-observation prediction, planning, simulation, or policy learning.Representative approaches include Dreamer Series, World Action Models, DreamZero, and Qwen-RobotWorld.
- Embodied World Models: Robot-only training datasets are substantially smaller and less diverse than foundation-model datasets, risking loss of inherited semantic richness, controllability, and visual diversity.The passage contrasts continual adaptation with the broader capabilities acquired during foundation pre-training.
- Controllable Editing and Multi-View Generation: Controllable editing methods provide instruction-guided synthesis and reference-conditioned editing, while multi-view methods enforce cross-view consistency but mainly target natural images or object-centric content.Representative editing methods include InstructPix2Pix, AnyDoor, OmniGen, GPT-Image-2.0, and FLUX Kontext; multi-view methods include MVDream, SyncDreamer, Wonder3D, Era3D, and Zero123++.
- Controllable Editing and Multi-View Generation: Xiaomi-Robotics-U0 unifies controllable image editing, multi-view embodied scene generation, and embodied transfer with structured controls for workspace layout, appearance, objects, targets, and lighting.The controls are disentangled into independent dimensions while preserving robot geometry and embodiment-related consistency.
- Video Generation and Robot Learning: Existing systems often generate isolated images, simulation assets, or short robot videos independently, whereas Xiaomi-Robotics-U0 jointly models scene generation, embodied transfer, subtask-subgoal prediction, and embodied video generation.The unified model also learns embodied videos at multiple temporal resolutions.
5 Conclusions
Xiaomi-Robotics-U0 is a unified embodied synthesis model built on a world foundation model through lightweight continual training, combining general-domain and embodied data. Its current limitations include depth-dependent embodied transfer and separate scene/video generation, motivating direct raw-observation editing and improved long-horizon rollout.
- Xiaomi-Robotics-U0 is built on a world foundation model through lightweight continual training on general-domain and embodied datasets.
- It unifies text-to-image generation, image editing, multi-view embodied scene generation, embodied transfer, and embodied video generation in one autoregressive framework.
- Embodied transfer relies on depth estimation as an intermediate representation, which can introduce artifacts and limit fine-grained control over textures and local appearances.
- Future work will investigate embodied multi-view image editing directly on raw observations, while separate scene and video generation makes long-horizon rollout susceptible to limitations.
Contributions
The contribution list distinguishes core contributors from an external contributor and notes that core contributors are alphabetically listed by first name.
- Contributions: Core contributors are listed alphabetically by first name, with an external contributor separately identified.The passage uses asterisks to mark core contributors and double asterisks to mark the external contributor.