PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

Jie Zhang, Xiaoyue Chen, Anzhe Chen, Dayiheng Liu, Deqing Li, Gengze Zhou, Hale Yin, Haoqi Yuan, Haoyang Li, Jiahao Li, Jiazhao Zhang, Jingren Zhou, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Pei Lin, Qihang Peng, Shengming Yin, Tianhe Wu, Tianyi Yan, Xiao Xu, Yan Shu, Yanran Zhang, Ye Wang, Yi Wang, Yilei Chen, Yixian Xu, Yiyang Huang, Yuxiang Chen, Zekai Zhang, Zhendong Wang, Zixing Lei, Zhixuan Liang, Zihao Liu, Zikai Zhou, Chenxu Lv, Xiong-Hui Chen, Chenfei Wu

arXiv:2606.17030v3cs.CV

TL;DR

Embodied world models need a scalable way to predict physically grounded futures across heterogeneous domains without relying on separate action interfaces. Qwen-RobotWorld unifies these domains through language-conditioned video generation and achieves strong cross-benchmark performance, including first-place results on EWMBench and DreamGen Bench.

  • Problem

    Embodied world modeling lacks a unified action representation across manipulation, driving, and navigation, despite the cost, inefficiency, and safety risks of real-world training.

  • Method

    Qwen-RobotWorld unifies heterogeneous embodied actions through language-conditioned video generation, combining a shared architecture, cross-embodiment data, and progressive general-to-expert training.

  • Results

    4.60 overall on EWMBench and 4.952 on DreamGen Bench, with first-place overall rankings and consistently strong performance across four embodied-world-model benchmarks.

  • Takeaways & Limitations

    The unified backbone supports synthetic data generation, policy evaluation, and action planning across embodied applications under a shared natural-language interface.

  • Takeaways & Limitations

    The model uses lower output resolution than general-purpose video generators, reducing pixel-level VBench quality scores.

Abstract

from arXiv · show

We introduce Qwen-RobotWorld, a language-conditioned video world model for embodied intelligence. With natural language as a unified action interface, it predicts physically grounded future visual trajectories from current observations across robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. This unified formulation provides three promising application directions: synthetic data generation for policy training augmentation, scalable virtual environments for policy evaluation, and language-guided planning signals for downstream robot control. This is achieved through a three-part design: a) Double-Stream MMDiT with MLLM Action Encoding, where a 60-layer double-stream diffusion transformer couples frozen Qwen2.5-VL semantics with video-VAE latents through layer-wise joint attention; b) Embodied World Knowledge (EWK), an 8.6M video-text corpus (200M+ frames) with action-language mapping over 20+ embodiments and 500+ action categories; and c) General+Expert Progressive Curriculum, a two-stage training strategy that first learns general visual priors and then injects embodied specialization under a shared language interface. Extensive results show strong competitiveness: ranks 1st overall on EWMBench and DreamGen Bench, outperforms all open-source models on WorldModelBench and PBench. Additional zero-shot analyses on RoboTwin-IF benchmark further support robust generalization and multi-view consistency.

1 Introduction

QWEN-ROBOTWORLD is a language-conditioned video world model that unifies embodied capabilities across scenarios through natural language actions. Its architecture, dataset, and progressive training support synthetic data generation, policy evaluation, action planning, and competitive benchmark performance.

  • Motivation: World models provide scalable, interactive alternatives to costly and risky real-world training for embodied agents across manipulation, driving, and indoor navigation.They learn environment dynamics from observational data and enable future visual trajectory prediction conditioned on current observations and actions.
  • Motivation: General video models lack embodied physics, while domain-specific models use scenario-bound action representations that hinder cross-domain generalization.The gap concerns contact dynamics, rigid-body constraints, and action-consequence relationships needed for physically plausible transitions.
  • Framework: QWEN-ROBOTWORLD uses natural language as a unified action interface to integrate complementary physical knowledge across manipulation, driving, navigation, and human-to-robot transfer.The model is designed as a unified backbone supporting synthetic data generation, policy evaluation, and action planning with task-specific adaptation.
  • Architecture: Its Double-Stream MMDiT couples frozen Qwen2.5-VL action semantics with video-VAE visual latents through layer-wise joint attention.The understanding stream represents the action, while the generation stream represents the visual state.
  • Data and Training: The EWK dataset contains approximately 8.6M video-text pairs and over 200M observation frames, while progressive training moves from general visual priors to embodied specialization.The corpus spans four embodied domains alongside general video data, and training jointly covers T2I, T2V, and TI2V before embodied SFT.
  • Evaluation: 1st overall on EWMBench (4.60), QWEN-ROBOTWORLD outperforms all open-source models on WorldModelBench (8.99, 3rd overall) and attains strong instruction following (2.33/3.0).It also achieves perfect physics adherence across Newton’s laws, mass conservation, fluid dynamics, and gravity, on par with leading closed-source models.

2 Data

The data challenge is representational heterogeneity across embodied domains, whose differing action formats require separate interfaces. An action-language mapping framework addresses this by converting heterogeneous actions across 20+ robot embodiment types into a unified representation.

  • Data: Representational heterogeneity—not data scale alone—is the central challenge, because manipulation, driving, and navigation use distinct action formats.Examples include joint angles or end-effector waypoints, steering commands and velocity profiles, and heading vectors.
  • Data: An action-language mapping framework converts heterogeneous actions across 20+ robot embodiment types into a shared interface.This addresses the need for separate models or interfaces across domains.

2.1 Action-Language Mapping

The framework projects fragmented embodied action signals into a shared natural-language interface, enabling one diffusion transformer to model visual transitions across physical domains. Hierarchical annotations, dual-granularity descriptions, quality controls, and broad embodiment coverage provide action-rich, verifiable training signals.

  • Unified Action Interface: Projecting all action signals into a shared natural-language space lets one diffusion transformer learn visual transitions across physical domains.The formulation predicts s_t+1 = f(s_t, a_t) regardless of the underlying embodiment or modality.
  • Unified Action Interface: Natural language encodes action sequences, goals, and physical constraints without requiring embodiment-specific kinematic knowledge.The framework is designed to generalize across embodiments by predicting the next visual state from a language action alone.
  • Hierarchical Annotation: 20+ robot embodiments and 500+ action categories are covered through a hierarchical five-layer annotation framework for action-rich captions.Coverage includes human hands, seven robot-arm configurations, ego vehicles, pedestrians or drones, and mobile navigation agents.
  • Description Granularity: Comprehensive descriptions contain 50–100 words, while concise descriptions contain 15–30 words for detailed specifications and brief inference-time commands.Both formats specify viewpoint and action information at different levels of granularity.
  • Quality Control and Training: Four quality principles enforce operation focus, viewpoint definition, objectivity, and physical verifiability, with comprehensive and concise descriptions sampled equally during training.The 50%–50% sampling teaches the model to handle detailed trajectory specifications and brief task-level commands.

2.2 Data Collection

The data collection framework builds a structured mixture spanning manipulation, autonomous driving, indoor navigation, and human-to-robot transfer. It combines multi-embodiment, multi-task, multi-scenario, and multi-view coverage with large-scale driving, physically grounded navigation, and automated cross-embodiment correspondence pipelines.

  • Manipulation: Manipulation data spans human hands, single-arm grippers, dual-arm dexterous systems, mobile manipulators, and full-body humanoids to separate task intent from embodiment-specific kinematics.Human manipulation recordings provide priors for dexterous hand–eye coordination.
  • Manipulation: Manipulation coverage combines atomic contact actions with extended procedures across short- and long-horizon datasets, grounding contact physics and object affordances at multiple temporal granularities.The corpus includes primitives such as grasping, pushing, and inserting.
  • Autonomous driving: 1,744,405 clips spanning 2,405 hours form the driving corpus, integrating real-world, simulated, egocentric, and drone videos through unified waypoint extraction, action segmentation, and captioning.The curated sources include Waymo E2E, NVIDIA PhysicalAI-AD, Bench2Drive, and Sekai.
  • Indoor navigation: 6,064 successful navigation episodes across 134 indoor scenes pair 256 × 256 RGB videos at 10 FPS with natural-language instructions and approximately 8.2 m traversals.The Isaac Sim data supports room-scale spatial understanding, depth reasoning, geometric consistency, and obstacle-aware planning.
  • Human-to-robot transfer: Human-to-robot transfer data retargets MANO-reconstructed hand keypoints to robot end-effectors, removes hands through inpainting, and renders 14 robot embodiments into shared scenes.Each episode provides aligned human, hand-removed, simulated, and robot-overlaid video streams.

2.3 Data Processing

The data-processing pipeline converts heterogeneous general and embodied videos into consistently formatted training samples through adaptive preprocessing, hierarchical annotation, and iterative quality control. It yields an approximately 8.6M-pair corpus spanning single-view, multi-view, navigation, and driving data.

  • Pipeline Overview: Four stages transform heterogeneous raw videos into training samples: collection, preprocessing, hierarchical annotation, and caption filtering with iterative prompt refinement.Preprocessing and annotation adapt to source-data characteristics.
  • Video Preprocessing: Five adaptive operations structure videos: frame extraction, interpolation, sub-task splitting, main-view selection, and multi-view concatenation.These operations capture complete atomic actions, preserve motion continuity, separate long-horizon procedures, remove redundant views, or align 2–4 synchronized viewpoints.
  • Hierarchical Annotation: The hierarchical annotation framework produces viewpoint-aware comprehensive captions of 50–100 words and concise captions of 15–30 words, sampled equally during training.The framework covers task goals, action details, and a comprehensive viewpoint–agent–action–feedback description before producing the concise command.
  • Quality Filtering: A closed-loop quality system combines LLM judging and human review, routing failed captions back for targeted re-annotation and prompt refinement.The judge checks factual accuracy, specificity, instruction clarity, and viewpoint consistency; refinement targets scenario, task, and embodiment categories.
  • Final Corpus: Approximately 8.6M video-text pairs and over 200M observation frames comprise the final corpus, with embodied data at 70% and general data at 30%.Single-view manipulation contributes ∼4.3M samples, multi-view concatenation ∼1.6M samples, and navigation and driving ∼200K samples.

3 Model

The model combines a frozen Qwen2.5-VL action encoder, Wan-VAE state encoder/decoder, and double-stream MMDiT transition function for language-conditioned video generation. SCENE2ROBOT extends this backbone with multi-segment conditioning for cross-embodiment synthesis without architectural modification.

  • Core Architecture: The architecture comprises an MLLM action encoder, VAE state encoder/decoder, and MMDiT transition function.The MLLM encodes actions, the VAE represents and reconstructs visual states, and MMDiT models transitions.
  • Core Architecture: Frozen Qwen2.5-VL maps input text S to last-layer hidden states h = ϕ(S), which serve as action-condition signals.The VAE uses Wan-VAE and supports both image and video modalities.
  • Core Architecture: 60 double-stream MMDiT blocks couple projected MLLM states with noisy VAE latents through joint attention at every block.The backbone uses 24 attention heads, head dimension 128, hidden size 3,072, and patch size 2×2.
  • Positional Encoding: Asymmetric 3D RoPE allocates 16 dimensions temporally and 56 dimensions to each spatial axis, totaling 128 dimensions.The temporal axis receives fewer dimensions because adjacent frames are strongly correlated, while spatial axes capture greater diversity.
  • SCENE2ROBOT: SCENE2ROBOT repurposes the same VAE–MMDiT backbone for cross-embodiment synthesis using scene-condition, robot-reference, and generation segments.The scene video supplies appearance and object state, the MuJoCo rendering supplies robot motion and morphology, and the generation segment is denoised into the final video.

4 Training

Training unifies general scene generation and robot manipulation prediction as conditional video generation under one natural-language interface, with shared optimization across data regimes. It combines broad world and human-action priors with joint multimodal training, progressive embodied-data mixing, and flow matching.

  • Joint Training: A shared natural-language conditional video-generation task jointly trains general scene generation and robot manipulation prediction, allowing both data regimes to update one backbone.The paradigm is designed for stable cross-scenario and cross-task co-training.
  • General World Priors: Over 200M real-world observation samples from 14 high-quality video platforms provide general priors for object motion, lighting variation, and collision dynamics.Multi-camera synchronized observations with 3D RoPE spatial encoding add preliminary cross-view geometric consistency.
  • Human Interaction Priors: First-person hand-manipulation data teaches grasping, tool use, and object manipulation, transferring human action priors and affordance understanding to later robot operation.The data includes Ego4D and EPIC-Kitchen.
  • Multi-Task Joint Training: T2I, T2V, and TI2V are jointly trained on a shared backbone, with T2I providing sharp visual representations that transfer to video generation.Task ratios gradually shift from pure T2I toward full three-task joint training to support multiple generation modes.
  • Progressive Embodied Knowledge Injection: A four-phase data-mixing schedule combines multi-embodiment robot and human manipulation data early, then increases wrist-view and third-person-view coverage.Human actions guide cross-embodiment commonalities while robot data strengthens concrete execution representations.
  • Training Objective: Flow matching trains VAE-encoded videos with standard-normal noise, Qwen2.5-VL text guidance, log-normal timesteps, and adaptive shifting by video sequence length.For TI2V, the first-frame timestep is fixed at 0.

5 Experiments · 5.1 Quantitative Evaluation

Quantitative evaluation shows Qwen-RobotWorld achieves leading results across embodied world-modeling, video-generation, physical-behavior, and physics-reasoning benchmarks. Its strongest results are paired with limitations in long-horizon behavior generalization and pixel-level visual quality at lower output resolution.

  • 5 Experiments: The experiments span embodied manipulation, physical reasoning, and general video quality, with qualitative analysis and cross-domain tests covering broader generalization.Qualitative evaluation examines language grounding, embodiment/task/viewpoint generalization, and zero-shot robustness; supplementary tasks include human-to-robot transfer, autonomous driving, and indoor navigation.
  • 5.1 Quantitative Evaluation: Quantitative comparisons include general video generators and embodied world models, with boldface and underlining marking the best and second-best values.General-video baselines include Sora2, Veo3, Wan2.6, Kling, and LTX-2; embodied baselines include Cosmos, WoW, LVP, Vidar, and GigaWorld.
  • 5.1.1 EWMBench: Embodied Motion Fidelity: 4.60 overall ranks Qwen-RobotWorld first on EWMBench, exceeding LVP’s 4.05 by +0.55.The model also reaches HSD 0.566 versus LVP’s 0.425, SceneC 0.914, and Logics 1.00.
  • 5.1.2 DreamGen Bench: 4.952 total ranks Qwen-RobotWorld first on DreamGen Bench, including first-place GR1-Object IF of 0.878.Physics alignment remains 0.828/0.840/0.781 across subsets, while GR1-Behavior IF is 0.832 versus LVP’s 0.889 and GigaWorld’s 0.884.
  • 5.1.3 PBench: Physical Behavior Evaluation: 0.804 overall outperforms all open-source models on PBench, with Domain Score 0.857 and motion smoothness 0.990.Aesthetic quality 0.455 and imaging quality 0.649 are lower because the embodied-task model uses lower output resolution, though that resolution suffices for robot control.
  • 5.1.4 WorldModelBench: Physical Reasoning and Instruction Following: 8.99 ranks Qwen-RobotWorld first among open-source models and third overall on WorldModelBench.It achieves perfect physics adherence of 1.00 across four categories and instruction following of 2.33/3.0, while lower resolution contributes to the common-sense gap.

5.2 Qualitative Analysis

QWEN-ROBOTWORLD qualitatively grounds language in fine-grained visual actions, generalizes across embodiments, tasks, environments, and synchronized viewpoints, and preserves coherent zero-shot execution against strong baselines and on RoboTwin-IF.

  • Cross-embodiment generalization: One instruction drives four robot morphologies without embodiment-specific adaptation, supporting natural language as a universal action interface.The morphologies are a single-arm gripper, dual-arm system, humanoid, and dexterous hand.
  • Fine-grained language grounding: Figure 5 shows that changing target object, destination, or action keywords produces precisely differentiated motions from identical initial frames.The figure also includes complex examples requiring multi-step execution or abstract goal inference.
  • Cross-task and multi-view consistency: Generations across fruit pick-and-place, bowl retrieval, cloth folding, and human–robot handover exhibit task-appropriate contact dynamics, while synchronized camera streams remain consistent.The multi-view setting jointly generates main, wrist-left, and wrist-right camera streams from the same episode.
  • Zero-shot baseline comparison: QWEN-ROBOTWORLD more consistently preserves language-grounded execution and coherent multi-view trajectories than LVP and Cosmos2.5-14B in zero-shot Unitree G1 comparisons.LVP more often produces incomplete execution, whereas Cosmos2.5-14B shows weaker instruction alignment in complex manipulation cases.
  • RoboTwin-IF zero-shot evaluation: Despite mixing only a small amount of open-source RoboTwin data during training, QWEN-ROBOTWORLD shows strong zero-shot RoboTwin-IF performance with stable multi-view consistency.Representative cases on newly constructed complex tasks preserve coherent execution and cross-view consistency across ten uniformly sampled frames.

5.3 Cross-Domain Generalization

The model generalizes beyond manipulation to human-to-robot transfer across eight embodiments and mobility scenarios spanning autonomous driving and language-guided indoor navigation.

  • Human-to-robot transfer: Across eight target embodiments, generated trajectories preserve human-demonstrated task intent while adapting motion to embodiment-specific kinematic constraints.Figure 9 compares human demonstrations with synthesized robot executions using five uniformly sampled frames per video.
  • Mobility generation: Mobility generation covers autonomous driving episodes from Bench2Drive, NVIDIA PhysicalAI-AD, Sekai, and Waymo, alongside language-guided egocentric indoor navigation from VLNVerse.Each episode uses five uniformly sampled frames.

6 Conclusion

QWEN-ROBOTWORLD unifies multiple embodied-intelligence domains under a shared natural-language action interface through a three-part world-model system. Its common backbone supports synthetic data generation, policy evaluation, and action planning, while benchmark and zero-shot results show strong performance and robust multi-view instruction-following generalization.

  • Unified framework: QWEN-ROBOTWORLD unifies robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer under a shared natural-language action interface.The framework is presented as a language-conditioned world model for embodied intelligence.
  • System design: The system combines double-stream MMDiT with MLLM action encoding, Embodied World Knowledge, and a third component in its unified design.The passage explicitly identifies the double-stream MMDiT and MLLM action encoding components, while the Embodied World Knowledge component name is truncated in the supplied text.
  • Applications: A common backbone supports synthetic data generation, policy evaluation, and action planning as three representative embodied world model applications.These applications are described as adaptation targets for the shared backbone.
  • Results: Across benchmark evaluations and zero-shot analyses, QWEN-ROBOTWORLD demonstrates strong, consistent performance and robust multi-view instruction-following generalization.The conclusion characterizes the results across both evaluation settings without reporting additional numerical values.

Authors

The paper lists 39 authors, with equal contribution marked by an asterisk and corresponding authors marked by a dagger.

  • Authors: 39 authors are listed, including Jie Zhang and Xiaoyue Chen as the first two authors.The author list contains 39 names in total.
  • Authors: * denotes equal contribution, while † denotes corresponding author.These symbols appear beside author names in the author list.
Loading 2606.17030v3…