Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

Motivation

• Language-conditioned world models address the gap between general video models that lack embodied physics and domain-specific models with robot-specific interfaces.Manipulation, driving, and indoor navigation contribute complementary physical knowledge, while natural language enables their joint training; video editing of human demonstrations offers a path to scale robot training data.Source: §1 · pp. 1–2

Framework

• QWEN-ROBOTWORLD predicts future visual trajectories from observations and language actions, and can support synthetic-data generation, policy evaluation, and action planning.The model is presented as a unified backbone for these applications with task-specific adaptation.Source: §1 · pp. 1–2

Architecture

• A double-stream MMDiT uses frozen Qwen2.5-VL action semantics and video-VAE visual latents, with joint attention coupling both streams at every layer.The MLLM encoder parses compositional instructions and supplies world knowledge that can constrain plausible transitions and reduce object deformation when combined with T2I co-training.Source: §1 · p. 2

Data

• The EWK dataset contains approximately 8.6M video-text pairs and over 200M frames, mapping actions across 20+ robot embodiments and 500+ categories into natural language.It spans manipulation, driving, indoor navigation, and human-to-robot transfer, with task-aware temporal segmentation and viewpoint-aware annotations.Source: §1 · p. 2

Training

• A two-stage curriculum first learns general visual priors through T2I, T2V, and TI2V pretraining, then progressively introduces embodied data during SFT.SFT uses a four-phase schedule from single-view manipulation through multi-view and complex cross-domain tasks; asymmetric 3D RoPE and multi-view concatenation support geometrically consistent synchronized views.Source: §1 · pp. 2–3

Evaluation

• 1st overall on EWMBench (4.60) and DreamGen Bench (4.952), while outperforming all open-source models on WorldModelBench (8.99, 3rd overall) and PBench (0.804).On EWMBench, HSD motion fidelity is 0.566 (+33% over the runner-up) and scene consistency is 0.914; WorldModelBench instruction following is 2.33/3.0, and PBench domain understanding is 0.857 while motion smoothness is 0.990. Zero-shot RoboTwin-IF results further support transfer under complex instructions.Source: §1 · p. 3

2 Data

• The framework maps heterogeneous actions from 20+ robot embodiment types and 500+ action categories into one natural-language interface for cross-scenario, cross-task joint training.It enables a Franka gripper, an autonomous vehicle, and an indoor navigation agent to share a single language-conditioned video-generation task without domain-specific control interfaces.Source: §2 · p. 3
• Approximately 6M high-quality embodied video–text pairs are augmented with general video data, which makes up 30% of the total, to construct EWK.The resulting corpus contains 8.6M video–text pairs and over 200M observation frames.Source: §2 · p. 3
• The data challenge is representational heterogeneity, with manipulation, driving, and navigation actions expressed in distinct forms rather than scale alone.Examples include joint angles or end-effector waypoints, steering commands and velocity profiles, and heading vectors.Source: §2 · p. 3

2.1 Action-Language Mapping

• Projecting fragmented action signals into shared natural language lets one diffusion transformer model next-state transitions across physical domains.The visual states already share pixel space, while action representations vary across modalities.Source: §2.1 · p. 3
• Language commands encode action sequences, goals, and physical constraints without robot-specific kinematics, but captions must support next-state prediction from video and language alone.The framework is intended to generalize across embodiments without robot-specific interface re-engineering or access to robot metadata or proprioception.Source: §2.1 · pp. 3–4
• A five-layer annotation pipeline decomposes transitions into task goal, action detail, physical feedback, and comprehensive and concise descriptions.Comprehensive descriptions (50–100 words) specify viewpoint–agent–action–feedback; concise descriptions (15–30 words) retain viewpoint, agent, and key action for brief commands.Source: §2.1 · p. 4
• Four quality controls enforce operation focus, explicit viewpoint, visible dynamics, and physically verifiable outcomes; comprehensive and concise descriptions are sampled equally in training.Source: §2.1 · p. 5
• The framework covers 20+ robot embodiments and 500+ action categories, spanning manipulation, long-horizon compositions, locomotion and navigation, and dynamic or deformable interactions.Agibot-World alone defines 84 manipulation primitives. EWK data also spans embodiment, task, scenario, and multi-view variation, alongside general video and image priors.Source: Figure 1 · p. 4, §2.1 · p. 5

2.2.1 General Data

• Internet videos and high-quality stills provide general visual data; natural-language annotations omit viewpoint-specific details, and AI-generated content is excluded.Videos are standardized to 24 FPS while supporting multiple resolutions and aspect ratios.Source: §2.2.1 · p. 6

2.2.2 Embodied Manipulation Data

• Manipulation data spans human and robot embodiments and short-horizon primitives through long-horizon procedures, supporting embodiment-invariant action semantics and prediction across temporal scales.Dynamic whole-body interactions add high-velocity motions with momentum and balance constraints.Source: §2.2.2 · p. 6
• Real-world and photorealistic simulated manipulation scenes broaden environmental coverage, while synchronized 2–4-view recordings provide geometric supervision and support viewpoint-specific or mutually consistent multi-view generation.Approximately 1.6M of 6M embodied samples include synchronized views; simulation also enables controlled variations in lighting, object pose, and camera placement.Source: §2.2.2 · pp. 6–7

2.2.3 Autonomous Driving Data

• 1,744,405 driving clips spanning 2,405 hours combine multi-view real and simulated driving with egocentric pedestrian and drone footage, processed through waypoint unification, 2–8-second action segmentation, and captioning.The data covers varied maneuvers and multi-agent traffic dynamics, with camera displacement providing supervision for 3D geometry.Source: §2.2.3 · p. 7

2.2.4 Egocentric Indoor Navigation Data

• 6,064 successful egocentric navigation episodes across 134 indoor scenes pair continuous RGB video with natural-language instructions in two formats, supporting room-scale, obstacle-aware trajectory learning.Episodes average 8.2 m, totaling roughly 49.8 km and 5.8 hours of video; the paired traversal data is intended to support spatial reasoning and action prediction.Source: §2.2.4 · p. 7

2.2.5 Human-to-Robot Transfer Data

• Human-to-robot transfer data retargets hand-keypoint demonstrations to 14 robot embodiments and pairs simplified MuJoCo renders with photorealistic views across four robot configurations and approximately 80K episodes.The paired renderings share geometry and viewpoint but differ in lighting and shadows, enabling learning of the mapping between simplified and photorealistic observations.Source: §2.2.5 · pp. 7–8

2.3.1 Stage 1: Raw Data Collection

• Raw data spans five categories: general video, manipulation, autonomous driving, indoor navigation, and paired human demonstrations with robot executions.Sources include VLNVerse episodes across 134 indoor scenes and human-to-robot transfer data across 14 robot types.Source: §2.3.1 · p. 8

2.3.2 Stage 2: Video Preprocessing

• Domain-adaptive preprocessing uses frame extraction, interpolation, sub-task splitting, main-view selection, or synchronized multi-view concatenation to structure clips for training.Concatenating 2–4 camera views preserves temporal alignment for learning cross-view geometric consistency and synchronized state transitions.Source: §2.3.2 · p. 9

2.3.3 Stage 3: Hierarchical Annotation

• The five-layer annotation framework captures task goals, action details, physical feedback, and viewpoint-aware comprehensive and concise captions.Comprehensive captions are 50–100 words, concise captions are 15–30 words, and both are sampled with equal probability during training.Source: §2.3.3 · p. 9

2.3.4 Stage 4: Caption Quality Filtering

• Caption filtering combines an LLM judge and human review, routing failures to targeted prompt redesign and re-evaluation until quality standards are met.The judge checks factual accuracy, specificity, instruction clarity, and viewpoint consistency; prompt retries target scenario, task, or embodiment failure modes.Source: §2.3.4 · pp. 9–10
• The final corpus contains approximately 8.6M video-text pairs and over 200M observation frames, with embodied data at 70% and general data at 30%.Single-view manipulation contributes ∼4.3M samples, multi-view concatenated data ∼1.6M, and navigation and driving ∼200K.Source: §2.3.4 · p. 10

3.1 Model Architecture

• A frozen Qwen2.5-VL action encoder, video VAE state encoder/decoder, and 60-block double-stream MMDiT transition function form the model.The streams interact through joint attention at each block; the MMDiT has 24 attention heads and 20B parameters, while the MLLM has 7B parameters and the VAE 127M. The model supports up to 48,360 video tokens. The components use Qwen2.5-VL (Bai et al., 2025), Wan-VAE (Wan et al., 2025), and MMDiT.Source: §3.1 · pp. 10–11, Figure 3 · p. 10

3.2 3D Rotary Position Encoding

• Asymmetric 3D RoPE assigns 16 dimensions to time and 56 each to height and width, with Scalable RoPE supporting varied inference resolutions and durations.The temporal axis receives fewer dimensions because adjacent frames are strongly correlated, while spatial axes receive more to represent diverse object positions and scene layouts (Heo et al., 2024; Wan et al., 2025).Source: §3.2 · p. 11

3.3 Scene2Robot

• Scene2Robot adapts the same VAE–MMDiT backbone without architectural changes to synthesize robot videos conditioned on a human demonstration scene and simulated robot trajectory.The three F-frame segments provide scene appearance and object state, robot morphology and motion, and generation latents; the first two are fixed at timestep t=0 and excluded from loss, while generation alone is trained. Joint attention combines both visual conditions with MLLM action semantics, enabling scene-preserving executions of the instructed manipulation behavior.Source: §3.3 · p. 12

4.1 Training Strategy

• The curriculum co-trains general scene generation and robot manipulation prediction as one conditional video task, allowing general world and embodied action priors to reinforce each other through a shared backbone.Pretraining establishes broad world foundations; SFT deepens embodied specialization while preserving the general-expert balance.Source: §4.1 · p. 12

4.1.1 Pretraining Stage: Establishing General World Foundation

• Pretraining combines 200M+ real-world samples, human manipulation demonstrations, synchronized multi-camera inputs, and joint T2I/T2V/TI2V training to establish general priors and visual consistency.Human data include Ego4D (Grauman et al. (2022)) and EPIC-Kitchen (Damen et al. (2018)); task ratios shift from T2I toward full three-task joint training.Source: §4.1.1 · pp. 12–13

4.1.2 SFT Stage: Embodied Specialization

• SFT uses four phases to combine human and robot action priors, expand viewpoints, train synchronized multi-view generation, and supplement complex or long-horizon tasks while retaining general-world data in every batch.Multi-view concatenation trains the model to generate subsequent frames for all camera views jointly, encouraging cross-view spatial correspondences and geometric consistency.Source: §4.1.2 · p. 13

4.2 Training Objective and Infrastructure

• Training uses flow matching with VAE video latents, Qwen2.5-VL text guidance, and a zero timestep for the conditioned first frame in TI2V.Timesteps follow an adaptively shifted log-normal distribution based on video length; Megatron-LM hybrid parallelism and selective activation recomputation balance memory use and throughput.Source: §4.2 · p. 13

Quantitative Evaluation

• Overall 4.60 on EWMBench is +0.55 over LVP; Qwen-RobotWorld ranks 1st overall on DreamGen Bench (Total 4.952) and 1st among open-source models on WorldModelBench (Total 8.99).The evaluations span embodied manipulation, physical reasoning, and general video quality.Source: §5 · p. 13
• The quantitative evaluation compares general video generation models with embodied world models.The general-video baselines include Sora2, Veo3, Wan2.6, Kling, and LTX-2; embodied-world-model baselines include Cosmos, WoW, LVP, Vidar, and GigaWorld.Source: §5 · p. 13

Qualitative Analysis

• Manipulation analysis evaluates fine-grained language grounding, generalization across embodiments, tasks, and viewpoints, and zero-shot robustness against strong baselines.These are the three progressive dimensions used to assess manipulation capabilities.Source: §5 · p. 13

Cross-Domain Generalization

• Supplementary cross-domain tests cover human-to-robot transfer, autonomous driving, and indoor navigation.Source: §5 · p. 13

5.1.1 EWMBench: Embodied Motion Fidelity

• 4.60 overall ranks 1st on EWMBench, beating LVP’s 4.05 by 0.55, with HSD 0.566, SceneC 0.914, and Logics 1.00.HSD surpasses LVP by 33%.Source: Table 2 · p. 14, §5.1.1 · p. 14

5.1.2 DreamGen Bench

• 4.952 total ranks 1st on DreamGen Bench; GR1-Object IF is 0.878, while GR1-Behavior IF (0.832) trails LVP (0.889) and GigaWorld (0.884).Physics alignment across the three subsets is 0.828/0.840/0.781.Source: Table 3 · p. 14, §5.1.2 · p. 14

5.1.3 PBench: Physical Behavior Evaluation

• 0.804 Overall Score beats all open-source PBench models; Domain Score is 0.857 (3rd overall), and Motion Smoothness is 0.990 (2nd among open-source models).Aesthetic quality (0.455) and imaging quality (0.649) are lower, primarily because the model is purpose-built for embodied tasks and uses lower output resolution; that resolution is sufficient for downstream robot control.Source: §5.1.3 · pp. 14–15, Table 4 · p. 15

5.1.4 WorldModelBench: Physical Reasoning and Instruction Following

• 8.99 ranks 3rd overall on WorldModelBench, ahead of all open-source models and behind only Wan2.6 and Veo3; Physics Adherence is 1.00 across all four categories.Instruction Following is 2.33/3.0; the common-sense gap is attributed to lower output resolution.Source: Table 5 · p. 15, §5.1.4 · p. 15

5.2.1 Fine-Grained Language Grounding

• With identical initial frames, changing a keyword elicits distinct motions grounded in target identity, destination, or action, while complex instructions yield temporally coherent multi-step execution.The examples also show abstract goal inference without explicit sub-task prompts.Source: §5.2.1 · p. 15, Figure 5 · p. 16

5.2.2 Generalization across Embodiments, Tasks, and Viewpoints

• One instruction drives four robot morphologies, while diverse tasks show appropriate contact dynamics and synchronized camera streams retain consistent object identities and motion trajectories.The examples span single-arm, dual-arm, humanoid, and dexterous-hand embodiments; tasks include fruit pick-and-place, bowl retrieval, cloth folding, and human–robot handover.Source: §5.2.2 · pp. 15–16, Figure 6 · p. 17

5.2.3 Zero-Shot Robustness on RoboTwin-IF

• On four Unitree G1 tasks, QWEN-ROBOTWORLD more consistently preserves language-grounded execution and coherent multi-view trajectories than LVP and Cosmos2.5-14B.LVP more often produces incomplete execution, while Cosmos2.5-14B shows weaker instruction-to-manipulation alignment in complex cases. RoboTwin-IF tests newly constructed complex tasks; despite using only a small amount of open-source RoboTwin training data, the model shows strong zero-shot performance and stable synchronized views. Figure 8 shows representative cases with ten uniformly sampled frames per task.Source: §5.2.3 · pp. 16–18, Figure 7 · p. 18, Figure 8 · p. 19

5.3 Cross-Domain Generalization

• Across eight target embodiments, human-to-robot transfer preserves task intent while adapting motion to embodiment-specific kinematic constraints.Figure 9 compares human demonstrations with synthesized robot executions for the same tasks.Source: §5.3 · p. 19, Figure 9 · p. 20
• Mobility generation covers autonomous driving episodes from Bench2Drive, NVIDIA PhysicalAI-AD, Sekai, and Waymo, plus egocentric indoor navigation from VLNVerse.Together, these scenarios indicate generalization beyond a single embodiment or scenario family.Source: §5.3 · p. 19, Figure 10 · p. 20

• QWEN-ROBOTWORLD unifies robotic manipulation, autonomous driving, indoor navigation, and human-to-robot transfer through a shared natural-language action interface.Source: §6 · p. 20
• Its three-part system combines double-stream MMDiT with MLLM action encoding, the cross-embodiment Embodied World Knowledge dataset, and a general-plus-expert progressive curriculum.The design couples broad visual priors with embodied specialization and is intended to enable synthetic data generation, policy evaluation, and action planning.Source: §6 · pp. 20–21
• Across benchmark evaluations and zero-shot analyses, QWEN-ROBOTWORLD shows strong, consistent performance and robust multi-view instruction-following generalization.Source: §6 · p. 21