Source-linked AI summary
DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation
DreamX Team, Rui Chen, Xiangxiang Chu, Geng Li, Jifan Li, Qingfeng Shi, Datao Tang, Jing Tang, Jun Wang, Pengfei Zhang
TL;DR
Faithful action-conditioned video prediction for robotic manipulation remains difficult because realistic rollouts may deviate from commanded motions or lose object consistency. DreamX-Phi 1.0 combines geometry-aware action conditioning with depth and object-centric supervision, achieving top WorldArena 2.0 performance across both tracks.
Problem
Realistic video generators may produce rollouts that deviate from commanded robot motions or fail to preserve manipulated-object interactions and scene geometry.
Method
DreamX-Phi combines arm-specific SE(3) conditioning, image-space motion cues, auxiliary depth supervision, and object-centric supervision for action-conditioned video prediction.
Results
The model ranked first on WorldArena 2.0 Track 1 with an EWMScore-P of 60.65 and tied for second on Track 2 with 67.19% success.
Takeaways & Limitations
DreamX-Phi demonstrates strong system-level performance for video prediction and world-model-based policy training.
Takeaways & Limitations
Evaluation is limited to WorldArena and RoboTwin, leaving generalization to other tasks, embodiments, and real robots unverified.
Abstract
from arXiv · showhide
We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.
1 Introduction
DreamX-Phi 1.0 is a geometry-aware action-conditioned video world model that predicts future observations from a single frame and prescribed bimanual actions. It preserves robot motion structure and enforces scene and manipulated-object consistency for faithful rollouts, achieving leading WorldArena rankings.
- Motivation: World models enable scalable pre-execution action evaluation by predicting future scene evolution from current observations and proposed action sequences, but photorealism alone does not ensure action faithfulness.Faithful predictions must preserve prescribed robot trajectories, object interactions, and action-irrelevant scene content.
- Model overview: DreamX-Phi predicts future observations from a single frame and prescribed bimanual actions using a geometry-aware, action-conditioned video world model.The model is based on Wan2.2-TI2V-5B.
- Training data: The training corpus combines egocentric video with robot interaction data from real and simulated environments, using action-agnostic clips for visual priors and aligned trajectories for control grounding.RoboTwin training videos are processed with DreamX-Refiner to improve resolution and visual quality.
- Geometry-aware action representation: Its action representation combines structured SE(3) trajectories with image-space motion cues, encoding commanded 3D end-effector motion and where its effects should appear.PRoPE-based attention preserves end-effector motion structure and relates it to corresponding image dynamics.
- Manipulation-aware supervision: Manipulation-aware supervision emphasizes depth-based scene geometry and object evolution, encouraging physically coherent robot–object interactions and consistent manipulated-object identity over time.The objective regularizes scene geometry and object-centric temporal consistency to support action-faithful rollouts.
- Results: WorldArena 2.0 submissions rank first on Track 1 and tie for second on Track 2, while WorldArena 1.0 Track 1 reaches an offline EWMScore-P of 76.88.These results are reported for the fixed WorldArena 2.0 snapshot and offline WorldArena 1.0 evaluation.
2 Related Work
Prior work develops action-conditioned video world models from scalable visual priors and external control signals, while robotic world action models connect video and control through action injection or joint policy-oriented modeling. Geometry-aware attention and depth supervision further target physical consistency, although geometry alone does not constrain the full scene response.
- Action-Conditioned Video World Models: Action-conditioned video world models predict scene evolution under external controls, extending broad-domain visual dynamics and pretrained video diffusion to learned action interfaces.Large generators such as Wan and Cosmos 3 provide scalable visual priors, while UniSim, iVideoGPT, DreamX-World, and AVID study controllable dynamics across domains.
- Action-Conditioned Video World Models: Robot-specific models make action conditioning concrete by aligning robot trajectories with video frames and introducing causal or interactive generation mechanisms.
- World Action Models for Robotics: World action models connect video and control through low-dimensional action tokens or adapters, or by jointly modeling visual futures and actions for policy or planning.The surveyed examples include IRASim, Vid2World, HMA, UVA, WorldVLA, LingBot-VA, DreamZero, and Cosmos Policy.
- Structured Control and Physical-Consistency Supervision: Geometry-aware attention injects relative SE(3) transformations or known camera geometry into attention, while depth supervision organizes scene geometry; geometry alone constrains robot motion rather than the full scene response.GTA uses relative SE(3) transformations, and projective relative positional encoding applies camera geometry to queries, keys, values, and outputs.
3 Data Curation
The curated corpus combines complementary visual and robotic data sources, then filters and organizes them to preserve manipulation-relevant dynamics and align observations with actions. Separate action-agnostic and action-conditioned pools support single-/multi-view pretraining and synchronized control-conditioned fine-tuning.
- Data Sources: The corpus combines action-free egocentric video, real-robot demonstrations, and simulated robot trajectories to cover visual dynamics, executed manipulation, and controlled task and scene variation.These sources provide complementary coverage for action-conditioned prediction.
- Curation and Normalization: Curation removes mobile-base motion, dexterous-hand behavior, and stationary segments while retaining failed executions as informative failure modes and non-ideal interaction dynamics.The filtered AgiBot imitation-learning split contains 178.7 ... after mobile-base and stationary segments are removed.
- Action-Agnostic Pretraining: The action-agnostic pool retains every curated video, keeping single-camera streams single-view and spatially concatenating synchronized multi-camera streams into unified multi-view videos.This view-adaptive organization preserves source information for both single-view and multi-view downstream use.
- Action-Conditioned Fine-Tuning: The action-conditioned pool includes only videos with synchronized action annotations, pairing each video with its robot trajectory and common-representation annotations for temporal visual-control alignment.Its RoboTwin component contains 25,000 bimanual clips spanning clean and randomized variants, which are refined before entering the curated pool.
4 Method
DreamX-Phi 1.0 conditions a Wan2.2-TI2V-5B video diffusion transformer on observations, language, and bimanual actions, using structured geometric attention to preserve arm-specific rigid motion. Auxiliary depth, object-aware supervision, and DMD2 distillation strengthen geometry and manipulated-object consistency while enabling few-step inference.
- Framework: The model predicts future RGB frames from an observed frame, language instruction, and prescribed bimanual end-effector and gripper trajectories using a Wan2.2-TI2V-5B diffusion transformer.Future-frame latents are trained with a flow-matching objective, while the first-frame latent supplies visual context.
- Geometric Action Conditioning: PRoPE-style attention injects per-arm relative SE(3) transforms and separate gripper states, preserving arm identity and coupling tokens through relative motion rather than absolute coordinates.Arms share a reference frame, occupy fixed attention-head groups, and use a residual branch whose adapters are initialized to zero.
- Depth Adaptation: A lightweight auxiliary depth branch supplies scene-geometry supervision through shared early layers, replicated late blocks, RGB-to-depth cross-attention, and latent-space mean-squared error.The one-way branch leaves RGB computation unchanged, so depth prediction remains optional during inference.
- Object-Aware Supervision: SAM3 masks reweight object tokens while a frozen V-JEPA teacher regularizes their spatiotemporal evolution, emphasizing contact-local errors and coupling object motion to arm contact.Masks are used offline for supervision only; no mask is required at inference, and mean-weight normalization stabilizes loss scale as mask area changes.
- Distillation: DMD2 distills the multi-step conditional generator into an N-step student for reduced denoising evaluations while retaining the observed frame, aligned bimanual actions, and language as conditions.The student predicts the clean future-video latent at sampled denoising steps.
5 Evaluation
DreamX-Phi 1.0 is evaluated for both visual rollout fidelity and usefulness as a learned environment for policy training, primarily on WorldArena 2.0. It ranks first on Track 1 and ties for second on Track 2 in the August 12, 2026 snapshot, while also reporting offline WorldArena 1.0 results.
- Evaluation protocol: WorldArena 2.0 Track 1 contains 1,000 episodes testing rollouts conditioned on language instructions or robot action trajectories, while Track 2 tests policy success after world-model interaction.Track 2 optimizes a π0.5 policy with organizer-provided initialization and a fixed reward model, then evaluates it on held-out Adjust Bottle episodes.
- Metrics: Track 1 reports EWMScore-P and 15 normalized metrics spanning visual quality, temporal dynamics, consistency, physical interaction, 3D structure, and conditioning fidelity.EWMScore-P averages the component scores, with WorldArena 2.0 capping Dynamic Degree, Flow Score, and Motion Smoothness by ground-truth references before aggregation.
- WorldArena 2.0: 60.65 EWMScore-P ranks DreamX-Phi first on WorldArena 2.0 Track 1, while 67.19% Adjust Bottle success ties for second on Track 2.These rankings come from the complete August 12, 2026 snapshot; WOVR-PLUS leads Track 2, while Lute shares second place.
- Qualitative evaluation: Qualitative Track 1 rollouts preserve arm, gripper, and manipulated-object coherence in standard and domain-randomized RoboTwin 2.0 scenes.The randomized settings vary backgrounds, textures, lighting, and distractor layouts.
- WorldArena 1.0: 76.88 EWMScore-P places DreamX-Phi 3.24 points above UNIS, the leading official entry, in the offline WorldArena 1.0 Track 1 comparison.The reported aggregate is retained despite a recomputation from displayed component values yielding 76.89 after rounding.
6 Limitations
The evaluation covers only WorldArena and RoboTwin, with Track 2 restricted to Adjust Bottle, leaving generalization to other tasks, embodiments, and real robots unverified. The full-system leaderboard scores do not isolate component contributions, and DreamX-Phi predicts externally supplied actions rather than generating them.
- Evaluation scope: Evaluation is limited to WorldArena and RoboTwin, with Track 2 covering only Adjust Bottle, so generalization remains unverified.The unverified scope includes other tasks, embodiments, and real robots.
- Attribution: Leaderboard scores evaluate the full system and therefore do not isolate the contributions of individual components.
- Action generation: DreamX-Phi predicts videos from externally provided actions rather than generating actions itself.
7 Conclusion
DreamX-Phi 1.0 addresses bimanual video prediction by conditioning each arm’s motion while preserving scene geometry and manipulated-object state. It combines arm-specific SE(3) conditioning with auxiliary depth and object-centric supervision, ranking first among 31 Track 1 entries with an EWMScore-P of 60.65 and supporting a Track 2 policy achieving 67.19% success on Adjust Bottle.
- Conclusion: DreamX-Phi combines arm-specific SE(3) conditioning with auxiliary depth and object-centric supervision to preserve prescribed arm motion, scene geometry, and manipulated-object state.These components target the central challenge of faithful bimanual video rollouts.
- Conclusion: 60.65 EWMScore-P ranked DreamX-Phi first among 31 Track 1 entries on the August 12, 2026 WorldArena 2.0 snapshot.The result is reported for the submission evaluated on that snapshot.
- Conclusion: A policy trained with the submitted world model as its rollout environment achieved 67.19% success on Adjust Bottle in Track 2, tying for second-highest performance.The policy used the submitted world model as its rollout environment.
8 Future Work
Future work will extend DreamX-Phi 1.0 beyond its forward-dynamics formulation, which predicts observations from externally provided actions but does not generate actions. The planned joint World Action Model will generate video and robot trajectories together to align proposed actions with their predicted visual consequences.
- 8 Future Work: DreamX-Phi 1.0 currently predicts future observations from externally provided action sequences rather than generating actions itself.This limitation motivates exploring alternative model formulations.
- 8 Future Work: Future work will develop a joint World Action Model that generates future video and robot action trajectories together.
- 8 Future Work: Jointly training video and action outputs is intended to align each proposed action with its predicted visual consequences.
Authors
The paper’s team members are listed alphabetically, and the ordering does not indicate relative contributions.
- Authors: Team members are Rui Chen, Xiangxiang Chu, Geng Li, Jifan Li, Qingfeng Shi, Datao Tang, Jing Tang, Jun Wang, and Pengfei Zhang.Names are listed alphabetically by last name, then by first name when last names are identical.