Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising

Jun Guo; Qiwei Li; Peiyan Li; Zilong Chen; Nan Sun; Yifei Su; Heyun Wang; Yuan Zhang; Xinghang Li; Huaping Liu

Source-linked AI summary

Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising

Jun Guo, Qiwei Li, Peiyan Li, Zilong Chen, Nan Sun, Yifei Su, Heyun Wang, Yuan Zhang, Xinghang Li, Huaping Liu

arXiv:2604.26694v2Apr 29, 2026cs.ROcs.AIcs.CV

TL;DR

Existing unified world-action models lack explicit 3D geometric grounding; X-WAM adds spatially aware 4D modeling with asynchronous denoising for efficient action execution, and outperforms baselines in policy success and 4D reconstruction quality.

Problem
Existing unified world-action models remain confined to 2D pixel-space observations, lacking explicit spatial awareness and 3D geometric grounding.
Method
X-WAM jointly models multi-view video, depth, robot states, and actions using a lightweight depth branch and Asynchronous Noise Sampling.
Results
X-WAM consistently outperforms baselines on RoboCasa, RoboTwin 2.0, and real-world experiments in policy success and visual and geometric 4D reconstruction metrics.
Takeaways & Limitations
The results support a single unified framework for jointly optimizing policy execution, visual generation, and spatial reconstruction.
Takeaways & Limitations
The depth design assumes that depth information can be inferred from RGB features without fully independent generation.

Abstract

from arXiv · show

We propose X-WAM, a Unified 4D World Model that unifies real-time robotic action execution and high-fidelity 4D world synthesis (video + 3D reconstruction) in a single framework, addressing the critical limitations of prior unified world models (e.g., UWM) that only model 2D pixel-space and fail to balance action efficiency and world modeling quality. To leverage the strong visual priors of pretrained video diffusion models, X-WAM imagines the future world by predicting multi-view RGB-D videos, and obtains spatial information efficiently through a lightweight structural adaptation: replicating the final few blocks of the pretrained Diffusion Transformer into a dedicated depth prediction branch for the reconstruction of future spatial information. Moreover, we propose Asynchronous Noise Sampling (ANS) to jointly optimize generation quality and action decoding efficiency. ANS applies a specialized asynchronous denoising schedule during inference, which rapidly decodes actions with fewer steps to enable efficient real-time execution, while dedicating the full sequence of steps to generate high-fidelity video. Rather than entirely decoupling the timesteps during training, ANS samples from their joint distribution to align with the inference distribution. Pretrained on over 5,800 hours of robotic data, X-WAM achieves 79.2% and 90.7% average success rate on RoboCasa and RoboTwin 2.0 benchmarks, while producing high-fidelity 4D reconstruction and generation surpassing existing methods in both visual and geometric metrics.

1 Introduction

X-WAM extends unified video-action modeling into explicit 4D world modeling by jointly targeting video generation, 3D reconstruction, policy success, and efficient execution. It combines lightweight depth adaptation with Asynchronous Noise Sampling to preserve pretrained visual priors while improving spatial modeling and action decoding.

Introduction: Existing unified world action models share representations for video generation and action prediction but remain confined to 2D pixel-space observations.This limitation motivates incorporating explicit spatial information into the unified modeling paradigm.
Introduction: X-WAM unifies high-fidelity video generation, 3D spatial reconstruction, policy success, and efficient action execution within one architecture.It builds on visual priors from a pretrained video foundation model.
Introduction: Replicating the pretrained DiT’s final blocks as a dedicated depth branch injects 3D awareness without doubling sequence lengths or disrupting pretrained visual priors.The lightweight structural adaptation avoids the computational overhead of concatenating depth tokens and the distribution shift caused by channel fusion.
Introduction: Asynchronous Noise Sampling jointly models video and actions using their joint distribution and an asynchronous denoising schedule.The design addresses the mismatch between many denoising steps for high-fidelity video and fewer steps for low-dimensional actions.
Introduction: X-WAM outperforms all baselines on RoboCasa, RoboTwin 2.0, and real-world earphone packing experiments while achieving superior visual and geometric 4D results.These results support jointly optimizing policy execution, visual generation, and spatial reconstruction in one framework.

2 Related Work

Related work spans policy models for direct real-time action prediction and world models for imagining future observations, with recent efforts unifying the two paradigms. A parallel research direction adds 3D information to improve spatial awareness, multi-view consistency, and spatial reasoning.

Embodied Models: Policy models, including predominantly VLA models, map observations directly to executable robot actions for real-time control, whereas world models learn environmental dynamics and imagine future observations.These paradigms are described as complementary but have largely evolved in isolation.
World Action Models: Recent World Action Models build end-to-end systems from video foundation models, including UWM and Motus as Unified World Models with flexible conditioning and multi-task generation.VideoVLA and Cosmos Policy instead append action tokens into video sequences for joint prediction.
World Action Models: Other approaches use Mixture of Transformer architectures with independent parameters and denoising timesteps for each modality.This line of work is identified alongside UWM, Motus, VideoVLA, and Cosmos Policy as recent video-foundation-model-based WAM research.
3D Spatial Modeling: Contemporary mainstream embodied models primarily operate in 2D, but limited explicit spatial awareness and modeling bottleneck spatial comprehension and out-of-distribution generalization.The passage attributes this limitation to reliance on purely data-driven fitting.
3D Spatial Modeling: Several world and world action models introduce 3D supervisory signals during video generation, providing multi-view consistency and superior spatial reasoning.ManiGaussian and GWM construct world models entirely within 3D representations using neural rendering with 3D Gaussian Splatting.

3 Methodology

X-WAM unifies future RGB-D video generation, 3D spatial reconstruction, proprioceptive-state prediction, and action execution in one model. Its lightweight depth branch and Asynchronous Noise Sampling jointly preserve pretrained video priors while enabling spatial modeling and efficient control.

Overall architecture: X-WAM jointly predicts future RGB videos, depth videos, proprioceptive states, and actions from language, initial state, and multi-view RGB observations.The model is fine-tuned from the pretrained Wan2.2-TI2V-5B Diffusion Transformer and processes the modalities in a unified denoising sequence.
Overall architecture: The asymmetric horizons produce H = 8 future RGB frames and states but K = 32 future actions, matching visual-generation and reconstruction needs with higher-frequency robot control.Initial RGB and state conditions remain clean during denoising, while actions use a higher control frequency for responsive execution.
Depth adaptation: The lightweight depth adaptation module replicates the final M < N DiT blocks into an auxiliary branch, using unilateral attention to predict inverse depth without altering the main branch.Depth is trained with mean squared error, and the auxiliary branch can be toggled during inference to reduce rollout overhead.
Asynchronous Noise Sampling: Asynchronous Noise Sampling assigns Ta < TO denoising steps to actions and states versus video, allowing noise-free actions to be dispatched after Ta forward passes while video continues denoising.During training, video and action noise levels are sampled from a coupled joint distribution rather than independent distributions to match asynchronous inference.
Training: X-WAM is fine-tuned with flow matching and inverse-depth supervision on over 5,800 hours of real-robot and simulated manipulation data unified across datasets.The training interface standardizes robot states and actions across heterogeneous datasets and coordinate systems.

4 Experiments

X-WAM is evaluated on policy execution, 4D reconstruction and generation, and joint ablations. It achieves strong manipulation success, superior spatial reconstruction, and improved action-generation efficiency through its depth architecture and asynchronous denoising schedule.

Policy execution: X-WAM achieves 79.2% average SR on RoboCasa and 89.8% Clean / 90.7% Randomized SR on RoboTwin 2.0, outperforming the reported baselines.On RoboCasa, it surpasses Cosmos Policy (67.1%) by 12.1 percentage points; on RoboTwin 2.0, it exceeds Motus (88.7% / 87.0%) across both protocols.
4D reconstruction and generation: X-WAM achieves the best performance across all reported visual, depth, and point-cloud reconstruction metrics on RoboCasa.The evaluation uses PSNR, SSIM, LPIPS, AbsRel, δ1 accuracy, and Chamfer Distance.
4D reconstruction and generation: 2.34 dB higher PSNR and Chamfer Distance reduced from 0.0680 to 0.0049 versus DreamZero + DA3 demonstrate stronger end-to-end spatial reconstruction.The comparison supports joint modeling over post-hoc depth estimation applied to independently generated videos.
Depth architecture design: 1888 ms latency for sequence concatenation and 1266 ms for channel concatenation show the overhead of explicit depth incorporation, whereas the interleaved branch matches the no-depth variant.Sequence concatenation achieves the best quality metrics but nearly doubles action latency because of its expanded sequence length.
Effect of ANS: 1033 ms asynchronous latency versus 4665 ms synchronous latency yields a 4.5× speedup by decoding actions in only the first 5 denoising steps.Synchronous variants use 25 joint steps, while asynchronous variants decode actions in 5 steps.

5 Conclusion · A Detailed Algorithms · B Training Details

X-WAM extends unified world-action modeling to spatially aware 4D dynamics through lightweight interleaved depth adaptation and Asynchronous Noise Sampling. The algorithms jointly process multimodal inputs, predict action-related velocities and depth, and align coupled training noise with asynchronous inference denoising.

5 Conclusion: X-WAM extends unified world-action modeling into spatially aware 4D dynamics simulation.The conclusion identifies this as the model’s central scope.
5 Conclusion: A lightweight depth adaptation module replicates final DiT blocks as an interleaved depth branch for high-quality spatial reconstruction.The depth branch avoids increasing sequence length and preserves pretrained visual priors.
5 Conclusion: Asynchronous Noise Sampling aligns training and inference noise distributions across modalities.ANS uses coupled noise sampling during training and an asynchronous denoising schedule during inference.
A Detailed Algorithms: Algorithm 1 jointly processes the multimodal sequence through a shared DiT trunk and interleaved depth branch.The procedure produces velocity predictions and depth estimates from the jointly processed representations.
A Detailed Algorithms: Algorithm 2 specifies Asynchronous Noise Sampling for both training and inference.Its purpose is to align coupled training noise sampling with the asynchronous denoising schedule used at inference.
A Detailed Algorithms: The denoising procedure concatenates noisy video, state, and action latents with learnable view embeddings before shared-trunk processing.The shared trunk runs before the main and depth branches are initialized.
A Detailed Algorithms: The algorithm returns predicted video, state, and action velocities together with predicted inverse depth.The outputs are denoted ˆvO, ˆvs, ˆva, and ˆD.
A Detailed Algorithms: During interleaved processing, each depth block attends to the main branch’s input while the main branch continues through the remaining DiT blocks.This establishes the asymmetric interaction between the main and depth branches.

B.1 Pretraining Data

X-WAM is pretrained on a large, diverse corpus spanning real-robot and simulated environments, totaling over 1.49 million episodes and approximately 5,874 hours. The data is systematically filtered and standardized before training.

Dataset scale and scope: 1.49 million episodes and approximately 5,874 hours comprise X-WAM’s pretraining data across real-robot and simulated environments.Table 5 summarizes the datasets used for pretraining.
Filtering: Preprocessing removes episodes with base locomotion, dexterous manipulation, or failed executions.Stationary frames are additionally removed from the DROID dataset following.
Standardization: All videos are uniformly downsampled to 3.75 FPS before training.The videos are also resized, although the supplied passage does not specify the target resolution.

B.2 Implementation Details · B.3 Baseline Details

X-WAM uses a unified end-effector-based state/action interface and asynchronous denoising implementation, with specified large-scale pretraining and benchmark inference settings. Baseline comparisons use reported or reproduced results with matched backbones where applicable and broadly comparable robotic-data regimes.

B.2 Implementation Details: X-WAM represents state as a 16-dimensional absolute vector and action as a 14-dimensional relative vector across two arms.Single-arm supervision uses the first 8 state and 7 action dimensions, with per-dataset quantile normalization and bias-free action scaling.
B.2 Implementation Details: ANS training couples video and action noise timesteps, including action-conditioned generation with probability p and asynchronous joint generation otherwise.In the asynchronous branch, tO is rescaled to [ta, 1], ensuring tO ≥ ta.
B.2 Implementation Details: ANS inference jointly denoises video, state, and action before continuing video-only denoising, making actions available after Ta steps and video after TO steps.The procedure initializes separate schedulers with Ta < TO and returns denoised video, state, and action outputs.
B.2 Implementation Details: X-WAM pretraining uses 256 NVIDIA H20 GPUs, total batch size 2,048, 40,000 steps, H = 8, M = 10, and p = 0.5.Optimization uses AdamW, peak learning rate 1 × 10^-4, 1,000-step warmup, cosine decay, and λs = λa = λD = 1.0.
B.2 Implementation Details: Benchmark inference uses Ta = 10 action denoising steps and TO = 50 video denoising steps with UniPC and classifier-free guidance scale 1.0.Each benchmark task is evaluated over 100 episodes, with success rates averaged under official protocols.
B.3 Baseline Details: For RoboCasa, π0, GR00T-N1.5, UWM, and Cosmos Policy results are taken from prior work, while DreamZero is reproduced with a Wan2.2-5B backbone.The backbone replacement is used for fair comparison.
B.3 Baseline Details: For RoboTwin 2.0, π0 and π0.5 results come from prior work, Motus and GigaWorld-Policy results from their papers, and UWM is reimplemented with Wan2.2-5B.The UWM backbone replacement is likewise intended to provide a fair comparison.
B.3 Baseline Details: Only DreamZero is fine-tuned from a general-purpose video model without prior robotic-data exposure; the other listed baselines use large-scale robotic pretraining datasets.The comparison is therefore conducted under broadly comparable data regimes.

C Detailed Results

This section reports X-WAM’s per-task success rates on the RoboCasa and RoboTwin 2.0 benchmarks.

X-WAM’s per-task success rates are reported for the RoboCasa and RoboTwin 2.0 benchmarks.

C.1 Per-Task Results on RoboCasa

Table 6 reports X-WAM’s success rate separately for all 24 manipulation tasks in the RoboCasa benchmark.

Per-Task Results: Table 6 reports X-WAM’s success rate on RoboCasa.The table covers the benchmark’s manipulation-task performance.
Per-Task Results: The RoboCasa evaluation spans 24 manipulation tasks.Results are presented for each task individually.
Per-Task Results: X-WAM is evaluated per task rather than only with an aggregate RoboCasa result.Table 6 organizes the success-rate results across the benchmark’s individual tasks.

C.2 Per-Task Results on RoboTwin 2.0

Table 7 reports per-task success rates on RoboTwin 2.0 under both Clean and Randomized settings.

Table 7 presents per-task success rates for RoboTwin 2.0.
The evaluation includes both Clean and Randomized settings.

D Real Robot Experiments

X-WAM is evaluated on a real dual-arm robot through a challenging long-horizon earphone-packing task requiring precise 3D spatial reasoning and tight-tolerance insertion. The deployment uses asynchronous inference and real-time chunking, while scalability and generalization results show stronger long-horizon reliability than XR-0.

Setup: The AC One dual-arm platform uses one main camera and two wrist-mounted cameras at 320 × 256, with approximately 20 hours of task demonstrations.Fine-tuning used 64 NVIDIA H20 GPUs, batch size 4 per GPU, and 40,000 steps.
Setup: Asynchronous inference uses 8 denoising steps and approximately 300 ms single-pass latency per action chunk, with RTC overlapping computation and execution.The robot operates at 15 Hz, executes 15 actions per one-second chunk, and uses a six-action RTC inference delay.
Task design: The earphone-packing episode comprises four sequential stages, each contributing 25% progress, and requires completion of all stages for 100% progress.The stages are opening the case, placing two earphones, closing the lid, and returning the case to the table.
Quantitative results: 93.8% progress on packing 2 earphones versus 79.1% for XR-0, and 68.0% versus 63.9% on packing 3 earphones, demonstrates X-WAM’s stronger long-horizon reliability.Both methods achieve perfect progress on the single-earphone task; the analysis attributes X-WAM’s advantage to explicit 3D spatial awareness.

E Limitations and Future Work

X-WAM’s main limitations are its fixed-length observation context and higher inference latency than dedicated policy models. Future work can extend temporal context through history-aware mechanisms and reduce latency through inference acceleration.

Context horizon: X-WAM processes only a fixed-length observation context without historical information or autoregressive rollout, limiting long-horizon task understanding.The limitation may cause suboptimal decisions when the current observation is insufficient to disambiguate task progress.
Inference latency: Approximately 300 ms per action chunk with 8 denoising steps makes X-WAM slower than specialized VLAs and lightweight WAMs.Real-time chunking overlaps computation with execution to support physical-robot deployment.
Future directions: X-WAM can be extended with history conditioning, KV caching, or autoregressive inference because its architecture and noise scheduling are orthogonal to long-context mechanisms.These extensions target longer temporal horizons.
Future directions: Model distillation, consistency models, and more aggressive asynchronous scheduling could further narrow the latency gap with dedicated policies.These approaches are proposed as inference-acceleration directions.