Source-linked AI summary
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
Jiaxu Wang, Yicheng Jiang, Tianlun He, Jingkai Sun, Qiang Zhang, Junhao He, Jiahang Cao, Zesen Gan, Mingyuan Sun, Qiming Shao, Xiangyu Yue
TL;DR
Robotic world models struggle to produce complete, geometrically consistent futures and to convert those futures into actions because inverse dynamics is ill-posed. The paper introduces a multi-view RGBD 4D world model with trajectory-latent test-time optimization and residual action refinement. Across three datasets and 34 manipulation tasks, it consistently outperforms strong baselines in 4D scene generation and downstream manipulation.
Problem
Existing methods provide image-based forecasts or partial 3D reasoning, while inverse dynamics cannot uniquely recover actions from perceptual transitions.
Method
The method generates geometry-consistent multi-view RGBD futures, optimizes a compact trajectory latent through the generator, and refines decoded actions with residual inverse dynamics.
Results
Across RoboTwin, RLBench, and a real-robot 4D multiview dataset with 14 tasks, the method consistently outperforms strong baselines in 4D scene generation and downstream manipulation.
Takeaways & Limitations
Geometry-consistent 4D prediction and test-time action inference improve embodied manipulation reliability, especially in cluttered and occluded environments.
Takeaways & Limitations
Test-time optimization requires multiple steps, introducing an inference-time computational cost.
Abstract
from arXiv · showhide
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.
1. Introduction
Existing robotic world models often forecast plausible images without complete geometric consistency, while converting imagined futures into actions remains ill-posed. The paper addresses both gaps with a multi-view RGBD 4D world model and trajectory-level action inference.
- Motivation: Image-space world models can violate geometric constraints, leaving imagined futures disconnected from executable manipulation.Single-view RGBD methods also produce incomplete geometry and can suffer scale and temporal drift in monocular depth.
- Motivation: Action inference is ill-posed because multiple actions can explain similar perceptual transitions, especially under partial observability and contact.Existing paradigms can also compound rollout errors or inherit pose-estimation fragility, while per-step actions overlook trajectory-level structure.
- Approach: The proposed model synthesizes geometry-consistent multi-view RGBD manipulation scenes from queried viewpoints using cross-modal and cross-view feature fusion.These mechanisms target appearance–geometry consistency and geometry-aligned information flow across views.
- Approach: Actions are inferred by optimizing a compact trajectory latent through the generative model, then refined with a residual inverse dynamics model.The trajectory latent captures temporal structure, while residual refinement supports executable action sequences.
- Evaluation: Experiments span 34 manipulation tasks across RoboTwin, RLBench, and a real-robot 4D multiview dataset with 14 tasks, consistently outperforming strong baselines.The reported evaluation covers both 4D scene generation and downstream manipulation.
2. Related Work
Related work extends generative modeling toward video, robotic world models, and 4D scene generation, but persistent limitations include weak geometric consistency, incomplete views, and poor scaling beyond two views.
- Video Generative Models: Video generation increasingly supports temporal modeling, latent synthesis, and camera-controlled novel views, but typically targets RGB without explicit multi-view geometric consistency.This limits direct suitability for contact-rich manipulation scenes.
- World Models for Robot Manipulation: Robotic world models support policy learning or imagined rollouts, yet most rely on 2D generation despite manipulation’s dependence on 3D geometry.Action inference in prior work uses inverse dynamics, intermediate features, or motion-relevant masks.
- 4D Video Generation: Early 4D video methods combine video generation with explicit 3D representations through SDS-style objectives, often requiring additional optimization.These approaches seek to distill dynamic 4D geometry from generated video.
- 4D Video Generation: RGB-DN reconstruction methods remain limited to single-view outputs, producing incomplete geometry despite post-optimization.This exemplifies the gap between 4D reconstruction and complete multi-view scene modeling.
- 4D Video Generation: Two-view pointmap methods provide consistent geometry but do not generalize well to more views and can depend sensitively on the source view.The limitation highlights the need for scalable multi-view generation.
3. Preliminaries
The imagine-then-act paradigm predicts instruction-conditioned futures before deriving actions, but partial observations and ill-posed inverse dynamics weaken action reasoning. The paper combines complementary-view prediction with trajectory-conditioned latent diffusion.
- Imagine-then-act Paradigm: Imagine-then-act uses a world model to predict instruction-related future observations before deriving actions.The agent receives an instruction and an initial RGB or RGB-D observation at time t0.
- Imagine-then-act Paradigm: Manipulation requires temporally consistent 3D relations, while limited viewpoints and occlusions make initial observations partial.Generated futures can therefore miss task-critical states and undermine action reasoning.
- Act Stage: Prior action heads and inverse dynamics degrade under partial predicted futures, with inverse dynamics remaining ill-posed.The proposed response is complementary-view prediction with cross-view-consistent appearance and geometry.
- Latent Video Diffusion Models: The latent video diffusion model learns a velocity field and generates samples by solving its probability-flow ODE with Euler steps.Inference starts from Gaussian noise, iteratively updates the latent, and decodes the final state with a VAE.
4. Methodology
The method generates geometrically consistent multi-view RGB-D futures from a single observation and uses trajectory-level latent conditioning to support action inference. Cross-modality, cross-view, and camera-aware fusion provide structured information for dynamic scene prediction.
- Problem formulation: Given one reference RGB-D observation, camera extrinsics, and a text instruction, the generator produces synchronized RGB-D sequences for the reference and target views.The outputs can be back-projected and fused into point-cloud sequences of the dynamic scene.
- Input representation: RGB and depth latents are fused within each view, while view latents are concatenated across height before Transformer processing.This tokenization controls neighborhood structure and supports variable numbers of input views.
- Cross-modality fusion: A learnable modality token identifies appearance and geometry streams before local cross-modality attention exchanges information through gated residual updates.Locality reduces cross-attention cost from global quadratic matching to O(Nk) and suppresses transfers under noise or imperfect alignment.
- View representation: Camera embeddings encode view identity using spherical coordinates around a shared look-at point, Fourier features, and log-radius rather than flattened extrinsics.The resulting representation exposes scale cues and distinguishes tokens from different viewpoints.
- Cross-view fusion: Geometry-aware deformable cross-view attention samples sparse correspondences along epipolar lines and refines locations with learned offsets before global self-attention.Known camera parameters provide the geometric basis, while query-conditioned refinement addresses coarse latent resolution.
- Trajectory conditioning and action inference: The action sequence is compressed into a low-dimensional trajectory latent, injected as a style code, and recovered at test time by optimizing the latent against a frozen generated rollout.A latent-consistency head reconstructs the conditioning tokens, and the optimized latent is decoded into actions with a pretrained TCN decoder.
5. Experiments
Experiments evaluate 4D scene generation and embodied action planning across synthetic and real-world datasets, including geometry, appearance, fusion, and trajectory-inference analyses. The results indicate more consistent multi-view geometry and stronger manipulation performance, with ablations supporting the proposed fusion and trajectory components.
- Experimental Setup: Experiments span RLBench, RoboTwin2, and a real-world robot dataset, covering 4D generation and manipulation on simulated and physical platforms.The datasets include over 8,000 RLBench trajectories, over 10,000 RoboTwin2 trajectories, and 14 real-world manipulation tasks.
- Experimental Setup: The evaluation measures appearance with FVD, SSIM, and PSNR, and geometry with depth and 3D point-cloud metrics.Geometry metrics include AbRel, RMSE, δ1, Chamfer Distance, and Earth Mover Distance.
- 4D Scene Generation: Back-projecting and fusing predicted multi-view RGB-D sequences produces more complete 3D point trajectories with fewer occlusion-induced holes and misalignments.The paper reports improved depth and point-cloud metrics and similar fused trajectories in the real-robot setting.
- Fusion Ablations: Explicit cross-view fusion improves geometric consistency, while removing it or replacing it with epipolar attention degrades performance and causes view misalignment.Observed failures include shifted objects and disagreement at depth boundaries across views.
- Fusion Ablations: Explicit cross-modality fusion keeps RGB predictions aligned with depth and object boundaries across views and time.Without modality fusion, RGB can drift from depth and lose correspondence with object boundaries.
- Embodied Action Planning: The full model consistently outperforms baselines across tasks and platforms, while trajectory-latent optimization and residual inverse dynamics improve action inference.Removing residual refinement or using a full inverse dynamics model underperforms the full model on RLBench; action initialization raises success from 72.6% to 76.5% on RLBench and from 43.0% to 46.6% on RoboTwin.
- Manipulation Ablations: Removing cross-view fusion, cross-modality fusion, or the compact trajectory representation consistently reduces manipulation success rates under the same 3-view RGB-D input.Direct RGB-D concatenation also performs worse because the pretrained video backbone is optimized for 3-channel RGB inputs.
6. Conclusion
The paper presents a multi-view 4D world model that jointly predicts geometry-consistent futures and infers executable actions. Across RoboTwin, RLBench, and a real-robot dataset, it reports consistent gains in 4D generation and manipulation.
- The method jointly supports geometry-consistent RGBD future prediction and action inference through explicit cross-view and cross-modality fusion.It uses a trajectory prior refined by a residual inverse dynamics model to address inverse-dynamics ambiguity.
- Experiments span three datasets, including a newly collected real-robot multiview dataset with 14 tasks and 34 manipulation tasks overall.
- The method consistently outperforms strong baselines in both 4D scene generation and downstream manipulation.
- Analyses validate the key design choices and suggest view-consistent 4D prediction as a geometric interface between imagined futures and executable actions.
Impact Statement
The method is intended to improve manipulation reliability through geometry-consistent 4D prediction and test-time action inference, particularly in cluttered and occluded environments. Scaling the number of generated views tends to improve reconstruction completeness and alignment, while deployment should account for misuse risks.
- Geometry-consistent 4D prediction and test-time action inference are especially beneficial in cluttered and occluded manipulation environments.
- Generating more views tends to produce more complete and better-aligned 4D reconstructions.The paper identifies increased compute or view budgets as a practical path toward stronger performance.
- Potential risks include misuse for producing realistic synthetic visual data or misleading reconstructions.The paper advocates responsible release and safety-aware deployment.
A. Implementation Details
The implementation combines masked latent video diffusion, trajectory conditioning, cross-modality and geometry-aware cross-view fusion, and residual point-based inverse dynamics. It supports variable-view generation and evaluates execution through decoded, interpolated actions.
- Diffusion and conditioning: Masked latent inputs train the diffusion model to generate 4D dynamics from a single frame and complete missing timesteps consistently.Inputs may contain the first frame, a full first-view video, or randomly masked video frames.
- Diffusion and conditioning: A TCN-VAE trajectory latent conditions generation and can later be replaced by an optimizable null trajectory token.
- Feature fusion: Local cross-modality attention exchanges information between aligned appearance and geometry features through symmetric RGB/depth fusion.
- Feature fusion: Geometry-aware deformable cross-view attention performs sparse fusion over multi-view latent features using camera geometry.
- Action execution: The residual inverse dynamics model crops interaction regions, back-projects RGB-D observations into 3D, and predicts action corrections from point-cloud transitions.The correction is conditioned on the trajectory-prior action decoded from the optimized latent.
- Action execution: Residual action learning uses an ℓ2 loss and focuses on local execution adjustments because the prior already supplies trajectory intent and temporal structure.
- Data and hardware: The real-world setup uses four RGB-D cameras, two robotic arms, synchronized 15 FPS streams, calibration refinement with ICP, and randomized distractors and target poses.
- Action execution: At evaluation, predicted RGB-D videos are fused into point-cloud sequences, mapped to actions, temporally interpolated, and executed for success-rate measurement.
E.1. Per-Task Success Rates on RLBench
Per-task results show broad manipulation robustness rather than gains concentrated in a small subset of behaviors. The method performs strongly on long-horizon, cluttered, contact-rich, and precise-object tasks across the reported benchmarks.
- RLBench: 91 on Close Drawer and 75 on Close Microwave demonstrate strong performance on coarse, long-horizon interactions.
- RLBench: 98 on Open Drawer indicates robust performance in cluttered or occluded settings.
- RLBench: 62 on Pick Up Cup, 89 on Push Button, and 97 on Play Jenga show strong performance on precise or contact-sensitive tasks.
- RoboTwin: On RoboTwin, the method achieves the highest success rates on Adjust Bottle (69), Beat Hammer (42), Click Bell (38), Grab Roller (68), Lift Pot (42), and Place Container (72).
- RoboTwin: Some RoboTwin tasks remain challenging, including Object Stand and Phone Stand, despite the method leading on a majority of tasks.
- Camera representation: Spherical camera embeddings improve depth accuracy and fused point-cloud geometry over Flatten Cam, although Flatten Cam attains slightly lower FVD.
G. Effect of the number of views on manipulation tasks
Increasing views improves RLBench manipulation success up to three views, after which gains are marginal while computation continues to rise. The reported failures indicate remaining challenges in action-direction ambiguity and precise spatial contact.
- RLBench average success rate rises from 68.6 with 1 view to 72.6 with 3 views, then reaches 72.9 and 73.1 with 4 and 5 views.The 3-view configuration provides most of the observed success-rate improvement.
- Runtime increases from 1.00 at 3 views to 1.20 with 4 views and 1.35 with 5 views.The additional views therefore impose steadily higher computation costs.
- The method underperforms mainly when partial observations make action direction ambiguous or when imprecise spatial grounding causes near-miss contacts.These failure modes concern both decision ambiguity and fine-manipulation precision.
- In the RoboTwin contact-sensitive failure, the policy reaches the target area but the end-effector is spatially misaligned and misses the red target block.
I. Limitation
Test-time action optimization improves consistency between actions and generated multi-view videos but introduces latency and depends on accurate calibration. These constraints can reduce responsiveness and action quality when system geometry changes.
- Test-time optimization requires multiple steps, increasing latency and potentially reducing real-time responsiveness.
- The method depends on accurate camera and robot calibration; changes in extrinsics or robot kinematics can reduce consistency and action quality.
J. Additional Qualitative Results of 4D Generation
Additional qualitative results visualize 4D generation across RoboTwin, RLBench, and real-world manipulation tasks. The figures cover multiview generation, qualitative RLBench outputs, and real-world cube-placement and drawer-opening scenarios.
- Additional RLBench qualitative results are provided in Figures 13, 14, and 15, with more visualizations and longer rollouts in the supplemental video.
- Figure 11 presents Mode-2 multiview generation results on RoboTwin alongside corresponding 4DGen results.
- Figure 12 presents Mode-1 multiview generation results on RLBench alongside corresponding 4DGen results.
- Figure 13 shows qualitative generation results on the RLBench dataset.
- Figures 14 and 15 show qualitative generation results for real-world cube placement into a cup and opening a drawer.