PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-aware 4D Video Generation for Robot Manipulation

Zeyi Liu, Shuang Li, Eric Cousineau, Siyuan Feng, Benjamin Burchfiel, Shuran Song

arXiv:2507.01099v4cs.CVcs.AIcs.LGcs.RO

TL;DR

Robotic video prediction must capture both temporal coherence and 3D consistency across camera views. This paper uses cross-view pointmap supervision to learn shared geometry, and reports improved generation quality and consistency alongside higher manipulation success against a baseline.

  • Problem

    Generating videos that are both temporally coherent and geometrically consistent across camera views remains challenging for robotic planning and interaction.

    Source: §1 · pp. 1–2

  • Method

    The model jointly predicts future RGB frames and pointmaps, using cross-view pointmap alignment to learn shared geometry without camera-pose inputs.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Over 50% higher task success rate on average than Dreamitate; the method also improves video-generation quality and 3D consistency over baselines.

    Source: §4.3 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    Generated 4D videos can provide robot end-effector trajectories through an off-the-shelf 6DoF pose tracker for manipulation tasks.

    Source: §1 · p. 2

    The current model’s relatively slow inference compared with end-to-end behavior cloning makes closed-loop planning difficult.

    Source: §5 · p. 10

Abstract

from arXiv · show

Understanding and predicting dynamics of the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains a significant challenge. To address this, we propose a 4D video generation model that enforces multi-view 3D consistency of generated videos by supervising the model with cross-view pointmap alignment during training. Through this geometric supervision, the model learns a shared 3D scene representation, enabling it to generate spatio-temporally aligned future video sequences from novel viewpoints given a single RGB-D image per view, and without relying on camera poses as input. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, yielding robot manipulation policies that generalize well to novel camera viewpoints.

1 INTRODUCTION

The paper targets future videos that preserve temporal coherence and cross-view geometry for robotic manipulation. It uses pointmap alignment to learn shared scene structure and support novel-view prediction.

1 INTRODUCTION

  • Robots need to anticipate interaction-driven object motion and occlusion over extended horizons, but generated videos must balance temporal coherence with cross-view 3D consistency.The introduction says existing models often capture one objective at the expense of the other.Source: §1 · pp. 1–2
  • Cross-view pointmap alignment trains the model to predict reference-view and projected pointmap sequences, learning a shared scene representation for novel viewpoints.The method minimizes differences between the reference and projected 3D points over time.Source: §1 · p. 2
  • Pretrained video diffusion weights provide temporal priors while joint RGB-frame and pointmap generation combines the original video loss with geometry-consistent supervision.The resulting RGB-D videos are described as spatio-temporally consistent.Source: §1 · p. 2
  • Future pointmaps can be generated from initial RGB-D observations without camera-pose inputs, bypassing extrinsics calibration for novel camera poses.The predicted multi-view RGB-D videos can also supply end-effector trajectories through an off-the-shelf 6DoF pose tracker.Source: §1 · p. 2
  • Predicted videos support trajectory extraction and manipulation evaluation on three simulated tasks with unseen training views, where the method outperforms the baseline.The introduction also describes a benchmark spanning simulation and real-world tasks recorded from diverse viewpoints.Source: §1 · p. 2

2 RELATED WORK

Related work spans video prediction, multi-view and 4D generation, and video-based robot planning. The paper positions cross-view pointmap alignment as a way to combine temporal and geometric consistency for dynamic manipulation scenes.

2 RELATED WORK

  • Earlier video prediction used recurrent networks and GANs, while later diffusion approaches added temporal layers or latent diffusion; video models also generate depth and pointmaps.This work adapts a latent video diffusion model to learn jointly from RGB and depth videos.Source: §2 · pp. 2–3
  • Camera-conditioned video generation associates poses with views, while prior 4D methods separately optimize temporal consistency and novel-view spatial consistency.The paper instead applies cross-view pointmap alignment on standard video diffusion for joint consistency in multi-object robot manipulation scenes.Source: §2 · p. 3
  • Robot-planning methods extract actions from future frames or jointly predict frames and actions, but accurate cross-view geometry remains challenging for manipulation.The figure depicts two-view RGB-D inputs, future pointmaps and RGB videos, and end-effector pose extraction with a 6DoF tracker.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD

The method combines video diffusion with cross-view pointmap supervision to predict temporally coherent, geometrically aligned future views. It also uses generated RGB-D sequences to recover robot gripper trajectories for downstream control.

3 METHOD

  • Stereo video prediction with 3D supervision addresses single-view models’ lack of geometric grounding and spatial consistency.Each pixel’s depth yields a 3D pointmap for its camera view.Source: §3 · pp. 3–4

3.1 DIFFUSION-BASED VIDEO GENERATION

  • Stable Video Diffusion encodes history frames into latents, predicts future latents with a U-Net, and decodes them into RGB frames.Training uses a clean-data prediction objective; inference progressively denoises Gaussian noise.Source: §3.1 · p. 4

3.2 GEOMETRY-CONSISTENT SUPERVISION

  • Two U-Net decoders use cross-attention so the second view predicts pointmaps in the reference view’s coordinate frame.The model is trained with diffusion losses for native and projected pointmaps; camera poses define projection during training but are not inference inputs.Source: §3.2 · pp. 4–5

3.3 JOINT TEMPORAL AND 3D CONSISTENCY OPTIMIZATION

  • Joint optimization combines RGB video diffusion and pointmap consistency losses across predicted timesteps and both camera views.The geometric-supervision weight λ is set to 1 in the experiments.Source: §3.3 · p. 5

3.4 ROBOT POSE ESTIMATION FROM 4D VIDEOS

  • An off-the-shelf 6DoF tracker estimates gripper poses from generated RGB-D frames, using an initial object mask, camera intrinsics, and gripper CAD model.Pose estimates from both views are ranked by confidence and transformed into the global frame; finger-point-cloud distances determine gripper open/close state.Source: §3.4 · p. 5
  • Table 1 compares cross-view consistency, RGB video quality, and depth quality across simulated and real-world datasets.Its caption describes the method as maintaining strong cross-view consistency alongside high-quality video and depth generation.Source: Table 1 · p. 6

4 EXPERIMENTS

Experiments evaluate video generation on simulated and real-world manipulation tasks, then test whether generated videos support policies on unseen camera views. The method generally leads baselines in generation quality and cross-view consistency, while its video-based policy achieves higher average task success than Dreamitate.

4.1 TASKS

  • Evaluation spans three simulated manipulation tasks and four real-world tasks, using novel camera viewpoints for testing.Simulation uses 25 demonstrations per task with 12 of 16 views for training and four for testing; the real-world dataset has 20 demonstrations per task from two cameras.Source: §4.1 · p. 6

4.2 4D VIDEO GENERATION RESULTS

  • Across simulation tasks, the method mostly achieves the best results, including depth and cross-view consistency; after around 15k fine-tuning steps, it consistently outperforms baselines on real-world tasks.Figure 3 qualitatively compares novel-view predictions; Figure 4 shows high-fidelity real-world RGB-D sequences capturing gripper motion over time.Source: Figure 3 · p. 7, §4.2 · p. 8, Figure 4 · p. 8

4.3 ROBOT POLICY RESULTS

  • The RGB-D-video policy achieves over 50% higher task success on average than Dreamitate across three simulated tasks.Diffusion Policy struggles with unseen viewpoints; DP3 improves on one task but not the other two, while direct gripper-pose tracking on generated RGB-D videos yields higher action accuracy and improves grasping success.Source: Table 2 · p. 9, §4.3 · p. 9

5 LIMITATION

The method requires RGB-D video datasets with varied camera viewpoints, which are challenging to collect in the real world. Its inference speed also makes closed-loop planning difficult relative to end-to-end behavior cloning.

5 LIMITATION

  • Real-world training data are difficult to collect because of hardware constraints and camera calibration requirements.The method requires RGB-D video datasets with varying camera viewpoints; such datasets are easy to generate in simulation.Source: §5 · pp. 9–10
  • High-quality depth is often difficult to obtain in real-world settings.The authors suggest RGB-based depth estimation as a possible aid to future data curation.Source: §5 · p. 10
  • Inference is relatively slow compared with end-to-end behavior cloning, making closed-loop planning difficult.The authors note faster video-generation approaches that could lead to more reactive robot policies.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents a 4D video generation model for spatio-temporally consistent RGB-D sequences, trained with cross-view pointmap supervision. It reports improved generation quality and 3D consistency over baselines and uses generated videos to extract robot actions.

6 CONCLUSION

  • The model produces spatio-temporally consistent RGB-D sequences using geometric supervision that projects pointmaps across camera views.It learns a shared geometric space and generates future videos from novel viewpoints without camera poses at inference.Source: §6 · p. 10
  • The method improves video-generation quality and 3D consistency compared with baseline methods.Source: §6 · p. 10
  • Generated 4D videos can be used with an off-the-shelf 6DoF pose tracker to extract robot actions, with higher success rates on several manipulation tasks.Source: §6 · p. 10

7 ETHICS STATEMENT

The work studies multi-view video generation for robot manipulation using synthetic and controlled-lab demonstrations, without human subjects or sensitive personal data. It identifies potential assistive benefits alongside misuse risks and limits its release to robotic data under a harmful-use-restricting license.

7 ETHICS STATEMENT

  • The work uses simulation data and controlled-lab robot demonstrations, with no human subjects or sensitive personal data.Source: §7 · p. 11
  • The authors cite assistance with daily tasks and elder care as potential societal benefits, while acknowledging misuse risks such as manipulative content generation.Source: §7 · p. 11
  • The authors limit the work to robotic data and plan to release code and models under a license restricting harmful use.Source: §7 · p. 11

8 REPRODUCIBILITY STATEMENT

The paper points to main-text and appendix sections for model, training, dataset, evaluation, and baseline details. It also reports a project website and plans to release code, datasets, and pretrained models upon acceptance.

8 REPRODUCIBILITY STATEMENT

  • Architecture, training objectives, and hyperparameters are detailed in Section 3 and Appendix Sections A.1, A.3, and A.4.Source: §8 · p. 11
  • Simulation and real-world dataset designs, camera sampling, evaluation metrics, and baseline implementations are described in the appendices and Section 4.Source: §8 · p. 11
  • A project website provides visualizations, and the authors plan to release code, datasets, and pretrained models upon acceptance.Source: §8 · p. 11

A APPENDIX

The appendix outlines where the paper provides additional model, dataset, training, evaluation, and LLM-use details.

A APPENDIX

  • The appendix directs readers to model architecture, camera sampling, training and inference details, additional results, and the LLM-use statement.It identifies Sections A.1–A.5 and the corresponding appendix material.Source: A APPENDIX · p. 16

A.1 MODEL DETAILS

The appendix details how the model conditions and encodes observations, exchanges information across views, and weights gripper-region training loss.

A.1 MODEL DETAILS

  • The model repeats the latest observation 10 times to provide the historical-frame input used for prediction.This follows the SVD implementation described in the passage.Source: §A.1 · p. 16
  • Separate image and pointmap VAEs encode each view; their features are concatenated with noisy future latents before entering the U-Net.The combined input has shape h × 16 × 32 × 40, with h = 10.Source: §A.1 · p. 16
  • Cross-attention after each decoder block lets the second-view branch attend to native-view features and capture spatial correspondences.The model adds 12 such layers.Source: §A.1 · p. 16, Figure 5 · p. 16
  • A gripper-region weight map reweights diffusion loss, doubling the contribution of pixels marked within the gripper region.The gripper masks are downsampled to latent resolution before being used as the spatial weight map.Source: §A.1 · p. 16

A.2.1 SIMULATION TASKS

The appendix documents evaluation settings and additional results, including qualitative multi-view generation, pose-tracking errors, and three-view performance.

A.2.1 SIMULATION TASKS

A.2.2 CAMERA SAMPLING

  • Camera sampling draws 16 poses per episode; training and evaluation cameras are distinguished in Figure 7, with positions restricted to the stated half-sphere region.The sampled region uses radii 0.7–1.2 m and coordinate bounds x = 0.2–0.6 m, y = −0.5–0.5 m, and z = 0.7–1.2 m.Source: Figure 7 · p. 17, §A.2.2 · p. 17

A.2.3 REAL WORLD TASKS

  • Figure 8 shows the tasks included in the real-world dataset.Source: Figure 8 · p. 18

A.3 TRAINING DETAILS

  • Task-specific training runs about 60 epochs on four RTX A6000 GPUs; inference uses 25 EulerEDMSampler steps.The method reports 30.0 s inference time, 47G training memory, and 2.4B parameters; it is slower than 4D Gaussian but produces stronger generation quality and predicts from one RGB-D image per view.Source: §A.3 · p. 18, Table 3 · p. 18

A.4 ADDITIONAL RESULTS

  • Across simulation and real-world tasks, the model produces visually faithful, spatio-temporally consistent multi-view videos; baseline comparisons show fewer cross-view inconsistencies and RGB or depth artifacts.The simulation examples include unseen views; the real-world model was fine-tuned for about 15k steps.Source: Figure 9 · p. 19, §A.4 · p. 19, Figure 10 · p. 20

A.5 THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • ChatGPT 5 assisted with manuscript editing, while Claude helped generate visualization code; authors report that technical content and conclusions were their own work.Source: §A.5 · p. 20

A.6 POSE TRACKING ERROR ANALYSIS

  • Pose-tracking MSE is generally around or below 0.001, with occasional peaks near 0.003 for the right gripper in PutSpatulaOnTable and the left gripper in StoreCerealBoxUnderShelf.The analysis separates tracker performance on ground-truth RGB-D from errors involving generated RGB-D and shows representative high-error frames.Source: §A.6 · p. 21, Figure 13 · p. 23

A.7 SCALING TO MORE INPUT VIEWS

  • The model can add a third or more views at inference without retraining, with inference time increasing approximately linearly; three-view performance degrades slightly but retains good visual quality and cross-view consistency.The three-view evaluation uses randomly sampled unseen viewpoints on StoreCerealBoxUnderShelf.Source: Table 4 · p. 22, §A.7 · pp. 22–23
Loading 2507.01099v4…