PaperlayerView exact revision on arXiv

Source-linked AI summary

DepthWorld: 3D World Model for Robot Manipulation

Jai Bardhan, Josef Sivic, Vladimir Petrik

arXiv:2610.08780v1cs.ROcs.AIcs.CV

TL;DR

Robot world models need geometrically faithful rollouts, but current RGB-trained models lack geometric supervision, while manipulation datasets lack high-quality 3D annotations and common architectural changes risk pretrained video priors. The paper calibrates DROID into a metric 3D dataset and trains a Stable Video Diffusion model to jointly predict multi-view RGB and depth. At equal training budget, depth supervision improves RGB prediction over an identical RGB-only baseline while providing metric depth for geometric reasoning.

  • Problem

    World models are used for policy evaluation, improvement, and planning, where geometric fidelity matters, but manipulation datasets lack supervision-quality 3D annotations and integrating geometry risks disturbing pretrained video priors.

    Source: §1 · pp. 1–2

  • Method

    The paper combines learned stereo depth with a joint factor graph to calibrate DROID, then trains a Stable Video Diffusion model to jointly predict multi-view RGB and depth using spatial latent tiling.

    Source: §1 · p. 2

  • Results

    +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while jointly predicting RGB and depth.

    Source: §6 · p. 9

  • Takeaways & Limitations

    DepthWorld yields metric depth alongside RGB predictions for downstream geometric reasoning.

    Source: §6 · p. 9

    As an autoregressive rollout model, DepthWorld inherits exposure bias and quality degradation common to such models.

    Source: §5 · p. 9

Abstract

from arXiv · show

World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

1 Introduction

World models support robot policy evaluation, improvement, and planning, but current RGB-trained models lack the geometric structure these uses require. The paper introduces calibrated 3D supervision and a world model that predicts RGB and depth together.

1 Introduction

  • Current RGB-only video world models lack geometric supervision, producing rollouts with incoherent depth across views and degraded object shape and identity.The paper identifies geometric fidelity as important to uses that read information from rollouts.Source: §1 · p. 1
  • Geometric supervision is difficult to add because manipulation datasets lack supervision-quality 3D annotations and conventional architecture changes risk disturbing pretrained video priors.Manipulation scenes include textureless and specular surfaces, and internally consistent camera rigs may still be misaligned with the robot frame.Source: §1 · p. 2
  • +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while DepthWorld jointly predicts multi-view RGB and depth.The approach uses spatial latent tiling and leaves the pretrained VAE unchanged.Source: §1 · p. 2
  • DROID-3D provides dense metric depth and recalibrated multi-view extrinsics for over 70,000 DROID episodes by pooling episodes to estimate shared robot kinematics.The calibration pipeline combines learned stereo depth with a joint factor graph.Source: §1 · p. 2

2 Related Work

Prior work has developed action-conditioned video world models and image-based geometric predictors, but robot manipulation lacks large-scale 3D supervision and current approaches to joint RGB–depth prediction alter pretrained backbones. DepthWorld addresses these gaps with calibrated manipulation data and spatial latent tiling.

Robotic World Models

  • Action-conditioned video models such as Ctrl-World [1] and IRASim [19] adapt pretrained diffusion backbones for manipulation rollouts, which presuppose geometric fidelity.These models are used for policy evaluation [1] [2], policy improvement [1] [3], and visual planning [4] [5].Source: §2 · p. 2

Geometric Foundation Models

  • DUSt3R [6], MASt3R [7], VGGT [8], and MapAnything [11] predict dense metric geometry, relying on datasets with depth and calibrated poses.Those datasets do not cover the visual distribution of real manipulation, including indoor scenes and large viewpoint differences between fixed and wrist cameras.Source: §2 · pp. 2–3

Depth-Aware Video Diffusion

  • Channel expansion [4] [26] and dual-branch architectures [27] add depth prediction but substantially modify pretrained backbones, whereas spatial latent tiling keeps the SVD VAE unchanged.Marigold [24] established that an image-diffusion VAE can encode and decode depth maps faithfully; this paper extends that approach to multi-view RGB–depth video prediction.Source: §2 · p. 3

Robot Datasets and Their Calibration

  • DROID offers synchronized multi-view stereo and a public URDF, but its geometric metadata is not suitable for downstream geometric learning without recalibration.PointWorld [34] previously attempted post-hoc recalibration using VGGT initialization and robot depth observations.Source: §2 · p. 3

3 Constructing Large-scale 3D Datasets for Robot Manipulation

The dataset pipeline turns multi-view stereo teleoperation data and a known robot model into a calibrated RGB-D corpus in the robot frame. It first builds consistent per-scene rigs, then jointly anchors camera extrinsics and shared robot kinematics across episodes.

3 Constructing Large-scale 3D Datasets for Robot Manipulation

  • The pipeline addresses the need for supervision-quality 3D by converting multi-view stereo teleoperation data with a known URDF into a calibrated RGB-D corpus in robot coordinates.Large manipulation datasets provide visual diversity and embodied dynamics, but their raw geometric annotations are not suitable for geometric pretraining.Source: §3 · p. 3

3.1 Stage 1: Initial Multi-View Calibration

  • Learned stereo matching supplies per-view metric depth, and dense feature matching supports a two-pass solve that aligns the external cameras before placing their pair in the robot frame.Textureless tabletops and specular arms challenge classical stereo; the resulting Stage 1 rig remains approximate in the robot frame because factory hand-eye and encoder errors persist.Source: §3.1 · p. 4

3.2 Stage 2: Robot-Grounded Joint Optimization

  • Pooling all episodes of each robot in a joint factor graph separates scene-specific extrinsic noise from shared kinematic errors and grounds camera poses to the robot frame.URDF-based rendering links robot-frame surface points to image pixels; the graph includes per-episode external-camera corrections and shared hand-eye and joint-encoder corrections.Source: §3.2 · pp. 4–5
  • The robust loss combines reprojection, metric-lifting, URDF-rendering, cross-camera, and featuremetric constraints, with joint-offset regularization.A GPU-accelerated Schur-complement solver eliminates per-scene blocks and reduces each iteration to a 13 × 13 global solve.Source: §3.2 · p. 5

3.3 DROID-3D

  • DROID-3D covers 71,100 scenes across 28 robots and 13 labs after filtering, using DROID’s raw 71k-trajectory release.The recordings use Franka Panda arms with two table-mounted ZED 2 cameras and a wrist-mounted ZED Mini.Source: §3.3 · p. 5

4 Building a 3D-Consistent World Model

DepthWorld uses DROID-3D’s metric depth and calibrated extrinsics to train a world model that predicts RGB and depth as a geometrically consistent scene. Spatial latent tiling preserves the pretrained SVD VAE, while robot-frame point-map supervision encourages consistency across cameras.

4 Building a 3D-Consistent World Model

  • DepthWorld consumes dense metric depth and calibrated extrinsics from DROID-3D to generate rollouts whose RGB and depth compose into a geometrically consistent 3D world.The design addresses the risk that modifying pretrained SVD can damage its appearance and motion priors, and that independent per-view depth supervision does not enforce cross-view consistency.Source: §4 · pp. 5–6

4.1 Joint RGB-Depth Prediction via Spatial Tiling

  • Spatial latent tiling encodes each RGB and depth image with the unmodified SVD VAE and places them side-by-side for joint U-Net denoising.Three cameras and two modalities form a 72 × 80 latent grid, processed across 11 frames: six history and five future.Source: §4.1 · p. 6

4.2 Robot-Frame Point-Map Supervision

  • A lightweight DPT head predicts robot-base-frame point maps from future latents, with predictions supervised against point maps derived from ground-truth depth and calibrated extrinsics.The head also predicts per-pixel confidence, used in a confidence-weighted robust geometric loss.Source: §4.2 · pp. 6–7

Training Schedule

  • Training first fits the joint RGB-depth model for 40,000 steps, then jointly trains all components for 50,000 more steps with point-map loss weight 0.005.The second stage initializes the DPT head from VGGT.Source: Training Schedule · p. 7

5 Experiments

The experiments evaluate DROID-3D calibration and DepthWorld prediction on held-out trajectories. Joint RGB-depth prediction improves RGB quality, while calibration improves geometric consistency and robot-frame accuracy over a per-scene baseline.

DROID-3D calibration accuracy

  • DROID-3D improves calibration over the per-scene baseline by 23× on EE, 13× on WE, 8× on RD, and +37 absolute points on IoU.The gap remains on scenes where the baseline converges; that baseline fails to converge on about 18% of scenes.Source: §5 · p. 7, Table 1 · p. 8

Accuracy to GT poses

  • On the GT setup, extrinsics reach 4.9 mm/0.39° median error and hand-eye reaches 3 mm/0.3° across five capture sets; joint offsets reach 0.14° mean error.The per-scene baseline obtains 72 mm / 3.0° median pose error on the same setup.Source: §5 · p. 7

Qualitative results

  • DepthWorld’s qualitative rollouts simulate grasping the red cup and moving an arm through an open wardrobe door, while the RGB-only model fails at both interactions.DepthWorld also produces an aligned depth map in the cup example.Source: Figure 4 · p. 8

World model prediction quality

  • +1.48 dB PSNR on external views and +1.02 dB on wrist views: RGBD models outperform the RGB-only baseline on all three RGB metrics.The point-map auxiliary loss further refines depth accuracy while maintaining RGB generation quality.Source: §5 · p. 8, Table 2 · p. 9

Cross-view consistency improvements

  • RGB+D improves cross-view reprojection PSNR over RGB by +1.0 dB (17.60 vs. 16.60), and PM-DPT reduces projected depth disagreement by 2.6 mm versus RGB+D.PM-DPT improves reprojection PSNR by a further +0.05–0.08 dB over RGB+D.Source: §5 · p. 8

Depth quality

  • Depth quality is view-dependent: δ1 exceeds 0.94 on external views but is about 0.82 on wrist views.The paper attributes the wrist gap to rapid camera motion, proximity to the object, and frequent arm self-occlusion.Source: §5 · p. 9

Limitations

  • DepthWorld’s autoregressive rollouts inherit exposure bias and quality degradation common to such models [1].The paper notes that RL post-training has improved rollout stability in recent work [2].Source: §5 · p. 9

6 Conclusion

The paper introduces DROID-3D, a calibrated 3D corpus, and DepthWorld, a world model that jointly predicts multi-view RGB and depth. At equal training budget, joint depth prediction improves RGB PSNR over an RGB-only baseline while yielding metric depth for geometric reasoning.

Contributions

  • DROID-3D provides dense metric depth and robot-grounded multi-view extrinsics across more than 70,000 episodes, using a factor graph that pools episodes against shared robot kinematics.DepthWorld uses spatial latent tiling to jointly predict multi-view RGB and depth with a Stable Video Diffusion-based model.Source: §6 · p. 9

Empirical finding

  • +1.48 dB PSNR: at equal training budget, joint RGB-depth prediction improves RGB over an RGB-only baseline while yielding metric depth.The authors frame the combined dataset and model as a path toward rollouts that compose into a coherent 3D scene.Source: §6 · p. 9

Supplementary Material for DepthWorld: 3D World Model for Robot Manipulation

The supplementary material documents the DROID-3D calibration pipeline, DepthWorld architecture and training, and the evaluation protocols.

Appendices A–B

  • Appendices A and B detail per-view depth and multi-view consistency, then the robot-grounded joint factor graph and its design choices.These appendices also specify residuals, hyperparameters, and discarded alternatives.Source: Supplementary Material for DepthWorld:… · p. 14

Appendix C

  • Appendix C covers DepthWorld’s architecture and training, from depth preprocessing and spatial latent tiling to inference rollout.It also describes the point-map head, objective, and optimization schedule.Source: Supplementary Material for DepthWorld:… · p. 14

Appendix D

A.1 Per-View Metric Depth

The calibration pipeline obtains metric depth from stereo disparity, aligns views with dense correspondences, then jointly anchors scene extrinsics to shared robot kinematics. Its stages address difficult manipulation-scene imagery and calibration noise at both per-scene and robot-wide scales.

A.1 Per-View Metric Depth

  • Metric depth is triangulated from predicted per-pixel disparity using each calibrated stereo pair’s known baseline and focal length.The pipeline uses the global-matching stereo network S2M2 for disparity estimation.Source: §A.1 · p. 15
  • On the GT setup with GT poses, S2M2 robot-surface median errors are 11.2–15.3 mm versus 12.6–16.9 mm for ZED NEURAL across all sets.The authors also report sharper results and faster runtime than FoundationStereo at 1280 × 720p.Source: §A.1 · p. 15

A.2 Robust Multi-View Consistency Optimization

  • Dense RoMa v2 matches filtered by MAGSAC support two-pass bundle adjustment: first align external cameras, then anchor their global pose to the wrist trajectory.Sparse matchers return too few reliable correspondences, especially for wrist–external views; those matches are noisier than external–external matches.Source: §A.2 · pp. 15–16

B Stage 2: Robot-Grounded Joint Optimization

  • Stage 2 jointly refines base-frame camera extrinsics and robot-specific hand-eye and joint-encoder parameters using all episodes from the same robot.Its inputs are Stage 1 per-scene rigs, the URDF, and raw joint encoder readings.Source: B Stage 2: Robot-Grounded Joint… · p. 16

B.1 Kinematic Anchoring via Rendering

  • LoMa-G matches rendered URDF arm points to real-image pixels, while tight crops increase the match count from 92 to 277.The method uses LoMa features, which the authors report align better and provide higher resolution than raw DINOv2 features.Source: §B.1 · p. 17

B.2 Joint Factor Graph: Residual Definitions

  • The joint graph assigns 12 DoF of external-camera corrections per episode and shares 13 DoF of hand-eye and joint-offset corrections across episodes; uncertain wrist–external matches are downweighted.The wrist pose is determined by the robot’s kinematic chain rather than given per-episode degrees of freedom.Source: §B.2 · pp. 17–18

B.3 Ablations on the Pose Estimation Pipeline

The ablation reports that jointly optimizing shared robot parameters improves pose quality; the solver uses robust losses and exploits the graph’s sparse structure to scale across episodes.

B.3 Ablations on the Pose Estimation Pipeline

  • Joint factor-graph optimization of shared hand-eye and joint-offset parameters significantly improves the obtained poses.The ablation is reported in Table A1; the deployed configuration amplifies the 3D term and omits the featuremetric term.Source: §B.3 · p. 22, §B.4 · p. 22

B.4 Hyperparameters and Solver

  • The deployed solver uses a robust Cauchy loss, weights the 3D metric factor by λ3d = 10^4, and regularizes joint offsets with λq = 10^-2.The 3D factor is up-weighted to balance metre-scale residuals against pixel-scale factors.Source: §B.4 · p. 22
  • A Schur-complement Levenberg–Marquardt solver reduces each iteration to a dense 13×13 shared-parameter solve plus parallel per-scene back-substitution.Typical convergence takes approximately 12.5 minutes per robot group; data loading, rather than linear algebra, is the practical bottleneck.Source: §B.4 · p. 22

B.5 Qualitative results of DROID-3D Calibration Dataset vs Baseline

The qualitative comparison contrasts robot-pose alignment against a PointWorld-based baseline. The accompanying depth-preprocessing steps preserve foreground geometry while adapting depth images to the SVD VAE input range.

B.5 Qualitative results of DROID-3D Calibration Dataset vs Baseline

  • On the examples shown, PointWorld-based poses have around 110–130 mm average robot depth error, versus around 15–20 mm for ours.The visualization shows points from segmented robot pixels and notes minor cleaning to remove floaters from imperfect segmentation.Source: §B.5 · p. 22, Figure A10 · p. 23

C.1 Depth Preprocessing Rationale (Supplement to Section 4.1)

  • Depth preprocessing uses unprojection with z-buffer min-pooling at 192×320, clips depth to the per-camera 95th percentile, applies a log transform, then normalizes to [0, 1] and replicates the channel three times.The min-pooling preserves foreground geometry and prevents flying-pixel artifacts at object boundaries.Source: §C.1 · p. 24

C.2 Spatial Latent Tiling vs. Other Depth Integration Architectures

Spatial latent tiling integrates RGB and depth in a wider latent grid, allowing one unmodified U-Net to denoise both modalities. In the reported ablation, it improves RGB metrics over the dual-branch alternative while matching or improving depth quality overall.

C.2 Spatial Latent Tiling vs. Other Depth Integration Architectures

  • Tiling places RGB and depth side-by-side in a wider latent grid, so a single unmodified U-Net jointly denoises both modalities without major architectural changes.This design avoids a parallel depth U-Net and the channel expansion of the alternative architecture.Source: §C.2 · p. 24
  • Spatial tiling improves every RGB metric on both views versus the dual-branch model, including +1.7 dB external and +1.5 dB wrist PSNR, while matching or improving depth quality.The dual-branch model is marginally ahead only on external RMSE; the authors note that the comparison is not one-to-one because of compute constraints.Source: §C.2 · p. 24

C.3 Effect of the Depth Source on DepthWorld

DepthWorld’s rollout quality depends strongly on the depth source: sparse, noisy ZED ULTRA depth worsens both depth and RGB metrics, while S2M2 and ZED NEURAL produce similar models.

C.3 Effect of the Depth Source on DepthWorld

  • ZED ULTRA raises AbsRel from 0.074 to 0.194 on the external view and from 0.266 to 0.461 on the wrist, while S2M2 and ZED NEURAL remain within 0.2 dB PSNR across views.S2M2 and ZED NEURAL depth values are correlated at ∼0.97; their small depth-metric differences vary by view.Source: §C.3 · p. 25

C.4 Auxiliary Point-Map Head (Supplement to Section 4.2)

  • Factory rather than refined DROID-3D extrinsics leave RGB quality unchanged but worsen wrist-view depth AbsRel from 0.220 to 0.233.The ablation compares point-map targets built from factory versus refined extrinsics.Source: Table A4 · p. 26, §C.4 · p. 26

C.5 Training Objective

  • The objective combines unchanged SVD denoising with confidence-weighted robust point-map supervision, drops the largest 5% of per-pixel terms, and EDM-weights future samples with min-SNR-γ 5.The point-map targets are robot-base XYZ maps backprojected from ground-truth depth using calibrated cameras.Source: §C.5 · p. 26

C.6 Training Schedule and Optimization

  • Training runs 90,000 steps: 40,000 for the joint RGB–depth model alone, then 50,000 with the point-map head attached at λpm = 0.005.The optimization uses a constant learning rate of 10^-5, global batch size 64, and EMA decay 0.9999.Source: §C.6 · p. 27

C.7 Inference and Autoregressive Rollout

  • Inference uses 11-frame windows at 5 Hz, with six history and five future frames, generating ten future chunks autoregressively from a repeated first-frame history.Each window uses 50 denoising steps and classifier-free guidance scale 2.Source: §C.7 · p. 27

C.8 Depth Reference and De-normalization for Metric Evaluation

  • Depth predictions are de-normalized back to metric units and scored against ground-truth depth after the same SVD VAE round trip, factoring out its irreducible reconstruction error.The reference bounds what a latent-space predictor can represent, so the metrics measure prediction relative to that representational ceiling.Source: §C.8 · p. 27

D.1 Calibration Baseline Modifications (Supplement to Section 5.1)

  • The calibration baseline uses the same S2M2 depth as the proposed method, isolating the comparison to per-scene versus robot-coupled joint-graph optimization.This modification removes differences in depth predictors as a source of the performance delta.Source: §D.1 · p. 27

D.2 Geometric Calibration Metrics

  • Calibration quality is assessed with four geometric proxies: external-to-external and wrist-to-external reprojection error, robot depth error, and robot-mask IoU.The metrics are proxies because DROID has no ground-truth extrinsics.Source: §D.2 · pp. 27–28

D.3 World-Model Evaluation Protocol

The supplement specifies held-out evaluation and representationally fair 3D comparisons, then contrasts DepthWorld with PointWorld and adapted TesserAct. Results show a long-horizon scene-geometry advantage over PointWorld and stronger RGB and depth metrics than TesserAct, alongside important comparison caveats.

D.3 World-Model Evaluation Protocol

  • A random 1% episode holdout tests same-robot/lab trajectory prediction, while RGB is retrained on the same 71,100 episodes, split, architecture, and budget as RGB+D.Metrics are averaged over rollout frames separately for external and wrist views; depth is scored in metric units without per-frame scale-and-shift alignment against VAE-decoded ground truth.Source: §D.3 · p. 28

D.4 Comparison with PointWorld (Supplement to Section 5)

  • PointWorld rolls out robot-frame seed points, whereas DepthWorld predicts camera depth at known pixels; both are lifted to shared tracked locations and scored against their own representationally reachable targets.DepthWorld’s VAE encode–decode round trip creates an error floor, so the fair comparison scores PointWorld against raw depth and DepthWorld against VAE-round-tripped depth.Source: §D.4 · pp. 28, 30
  • Tracker-derived 3D tracks are filtered to workspace points, and robot-arm points are excluded because both models receive robot actions and the arm would dominate moving scores.Tracks are lifted from tracked pixels using ground-truth depth and calibrated poses; the robot mask also drops some points on objects grasped near the gripper.Source: §D.4 · p. 29
  • A fixed persistent moving-point set avoids scores improving merely as occluded tracks disappear; Chamfer separately measures cloud occupancy rather than point identity.Tracked-point ℓ2 compares the predicted and target 3D position of the same point, while whole-scene Chamfer is dominated by static structure.Source: §D.4 · pp. 29–31
  • At 8 s, tracked moving-point ℓ2 error is 46 vs. 56 mm and whole-scene Chamfer is 9 vs. 18 mm (DepthWorld vs. PointWorld).PointWorld is more accurate on tracked moving points at 2 s; its whole-scene Chamfer distance grows over the rollout while DepthWorld’s stays flat.Source: Table A5 · p. 31, §D.4 · p. 31

D.5 Comparison with TesserAct (Supplement to Section 5)

  • TesserAct’s transformer stays unchanged, but action tokens replace text conditioning; it is retrained on external-camera single-view clips from DROID-3D for DepthWorld’s 50k-step budget.The comparison uses only external cameras because TesserAct has no multi-view rig, and both models start from generic pretrained weights rather than depth-pretrained checkpoints.Source: §D.5 · pp. 31–32
  • 19.43 dB PSNR and 0.154 AbsRel for adapted TesserAct trail DepthWorld’s 24.07 dB PSNR and 0.074 AbsRel at 50k steps, with quality decaying over the rollout.TesserAct’s first generated chunk is reasonable, but quality falls steadily through the fortieth frame.Source: §D.5 · p. 32
Loading 2610.08780v1…