Source-linked AI summary
Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction
Zeren Jiang, Chuanxia Zheng, Iro Laina, Diane Larlus, Andrea Vedaldi
TL;DR
Geo4D addresses monocular feed-forward 4D reconstruction in challenging uncontrolled videos by adapting a pre-trained video generator to predict and fuse complementary geometric modalities. Trained only on synthetic data, it generalizes to real data and outperforms state-of-the-art methods for video depth and camera rotation prediction, especially in dynamic scenes.
Problem
Monocular feed-forward 4D reconstruction of dynamic scenes is challenging in uncontrolled videos, despite its importance for video understanding, computer graphics, and robotics.
Method
Geo4D adapts a pre-trained video diffusion generator to predict point, disparity, and ray maps, then fuses these modalities through optimization for dynamic 4D reconstruction.
Results
Geo4D generalizes to real data despite synthetic-only training and outperforms state-of-the-art methods on video depth and camera rotation prediction, particularly in challenging dynamic scenes.
Takeaways & Limitations
Pre-trained video generators provide useful motion and scene-geometry priors for monocular 4D reconstruction, while multimodal prediction and fusion improve generalization.
Takeaways & Limitations
The method can struggle with substantial focal-length changes or extreme camera motion, likely because its synthetic training data lacks focal-length variation, and it currently supports only monocular input.
Abstract
from arXiv · showhide
We introduce Geo4D, a method to repurpose video diffusion models for monocular 3D reconstruction of dynamic scenes. By leveraging the strong dynamic priors captured by large-scale pre-trained video models, Geo4D can be trained using only synthetic data while generalizing well to real data in a zero-shot manner. Geo4D predicts several complementary geometric modalities, namely point, disparity, and ray maps. We propose a new multi-modal alignment algorithm to align and fuse these modalities, as well as a sliding window approach at inference time, thus enabling robust and accurate 4D reconstruction of long videos. Extensive experiments across multiple benchmarks show that Geo4D significantly surpasses state-of-the-art video depth estimation methods.
1. Introduction
Geo4D addresses monocular feed-forward 4D reconstruction by adapting a pre-trained video generator to predict and fuse complementary geometric representations. The approach targets difficult dynamic scenes and reports stronger reconstruction, video-depth, and camera-orientation performance.
- Motivation: Monocular 4D reconstruction is especially difficult in uncontrolled videos because dynamic scenes are more ambiguous than static 3D scenes.The setting includes handheld-camera and Internet videos, with applications spanning video understanding, computer graphics, and robotics.
- Motivation: Existing dynamic reconstruction models require substantial real 3D annotations, while synthetic data is difficult to scale and may generalize poorly to real scenes.These limitations motivate using priors from pre-trained video generators.
- Approach: Geo4D adapts an off-the-shelf video generator to recover explicit 4D geometry, including camera motion and dynamic 3D structure.The method builds on video models whose generated videos encode effects such as camera motion, perspective, and object motion.
- Approach: The model predicts viewpoint-invariant point maps, disparity maps, and camera ray maps to represent complementary aspects of dynamic scene geometry.Point maps align static scene regions across frames, while disparity and ray maps provide better dynamic range or remain defined independently of scene geometry.
- Approach: At test time, Geo4D reconciles the modalities through fast global optimization, incorporating uncertainty maps into multi-modal alignment.The authors report more robust 4D reconstruction and strong results in video depth estimation and camera-orientation recovery.
- Results: Geo4D reconstructs highly dynamic scenes and outperforms current video depth and camera rotation estimation methods.The reported contribution emphasizes improved 4D prediction accuracy through lightweight multi-modal fusion.
2. Related Work
Prior work spans static and iterative 3D/4D reconstruction, feed-forward dynamic reconstruction, and diffusion-based geometric prediction. Geo4D differs by using a diffusion-driven feed-forward framework for dynamic 4D reconstruction without per-video bundle adjustment or external depth models.
- Static and iterative reconstruction: Static reconstruction methods use representations such as voxels, meshes, point clouds, implicit fields, and 3D Gaussian Splatting, but primarily target static scenes.DUSt3R introduced point maps for scene-level 3D reconstruction, which Geo4D also uses as a foundation.
- Static and iterative reconstruction: Iterative 4D methods fit monocular videos through optimization, while NeRF-based approaches can be computationally expensive.Recent methods such as MegaSaM and Uni4D achieve dynamic reconstruction or pose estimation but rely on additional priors or multi-stage optimization.
- Feed-forward dynamic reconstruction: Feed-forward dynamic reconstruction methods generate multi-view videos or adapt static reconstruction networks to recover temporally consistent 4D representations.Examples include L4GM, Animate3D, and MonST3R, while Geo4D uses video diffusion instead.
- Diffusion-based geometric prediction: Pre-trained diffusion models have been adapted for tasks including view synthesis, depth, normals, and camera-parameter prediction, but most such methods assume static environments.Geo4D addresses dynamic 4D reconstruction, where object motion adds substantial difficulty.
- Diffusion-based geometric prediction: GeometryCrafter uses point-map latent modeling but requires segmentation and tracking to recover global point maps and camera poses.Geo4D instead presents a diffusion-driven feed-forward framework that avoids per-video bundle adjustment and external depth estimation models.
3. Method
Geo4D uses a video diffusion model to predict complementary geometric representations from monocular videos, then aligns and fuses them into coherent 4D reconstructions. Its inference procedure handles long videos through overlapping temporal windows and uncertainty-aware global alignment.
- 3.2. Multi-modal Geometric 4D Diffusion: The network maps a monocular video to per-frame disparity maps, viewpoint-invariant point maps, and camera ray maps.These modalities collectively represent dynamic scene geometry and time-varying camera parameters without camera inputs.
- 3.1. Preliminaries: Video Diffusion Model: Geo4D builds on a pre-trained video diffusion model to exploit motion and scene-geometry priors encoded in video generation.The model uses latent diffusion with a VAE representation and denoising network.
- 3.2. Multi-modal Geometric 4D Diffusion: Viewpoint-invariant point maps express each pixel’s 3D point in the first camera’s reference frame, allowing static points across frames to align.They also support recovery of camera parameters and scene depth through alignment with a pinhole camera model.
- 3.2. Multi-modal Geometric 4D Diffusion: Disparity and ray maps complement point maps by providing better dynamic range and ray representations defined independently of scene geometry.Disparity is inverse depth, while ray maps encode camera parameters for all image pixels.
- 3.3. Multi-Modal Alignment: At inference, overlapping sliding windows are aligned and fused using group-wise global optimization over transformations, scales, and uncertainty-weighted point-map residuals.The temporal stride determines the overlapping clips used to reconstruct the full video.
4. Experiments
Geo4D is evaluated on synthetic and real video benchmarks for depth, camera pose, and 4D reconstruction. It consistently improves reconstruction quality and achieves strong efficiency through multi-modal fusion and sliding-window inference.
- Experimental Setup: Geo4D is trained exclusively on five synthetic datasets and evaluated on Sintel, KITTI, and Bonn under the MonST3R protocol.The evaluation includes synthetic and real-world video sequences with globally aligned depth metrics.
- Video Depth Estimation: The method outperforms state-of-the-art video depth methods, including DepthCrafter and MonST3R, across the reported benchmarks.This includes methods based on video diffusion and dynamic 3D reconstruction.
- Video Depth Estimation: 24.0% lower Abs Rel than DepthCrafter on Sintel and 17.3% lower Abs Rel on KITTI are reported despite Geo4D solving a more general problem.Geo4D and DepthCrafter use the same DynamiCrafter video diffusion model.
- Camera Pose Estimation: Geo4D achieves much better camera rotation prediction and comparable camera translation prediction than existing non-generative alternatives.Evaluation uses ATE, RPE-T, and RPE-R after trajectory alignment with the Umeyama algorithm.
- 4D Reconstruction: Group-wise alignment improves temporal consistency and produces more stable, globally coherent 4D reconstructions in highly dynamic scenes.Qualitative comparisons show improved tracking of fast-moving objects and correct reconstruction of reflections in water.
- Ablation Study: Multi-modal training and inference improve performance: all three modalities outperform point-map-only training, while multi-modal alignment achieves the best inference result.The modalities are point maps, disparity maps, and ray maps.
- Ablation Study: Shorter sliding-window strides improve results by averaging more stochastic estimates, while stride s = 4 balances runtime and performance.Under the same setting, Geo4D is 1.27 times faster than MonST3R, which requires 2.41 seconds per frame.
5. Discussion and Conclusion
Geo4D adapts a pre-trained video generator for dynamic 4D reconstruction, generalizing to real data despite synthetic-only training. It improves video depth and camera rotation prediction, while point-map accuracy remains a bottleneck.
- 5. Discussion and Conclusion: Geo4D adapts a pre-trained video generator for dynamic 4D reconstruction and generalizes to real data despite synthetic-only training.The approach also predicts multiple modalities and fuses them at test time via optimization.
- 5. Discussion and Conclusion: Point-map encoder-decoder accuracy remains a bottleneck for overall reconstruction quality.
- 5. Discussion and Conclusion: The approach opens a path toward integrating 4D geometry into video foundation models for applications such as text-driven 3D animation and world-model signals.
6. Implementation Details
Geo4D is trained on five synthetic datasets and reconstructs geometry through modality prediction followed by alignment and camera initialization. Its optimization combines multiple modality-specific losses with fixed weights.
- 6. Implementation Details: Geo4D training uses five synthetic datasets: Spring, BEDLAM, PointOdyssey, TarTanAir, and VirtualKitti.
- 6. Implementation Details: The method predicts three modality maps for each video clip, roughly aligns point maps with Umeyama, and initializes camera parameters using projection-error minimization and RANSAC PnP.
- 6. Implementation Details: The joint optimization weights the four loss terms as α1 = 1, α2 = 2, α3 = 0.005, and α4 = 0.015.
- 6. Implementation Details: Table 6 reports an ablation study of DDIM sampling steps on the Sintel dataset.
7. Additional Analysis
Additional analyses examine inference denoising, point-map VAE fine-tuning, and complementary geometric modalities. The results support efficient sampling, fine-tuning benefits, and multimodal fusion for difficult geometric cases.
- 7. Additional Analysis: Geo4D achieves optimal performance after around 5 denoising steps, reflecting the more deterministic nature of 4D reconstruction compared with video generation.
- 7. Additional Analysis: Additional qualitative results visualize geometric modality maps and reconstruction behavior across the analyzed scenes.
- 7. Additional Analysis: Fine-tuning the point-map VAE improves both video depth estimation and camera pose estimation over the original pre-trained image VAE.
- 7. Additional Analysis: Point and disparity maps are complementary: disparity better represents near objects, while point maps are more depth-agnostic.
- 7. Additional Analysis: Ray maps help infer camera pose when point maps fail to represent points at infinity.
8. Visualization
Visualization results cover indoor, outdoor, and driving scenes, showing that a model trained only on synthetic datasets generalizes to real-world data with diverse objects and motions.
- 8. Visualization: Figure 5 visualizes indoor, outdoor, and driving scenes with diverse objects and motions.The model was trained only on synthetic datasets yet generalizes to real-world data.
9. Limitations
Geo4D can struggle with significant focal-length changes or extreme camera motion. The authors attribute this limitation partly to missing focal-length variation in training data and currently support only monocular video input.
- Geo4D can struggle when focal length changes significantly or camera motion is extreme throughout a sequence.The authors suggest adding training sequences with more diverse camera movements and zooming effects.
- The method currently supports only monocular video because its architecture uses temporal attention.Extending it to multi-view images or videos is identified as future work.