Source-linked AI summary
Video Generative Models as Geometry Learner
Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng
TL;DR
Monocular depth and surface-normal estimation remains difficult, while prior image-diffusion approaches either separate tasks or require architecture changes and more labeled data. GeoNeXt repurposes pretrained video diffusion as unified next-frame prediction with joint image–geometry modeling. It achieves competitive or superior zero-shot results across diverse datasets using substantially less training data, including nearly 100× less than DepthAnything.
Problem
Prior image-diffusion geometry methods either train depth and normal models separately or modify diffusion architectures, limiting joint modeling and increasing data requirements.
Method
GeoNeXt repurposes a pretrained video generative model and formulates depth and surface-normal estimation as next-frame prediction conditioned on an RGB image.
Results
GeoNeXt outperforms or matches task-specific and unified generative competitors across diverse datasets and rivals discriminative models trained on much more data.
Takeaways & Limitations
Video generative priors support unified, data-efficient zero-shot estimation of depth and surface normals across diverse datasets.
Takeaways & Limitations
The approach adapts video-model priors to geometry without explicit geometric supervision during pretraining.
Abstract
from arXiv · showhide
Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.
1 Introduction
GeoNeXt addresses limitations in monocular geometry estimation by repurposing pretrained video generative models for unified next-frame prediction of images, depth, and normals. The approach leverages video priors and joint image–geometry modeling to improve data efficiency.
- Motivation: Monocular geometry estimation remains challenging because recovering precise 3D structure from one image requires comprehensive geometric reasoning.Existing approaches also face generalization and fine-detail limitations when relying on large-scale pseudo-labeled data.
- Limitations of prior work: Prior image-diffusion methods either train separate task-specific geometry models or modify diffusion architectures for joint prediction.The separate-model track misses joint target correlations, while architectural modifications increase the gap between pretraining and fine-tuning.
- Motivation: Video generative models provide richer priors from image-and-video pretraining and temporal-attention priors that capture cross-frame dependencies.These priors are adapted to model interactions between images and geometry targets.
- GeoNeXt: GeoNeXt reformulates geometry estimation as next-frame prediction, modeling depth and surface normals as frames conditioned on an RGB image.The input image is replicated across geometry slots, while image and geometry are synthesized in lockstep.
- Contributions: GeoNeXt jointly estimates depth and surface normals with a unified video-generative geometry learner and is evaluated for zero-shot estimation using substantially less training data.The experiments compare against task-specific and unified generative competitors and discriminative models trained with much larger datasets.
2 Related Work
Prior work spans discriminative, diffusion-based, and video-based approaches to monocular depth and normal estimation. GeoNeXt extends pretrained video diffusion toward unified joint geometry prediction while targeting generalization and data efficiency.
- Monocular depth and normal estimation: Depth and surface normals provide complementary information about 3D scene geometry, but monocular estimation is ill-posed and difficult in unconstrained environments.Generalization beyond the training distribution remains a practical challenge.
- Monocular depth and normal estimation: Recent methods improve generalization through large diverse datasets, but metric depth depends on camera intrinsics and is often replaced by affine-invariant depth.Absolute metric-depth models may require fixed cameras or explicit intrinsic-parameter conditioning.
- Monocular depth and normal estimation: Discriminative multi-task methods share a backbone while retaining task-specific decoder branches, yet often use limited data and generalize poorly.These methods can miss fine geometric details despite enabling information exchange between tasks.
- Diffusion models for geometry estimation: Diffusion-based geometry methods use pretrained latent diffusion models for depth and normal prediction, often with modest fine-tuning data but limited domain coverage.Earlier diffusion frameworks also combine geometric prediction with discriminative pipelines.
- Diffusion models for geometry estimation: Joint diffusion systems such as GeoWizard and Orchid modify architectures or retrain broader components, increasing data requirements and widening the pretraining–fine-tuning gap.GeoNeXt instead explores pretrained video diffusion for joint monocular geometry estimation.
- GeoNeXt: GeoNeXt uses a frozen image VAE and fine-tunes only its U-Net while optimizing over image, depth, and normal latents.The protocol is designed to maintain fine-grained alignment and consistency across modalities.
3 Method
GeoNeXt formulates joint monocular depth and normal estimation as image-conditioned video diffusion in latent space. It uses a pretrained video U-Net, frozen VAE encoding, replicated image conditioning, and simultaneous image–geometry denoising.
- 3.1 Preliminaries of Diffusion Models: Diffusion models learn data distributions through forward noising and a learned reverse denoising process.Stable Video Diffusion follows an EDM formulation that progressively denoises latent sequences from high noise.
- 3.1 Preliminaries of Diffusion Models: In EDM, Gaussian noise with variance σ_t^2 is added to a clean latent, and a preconditioned denoiser combines the noisy input with network output.Noise-dependent functions control input scaling, output scaling, the skip path, and noise embedding.
- 3.2 Geometry as Next Frames: GeoNeXt models depth and surface normals as subsequent frames following an input image and learns the conditional distribution p(g|x), where g contains both geometry targets.This converts joint geometry estimation into an image-conditioned image-to-video diffusion problem.
- 3.2 Geometry as Next Frames: The clean image–geometry triplet is encoded into latent space with a VAE and decoded back into reconstructed image, depth, and normal outputs.Depth is tiled to three channels for the RGB VAE and averaged back to one channel during decoding.
- 3.2 Geometry as Next Frames: At inference, geometry latents begin as Gaussian noise and are progressively denoised while the image is simultaneously reconstructed to maintain image–geometry consistency.Training uses the standard EDM noise-prediction objective.
- 3.2 Geometry as Next Frames: The adapted U-Net concatenates geometry latents with the image latent, replicates image conditioning across geometry slots, and uses temporal attention to propagate structural information.CLIP-based conditioning is disabled because resizing the image can distort geometry maps.
4 Experiments
Experiments show that GeoNeXt delivers strong zero-shot depth and surface-normal estimation across unseen datasets, while ablations support joint image–geometry modeling and unified prediction.
- Experimental Setup: GeoNeXt is implemented on the image-to-video variant of Stable Video Diffusion and trained using synthetic indoor and outdoor datasets.The training data include Hypersim and Virtual KITTI 2.
- Depth Estimation: GeoNeXt uses nearly 100× less data than DepthAnything while achieving highly competitive depth results and surpassing it on several benchmarks.On ETH3D, GeoNeXt achieves an AbsRel of 13.1 versus 5.6 for DepthAnything.
- Depth Estimation: GeoNeXt achieves superior or comparable depth performance to image-diffusion generative methods while jointly modeling geometric consistency in one framework.Lotus-G-depth uses a similar 59K-sample scale but requires a separate depth model.
- Depth Estimation: GeoNeXt outperforms GeoWizard on reported KITTI and DIODE depth comparisons despite using substantially less training data.The reported margins include 6.2 AbsRel and 9.4 δ1 on KITTI, plus 10.4 AbsRel and 1.8 δ1 on DIODE.
- Surface Normal Estimation: For surface normals, GeoNeXt achieves competitive or superior accuracy to discriminative baselines and the strongest overall results among unified geometry methods.Reported mean angular errors are 16.4 on iBims-1 and 33.0 on Sintel.
- Ablation Study: Ablations show that image reconstruction and joint depth–normal training improve performance, while removing CLIP conditioning slightly improves it.Swapping depth and normal reconstruction order produces nearly identical results, indicating order robustness.
- Cost-effectiveness Analysis: The unified approach provides comparable inference time and lower storage and deployment overhead than separately maintained depth and normal models.Reported runtimes exclude additional real-world GPU I/O overhead from loading separate models.
5 Conclusion
GeoNeXt repurposes pretrained video generative models as a unified, data-efficient framework for jointly estimating depth and surface normals from a single image. It formulates geometry estimation as next-frame prediction and systematically studies generative formulation, architectural adaptations, and reconstruction order.
- GeoNeXt jointly estimates depth and surface normals from a single image using a pretrained video generative model.The approach treats the video model as a unified geometry learner rather than maintaining separate task-specific models.
- Geometry estimation is reformulated as next-frame prediction, with video-model priors adapted for image–geometry joint modeling.The framework leverages temporal coherence and rich generative priors to connect images with geometry targets.
- GeoNeXt systematically examines joint generation, architectural adaptations, and robustness to reconstruction order.The study also identifies preserving fine-grained depth and surface-normal details as important to the training protocol.
- Extensive quantitative and qualitative evaluations show strong performance even with limited training data.