Source-linked AI summary
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, Konrad Schindler
TL;DR
Monocular depth estimation is geometrically ill-posed, and generalization is limited when images contain unfamiliar content and layouts. Marigold adapts Stable Diffusion into an affine-invariant depth estimator using synthetic-data fine-tuning, achieving state-of-the-art performance across varied datasets while retaining a visual scene-understanding prior.
Problem
Monocular depth estimation must recover per-pixel scene depth from a single image despite the task's geometric ill-posedness and limited generalization to unfamiliar visual content.
Method
Marigold uses a resource-efficient fine-tuning protocol to convert a pretrained Stable Diffusion latent diffusion model into an image-conditional depth estimator, trained exclusively on synthetic depth data.
Results
Marigold is reported as a state-of-the-art affine-invariant depth estimator with excellent performance across a wide variety of natural images.
Takeaways & Limitations
The results support sourcing a detailed visual scene-understanding prior for depth estimation from a pretrained text-to-image diffusion model.
Takeaways & Limitations
Synthetic-to-real domain gaps can sometimes limit generalization ability, leaving sufficient data diversity as a concern.
Abstract
from arXiv · showhide
Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learning has led to a breakthrough. The impressive progress of monocular depth estimators has mirrored the growth in model capacity, from relatively modest CNNs to large Transformer architectures. Still, monocular depth estimators tend to struggle when presented with images with unfamiliar content and layout, since their knowledge of the visual world is restricted by the data seen during training, and challenged by zero-shot generalization to new domains. This motivates us to explore whether the extensive priors captured in recent generative diffusion models can enable better, more generalizable depth estimation. We introduce Marigold, a method for affine-invariant monocular depth estimation that is derived from Stable Diffusion and retains its rich prior knowledge. The estimator can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases. Project page: https://marigoldmonodepth.github.io.
1. Introduction
Monocular depth estimation infers a depth value for every image pixel from a single photograph, an ill-posed task requiring scene understanding. Marigold adapts Stable Diffusion into a broadly applicable depth estimator using an efficient fine-tuning protocol and synthetic data.
- Monocular depth estimation regresses a range value for every pixel from a photographic image when direct range or stereo measurements are unavailable.
- Because projecting a 3D scene into 2D is geometrically ill-posed, depth estimation requires prior knowledge about objects, layouts, and occlusions.
- Marigold derives an affine-invariant depth estimator from Stable Diffusion while preserving its latent space and rich visual prior.
- The method converts a pretrained latent diffusion image generator into an image-conditional depth estimator through resource-efficient fine-tuning.
- Marigold uses only synthetic RGB-D data and a few GPU days on a single consumer graphics card for adaptation.
2. Related Work
Prior work addresses monocular depth through structured regression, diverse datasets, and diffusion or foundation-model adaptation. Marigold instead directly tunes Stable Diffusion, using broad image priors to estimate depth across diverse real-world settings with limited synthetic data.
- 2.1. Monocular Depth: Monocular depth estimation is a dense, structured regression task addressed with architectures and priors including multi-scale networks, ordinal regression, transformers, and variational methods.
- 2.1. Monocular Depth: Affine-invariant depth estimates unknown global offset and scale, offering geometrically meaningful relative differences while accommodating unfamiliar scenes.
- 2.1. Monocular Depth: Existing approaches improve generality through large diverse datasets, but handling unknown test-camera intrinsics remains outside methods requiring known cameras.
- 2.2. Diffusion Models: Latent diffusion models run denoising in an efficient latent space and can encode broad scene-understanding priors from internet-scale image data.
- 2.2. Diffusion Models: Prior diffusion-depth methods perform well in specific training domains, whereas this work targets monocular depth estimation across diverse real-world settings.
- 2.3. Foundation Models: Marigold directly tunes Stable Diffusion as a foundation model, obtaining state-of-the-art estimates on real datasets from as few as 74k synthetic depth samples.
3. Method
Marigold repurposes Stable Diffusion as an image-conditioned latent diffusion model for affine-invariant monocular depth estimation. Its protocol combines latent-space U-Net fine-tuning, synthetic depth training, modified noise scheduling, and stochastic inference with optional ensembling.
- Generative formulation: Marigold models monocular depth estimation as conditional denoising diffusion, learning the distribution of depth maps conditioned on RGB images.
- Generative formulation: Training noises depth maps at random timesteps, predicts the added noise, and minimizes a standard denoising diffusion objective.
- Fine-tuning protocol: The method starts from pretrained Stable Diffusion, freezes its VAE, and fine-tunes the U-Net with concatenated image and depth latent codes.
- Fine-tuning protocol: Depth maps are normalized using an affine transformation based on their 2% and 98% percentiles, producing a canonical representation primarily within [−1, 1].
- Fine-tuning protocol: Training uses exclusively synthetic depth because it is dense, complete, and clean, while remaining diversity and domain gaps between synthetic and real data constrain generalization.
- Inference: At inference, depth latent noise is progressively denoised from an input-image encoding, decoded through the VAE, and postprocessed by channel averaging.
- Inference: Because initialization noise changes predictions, test-time ensembling aligns multiple affine-invariant outputs and trades computation for prediction quality.
4. Experiments
Marigold is evaluated on unseen real indoor and outdoor datasets, where it generally outperforms prior affine-invariant depth estimators. Ablations show that training noise, dataset diversity, test-time ensembling, and denoising-step choices materially affect performance and efficiency.
- 4.2. Evaluation: Marigold outperforms other methods on indoor and outdoor zero-shot benchmarks in most cases without training on real depth samples.The comparison uses five real datasets not seen during training and reports the highest overall ranking.
- 4.2. Evaluation: Qualitative comparisons show better preservation of scene layout, thin structures, flat surfaces, and detailed structures.Examples include chair legs, walls, and furniture relationships across ETH3D and DIODE visualizations.
- 4.3. Ablation Studies: Multi-resolution training noise improves depth accuracy over Gaussian noise, while annealing provides an additional improvement and enhances prediction consistency.The reported consistency is measured across different initial inference noises.
- 4.3. Ablation Studies: More diverse and photorealistic synthetic training data improves performance across indoor and outdoor scenes, and adding another domain also benefits the original domain.Hypersim provides strong results, while Virtual KITTI improves outdoor performance.
- 4.3. Ablation Studies: Although trained with 1000 DDPM steps, 50 DDIM inference steps are sufficient for accurate results, and comparable performance can be retained with 10 or fewer steps.The marginal-return elbow is dataset-dependent but always under 10 steps.
5. Conclusion
Marigold combines a Stable Diffusion fine-tuning protocol with affine-invariant depth estimation, while highlighting efficiency, output consistency, and distant-scene handling as remaining challenges.
- Marigold is a fine-tuned Stable Diffusion model for state-of-the-art affine-invariant depth estimation.
- The method sources detailed visual scene-understanding priors from a pretrained text-to-image diffusion model.
- Future work should improve inference efficiency, consistency across similar inputs, and handling of distant scene parts.
A.1. Mixed Dataset Training
Training uses mixed synthetic datasets and an annealed multi-resolution noise scheme, with affine-invariant predictions aligned through least-squares fitting.
- Training mixes Hypersim and Virtual KITTI by probabilistically selecting a dataset for each batch.The datasets differ in image resolution and aspect ratio.
- Annealed multi-resolution noise reduces lower-resolution influence near the noise-free end of the diffusion schedule.The i-th pyramid level at timestep t receives weight (s^t/T)^i.
- Standard multi-resolution noise combines Gaussian noise images across pyramid resolutions through upsampling, weighted averaging, and renormalization.
- Affine-invariant predictions are aligned to ground truth using least-squares scale and shift fitting over valid pixels.Depth predictions are aligned to inverse ground-truth depth when the prediction type is disparity.
- For in-the-wild samples, metric-space unprojection requires manually estimated scale, shift, and camera intrinsics when metadata is unavailable.Camera intrinsics can sometimes be extracted from EXIF metadata.
B. Experimental Results
Supplementary experiments examine whether Stable Diffusion’s latent space can represent depth reliably, using VAE reconstruction and channel-consistency analyses.
- Stable Diffusion’s pretrained VAE was tested for depth-map reconstruction on 800 Hypersim training samples.Depth maps were normalized, replicated across three RGB channels, decoded, and averaged across channels.
- Channelwise discrepancies after VAE decoding are small relative to the decoder output range [−1, 1].The evaluation uses NYUv2 and a subsampled Eigen split of KITTI.
B.3. Prediction Variance and Training Noise
Because Marigold is generative, prediction variance depends on the initial diffusion noise; multi-resolution training improves consistency, with annealing providing further improvement.
- Marigold predictions vary with the initial noise used to start diffusion.Consistency is evaluated for models trained with Gaussian, multi-resolution, and annealed multi-resolution noise.
- Multi-resolution noise increases inference consistency, and annealed multi-resolution noise improves it further.Dataset-level statistics are reported in Table S2.
- Figure S1 compares predictions from the three noise-trained models on the same input, with the annealed multi-resolution model exhibiting the least variance.
B.4. Ratio of Mixed Training Datasets
The study finds that mixing Hypersim with a small portion of Virtual KITTI improves transfer across indoor and outdoor real datasets. Zero-shot transfer is achieved at a 10% Virtual KITTI ratio, while inference remains slower than feed-forward methods.
- Ratio of Mixed Training Datasets: Mixed Hypersim and Virtual KITTI training improves results on both indoor and outdoor real datasets.The validation datasets are NYUv2 and KITTI.
- Ratio of Mixed Training Datasets: 10% Virtual KITTI is sufficient to attain zero-shot transfer.Hypersim alone already delivers strong results, while Virtual KITTI further enhances outdoor performance.
- Inference Speed: Marigold trades slower inference for higher quality compared with feed-forward methods.The authors suggest distillation and smaller ensemble sizes as possible speed improvements.
- In-the-Wild: Marigold predicts accurate depth maps on daily-life and internet-sourced in-the-wild images.The qualitative results exhibit better overall layout and fine details.
C.2. Test Datasets
The evaluation compares monocular depth-estimation methods across five test datasets using qualitative depth and normal-map visualizations. Marigold is reported to capture fine scene details and global layout, with consistent color coding across depth maps.
- Test Datasets: Qualitative comparisons evaluate Marigold against competing methods on five test datasets.Depth maps appear in Fig. S4, and normal maps appear in Fig. S5.
- Test Datasets: Marigold excels at capturing fine scene details and reflecting the global scene layout.The comparison includes DiverseDepth, MiDaS, LeReS, DPT, and Omnidata visualizations.
- Test Datasets: In in-the-wild comparisons, Marigold and LeReS show depth while MiDaS shows disparity.Red denotes closer and blue farther distances for depth; yellow denotes closer and purple farther distances for disparity.
- Test Datasets: Depth predictions are aligned to ground truth, with consistent color coding across all depth maps.This supports visual comparison across samples and methods.
- Test Datasets: Normal-map comparisons use unprojected predictions, with ground-truth normals derived from ground-truth depth maps.The visualization provides an additional geometric view of the depth estimates.