Source-linked AI summary
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Igor Pavlovic, Thiemo Wandel, Anton Obukhov, Luca Bartolomei, Andrey Davydov, Fabio Tosi, Matteo Poggi, Sabine Süsstrunk, Dengxin Dai
TL;DR
Monocular depth estimation is ill-posed, and existing models can struggle with out-of-distribution generalization and sharp, detailed depth maps. Marigold V2 repurposes a diffusion transformer using ground-truth representation alignment and two-stage Sinkhorn-based fine-tuning. It achieves state-of-the-art depth results and extends to other dense regression tasks while retaining fine details.
Problem
Monocular depth estimation remains ill-posed, and diffusion-based estimators commonly lose fine-grained details and produce oversmoothed boundaries.
Method
Marigold V2 repurposes an image-editing diffusion transformer using ground-truth depth representation alignment and a two-stage protocol with SinkLoss.
Results
Marigold V2 achieves state-of-the-art monocular depth results and extends state-of-the-art performance to depth completion, see-through depth, surface normals, and intrinsic image decomposition.
Takeaways & Limitations
The recipe provides an accessible diffusion-based approach for high-quality computational photography and multiple dense regression tasks.
Takeaways & Limitations
The large image-editing backbone precludes real-time use, while reflections, motion, and defocus blur remain ambiguous.
Abstract
from arXiv · showhide
Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web
1 Introduction
Marigold V2 repurposes an image-editing diffusion transformer into a cost-effective monocular depth estimator, addressing detail loss through representation alignment and Sinkhorn-based refinement. It achieves state-of-the-art depth results and extends to other dense regression tasks.
- Motivation: Monocular depth estimation recovers per-pixel depth from a single image and supports applications including rendering, computational photography, scene reconstruction, and 3D content creation.These applications include novel view synthesis, bokeh simulation, relighting, image editing, object insertion, augmented reality compositing, and scene lifting.
- Contributions: Marigold V2 repurposes Qwen-Image-Edit into a state-of-the-art monocular depth estimator through a cost-effective fine-tuning protocol.The protocol uses 4-bit quantization with QLoRA and requires a single consumer GPU, a modest dataset, and a few days of training.
- Contributions: Stage 1 aligns model representations with semantic features extracted from ground-truth depth rather than RGB, supplying semantic and geometric information.The alignment is intended to ease convergence and improve visual quality.
- Contributions: SinkLoss is a Sinkhorn matching-based objective used in Stage 2 to improve edge sharpness while preserving fur, hair, and thin structures.The second-stage tuning targets fine-grained detail loss and oversmoothed boundaries in diffusion-based depth estimators.
- Results: Marigold V2 achieves state-of-the-art monocular depth results on standard benchmarks, outperforming diffusion-based alternatives and other competitors.The recipe also reaches state-of-the-art results for depth completion, see-through depth, surface normals, and intrinsic image decomposition.
2 Related Work
Related work spans discriminative and generative approaches to monocular depth estimation, with persistent constraints in data quality, inference cost, and fine-detail reconstruction. Marigold V2 builds on diffusion-transformer adaptation, representation alignment, and detail-preserving losses.
- Discriminative Monocular Depth Estimation: Discriminative depth models benefit from deep learning but remain constrained by scarce ground-truth data and annotation weaknesses on non-Lambertian, transparent, or reflective surfaces.Sensor noise and poor handling of these surfaces limit annotation quality and are inherited by trained models.
- Generative Depth Estimation: Generative depth approaches trade off inference speed, sensitivity, and quality across multi-step diffusion, single-pass feed-forward, and related families.Multi-step methods have high latency and sensitivity to noise initialization, while single-pass approaches trade quality for speed.
- Diffusion Transformers: Repurposing DiTs is more expensive than adapting earlier U-Net backbones because their higher complexity can require tens of GPUs for training.LoRA does not always remove this computational burden.
- Representation Alignment: Representation alignment with pretrained visual features improves semantic fidelity and training convergence in diffusion models and has been adapted for depth prediction.Prior work aligns intermediate features with semantic representations to bring semantic details into depth maps and facilitate convergence.
- Handling Depth Quality and Artifacts: Faithful boundary reconstruction and fine-grained surface detail remain difficult because prior edge-aware, architectural, diffusion, and distillation methods do not reliably prevent oversmoothing.These qualities directly affect novel view synthesis, 3D reconstruction, and computational photography.
3 Method
Marigold V2 adapts a pretrained image-editing diffusion transformer for single-step affine-invariant monocular depth estimation through two-stage training. Semantic feature alignment improves structural reconstruction, while SinkLoss refines boundaries and reduces artifacts caused by ambiguous supervision.
- Diffusion Transformer Adaptation: Marigold V2 repurposes a pretrained image-editing diffusion transformer to predict affine-invariant depth from a single RGB image.The model is initialized from Qwen-Image-Edit-2509 and uses normalized depth targets compatible with the RGB backbone.
- Stage 1: Stage 1 combines latent rectified-flow supervision with pixel-space reconstruction, gradient, and semantic feature losses to improve global structure and local detail.The DiT directly transforms RGB latents into depth latents, and fixed-timestep rectified flow enables single-pass inference.
- Depth Normalization: The target depth is converted to robustly clipped, normalized log-depth and encoded as a grayscale RGB image, removing global scale and shift ambiguity.The 2% and 98% log-depth quantiles define clipping before mapping values to [-1, 1].
- Semantic Feature Alignment: Ground-truth depth features outperform RGB-derived features for iREPA regularization, improving AbsRel and δ1 while strengthening semantic and structural consistency in dense regions.The depth-domain features provide more relevant information for geometric reconstruction than RGB-derived features.
- Stage 2: Stage 2 applies SinkLoss to match predicted and ground-truth depth values within local K×K blocks without requiring strict pixel-to-pixel alignment.Entropy-regularized Sinkhorn matching handles noisy or ambiguous pixels while preserving valid supervision.
4 Experiments
Experiments evaluate Marigold V2 under zero-shot depth protocols, ablations, edge-sensitive metrics, and efficiency tests. The model achieves strong accuracy and detail preservation while offering favorable resolution scalability on constrained hardware.
- Zero-Shot Affine Depth Estimation: Marigold V2 is evaluated on five zero-shot datasets using RANSAC-aligned AbsRel and δ1 metrics.The protocol removes global scale and shift mismatches before measuring scene geometry, local structure, and depth discontinuities.
- Zero-Shot Affine Depth Estimation: AbsRel 2.8 on ETH3D improves over the strongest baseline result of 3.8.The method achieves the best zero-shot performance across evaluated datasets among methods trained on comparable data.
- Edge-Aware Evaluation: Marigold V2 achieves the best results across all reported edge-aware metrics against detail-preserving and flying-pixel-suppressing methods.The comparison uses SEE3, SEE5, and SEE7 on the HyperSim test set.
- Analysis and Ablation Studies: iREPA-depth provides the strongest Stage-1 ablation performance, while pixel-space supervision consistently improves δ1 across datasets.At 160K steps, quantitative differences become less pronounced, but iREPA still visibly improves fine-structure quality.
- Analysis and Ablation Studies: Stage-2 SinkLoss substantially improves SEE3 while preserving standard metrics comparable to the Stage-1 checkpoint.The refinement also produces fewer flying pixels and cleaner discontinuities near thin structures and vegetation.
- Efficiency: At 2048 × 2048, Marigold V2 remains feasible on a single 32GB GPU while several prior diffusion estimators run out of memory.The model is not fastest overall but offers better resolution scalability than prior diffusion-based estimators.
5 Other Dense Regression Tasks
The Marigold V2 recipe transfers beyond affine-invariant depth to metric completion, see-through depth, surface normals, and albedo estimation. These adaptations retain strong benchmark performance while addressing task-specific geometry or appearance targets.
- Metric Depth Completion: Metric depth completion adapts a frozen affine-invariant prior with a test-time LoRA and learned scale and shift from sparse measurements.The approach uses a second zero-initialized rank-16 LoRA on the last 12 transformer blocks and optimizes sparse-depth residuals.
- Metric Depth Completion: Marigold V2 achieves the lowest RMSE on all four metric depth completion benchmarks.It outperforms Marigold-SSD and Marigold-DC despite using an affine-invariant prior.
- See-Through Depth: See-through depth addresses transparent surfaces where HyperSim annotations place depth at the glass rather than behind it.The base model is biased toward closer depths in these regions, motivating a specialized see-through model.
- Surface Normals: For surface normals, the adapted model replaces pixel-space supervision with angular loss while retaining iREPA and adding SinkLoss.Sinkhorn weights are computed from prediction–ground-truth L1 distances, and SinkLoss consistently improves SAEE.
- Albedo Estimation: For albedo estimation, the adapted model achieves the best PSNR and LPIPS while remaining competitive in SSIM.Training uses L1 reconstruction together with iREPA computed directly on ground-truth albedo.
6 Conclusion
Marigold V2 combines quantitative depth accuracy with perceptual detail through a two-stage diffusion-based fine-tuning recipe. The same approach extends to several dense regression tasks, although its large backbone and difficult visual conditions limit deployment scope.
- Conclusion: Marigold V2 derives from Qwen-Image-Edit through two-stage fine-tuning steered by iREPA and SinkLoss.The model targets both state-of-the-art monocular depth estimation and preservation of high-frequency details.
- Broader Applicability: The recipe extends beyond relative depth to metric depth completion, see-through depth, surface normals, and albedo estimation.These extensions cover multiple dense regression tasks within computational photography and related vision applications.
- Limitations: The large image-editing backbone precludes real-time use, while reflections, motion, and defocus blur remain ambiguous.The paper identifies uncertainty-aware or multi-layer extensions as directions for addressing these conditions.