PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image

Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, Xiaoxiao Long

arXiv:2403.12013v1cs.CV

TL;DR

Monocular geometry estimation is ill-posed and constrained by limited-diversity, low-quality datasets, motivating methods that generalize across scenes while preserving geometric detail. GeoWizard extends stable diffusion to jointly predict depth and normal, and separates scene distributions to address layout ambiguity. It achieves state-of-the-art zero-shot depth and normal prediction and supports downstream geometry applications, while remaining limited by slow iterative denoising and video inconsistency from random initialization.

  • Problem

    Monocular geometry estimation is ill-posed, while public datasets have limited diversity and poor accuracy across scenarios.

  • Method

    GeoWizard extends stable diffusion with joint depth-normal prediction and a scene distribution decoupler for distinct scene layouts.

  • Results

    GeoWizard achieves state-of-the-art zero-shot estimation for both depth and normal.

  • Takeaways & Limitations

    The model enhances downstream applications including 3D reconstruction, 2D content creation, and novel-view synthesis.

  • Takeaways & Limitations

    The framework is limited by time-consuming iterative denoising and inconsistencies in video sequences caused by randomly initialized noise.

Abstract

from arXiv · show

We introduce GeoWizard, a new generative foundation model designed for estimating geometric attributes, e.g., depth and normals, from single images. While significant research has already been conducted in this area, the progress has been substantially limited by the low diversity and poor quality of publicly available datasets. As a result, the prior works either are constrained to limited scenarios or suffer from the inability to capture geometric details. In this paper, we demonstrate that generative models, as opposed to traditional discriminative models (e.g., CNNs and Transformers), can effectively address the inherently ill-posed problem. We further show that leveraging diffusion priors can markedly improve generalization, detail preservation, and efficiency in resource usage. Specifically, we extend the original stable diffusion model to jointly predict depth and normal, allowing mutual information exchange and high consistency between the two representations. More importantly, we propose a simple yet effective strategy to segregate the complex data distribution of various scenes into distinct sub-distributions. This strategy enables our model to recognize different scene layouts, capturing 3D geometry with remarkable fidelity. GeoWizard sets new benchmarks for zero-shot depth and normal prediction, significantly enhancing many downstream applications such as 3D reconstruction, 2D content creation, and novel viewpoint synthesis.

1 Introduction

GeoWizard addresses monocular 3D geometry estimation as an ill-posed problem limited by low-diversity and poor-quality datasets. It uses diffusion priors, jointly predicts depth and normal, and decouples scene distributions to improve zero-shot geometry estimation across scenarios.

  • Motivation: Monocular depth and surface-normal estimation is fundamental to 3D vision but geometrically ambiguous because 3D scenes are projected into 2D images.The task supports applications including autonomous driving, 3D reconstruction, novel-view synthesis, and inverse rendering.
  • Motivation: Public datasets limit progress through low diversity across scenarios and poor accuracy from pseudo-label generation.Existing datasets often focus on autonomous driving or indoor environments, while pseudo labels can be inaccurate.
  • Approach: GeoWizard uses diffusion priors from a pretrained stable diffusion model to estimate high-quality depth and normal for images across varied scenarios.The stated goal includes real, out-of-domain, and AIGC-generated images.
  • Approach: GeoWizard jointly estimates depth and normal with a geometry switcher, enabling information exchange and consistency between the two representations.The unified framework avoids treating depth and normal as separate estimation tasks.
  • Approach: The scene distribution decoupler separates mixed scene data into sub-distributions such as outdoor, indoor, and background-free objects to resolve layout ambiguities.This guides the model toward distinct scene layouts and supports faithful geometric detail capture.
  • Results: GeoWizard achieves state-of-the-art zero-shot estimation for both depth and normal and enhances downstream applications.The paper highlights 3D reconstruction, 2D content creation, and novel-view synthesis.

2 Related Work

Prior work applies discriminative or diffusion-based models to depth and normal estimation, but remains constrained by limited training scopes, computational costs, or separate modality models. GeoWizard instead uses generative diffusion priors for joint geometry estimation.

  • Joint Depth and Normal Estimation: Prior joint depth-normal methods use geometric constraints, solvers, or multiple branches to exchange information between modalities.These approaches include deriving normals from depth, joint refinement, and latent-feature propagation.
  • Joint Depth and Normal Estimation: Existing joint-estimation approaches rely on discriminative models and limited training datasets, resulting in poor generalization and limited geometric detail capture.The related-work discussion contrasts these limitations with GeoWizard’s generative approach.
  • Diffusion Models for Geometry: Diffusion models have been applied to optical flow, view synthesis, depth estimation, and normal estimation.This establishes diffusion as a broader approach for 3D-related prediction tasks.
  • Diffusion Models for Geometry: Diffusion-based depth methods still face low efficiency and slow convergence when scaled to internet-scale data.The cited methods use improved diffusion processes or perception backbones but retain substantial resource demands.
  • Diffusion Models for Geometry: JointNet’s model size and resource costs increase linearly with the number of estimated modalities.Other methods instead model selected modality combinations or train depth and normal diffusion models separately.

3 Methodology

GeoWizard extends pretrained stable diffusion to jointly estimate depth and surface normals, while using cross-domain interaction and scene-specific conditioning to resolve geometric ambiguity across layouts.

  • Geometric Diffusion Model: GeoWizard uses a single diffusion model with geometry switchers to generate depth and normal maps jointly from an input image.The switchers indicate the target geometry domain, while shared modeling supports mutual information exchange between representations.
  • Geometric Diffusion Model: The model conditions generation through both CLIP image embeddings and latent concatenation with geometric latents.CLIP provides global guidance through cross-attention, while latent concatenation offers more precise control and reduces generation randomness.
  • Geometric Transformer Block: Cross-domain geometric self-attention combines depth and normal features to encourage spatial alignment, geometric consistency, and faster convergence.Queries are formed within each modality, while keys and values incorporate concatenated depth-normal features.
  • Scene Distribution Decoupler: GeoWizard decouples mixed scene geometry into indoor, outdoor, and object sub-distributions using one-hot scene prompts encoded as positional embeddings.The prompts are added to the time embedding to guide diffusion toward the spatial layout of each scene type.
  • Loss Function: Multi-resolution noise is used to preserve low-frequency geometric details, and both geometry branches share the same time-step scheduler.The paper reports that this noise strategy is more efficient than a single-scale schedule.

4 Experiment

GeoWizard is evaluated on diverse zero-shot depth and normal benchmarks, qualitative scene domains, geometric consistency, and downstream applications. The experiments report strong geometric detail, spatial structure, and usefulness for reconstruction, novel-view synthesis, and content generation.

  • Evaluation Setup: The evaluation covers six zero-shot relative-depth benchmarks and five zero-shot surface-normal benchmarks across indoor, outdoor, and object domains.Depth uses NYUv2, KITTI, ETH3D, ScanNet, DIODE, and OmniObject3D; normal evaluation uses NYUv2, ScanNet, iBim-1, DIODE-outdoor, and OmniObject3D.
  • Depth Estimation: DepthAnything achieves the best quantitative results on three real datasets but drops significantly on unreal images, while GeoWizard captures richer geometric details and foreground-background relationships.The comparison attributes DepthAnything’s weaker unreal-image generalization to its discriminative nature and contrasts GeoWizard with Marigold in outdoor scenes.
  • Normal Estimation: GeoWizard achieves superior surface-normal performance and recovers finer-grained details and more robust structures than DSINE on unseen terrain.Reported examples include hairline, architectural texture, and limbs, which DSINE struggles to reproduce at high frequency.
  • Ablation: Removing the geometry switcher decreases geometric consistency, especially in far regions, while the Distribution Decoupler prevents scene-layout errors across domains.Without distribution decoupling, depth and normal predictions can incorrectly perceive layouts such as Earth or Sky.
  • Downstream Applications: GeoWizard produces finer details and more accurate spatial structures for 3D reconstruction, better coherent novel views, and content-generation outputs that preserve the input’s 3D structure.The downstream evaluations compare against Omnidata v2, Midas V3.1, DepthAnything, and DSINE in the respective applications.

5 Conclusion

The conclusion presents GeoWizard as a diffusion-based model for high-fidelity monocular geometry estimation. It jointly predicts depth and normal while decoupling scene distributions to maintain correct spatial layouts, and identifies faster inference as future work.

  • Conclusion: GeoWizard distills knowledge from pre-trained stable diffusion to improve high-fidelity depth and normal estimation.The model uses diffusion priors rather than relying only on scaling data and computation.
  • Conclusion: A geometry switcher enables one model to jointly produce depth and normal, while scene-distribution decoupling supports correct spatial layouts across scene types.The conclusion describes mixed scene distributions as being separated into distinct sub-distributions.
  • Conclusion: Reducing denoising steps is identified as future work, with latent consistency models proposed as a route toward inference below one second.The stated target is to decrease inference time to less than 1 second.
  • Conclusion: The supplementary material provides additional implementation details, experiments, qualitative comparisons, and discussion of limitations and potential negative impact.These materials are organized in Appendices B through E.

B.1 Data Preprocessing

The preprocessing standardizes image resolution while preserving aspect ratio through resizing and cropping, and applies domain-specific photometric and grayscale augmentations.

  • B.1 Data Preprocessing: Samples are standardized to 576×768 by resizing the shorter side to 576 and randomly cropping along the longer side.This procedure is intended to blend samples from different scene distributions while maintaining the original aspect ratio.
  • B.1 Data Preprocessing: Photometric distortion and greyization probabilities are assigned separately for indoor, outdoor, and object-level data.The distortion probabilities are 0.05, 0.1, and 0.05, while greyization probabilities are 0.1, 0.2, and 0.1, respectively.

B.2 Our Synthetic Urban Dataset

The synthetic urban dataset replaces low-resolution Virtual KITTI normals with high-resolution Unreal Engine city samples covering varied entities and environmental conditions.

  • B.2 Our Synthetic Urban Dataset: Virtual KITTI was excluded because its 375×1242 depth maps produced low-quality normal maps.The authors instead created high-resolution urban samples using Unreal Engine.
  • B.2 Our Synthetic Urban Dataset: The replacement urban samples have 1440×3840 resolution and derive normal maps from depth using a least-square algorithm.The synthetic data spans varied city entities under different environmental conditions.

C.1 Limitation on Normal GT

The supplementary evaluation identifies substantial noise and missing fine detail in several normal-map ground truths, limiting the interpretability of quantitative comparisons.

  • Normal ground truths in traditional benchmarks contain noise that can affect measurement precision.This issue was observed during zero-shot testing and is explicitly linked to evaluation precision.
  • NYUv2: NYUv2 normal maps miss details such as book outlines, shelf edges, folds, and flat-wall orientations.The maps may therefore misrepresent both object boundaries and planar surfaces.
  • iBims-1 and ScanNet normal maps also suffer from limited resolution, surface undulation, or poor fine-detail capture.These defects further reduce confidence in benchmark-based normal comparisons.
  • Quantitative comparisons in the main paper may only partially reflect the ground truth.The stated caveat follows from the documented quality problems in the benchmark normal maps.

C.2 More Ablation Studies

Additional ablations and qualitative studies examine scene indicators, geometry-switcher alternatives, and downstream outputs. The results support the specialized scene-guidance design while exposing evaluation and transformation caveats.

  • Scene indicators: Wrong scene-domain indicators reduce both depth and normal performance, especially when object indicators are used on indoor or outdoor tests.Geometric consistency can nevertheless remain stable or improve, changing from 14.7 to 14.4 for an indoor test with an outdoor indicator.
  • Geometry-switcher ablation: Replacing the geometry switcher with shared geometry causes slower convergence and lower depth and normal quality.The reported quality changes are 6.7→7.2 and 14.8→15.3, while geometric consistency remains relatively unchanged.
  • Zero-shot qualitative comparisons: Across seven zero-shot datasets, GeoWizard consistently produces more detailed high-frequency depth and normal structures than the compared baselines.The difference is less discernible on OmniObject3D because its object structures are comparatively simple.
  • In-the-wild evaluation: In-the-wild comparisons show high-fidelity details and correct spatial layouts relative to Marigold and DepthAnything.The supplementary figures cover publicly available, daily-life, and AI-generated images, with manual scale and shift adjustment for clearer depth visualization.
  • 3D reconstruction: GeoWizard-based reconstructions recover high-frequency details and structural layouts that align more closely with the input image.Examples include hair, clothing folds, metal and wood textures, and thin handrails, using only normal maps as input to BiNI.
  • Novel view synthesis: GeoWizard outperforms MiDaS V3.1 for coherent novel views of challenging objects, while its difference from DepthAnything is relatively minor.Manual conversion to inverse depth introduces accuracy loss because the downstream method was pretrained on inverse depth.

E Limitation and Potential Negative Social Impact

The authors describe practical limitations of diffusion-based geometry estimation and potential misuse risks. They propose restricting public release through academic-use licensing.

  • Limitations: Iterative denoising is time-consuming when GeoWizard is applied to large-scale collections.This limits deployment efficiency for batch processing.
  • Limitations: Randomly initialized depth and normal noise can produce inconsistencies across video sequences.The limitation specifically concerns temporal consistency rather than single-image estimation.
  • Limitations: Pseudo scale and shift derived from combined depth and normal maps may be inaccurate in some reconstructions.This constrains reconstruction accuracy in affected cases.
  • Potential negative social impact: The model could be used to create fake but realistic 3D assets or support surveillance applications.The authors identify these as concerns associated with making the model publicly available.
  • Risk mitigation: The authors plan to limit code applications to academic research through license stipulations.The proposed licensing measure is intended to mitigate the identified misuse concerns.
Loading 2403.12013v1…