Source-linked AI summary
Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion
Jiayi Yuan, Na Zhao, De Wen Soh
TL;DR
Unguided depth completion lacks explicit visual cues, while image-guided methods depend on RGB inputs. GUDC generates depth-conditioned pseudo-images and uses distillation and semantic attention to guide completion. Experiments on KITTI and NYUv2 demonstrate the approach's effectiveness and generalization.
Problem
Unguided depth completion lacks explicit visual cues, while RGB-guided methods depend on image quality and alignment.
Source: §1 · pp. 1–2Method
GUDC generates depth-conditioned pseudo-images with ControlNet, aligns sparse-depth representations to dense-depth teacher features, and uses semantic attention to fuse pseudo-image cues.
Source: §1 · pp. 2–3Results
Experiments on KITTI and NYUv2 demonstrate GUDC's effectiveness and generalization.
Source: §5 · p. 14Takeaways & Limitations
GUDC converts unguided completion into a semantically guided setting using generated pseudo-images rather than real RGB guidance.
Source: §1 · pp. 2–3
Abstract
from arXiv · showhide
Guided depth completion methods heavily depend on RGB quality and alignment, while unguided ones often suffer from limited precision due to the absence of explicit visual cues. In this paper, we present Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion (GUDC), a new completion paradigm that innovatively bridges advanced 2D generative models with unguided depth completion, enabling semantics-aware depth inference without real RGB inputs. Our key idea is to exploit ControlNet's powerful depth-conditioned generation capability to synthesize pseudo-images directly from sparse depth, effectively converting the original unguided setting into a semantics-guided one. To address the potential image-depth misalignment caused by depth sparsity, we propose a multi-level dense-to-sparse representation distillation strategy for ControlNet fine-tuning, where dense-depth features act as teacher signals to distill consistent structural representations for sparse-depth inputs. Furthermore, during pseudo-image-guided completion, we propose a pseudo-image semantic attention fusion module to adaptively extract informative semantic cues from pseudo-images while suppressing artifacts (e.g., texture hallucinations). Extensive experiments on KITTI and NYUv2 validate that our GUDC achieves superior accuracy and robustness over existing methods.
1 Introduction
RGB-guided completion depends on image availability and quality, while unguided methods lack explicit visual cues. GUDC generates depth-conditioned pseudo-images and uses distillation and semantic attention to provide structural and semantic guidance without real RGB inputs.
1 Introduction
- RGB-guided completion depends on reliable images, while unguided approaches can lack visual cues for recovering boundaries and structural detail.RGB inputs may be unreliable or unavailable in settings such as nighttime driving, low-light environments, or LiDAR-only sensing.Source: §1 · pp. 1–2
- GUDC uses depth-conditioned ControlNet to generate geometrically consistent pseudo-images as explicit semantic and structural guidance for unguided completion.The approach presents pseudo-images as guidance during both training and inference.Source: §1 · pp. 2–3
- Multi-level dense-to-sparse distillation fine-tunes ControlNet by using dense-depth features as teacher signals for sparse-depth representations.The strategy addresses image-depth misalignment under sparse conditions and seeks structural consistency and generation fidelity.Source: §1 · p. 3
- The framework is model-agnostic and integrates with diverse completion backbones; experiments on indoor and outdoor benchmarks demonstrate effectiveness and robustness in challenging scenarios.The paper describes the framework as plug-and-play.Source: §1 · p. 3
- PSAF uses diffusion features as semantic anchors to guide affinity-based image-feature correction and cross-modality fusion while mitigating pseudo-image uncertainty.The identified uncertainties include texture hallucinations and spurious edges.Source: §1 · p. 3
2 Related Work
Prior depth-completion work either relies on aligned RGB images or uses unguided geometric and implicit visual cues, each with stated limitations. GUDC draws on depth-conditioned diffusion to generate pseudo-images as explicit guidance for completion.
2 Related Work
- RGB-guided methods fuse aligned images with sparse depth, but their applicability is limited when RGB quality or calibration is poor or images are unavailable.Earlier image-processing methods can over-smooth and generalize poorly in complex scenes.Source: §2 · pp. 3–4
- Unguided methods use masks or confidence maps to handle sparse measurements, but purely geometric approaches can struggle in complex scenes without visual semantics and structural priors.The review also describes auxiliary image branches and latent image features, which still lack explicit visual guidance at inference.Source: §2 · p. 4
- Diffusion models synthesize fine-grained images, and ControlNet enables spatially aligned generation from structured inputs such as sparse depth.The paper identifies generative guidance for unguided depth completion as largely unexplored.Source: §2 · p. 4
- GUDC generates a pseudo-image from sparse depth, then combines it with diffusion-derived semantic cues and depth to predict dense depth.The pseudo-image supplies structural and contextual guidance, while Stable Diffusion serves as a semantic encoder.Source: §2 · p. 4, Figure 2 · p. 5
3 Method
GUDC generates pseudo-images from sparse depth to provide semantic guidance without real RGB inputs. It aligns depth-conditioned generation through dense-to-sparse distillation, then uses semantic features and adaptive fusion to incorporate pseudo-image guidance into depth completion.
3.1 Overview
- GUDC generates pseudo-images from sparse depth, turning unguided completion into semantically guided completion without requiring real RGB inputs.The framework is model-agnostic and can be incorporated into existing guided depth-completion architectures.Source: §3.1 · pp. 5–6
3.2 Geometry-aligned Pseudo-image Generation
- Dense-depth teacher features guide a sparse-depth ControlNet student, addressing unstable geometry and poor image-depth consistency under sparse conditioning.The teacher is pretrained on dense ground-truth depth; multi-level feature distillation is applied during fine-tuning alongside sparse-depth denoising and pixel-level supervision.Source: §3.2 · p. 7
3.3 Pseudo-image Semantic-guided Completion
- Diffusion-derived semantic features correct pseudo-image feature distortions, while a multi-source affinity kernel aggregates features according to structural, semantic, and geometric similarity.The selected Stable Diffusion U-Net decoder features come from layer 6, balancing global context with local structural detail.Source: §3.3 · p. 9
- Depth features query enhanced pseudo-image features through cross-attention, and the fused representation feeds the decoder for dense depth prediction.This replaces direct feature concatenation with adaptive image-depth interaction.Source: §3.3 · p. 10
Experiment
The experiments assess completion accuracy on KITTI and NYUv2 and robustness under adverse visual conditions, then examine the effects of fine-tuning, fusion, and backbone choices.
4.1 Experimental Setup
- Experiments use KITTI with approximately 5% valid depth pixels and NYUv2 with 200 or 500 sampled points per frame; ControlNet tuning uses 6K and 5K pairs, respectively.Depth completion uses BP-Net and less than 30% of the available training data.Source: §4.1 · p. 11, Experiment · p. 11
4.2 Comparison with SOTAs
- GUDC leads KITTI across all metrics, outperforms prior unguided methods on NYUv2 at both sparsity levels, and beats guided baselines in fog and nighttime.The comparisons are on the KITTI test set, NYUv2 with 200 or 500 points per frame, and Foggy-KITTI and Nighttime-KITTI.Source: Table 1 · p. 11, Table 2 · p. 11, Table 3 · p. 11
- Qualitative results show sharper depth boundaries and finer structural reconstructions on NYUv2, while GUDC recovers sharper structures and more coherent depth under visual corruption.The NYUv2 examples include furniture edges and room layouts; guided methods show blurred boundaries or distorted structures in adverse conditions.Source: Figure 7 · p. 12, §4.2 · pp. 12–13, Figure 8 · p. 12
4.3 Ablation Study
- Distillation-based ControlNet tuning produces clearer geometry and better depth-layout consistency than naïve tuning, with faster convergence and improved performance.Multi-level distillation improves over single-layer alignment, while increasing aligned layers from 2 to 3 gives only marginal gains.Source: §4.3 · p. 13, Table 4 · p. 13, Figure 9 · p. 14
- ControlNet fine-tuning performance improves with more training data but saturates around 6K samples; even 1K samples yields satisfactory performance.The study therefore uses 6K samples as its default configuration.Source: §4.3 · pp. 13–14
- The full PSAF module performs best, while removing either component causes noticeable performance drops.The ablation compares full PSAF, direct feature concatenation, and variants missing either the image-feature correction unit or global image-depth interaction block.Source: Table 4 · p. 13, §4.3 · p. 14
- The generative completion framework serves as a plug-and-play enhancement across BP-Net, MSG-CHN, and DFUNet.The passage describes these as representative backbones with differing architectural designs.Source: §4.3 · p. 14
5 Conclusion
GUDC uses depth-conditioned ControlNet pseudo-images as structural and semantic guidance for unguided depth completion without real RGB. Dense-to-sparse distillation and semantic-attention fusion address image-depth misalignment and hallucinated artifacts, and experiments on KITTI and NYUv2 demonstrate effectiveness and generalization.
5 Conclusion
- GUDC uses depth-conditioned ControlNet to generate geometrically aligned pseudo-images that guide unguided depth completion without real RGB.The pseudo-images provide structural and semantic guidance.Source: §5 · p. 14
- Multi-level dense-to-sparse representation distillation addresses misalignment caused by depth sparsity and boosts image quality.Source: §5 · p. 14
- Pseudo-image Semantic-Attention Fusion extracts reliable semantic cues while suppressing hallucinated artifacts.Source: §5 · p. 14
- Experiments on KITTI and NYUv2 demonstrate the method's effectiveness and generalization capability.Source: §5 · p. 14