PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Depth Posterior (DDP) from Single Image and Sparse Range

Yanchao Yang, Alex Wong, Stefano Soatto

arXiv:1901.10034v2cs.CV

TL;DR

Dense depth completion must infer scene geometry from an image and sparse range measurements despite multiple compatible depth maps. The paper learns an image-conditioned depth prior, combines it with a sparse-measurement likelihood, and optionally uses stereo during training; it achieves state-of-the-art KITTI performance in both supervised and unsupervised settings while outputting a full posterior.

  • Problem

    Images and sparse point clouds can support multiple dense depth maps, motivating posterior estimation rather than only a point estimate.

  • Method

    A Conditional Prior Network scores dense-depth compatibility from an image and training data, and the model combines this prior with a sparse-depth likelihood, optionally adding stereo supervision.

  • Results

    The method achieves state-of-the-art performance on both supervised and unsupervised KITTI depth completion while providing confidence measures over depth.

  • Takeaways & Limitations

    The system produces a full posterior that supports confidence reasoning and downstream processing beyond a single depth estimate.

  • Takeaways & Limitations

    Real-world deployment may introduce time-varying sensor misalignment, calibration faults, and other nuisance variability not represented in KITTI evaluation.

Abstract

from arXiv · show

We present a deep learning system to infer the posterior distribution of a dense depth map associated with an image, by exploiting sparse range measurements, for instance from a lidar. While the lidar may provide a depth value for a small percentage of the pixels, we exploit regularities reflected in the training set to complete the map so as to have a probability over depth for each pixel in the image. We exploit a Conditional Prior Network, that allows associating a probability to each depth value given an image, and combine it with a likelihood term that uses the sparse measurements. Optionally we can also exploit the availability of stereo during training, but in any case only require a single image and a sparse point cloud at run-time. We test our approach on both unsupervised and supervised depth completion using the KITTI benchmark, and improve the state-of-the-art in both.

1. Introduction

The paper frames dense depth completion as posterior inference because images and sparse point clouds admit multiple scene geometries. It combines learned image-conditioned priors with sparse measurements to produce confidence-aware depth estimates and reports top performance on KITTI benchmarks.

  • Motivation: The posterior representation preserves uncertainty over multiple depth maps, enabling confidence reasoning and evidence integration beyond a single point estimate.The authors use a simple MAP estimate only for benchmark evaluation.
  • Motivation: A single image and sparse lidar are ambiguous, while previously observed image-depth pairs provide contextual regularities for dense completion.The motivation includes ambiguity from occlusions, scene topology, and uninformative lidar regions.
  • Application: The system targets depth completion for downstream decision and control, using sparse range measurements to anchor the inferred posterior in metric space.Additional training data such as stereo may be incorporated, while runtime input remains an image and sparse point cloud.
  • Method: The method combines a Conditional Prior Network P(d|I, D) with the likelihood of sparse measurements to form a posterior and derive a MAP depth map.The prior scores dense-depth compatibility with the image and training dataset, while the likelihood accounts for observed sparse ranges.
  • Results: The approach achieves top performance on KITTI unsupervised completion with 5% pixel coverage and supervised completion with 30% coverage.The paper evaluates a MAP estimate of the learned posterior on both benchmark settings.

2. Related Work

Prior depth-completion work spans sparse-to-dense supervised regression, photometric and smoothness-based unsupervised learning, stereo supervision, and semantic or contextual cues. This paper explicitly models contextual regularities through a Conditional Prior Network and a likelihood-based MAP formulation.

  • Problem regime: The paper focuses on very sparse point clouds with more than 90% missing values, distinguishing its setting from semi-dense inpainting regimes with about 20% missing measurements.Related work also explores explicit semantic labels, object boundaries, anisotropic diffusion, and multitask learning.
  • Supervised depth completion: Supervised methods learn from sparse RGB-depth inputs and dense ground truth using regression, sparse convolutions, confidence propagation, morphological layers, recurrent auto-encoders, or scene-structure predictions.The cited approaches include surface normals and occlusion boundaries as coarse structural cues.
  • Unsupervised depth completion: Unsupervised methods rely on strong priors because dense ground truth is unavailable, commonly combining photometric consistency with smoothness and geometric estimation.One cited approach uses PnP and RANSAC rather than a separate pose network.
  • Stereo as supervision: Stereo-based methods use view synthesis to reconstruct an unseen view, typically alongside local smoothness assumptions, to supervise disparity prediction.The cited stereo formulations predict disparities from one image and reconstruct its paired view.
  • Contextual and semantic cues: The paper differs from prior depth-completion methods by explicitly modeling semantic regularities through a Conditional Prior Network and factorized MAP inference.The CPN scores depth compatibility with a single image, while the likelihood incorporates sparse matches.

3. Method

The method combines a Conditional Prior Network with sparse-depth likelihood modeling to estimate dense depth, optionally incorporating stereo during training. It evaluates supervised and unsupervised KITTI completion, including comparisons showing state-of-the-art performance.

  • 3. Method: A Conditional Prior Network scores candidate dense depth maps given an image and training data, providing the learned prior used in posterior estimation.The CPN maps an image and putative depth map to a positive score representing conditional probability or prior.
  • 3. Method: The posterior combines the CPN prior with a likelihood for sparse depth measurements, while training losses use selectable norms γ and η for the likelihood and prior terms.The sparse measurements constrain the hypothesized depth on the observed subset, and γ and η determine the corresponding norms.
  • 3.3. Disparity Supervision: Stereo supervision adds an image-reconstruction likelihood by relating disparity to predicted depth through s = FB/d, without requiring stereo at inference.The stereo term can use raw photometric error or SSIM, and the final stereo loss composes these terms with tunable weights.
  • Results: On supervised KITTI, the method reaches state-of-the-art performance on iRMSE, iMAE, and MAE, while [20] remains 2.6% better on RMSE.The reported margins over [20] are 24.3%, 28.9%, and 17.8% on iRMSE, iMAE, and MAE, respectively.
  • Results: On unsupervised KITTI, the CPN baseline beats [20] on iRMSE, iMAE, and RMSE, and the full photometric model beats [20] on every metric.[20] is marginally better on MAE by 0.8% before the photometric term is added.

4. Implementation Details

The implementation modifies CPN for dense depth, uses a symmetric two-branch architecture, and trains the prior on Virtual KITTI. Figure 3 studies norm choices and α using RMSE.

  • Implementation: The CPN implementation replaces its input encoding branch with dense depth, concatenates depth and image encodings, and decodes a dense reconstruction.The modified network adds skip connections and uses ResNet blocks in the encoders.
  • Implementation: The depth-completion network encodes sparse depth and image inputs separately, fuses them during decoding, and avoids specially designed sparse-input layers.This delayed fusion uses fewer learnable parameters than early fusion.
  • Training Procedure: The CPN is trained on Virtual KITTI because the authors identify no suitable real-world outdoor dataset with paired images and dense depth maps.Once trained, the CPN is used in the training loss and is not needed during inference.
  • Training Procedure: Figure 3 varies α for fixed γ and η to study norm choices in the likelihood and conditional-prior terms, measuring performance with RMSE.The experiment examines how γ and η affect the norms used in the two terms.

5. Experiments

The method achieves state-of-the-art KITTI depth-completion performance in supervised and unsupervised settings, with strong quantitative and qualitative results. It improves fine and distant structures while reducing visual artifacts.

  • 5.2. Supervised Depth Completion: Our method achieves state-of-the-art supervised KITTI depth completion, leading three metrics and obtaining the best overall average rank.It improves over [15] by 2.3% in iRMSE and 9.5% in iMAE, and over [7] by 11.9% in MAE, while [20] leads RMSE by 2.6%.
  • 5.2. Supervised Depth Completion: Qualitatively, our supervised predictions better preserve fine and far structures and contain fewer visual artifacts than [20].The comparison uses sparse inputs, dense predictions, and corresponding error maps; warmer error-map colors indicate higher error.
  • 5.3. Unsupervised Depth Completion: In unsupervised completion, the full model is state-of-the-art and outperforms [20] on every reported metric.The base loss loses to [20] by 0.8% on MAE, while adding reconstruction loss surpasses [20] across metrics and the full model improves further.
  • 5.3. Unsupervised Depth Completion: The unsupervised model avoids artifacts found in [20], including circles, and produces predictions that are globally consistent with scene geometry.Training modifications prevent inefficient copying of sparse inputs by changing the first-layer stride and using nearest-neighbor decoder upsampling.

6. Discussion

The system estimates a full depth posterior from an image and sparse aligned measurements, optionally using stereo during training. On KITTI, it reaches state-of-the-art performance in both coverage regimes while retaining confidence information for downstream decisions.

  • 6. Discussion: The system combines a Conditional Prior Network with a likelihood term for sparse depth measurements to infer posterior depth probabilities.Stereo imagery can additionally provide a photometric reconstruction term that constrains predictions to scene geometry.
  • 6. Discussion: The method achieves state-of-the-art KITTI performance in both supervised and unsupervised settings, corresponding to about 30% and 5% depth coverage.Unlike a point estimate alone, its posterior provides confidence information usable for planning, control, and decision making.
  • 6. Discussion: Real-world deployment may introduce time-varying sensor misalignment, calibration errors, and other variability absent from carefully controlled KITTI evaluation.The authors identify expansion to real-world environments as beyond the scope of this paper.

7. Supplementary

The supplementary experiments examine sensitivity to sparse-depth density, parameter efficiency, visual comparisons, and the effects of supervision and the Conditional Prior Network. Results show degradation with lower input density, a smaller network than prior art, and benefits from the proposed prior.

  • An Ablation Study on NYUv2 Indoor Dataset: Decreasing sparse-depth input density degrades performance on the NYUv2 depth-completion experiment.Table 3 varies the percentage of valid sparse-depth measurements sampled from dense depth maps.
  • Visual Comparisons: The proposed method avoids the “O” artifacts shown by [20] in KITTI unsupervised depth completion by using a scene prior compatible with the RGB image.The comparison is presented in Fig. 6.
  • More Visual Comparisons: The supervised comparison evaluates state-of-the-art methods on the supervised depth-completion task.The comparison is shown in Fig. 7.
  • CPN and Supervision: The Conditional Prior Network improves performance, and the monocular setting still outperforms [20].The ablation compares different supervision and prior configurations.
Loading 1901.10034v2…