PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End

Abdelrahman Eldesokey, Michael Felsberg, Karl Holmquist, Mikael Persson

arXiv:2006.03349v1cs.CVcs.LG

TL;DR

Depth completion uncertainty is under-addressed despite noisy, sparse depth measurements, while prior NCNN confidence does not reliably track prediction error. The paper learns input confidence and adds probabilistic NCNNs; on KITTI, its small network reports state-of-the-art accuracy and accurate uncertainty, outperforming Bayesian deep-learning approaches across accuracy, uncertainty quality, and computational efficiency.

  • Problem

    Depth completion uses inherently noisy, sparse sensor data, while prior approaches do not provide probabilistically valid confidence or uncertainty for the final prediction.

    Source: §1 · p. 1, §3.3 · p. 4, §4 · p. 4

  • Method

    The approach self-supervises an input-confidence estimator through NCNN prediction loss and introduces probabilistic NCNNs to estimate final-prediction uncertainty.

    Source: §3.3 · p. 4, §4.3 · p. 5

  • Results

    With 670k parameters, the network achieves state-of-the-art KITTI depth-completion accuracy and accurate uncertainty, outperforming Bayesian deep-learning approaches in accuracy, uncertainty quality, and computational efficiency.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The results show that a small network can provide accurate depth predictions alongside uncertainty estimates on KITTI depth completion.

    Source: §6 · p. 8

    The generalized least-squares formulation assumes zero-mean observation noise with variance σ^2V, where σ^2 is global for each signal.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

The focus in deep learning research has been mostly to push the limits of prediction accuracy. However, this was often achieved at the cost of increased complexity, raising concerns about the interpretability and the reliability of deep networks. Recently, an increasing attention has been given to untangling the complexity of deep networks and quantifying their uncertainty for different computer vision tasks. Differently, the task of depth completion has not received enough attention despite the inherent noisy nature of depth sensors. In this work, we thus focus on modeling the uncertainty of depth data in depth completion starting from the sparse noisy input all the way to the final prediction. We propose a novel approach to identify disturbed measurements in the input by learning an input confidence estimator in a self-supervised manner based on the normalized convolutional neural networks (NCNNs). Further, we propose a probabilistic version of NCNNs that produces a statistically meaningful uncertainty measure for the final prediction. When we evaluate our approach on the KITTI dataset for depth completion, we outperform all the existing Bayesian Deep Learning approaches in terms of prediction accuracy, quality of the uncertainty measure, and the computational efficiency. Moreover, our small network with 670k parameters performs on-par with conventional approaches with millions of parameters. These results give strong evidence that separating the network into parallel uncertainty and prediction streams leads to state-of-the-art performance with accurate uncertainty estimates.

1. Introduction

Depth completion must account for noisy sparse measurements, but prior confidence-based methods do not provide probabilistically valid confidence or uncertainty for the final prediction. The paper learns input confidence and introduces probabilistic NCNNs; on KITTI, it reports strong accuracy and uncertainty with a small network.

1. Introduction

  • Depth completion combines sparse, noisy sensor data with the need for reliable predictions, but prior approaches lack probabilistically valid intermediate confidence and uncertainty for the final prediction.Depth sensors are inherently noisy, and existing confidence masks have not demonstrated probabilistic validity.Source: §1 · p. 2

1.1. Contributions

  • A self-supervised estimator learns input confidence by using prediction-error gradients to assign lower confidence to disturbed measurements and higher confidence to valid ones.The approach uses NCNNs’ dependence on input confidence to train the estimator.Source: §1.1 · p. 2
  • The probabilistic NCNN framework uses maximum-likelihood training to produce uncertainty estimates with probabilistic meaning, unlike standard NCNN output confidence.The paper derives a loss for training the probabilistic framework.Source: §1.1 · p. 2
  • A 670k-parameter network achieves state-of-the-art prediction accuracy and accurate uncertainty on KITTI depth completion, outperforming Bayesian deep-learning approaches in accuracy, uncertainty quality, and computational efficiency.The paper also reports performance on par with conventional methods with millions of parameters.Source: §1.1 · p. 2

2. Related Work

Prior depth-completion networks can implicitly handle disturbed measurements or use confidence masks, but do not provide uncertainty for the final prediction. This work combines learned input confidence with probabilistic NCNNs to estimate that uncertainty.

2. Related Work

  • Earlier depth-completion approaches implicitly handle disturbed measurements but do not model uncertainty in either the data or the prediction.These approaches map sparse depth, sometimes fused with RGB or 3D representations, to dense depth.Source: §2 · pp. 2–3
  • Several methods use confidence masks to filter noisy depth measurements or fuse network streams, but they do not provide uncertainty for the final prediction.The cited methods learn masks from RGB or predict confidence masks for use within their networks.Source: §2 · p. 3
  • NCNNs propagate input confidence and produce output confidence, but binary input confidence is problematic for disturbed data and output confidence is not probabilistically meaningful or strongly tied to prediction error.The paper’s approach learns input confidence and derives a probabilistic NCNN to address these limitations.Source: §2 · p. 3

3. Self-supervised Input Confidence Learning

The section builds from normalized convolution and NCNN confidence propagation to a self-supervised estimator that learns input confidence from prediction error. This enables the network to handle disturbed sparse measurements rather than assuming binary confidence.

3. Self-supervised Input Confidence Learning

  • Normalized convolution separates signal from confidence and reconstructs non-confident values from their vicinity using an applicability function.The approach uses confident signal points in the convolution and estimates other points from neighboring values.Source: §3 · p. 3

3.1. The Normalized Convolution

  • Weighted least squares estimates basis coordinates from the signal using weights formed from applicability and confidence.The reconstructed signal is represented in the subspace spanned by the basis functions.Source: §3.1 · p. 3

3.2. Normalized Convolutional Neural Networks

  • NCNN layers learn an applicability function and propagate each layer’s output confidence as the next layer’s input confidence; the standard setup assumes binary input confidence.The naїve basis is used for the NCNN layer described here.Source: §3.2 · pp. 3–4

3.3. Self-Supervised Input Confidence Estimation using NCNNs

  • The proposed estimator learns input confidence by back-propagating prediction-loss gradients through NCNNs, whose outputs depend on those confidences.This targets disturbed measurements that can cause artifacts under the binary-confidence assumption.Source: §3.3 · p. 4

4. Probabilistic NCNNs

The section turns NCNN output confidence into a probabilistic uncertainty estimate by relating normalized convolution to generalized least squares and training the resulting model with a likelihood-based loss. Its architecture estimates input confidence and noise variance in separate stages to produce prediction and uncertainty.

4. Probabilistic NCNNs

  • The probabilistic NCNN framework derives a likelihood-based training loss so its final uncertainty estimates have probabilistic meaning, unlike standard NCNN output confidence.The authors connect normalized convolution to statistical least-squares methods and formulate training as maximum likelihood estimation.Source: §4 · p. 4

4.1. Connection between NCNN and Generalized Least-Squares

  • The NCNN least-squares solution is equivalent to generalized least squares when observation covariance is diagonal, with variances determined by confidence and applicability.The model treats noise variance σ² as unknown, while the applicability and confidence terms are non-stochastic.Source: §4.1 · pp. 4–5

4.2. Output Uncertainty for NCNNs

  • The standard output confidence omits stochastic noise variance, so the proposed method estimates that variance from the last layer’s output confidence.For the naïve basis, the uncertainty is reduced to the center signal estimate.Source: §4.2 · p. 5

4.3. The Loss Function for Probabilistic NCNNs

  • The probabilistic NCNN models its estimate as normally distributed and minimizes a negative log-likelihood whose uncertainty term encodes observation-noise variance and output confidence.The cost can also be derived with an exponential-family error model, including a Laplace distribution.Source: §4.3 · p. 5

4.4. Probabilistic NCNN Architecture

  • The pipeline estimates input confidence from disturbed depth, passes it with the input to the NCNN, then uses final-layer confidence to estimate noise variance and uncertainty.A compact UNet estimates the noise parameter from the NCNN output confidence alone.Source: §4.4 · pp. 5–6

5. Experiments

On KITTI unguided depth completion, the probabilistic NCNN variants compare favorably with statistical and non-statistical approaches on prediction accuracy and uncertainty quality. The experiments also compare variants, including a modified loss intended to handle outliers to the presumed Gaussian error model.

5. Experiments

  • The experiments evaluate unguided depth completion on KITTI-Depth, comparing prediction error and uncertainty quality against Bayesian and conventional approaches.The uncertainty measures are sparsification error plots and AUSE; prediction metrics include MAE, RMSE, and their inverses.Source: §5 · p. 6, §5.1 · p. 6

5.1. Experimental Setup

  • The setup compares NCNN-Conf trained with L1 or L2, pNCNN trained with the proposed loss, and pNCNN-Exp trained with a modified loss to robustify against outliers to the presumed Gaussian error model.Training uses Adam with initial learning rate 0.01, decayed by a factor of 10^-1 every three epochs.Source: §5.1 · p. 6

5.2. Results Compared to Statistical Methods

  • AUSE 0.053 gives pNCNN the best uncertainty measure, outperforming a 32-network ensemble; pNCNN-Exp matches NCNN-Conf RMSE with slightly worse AUSE.Both proposed variants outperform other statistical methods on RMSE, while pNCNN has a moderate RMSE gap behind NCNN-Conf.Source: Figure 3 · p. 6, §5.2 · p. 6

5.3. Results Compared to Non-Statistical Methods

  • The probabilistic model and pNCNN-Exp perform equally well to NCNN-Conf-L2 while additionally providing output uncertainties.NCNN-Conf-L1 outperforms other non-statistical methods on three of four metrics, except Spade, against which it is better on two and on-par on one; the approach reports an approximately 45% performance boost over standalone NCNN.Source: §5.3 · pp. 6–7, Table 1 · p. 7

5.4. Ablation Study

The ablations examine how confidence estimation and variance estimation affect predictions and uncertainty, while ensemble fusion and MPI correction test further uses of the approach.

5.4. Ablation Study

  • On KITTI-Depth, the full pipeline identifies disturbed measurements and makes output uncertainty highly correlated with prediction error.Standard NCNN assigns all measurements full confidence and produces a corrupted prediction; input confidence estimation identifies disturbed measurements, but its output confidence alone shows little correlation with accuracy.Source: §5.4 · p. 7, Figure 4 · p. 7
  • Discarding confidence estimation degrades RMSE, discarding variance estimation severely degrades RMSE, and Laplace loss improves MAE while degrading RMSE-based AUSE.Using the final prediction as input to variance estimation brings no AUSE improvement; the authors say uncertainty is encoded in the NCNN confidence stream without looking at the prediction.Source: §5.4 · p. 7, Table 2 · p. 7

5.5. Ensemble of pNCNN

  • Among four-network fusion schemes, maxConf performs best; wMean and MLE also outperform the standard mean.The ensemble uses pNCNN networks trained on random subsets of KITTI-Depth.Source: §5.5 · p. 7, Table 3 · p. 8

5.6. Mutli-Path Interference (MPI) Correction

  • On FLAT data, the approach outperforms bilateral filtering in RMSE on noisy and clean data; bilateral filtering is worse than no processing unless edges are excluded from evaluation.With edges excluded, bilateral filtering improves slightly but remains outperformed by the proposed approach.Source: §5.6 · p. 8

5.7. Sparse Optical Flow Rectification

The paper illustrates sparse optical-flow rectification and tests confidence estimation when depth inputs are undisturbed. On NYU, both confidence-based methods improve over standalone NCNN.

5.7. Sparse Optical Flow Rectification

  • The optical-flow example shows rectification and densification of a corrupted sparse flow field, with RGB, raw flow, ground truth, and estimated flow shown in that order.Input flow is generated with Lucas–Kanade on image pairs; ground truth is geometrically verified over several frames under a multiple rigid body assumption.Source: §5.7 · p. 8, Figure 5 · p. 8

5.8. What happens if the input is undisturbed?

  • On NYU, NCNN-Conf and pNCNN improve over standalone NCNN when 1,000 depth points are sampled from ground truth at 0.6% sparsity.The authors attribute the gain to confidence assignments based on proximity to edges, which act like nonlinear filtering and yield sharper edges and better object reconstruction.Source: §5.8 · p. 8, Figure 6 · p. 8

6. Conclusion

The conclusion reports that self-supervised input-confidence estimation and probabilistic NCNNs provide accurate uncertainty for unguided depth completion, with strong KITTI results and applications to other sparse problems.

6. Conclusion

  • On KITTI unguided depth completion, the 670k-parameter network achieves state-of-the-art prediction accuracy and accurate uncertainty, outperforming existing probabilistic methods in accuracy, uncertainty quality, and computational efficiency.It performs on par with state-of-the-art non-statistical methods that use millions of parameters.Source: §6 · p. 8
  • The authors also report that the approach applies to other sparse problems and conclude that signal/uncertainty modeling gains over conventional black-box models.Source: §6 · p. 8

Supplementary Material for Uncertainty-Aware CNNs for Depth Completion:

The supplementary material is titled “Uncertainty from Beginning to End.”

Supplementary Material for Uncertainty-Aware CNNs for Depth Completion:

1. Implementation Details

The implementation uses learned confidence and noise-variance estimates with pNCNN, and defines uncertainty through both output confidence and observation noise.

1.1. The Loss Function

  • The proposed uncertainty measure depends on both the last layer’s output confidence and the observations’ noise variance.Higher noise variance reduces output confidence, and vice versa.Source: §1.1 · p. 11

1.2. The Architecture

  • Two similarly designed compact UNets learn input confidence and observation noise variance end-to-end with NCNN.Each uses three scales rather than the original UNet’s four, which did not improve estimation empirically.Source: §1.2 · p. 11
  • SoftPlus is used at the final layer to produce nonnegative confidence values without ReLU’s gradient discontinuity at zero.Source: §1.2 · p. 11

2. Ensemble methods

The appendix defines ensemble fusion schemes for pNCNN predictions, including confidence-based weighting and maximum-likelihood estimation alongside standard mean fusion.

2.1. The Standard Mean

  • The standard mean fusion averages network predictions at each pixel without accounting for uncertainty.Source: §2.1 · p. 11

2.2. The Weighted Mean

  • Weighted mean fusion uses each prediction’s confidence when combining network predictions.Source: §2.2 · p. 12

2.3. Max Voting

  • Max voting selects the prediction with the highest confidence at each pixel.Source: §2.3 · p. 12

2.4. Maximum Likelihood Estimate

  • Maximum-likelihood fusion treats predictions as Gaussian-mixture components and formulates an inference procedure for each pixel.The optimization starts from each prediction and selects the best local maximum found, using ADAM for up to 500 steps; component variance is set to 0.1.Source: §2.4 · p. 12

3. Additional Results

These additional results examine KITTI depth completion, uncertainty quality, missing-data cases, undisturbed input, and sparse optical-flow rectification.

3.1. Qualitative Results for The KITTI-Depth dataset

  • NCNN-Conf-L1 discards error-increasing measurements to produce sharper, artifact-free edges, while pNCNN provides uncertainty across observations.pNCNN-Exp improves prediction around disturbed measurements where the presumed Gaussian error model does not hold.Source: Figure 9 · p. 13, §3.1 · pp. 13–14, Figure 11 · p. 15

3.2. The Quality of the Proposed Uncertainty Measure

  • pNCNN’s uncertainty closely tracks the error oracle in sparsification plots, while NCNN-Conf-L1 is not correlated with the oracle; pNCNN-Exp handles outliers better.Source: Figure 10 · p. 14, §3.2 · p. 14

3.5. Sparse Optical Flow Rectification

3.3. Multi-Path Interference Correction

  • On FLAT, pNCNN handles small missing-data regions and assigns high uncertainty to larger regions that cannot be handled.Source: §3.3 · p. 14, Figure 12 · p. 16

3.4. What happens when the input is undisturbed?

  • With undisturbed NYU input, NCNN-Conf-L1 preserves edges by learning input confidences, unlike NCNN’s equal trust in foreground and background samples.Source: §3.4 · p. 14, Figure 13 · p. 16
Loading 2006.03349v1…