PaperlayerView exact revision on arXiv

Source-linked AI summary

Confidence Propagation through CNNs for Guided Sparse Depth Regression

Abdelrahman Eldesokey, Michael Felsberg, Fahad Shahbaz Khan

arXiv:1811.01791v2cs.CV

TL;DR

Sparse and irregular sensor data challenge CNNs because missing regions require reconstruction and reliability tracking, while high-resolution depth completion often uses large networks. The paper develops a lightweight normalized-convolution framework with continuous confidence propagation, non-negative filters, confidence-aware training, and RGB fusion. On KITTI-Depth and NYU-Depth-v2, it reports state-of-the-art performance with substantially fewer parameters than existing methods.

  • Problem

    Sparse active-sensor outputs require dense depth reconstruction and reliability estimation, while existing high-resolution CNN methods demand millions of parameters.

  • Method

    The paper uses normalized convolution with continuous confidence propagation, non-negative learned filters, a confidence-aware loss, and RGB fusion with output confidence as guidance.

  • Results

    The proposed method achieves state-of-the-art depth-completion performance on KITTI-Depth and NYU-Depth-v2 with approximately 356k and 484k parameters, respectively.

  • Takeaways & Limitations

    The framework provides accurate scene depth completion with significantly fewer network parameters than existing state-of-the-art methods.

  • Takeaways & Limitations

    The applicability function is learned rather than handcrafted, with handcrafted applicability outside the paper's scope.

Abstract

from arXiv · show

Generally, convolutional neural networks (CNNs) process data on a regular grid, e.g. data generated by ordinary cameras. Designing CNNs for sparse and irregularly spaced input data is still an open research problem with numerous applications in autonomous driving, robotics, and surveillance. In this paper, we propose an algebraically-constrained normalized convolution layer for CNNs with highly sparse input that has a smaller number of network parameters compared to related work. We propose novel strategies for determining the confidence from the convolution operation and propagating it to consecutive layers. We also propose an objective function that simultaneously minimizes the data error while maximizing the output confidence. To integrate structural information, we also investigate fusion strategies to combine depth and RGB information in our normalized convolution network framework. In addition, we introduce the use of output confidence as an auxiliary information to improve the results. The capabilities of our normalized convolution network framework are demonstrated for the problem of scene depth completion. Comprehensive experiments are performed on the KITTI-Depth and the NYU-Depth-v2 datasets. The results clearly demonstrate that the proposed approach achieves superior performance while requiring only about 1-5% of the number of parameters compared to the state-of-the-art methods.

1 INTRODUCTION

The paper addresses sparse depth completion by propagating continuous confidence through normalized CNNs and combining sparse depth with RGB guidance. It targets accurate, resource-efficient dense depth estimation for applications requiring trustworthy outputs.

  • Motivation: Sparse active sensors leave empty regions and nonuniform data relevance, requiring methods that reconstruct or ignore missing areas.The challenge affects ToF cameras, LiDAR, RGB-D, and event cameras.
  • Motivation: Scene depth completion reconstructs dense depth maps from sparse active-sensor outputs, while confidence indicates output trustworthiness for safety and decision-making applications.The paper specifically connects confidence to obstacle detection and avoidance in autonomous driving.
  • Problem Setting: Continuous confidence fields avoid validity-mask saturation in multi-stage CNNs and enable reliability tracking throughout the processing pipeline.Binary masks use zeros for missing regions and ones otherwise, whereas continuous confidence describes data reliability.
  • Problem Setting: High-resolution depth-completion methods often require CNNs with millions of parameters, hindering deployment in resource-limited autonomous systems.The paper motivates smaller networks that can still propagate confidence for real-world applications.
  • Contributions: The proposed normalized convolution layer uses non-negative filters, confidence propagation, and a loss balancing data error with output confidence.The non-negative filters act as neighborhood weighting functions and are intended to accelerate convergence.
  • Contributions: RGB guidance is investigated through early and late fusion, with output confidence also supplied as auxiliary guidance; KITTI-Depth and NYU-Depth-v2 results use about 356k and 484k parameters, respectively.The reported parameter counts are compared with existing state-of-the-art methods using millions of parameters.

2 RELATED WORK

Prior depth-completion methods use compressed sensing, encoder-decoder networks, sparsity-invariant convolutions, and RGB fusion. The paper distinguishes its approach through normalized convolution with continuous confidence, non-negative filters, propagated output confidence, and lightweight late fusion.

  • Unguided Depth Completion: Earlier unguided methods handled sparse depth using compressed sensing, encoder-decoder prediction, or binary validity masks in sparsity-invariant convolution.These methods take sparse depth alone or pair it with a binary validity mask.
  • Guided Depth Completion: Guided approaches combine sparse depth with RGB or other auxiliary data using early or late fusion, with different studies reporting advantages for each strategy.The cited literature includes early-fusion and late-fusion architectures.
  • Proposed Approach: The proposed normalized convolution differs by using continuous confidence, non-negative learned filters, and confidence propagation to produce point-wise output confidence.These design choices are presented as enabling faster convergence and fewer parameters than comparable methods.
  • Proposed Approach: RGB guidance is extended with output-confidence guidance, and the proposed multi-stream late-fusion architecture reports strong results with substantially fewer parameters than comparable methods.The paper states that output-confidence guidance significantly improves performance.

3 NORMALIZED CONVOLUTION

Normalized convolution separates signal values from confidence and uses confidence-weighted local modeling to densify sparse fields. In the paper's CNN formulation, learned non-negative applicability functions determine neighborhood weighting while output confidence summarizes input support and data coherence.

  • Confidence-Equipped Signals: Normalized convolution models a discrete signal together with a non-negative confidence field describing the reliability of each sample.Zero confidence indicates an absent sample in a sparse depth field.
  • Confidence-Equipped Signals: Local signal reconstruction projects samples onto basis functions and estimates their coordinates with a confidence-weighted least-squares problem.The applicability function weights the basis, while confidence weights the signal samples.
  • Convolution Formulation: Diagonal confidence and applicability matrices allow the formulation to be implemented through convolution-like point-wise operations.The diagonal structure replaces matrix multiplication with Hadamard and scalar-product operations.
  • 3.1 The Na¨ıve Basis: With a constant naïve basis, the applicability function acts as the convolution filter and the formulation can densify sparse depth using point-wise confidence.A suitable applicability function produces a dense depth field from sparse samples.
  • 3.2 The Applicability Function: The applicability function must be non-negative, can emphasize central samples or adapt to local structure, and is learned rather than handcrafted in this paper.The paper notes that anisotropic applicability is relevant to image inpainting, while handcrafted applicability is outside its scope.
  • 3.3 Propagating Confidences: Normalized convolution separates signal and confidence, enabling confidence-adaptive processing and output-confidence estimation based on input density and data coherence under the chosen basis.The paper discusses geometric and alternative confidence measures for normalized convolution.

4 UNGUIDED NORMALIZED CNNS

The unguided normalized CNN uses non-negative applicability functions, confidence-aware convolution, and propagated continuous confidence to complete highly sparse depth inputs. Its multi-scale architecture and confidence-aware loss support dense predictions while balancing data accuracy and output confidence.

  • 4.1 Training the Applicability: Normalized convolution learns a non-negative applicability function by applying a differentiable non-negative function to convolution kernels.The function acts as a windowing or neighborhood-weighting function, with back-propagation handled through the chain rule.
  • 4.3 The Normalized CNN Layer: The normalized convolution layer jointly processes data and confidence, producing corresponding data and confidence terms.Its forward pass uses confidence from the previous layer, applicability weights, bias, and a constant preventing division by zero.
  • 4.2 Propagating the Confidence: Confidence is propagated between CNN layers using a measure derived from the normalized-convolution forward path, avoiding validity-mask saturation.The measure reuses already computed forward-pass terms and produces a point-wise confidence map.
  • 4.3 The Normalized CNN Layer: The multi-scale architecture downsamples by max pooling confidence maps, selects highest-confidence features, and fuses scales through upsampling, concatenation, and normalized convolution.A final 1 × 1 normalized convolution merges channels into a dense output and an output confidence map.
  • 4.4 The Loss Function: The loss function combines Huber data error with confidence maximization, while dividing the confidence term by the epoch number to prevent it dominating after data-error convergence.The Huber norm behaves like L2 below δ and L1 otherwise.

5 GUIDED NORMALIZED CNNS

The guided framework combines sparse depth, RGB structure, and propagated output confidence to produce dense depth maps, using either early or late fusion in multi-stream and encoder-decoder architectures.

  • RGB guidance addresses larger errors at edges and rough surfaces, where the unguided normalized convolution network lacks structural information.
  • Output confidence is fused with RGB to indicate the reliability of each pixel in the unguided prediction.The paper reports that using output confidence improves depth results by approximately 10% and correlates with prediction error.
  • The multi-stream architecture separates depth from RGB-plus-confidence feature extraction before concatenating both streams in a fusion network.Both late-fusion and early-fusion variants are evaluated.
  • The encoder-decoder architecture processes depth and RGB-plus-confidence streams separately, then combines corresponding-scale features through decoder skip connections.Early-fusion variants concatenate the depth output, confidence, and RGB image before the encoder-decoder.
  • All guided networks use a fixed feature-channel budget of 16 features per input channel to reduce parameter count for computational efficiency.This gives 16 depth-stream features and 64 features for RGB plus output confidence.

6 EXPERIMENTS

Experiments evaluate the proposed guided depth-completion networks on KITTI-Depth and NYU-Depth-v2 using established depth-error metrics and comparisons with published methods. The results show strong benchmark performance, especially for the multi-stream late-fusion and encoder-decoder early-fusion variants.

  • Datasets and metrics: The evaluation uses KITTI-Depth and NYU-Depth-v2 datasets for sparse-to-dense depth completion.KITTI depth inputs contain only 5% available pixels, while NYU-Depth-v2 uses roughly 48k training pairs and 654 test pairs.
  • Datasets and metrics: KITTI-Depth evaluation reports MAE, RMSE, iMAE, and iRMSE, while NYU-Depth-v2 evaluation reports RMSE, REL, and inliers ratio.The inverse metrics are computed on disparity rather than depth.
  • Architecture comparison: MS-Net[LF] outperforms the other evaluated fusion architectures across all KITTI-Depth validation metrics.MS-Net[EF] has similar MAE and iMAE but slightly higher RMSE and iRMSE, while EncDec-Net[EF] outperforms EncDec-Net[LF].
  • KITTI-Depth comparison: ∼355k parameters let MS-Net[LF]-L1 (gd) achieve state-of-the-art KITTI-Depth results, compared with ∼5.5M for Sparse-to-Dense (gd).MS-Net[LF]-L1 (gd) achieves the best MAE and outperforms Spade (gd) on iMAE.
  • NYU-Depth-v2 comparison: For very sparse NYU-Depth-v2 input with 200 samples, EncDec-Net[EF] achieves the best results by a large margin, while MS-Net[LF] ranks second.With 500 samples, EncDec-Net[EF] ranks second, close to UNet+CSPN.
  • Qualitative comparison: On NYU-Depth-v2 qualitative examples, EncDec-Net[EF] and MS-Net[LF] produce sharper, more detailed reconstructions than Sparse-to-Dense.EncDec-Net[EF] is smoother and more consistent than MS-Net[LF], especially along edges.

7 ANALYSIS

The analysis tests robustness to input sparsity and examines the SoftPlus choice for non-negative filters. Multi-scale processing helps EncDec-Net[EF] remain effective across sparsity levels, while SoftPlus with β = 10 approximates ReLU smoothly.

  • Varying Degree of Sparsity: EncDec-Net[EF] performs well across varying NYU-Depth-v2 sparsity levels, including approximately 0.01% input density.Its multiple scales exploit depth information at different scales, whereas single-scale MS-Net[LF] performs worse for very sparse input.
  • Varying Degree of Sparsity: As more depth samples are provided, MS-Net[LF] approaches EncDec-Net[EF], while Sparse-to-Dense does not significantly benefit from the additional samples.
  • Non-negative function: SoftPlus is used as a continuous approximation to ReLU for enforcing non-negative learned filters.At β = 10, it provides a good differentiable approximation of ReLU while retaining a continuous derivative.

7.3 The Non-negativity Constraint Impact

The analysis evaluates non-negativity constraints and the confidence term in the loss. Removing non-negativity degrades guided results, while the proposed confidence-aware loss increases confidence and lowers MAE.

  • The Non-negativity Constraint Impact: Discarding the non-negativity constraint significantly degrades the guided network across all evaluation metrics.The paper suggests this may reflect lost output-confidence guidance or poorer unguided depth estimates.
  • Confidence term: The proposed confidence-aware loss produces monotonically increasing output confidence and lower MAE than using only the Huber data term.Confidence statistics are reported as means and standard deviations over the selected validation images.

7.5 The Learned Filters

The learned filters form a multi-scale estimator: early layers reconstruct missing pixels from neighboring values, while later layers rescale outputs for multi-scale fusion. Ablation evidence also shows that output confidence and depth refinement materially affect guided-network performance.

  • Learned filter structure: NCONV[1-3] learn asymmetric low-pass filters that construct missing pixels from their neighborhood.These filters operate in the early layers of the multi-scale estimator.
  • Learned filter structure: NCONV[4-6] learn linear-ramp-like filters that scale each output for efficient fusion across scales.These filters support the later-stage combination of multi-scale estimates.
  • Ablation study: Removing depth refinement layers has nearly the same performance influence as removing output confidence, while end-to-end training slightly degrades results.The text attributes the latter degradation possibly to vanishing gradients in the deeper network.
  • Ablation study: Removing output confidence increases error by approximately 10%, indicating that confidence supplies the RGB stream with information about low-confidence regions.Low-confidence regions are described as highly likely to contribute to prediction error.
  • Ablation study: 1122.51 [mm] RMSE is obtained by a doubled-depth network without NConv, versus 1007.71 [mm] for the proposed lightweight architecture.The comparison removes confidence propagation and normalization from the larger network.

7.7 Number of Parameters and Runtime Comparison

The proposed networks substantially reduce computational demands while retaining competitive depth-completion performance. The guided model combines a very small parameter count with the lowest reported guided runtime, and the method remains effective on NYU-Depth-v2.

  • Guided networks: 356k parameters and 0.02 seconds runtime are reported for the guided MS-Net-L2[LF] (gd), with at least one order of magnitude fewer parameters than other guided methods.The reported runtime satisfies real-time constraints in the comparison.
  • Unguided networks: The unguided NConv-CNN (d) has the lowest parameter count and runtime among the compared unguided methods.The paper identifies this profile as suitable for embedded applications with limited computational resources.
  • NYU-Depth-v2: Approximately 1% of Sparse-to-Dense’s parameter count is used on NYU-Depth-v2, while achieving better results at different sparsity levels.This comparison is reported for the paper’s method on the NYU-Depth-v2 dataset.

7.8 Output Confidence/Error Correlation

The paper evaluates whether output confidence tracks prediction error by correlating absolute error with the negative logarithm of confidence. Its learned confidence substantially outperforms a naive interpolated-confidence baseline on both datasets.

  • Correlation analysis: The analysis correlates prediction absolute error with the negative logarithm of output confidence, similar to log likelihood.Pearson’s correlation is used after histogram equalization because the error and confidence distributions are unknown.
  • Baseline: The naive interpolated-confidence baseline reaches average Pearson correlation 0.1 on NYU-Depth-v2 and - 0.25 on KITTI-Depth.The baseline uses the naive basis and a Gaussian applicability, with confidence attenuating away from valid input points.

8 CONCLUSION

The paper concludes that normalized convolution can complete highly sparse depth data while propagating continuous confidence and integrating RGB guidance. Experiments on KITTI-Depth and NYU-Depth-v2 report superior performance with substantially fewer parameters than state-of-the-art methods.

  • Contributions: The normalized convolution layer treats validity masks as continuous confidence fields for unguided scene depth completion on highly sparse data.Confidence propagation produces a point-wise continuous confidence map alongside the network’s dense output.
  • Contributions: Non-negative filter constraints act as neighborhood weighting functions, while the loss jointly minimizes data error and maximizes output confidence.These design choices are presented as part of the normalized convolution framework.
  • Contributions: The framework includes a fusion strategy combining depth and RGB information to incorporate structural information.The conclusion presents RGB-depth fusion as part of the proposed method.
  • Conclusion: Experiments on KITTI-Depth and NYU-Depth-v2 achieved superior performance with significantly fewer network parameters than state-of-the-art methods.The conclusion states this as the overall empirical outcome.
Loading 1811.01791v2…