PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-Local Spatial Propagation Network for Depth Completion

Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, In So Kweon

arXiv:2007.10042v1cs.CV

TL;DR

Sparse depth measurements and mixed-depth boundaries make reliable depth completion difficult. The paper proposes a non-local propagation network that predicts relevant neighbors, affinities, confidence, and initial depth, then iteratively refines predictions. Experiments show improved accuracy and robustness over fixed-local propagation and conventional affinity methods.

  • Problem

    Sparse depth measurements and unreliable boundary pixels complicate depth completion because propagation can incorporate irrelevant or noisy information.

  • Method

    The network predicts non-local neighbors, affinities, initial depth, and confidence, then iteratively propagates depth using learnable affinity normalization.

  • Results

    Non-local neighbors consistently outperform fixed-local neighbors, while confidence-incorporated affinity normalization outperforms alternative confidence-aware propagation methods.

  • Takeaways & Limitations

    The method focuses propagation on relevant neighbors and suppresses harmful contributions from unreliable depth values.

  • Takeaways & Limitations

    The optimal Tanh−C value may vary with the training task, neighbor count, activation functions, and dataset.

Abstract

from arXiv · show

In this paper, we propose a robust and efficient end-to-end non-local spatial propagation network for depth completion. The proposed network takes RGB and sparse depth images as inputs and estimates non-local neighbors and their affinities of each pixel, as well as an initial depth map with pixel-wise confidences. The initial depth prediction is then iteratively refined by its confidence and non-local spatial propagation procedure based on the predicted non-local neighbors and corresponding affinities. Unlike previous algorithms that utilize fixed-local neighbors, the proposed algorithm effectively avoids irrelevant local neighbors and concentrates on relevant non-local neighbors during propagation. In addition, we introduce a learnable affinity normalization to better learn the affinity combinations compared to conventional methods. The proposed algorithm is inherently robust to the mixed-depth problem on depth boundaries, which is one of the major issues for existing depth estimation/completion algorithms. Experimental results on indoor and outdoor datasets demonstrate that the proposed algorithm is superior to conventional algorithms in terms of depth completion accuracy and robustness to the mixed-depth problem. Our implementation is publicly available on the project page.

1 Introduction

Depth completion estimates dense depth from sparse measurements, but existing methods can produce blurry or mixed-depth values near boundaries. NLSPN addresses this by predicting non-local neighbors and spatially varying affinities to aggregate relevant information during propagation.

  • 1 Introduction: Depth completion is needed to estimate dense depth from sparse depth measurements obtained by sensors such as LiDAR.Depth sensors provide accurate measurements, but their acquired depth is sparse, motivating dense-depth estimation.
  • 1 Introduction: Early sparse-measurement methods produce blurry and mixed-depth values, while direct CNN methods still blur depth maps near boundaries.RGB guidance and affinity-based spatial propagation methods were introduced to improve dense depth estimation and alleviate boundary artifacts.
  • 1 Introduction: NLSPN predicts non-local neighbors and spatially varying affinities, allowing each pixel to receive relevant propagated information rather than relying on fixed local neighborhoods.The neighbors indicate where information should come from, while affinities determine how much information is propagated.

2 Related Work

Related work spans depth estimation and completion, spatial propagation with learned local affinities, and non-local modeling in vision. The proposed method extends these directions by predicting non-local neighbors and affinities, using learnable confidence-aware affinity normalization for iterative propagation.

  • Depth Estimation and Completion: Depth estimation generates dense predictions from inputs including RGB images, multi-view images, and sparse LiDAR measurements, with conventional methods often relying on a single modality.
  • Spatial Propagation Network: Spatial propagation learns task-specific local affinities to guide sparse-to-dense prediction from multimodal inputs such as RGB images.
  • Non-Local Network: Non-local networks calculate pairwise affinities and process features, demonstrating the value of non-local information across vision tasks.
  • Our Work: Unlike prior methods, the proposed network predicts non-local neighbors and affinities, then iteratively propagates depth using confidence-incorporated learnable affinity normalization.Given RGB and sparse depth inputs, it predicts initial dense depth, confidence, non-local neighbors, and affinities before propagation.
  • Our Work: Confidence-aware affinity normalization minimizes propagation of unconfident initial depth values while searching for an optimal affinity space.

3 Non-Local Spatial Propagation

This section motivates non-local spatial propagation by showing that fixed-local neighborhoods can mix foreground and background depths. It introduces learned, color- and depth-guided non-local neighbors that focus propagation on relevant pixels and support sub-pixel accuracy.

  • Spatial Propagation: Spatial propagation refines missing or less-confident values by combining the reference pixel with neighboring observations weighted by learned affinities.The reference affinity controls preservation of the original value, while neighbor affinities weight propagated observations.
  • Non-Local Spatial Propagation: Unlike SPN and CSPN, NLSPN permits flexible neighbor configurations, including real-valued offsets that define neighbors with sub-pixel accuracy.The visual comparison contrasts three-way, fixed-local, and non-local configurations, including desirable neighbors near depth boundaries.
  • Motivation: Fixed-local neighborhoods ignore local object and depth distributions, often mixing foreground and background values during propagation and limiting SPN/CSPN improvement.Predicted affinities can reduce mixing between irrelevant pixels but cannot reliably avoid incorrect local-neighbor predictions.
  • Non-Local Spatial Propagation: The proposed network predicts each pixel’s non-local neighbors from RGB and sparse-depth information over a wide area, concentrating propagation on relevant pixels.An encoder-decoder CNN estimates K neighbors using learnable parameters, while affinity learning becomes easier when irrelevant neighbors are excluded.

4 Confidence-Incorporated Affinity Learning

The section identifies bias in conventional affinity normalization and proposes a learnable, confidence-incorporated normalization that stabilizes propagation while reducing the influence of unreliable depth values. The method combines learned normalization with pixel confidences to improve propagation around noisy or boundary pixels.

  • Confidence-Incorporated Affinity Learning: Conventional Abs−Sum normalization restricts normalized affinities to a narrow high-dimensional space, limiting potentially advantageous affinity configurations.In the two-neighbor case, affinities lie on |w1| + |w2| = 1, excluding configurations with |w1| + |w2| < 1.
  • Confidence-Incorporated Affinity Learning: Tanh−C reduces this bias by limiting raw affinities, while the proposed learnable γ combines the advantages of Tanh−C and Abs−Sum∗.The condition C ≥ K preserves the stability guarantee, and the learned factor adapts normalization to the training task.
  • Confidence-Incorporated Affinity Learning: Confidence incorporation suppresses propagation from noisy or boundary pixels, avoiding artifacts that arise when unreliable depth values are treated equally.The confidence-agnostic example shows noisy, low-confidence depths harming neighboring pixels, whereas confidence-based normalization eliminates their impact and produces a more accurate result.
  • Confidence-Incorporated Affinity Learning: The proposed confidence-incorporated Tanh−γ−Abs−Sum∗ learns the normalization factor jointly with non-local affinities and uses predicted pixel confidences during normalization.The confidence map weights pixels by reliability, reducing disturbances from unreliable depths during propagation.

5 Depth Completion Network

NLSPN combines an encoder-decoder that predicts initial depth, confidence, non-local neighbors, and raw affinities with iterative non-local spatial propagation using learnable affinity normalization. Training uses depth reconstruction loss, while confidence is learned indirectly because no confidence ground truth is available.

  • NLSPN comprises an encoder-decoder for initial depth, confidence, non-local neighbors, and raw affinities, followed by propagation with learnable affinity normalization.The network is organized into these two main components.
  • Shared encoder-decoder features support initial dense-depth, confidence, neighbor, and affinity estimation before iterative non-local spatial propagation.The encoder-decoder is built on residual networks and combines low-level and high-level features through encoder-decoder connections.
  • Differentiable sampling incorporates fractional-coordinate non-local neighbors during propagation.The method adopts differentiable sampling to train with non-local neighbors whose coordinates may be fractional.
  • Depth reconstruction uses ℓ1 or ℓ2 loss against ground-truth depth, while confidence receives no direct supervision and is trained indirectly through reconstruction loss.The loss exponent ρ is set to 1 for ℓ1 loss and 2 for ℓ2 loss.

6 Experimental Results

Experiments on indoor and outdoor depth-completion benchmarks show that the proposed non-local propagation method improves accuracy and robustness, particularly around mixed-depth boundaries. Ablations further support non-local neighbors, affinity normalization, and confidence incorporation.

  • Quantitative comparisons: The method achieves the best NYUv2 result and outperforms competing methods by a large margin, including an RMSE of 0.020m.The method is characterized as implicitly geometry-aware because it explores geometrically relevant neighbors during propagation.
  • Qualitative comparisons: Qualitative results show that non-local propagation preserves tiny structures and depth boundaries better than competing methods affected by mixed-depth artifacts.The advantage is reported on both NYUv2 comparisons with S2D and CSPN and KITTI DC examples involving small structures near depth boundaries.
  • Quantitative comparisons: On KITTI DC, the proposed method outperformed all peer-reviewed methods on the KITTI online leaderboard.The evaluation also emphasizes confidence prediction as beneficial for reducing LiDAR mixed-depth noise.
  • Ablation studies: Non-local neighbors reduce average depth variance from 22.7mm with fixed-local neighbors to 11.6mm, indicating more relevant propagation neighbors.The non-local configuration also consistently outperforms the fixed-local configuration in quantitative comparisons.
  • Ablation studies: Ablations compare affinity-normalization variants and confidence-aware propagation, testing whether normalization and confidence handling improve robustness to limited affinity combinations and outliers.The study evaluates alternative confidence schemes, including thresholded masks and weighted summation, alongside the proposed confidence incorporation.

7 Conclusion

The paper proposes an end-to-end trainable non-local spatial propagation network for depth completion that flexibly selects propagation neighbors and improves affinity learning. Unlike fixed-local methods, it excludes irrelevant neighbors and focuses propagation on synergies among relevant ones.

  • 7 Conclusion: The proposed network enables flexible neighbor selection for depth propagation, supporting accurate propagation and easing the affinity learning problem.Its end-to-end trainable design jointly supports neighbor selection and affinity learning.
  • 7 Conclusion: Compared with fixed-local propagation, the method efficiently excludes irrelevant neighbors during propagation.This allows propagation to focus on relevant rather than indiscriminate local neighbors.
  • 7 Conclusion: The method enforces propagation to focus on a synergy among relevant neighbors.This non-local propagation strategy is presented as a central distinction from previous fixed-local algorithms.
Loading 2007.10042v1…