PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints

Yan Xu, Xinge Zhu, Jianping Shi, Guofeng Zhang, Hujun Bao, Hongsheng Li

arXiv:1910.06727v1cs.CV

TL;DR

Depth completion must recover dense maps from sparse measurements, while direct CNN mappings struggle to exploit 3D geometry and practical sensor noise. The paper introduces a unified CNN framework that couples depth and surface normals through diffusion and predicts sparse-measurement confidence. Experiments on KITTI and NYU-Depth-V2 report state-of-the-art performance, generalization capability, and stability.

  • Problem

    Existing depth-completion methods directly learn sparse-to-dense mappings and have difficulty using 3D geometric constraints and handling practical sensor noise.

  • Method

    A unified two-stage CNN framework uses plane-origin distance diffusion to enforce depth-normal constraints and a confidence branch to mitigate noisy LiDAR inputs.

  • Results

    Experiments on KITTI depth completion and NYU-Depth-V2 demonstrate state-of-the-art performance.

  • Takeaways & Limitations

    The framework provides end-to-end depth completion with reported generalization capability and stability while addressing geometric constraints and sensor noise.

Abstract

from arXiv · show

Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse LiDAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model.

1. Introduction

Depth completion addresses the need for dense depth from sparse sensor measurements, but direct CNN mappings struggle to use 3D geometry and handle sensor noise. The paper proposes a unified framework that couples depth and normal constraints with confidence-aware refinement.

  • Motivation: Sparse depth sensors provide incomplete measurements, while dense depth maps support applications including semantic segmentation, 3D reconstruction, and SLAM.The paper cites limited sensing scope, environmental interference, and economic considerations as causes of sparsity.
  • Limitations of Existing Methods: Direct encoder-decoder methods learn sparse-to-dense mappings as black boxes and face difficulties producing satisfactory completion results.The paper identifies the lack of explicit geometric constraints as a central concern.
  • Proposed Framework: The method associates depth and surface normal in plane-origin distance space, then applies recurrent diffusion to enforce their locally linear orthogonality.This shifts geometric modeling from 2D space toward 3D plane-based structure.
  • Proposed Framework: A confidence prediction branch reduces the influence of noisy practical sensor measurements by impeding propagation of information associated with noise.The confidence mechanism is integrated into the unified two-stage CNN framework.
  • Results: The framework is trained end-to-end and achieves state-of-the-art performance while retaining good generalization capability.The paper reports this outcome from extensive experiments.

2. Related Work

Prior depth-completion research includes handcrafted, learning-based, and geometry-based methods, while anisotropic diffusion provides a related mechanism for propagating information. The paper combines these ideas in an end-to-end framework using depth-normal constraints and learned guidance.

  • Depth Completion: Depth completion methods address incomplete measurements from structured-light scanners and LiDAR, with LiDAR methods often targeting real-time robotic navigation and autonomous driving.Structured-light approaches are commonly used in 3D-reconstruction post-processing.
  • Depth Completion: Classic completion methods use handcrafted features or kernels, whereas learning-based methods improve performance but can remain task-specific or rely only on sparse depth.The related work contrasts these approaches with newer CNN-based methods.
  • Depth and Normal: Previous depth-normal approaches derive one representation from the other or solve geometric constraints with a linear system, but some are difficult to integrate into end-to-end or real-time frameworks.The cited RGB-D post-processing approach is described as having limited suitability for real-time processing.
  • Proposed Framework: The proposed framework uses a prediction network and refinement network to estimate depth and normals, transform them into plane-origin distance space, and refine them with diffusion.The framework is presented as a two-stage end-to-end design.
  • Anisotropic Diffusion: Anisotropic diffusion has been used for denoising, depth completion, and segmentation; earlier conductance definitions rely on diffusion-space or guidance-map similarity.The paper positions learned high-dimensional guidance as a related design direction.

3. Method

The method uses a two-stage CNN framework that predicts normals, coarse depth, and sparse-depth confidence, then refines plane-origin distances through recurrent diffusion. This couples depth completion with depth-normal geometry while reducing the influence of noisy LiDAR measurements.

  • Plane-origin Distance: The framework assumes scenes consist of piecewise planes, making plane-origin distance piecewise constant and suitable for geometric refinement.For a plane, P = N(x) · X is constant and represents the distance from the plane to the camera-center origin.
  • Prediction Network: The prediction network takes sparse depth and color inputs and separately predicts surface normals, coarse depth, and a confidence map using a U-Net architecture.A ResNet-34 variant serves as encoder, with cascaded upsampling decoders.
  • Prediction Network: Confidence is learned from noisy sparse depth and ground-truth depth to mitigate noise propagation, while refinement feedback further improves its prediction.The confidence target is modeled using a Laplace-inspired function, with an L2 loss applied to the prediction.
  • Plane-origin Distance: Depth and normal are transformed into plane-origin distance using camera intrinsics, and refined distance is converted back to depth through the inverse relation.The recovery step is D(x) = P(x)/(N(x)C^-1x).
  • Plane-origin Distance Diffusion for Depth Refinement: Anisotropic diffusion refines the plane-origin distance map by exchanging information among neighboring pixels likely to share a plane.CNN features determine conductances, while asymmetric transformations f and g provide directional flexibility.
  • Plane-origin Refinement and Depth Recovery: Each refinement iteration uses confident sparse measurements as seeds, combines them with coarse predictions, and diffuses the resulting plane-origin distances recurrently.The confidence map suppresses unreliable measurements while allowing confident sparse depth and predicted depth to complement each other.

4. Experiments

Experiments evaluate the framework on KITTI and NYU-Depth-v2, showing strong benchmark performance, effective geometric and confidence components, and stability under refinement and sparsity changes.

  • Experimental Setup: The evaluation uses KITTI depth completion and NYU-Depth-v2, with dataset-specific depth-input settings and standard depth-completion metrics.KITTI evaluation includes RMSE, MAE, iRMSE, and iMAE; NYU-Depth-v2 uses RMSE, relative error, and threshold accuracies.
  • Benchmark Results: Our method ranks 1st among peer-reviewed methods on the KITTI test set according to RMSE.The paper also presents qualitative completion, detail, and error-map comparisons.
  • Ablation Study: Removing proposed components degrades performance, while the w/o replacement variant outperforms w/o refinement, supporting the geometric refinement design.The ablation compares depth-only prediction, removal of refinement and confidence branches, and diffusion without seed replacement.
  • Ablation Study: The proposed asymmetric conductance function outperforms symmetric, Euclidean-distance, and dot-product alternatives.These variants alter the diffusion module’s feature transformation or similarity function.
  • Ablation Study: Confidence prediction improves performance by limiting error propagation, while excessively large or small tolerance b values degrade results.Lower-confidence regions concentrate around moving objects and object boundaries, where noise commonly occurs.
  • Generalization Ability: The model achieves state-of-the-art performance on NYU-Depth-v2 and remains competitive with a ResNet-34 encoder, while ResNet-50 further improves performance.NYU-Depth-v2 experiments use uniformly sampled sparse depth inputs.
  • Stability Analysis: Error decreases and stabilizes with more refinement iterations, and the model still provides reasonable results with 1/10 of the original sparse inputs.Performance drops as the sampling ratio decreases, as expected.

5. Conclusion

The paper presents a unified two-module framework that uses depth-normal geometry and confidence prediction to complete sparse depth maps, achieving state-of-the-art results on outdoor and indoor datasets.

  • Conclusion: The framework combines a prediction network with a refinement network to complete depth maps from sparse inputs.It shifts processing from 2D space toward 3D plane-origin distance space.
  • Conclusion: A diffusion model uses depth-normal constraints and adaptive feature-space conductance to regularize completion.The method assumes scenes are composed of piecewise planes and recurrently refines plane-origin distances.
  • Conclusion: A confidence decoder branch addresses LiDAR noise by impeding error propagation during refinement.
  • Conclusion: Extensive experiments demonstrate state-of-the-art performance on both outdoor and indoor datasets.

Abstract

Depth completion recovers dense depth from sparse measurements, but existing direct CNN mappings struggle with 3D geometric constraints and practical sensor noise. The paper proposes a unified CNN framework combining depth-normal diffusion with sparse-measurement confidence prediction, and reports state-of-the-art results on KITTI and NYU-Depth-V2.

  • Abstract: Depth completion recovers dense depth maps from sparse measurements, but direct CNN mappings struggle with geometric constraints and sensor noise.
  • Abstract: The unified CNN framework models depth-normal geometry in a diffusion module and predicts confidence for sparse LiDAR measurements.
  • Abstract: The encoder-decoder predicts surface normals, coarse depth, and LiDAR confidence before diffusion refinement produces the final completion.
  • Abstract: Experiments on KITTI depth completion and NYU-Depth-V2 report state-of-the-art performance, with analyses supporting generalization and stability.

1. Introduction

The paper presents a unified CNN framework for depth completion that combines depth–normal geometric constraints with confidence prediction to improve robustness against sparse-measurement noise.

  • The framework jointly predicts surface normals, coarse depth, and LiDAR confidence before diffusion-based refinement produces the final dense completion.
  • It addresses limitations of direct sparse-to-dense CNN mapping, particularly weak use of 3D geometric constraints and sensitivity to practical sensor noise.
  • Experiments on KITTI depth completion and NYU-Depth-V2 report state-of-the-art performance, with ablations examining generalization and stability.
Loading 1910.06727v1…