Source-linked AI summary
Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints
Yan Xu, Xinge Zhu, Jianping Shi, Guofeng Zhang, Hujun Bao, Hongsheng Li
TL;DR
Sparse depth completion seeks dense maps, while direct sparse-to-dense learning has difficulty using 3D geometric constraints and handling sensor noise. This paper combines depth-normal-constrained diffusion refinement with confidence prediction and reports state-of-the-art performance with good generalization capability.
Problem
Direct sparse-to-dense CNN methods have difficulty incorporating 3D geometric constraints, while sparse depth completion also must address practical sensor noise.
Source: §1 · pp. 1–2Method
A unified CNN predicts depth, surface normals, and sparse-measurement confidence, then refines depth-normal constraints through diffusion in plane-origin distance space.
Source: §1 · p. 2Results
The framework achieves state-of-the-art performance while retaining good generalization capability.
Source: §1 · p. 2Takeaways & Limitations
Ablations indicate that geometric refinement and confidence prediction each contribute to the framework’s performance.
Source: §4.4 · pp. 7–8
Abstract
from arXiv · showhide
Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse LiDAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model.
1. Introduction
The paper proposes depth completion that enforces 3D geometric constraints between depth and surface normals while predicting confidence for sparse LiDAR measurements to mitigate sensor noise.
1. Introduction
- Depth sensors produce sparse measurements, whereas high-level applications require dense depth; depth completion generates dense maps from sparse measurements.The introduction attributes sparsity to limited sensing scope, environmental interference, and economic considerations.Source: §1 · p. 1
- Existing CNNs learn sparse-to-dense mappings as black boxes [16] [36] [15] [10] [23] [2], while normal-guided optimization [46] is costly and has limited driving-scenario generalization.The paper also notes that the earlier method separates normal training from dense-depth optimization, preventing joint data-driven optimization.Source: §1 · pp. 1–2
- The method represents depth and surface normals in plane-origin distance space and recurrently diffuses estimates under piecewise-planar geometric constraints.The paper argues this 3D representation enforces depth-normal constraints during training and improves accuracy and stability at inference.Source: §1 · p. 2
- A two-stage CNN estimates normals, depth, and LiDAR confidence, using the confidence branch to reduce propagation of sensor noise.The framework is trained end to end.Source: §1 · p. 2
- The framework is reported to achieve state-of-the-art performance while retaining good generalization capability.The paper reports extensive experimental results supporting this claim.Source: §1 · p. 2
2. Related Work
Related work spans hand-crafted and learned depth completion, geometric constraints between depth and normals, and anisotropic diffusion. The cited approaches leave challenges in generalization, local refinement, LiDAR noise handling, and efficient integration of geometric constraints.
Depth Completion
- Depth completion targets structured-light reconstruction and real-time LiDAR scenarios; classic hand-crafted methods [13, 1, 8, 12, 27, 40, 19, 25, 17] are task-specific and limited in generalization.Learned approaches use sparse depth, including sparsity-invariant convolution [36] and confidence propagation [11], as well as color inputs and encoder-decoder architectures, [23], [14] [16] [33] [4].Source: §2 · p. 2
- Encoder-decoder methods can miss local areas, while Cheng et al. [2] refine in 2D using a locally constant-depth assumption; prior approaches also ignore inevitable LiDAR noise.The cited refinement is suboptimal for depth completion, and its outdoor-scene performance is described as barely satisfactory.Source: §2 · p. 2
Depth and Normal
- Depth-normal constraints support monocular estimation by deriving normals from depth and inversely recovering depth; Zhang et al. [46] formulate geometric constraints as a linear system.That system is difficult to optimize jointly end-to-end and is unsuitable for real-time processing, although it suits post-processing RGB-D camera data.Source: §2 · pp. 2–3
Anisotropic Diffusion
3. Method
The method combines a prediction network with recurrent diffusion refinement, using depth–normal geometry and predicted LiDAR confidence to regularize dense depth completion. It assumes piecewise-planar scenes, where plane-origin distance is piecewise constant.
3. Method
- The framework predicts surface normals, coarse depth, and sparse-input confidence, then recurrently refines depth in plane-origin distance space using depth–normal constraints.This two-stage design is intended to regularize completion while mitigating noisy sparse measurements.Source: §3 · p. 3
3.1. Prediction Network
- A U-Net with a ResNet-34 variant encoder and separate decoders predicts normals, coarse depth, and confidence from sparse depth and the color image.Normal targets come from PCA of nearby 3D ground-truth points and use negative cosine loss [9]; confidence is modeled from sparse-depth error using a Laplace-inspired function.Source: §3.1 · pp. 3–4
3.2. Recurrent Refinement Network
- The recurrent refinement network addresses the prediction backbone’s lack of geometric regularization by diffusing in plane-origin distance space under a piecewise-planar scene assumption.The method uses anisotropic diffusion to enforce piecewise-constant plane-origin distance.Source: §3.2 · p. 4
3.2.1 Plane-origin Distance
- For a 3D point, plane-origin distance is the normal–point dot product and is constant across points on the same plane; refined depth is recovered by inverse transformation.The method converts depth to plane-origin distance using the normal and camera intrinsics, then uses the inverse relation to recover depth.Source: §3.2.1 · p. 4
3.2.2 Plane-origin Distance Diffusion for Depth Refinement
- Diffusion updates each plane-origin distance from neighbors, with learned high-dimensional geometric features determining whether pixels likely share a plane.Separate feature transforms make conductances asymmetric, allowing confident sparse-depth locations to resist incoming information and unreliable locations to limit propagation.Source: §3.2.2 · p. 5
3.2.3 Plane-origin Refinement and Depth Recovery
- At each refinement iteration, confidence-weighted sparse plane-origin distances serve as seeds; the final refined distance map is converted back to depth.The confidence map limits noise propagation while allowing reliable sparse measurements and the predicted depth to complement one another, coupling depth and normals during training.Source: §3.2.3 · p. 5
3.3. Loss Functions
- End-to-end training adds an L2 loss on refined depth to the prediction network’s coarse-depth, normal, and confidence losses.The prediction losses use L2 reconstruction for coarse depth, negative cosine for normals [9], and L2 supervision for confidence.Source: §3.1 · pp. 3–4, §3.3 · p. 5
4. Experiments
Experiments evaluate the method on outdoor KITTI depth completion and indoor NYU-Depth-v2, then examine its benchmark performance, components, generalization, and stability.
4.1. Dataset and Metrics; 4.2. Experimental Setup
- Experiments use KITTI and NYU-Depth-v2 with dataset-specific splits and metrics; training uses PyTorch, ADAM, batch size 16, and an initial learning rate of 4 × 10−4.KITTI has over 93k annotated depth maps; NYU training uses about 43k RGB-depth pairs, down-sampled and center-cropped to 304 × 224. KITTI metrics are RMSE, MAE, iRMSE, and iMAE; NYU metrics include RMSE and relative error.Source: §4.1 · p. 6, §4.2 · p. 6
4.3. Comparison with the State-of-the-Arts
- Our method ranks 1st among peer-reviewed methods on KITTI test RMSE, while geometry-aware diffusion reduces errors and recovers more detail than competing methods.The refinement also mostly eliminates telegraph-pole outliers in the reported qualitative comparison.Source: §4.3 · p. 6, Table 1 · p. 6, Figure 6 · p. 7
4.4. Ablation Study
- Removing components degrades performance; the variant without seed replacement still outperforms the one without refinement, supporting the geometric constraints’ contribution.The study tests depth-only prediction, removal of refinement and confidence, and diffusion without seed replacement.Source: §4.4 · pp. 6–7
- The proposed asymmetric diffusion conductance outperforms symmetric conductance and variants using Euclidean distance or dot-product similarity.All tested ablation variants perform worse than the full model.Source: §4.4 · pp. 6–7
- Removing confidence prediction degrades performance, and confidence is lowest mainly around moving objects and boundaries where noise occurs.Both too-large and too-small tolerance b values degrade performance: the former tolerates noise, while the latter withholds high confidence from valid measurements.Source: Figure 5 · p. 6, Figure 6 · p. 7, §4.4 · p. 8
4.5. Analysis of Generalization Ability and Stability
- On NYU-Depth-v2, our method achieves state-of-the-art performance; its ResNet-34 version matches or exceeds prior ResNet-50 methods, and ResNet-50 improves further.The comparison uses uniformly sampled sparse depth inputs; Table 3 evaluates methods with 500 samples.Source: §4.5 · p. 8, Table 3 · p. 8
- RMSE decreases and stabilizes with more refinement iterations, while the model still produces reasonable results with 1/10 of the original sparse inputs.Performance drops as input sampling decreases, as expected.Source: §4.5 · p. 8, Figure 7 · p. 8
5. Conclusion … 2. Method
The paper proposes a two-module depth-completion framework that uses depth-normal constraints in a diffusion refinement process and predicts sparse-input confidence to reduce noise effects. It reports state-of-the-art performance on outdoor and indoor datasets.
5. Conclusion
- The method achieves state-of-the-art performance on both outdoor and indoor datasets.Source: §5 · p. 8
- The unified framework combines a prediction network with a refinement network for depth completion from sparse inputs.Source: §5 · p. 8
- Depth-normal constraints regularize completion through a diffusion model in plane-origin distance space, shifting the focus from 2D to 3D geometry.Source: §5 · p. 8
- The diffusion model adaptively adjusts conductance between vertex pairs according to their similarities in high-dimensional feature space.Source: §5 · p. 8
- A decoder branch predicts confidence in sparse LiDAR inputs to handle measurement noise and impede error propagation during refinement.Source: §5 · p. 8