Source-linked AI summary
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image
Jiaxiong Qiu, Zhaopeng Cui, Yinda Zhang, Xingdi Zhang, Shuaicheng Liu, Bing Zeng, Marc Pollefeys
TL;DR
The paper addresses the lack of affordable dense outdoor depth by proposing an end-to-end network that combines sparse LiDAR with a color image through surface-normal estimation. Its confidence masking and attention-based integration address occlusion and distance-related challenges, and experiments report state-of-the-art benchmark performance with generalization to sparser and indoor inputs.
Problem
Affordable immediate access to dense and accurate outdoor depth remains unavailable because high-end LiDAR is expensive and commodity LiDAR has low resolution.
Method
An end-to-end network uses surface normals as an intermediate representation, fuses sparse LiDAR with color through modified encoder-decoder pathways, and learns confidence masks and attention maps.
Results
The method achieves state-of-the-art performance on the benchmark and generalizes well to higher input sparsity and indoor scenes.
Takeaways & Limitations
Surface normals are effective for outdoor depth completion, while learned confidence and attention mechanisms address occlusion and improve far-distance integration.
Abstract
from arXiv · showhide
In this paper, we propose a deep learning architecture that produces accurate dense depth for the outdoor scene from a single color image and a sparse depth. Inspired by the indoor depth completion, our network estimates surface normals as the intermediate representation to produce dense depth, and can be trained end-to-end. With a modified encoder-decoder structure, our network effectively fuses the dense color image and the sparse LiDAR depth. To address outdoor specific challenges, our network predicts a confidence mask to handle mixed LiDAR signals near foreground boundaries due to occlusion, and combines estimates from the color image and surface normals with learned attention maps to improve the depth accuracy especially for distant areas. Extensive experiments demonstrate that our model improves upon the state-of-the-art performance on KITTI depth completion benchmark. Ablation study shows the positive impact of each model components to the final performance, and comprehensive analysis shows that our model generalizes well to the input with higher sparsity or from indoor scenes.
1. Introduction
The paper targets affordable dense outdoor depth from sparse LiDAR and color imagery, using surface normals as an intermediate representation while addressing fusion, occlusion, and distance-dependent errors.
- Dense and accurate outdoor depth is important for autonomous driving and unmanned aerial vehicles, but affordable immediate access remains unavailable.
- The proposed system produces dense depth from sparse LiDAR and a color image using surface normals as an intermediate representation.The authors investigate whether this representation, previously used indoors, is suitable for outdoor scenes.
- Surface-normal depth reconstruction becomes more sensitive to normal noise as distance increases, motivating special treatment of distant regions.With the same 15° normal error, the farther case produces much larger depth error.
- The architecture addresses data fusion through a modified encoder-decoder design and combines color- and normal-based depth estimates with learned attention maps.The attention maps select the pathway likely to perform better in each area.
- The network learns a confidence mask to handle mixed foreground and background LiDAR signals caused by occlusion near depth boundaries.This replaces reliance on a simple binary availability mask and estimates sparse-depth reliability within the network.
2. Related Work
Related work covers sparse-depth completion, indoor depth refinement, and monocular depth estimation; DeepLiDAR differs by explicitly using surface normals as an intermediate representation for outdoor completion.
- Sparse-depth completion methods densify accurate but low-resolution sensor measurements using techniques including wavelet analysis, sparsity-invariant CNNs, and encoder-decoder networks.
- DeepLiDAR is more physically driven than prior approaches because it explicitly exploits surface normals as the intermediate representation.
- Indoor depth-refinement methods use aligned high-resolution color images for depth super-resolution or color-guided depth inpainting.Depth super-resolution methods assume a low-resolution but dense depth map without missing signals.
- Single-RGB depth methods estimate depth or normals with deep networks, but they generally do not handle sparse depth as an additional input.
- The paper presents surface normals as an intermediate representation specifically for outdoor depth completion, distinguishing it from prior independent or indoor-focused predictions.
3. Method
The method uses parallel color and surface-normal pathways to complete sparse LiDAR depth, with learned confidence and attention mechanisms addressing outdoor occlusions and pathway reliability.
- Overall architecture: The network takes RGB and reprojected sparse depth inputs, then integrates outputs from color and surface-normal pathways using a learned weighted sum.The surface-normal pathway predicts normals before combining them with sparse depth and a learned confidence mask.
- Deep completion unit: Each pathway is implemented with deep completion units that fuse RGB or normal features with sparse-depth features in the decoder.The unit uses late fusion and sums features from both inputs at each decoder resolution.
- Attention-based integration: Attention maps determine how strongly to combine depths predicted from the color and surface-normal pathways according to the current context.Two score maps are converted through softmax into combination weights for the final dense depth.
- Confidence prediction: The model learns a confidence mask to reduce the influence of sparse-depth values corrupted by foreground/background mixing near occlusion boundaries.The mask replaces a binary availability mask and is learned without corresponding ground-truth masks.
- Loss function: The overall loss jointly supervises pathway depths, final depth, and surface normals, with staged training used for stable convergence.Depth terms use L2 and converted-normal cosine losses, while normal estimation uses cosine loss.
- Training data: Training uses synthetic urban-driving data with rendered normals, followed by KITTI depth-completion data for real-data finetuning and evaluation.The synthetic dataset contains 50K rendered training samples.
4. Experiments
Experiments on KITTI show that DeepLiDAR achieves leading depth-completion performance, while ablations identify surface normals, attention integration, confidence masks, and the deep completion unit as important components. The model also remains effective with extremely sparse LiDAR input and transfers well to indoor scenes.
- KITTI comparison: The method ranked first on the KITTI test leaderboard and significantly outperformed the second-ranked method.RMSE was the leaderboard ranking metric; the model ran at 0.07 seconds per image on an Nvidia GTX 1080Ti.
- KITTI comparison: On KITTI validation, the method significantly outperformed bilateral, fast bilateral, TGV, and Zhang et al. methods.Qualitatively, bilateral methods oversmooth boundaries, TGV produces noisy smooth surfaces, and Zhang et al. performs worse in far areas.
- Ablation study: The full model performs best in ablation studies; removing the normal pathway increases RMSE by about 87mm, while removing the deep completion unit also causes a significant drop.These results support surface normals as an outdoor depth representation and the modified fusion architecture as useful components.
- Ablation study: Removing attention integration reduces every evaluation metric because surface-normal estimates are stronger nearby while the color pathway helps at far distances.The learned attention maps combine the two pathways according to their relative performance across regions.
- Ablation study: Replacing the learned confidence mask with a binary availability mask increases RMSE by 69mm compared with the full model.Low confidence is assigned near foreground boundaries where occlusion mixes foreground and background LiDAR depths.
- Generalization capability: With only 0.0168% of pixels containing depth, the method still performs reasonably well and outperforms traditional methods given their full 4.3% input.The indoor NYUv2 evaluation also finds comparable or better state-of-the-art performance, ranking first on four of five metrics.
5. Conclusion
The proposed end-to-end network predicts dense outdoor depth from sparse LiDAR and a single color image, using surface normals and outdoor-specific mechanisms. It achieves state-of-the-art benchmark performance and generalizes to sparser inputs and indoor scenes.
- The network predicts dense depth from sparse LiDAR data and a single color image using surface normals as an intermediate representation.It is trained end-to-end and applies the surface-normal representation to outdoor scenes.
- A deep completion unit fuses the color image with sparse depth, while attention-based integration improves performance in far-distance regions.The architecture also estimates a confidence mask to handle occlusion-related mixed LiDAR signals.
- The method achieves state-of-the-art benchmark performance and generalizes well to sparser input and indoor scenes.