Source-linked AI summary
Sparse and noisy LiDAR completion with RGB guidance and uncertainty
Wouter Van Gansbeke, Davy Neven, Bert De Brabandere, Luc Van Gool
TL;DR
Sparse LiDAR depth completion must produce precise dense maps despite irregular sampling and noisy inputs. The paper combines global and local processing with RGB guidance and confidence-weighted fusion, ranking first on KITTI with and without RGB. Its conclusion supports a real-time framework for completing and correcting sparse LiDAR frames.
Problem
Depth completion seeks dense, precise depth from sparse and irregular LiDAR, whose sparsity and errors challenge autonomous-vehicle perception.
Method
The framework combines global and local depth information, uses monocular RGB as guidance, and learns confidence maps for per-pixel late fusion.
Results
The method ranks first on the KITTI depth completion benchmark with and without RGB images.
Takeaways & Limitations
The framework completes and corrects sparse LiDAR frames while meeting real-time requirements with 20 ms inference time.
Takeaways & Limitations
The framework assumes global information can regularize local predictions and help detect LiDAR artifacts and shared-depth structures.
Abstract
from arXiv · showhide
This work proposes a new method to accurately complete sparse LiDAR maps guided by RGB images. For autonomous vehicles and robotics the use of LiDAR is indispensable in order to achieve precise depth predictions. A multitude of applications depend on the awareness of their surroundings, and use depth cues to reason and react accordingly. On the one hand, monocular depth prediction methods fail to generate absolute and precise depth maps. On the other hand, stereoscopic approaches are still significantly outperformed by LiDAR based approaches. The goal of the depth completion task is to generate dense depth predictions from sparse and irregular point clouds which are mapped to a 2D plane. We propose a new framework which extracts both global and local information in order to produce proper depth maps. We argue that simple depth completion does not require a deep network. However, we additionally propose a fusion method with RGB guidance from a monocular camera in order to leverage object information and to correct mistakes in the sparse input. This improves the accuracy significantly. Moreover, confidence masks are exploited in order to take into account the uncertainty in the depth predictions from each modality. This fusion method outperforms the state-of-the-art and ranks first on the KITTI depth completion benchmark. Our code with visualizations is available.
1 Introduction
Depth completion generates dense maps from sparse, irregular LiDAR points, but sparsity, multimodal fusion, and incomplete annotations make the task challenging. The proposed framework combines global and local information, RGB guidance, and confidence-weighted late fusion, ranking first on KITTI with and without RGB.
- Motivation: Depth completion predicts dense depth maps from sparse point clouds for applications requiring precise depth cues.The paper focuses on self-driving cars using sparse LiDAR and monocular RGB images.
- Challenges: Random point spacing, multimodal fusion choices, and partially completed annotations create three central challenges.Limited LiDAR scan lines produce highly sparse inputs, while pixel-wise ground truth is expensive to construct.
- Contributions: Global and local information are combined to complete and correct sparse LiDAR input, with monocular RGB images providing guidance.The contributions explicitly pair global-local processing with RGB guidance.
- Contributions: Confidence maps are learned for both branches and used to weight their predicted depth maps in late fusion.The confidence maps are learned in an unsupervised manner.
- Results: The method ranks first on the KITTI depth completion benchmark with and without RGB images and requires no additional data or postprocessing.This is reported as the paper’s benchmark contribution.
2 Related Work
Prior depth-completion work addresses sparse inputs through handcrafted interpolation, CNNs, sparsity-aware operations, and multimodal fusion. RGB guidance and fusion remain active research directions, motivating the paper’s combined strategy.
- Background: Depth completion relates to inpainting, denoising, and superresolution because it completes missing information while correcting input errors.These related sub-tasks broaden the relevance of depth completion.
- Handling sparse data: Handcrafted interpolation and morphological preprocessing methods can be prone to errors in the LiDAR frame.The paper contrasts these approaches with CNN-based methods.
- Handling sparse data: Recent CNN methods encode sparse values with zeros, while this paper argues that combining local and global networks yields better results.The cited prior work frames deep networks as necessary, whereas this paper proposes a different architecture.
- Handling sparse data: Sparsity-invariant and confidence-propagating convolutions explicitly account for varying input sparsity during network processing.These methods propagate validity or confidence information through layers.
- RGB guidance and fusion: RGB fusion methods use semantic edges, RGB-D streams, surface normals, and early, late, or multi-level fusion to improve depth completion.The literature includes both guidance and feature-combination strategies.
- RGB guidance and fusion: The paper’s framework combines global and local branches, using global guidance for the local network and confidence-based late fusion.The figure description presents the global branch above the local branch and their confidence-weighted combination.
3 Method
The method treats projected LiDAR depth completion as supervised regression using semi-sparse targets, combining global and local information with RGB guidance and learned uncertainty. Confidence maps weight the branches per pixel to produce the final depth map.
- Method: The approach regresses dense depth from a 2D projection of a 3D point cloud, encoding missing LiDAR values with zeros.Training uses sparse input and semi-sparse ground truth built with SGM and temporal information.
- Method: Global and local networks jointly complete the input, with the global branch using LiDAR and RGB information and the local branch interpreting local structure.Both parts are used because LiDAR contains errors from moving objects and moving sensors.
- Global information: The global branch acts as a prior that helps the local path detect LiDAR artifacts and reconstruct sparse input more accurately.RGB color information helps identify object borders and structures likely to share depth.
- Local information: The local network fuses the LiDAR map with global guidance and uses residual learning through skip connections over small local networks.The guidance is intended to help focus on correct and confident LiDAR points.
- Uncertainty fusion: Both branches predict confidence maps that weight their depth predictions per pixel in an unsupervised late-fusion process.The network can prefer local information where LiDAR is accurate and global information where LiDAR is scarce or incorrect.
- Network: The global branch uses an ERFNet-based encoder-decoder, while the local branch uses a stacked hourglass network with two modules and approximately 350k parameters.The local network downsamples twice and omits batch normalization in specified early layers because input sparsity can vary.
4 Experiments
Experiments evaluate the framework on KITTI through ablation, loss analysis, benchmark comparison, and qualitative examples. The results show benefits from global-local fusion, uncertainty guidance, and RGB information while maintaining efficient inference.
- Experimental setup: KITTI evaluation uses 85,898 training frames, 1,000 evaluation frames, and 1,000 testing frames, with RMSE as the primary benchmark metric.MAE is also reported, and losses are computed over all ground-truth pixels because some LiDAR inputs are incorrect.
- Loss analysis: The proposed focal-MSE loss slightly outperforms vanilla-MSE and BerHu for depth prediction.Its focal term gives wrongly predicted points somewhat higher training weight.
- Ablation study and analysis: Combining global and local networks produces strong ablation results, while uncertainty-based late fusion provides a large accuracy gain.The final loss combines global, local, and output depth predictions with weights w1 = 0.1, w2 = 0.1, and w3 = 1.
- Benchmark comparison: The framework outperforms F. Ma et al. on all reported KITTI test metrics while achieving four times higher frame rate.It also ranks first using LiDAR alone, without RGB images.
- Qualitative analysis: Confidence maps are more certain around edges and locations where the LiDAR sensor is incorrect, supporting correction of sparse-input artifacts.Qualitative comparisons also show more accurate depth around close and far objects, especially near object boundaries.
5 Conclusion
The paper presents a framework that completes and corrects sparse LiDAR using RGB guidance, global-local processing, and confidence-based late fusion. It reports 20 ms inference time and first place on the KITTI depth completion benchmark.
- Conclusion: The framework uses RGB-guided global information and confidence maps to combine inputs according to their uncertainty in late fusion.It is designed to complete and correct sparse LiDAR frames.
- Conclusion: 20 ms inference time meets the stated real-time requirements for self-driving cars with a large margin.
- Conclusion: The method ranks first on the KITTI depth completion benchmark.