Source-linked AI summary
Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera
Fangchang Ma, Guilherme Venturelli Cavalheiro, Sertac Karaman
TL;DR
Depth completion must infer dense depth from sparse, irregular LiDAR while incorporating color images and avoiding costly dense labels. The paper introduces a direct regression network and a self-supervised framework using RGB and sparse-depth sequences; supervised training reaches state-of-the-art KITTI performance, and self-supervised training outperforms several semi-dense-label methods.
Problem
Depth completion is challenged by irregular sparse measurements, multimodal depth-color inputs, and the scarcity of scalable dense pixel-level annotations.
Method
The paper develops a deep regression network mapping sparse depth and optional color images to dense depth, plus self-supervised training using RGB and sparse-depth sequences.
Results
The network achieves state-of-the-art KITTI depth completion performance, while the self-supervised framework outperforms existing solutions trained with semi-dense annotations.
Takeaways & Limitations
Depth completion networks can be trained without dense labels using sequences of RGB and sparse depth images, while retaining competitive performance against semi-supervised alternatives.
Takeaways & Limitations
Evaluation metrics may underestimate self-supervised prediction quality because the ground truth is low-density and lacks labels in the top image region.
Abstract
from arXiv · showhide
Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) to dense depth. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that our network, when trained with semi-dense annotations, attains state-of-the- art accuracy and is the winning approach on the KITTI depth completion benchmark at the time of submission. Furthermore, the self-supervised framework outperforms a number of existing solutions trained with semi- dense annotations.
1 Introduction
Depth completion is valuable for robotics and autonomous vehicles but is difficult because LiDAR measurements are sparse and irregular, multimodal fusion is non-trivial, and dense labels are costly. The paper addresses these challenges with a direct regression network and a scalable self-supervised training framework.
- Depth completion estimates dense depth from sparse LiDAR measurements, supporting obstacle avoidance, 3D mapping, and localization.
- Sparse LiDAR measurements are irregularly spaced, while combining depth with color images involves different sensor modalities.
- Dense pixel-level ground truth is generally unavailable because acquiring it is labor-intensive and non-scalable.
- The proposed architecture learns a direct mapping from sparse depth and optionally color images to dense depth.
- The network achieves state-of-the-art KITTI depth completion accuracy, while the self-supervised framework outperforms some methods trained with semi-dense annotations.
- The self-supervised framework trains depth completion networks using sparse 3D LiDAR and monocular color-image sequences without dense labels.
2 Related Work
Related work distinguishes depth completion settings by sensor modality and input density. Sparse LiDAR completion is substantially more challenging than filling small holes in relatively dense depth images and has motivated deep, interpolation-based, and sparse-convolution approaches.
- Depth completion spans different input densities and modalities, including sparse or relatively dense depth and optional color guidance.
- Structured-light completion typically fills small holes in images where over 80% of pixels are observed, making simple filtering effective.
- Low-density completion is more challenging because the inverse problem is ill-posed; autonomous-driving LiDAR projections cover roughly 4% of image pixels.
- Prior sparse-depth approaches include piecewise-linearity assumptions, end-to-end deep regression, CPU interpolation, and sparse convolution.
- Monocular depth prediction has evolved from handcrafted features to deep learning, but most learning-based methods relied on pixel-level ground truth.
3 Network Architecture
The model formulates depth completion as deep regression using an encoder-decoder architecture. Sparse depth and color inputs are initially processed separately, fused before residual blocks, and connected across scales through skip connections.
- The network uses an encoder-decoder paradigm, with convolutions downsampling features and transposed convolutions restoring spatial resolution.
- Sparse depth and RGB inputs are separately convolved before their outputs are concatenated.
- The fused representation is processed by ResNet-34 residual blocks, while encoder outputs connect to corresponding decoder layers through skip connections.
- A final 1x1 convolution produces a single prediction image at the network input resolution.
- Without color images, the RGB branch is removed and the number of filters is reduced by half.
4 Self-supervised Training Framework
The framework trains depth completion using synchronized monocular color and sparse LiDAR depth sequences, combining sparse-depth supervision with geometric and smoothness losses. It estimates pose modelically and uses differentiable image warping for photometric supervision.
- The framework requires only synchronized monocular color/intensity sequences and sparse LiDAR depth images, without additional sensors, manual labeling, or learned building blocks.
- Sparse Depth Supervision: Sparse-depth supervision penalizes differences between input and output at known-depth pixels, encouraging identity mapping and improving accuracy, stability, and convergence.
- Model-based Pose Estimation: Relative pose is estimated with PnP and RANSAC from matched RGB-depth and RGB feature correspondences, producing scale-accurate and failure-aware transformations.
- Photometric Loss as Depth Supervision: The photometric term inversely warps a nearby color image using predicted depth, camera intrinsics, and the estimated transformation, with bilinear interpolation making the loss differentiable.
- Photometric Loss as Depth Supervision: Photometric supervision assumes static environments with limited occlusion and is effective when projected correspondences are within 1 pixel of the truth.
- Smoothness Loss: A smoothness term addresses incorrect, discontinuous depth solutions from photometric loss alone by penalizing second-order depth derivatives to encourage piecewise-linear predictions.
- The final self-supervised objective combines depth, photometric, and smoothness terms, with β1 = 0.1 and β2 = 0.1.
5 Implementation
Experiments use KITTI depth completion data, with semi-dense ground truth for training and validation but no ground truth for the test split. Training uses randomized temporal neighbors and specified PyTorch optimization settings.
- The KITTI dataset contains 85,898 training samples, 1,000 validation samples, and 1,000 test samples without ground truth.
- KITTI semi-dense ground truth aggregates LiDAR scans from 11 consecutive frames and annotates roughly 30% of pixels.
- Each epoch pairs the current frame with a randomly selected neighbor among the 6 nearest temporal frames, while sparse depth is dilated with a 4 × 4 kernel for PnP.
- The implementation uses PyTorch, batch sizes of 8 for the RGBd-network and 16 for the d-network, and Adam with an initial learning rate of 10^-5.
- Training uses 8 Tesla V100 GPUs; 12 epochs take roughly 12 hours for the RGBd-network and 4 hours for the d-network.
6 Results
The experiments evaluate supervised benchmarking, architectural ablations, self-supervised training, and robustness to sparse or lower-resolution LiDAR inputs. The proposed methods lead prior benchmark results, while self-supervision remains effective with sufficient measurements but is evaluated against biased ground truth.
- 6.1 Comparison with State-of-the-art Methods: The RGBd-network leads all submissions on the KITTI depth completion benchmark, while the d-network leads prior work with a large margin in almost all metrics.Predictions also have cleaner and sharper object boundaries.
- 6.2 Ablation Studies: Using RGBd input and L2 loss are the most effective architectural choices for improving final accuracy.Skip connections, training from scratch, avoiding max pooling, greater depth, more encoder-decoder pairs, and a 16/48 RGB-depth filter split also improve results.
- 6.2 Ablation Studies: Dropout combined with weight decay degrades performance, while interpolated sparse-depth encodings do not improve prediction accuracy.The latter result indicates that the proposed network handles highly sparse input images directly.
- 6.3 Evaluation of the Self-supervised Framework: rmse = 1384 is achieved by the self-supervised framework, outperforming some prior methods trained with semi-dense annotations, including SparseConvs.The comparison evaluates photometric-only, complete self-supervised, and purely supervised training methods.
- 6.3 Evaluation of the Self-supervised Framework: Self-supervised evaluation metrics underestimate prediction quality because the ground truth lacks density and top-region labels where self-supervision provides supervision.Figure 5 highlights supervision at the top that semi-dense annotations do not provide.
- 6.4 On Input Sparsity: With sufficiently many measurements, self-supervised validation error decreases as a power function and works with lower-resolution LiDAR; at least 4 scanlines are cited as effective.Uniform sampling requires an equivalent number of samples to at least 2 scanlines.
- 6.4 On Input Sparsity: Under semi-dense training, uniform random sub-sampling is more accurate than reduced scan lines, and RGBd especially helps when depth measurements are few.Increasing measurements yields diminishing returns, with error decreasing as a power function of input depth measurements.
7 Conclusions
The paper presents a deep regression model for sparse-LiDAR depth completion and a scalable self-supervised framework requiring only RGB and sparse-depth sequences. The model achieves state-of-the-art KITTI performance, while the self-supervised framework outperforms several methods trained with semi-dense annotations.
- The deep regression model achieves state-of-the-art performance on the KITTI depth completion benchmark and significantly outperforms existing published work at submission time.
- The self-supervised framework requires only sequences of RGB and sparse depth images, avoiding dense depth labels.
- The self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations.
- The authors conjecture that RGB weights may become negligible during early self-supervised training iterations, but this explanation remains unverified.
- Future work will investigate improved loss functions and methods that account for dynamic objects.