Source-linked AI summary
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation
Maximilian Jaritz, Raoul de Charette, Emilie Wirbel, Xavier Perrotton, Fawzi Nashashibi
TL;DR
CNNs are built for dense inputs, yet many vision measurements are sparse and missing information must be completed. This paper introduces a sparse-training CNN method with optional late fusion of dense RGB, using the same network for depth completion and semantic segmentation. It reports state-of-the-art Kitti depth-completion results and operation with only 8 Lidar layers.
Problem
CNNs are designed for dense data, although vision inputs can become sparse through reprojection, corrupted measurements, or inherently sparse processes.
Method
The method adapts NASNet with an encoder-decoder, sparse training, late fusion of dense RGB and sparse depth, and no additional validity mask.
Results
The method outperforms all published methods on the Kitti depth-completion benchmark and produces qualitatively remarkable depth maps with only 8 Lidar layers.
Takeaways & Limitations
Sparse depth can be used directly for semantic segmentation, while dense RGB and sparse depth can be efficiently fused for vision tasks with varying input densities.
Takeaways & Limitations
Ground-truth sparsity and limited Lidar field of view leave some regions, such as light poles, unsupervised or unsegmented in the reported settings.
Abstract
from arXiv · showhide
Convolutional neural networks are designed for dense data, but vision data is often sparse (stereo depth, point clouds, pen stroke, etc.). We present a method to handle sparse depth data with optional dense RGB, and accomplish depth completion and semantic segmentation changing only the last layer. Our proposal efficiently learns sparse features without the need of an additional validity mask. We show how to ensure network robustness to varying input sparsities. Our method even works with densities as low as 0.8% (8 layer lidar), and outperforms all published state-of-the-art on the Kitti depth completion benchmark.
1. Introduction
Vision data is often sparse, while CNNs are designed for dense inputs. The paper proposes a mask-free sparse-data method that supports depth completion and semantic segmentation across varying densities.
- Motivation: CNNs are designed for dense data, but reprojection, corrupted measurements, and inherently sparse processes produce sparse vision inputs.Examples include Lidar projected into the camera plane, invalid or noisy measurements, and stereo disparity.
- Approach: The method handles sparse depth with or without dense RGB for depth completion or semantic segmentation, requiring only minor adaptation.Figure 1 presents the shared use of sparse depth alone or fused with dense RGB across both tasks.
- Motivation: 6% pixels from a 64-layer Velodyne Lidar cover the Kitti image space, with lower densities from fewer layers.The supplied passage reports 5.9% coverage; the accompanying density passage simulates 13.0%, 1.6%, and 0.8% for 32, 16, and 8 layers.
- Approach: Unlike sparse convolution approaches using a validity mask, the proposed normal CNN with sparse training learns features invariant to missing data without an additional mask.The paper reports that proper training lets CNNs learn where valid inputs are and how to use them.
- Evaluation: The experiments cover synthetic and real data, varying densities, sparse-depth inputs, and sparse-depth plus dense-RGB fusion.The contribution passage also reports an ablation study targeting robustness at lower data density.
2. Related Work
Prior work develops dense-input pixel-wise inference, sparse-input processing, and limited sparse-plus-dense fusion strategies. The paper positions its method within these approaches to produce dense image-plane outputs from sparse data.
- Scope: Pixel-wise inference tasks produce dense image-plane outputs, motivating overviews of dense, sparse, and sparse-plus-dense processing.The related-work framing covers depth prediction and semantic segmentation.
- Dense Inputs: Dense-input methods use encoder-decoder or multiscale CNNs, with skip connections and pyramid pooling helping preserve spatial detail and context.FCNs upsample downsampled features, while skip connections connect corresponding encoder and decoder maps.
- Dense Inputs: Depth prediction can learn priors from single images, while supervision may use sparse Lidar measurements or self-supervised stereo reprojection.Semantic segmentation differs in requiring costly annotation, according to the passage.
- Sparse Inputs: Sparse-input patterns differ by sensor: Lidar is structured, stereo or structured light has patches of missing data, and synthetic sparsity is typically uniform.Classical methods address sparse 2D inputs using local interpolation or guided optimization.
- Sparse Inputs: CNN sparse-processing methods commonly grid sparse data and may add validity masks, while alternatives operate on the data manifold or use order-invariant operations.Validity-aware convolution considers only valid locations and normalizes by the number of valid pixels.
- Sparse + Dense Inputs: Sparse-plus-dense fusion remains little addressed, with prior methods using sparse depth to guide RGB inference or concatenating RGB and sparse depth.The cited approaches include optimization guided by sparse depth and direct modality combination.
3. Method
The method adapts an encoder-decoder CNN to sparse depth, optionally fusing dense RGB, while using sparse training to handle varying input densities without a validity mask.
- Method scope: The method targets sparse depth completion with or without dense RGB and also supports semantic segmentation through minor adjustments.Its common network structure is intended for pixel-wise inference with sparse inputs and optional data fusion.
- 3.1. Network Architecture: An adapted NASNet encoder-decoder processes sparse inputs, with batch normalization removed after the first strided convolution because missing pixels are represented by zeros.The decoder uses transposed convolutions, normal convolutions, and skip connections between equivalent-resolution stages.
- 3.2. Analysis of Validity Mask: Validity masks are omitted because large networks can learn sparsity-invariant features while preserving spatial information about valid pixels.Mask propagation can quickly saturate at deeper layers, and adding mask channels did not improve performance for large NASNet models.
- 3.3. Sparse Data Training: Training on synthetic densities sampled across ]0, 1] improves robustness to varying input sparsity.The paper contrasts this with prior work that uses fixed density during training.
- Training objective: Losses for depth completion are computed only on unobserved ground-truth pixels, prioritizing prediction of unknown values over reproducing measured data.The method trains with inverse depth and uses an L1 loss, which performed slightly better than L2 for depth prediction.
- 3.4. Sparse Depth + RGB Fusion: Late fusion transforms RGB and sparse-depth modalities into feature spaces before channel-wise concatenation and a following convolution.This design accommodates their different representations while allowing the branches to provide distinct information.
4. Experiments
Experiments evaluate sparse-depth processing for depth completion and semantic segmentation on synthetic and real datasets, using sparse depth, RGB, or their fusion. The method handles varying densities, achieves strong Kitti benchmark results, and benefits from late RGB–depth fusion.
- Experimental setup: Experiments test sparse depth, dense RGB, and their fusion for depth completion and semantic segmentation on synthetic and real public datasets.The datasets include Synthia, Kitti, and Cityscapes.
- Depth completion on Synthia: At 0.02 input density, sparse depth outperforms RGB-only depth completion, while late fusion achieves the best reported iMAE of 2.96.Sparse depth alone reports 4.05, compared with 13.56 for RGB alone.
- Depth completion on Synthia: Late fusion outperforms early fusion because separate modality branches map RGB and sparse-depth features into a similar feature space before fusion.Early fusion performs approximately as well as sparse depth alone, suggesting weak integration of the less informative modality.
- Varying density: Training across varying densities enables performance across a large variety of sensor densities, whereas fixed-density training is evaluated at its trained density for best test-time results.The varying-density method is highlighted as especially relevant to Lidar applications.
- Depth completion on Kitti: On the Kitti depth-completion benchmark, the method ranks first among published methods on all metrics and third on RMSE when anonymous submissions are included.The benchmark comparison uses iRMSE, iMAE, RMSE, and MAE.
- Semantic segmentation: For semantic segmentation, RGB plus sparse depth improves over the RGB-only baseline, while sparse depth alone remains usable on synthetic and real data.The method also handles Cityscapes stereo disparity, although thin structures such as light poles are never segmented.
- Semantic segmentation: The method directly processes sparse depth for semantic segmentation without first generating a dense depth map and supports different densities and sparsity types when train and test sparsity match.This extends the demonstrated approach beyond depth completion with only minor task-specific adjustment.
5. Conclusion
The method handles sparse depth through a modified NASNet, decoder, sparse training, and late RGB fusion without an additional mask. It outperforms published Kitti depth-completion methods, works with 8-layer lidar, and supports semantic segmentation by changing only the last layer.
- The method combines a modified NASNet, decoder, sparse training strategy, and late fusion of dense RGB with sparse depth.
- It learns sparsity-invariant features without using an additional validity mask.
- It outperforms all published methods on the Kitti depth-completion benchmark and produces qualitatively remarkable results with only 8-layer lidar.
- Changing only the last layer enables semantic segmentation on synthetic and real datasets.
- The proposed method efficiently fuses dense RGB and sparse depth, with potential benefits for vision tasks using inputs of various densities.