Source-linked AI summary
HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion
Zixuan Huang, Junming Fan, Shenggan Cheng, Shuai Yi, Xiaogang Wang, Hongsheng Li
TL;DR
Sparse LIDAR depth maps motivate depth completion, which generates dense maps for vision tasks. HMS-Net introduces sparsity-invariant operations and a multi-scale encoder-decoder, achieving strong benchmark performance with and without RGB guidance.
Problem
LIDAR depth maps are generally sparse, creating a need to generate dense depth maps from sparse inputs for computer vision applications.
Method
HMS-Net combines three sparsity-invariant operations with a hierarchical multi-scale encoder-decoder and optional RGB features.
Results
The method outperforms state-of-the-art methods and ranks first without RGB guidance and second among RGB-guided methods on the KITTI leaderboard.
Takeaways & Limitations
The proposed operations enable multi-scale feature fusion for sparse depth completion and show robustness to different levels of data corruption.
Takeaways & Limitations
Existing sparsity-invariant convolution loses spatial information at later stages, and standard multi-scale operations cannot directly handle sparse feature maps.
Abstract
from arXiv · showhide
Dense depth cues are important and have wide applications in various computer vision tasks. In autonomous driving, LIDAR sensors are adopted to acquire depth measurements around the vehicle to perceive the surrounding environments. However, depth maps obtained by LIDAR are generally sparse because of its hardware limitation. The task of depth completion attracts increasing attention, which aims at generating a dense depth map from an input sparse depth map. To effectively utilize multi-scale features, we propose three novel sparsity-invariant operations, based on which, a sparsity-invariant multi-scale encoder-decoder network (HMS-Net) for handling sparse inputs and sparse feature maps is also proposed. Additional RGB features could be incorporated to further improve the depth completion performance. Our extensive experiments and component analysis on two public benchmarks, KITTI depth completion benchmark and NYU-depth-v2 dataset, demonstrate the effectiveness of the proposed approach. As of Aug. 12th, 2018, on KITTI depth completion leaderboard, our proposed model without RGB guidance ranks first among all peer-reviewed methods without using RGB information, and our model with RGB guidance ranks second among all RGB-guided methods.
1 INTRODUCTION
Depth completion addresses sparse LIDAR measurements by generating dense depth maps for computer vision and robotics. HMS-Net introduces sparsity-invariant operations and a hierarchical multi-scale encoder-decoder to fuse features for depth completion.
- Motivation: LIDAR hardware limitations produce sparse depth maps that may be insufficient for applications such as segmentation, 3D detection, odometry, and SLAM.The paper motivates depth completion for self-driving cars and UAVs.
- Prior limitations: Non-learning methods rely on hand-crafted priors, lack robustness to sensor noise, and can produce inaccurate object boundaries.The paper illustrates noise and boundary errors in a hand-crafted method.
- Prior limitations: Existing sparsity-invariant convolution loses spatial information at later stages and cannot directly support multi-scale encoder-decoder operations.Multi-scale fusion also requires sparsity-invariant upsampling, averaging, concatenation, and convolution.
- Proposed method: HMS-Net introduces three sparsity-invariant operations and uses sparsity masks to process sparse inputs and feature maps across an encoder-decoder network.The operations are sparsity-invariant upsampling, sparsity-invariant average, and joint sparsity-invariant concatenation and convolution.
- Results: The method outperforms state-of-the-art depth-completion methods on the KITTI benchmark, with RGB guidance further incorporated in the framework.The supplied introduction also reports separate leaderboard rankings for models with and without RGB guidance.
2 RELATED WORK
Related work distinguishes sparse depth completion from depth enhancement and depth upsampling, while reviewing learning-based, hand-crafted, RGB-guided, and multi-scale approaches. HMS-Net addresses the incompatibility between sparse inputs and standard encoder-decoder operations.
- Depth-completion categories: Sparse depth completion recovers dense maps from spatially sparse inputs, whereas depth enhancement fills small holes in otherwise dense RGB-D depth maps.Regularly sampled inputs define depth upsampling as a special depth-completion case.
- Sparse depth completion: Prior sparse-completion methods include sparsity-invariant CNNs, hand-crafted image processing, and RGB-depth neural networks.These approaches differ in whether they learn from data and whether they use RGB information.
- Depth enhancement: Depth enhancement methods generally process dense depth maps with irregular, relatively small holes using neighborhood, bilateral-filter, or related image-processing strategies.Their input setting differs from sparse depth completion.
- Depth upsampling: Depth upsampling methods increase the resolution of low-resolution depth maps, often using RGB edges or images as guidance.The literature includes both guided and unguided approaches.
- Multi-scale networks: Multi-scale encoder-decoder networks fuse low- and high-level features, but standard versions cannot directly process sparse data because many operations lack sparse-feature support.HMS-Net’s sparsity-invariant operations are presented as the solution to this compatibility problem.
3 METHOD
The method section introduces three sparsity-invariant operations and then builds HMS-Net, a hierarchical multi-scale encoder-decoder for combining features from different CNN layers.
- Framework overview: The framework first reviews sparsity-invariant convolution, then presents three operations for processing sparse inputs and feature maps.These operations enable multi-scale encoder-decoder processing.
- Framework overview: HMS-Net uses the proposed operators to combine multi-scale features for depth completion.The network is described as a hierarchical multi-scale encoder-decoder.
3.1 Sparsity-invariant Convolution
Sparsity-invariant convolution processes a sparse feature map together with a binary mask and normalizes convolution using only valid input locations. The mask is updated across stages to track feature validity.
- Inputs and outputs: Sparsity-invariant convolution takes a sparse feature map and binary sparsity mask as inputs, producing output features at each spatial location.The feature map and mask share spatial dimensions.
- Masked computation: The mask selects valid input features for convolution, while normalization accounts for the number of valid locations and avoids division by zero.A small epsilon is used in the denominator.
- Mask propagation: Each output mask location is set valid when its receptive field contains at least one valid input location.Output masks are modified at every processing stage to match the resulting features.
3.2 Sparsity-invariant operations
The paper extends sparsity-invariant processing beyond convolution so sparse feature maps can pass through multi-scale encoder-decoder networks. The operations track validity with sparsity masks while preserving usable features during upsampling, averaging, and concatenation.
- Three operations—sparsity-invariant upsampling, sparsity-invariant average, and joint sparsity-invariant concatenation and convolution—support sparse feature maps across encoder-decoder networks.They form the foundations of the paper’s multi-scale framework.
- Sparsity masks track valid feature locations and guide operation calculations throughout the network.The masks regularize sparse-feature processing and are updated as feature maps change.
- Sparsity-invariant upsampling: Bilinear upsampling masks invalid inputs, upsamples masked features and masks, then normalizes features using the upsampled mask.The output mask marks locations where the upsampled mask is nonzero.
- Sparsity-invariant average: Sparsity-invariant average combines masked feature maps and masks, then averages valid pairs or copies the sole valid feature.The output mask is valid when either input mask is valid, and normalization keeps the two cases at similar magnitude.
- Feature addition without an output sparsity mask cannot support subsequent sparse convolutions and is therefore limited to a network’s last layer.The proposed average explicitly generates an output mask for continued sparse processing.
- Joint sparsity-invariant concatenation and convolution: Concatenation creates spatial and channel sparsity, so the following convolution must handle distinct validity patterns rather than apply one kernel everywhere.The proposed joint operation uses adaptive 1 × 1 kernels for the different concatenation scenarios while retaining a single-channel output mask.
3.3 Hierarchical Multi-scale Network (HMS-Net) for depth completion
HMS-Net adapts hierarchical multi-scale encoder-decoder design to sparse depth completion, repeatedly transforming and fusing low-, mid-, and high-level features. Its specialized blocks and shortcuts are designed to preserve full-resolution information while incorporating higher-level context.
- Depth completion requires low-level features to undergo nonlinear transformation and repeated fusion with mid-level and high-level features.
- HMS-Net applies sparsity-invariant operations within a hierarchical multi-scale encoder-decoder network for sparse depth completion.
- Two-scale blocks transform full-resolution low-level features alongside downsampled features, then upsample and add higher-level features back to the full-resolution path.
- Three-scale blocks fuse features from two higher levels into the low-level path to provide additional global information.
- The final network uses three multi-scale blocks, sparsity-invariant max-pooling, three upsampling stages, and a 1 × 1 convolution for per-pixel prediction.
- Compared with conventional encoder-decoder structures, the proposed design includes key shortcuts and simultaneous multi-scale feature fusion, which are evaluated experimentally.
3.4 RGB-guided multi-scale depth completion
Because LIDAR depth maps are paired with aligned RGB images, HMS-Net incorporates RGB guidance through an additional feature path. This path extracts mid-level RGB features using an ERFNet-based sub-network.
- Aligned RGB images can provide auxiliary guidance for completing sparse LIDAR depth maps.
- The RGB-guided network adds an RGB feature path whose sub-network extracts mid-level features from the input image.
- The RGB sub-network follows the first six ERFNet blocks, comprising two downsampling blocks and four residual blocks.
3.5 Training scheme
The networks are trained with masked MSE supervision on valid ground-truth locations, while sparsity masks are propagated structurally through the model. Training uses Adam with learning-rate decay for 50 epochs, and the RGB path starts from CityScapes-pretrained parameters.
- MSE loss is evaluated only at coordinates with ground-truth depth annotations when datasets provide sparse ground truth.
- Training uses Adam with an initial learning rate of 0.01 for 50 epochs, with the learning rate decayed during training.
- Sparsity masks are generated without learnable parameters, propagated between layers, and used to filter invalid features and gradients.
- Table 1 reports depth-completion errors for different methods on the KITTI test set.
- The RGB sub-network is initialized from CityScapes-pretrained parameters, while both network paths are trained end-to-end until convergence.
4 EXPERIMENTS
Experiments on KITTI evaluate the proposed network, its components, alternative encoder-decoder structures, and RGB guidance. Multi-scale fusion and sparsity-invariant operations improve completion accuracy, while the proposed structure achieves the lowest RMSE and MAE among compared encoder-decoder networks.
- Data and evaluation: 85,898 depth maps were used for training, with 1,000 each for validation and testing; inputs generally contain fewer than 10% valid depth points.
- Data and evaluation: KITTI evaluation uses RMSE and MAE in millimeters, plus inverse-depth iRMSE and iMAE in 1/km.
- Data and evaluation: RMSE is the main leaderboard metric because squaring magnifies large errors; inverse-depth metrics reduce dominance by errors at large depths.
- Comparison with state-of-the-arts: Without RGB guidance, the proposed network outperforms all other peer-reviewed methods in RMSE, although Spade-sD performs better on MAE, iRMSE, and iMAE.
- Comparison with state-of-the-arts: With RGB guidance, the method ranks second in RMSE behind Sparse-to-dense(gd), which uses additional temporal supervision unavailable to the proposed method.
- Ablation study: RMSE decreases from 1819.81 to 1137.42 when the baseline adds the multi-scale encoder-decoder structure, which fuses information across scales and enlarges receptive fields.
- Ablation study: RMSE improves from 1137.42 to 994.14 after replacing dense operations with sparsity-invariant operations that maintain mask flow for sparse inputs and feature maps.
- Ablation study: RGB feature fusion further improves guidance and produces sharper depth-map boundaries than the corresponding model without RGB guidance.
4.2 Robustness testing on KITTI benchmark
The robustness experiments test noisy and increasingly sparse KITTI inputs without retraining on corrupted data. The proposed method outperforms competing methods under different noise levels and shows the highest tolerance to input sparsity.
- Noise robustness: Robustness tests compare models trained on original data while directly evaluating them on noisy inputs, reflecting the importance of handling LIDAR noise in practice.
- Noise robustness: Scene-level Gaussian noise is added to randomly selected 10% of depth points, with standard deviations ranging from 5 to 50 meters.
- Sparsity robustness: Figure 11 summarizes robustness tests for scene-level noise, region-level noise, and random abandonment of input depth points.
- Noise robustness: The proposed method outperforms SparseConvs and IP-Basic across different noisy depth values.
- Noise robustness: Region-level tests add Gaussian noise to 50% of points in eight randomly selected 25 × 25 pixel regions, simulating large glasses or mirrors.
- Sparsity robustness: After randomly removing 10%-90% of valid depth points without fine-tuning, the proposed method shows the highest tolerance across sparsity levels.
4.3 NYU-depth-v2 dataset
NYU-depth-v2 experiments create synthetic sparse inputs by retaining N depth points from dense maps and evaluate completion across varying sparsity levels. The study reports comparisons with prior methods and RGB-guided results.
- Dataset and setup: NYU-depth-v2 pairs aligned RGB images with dense depth maps and uses synthetic sparse inputs because the dataset was not originally designed for sparse depth completion.
- Dataset and setup: Each input retains N = 20, 50, or 200 randomly selected depth points, with RMSE and REL reported for each setting.
- Comparison: The method is compared with approaches from Ma et al., Jaritz et al., and He et al., using added pooling and batch normalization for the much sparser NYU inputs.
- Results and visualization: The study also reports results with RGB guidance and visual examples for different N values, including inputs, RGB images, predicted dense maps, and ground truth maps.
5 CONCLUSIONS
The paper concludes that its sparsity-invariant operations support a multi-scale encoder-decoder that fuses CNN features for depth completion. Experiments and component analysis show advantages over state-of-the-art methods and robustness to data corruption.
- The proposed sparsity-invariant operations handle sparse feature maps and enable an encoder-decoder network that fuses multi-scale CNN features.
- RGB features are integrated into the framework to provide additional guidance for depth completion.
- Experiments and component analysis report advantages over state-of-the-art methods and robustness against different levels of data corruption.