Source-linked AI summary
Sparsity Invariant CNNs
Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, Andreas Geiger
TL;DR
Sparse and irregular inputs make standard CNN convolutions difficult to define and learn robustly, even when observation masks are supplied. The paper introduces a validity-aware sparse convolution with a separate validity stream, achieving improved performance and generalization across datasets and sparsity levels while providing a 93k-image depth dataset.
Problem
Standard CNNs struggle with sparse, irregular inputs because varying input locations and binary observation patterns complicate robust representation learning.
Method
The paper introduces a sparse convolution layer that weights kernel elements by input validity and propagates validity information through a second network stream.
Results
The proposed network improves performance over regular convolution modules and generalizes across novel datasets and sparsity levels, with evaluation on synthetic and real data.
Takeaways & Limitations
Sparsity-level invariance improves robustness to random data perturbations and supports generalization across different depth sensors.
Takeaways & Limitations
The method does not explicitly consider structured missing-data patterns encoded in visibility masks, which can benefit standard convolutions for some active-sensor data.
Abstract
from arXiv · showhide
In this paper, we consider convolutional neural networks operating on sparse inputs with an application to depth upsampling from sparse laser scan data. First, we show that traditional convolutional networks perform poorly when applied to sparse data even when the location of missing data is provided to the network. To overcome this problem, we propose a simple yet effective sparse convolution layer which explicitly considers the location of missing data during the convolution operation. We demonstrate the benefits of the proposed network architecture in synthetic and real experiments with respect to various baseline approaches. Compared to dense baselines, the proposed sparse convolution network generalizes well to novel datasets and is invariant to the level of sparsity in the data. For our evaluation, we derive a novel dataset from the KITTI benchmark, comprising 93k depth annotated RGB images. Our dataset allows for training and evaluating depth upsampling and depth prediction techniques in challenging real-world settings and will be made available upon publication.
1. Introduction
Sparse and irregular inputs make standard CNN convolutions difficult to define and can produce poor depth completion results. The paper addresses this with sparsity-aware convolutions and evaluates robustness across datasets and sparsity levels.
- Motivation: When only a fraction of pixels carries information, each filter location receives varying numbers and placements of inputs.This makes the standard convolution operation ambiguous for sparse, irregular data.
- Motivation: Standard CNN processing of sparse laser measurements produces noisy depth maps, whereas the proposed method predicts smoother, more accurate maps.The proposed convolution explicitly considers sparsity during processing.
- Method: The proposed sparse convolution weights kernel elements by input-pixel validity and propagates validity information through a second stream.This design is intended to handle large sparsity levels without significantly compromising accuracy.
- Robustness: The representation remains robust when training and testing use different sparsity levels, supporting applications with changing sensor configurations.The paper specifically identifies robotics as an application requiring robustness to sensor changes.
- Application: The method projects sparse laser scans onto a 2D image plane, enabling depth regression while allowing dense RGB information to be integrated.Projected scans remain sparse and irregular, which causes poor results with standard CNNs.
- Evaluation: Evaluation uses Synthia and a newly proposed KITTI-derived dataset containing 93k depth-annotated images.The experiments include ablations, state-of-the-art baselines, and tests of generalization across datasets and sparsity levels.
2. Related Work
Related work spans sparse-input processing, invariant representations, and depth upsampling. Existing approaches either assume regular data, use masks or default values, require guidance, or primarily target computational efficiency.
- CNNs with Sparse Inputs: Naïve sparse-input CNN methods zero invalid values or append a binary validity mask, but both strategies are reported as suboptimal.The paper contrasts them with explicitly incorporating sparsity inside convolution layers.
- CNNs with Sparse Inputs: The bilateral-filter CNN approach handles irregular inputs but requires guidance and is computationally expensive on large grids.The paper reports that its own sparse convolution networks achieve better depth-upsampling results with regular-CNN efficiency.
- CNNs with Sparse Inputs: Sparse 3D methods partition space to reduce computational and memory demands, while still employing regular convolutions.The paper argues that these convolutions retain the drawbacks of naïve sparse-input processing.
- Sparsity in CNNs: Prior CNN-sparsity work focuses on parameter redundancy, structural regularization, or hardware-friendly speedups rather than input sparsity.Reported examples include eliminating more than 90% of parameters with less than 1% accuracy loss and achieving 3-to-5× speedups.
- Invariant Representations: This paper studies representations invariant to the input’s sparsity level, including cases where training and testing sparsity differ significantly.The stated application is replacing sensors such as laser scanners without retraining.
- Depth Upsampling: Depth upsampling methods address applications including stereo vision, optical flow, and 3D reconstruction from laser scans.Some methods operate directly on depth, while others use high-resolution guidance.
- Depth Upsampling: Many established depth and image superresolution methods assume data lies on a regular grid.This assumption limits their direct applicability to irregular projected laser scans.
- Depth Upsampling: Guided upsampling exploits shared structure between a target domain and a high-resolution guidance image, often through filtering or global optimization.Examples include guided bilateral filtering, energy minimization, compressive sensing, and semantic information.
3. Method
The method explicitly incorporates observation masks into convolution and related network operations, so outputs depend on observed inputs rather than treating missing values as ordinary data. It also propagates visibility information across layers and uses normalized aggregation in skip connections.
- Naïve Approach: Standard handling either replaces missing values with a default such as zero or supplies an observation mask as an additional channel, but both approaches struggle with sparse inputs.The network must distinguish invalid values from observed ones, while possible binary activation patterns grow exponentially with kernel size.
- Sparse Convolutions: The sparse convolution evaluates only observed inputs and normalizes their weighted contribution to account for varying input validity.A small epsilon prevents division by zero when no inputs are observed; with dense input, the operation becomes a scaled standard convolution.
- Sparse Convolutions: The proposed normalization is intended to make filter outputs invariant to the number of observed inputs, without interpolating sparse data or adding distractors.This addresses the irregular variation in observed inputs across filter locations directly on the original input.
- Sparse Convolutions: Observation masks are propagated between layers, marking an output unobserved when none of its filter inputs is observed.Max pooling sets the subsequent mask to 1 when at least one observed variable is visible and to 0 otherwise.
- Skip Connections: Layers that combine multiple preceding outputs, including skip connections, can use normalized sums over observed inputs instead of ordinary aggregation.When all inputs are observed, the normalized expression simplifies to the standard operation.
- Architecture: The network takes a sparse depth map and binary observation mask through sparse convolution layers whose kernel sizes decrease from 11×11 to 3 × 3.The sparse convolution operation supports single-channel and multi-channel input features.
4. Large-Scale Dataset
The paper introduces a fully automatic large-scale KITTI-derived dataset to address the lack of sufficiently large real-world depth annotations for end-to-end training. Its cleaned depth maps achieve high accuracy and substantially greater density than raw LiDAR scans, with remaining errors concentrated mainly on dynamic objects and distant regions.
- Motivation: Synthetic datasets may not reflect challenging real-world performance, while existing sanitized real-world benchmarks are too small for end-to-end training of high-capacity networks.Middlebury provides roughly a dozen images, Make3D 500 low-resolution images, and KITTI 400 street-scene images.
- Dataset Construction: The cleanup pipeline removes LiDAR outliers by enforcing consistency between laser scans and stereo reconstructions, targeting streaking and depth-bleeding artifacts.Large relative errors between LiDAR depth and SGM-derived depth are used to remove inconsistent LiDAR points.
- Evaluation: The generated dataset outperforms all baselines on RMSE, KITTI outliers, and δ inlier ratios while producing depth maps four times denser than raw LiDAR scans.Its MAE is approximately equal to raw LiDAR, whereas the other reported metrics favor the generated dataset.
- Evaluation: Remaining errors are mostly found on dynamic objects or at high distances, while SGM results become inaccurate at large distances.The foreground/background evaluation examines 142 KITTI images and separates overall, foreground, and background errors.
5. Experiments
Experiments evaluate sparse depth completion across synthetic and real datasets, varying sparsity, domains, and RGB guidance. Sparse convolutions consistently generalize better than standard convolutional baselines, including under cross-domain and extreme-sparsity conditions.
- Depth Upsampling: Depth completion reconstructs full-resolution depth maps from sparse, irregular projected laser scans without RGB guidance.
- Depth Upsampling: The Synthia evaluation varies input sparsity through random dropout and compares plain convolutions, masked convolutions, and Sparse Convolution Layers.Dropout probabilities range from 0% to 95%.
- Depth Upsampling: Sparse convolutions outperform plain and validity-mask convolutions, which perform poorly especially with very sparse inputs.Plain convolutions treat valid and invalid pixels alike, while concatenating a validity mask provides only a slight improvement.
- Depth Upsampling: Sparse Convolution Networks maintain performance across sparsity levels, whereas both baselines fail on inputs that are more sparse or denser than training data.With 5% training sparsity and 20% evaluation sparsity, standard CNNs degrade sharply while the proposed network performs equally well.
- Synthetic-to-Real Domain Adaptation: Training on Synthia and evaluating on KITTI shows that sparse convolutions adapt to the new input modality, while dense baselines fail completely.The comparison uses networks with the same number of parameters.
- RGB Guidance: On KITTI, the method performs comparably to RGB-guided state-of-the-art upsampling and outperforms depth-only variants.It ranks second in RMSE on both validation and test splits.
- Sparsity Evaluation on KITTI: KITTI sparsity experiments show that regular convolutions and several baselines perform poorly with sparse inputs, while the proposed network generalizes across input densities.RGB-guided Barron et al. and Ferstl et al. perform slightly better on very sparse data but require dense high-resolution RGB guidance.
- Cross-Dataset Evaluation: The proposed sparse convolution module also outperforms two baselines when trained on Synthia Sequence frames and evaluated on the Synthia Cityscapes subset.The validation subset contains more people and substantially different viewpoints.
6. Conclusion
The paper proposes a sparse convolution module that improves performance while generalizing across domains and sparsity levels, and releases a 93k-image depth-annotated dataset. Future work targets efficiency and sparse 3D CNNs.
- The proposed sparse convolution module replaces regular convolutions, improves performance, and generalizes to novel domains or sparsity levels.
- The authors provide a newly annotated dataset with 93k depth-annotated images for depth prediction and depth upsampling research.
- Future work will combine sparse convolution networks with compression techniques and investigate sparse irregular inputs for 3D CNNs.
Convergence Analysis
The convergence analysis compares Sparse Convolutions with standard convolutions under identical solver settings. Sparse Convolutions converge much faster for most input-output combinations, especially on irregularly sparse Synthia depth inputs.
- Sparse Convolutions converge much faster than standard convolutions for most input-output combinations, especially with irregularly sparse Synthia depth inputs.The comparison uses identical solver settings and reports mean average error in meters over training.
Semantic Segmentation
Sparse convolutions improve semantic labeling from sparse depth maps, including real SUN-RGBD data, while missing-data patterns can sometimes carry useful task information.
- Synthia: Sparse convolutions learn less likely semantic classes that standard convolutions on sparse inputs struggle to classify correctly.The comparison uses 5% sparse input and evaluates class-wise IoU across VGG-like variants.
- SUN-RGBD: The method applies sparse convolutions directly to incomplete depth channels instead of artificially filling depth maps before dense convolution.This targets sparse depth inputs used for semantic segmentation.
- SUN-RGBD: Sparse convolutions outperform standard convolutions on both raw and filled SUN-RGBD depth maps, with or without a concatenated visibility map.Standard convolutions perform particularly poorly on raw sparse depth, although adding visibility information improves them.
- Scope: Missing data can encode useful shape or category information in digit classification, 3D CAD recognition, and sensor-specific object patterns.The paper distinguishes these cases from irregularly distributed missing data that lacks additional information.
Detailed Dataset Evaluation
The generated KITTI depth maps are denser and substantially cleaner than raw LiDAR, with especially large outlier reductions on dynamic objects.
- Overall evaluation: Four times denser and five times less outlier-prone, the generated depth maps outperform the other investigated variants on average.Cleaning sacrifices almost 50% of accumulated LiDAR density but improves quality substantially.
- Foreground and background: Almost 20 times fewer outliers occur on dynamic objects after cleaning, with a similar improvement on the static environment.The comparison separates foreground and background pixels using manually extracted dynamic-object regions.
- Benchmark comparison: The cleaned depth maps achieve KITTI outlier rates of 0.23 for background, 2.99 for foreground, and 0.84 for all pixels.These values are compared with a published stereo result reporting 2.48, 3.59, and 2.67 for the same categories.
Further Depth Upsampling Results
Sparse convolutions upsample randomly distributed Velodyne depth projections while preserving fine structures and smooth planar regions.
- Method: Sparse convolutions apply learned filters only at input pixels with valid depth measurements.The approach internally incorporates input sparsity rather than treating the projected scan as a dense regular image.
- Qualitative results: The depth upsampling results retain fine structures while remaining smooth on flat surfaces.The evaluation uses sparse, randomly distributed Velodyne HDL64 projections.
Boosting Single-Image Depth Prediction
The generated dense depth dataset improves single-image depth prediction, particularly in outlier-prone and previously unseen image regions, with absolute-depth training performing best in the tested setup.
- Training representation: Absolute depth representation outperforms inverse depth for training single-image depth prediction, while L1 and L2 losses perform comparably.The selected setup uses absolute depth with L2 loss because it converges faster.
- Test performance: The generated depth maps outperform raw LiDAR and other promising depth maps on the evaluated test split.The comparison uses the best-performing absolute-depth, L2-loss setup.
- Qualitative effects: Denser, cleaned training data produces visually more pleasant predictions with fewer outliers in occluded regions and better generalization to unseen upper image areas.The qualitative comparison contrasts training with generated dense data against sparse raw LiDAR data.