Source-linked AI summary
CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion
Xinjing Cheng, Peng Wang, Chenye Guan, Ruigang Yang
TL;DR
Depth completion must recover dense depth from sparse measurements and images, despite noisy inputs and fixed propagation choices. CSPN++ learns pixel-wise kernel sizes and iteration counts, using assembly for context awareness and selection for resource awareness, with gating for sparse-depth reliability. The resulting variants reduce estimation error or improve efficiency, and resource use can be adapted to computational budgets.
Problem
Noisy sparse depths and fixed, uniformly treated CSPN propagation choices limit how depth completion can preserve structure and allocate context.
Method
CSPN++ learns pixel-wise kernel sizes and iteration counts through weighted assembly or selection, adds budget-aware resource allocation, and uses a gate for sparse-depth preservation.
Results
CSPN++ improves over CSPN, with context-aware CSPN reducing estimation error and resource-aware CSPN achieving much better efficiency at comparable accuracy.
Takeaways & Limitations
Propagation context and computational resources can be dynamically assigned per pixel, while resource-aware CSPN can adapt to a provided computational budget.
Abstract
from arXiv · showhide
Depth Completion deals with the problem of converting a sparse depth map to a dense one, given the corresponding color image. Convolutional spatial propagation network (CSPN) is one of the state-of-the-art (SoTA) methods of depth completion, which recovers structural details of the scene. In this paper, we propose CSPN++, which further improves its effectiveness and efficiency by learning adaptive convolutional kernel sizes and the number of iterations for the propagation, thus the context and computational resources needed at each pixel could be dynamically assigned upon requests. Specifically, we formulate the learning of the two hyper-parameters as an architecture selection problem where various configurations of kernel sizes and numbers of iterations are first defined, and then a set of soft weighting parameters are trained to either properly assemble or select from the pre-defined configurations at each pixel. In our experiments, we find weighted assembling can lead to significant accuracy improvements, which we referred to as "context-aware CSPN", while weighted selection, "resource-aware CSPN" can reduce the computational resource significantly with similar or better accuracy. Besides, the resource needed for CSPN++ can be adjusted w.r.t. the computational budget automatically. Finally, to avoid the side effects of noise or inaccurate sparse depths, we embed a gated network inside CSPN++, which further improves the performance. We demonstrate the effectiveness of CSPN++on the KITTI depth completion benchmark, where it significantly improves over CSPN and other SoTA methods.
Introduction
CSPN++ extends CSPN depth completion by adapting propagation context and computational resources per pixel, while addressing noisy sparse depths with confidence-aware preservation.
- Motivation: Depth completion converts sparse sensor- or algorithm-derived depth maps into dense per-pixel maps using reference images.The paper highlights applications including self-driving and indoor robots, and augmented reality.
- Motivation: CSPN is a state-of-the-art local linear propagation model that learns pixel affinities with a CNN.Its design emphasizes depth preservation, structure alignment, and smooth transitions between sparse points and neighborhoods.
- Motivation: Noisy sparse depths and equal treatment of pixels motivate CSPN++ to learn propagation context suited to image content.The paper specifically identifies geometrical edges and object boundaries as regions requiring greater focus.
- Proposed approach: CSPN++ weights predefined kernel-size and iteration configurations to assemble or select propagation outputs at each pixel.The resulting variants are context-aware CSPN for assembling outputs and resource-aware CSPN for selecting configurations.
- Proposed approach: 2ˆ „5ˆ speed up CSPN significantly with improved accuracy is reported for resource-aware CSPN in the experiments.The method also adapts to computational budgets during training and inference.
- Proposed approach: A gate network guides depth preservation so outputs become more robust to inaccurate sparse depths.The gate is introduced specifically in response to errors observed in sparse depth inputs.
Related Work
The related work positions CSPN++ among image-guided depth completion, context-assembly, and adaptive-inference methods, extending these ideas to pixel-wise propagation choices.
- Depth completion: Depth completion research uses sparse LiDAR, SfM, or SLAM depths with images to recover dense depth and improve 3D perception.Prior approaches include methods without image guidance and CNN predictors that combine images with sparse depth.
- Depth completion: CSPN is selected as the baseline because of its clear motivation and stability guarantees for training and inference.CSPN++ is presented as improving both effectiveness and efficiency over this baseline.
- Context assembly: Context-assembly methods combine outputs horizontally across branches or vertically across network layers.Examples include Inception-style branches, pyramid pooling, atrous spatial pyramids, and multi-layer modules.
- Context assembly: CA-CSPN extends context assembly to CSPN by combining multiple kernel sizes and outputs from different iteration stages.The paper relates this design to affinity-guided context fusion in depth completion.
- Context assembly: Although one-step CA-CSPN is mathematically equivalent to a single largest-kernel CSPN in forward propagation, auxiliary weighting parameters yield different backward learning.The paper reports significantly better results for the assembled formulation.
- Adaptive inference: Adaptive inference stops computation after sufficient context is obtained, and RA-CSPN applies this idea to depth completion.Each pixel selects a kernel size and iteration count, while latency or memory budgets can be included in optimization.
Preliminaries
CSPN propagates a hidden depth representation through learned local affinities, then replaces propagated values at valid sparse-depth pixels to preserve observations.
- CSPN formulation: CSPN updates a hidden depth representation through repeated local propagation using a predefined convolutional kernel and learned normalized affinities.The process iterates N times, with kernel size k and iteration count N tuned because they affect performance.
- Sparse-depth replacement: A replacement operation preserves valid sparse-depth values after each propagation step.The update blends the propagated representation with the sparse-depth embedding using a validity indicator.
Context and Resource Aware CSPN
CSPN++ predicts per-pixel propagation configurations from image content, either assembling multiple contexts for accuracy or selecting configurations for efficiency and budget control, while gating sparse-depth replacement against noise.
- Context and Resource Aware CSPN: CSPN++ predicts per-pixel weights for kernel sizes and iteration counts from image content, enabling adaptive propagation configurations.The parameters αx weight kernel-size proposals, while λx weights outputs associated with different iteration counts.
- Context-aware CSPN: Context-aware CSPN assembles outputs across kernel sizes and propagation stages using normalized weighting parameters.The normalization is intended to preserve CSPN’s stabilization property during training and inference.
- Noise-robust depth preservation: A gated network predicts confidence for valid sparse-depth points, making depth preservation conditional on estimated input accuracy.The gate modifies the replacement operation rather than unconditionally retaining every sparse-depth value.
- Complexity and computational resource analysis: CSPN++ uses estimated memory and FLOPs as practical resource indicators, with costs depending on kernel size, iterations, feature dimensions, and device configuration.The paper notes that latency estimates vary across devices and that direct testing on the target device can be preferable.
- Efficient testing: Regional im2col groups pixels with the same predicted kernel size for simultaneous convolution, while stopped pixels copy features forward to later stages.This implementation addresses both heterogeneous per-pixel convolutions and continued computation when neighboring propagation persists.
- Budget control: The resource-aware model can incorporate target memory and latency costs into training and apply rounding to enforce computational budgets.The optimization adds budget constraints through Lagrange multipliers and uses a rounding strategy because learned configurations may violate hard limits.
Experiments
Experiments evaluate CSPN++ on KITTI and NYU v2, using ablations, efficiency comparisons, qualitative results, and KITTI benchmark comparisons. The studies show that its context-aware variant improves accuracy, while resource-aware selection reduces computation and adapts to budgets.
- Datasets: KITTI Depth Completion provides 86k training, 7k validation, and 1k testing depth maps with corresponding LiDAR scans and reference images.The validation set uses the official 1k validation images, while the remaining images are merged into training.
- Context-aware CSPN: Pixels on ground and walls generally select small kernels and few iterations, whereas distant pixels and boundaries use larger kernels and more iterations.The learned configurations assign context according to image content and scene structure.
- Resource-aware CSPN: Under constraints below 35% of vanilla CSPN memory and latency, expected iterations decrease from 0.439 to 0.303 while expected kernel size increases from 0.268 to 0.333.The method adjusts kernel sizes and iteration counts automatically while producing better results.
- Cross-dataset evaluation: On NYU v2, the experiments draw similar conclusions to KITTI, supporting the method’s effectiveness in indoor scenes.NYU v2 testing uses 654 images from the standard labeled test set.
- Benchmark comparison: CA-CSPN ranks 2nd on the KITTI benchmark and reports better results than competing methods on three of four metrics.The comparison uses a model trained only on KITTI, unlike some competitors that use additional datasets.
- Qualitative comparison: Qualitative KITTI comparisons report better detailed scene-structure recovery than UberATG-FuseNet.The comparison includes dense depth maps and corresponding error maps.
Conclusion
CSPN++ extends CSPN through context-aware and resource-aware variants built with a shared model-selection framework. The context-aware variant reduces estimation error, while the resource-aware variant improves efficiency with comparable accuracy.
- Conclusion: CSPN++ uses a shared model-selection framework to create context-aware and resource-aware CSPN variants.The framework is presented as an improvement over CSPN for depth completion.
- Conclusion: Context-aware CSPN significantly reduces estimation error relative to CSPN.The conclusion describes this as the principal accuracy benefit of the context-aware variant.
- Conclusion: Resource-aware CSPN achieves much better efficiency with accuracy comparable to the context-aware variant.The conclusion presents efficiency and comparable accuracy as the resource-aware variant’s main outcome.