Source-linked AI summary
Dynamic Spatial Propagation Network for Depth Completion
Yuankai Lin, Tao Cheng, Qi Zhong, Wending Zhou, Hua Yang
TL;DR
Depth completion must infer dense maps from sparse measurements and RGB images, but fixed-affinity SPNs have limited representation and can over-smooth during iterations. DySPN uses nonlinear attention-based propagation, diffusion suppression, and lower-complexity variants; it reports stronger performance with fewer iterations and state-of-the-art results on KITTI and NYU Depth v2.
Problem
Fixed-affinity SPNs struggle to represent both long-range and local relationships and can over-smooth dense depth during propagation.
Method
DySPN uses nonlinear propagation with spatial-sequential attention, diffusion suppression, and three variants that reduce computational complexity.
Results
DySPN requires fewer iterations to match other SPNs and achieves stronger results overall, including state-of-the-art performance on KITTI and NYU Depth v2.
Takeaways & Limitations
DySPN provides speed-accuracy trade-offs through variants that reduce neighbors and attention requirements while retaining similar accuracy.
Takeaways & Limitations
The basic nonlinear propagation model has complexity O(m^2n^2N), while estimating N times more feature maps adds substantial computation.
Abstract
from arXiv · showhide
Image-guided depth completion aims to generate dense depth maps with sparse depth measurements and corresponding RGB images. Currently, spatial propagation networks (SPNs) are the most popular affinity-based methods in depth completion, but they still suffer from the representation limitation of the fixed affinity and the over smoothing during iterations. Our solution is to estimate independent affinity matrices in each SPN iteration, but it is over-parameterized and heavy calculation. This paper introduces an efficient model that learns the affinity among neighboring pixels with an attention-based, dynamic approach. Specifically, the Dynamic Spatial Propagation Network (DySPN) we proposed makes use of a non-linear propagation model (NLPM). It decouples the neighborhood into parts regarding to different distances and recursively generates independent attention maps to refine these parts into adaptive affinity matrices. Furthermore, we adopt a diffusion suppression (DS) operation so that the model converges at an early stage to prevent over-smoothing of dense depth. Finally, in order to decrease the computational cost required, we also introduce three variations that reduce the amount of neighbors and attentions needed while still retaining similar accuracy. In practice, our method requires less iteration to match the performance of other SPNs and yields better results overall. DySPN outperforms other state-of-the-art (SoTA) methods on KITTI Depth Completion (DC) evaluation by the time of submission and is able to yield SoTA performance in NYU Depth v2 dataset as well.
Introduction
Image-guided depth completion uses RGB guidance to densify sparse sensor measurements, while SPNs remain limited by fixed affinities and over-smoothing. DySPN addresses these issues with dynamically adjusted affinities, diffusion suppression, and lower-complexity variants.
- Motivation: Dense depth maps support 3D vision tasks, but LiDAR and ToF sensors leave many empty regions that require completion.These applications include autonomous driving and robot perception.
- Motivation: RGB-guided methods improve depth completion by using color and texture cues alongside sparse depth measurements.They commonly use encoder-decoder architectures and learnable edge-preserving strategies.
- Problem: CSPN's fixed affinity matrix distributes most 7×7 kernel weights toward distant neighbors, limiting representation of pixel-neighbor relationships.The comparison motivates adaptive affinities for propagation.
- DySPN: DySPN dynamically adjusts affinity weights through distance-specific attention, emphasizing far neighbors early and near neighbors later during recursive refinement.This design targets long-range hole filling first and edge preservation as the depth map becomes denser.
- DySPN: Diffusion suppression adaptively terminates propagation to preserve good depth values and reduce over-smoothing.The operation estimates refining-depth affinities at each step with nearly no additional computational cost.
Related Work
Prior depth-completion work combines RGB and sparse-depth features or refines depth with spatial propagation networks. DySPN extends SPNs with attention-generated adaptive affinities to address fixed-propagation limitations.
- Image-guided depth completion: Image-guided depth-completion methods use RGB surface, edge, and semantic information to improve reconstruction from sparse measurements.Later approaches fuse RGB and depth features through separate branches or multimodal guidance modules.
- Spatial propagation networks: SPNs refine depth with learned affinity matrices, while CSPN improves efficiency through recursive convolution and CSPN++ combines independent modules.NLSPN and DSPN further modify neighbor selection or receptive fields.
- Spatial propagation networks: Fixed affinity weights make it difficult for SPNs to model both long-range dependencies and local information, requiring more propagation iterations.This is identified as a bottleneck of the linear propagation model.
- Dynamic propagation: DySPN's framework generates an initial depth map, affinity matrix, and attention maps before applying them in a six-step propagation process.The attention maps are split into six parts for the DySPN module.
- Dynamic propagation: DySPN uses spatial and sequential attention to generate a series of adaptive affinity matrices for nonlinear propagation.The approach is inspired by dynamic filters that modify filter weights according to input features.
Our Approach
DySPN replaces fixed SPN propagation with a non-linear model that dynamically generates adaptive affinities, suppresses diffusion when refinement converges, and reduces computation through neighborhood-decoupling variants.
- Naive Non-linear Propagation Model: DySPN uses a non-linear propagation model with spatial and sequential attention to generate adaptive affinity matrices during propagation.The model represents propagation with iteration-dependent transformation and attention matrices rather than a fixed affinity.
- Diffusion Suppression: Diffusion suppression estimates refined-depth affinity at each iteration so propagation can converge early and avoid over-smoothing.At convergence, the propagation becomes an identity update, leaving the refined depth unchanged.
- Computational Considerations: The basic N-step NLPM has O(m^2n^2N) global-attention complexity, motivating variants that reduce sampled neighbors and attention calculations.The base network also estimates N times more feature maps than linear propagation, adding further computation.
- Efficient Neighborhood Decoupling: Neighborhood decoupling assigns neighbors to distance-based parts, reducing decoupled attention complexity to O(kN), independent of image size.The neighborhood representation samples neighbors by pixel distance and normalizes their affinity weights.
- Efficient Neighborhood Decoupling: The three variants trade neighborhood structure against receptive field: standard DySPN uses distance parts, Dilated DySPN uses two 3×3 neighborhoods, and Deformable DySPN uses two deformable 3×3 neighborhoods.The deformable variant estimates neighbor offsets, while the dilated variant uses dilation to enlarge the receptive field with fewer neighbors.
- Implementation: The implementation uses an encoder-decoder base network whose head predicts attention maps, an initial depth map, an affinity matrix, and sparse-depth confidence.Training combines L1 and L2 losses for the refined depth prediction.
Experiment
DySPN is evaluated on NYU Depth v2 and KITTI DC against SPN-based methods, with ablations testing its propagation, diffusion suppression, training strategy, and computational variants. It achieves strong indoor and outdoor results, including first-place KITTI DC performance and improved speed-accuracy trade-offs.
- Evaluation on NYU Depth v2 Dataset: On NYU Depth v2, the proposed algorithms achieve the best results and outperform other state-of-the-art methods.Attention visualizations show changing neighborhood behavior across iterations, with larger receptive fields early and smaller kernels near boundaries later.
- Evaluation on KITTI DC Dataset: DySPN ranks first on KITTI DC across RMSE, MAE, iRMSE, and iMAE, improving RMSE by nearly 21mm over competing methods.The reported KITTI result uses Deformable DySPN, identified as the best of the three variations.
- Ablation Studies: At six iterations, DySPN reduces RMSE from 811.0mm for CSPN to 769.0mm, and increasing iterations to twelve does not change its RMSE.This ablation indicates better convergence than the CSPN baseline under the reported comparison.
- Ablation Studies: Stochastic depth alone performs better than combining stochastic depth with weight decay in the reported training-strategy experiments.The comparison evaluates stochastic depth as an alternative strategy for reducing over-fitting.
- Ablation Studies: Dilated and Deformable DySPN use fewer iterations and neighbor samples; Dilated DySPN is over 2× faster than PENet’s C2 module while improving RMSE from 757.2mm to 748.5mm.Deformable DySPN is reported as the precision-oriented variation and ranks first on the KITTI DC benchmark.
Conclusion
DySPN combines spatial and sequential attention with diffusion suppression to improve depth completion while offering variants with different speed–accuracy trade-offs. Ablations support its performance on indoor and outdoor datasets, with potential applications beyond image-guided depth completion.
- DySPN uses spatial and sequential attention to generate adaptive affinity matrices through a non-linear propagation model.
- Diffusion suppression addresses over-smoothing during propagation.
- The 7×7, Dilated, and Deformable DySPN variants provide different speed–accuracy trade-offs.
- Ablation studies show superior performance for DySPN variants on indoor and outdoor depth completion datasets.
- DySPN may also apply to image segmentation and cellular automata modeling.