Source-linked AI summary
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network
Xinjing Cheng, Peng Wang, Ruigang Yang
TL;DR
Single-image depth estimation needs accurate per-pixel distances, while sparse LiDAR measurements must be converted into dense maps. The paper proposes CSPN, which learns image-dependent affinities and performs recurrent local propagation simultaneously across pixels. CSPN is reported to improve depth-refinement quality and speed over SPN, while its sparse-depth extension preserves measured values and supports dense prediction.
Problem
Single-image depth estimation and sparse-to-dense conversion require improved depth prediction while existing propagation can be serial and sparse measurements may not be preserved.
Method
CSPN learns image-dependent local affinities with a CNN and recurrently propagates depth simultaneously across pixels, optionally embedding sparse depth samples.
Results
CSPN is reported as 2-5× faster and over 30% more accurate than SPN for depth refinement, with superior improvement over other SOTA methods for sparse-to-dense prediction.
Takeaways & Limitations
The framework supports efficient depth refinement and real-time sparse-depth fusion for robotics and autonomous driving applications.
Takeaways & Limitations
The sparse-to-dense setting assumes sparse depth measurements whose values can be embedded and preserved during propagation.
Abstract
from arXiv · showhide
Depth estimation from a single image is a fundamental problem in computer vision. In this paper, we propose a simple yet effective convolutional spatial propagation network (CSPN) to learn the affinity matrix for depth prediction. Specifically, we adopt an efficient linear propagation model, where the propagation is performed with a manner of recurrent convolutional operation, and the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). We apply the designed CSPN to two depth estimation tasks given a single image: (1) To refine the depth output from state-of-the-art (SOTA) existing methods; and (2) to convert sparse depth samples to a dense depth map by embedding the depth samples within the propagation procedure. The second task is inspired by the availability of LIDARs that provides sparse but accurate depth measurements. We experimented the proposed CSPN over two popular benchmarks for depth estimation, i.e. NYU v2 and KITTI, where we show that our proposed approach improves in not only quality (e.g., 30% more reduction in depth error), but also speed (e.g., 2 to 5 times faster) than prior SOTA methods.
1 Introduction
The paper introduces CSPN to refine monocular depth and densify sparse depth using simultaneous local propagation, targeting better efficiency and accuracy than serial SPN-based propagation.
- Single-image depth estimation predicts per-pixel distance and supports applications including augmented reality, autonomous driving, and robotics.
- SPN learns image-dependent affinity with a deep CNN but propagates serially along scan-lines or scan-columns.Pixels farther along a scan direction must wait for preceding pixels to update.
- CSPN updates all pixels simultaneously within a local convolutional context and obtains longer-range context through recurrence.The method is designed as a parallel alternative to serial propagation.
- CSPN also converts sparse, accurate LiDAR depth samples into dense maps by combining the samples with image information during propagation.The extension targets robotics and autonomous-car settings where LiDAR measurements are sparse but accurate.
- 2-5× acceleration is reported for the proposed strategy compared with SPN, alongside improved depth-refinement quality.
- CSPN preserves provided sparse depths in the final dense map while propagating their information to neighboring pixels.The contribution also reports real-time suitability for applications that fuse LiDAR and image data.
2 Related Work
Prior work uses CNNs, CRFs, propagation networks, and sparse-to-dense methods for depth estimation and enhancement, but existing approaches retain important efficiency or input-density limitations.
- CNN-based methods estimate single-view depth using supervised, semi-supervised, or unsupervised training, with architectural additions such as skip and mirror connections.
- Deep-learning enhancement methods map low-resolution inputs to high-resolution depth outputs end-to-end but are commonly trained with perfect corresponding depth pairs.The paper contrasts this setting with non-perfect inputs and ground truth from low-cost LiDAR or a network.
- Random-walk affinity learning for segmentation requires additional supervision from ground-truth sparse pixel pairs, limiting potential pixel connections.
- SPN learns local linear propagation for output enhancement, but its scan-based whole-image updates are less efficient for depth refinement that commonly needs local context.The paper reports that CSPN is more efficient and provides better results in experiments.
- Some sparse-to-dense methods omit image input and require sufficiently dense sparse points to reveal scene structure.That density is described as unavailable in the paper’s scenario.
3 Our Approach
CSPN learns image-dependent affinities and propagates depth through simultaneous local convolutional updates, with recurrent steps supplying broader context. The method also embeds sparse depth measurements to refine depth maps while preserving measured values and maintaining smooth, image-aligned boundaries.
- 3.1 Convolutional Spatial Propagation Network: CSPN formulates depth refinement as anisotropic diffusion whose image-dependent transformation kernel is learned by a deep CNN.The input depth map is embedded into a hidden representation before recurrent propagation.
- 3.1 Convolutional Spatial Propagation Network: The learned kernels are normalized so off-center weights have total absolute value at most 1, supporting stabilization and convergence over N iterations.The center weight is set to one minus the sum of neighboring weights.
- 3.1 Convolutional Spatial Propagation Network: Unlike SPN’s sequential image scans, CSPN updates all pixels simultaneously within a k×k local context and gains larger context through recurrent operation.Its recurrent context acquisition rate is O(kN).
- 3.2 Spatial Propagation with Sparse Depth Samples: Fig. 3 reports smaller gradient error for CSPN than sparse depth replacement, while Fig. 4 compares the corresponding depth-map smoothness around sparse points.The replacement baseline can preserve sampled values but produces non-smooth surrounding gradients.
- 3.2 Spatial Propagation with Sparse Depth Samples: Sparse depth samples are embedded after propagation updates, guaranteeing exact measured values while spreading their information to neighboring pixels.This preserves smooth gradients around sparse measurements and aligns the final map with image structures.
- 3.3 Complexity Analysis: CSPN’s N-step propagation has complexity O(log2(k2)N), independent of image size, and with k = 3 and N = 12 it outperforms SPN by a relative 30%.Parallel convolution and CUDA reduction make the propagation efficient; the network uses a final CSPN layer for dense depth output and shares features for kernel prediction.
4 Experiments
Experiments on NYU v2 and KITTI evaluate CSPN for depth refinement and sparse-to-dense prediction, including parameter effects, speed, comparisons, and qualitative structure recovery.
- Datasets and Metrics: CSPN was evaluated on NYU v2 and KITTI using standard depth-estimation metrics, with NYU v2 also using 500 randomly selected sparse depth samples per image.NYU v2 testing uses 654 images; KITTI evaluation uses 3,200 images and evaluates valid pixels in the bottom 912×228 region.
- Parameter Tuning and Speed Study: CSPN outperformed SPN after four iterations, with further gains from additional iterations; kernel size and iteration count present an accuracy–memory trade-off.Kernel size 7 with 12 iterations performs similarly to kernel size 3 with 20 iterations, while the latter uses less memory; the selected setting is kernel size 3 with 24 iterations.
- Parameter Tuning and Speed Study: 0.531, 0.158, and 0.137 RMSE were obtained when mirror connections were concatenated after convolution, batch normalization, and ReLU, respectively.The proposed concatenation endpoint was adopted based on these results.
- Parameter Tuning and Speed Study: 3.689 ms versus 127.902 ms was required for four CSPN iterations versus SPN on one 1024 × 768 image.At high iteration counts and large images, implementation memory paging reduces CSPN’s scaling advantage, but it remains twice as fast as SPN without coding optimization.
- Comparisons: UNet + CSPN achieved the best NYU v2 sparse-to-dense results in all cases and improved substantially over ASAP, SPN, and UNet + SPN.All sparse-to-dense results preserve the provided sparse depth values.
- Comparisons: Qualitative comparisons show CSPN recovering image structures better than competing strategies, including details and thin structures in KITTI scenes.Reported examples include poles, a grass trunk, and a car silhouette under shadow that SPN fails to refine.
5 Conclusion
The paper concludes that CSPN provides efficient, accurate propagation for depth refinement and can incorporate sparse measurements for dense depth prediction. It reports improvements over prior methods and identifies broader applications as future work.
- Conclusion: CSPN is a convergent linear diffusion process that can be jointly learned with any type of CNN.Compared with SPN, it is reported as 2-5× faster and to provide over 30% improvement for depth refinement.
- Conclusion: Embedding sparse depth samples into CSPN provides superior improvement over other state-of-the-art sparse-to-dense methods while preserving the sparse input values.The framework is described as suitable for real-time robotics and autonomous-driving applications.