PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Depth with Convolutional Spatial Propagation Network

Xinjing Cheng, Peng Wang, Ruigang Yang

arXiv:1810.02695v3cs.CV

TL;DR

Depth estimation requires accurate dense predictions for multiple vision tasks, while existing propagation can be serial or produce blurry, structure-misaligned results. The paper introduces CSPN, an efficient learned linear propagation module, extends it to sparse depth completion and 3D stereo cost volumes, and reports improved accuracy and speed across benchmarks.

  • Problem

    Existing depth-completion predictions can be blurry and misaligned with image structures, while SPN propagation is serial and order dependent.

  • Method

    CSPN learns image-dependent affinities for recurrent convolutional propagation, with sparse-depth embedding for completion, 3D propagation for stereo, and learned pyramid aggregation.

  • Results

    CSPN is reported as 2-5× faster and over 30% more accurate than SPN for depth completion, while CSPN-based approaches improve both depth-completion and stereo results over other SOTA methods.

  • Takeaways & Limitations

    The modules can be jointly learned with different depth-estimation networks and provide task-specific propagation across image, disparity, scale, and pyramid dimensions.

  • Takeaways & Limitations

    The stability guarantee relies on the temporal Jacobian norm being no greater than one, and depth-completion experiments use 500 sparse samples.

Abstract

from arXiv · show

Depth prediction is one of the fundamental problems in computer vision. In this paper, we propose a simple yet effective convolutional spatial propagation network (CSPN) to learn the affinity matrix for various depth estimation tasks. Specifically, it is an efficient linear propagation model, in which the propagation is performed with a manner of recurrent convolutional operation, and the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). We can append this module to any output from a state-of-the-art (SOTA) depth estimation networks to improve their performances. In practice, we further extend CSPN in two aspects: 1) take sparse depth map as additional input, which is useful for the task of depth completion; 2) similar to commonly used 3D convolution operation in CNNs, we propose 3D CSPN to handle features with one additional dimension, which is effective in the task of stereo matching using 3D cost volume. For the tasks of sparse to dense, a.k.a depth completion. We experimented the proposed CPSN conjunct algorithms over the popular NYU v2 and KITTI datasets, where we show that our proposed algorithms not only produce high quality (e.g., 30% more reduction in depth error), but also run faster (e.g., 2 to 5x faster) than previous SOTA spatial propagation network. We also evaluated our stereo matching algorithm on the Scene Flow and KITTI Stereo datasets, and rank 1st on both the KITTI Stereo 2012 and 2015 benchmarks, which demonstrates the effectiveness of the proposed module. The code of CSPN proposed in this work will be released at https://github.com/XinJCheng/CSPN.

1 INTRODUCTION

The paper targets depth completion and stereo depth estimation with CSPN, a parallel recurrent propagation approach designed to improve depth quality and efficiency. It extends propagation to sparse depth inputs, disparity and scale dimensions, and pyramid feature aggregation.

  • Depth estimation supports applications including augmented reality, autonomous driving, robotics, 3D reconstruction, and recognition.
  • Depth completion: Depth completion converts sparse, accurate sensor measurements and a reference image into a full-frame dense depth map.The paper highlights LiDAR as a source of sparse but accurate depth measurements.
  • Depth completion: Contemporary depth-completion outputs can remain blurry and misaligned with image structures such as object silhouettes.
  • CSPN: CSPN updates all pixels simultaneously within a local convolutional context while recurrent processing supplies longer-range context.This removes the serial scan-line or scan-column propagation used by SPN.
  • Stereo and pyramid extensions: 3D CSPN propagates information through disparity and scale spaces in stereo cost volumes, while CSPN-based pyramid aggregation learns affinities within and across pyramids.The paper presents SPP and ASPP as special cases of CSPN and proposes learned affinity aggregation for further gains.
  • Evaluation: Depth completion experiments report about 30% relative improvements on key measurements and up to 5× acceleration over SPN on large images.The experiments use NYU v2 and KITTI; stereo experiments use Scene Flow and KITTI Stereo.

2 RELATED WORK

Prior work combines deep networks, filtering, propagation, cost volumes, and pyramid features for depth estimation, but leaves efficiency, refinement, or flexibility gaps that motivate CSPN.

  • Single-view depth and enhancement: Earlier depth methods use CNNs and CRFs, while neighboring-pixel affinity is often manually designed from color similarity or intervening contours.CSPN instead learns data-dependent affinity through the network.
  • Depth enhancement: Deep depth-enhancement methods learn low-resolution-to-high-resolution mappings end-to-end but commonly rely on perfect corresponding ground-truth depth pairs and black-box models.
  • Spatial propagation: SPN converts large-affinity diffusion into local linear propagation, but its serial scanning is less suited to local-context refinement for sparse depth completion.
  • Depth completion settings: Some LiDAR-only approaches require sufficient valid-depth density to reveal scene structure, whereas RGB-guided in-painting methods use a different setup.
  • Stereo matching: Stereo methods evolved from CNN matching-cost computation toward stronger fully learnable architectures using cost volumes, epipolar constraints, and pyramid strategies.Recent systems such as GCNet and PSMNet use 3D convolutions over disparity cost volumes.
  • Propagation comparison: Figure 2 contrasts sequential four-direction SPN propagation with simultaneous local 2D CSPN propagation and 3D propagation across an added feature dimension.

3 OUR APPROACH

The paper introduces CSPN as a parallel, recurrent convolutional propagation model that learns image-dependent affinities for depth refinement. It extends CSPN to sparse-depth completion and 3D disparity/scale propagation, while using learned propagation for pyramid feature integration.

  • CSPN: The affinity network predicts spatially dependent transformation kernels from the input image, with normalized weights supporting stable propagation.Kernel weights are normalized to (−1,1), and stability follows when the stated absolute-weight condition is satisfied.
  • CSPN: CSPN updates all pixels simultaneously within a k×k local context, while recurrent iterations provide longer-range information.Its context acquisition rate is O(kN), and convolution enables efficient image-vectorized implementation.
  • CSPN: CSPN has theoretical complexity O(log2(k2)N), compared with SPN complexity O(log2(k)(m + n)).The parallel formulation avoids SPN’s full-image row/column scanning and is designed for GPU execution.
  • CSPN: With k = 3 and N = 12, CSPN runs 6x faster than SPN and is 30% more accurate in the reported experiment.The timing was measured on a Titan X GPU.
  • CSPN variants: SPP is treated as a special case of one-step CSPN, enabling learned kernels for pyramid pooling and 3D CSPN for cross-pyramid feature fusion.The resulting CSPF combines convolutional spatial pyramid processing with learned feature fusion; ACSPF is reported as the best setting in the experiments.
  • Depth completion: For depth completion, sparse depths are embedded into propagation, while reported gradients near sparse points are smoother than with direct replacement.Using 500 sparse samples, the paper reports smaller average gradient error and mostly error-free pixels for its method.
  • 3DCSPN: 3DCSPN adds a propagation dimension for disparity-space processing and connects multi-stage disparity predictions across scale space.For scale fusion, the method concatenates stage outputs and applies 3D CSPN before producing a single disparity map.

4 EXPERIMENTS

The experiments evaluate CSPN and its variants, including CSPF and 3D CSPN, across depth completion and stereo matching benchmarks.

  • The evaluation covers depth completion and stereo matching using CSPN, CSPF, and 3D CSPN.The experiments use several popular depth-estimation benchmarks and compare the proposed variants across both tasks.

4.1 Depth completion

The depth-completion experiments evaluate CSPN across datasets, propagation settings, hyperparameters, efficiency, and architectural variants. CSPN improves depth quality over SPN and other baselines while retaining strong efficiency, with performance affected by iteration count, kernel size, and mirror-connection placement.

  • Ablation study: CSPN outperforms SPN after only four iterations, with accuracy saturating at 24 iterations for final evaluation.The experiments use a kernel size of 3 for this iteration study.
  • Ablation study: Larger kernels improve propagation performance up to size 7, while performance drops slightly at size 9 under fixed 40-epoch training.With N = 12, larger kernels provide context similar to increasing the iteration count.
  • Ablation study: The after-relu mirror-connection endpoint achieves RMSE 0.137, compared with 0.531 after conv and 0.158 after bn.The after-relu endpoint is therefore adopted in the model.
  • Efficiency: Four CSPN iterations take 3.689 ms on a 1024 × 768 image, versus 127.902 ms for SPN, while high-iteration CSPN remains twice as fast without code optimization.CSPN is comparable to SPN on small images, but practical cost can grow with image size at high iteration counts.
  • Comparisons: Replacing SPN with CSPN further improves results across refinement settings and performs best in the reported NYU v2 cases.The comparison includes settings that either exclude or preserve sparse-depth samples.
  • Comparisons: On KITTI, the final MC + CSPN + ACSPF model improves δ < 1.02 from 30% for the baseline to 72%.The model also recovers thin structures and difficult shaded objects more effectively than baseline and SPN refinements.

4.2 Stereo depth estimation

The stereo experiments extend spatial propagation into disparity and scale dimensions and evaluate these components on Scene Flow and KITTI benchmarks. Three-dimensional propagation and enhanced spatial-pyramid fusion improve error metrics, while the complete method achieves the strongest reported KITTI benchmark results.

  • Evaluation: The stereo evaluation uses EPE on Scene Flow and erroneous-pixel percentages with benchmark-specific thresholds on KITTI 2012 and 2015.KITTI 2015 marks errors using a disparity threshold of 3 pixels or relative 5%, whichever is less stringent.
  • 3D module: 2D CSPN reduces Scene Flow EPE from 1.119 to 0.992, while 3D CSPN in disparity space further reduces it to 0.971.Joint propagation across disparity and scale spaces performs best among the 3D module variants.
  • 3D module: Combining 3D CSPN with ACSPF additionally reduces Scene Flow error by around 30% relative to the baseline.This result combines the proposed 3D propagation and enhanced spatial-pyramid modules.
  • CSPF module: ACSPP lowers EPE to 0.902, while CFF lowers it to 0.905 from the SPP-based 0.954 result.ACSPF, combining ACSPP and CFF, is selected as the final spatial-pyramid module and gives the best performance.
  • Comparisons: The complete method outperforms all compared methods by more than 10% and leads the major metrics on both KITTI 2012 and 2015.Qualitative comparisons also report improved overall scene structure and detailed structures over PSMNet.

5 CONCLUSION

The paper presents CSPN as a learnable propagation module for depth completion and stereo depth estimation, extending it to 3D and combining it with spatial-pyramid fusion. The reported experiments show faster and more accurate depth completion than SPN and strong improvements over other state-of-the-art methods across both tasks.

  • Contribution: CSPN is a learnable module for depth completion and stereo depth estimation that can be jointly trained with depth-estimation networks.The method is described as a linear diffusion process with model-stability guarantees.
  • Results: Compared with previous SPN, CSPN is 2–5× faster in practice and achieves over 30% improvement for depth completion.The efficiency advantage is especially reported for large images.
  • Extensions: The framework extends CSPN to 3D and combines it with SPP as CSPF, with task-specific propagation for sparse depth and disparity-scale features.The stereo variant diffuses along disparity and scale dimensions.
  • Conclusion: The proposed CSPN-based approaches provide superior improvement over other state-of-the-art methods on the two evaluated depth-estimation tasks.The paper identifies future applications in image segmentation and image enhancement for its general components.
Loading 1810.02695v3…