PaperlayerView exact revision on arXiv

Source-linked AI summary

Group-wise Correlation Stereo Network

Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, Hongsheng Li

arXiv:1903.04025v1cs.CV

TL;DR

Stereo matching needs accurate disparity estimates for depth sensing, while prior cost volumes either compress feature similarity into a single correlation map or require aggregation networks to learn similarity from concatenated features. GwcNet uses group-wise correlation to form multi-proposal cost volumes and improves the stacked 3D hourglass network. The method achieves better performance on Scene Flow, KITTI 2012, and KITTI 2015, including gains under limited computational cost.

  • Problem

    Prior full correlation loses information through single-channel disparity maps, while concatenation volumes lack feature-similarity information and require more aggregation parameters.

  • Method

    GwcNet splits left and right features into corresponding channel groups, correlates each group across disparities, packs the maps into a cost volume, and modifies the stacked 3D hourglass network.

  • Results

    GwcNet achieves better performance than previous methods on Scene Flow, KITTI 2012, and KITTI 2015, with smaller performance reductions when 3D aggregation cost is limited.

  • Takeaways & Limitations

    Group-wise correlation provides matching features that improve performance and reduce aggregation-network parameter requirements, especially under limited computational cost.

Abstract

from arXiv · show

Stereo matching estimates the disparity between a rectified image pair, which is of great importance to depth sensing, autonomous driving, and other related tasks. Previous works built cost volumes with cross-correlation or concatenation of left and right features across all disparity levels, and then a 2D or 3D convolutional neural network is utilized to regress the disparity maps. In this paper, we propose to construct the cost volume by group-wise correlation. The left features and the right features are divided into groups along the channel dimension, and correlation maps are computed among each group to obtain multiple matching cost proposals, which are then packed into a cost volume. Group-wise correlation provides efficient representations for measuring feature similarities and will not lose too much information like full correlation. It also preserves better performance when reducing parameters compared with previous methods. The 3D stacked hourglass network proposed in previous works is improved to boost the performance and decrease the inference computational cost. Experiment results show that our method outperforms previous methods on Scene Flow, KITTI 2012, and KITTI 2015 datasets. The code is available at https://github.com/xy-guo/GwcNet

1 The Chinese University of Hong Kong 2 SenseTime Research

The listed affiliations include The Chinese University of Hong Kong and SenseTime Research.

  • The contact addresses use the ee.cuhk.edu.hk and sensetime.com domains.

1. Introduction

The introduction motivates accurate stereo disparity for depth sensing and presents group-wise correlation as a cost-volume construction, alongside an improved 3D hourglass network.

  • Accurate disparity prediction improves depth precision in stereo-based depth sensing.The paper connects disparity d and depth through Fl/d, where F is focal length and l is the camera-center distance.
  • Matching cost computation measures similarity between left-image patches and possible corresponding right-image patches.Cost aggregation and optimization then incorporate contextual costs and regularization for more robust disparity predictions.
  • Group-wise correlation splits left and right features by channel groups and correlates corresponding groups across disparity levels.The resulting correlation maps are packed into a 4D cost volume for the 3D aggregation network.
  • The method modifies the stacked 3D hourglass aggregation network with 1×1×1 convolutions in shortcut connections.The modification targets improved performance without much additional computational cost and reduced inference computation.
  • Experiments report better performance than previous methods on Scene Flow, KITTI 2012, and KITTI 2015.The introduction also reports smaller performance reductions than PSMNet when the 3D aggregation computational cost is limited.

2. Related Work

Related work spans traditional matching costs, learned correlation volumes, concatenation-based volumes, and 3D cost aggregation for stereo disparity estimation.

  • Traditional stereo matching computes costs across pixels and possible disparities before aggregation, optimization, and post-processing.Common costs include SAD, SSD, and NCC; local methods aggregate neighboring costs and often use winner-take-all selection.
  • Learned methods use neural networks to predict matching costs and combine them with cost aggregation algorithms.Prior work includes neural matching costs, accelerated unary-feature correlation, multiple-cost evidence, learned SGM penalties, and CNN-correlation with CRF integration.
  • Correlation-volume approaches directly regress disparity maps from left-right feature correlations across disparity levels.CRL and iRes-Net extend this direction with stacked refinement subnetworks, while other methods add edge or semantic features.
  • Concatenation-volume methods concatenate left and right features at each disparity level and use 3D convolution for context aggregation.GC-Net introduced 3D convolution for cost-volume context aggregation, and PSMNet added spatial pyramid pooling and stacked 3D hourglass refinement.
  • Other stereo systems address multiple aggregation proposals, open-world video, consistency refinement, real-time inference, or application-friendly deployment.

3. Group-wise Correlation Network

GwcNet combines concatenation and group-wise correlation cost volumes with a modified stacked 3D hourglass aggregation network. Group-wise correlation supplies multiple feature-similarity proposals while the architecture changes target performance and inference efficiency.

  • Network architecture: GwcNet constructs cost volumes from concatenation and group-wise correlation components before 3D aggregation and disparity prediction.The network includes unary feature extraction, cost-volume construction, 3D aggregation, and output modules.
  • 3D aggregation network: The aggregation network uses a pre-hourglass module and three stacked 3D hourglass networks to regularize feature volumes and refine ambiguities.The hourglass structures use encoder-decoder processing, while 1×1×1 convolutions are added within shortcut connections.
  • Group-wise correlation volume: Group-wise correlation splits left and right unary features into channel groups and correlates corresponding groups across disparity levels.The resulting maps are packed into a cost volume with shape [Dmax/4, H/4, W/4, Ng].
  • Group-wise correlation volume: The group-wise correlation volume represents Ng matching-cost proposals, retaining richer similarity information than full correlation while reducing aggregation parameter demands.Each proposal is computed from one corresponding feature group, and the 3D aggregation network combines them to regress disparity maps.
  • 3D aggregation network: Removing connections between hourglass modules allows auxiliary output modules 0, 1, and 2 to be removed during inference, saving computational cost.An additional auxiliary output module provides a training loss for lower-layer features.
  • Output module and loss function: Each output module predicts a disparity map from a probability volume, and training combines smooth L1 losses over four disparity predictions.Disparity is estimated with soft argmin from probabilities over disparity levels; λi weights each prediction against the ground truth.

4. Experiment

Experiments evaluate group-wise correlation and the modified stacked hourglass on Scene Flow and KITTI, including ablations, computational limits, and test-set comparisons. Group-wise correlation improves accuracy and retains advantages as aggregation channels decrease, while the hourglass changes improve accuracy and reduce inference time.

  • Experimental setup: Experiments cover Scene Flow, KITTI 2012, and KITTI 2015 using ablations of volume types, group settings, hourglass modules, and channel counts.Scene Flow uses EPE; KITTI reports erroneous-pixel, mean-error, and D1 metrics.
  • Group-wise correlation: 40 groups with 8 channels per group are selected because gains become minor beyond 40 groups while memory and computational cost remain considerations.The Gwc-Base endpoint error stays around 1.2px when the group number exceeds 40.
  • Group-wise correlation: 0.1px lower endpoint error and 0.75% lower 3-pixel error are achieved by Gwc40 than Cat-Base with nearly identical time consumption.Combining group-wise correlation and concatenation further improves performance in Gwc40-Cat24-Base.
  • Improved stacked hourglass: 7.8% higher Scene Flow EPE and 5.8% higher KITTI 2015 performance are reported for Cat64 with the proposed hourglass versus the original-hourglass variant.Inference time for 640×480 inputs decreases by 42.7ms because auxiliary output modules can be removed during inference.
  • Computational cost: As channels decrease, Gwc-Cat outperforms Cat by increasingly larger margins because group-wise correlation supplies matching-cost representations to the 3D aggregation network.The channel reduction applies equally to the cost volume and all 3D convolutions, supporting use in computationally limited real-time stereo networks.
  • KITTI test results: 0.21% and 0.14% lower KITTI 2015 D1-all errors than PSMNet and SegStereo are reported, respectively.The submitted Gwc40 model is selected using validation performance before evaluation on the test server.
  • KITTI test results: 0.19% lower 3-pixel error and 0.1px lower mean disparity error than PSMNet are reported on the KITTI 2012 test set.The submitted model is the best Gwc40-Cat24 model on the validation set.

5. Conclusion

GwcNet uses group-wise correlation to construct cost volumes for stereo disparity estimation. The method improves aggregation efficiency and performance, with experiments demonstrating effectiveness on Scene Flow and KITTI datasets.

  • GwcNet estimates stereo disparity maps using group-wise correlation to construct cost volumes.The method incorporates group-wise correlation into the cost-volume construction process.
  • Group-wise correlation volumes provide matching features that improve 3D aggregation performance while reducing its parameter requirements.The conclusion attributes both performance improvement and lower aggregation-network parameter requirements to these volumes.
  • Under limited computational cost, GwcNet achieves larger gains than concatenation-volume-based stereo networks.
  • An improved stacked hourglass network further improves performance and reduces inference time.
  • Experiments demonstrate the effectiveness of the proposed method on Scene Flow and KITTI datasets.
Loading 1903.04025v1…