Source-linked AI summary
Group-wise Correlation Stereo Network
Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, Hongsheng Li
TL;DR
Stereo cost-volume designs face trade-offs between full correlation and concatenation. GwcNet uses group-wise correlation and an improved 3D stacked hourglass, and outperforms prior methods on Scene Flow, KITTI 2012, and KITTI 2015.
Problem
Stereo cost volumes either discard feature information through full correlation or require aggregation to learn similarities from concatenation, which uses more parameters.
Source: §3.2 · p. 4Method
Results
GwcNet outperforms prior methods on Scene Flow, KITTI 2012, and KITTI 2015; on KITTI 2015 D1-all, it surpasses PSMNet by 0.21%.
Source: §1 · p. 2, §4.6 · p. 8Takeaways & Limitations
When 3D aggregation compute is limited, the proposed network has a much smaller performance reduction than PSMNet.
Source: §1 · p. 2, §4.5 · p. 8
Abstract
from arXiv · showhide
Stereo matching estimates the disparity between a rectified image pair, which is of great importance to depth sensing, autonomous driving, and other related tasks. Previous works built cost volumes with cross-correlation or concatenation of left and right features across all disparity levels, and then a 2D or 3D convolutional neural network is utilized to regress the disparity maps. In this paper, we propose to construct the cost volume by group-wise correlation. The left features and the right features are divided into groups along the channel dimension, and correlation maps are computed among each group to obtain multiple matching cost proposals, which are then packed into a cost volume. Group-wise correlation provides efficient representations for measuring feature similarities and will not lose too much information like full correlation. It also preserves better performance when reducing parameters compared with previous methods. The 3D stacked hourglass network proposed in previous works is improved to boost the performance and decrease the inference computational cost. Experiment results show that our method outperforms previous methods on Scene Flow, KITTI 2012, and KITTI 2015 datasets. The code is available at https://github.com/xy-guo/GwcNet
1. Introduction
Stereo matching estimates disparity from rectified image pairs, enabling depth sensing; this paper proposes group-wise correlation cost volumes and a modified 3D hourglass network to improve matching and reduce inference cost.
1. Introduction
- Stereo matching estimates disparity from rectified image pairs, and greater disparity precision improves depth precision because depth is Fl/d.Stereo matching is presented as passive depth sensing for applications including autonomous driving and robot navigation.Source: §1 · p. 1
- Prior learning-based methods use single-channel full correlation [14], parameter-intensive concatenation volumes [6] [2], or hand-crafted costs that are not end-to-end optimized.Full correlation loses information, while concatenation requires the aggregation network to learn similarity measurement from scratch.Source: §1 · p. 1
- Group-wise correlation splits left and right features into corresponding channel groups, cross-correlates them across disparities, and packs the resulting matching-cost proposals into a 4D cost volume.Multiple proposals avoid the information loss of full correlation while providing similarity measures for 3D aggregation.Source: §1 · pp. 1–2
- The 3D stacked hourglass network from PSMNet [2] is modified with 1×1×1 convolutions in shortcut connections to improve performance without substantially increasing inference cost.The modification is intended to decrease inference computational cost.Source: §1 · p. 2
- The method outperforms previous methods on Scene Flow, KITTI 2012, and KITTI 2015, and loses less performance than PSMNet when 3D aggregation cost is limited.The authors describe group-wise correlation as valuable for real-time stereo networks under constrained computational cost.Source: §1 · p. 2
2. Related Work
The related work moves from traditional stereo pipelines to learned approaches that improve matching costs, cost aggregation, and disparity prediction. Learned methods span hand-crafted-style cost aggregation with neural matching costs, correlation-volume regression, and concatenation-based volumes refined by 3D convolutions.
2.1. Traditional methods
- Traditional stereo matching computes costs, aggregates them, optimizes disparity, and may post-process results; local, global, and semi-global methods differ in aggregation and optimization.Local methods commonly use winner-take-all, global methods optimize costs with smoothness priors, and SGM approximates global optimization with dynamic programming.Source: §2.1 · p. 2
2.2. Learning based methods
- Learned methods span neural matching costs [29], unary-feature correlation, combined cost evidence, scanline candidate classification, and learned SGM penalties.Another approach combines CNN-predicted correlation costs with a CRF for long-range interactions [10].Source: §2.2 · p. 2
- Following DispNetC [14], learned stereo methods regress disparity from correlation cost volumes, with later work adding refinement networks or edge and semantic features [17] [12] [22] [26].The volume is computed at each disparity from left- and right-feature correlations.Source: §2.2 · p. 2
- GC-Net [6] first used 3D convolutions to aggregate cost-volume context, concatenating left and right features into a 4D volume; PSMNet [2] followed with spatial pyramid pooling and stacked 3D hourglass networks for cost-volume refinement.Source: §2.2 · p. 2
3. Group-wise Correlation Network
GwcNet extends PSMNet with a group-wise correlation cost volume and a modified 3D stacked-hourglass aggregator. Its design combines multiple feature-similarity proposals with a concatenation volume and streamlines inference by removing auxiliary output modules at inference time.
3. Group-wise Correlation Network
- GwcNet extends PSMNet [2] with group-wise correlation costs and an improved 3D stacked-hourglass network to address the parameter and computation demands of learning similarity from concatenated features.The paper contrasts this with full correlation [14], which measures feature similarity efficiently but loses information.Source: §3 · pp. 2–3
3.1. Network architecture
- The network comprises unary feature extraction, a cost volume combining concatenation and group-wise correlation, 3D aggregation, and disparity prediction.The concatenation volume uses compressed left and right features; the two volumes are concatenated as input to the 3D aggregator.Source: §3.1 · p. 3, Figure 1 · p. 3
3.3. Improved 3D aggregation module
- The 3D aggregation module uses a pre-hourglass stage and three stacked hourglasses, adds an auxiliary output for pre-hourglass features, and removes inter-output residual connections so auxiliary modules can be dropped at inference.It also adds 1×1×1 convolutions to shortcut connections within hourglass modules; these have 1/27 the multiplication operations of 3×3×3 convolutions.Source: §3.1 · p. 3, Figure 2 · p. 3, §3.3 · pp. 4–5
3.2. Group-wise correlation volume
- Group-wise correlation splits unary feature channels into Ng groups, computes correlations at each disparity, and packs them into an Ng-channel cost volume of multiple similarity proposals.The 3D aggregator combines these proposals; the correlation volume complements concatenation and requires fewer aggregation parameters to achieve favorable results.Source: §3.2 · p. 4
3.4. Output module and loss function
- Each of four output modules converts its cost volume into disparity probabilities and applies soft argmin, with the predicted maps trained using weighted smooth L1 loss.The four predictions are denoted ed0, ed1, ed2, and ed3.Source: §3.4 · p. 5
4. Experiment
Experiments evaluate the proposed stereo models on Scene Flow and KITTI, testing group-wise correlation, an improved stacked hourglass, and performance under constrained computation. The paper also reports final KITTI test-set comparisons against prior methods.
4.1. Datasets and evaluation metrics
- Evaluation used Scene Flow Finalpass and KITTI 2012/2015, with EPE on Scene Flow and dataset-specific error and outlier metrics on KITTI.Scene Flow contains 35,454 training and 4,370 testing images; KITTI 2012 and 2015 provide 194/195 and 200/200 training/testing pairs, respectively. KITTI 2015 D1 outliers exceed max(3px, 0.05d*).Source: §4.1 · p. 5
4.2. Implementation details
- Training used PyTorch and Adam; models ran 16 epochs on Scene Flow and were fine-tuned for 300 epochs on KITTI.Training used batch size 16 across eight TITAN Xp GPUs.Source: §4.2 · p. 5
4.3. The effectiveness of Group-wise correlation
- Gwc40 lowers EPE by 0.1px and 3-pixel error by 0.75% versus Cat-Base at nearly equal time; combining volumes improves further, with gains leveling beyond 40 groups.The selected setting uses 40 groups of 8 channels; group-wise correlation supplies accurate matching features while concatenation adds complementary semantic information.Source: §4.3 · p. 6
4.4. Improved stacked hourglass
- The proposed hourglass changes EPE by 7.8% on Scene Flow and 5.8% on KITTI 2015 versus Cat64-original-hg, while cutting inference time by 42.7ms.Auxiliary output modules can be removed during inference to save time.Source: §4.4 · pp. 6, 8
4.5. Limit the computational cost of 3D network
- As 3D-network channels decrease, Gwc-Cat outperforms Cat by increasingly larger margins, supporting group-wise correlation for stereo models with limited computational cost.Group-wise correlation supplies matching-cost representations, whereas concatenation-only aggregation must learn similarity from scratch and typically requires more parameters and computation.Source: §4.5 · p. 8
4.6. KITTI 2012 and KITTI 2015
- On KITTI 2015 test, Gwc40 surpasses PSMNet by 0.21% and SegStereo by 0.14% on D1-all.The submitted model was Gwc40, without concatenation volume.Source: §4.6 · p. 8, Table 4 · p. 8
- On KITTI 2012 test, the method surpasses PSMNet by 0.19% on 3-pixel-error and 0.1px on mean disparity error.The submitted model was the best-validation Gwc40-Cat24 model.Source: §4.6 · p. 8, Table 5 · p. 8
5. Conclusion
GwcNet builds stereo cost volumes with group-wise correlation, providing matching features for 3D aggregation while reducing its parameter requirements. The paper reports larger gains than concatenation-volume networks under limited computational cost and says its improved stacked hourglass further boosts performance and reduces inference time.
5. Conclusion
- GwcNet constructs stereo cost volumes using group-wise correlation, whose volumes provide good matching features for the 3D aggregation network.The aggregation network requires fewer parameters.Source: §5 · pp. 8–9
- Under limited computational cost, GwcNet achieves larger gains than previous concatenation-volume stereo networks, with experiments demonstrating effectiveness on Scene Flow and KITTI.The conclusion also attributes further performance improvement and reduced inference time to the improved stacked hourglass networks.Source: §5 · p. 9
- The improved stacked hourglass networks further improve performance and reduce inference time.Source: §5 · p. 9