PaperlayerView exact revision on arXiv

Source-linked AI summary

Correlate-and-Excite: Real-Time Stereo Matching via Guided Cost Volume Excitation

Antyanta Bangunharcana, Jae Won Cho, Seokju Lee, In So Kweon, Kyung-Soo Kim, Soohyun Kim

arXiv:2108.05773v1cs.CVcs.AIcs.LG

TL;DR

Volumetric stereo matching uses 3D convolutions to aggregate cost volumes, while spatially varying aggregation can add complexity, computation, and memory use. CoEx combines image-guided cost-volume excitation with top-k soft-argmin regression; it is reported to outperform speed-oriented methods while remaining competitive with state-of-the-art models.

  • Problem

    Volumetric cost aggregation relies on computationally expensive 3D convolutions, and spatially varying aggregation methods add computation, memory use, and implementation complexity.

    Source: §I · p. 1

  • Method

    CoEx combines image-guided channel excitation of cost-volume features with top-k selection before soft-argmin disparity regression in an end-to-end stereo network.

    Source: §I · pp. 1–2

  • Results

    CoEx outperforms other speed-oriented methods and is competitive with state-of-the-art models; full GCE and top-2 regression add 1ms computation overhead and give 0.17 lower test EPE.

    Source: §I · p. 2, D. Ablation study · p. 6

  • Takeaways & Limitations

    The authors conclude that lightweight spatially varying aggregation and top-k disparity regression can improve stereo matching performance in a real-time model.

    Source: §V · p. 6

Abstract

from arXiv · show

Volumetric deep learning approach towards stereo matching aggregates a cost volume computed from input left and right images using 3D convolutions. Recent works showed that utilization of extracted image features and a spatially varying cost volume aggregation complements 3D convolutions. However, existing methods with spatially varying operations are complex, cost considerable computation time, and cause memory consumption to increase. In this work, we construct Guided Cost volume Excitation (GCE) and show that simple channel excitation of cost volume guided by image can improve performance considerably. Moreover, we propose a novel method of using top-k selection prior to soft-argmin disparity regression for computing the final disparity estimate. Combining our novel contributions, we present an end-to-end network that we call Correlate-and-Excite (CoEx). Extensive experiments of our model on the SceneFlow, KITTI 2012, and KITTI 2015 datasets demonstrate the effectiveness and efficiency of our model and show that our model outperforms other speed-based algorithms while also being competitive to other state-of-the-art algorithms. Codes will be made available at https://github.com/antabangun/coex.

I. INTRODUCTION

CoEx targets the accuracy–efficiency challenge in stereo matching with image-guided cost aggregation and top-k disparity regression. The paper reports competitive real-time performance and presents the approach as outperforming speed-oriented methods.

I. INTRODUCTION

  • Stereo matching estimates depth from paired images and is relevant to robotics, autonomous driving, and computer vision, where fast computation can matter.The introduction contrasts accurate but often slow models with fast methods that may sacrifice accuracy.Source: §I · p. 1
  • CoEx is reported as competitive with state-of-the-art models while real-time on the KITTI 2015 leaderboard, and as outperforming other speed-oriented methods.The leaderboard figure plots D1-all% error against frame rate.Source: Figure 1 · p. 1, §I · p. 2
  • Spatially varying cost aggregation can improve accuracy, but existing approaches are described as adding computational cost, memory use, and implementation complexity.Deep 3D-convolution stacks are also characterized as computationally expensive and memory-inefficient.Source: §I · p. 1
  • Guided Cost volume Excitation (GCE) uses image features to guide cost aggregation, improving performance with lightweight channel excitation.The method is also described as letting correlation-based cost volumes use image information and perform at similar accuracy to concatenation-based models.Source: §I · pp. 1–2
  • Top-k disparity regression computes disparity from only the top-k values in the distribution, addressing cases where a soft-argmin expectation may be unsuitable.The passage identifies multiple peaks and nearly uniform distributions as examples of non-unimodal disparity distributions.Source: §I · p. 2

II. RELATED WORKS

Related work spans end-to-end stereo methods that either regress disparity directly or aggregate a volumetric cost representation. CoEx builds on volumetric aggregation while targeting the accuracy–speed trade-off and improving soft-argmin regression.

II. RELATED WORKS

  • End-to-end stereo methods are broadly divided into direct regression with 2D convolutions and volumetric approaches that aggregate cost volumes using 3D convolutions.Volumetric methods account for geometric constraints and commonly use hourglass architectures.Source: §II · p. 2
  • Spatially dependent aggregation methods such as GANet and CSPN improve accuracy but require more computation, while real-time methods have sought speed with varying accuracy trade-offs.The passage notes that accuracy in real-time stereo models has been getting closer to that of the best-performing models.Source: §II · p. 2
  • Soft-argmin takes the expected value of a disparity-cost distribution and may fail in ambiguous regions; prior approaches improved its output distribution with novel losses.This work instead modifies the soft-argmin operation itself.Source: §II · p. 2

III. METHOD

The method builds a volumetric stereo network with correlation-based matching and image-feature guidance for cost aggregation. Its GCE module excites cost-volume channels using image-derived weights while leaving neighborhood aggregation to 3D convolutions.

III. METHOD

  • The end-to-end model combines matching-cost computation, cost aggregation, and disparity regression, with GCE and top-k soft-argmin designed for volumetric stereo baselines.The proposed modules are described as integrating without significant computation overhead.Source: §III · pp. 2–3

A. Matching cost computation

  • MobileNetV2 extracts features from both images, and left-right features at quarter resolution form a correlation cost volume with maximum disparity D = 192.The feature extractor uses a U-Net-style upsampling module with long skip connections and supplies features at each scale for guidance.Source: A. Matching cost computation · p. 3

B. Guided Cost volume Excitation (GCE)

  • GCE computes image-guided channel weights for cost-volume features, relying on 3D convolutions to capture geometric neighborhood information instead of adding neighborhood aggregation.The guidance sub-network outputs c weights per pixel from an image feature map at the same scale.Source: B. Guided Cost volume Excitation (GCE) · p. 3
  • GCE uses a 2D point-wise convolution and sigmoid to compute weights that are shared across disparity and broadcast-multiplied with cost-volume features.In CoEx, GCE is inserted at every scale in a reduced-depth, reduced-channel 3D-convolution hourglass derived from GC-Net.Source: B. Guided Cost volume Excitation (GCE) · p. 3

C. Top-k disparity regression

Top-k disparity regression addresses cases where a disparity distribution has multiple peaks or is nearly uniform, where its expected value can diverge from ground truth. It estimates disparity from only the top-k matching values at each pixel.

C. Top-k disparity regression

  • When disparity distributions have multiple peaks or are nearly uniform, their expected value can diverge significantly from ground truth.The paper identifies these distributions at edge boundaries and in textureless regions.Source: C. Top-k disparity regression · p. 3
  • Top-k regression masks all but the k largest aggregated cost values at each pixel, normalizes them with softmax, and computes a weighted average of their disparity indices.When k equals the number of disparities D, it reduces to soft-argmax; when k = 1, it becomes a non-trainable argmax.Source: C. Top-k disparity regression · p. 3
  • The final disparity regression is computed at 1/4 input resolution, then upsampled using learned weights over a surrounding 3 × 3 superpixel.A separate CNN branch predicts the superpixel weights.Source: C. Top-k disparity regression · p. 4

IV. EXPERIMENTS

The experiments section presents evaluations and ablations of CoEx, alongside detailed discussion of the method.

IV. EXPERIMENTS

  • The section reports experiments and ablations to assess CoEx and discusses the method in detail.Source: §IV · p. 4

A. Datasets and Evaluation metrics

The evaluation covers synthetic SceneFlow and real-world KITTI 2012 and 2015, with dataset-specific metrics. The section also specifies training implementation and identifies a same-hardware timing comparison.

A. Datasets and Evaluation metrics

  • SceneFlow evaluation uses EPE and excludes pixels with disparity at or above the model's maximum of 192.The dataset has 35,454 training and 4,370 testing images; the ‘finalpass’ version is used.Source: A. Datasets and Evaluation metrics · p. 4
  • Table II compares model runtimes in milliseconds on the same hardware.Source: Table II · p. 4
  • KITTI 2012 uses Out-All at a 3-pixel error threshold, while KITTI 2015 reports D1-all; training data is split 90%/10% for training and validation.Both are real-world datasets with sparse LiDAR ground truth.Source: A. Datasets and Evaluation metrics · p. 4

B. Implementation details

  • Training uses ImageNet-pretrained MobileNetV2, Adam with SWA, and 576 × 288 crops.SceneFlow training runs for 10 epochs; KITTI is fine-tuned for 800 epochs, and an RTX 2080Ti is used for training and testing.Source: B. Implementation details · p. 4

C. Performance of CoEx

CoEx compares favorably with speed-oriented stereo models, and ablations show gains from GCE and top-k regression with little added computation. The ablations also identify benefits from integrating GCE across scales and using top-2 regression.

C. Performance of CoEx

  • StereoNet is fastest at 15 ms, but its accuracy is lower than CoEx by 0.411 EPE on SceneFlow and 2.7% on KITTI 2015.KITTI results come from the Stereo Matching Leaderboard; SceneFlow EPE and runtime are reported values from each work.Source: C. Performance of CoEx · p. 4
  • 3.3× faster than AANet, CoEx gives 0.18 EPE lower, 0.46% better KITTI 2012 3px out-all%, and 0.42% better D1-all% on KITTI 2015.The runtime comparison used the same RTX 2080Ti hardware and official open-source models.Source: C. Performance of CoEx · p. 4

D. Ablation study

  • A single GCE layer brings correlation-based PSMNet accuracy to a similar value as concatenation-based PSMNet, while correlation reduces computation time.The authors attribute this to GCE letting the network use image-feature representations missed by correlation.Source: D. Ablation study · p. 5
  • With PSMNet's D=192, reducing k from 192 generally improves accuracy until too-low k degrades it, possibly from weaker gradient flow; full-volume sorting is costly.The PSMNet cost volume is upsampled to the original image resolution before regression.Source: D. Ablation study · pp. 5–6
  • CoEx regresses disparity at quarter resolution and uses superpixel upsampling; with maximum k=48, top-k adds little computation time and lower k improves accuracy.The maximum k is 192/4 because regression is performed at one-quarter input resolution.Source: D. Ablation study · p. 6
  • Full-scale GCE with top-2 regression adds 1 ms over base CoEx and lowers test EPE by 0.17.This is the best model reported in the ablation study.Source: Table III · p. 5, D. Ablation study · p. 6

1) GCE:

The section compares image-guided cost-volume addition with excitation, then compares GCE with image-guided neighborhood aggregation. GCE is reported as the more effective approach in both comparisons.

1) GCE:

  • Cost-volume excitation was more effective than image-feature addition, which gave only a slight accuracy improvement over the baseline.The comparison was performed on SceneFlow.Source: 1) GCE · p. 6
  • The two image-guided approaches investigated were feature addition and cost-volume excitation.Addition used a broadcast operation that acted like a UNet-style skip connection.Source: 1) GCE · p. 6
  • GCE performed better and more efficiently than the implemented spatially varying neighborhood aggregation.That comparator uses image-derived, spatially varying graph edges to aggregate neighboring features.Source: 1) GCE · p. 6

2) Top-2 disparity regression:

Top-2 regression is reported to improve disparity estimates across three distribution shapes, while replacing trained soft-argmin with top-k only at test time did not improve accuracy. The conclusion presents CoEx as a lightweight, real-time stereo model.

2) Top-2 disparity regression:

  • Top-2 regression produced more accurate estimates than soft-argmin in unimodal, two-peak, and relatively flat disparity distributions.It uses peak matching values and suppresses values far from the correct matching peak.Source: 2) Top-2 disparity regression · p. 6
  • Replacing soft-argmin with top-k only at test time gave no accuracy improvement; models need to learn top-k regression during training.Source: 2) Top-2 disparity regression · p. 6

V. CONCLUSION

  • CoEx combines lightweight spatially varying aggregation and top-k disparity regression in a model described as fast enough for real-time applications.The conclusion presents its speed as a springboard for future real-time stereo-matching research in real-world application settings.Source: §V · p. 6

APPENDIX

The appendix identifies Table VI as the detailed cost-aggregation module and clarifies the notation for its convolution parameters and left-image features.

APPENDIX

B. Neighborhood aggregation

The neighborhood-aggregation comparator updates a cost-volume feature using image-guided weights over a local neighborhood. Its simplest pointwise-weight implementation requires at least n×n more computation than GCE.

B. Neighborhood aggregation

Loading 2108.05773v1…