PaperlayerView exact revision on arXiv

Source-linked AI summary

IGEV++: Iterative Multi-range Geometry Encoding Volumes for Stereo Matching

Gangwei Xu, Xianqi Wang, Zhaoxing Zhang, Junda Cheng, Chunyuan Liao, Xin Yang

arXiv:2409.00638v3cs.CV

TL;DR

Stereo matching still faces matching ambiguities in ill-posed regions and difficulty handling large disparities. IGEV++ constructs multi-range geometry encoding volumes and uses them to iteratively update disparity. It achieves the best Scene Flow performance across disparity ranges up to 768px and state-of-the-art accuracy on four benchmarks.

  • Problem

    Stereo matching remains challenged by ambiguities in ill-posed regions, while full-range cost volumes for large disparities carry substantial computational and memory costs.

    Source: §I · pp. 1–2

  • Method

    IGEV++ builds multi-range geometry encoding volumes with coarse geometry for ill-posed regions and large disparities and fine geometry for details and small disparities, then selectively fuses features for iterative disparity updates.

    Source: §V · pp. 12–13

  • Results

    IGEV++ achieves the best Scene Flow performance across disparity ranges up to 768px and state-of-the-art accuracy on Middlebury, ETH3D, KITTI 2012, and KITTI 2015.

    Source: §V · p. 13

  • Takeaways & Limitations

    The method also exhibits impressive generalization ability to unseen real-world datasets.

    Source: §V · p. 13

Abstract

from arXiv · show

Stereo matching is a core component in many computer vision and robotics systems. Despite significant advances over the last decade, handling matching ambiguities in ill-posed regions and large disparities remains an open challenge. In this paper, we propose a new deep network architecture, called IGEV++, for stereo matching. The proposed IGEV++ constructs Multi-range Geometry Encoding Volumes (MGEV), which encode coarse-grained geometry information for ill-posed regions and large disparities, while preserving fine-grained geometry information for details and small disparities. To construct MGEV, we introduce an adaptive patch matching module that efficiently and effectively computes matching costs for large disparity ranges and/or ill-posed regions. We further propose a selective geometry feature fusion module to adaptively fuse multi-range and multi-granularity geometry features in MGEV. Then, we input the fused geometry features into ConvGRUs to iteratively update the disparity map. MGEV allows to efficiently handle large disparities and ill-posed regions, such as occlusions and textureless regions, and enjoys rapid convergence during iterations. Our IGEV++ achieves the best performance on the Scene Flow test set across all disparity ranges, up to 768px. Our IGEV++ also achieves state-of-the-art accuracy on the Middlebury, ETH3D, KITTI 2012, and 2015 benchmarks. Specifically, IGEV++ achieves a 3.23\% 2-pixel outlier rate (Bad 2.0) on the large disparity benchmark, Middlebury, representing error reductions of 31.9\% and 54.8\% compared to RAFT-Stereo and GMStereo, respectively. We also present a real-time version of IGEV++ that achieves the best performance among all published real-time methods on the KITTI benchmarks. The code is publicly available at https://github.com/gangweix/IGEV and https://github.com/gangweix/IGEV-plusplus.

I. INTRODUCTION

IGEV++ combines filtering-based geometry encoding with iterative disparity updates to address matching ambiguities in ill-posed regions and large disparities. It reports strong results across disparity ranges and benchmarks, including a real-time version for KITTI.

I. INTRODUCTION

  • Filtering-based methods provide non-local geometry through cost-volume regularization, while iterative methods avoid costly aggregation but struggle with ill-posed regions using all-pairs correlations.Fixed-range cost volumes also limit large-disparity handling; full-range volumes carry substantial computational and memory costs.Source: §I · pp. 1–2
  • IGEV++ builds MGEV with coarse-grained geometry for ill-posed regions and large disparities while preserving fine-grained geometry for details and small disparities.Adaptive patch matching constructs MGEV, and selective geometry feature fusion integrates features across ranges and granularities.Source: §I · pp. 2–3
  • MGEV provides geometry features for iterative ConvGRU updates, and IGEV++ reaches EPE 0.79 in 4 iterations versus DLNR’s inference EPE 0.81 in 32 iterations.The authors attribute faster convergence to more comprehensive yet concise information for ConvGRUs.Source: §I · p. 2, Figure 3 · p. 3
  • RT-IGEV achieves real-time inference and the best accuracy among published real-time methods on the KITTI benchmarks.The paper presents it as a version for time-constrained applications.Source: §I · pp. 2–3
  • IGEV++ achieves state-of-the-art accuracy on Middlebury, ETH3D, KITTI 2012, and KITTI 2015, and the highest Scene Flow accuracy across disparity ranges up to 768px.The comparisons span disparity ranges and benchmark leaderboards.Source: Figure 1 · p. 1, §I · p. 3

A. Cost Volume Filtering-based Methods

The related-work sections contrast cost-volume filtering, iterative optimization, and transformer stereo methods, each with distinct strengths and limitations. IGEV++ uses geometry encoding volumes with ConvGRU updates, supported by multi-scale feature and context networks.

A. Cost Volume Filtering-based Methods

  • Cost-volume filtering regularizes ambiguous matches with 3D convolutions, but high memory and computational costs limit high-resolution use; coarse-to-fine methods risk accumulated errors.IGEV++ instead updates disparity at high resolution through ConvGRUs.Source: A. Cost Volume Filtering-based Methods · pp. 3–4

B. Iterative Optimization-based Methods

  • Iterative methods update disparity using local costs from all-pairs correlations, but those correlations lack non-local information and leave ill-posed ambiguities challenging.IGEV++ adds MGEV geometry and context information to ConvGRU updates and provides an accurate initial disparity map for faster convergence.Source: B. Iterative Optimization-based Methods · p. 4

C. Transformer-based Stereo Methods

  • Transformer stereo methods model long-range dependencies and can handle large disparities to some extent, but lack of an explicit cost volume hinders resolving local ambiguities in ill-posed regions.The passage identifies occlusions and reflective regions as examples.Source: C. Transformer-based Stereo Methods · p. 4

III. METHOD

  • IGEV++ comprises multi-scale feature extraction, multi-range geometry encoding volume construction, and a ConvGRU-based update operator.The feature network supports cost construction and aggregation guidance, while the context network supplies ConvGRU hidden-state initialization and updates.Source: §III · p. 4, A. Multi-scale Feature Extraction · p. 4

A. Multi-scale Feature Extraction

  • The feature network extracts multi-scale features from both images, while multi-scale context features initialize ConvGRU hidden states and are inserted during disparity updates.The context features are produced at 1/4, 1/8, and 1/16 input resolution with 128 channels.Source: A. Multi-scale Feature Extraction · p. 4

B. Multi-range Geometry Encoding Volumes Construction

IGEV++ builds correlation volumes at multiple disparity ranges, then regularizes them into geometry encoding volumes for multi-range and multi-granularity guidance. Adaptive patch matching represents large disparities with fewer candidates, while a lightweight 3D network supplies geometry context.

B. Multi-range Geometry Encoding Volumes Construction

  • Small, medium, and large group-wise correlation volumes cover disparity ranges below 192, 384, and 768 pixels, respectively.Group-wise correlation splits features into eight channel groups and computes correlations group by group.Source: B. Multi-range Geometry Encoding… · p. 5
  • For large disparities, adaptive point-to-patch matching uses 48 candidates to represent the 768-pixel range, reducing aggregation cost and easing disparity regression.The matching weights within a right-image patch are learned during training.Source: B. Multi-range Geometry Encoding… · p. 5
  • The 3D regularization network is a lightweight UNet with three down-sampling and three up-sampling blocks.The down-sampling blocks have 16, 32, and 48 channels, respectively.Source: B. Multi-range Geometry Encoding… · p. 5
  • A lightweight 3D regularizer transforms the multi-range correlation volumes into geometry encoding volumes that capture multi-scale geometry and help resolve matching noise and ambiguity.Guided cost-volume excitation is incorporated into the regularization network.Source: B. Multi-range Geometry Encoding… · p. 5

C. ConvGRU-based Update Operator

The update operator initializes disparity from the geometry volumes, then iteratively refines it with ConvGRUs using selectively fused geometry features. SGFF weights features by disparity range and region type.

C. ConvGRU-based Update Operator

  • Soft argmin on the small-, medium-, and large-range geometry volumes produces initial disparity maps, and the small-range map starts the ConvGRU iterations to speed convergence.The previous disparity indexes geometry features for each update.Source: C. ConvGRU-based Update Operator · p. 5
  • SGFF favors long-range, coarse geometry for large-disparity or ill-posed regions and short-range, fine geometry for small-disparity or detailed regions.It predicts selective weights from the initial disparity maps and left-image features.Source: C. ConvGRU-based Update Operator · p. 6
  • At each iteration, indexed geometry features are fused with all-pairs correlation values and the previous disparity, then passed through ConvGRUs to update the disparity map.The fused feature and disparity are encoded before entering the recurrent update.Source: C. ConvGRU-based Update Operator · pp. 5–6

D. Spatial Upsampling

The upsampling module uses higher-resolution context to predict weights for combining neighboring low-resolution disparities into a full-resolution map.

D. Spatial Upsampling

  • Full-resolution disparity is a weighted combination of neighboring points in the low-resolution map, with weights predicted from upsampled hidden-state features and left-image context.The hidden state is upsampled to half resolution, concatenated with left-image features, and used to produce the weights.Source: D. Spatial Upsampling · p. 6

E. Loss Function

The loss combines regularization on disparity maps regressed from MGEV with iteration loss on updated disparity maps. The supplied passages also describe MGEV’s coarse- and fine-grained geometry encoding.

E. Loss Function

  • The regularization loss compares MGEV-regressed small-, medium-, and large-range disparity maps with ground-truth disparity.The passage names these maps as ds 0, dm 0, and dl 0.Source: E. Loss Function · p. 6
  • The iteration loss is computed between the updated disparity maps {di}N.Source: E. Loss Function · p. 6
  • The total loss is represented as a combination of regularization loss and iteration loss.MGEV encodes coarse- and fine-grained geometry information for large disparities and subtle details.Source: E. Loss Function · p. 6, Figure 7 · p. 7

A. Datasets and Evaluation Metrics

The section describes the evaluation datasets and training setup, then compares IGEV++ with representative methods on Scene Flow across disparity ranges. The reported results emphasize robust large-disparity performance and lower memory use.

A. Datasets and Evaluation Metrics

  • Scene Flow uses 35,454 training pairs and 4,370 test pairs; evaluation uses EPE and Bad 3.0, the rate of pixels with disparity error above 3 px.Source: A. Datasets and Evaluation Metrics · p. 7
  • KITTI datasets provide sparse LiDAR disparities; Middlebury and ETH3D provide indoor and indoor/outdoor evaluation settings, respectively.Middlebury evaluation includes full- and half-resolution data for cross-domain large-disparity testing, using Bad 2.0.Source: A. Datasets and Evaluation Metrics · p. 7

B. Implemention Details

  • Training uses 22 update iterations, a one-cycle learning-rate schedule at 2e-4, and λs, λm, and λl weights of 1.0, 0.5, and 0.2.Source: B. Implemention Details · p. 8

C. Comparisons with Existing Representative Methods

  • IGEV++ consistently outperforms representative methods across Scene Flow disparity ranges and remains robust as disparity range grows; it uses 7× less memory than PCWNet for large disparities.PCWNet’s EPE rises from 0.78 to 1.52 when its disparity range is extended.Source: C. Comparisons with Existing… · pp. 8–9

D. Ablation Study

The ablations show that combining geometry ranges and using adaptive patch matching address complementary disparity and efficiency challenges. Benchmark results report strong KITTI, Middlebury, and ETH3D performance, while reduced iterations retain accuracy.

D. Ablation Study

  • MGEV significantly improves accuracy within 192px and in large-disparity regions; the final IGEV++ model achieves the best performance across all disparity ranges.The selective geometry feature fusion module fuses features from three ranges and three granularities for iterative updates.Source: D. Ablation Study · p. 9
  • Small-range GEV handles small disparities but struggles with large ones, while large-range GEV handles large disparities but tends to lose small-range details; MGEV combines their strengths.Source: D. Ablation Study · p. 10
  • APM constructs large- and medium-range GEVs with 48 disparity candidates for cost aggregation, rather than 192 candidates for the large-range volume without APM.The passage links large cost volumes with greater aggregation and regression complexity, especially in textureless regions and at large disparities.Source: D. Ablation Study · p. 10
  • With one iteration, IGEV++ surpasses RAFT-Stereo and DLNR by 72.91% and 57.27%, respectively, and retains state-of-the-art performance with few iterations.The iteration count offers a time-efficiency and accuracy trade-off.Source: D. Ablation Study · p. 10

E. Benchmark Results

  • 17.89% on KITTI 2012 2-noc versus LEAStereo and 14.69% on KITTI 2015 D1-all versus GMStereo are reported IGEV++ improvements.IGEV++ is reported best among published methods for almost all KITTI 2012 and 2015 metrics, with 100 ms additional runtime versus IGEV.Source: E. Benchmark Results · p. 10
  • IGEV++ outperforms most published methods on Middlebury and ETH3D; on ETH3D it performs better than RAFT-Stereo and GMStereo in textureless regions.Source: E. Benchmark Results · p. 11

F. Performance in Ill-posed Regions

In reflective KITTI 2012 regions, IGEV++ outperforms RAFT-Stereo. A real-time variant, RT-IGEV, modifies the architecture and ranks first among published real-time methods on KITTI benchmarks.

F. Performance in Ill-posed Regions

  • 31.30% improvement over RAFT-Stereo on KITTI 2012 reflective-region 3-noc is reported for IGEV++.The authors attribute the result to MGEV encoding non-local geometry information and scene prior knowledge for ill-posed-region matching.Source: F. Performance in Ill-posed Regions · p. 11

G. Real-Time Version of IGEV++

  • RT-IGEV removes the context network, uses a single-range 192px GEV and single-level ConvGRU, and reduces hidden-state channels from 128 to 96.The variant enables real-time inference on KITTI-resolution images (1248 × 384).Source: G. Real-Time Version of IGEV++ · pp. 11–12
  • RT-IGEV ranks first among published real-time methods on KITTI 2012 and 2015; it improves over HITNet by 8.51% on 3-noc and 13.64% on D1-all.Source: G. Real-Time Version of IGEV++ · p. 12

H. Zero-shot Generalization Performance

Zero-shot evaluations report strong performance on Middlebury V3 and SCARED, while the conclusion reports broad benchmark results for IGEV++ and introduces a real-time variant. The paper attributes its handling of large disparities and ill-posed regions to multi-range geometry encoding volumes and their construction and fusion modules.

H. Zero-shot Generalization Performance

V. CONCLUSION

  • MGEV encodes coarse geometry for ill-posed regions and large disparities while preserving fine geometry for details and small disparities.Adaptive patch matching constructs MGEV, and selective geometry feature fusion combines features across ranges and granularities; the geometry volume is iteratively indexed to update disparity.Source: §V · pp. 12–13
  • IGEV++ achieves the best Scene Flow test performance across disparity ranges up to 768px and state-of-the-art accuracy on Middlebury, ETH3D, KITTI 2012, and 2015.The conclusion also reports impressive generalization to unseen real-world datasets.Source: §V · p. 13
  • RT-IGEV is the real-time version, delivering real-time inference and the best performance among published real-time methods.It is presented as a solution for time-constrained applications.Source: §V · p. 13

Related papers

Loading 2409.00638v3…