Source-linked AI summary
MoCha-Stereo: Motif Channel Attention Network for Stereo Matching
Ziyang Chen, Wei Long, He Yao, Yongjun Zhang, Bingshu Wang, Yongbin Qin, Jia Wu
TL;DR
Learning-based stereo methods can lose geometric structure in feature channels, leaving edge details mismatched. MoCha-Stereo uses motif channels to build matching costs and applies reconstruction-error frequency information to refine disparity. It ranks first on KITTI 2015 and KITTI 2012 Reflective leaderboards and reports strong results across other evaluations.
Problem
Learning-based stereo methods can lose geometric detail in feature channels, blurring edges and making edge matching difficult.
Source: §1 · p. 2, §2.2 · p. 3Method
MoCha-Stereo uses repeated geometric contours in motif channels to construct MCCV matching costs and applies REMP to refine full-resolution disparity using reconstruction-error frequency information.
Source: §1 · p. 2Results
MoCha-Stereo ranks first on KITTI 2015 and KITTI 2012 Reflective, reports SOTA results on Scene Flow, zero-shot, and MVS, and achieves superior results with fewer iterations.
Source: §1 · p. 2Takeaways & Limitations
Its iteration results let users choose between efficient or high-precision settings according to their preferences.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Learning-based stereo matching techniques have made significant progress. However, existing methods inevitably lose geometrical structure information during the feature channel generation process, resulting in edge detail mismatches. In this paper, the Motif Cha}nnel Attention Stereo Matching Network (MoCha-Stereo) is designed to address this problem. We provide the Motif Channel Correlation Volume (MCCV) to determine more accurate edge matching costs. MCCV is achieved by projecting motif channels, which capture common geometric structures in feature channels, onto feature maps and cost volumes. In addition, edge variations in %potential feature channels of the reconstruction error map also affect details matching, we propose the Reconstruction Error Motif Penalty (REMP) module to further refine the full-resolution disparity estimation. REMP integrates the frequency information of typical channel features from the reconstruction error. MoCha-Stereo ranks 1st on the KITTI-2015 and KITTI-2012 Reflective leaderboards. Our structure also shows excellent performance in Multi-View Stereo. Code is avaliable at https://github.com/ZYangChen/MoCha-Stereo.
1. Introduction
Learning-based stereo methods can lose geometric detail in feature channels, producing mismatched object edges. MoCha-Stereo restores repeated geometric contours to improve edge matching and disparity refinement.
1. Introduction
- Geometric detail loss in feature channels blurs object edges, making edge matching difficult and motivating closer alignment between disparity-map edges and the RGB image.The paper links this loss to nonlinear transformations in network blocks, which can saturate or inadequately activate some channels.Source: §1 · pp. 1–2
- MoCha-Stereo uses repeated geometric contours in motif channels to restore feature detail, constructs MCCV for matching costs, and applies REMP to refine full-resolution disparity using reconstruction-error frequency information.MCA projects motif channels, while REMP uses high- and low-frequency signals from the reconstruction error.Source: §1 · p. 2
- MoCha-Stereo ranks first on KITTI 2015 and KITTI 2012 Reflective, reports SOTA on Scene Flow, zero-shot, and MVS, and achieves superior results with fewer iterations.Figure 2 compares leaderboard results and Scene Flow performance as iteration counts change.Source: §1 · p. 2, Figure 2 · p. 2
2. Related Work
The related work introduces motif mining as a way to identify repeated patterns, then adapts that idea to geometric structures in image features. It also reviews learning-based stereo methods and motivates restoring channel geometry and using error-map frequency information.
2.1. Motif Mining for Time series analysis
- Motif mining identifies the most similar pair of equal-length subsequences; MoCha-Stereo adapts repeated-pattern discovery to two-dimensional image geometry using sliding windows to form motif channels.The paper notes that direct similarity computation over multi-channel image sub-patches would be computationally expensive.Source: §2.1 · pp. 2–3
2.2. Learning-based Stereo Matching
- Learning-based stereo work includes GWC, RAFT-Stereo's APC pyramid, and IGEV-Stereo's combined APC/GWC volume with iterative disparity updates; disparity post-processing methods also use reconstruction errors.The cited post-processing approaches include iResNet, DLNR, and DRNet.Source: §2.2 · p. 3
- The paper identifies limited prior use of repeated geometric profiles to restore feature-map edge texture and limited attention to frequency information in potential motif channels from error maps.These gaps motivate MCCV and REMP, which project motif-channel information to reinstate feature-channel geometry.Source: §2.2 · p. 3
3. Method
MoCha-Stereo projects motif channels onto feature maps and cost volumes to restore geometric structure in channel features. Its method combines motif-guided matching costs with reconstruction-error refinement of disparity.
3.1. Context Extractor & Feature Extractor
- The context extractor produces 1/4, 1/8, and 1/16-scale context, while the feature extractor uses an ImageNet-pretrained frozen backbone and multi-scale outputs.The feature channels are denoted C_i for i = 4, 8, 16, 32; f_l and f_r denote left- and right-view features.Source: §3.1 · pp. 3–4
3.2. Motif Channel Correlation Volume
- MCCV mines repeated geometric patterns with adaptive 3 × 3 sliding windows, then uses motif-to-normal-channel affinities to reinforce spatial patterns in matching costs.The motif features are extracted from frequency-domain features; the cost volume combines the channel correlation with group-wise correlation, broadcasting the shared geometric structure across groups.Source: §3.2 · p. 4
3.3. Iterative Update Operator
- At each iteration, the disparity map is updated at 1/4 resolution as d_k = d_(k−1) + Δd_k.The iterative update operator follows [37].Source: §3.3 · p. 4
3.4. Reconstruction Error Motif Penalty
- REMP refines the upsampled disparity using reconstruction error and the last-iteration disparity, separating low-frequency, motif, and high-frequency feature branches.Pooling in LFE retains low-frequency information, LMC learns motif features, and HFE preserves high-resolution detail; their mapped feature errors act as penalties on disparity.Source: §3.4 · p. 5, Figure 4 · p. 5
3.5. Loss Function
- The initial disparity is generated from concatenated group correlation volumes, and the total loss combines an initial-disparity term with weighted L1 losses over iterative predictions.The stated weight is γ = 0.9, and d_gt denotes ground-truth disparity.Source: §3.5 · p. 5
4. Experiment
MoCha-Stereo is evaluated on benchmark performance, zero-shot generalization, Multi-View Stereo, and ablations. Results report strong benchmark performance and show how motif-channel components and iteration count relate to accuracy and inference efficiency.
4.1. Implementation Details
- Training and ablation experiments use Scene Flow for 200k epochs, batch size 8, and two NVIDIA A6000 GPUs; evaluations include five named datasets.The training and testing settings follow [18] [37].Source: §4.1 · p. 5
4.2. Comparisons with State-of-the-art
- MoCha-Stereo ranks first among submitted methods on KITTI-2012 and KITTI-2015, and achieves Scene Flow EPE 0.41, surpassing IGEV-Stereo [37] by 12.77%.It ranks first in reflective regions, keeps Avg-Noc within 0.8 px under a 5 px error threshold, and keeps Out-Noc below 4% at a 3 px threshold.Source: §4.2 · p. 6
4.3. Zero-shot Generalization
- Without fine-tuning, MoCha-Stereo achieves state-of-the-art performance on Middlebury and ETH3D zero-shot evaluations.The evaluation uses Scene Flow-trained models; Table 4 specifies a 2-pixel error rate for Middlebury and 1-pixel error rate for ETH3D.Source: §4.3 · p. 6, Table 4 · p. 7
4.4. Extension to MVS
- MoCha-Stereo is extended as MoCha-MVS and outperforms methods designed for Multi-View Stereo while balancing accuracy and completeness.The reported quantitative evaluation is on the DTU dataset.Source: §4.4 · p. 6, Table 5 · p. 7
4.5. Ablations
- MCCV’s MCA and CAMP components act on feature maps and correlation volumes, respectively; together they reduce EPE by 7.6%, from 0.458 to 0.423.The reported explanation is that MCCV addresses geometric edge information lost in some feature channels, improving matching-cost computation.Source: §4.5 · p. 7
- REMP uses reconstruction-error motif information about high- and low-frequency errors as a disparity penalty, reducing EPE by 2.8%, from 0.458 to 0.445.MoCha-Stereo also reports strong results with fewer iterations: eight iterations reduce EPE by 40.8% versus UPFNet at comparable inference time, while four iterations outperform IGEV-Stereo by over 2.1% in accuracy and save 40.5% inference time.Source: §4.5 · p. 8
5. Conclusion and Future Work
MoCha-Stereo uses motif-channel structure to refine edge matching and reports strong results across stereo benchmarks and multi-view stereo. The authors plan to extend motif-channel attention to more stereo-matching processes.
5. Conclusion and Future Work
- MoCha-Stereo ranks 1st on the KITTI-2015 and KITTI-2012 Reflective online benchmarks and reports SOTA performance on ETH3D, Middlebury, Scene Flow, and MVS.The authors describe robust cross-dataset generalization.Source: §5 · p. 8
- MCCV uses repeated geometric patterns in channel features to restore missing edge details and reconstruct the cost volume.REMP penalizes full-resolution disparity generation using high- and low-frequency information from potential motif channels in the reconstruction error.Source: §5 · p. 8
- The authors plan to extend motif-channel attention to more processes in stereo matching to further enhance edge-matching capability.Source: §5 · p. 8