Source-linked AI summary
WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching
Yihan Wang, Jia Deng
TL;DR
Stereo matching commonly relies on cost volumes that add memory overhead and low-resolution processing, motivating alternatives that preserve strong correspondence estimation. WAFT-Stereo uses high-resolution warping with iterative updates and an initial disparity classification step, achieving strong benchmark accuracy, zero-shot generalization, and improved efficiency.
Problem
Cost volumes used by leading stereo methods incur memory overhead that scales with disparity range or lookup radius, while low-resolution processing can hurt detailed-region accuracy.
Method
WAFT-Stereo replaces cost-volume-specific designs with high-resolution warping and iterative updates, adding classification of discretized disparity before recurrent refinement.
Results
WAFT-Stereo achieves the best ETH3D BP-0.5 among existing zero-shot submissions, reducing error by 81% and reducing KITTI-2015 D1 by 9% versus leading zero-shot methods.
Takeaways & Limitations
Strong stereo performance can be obtained without cost volumes, using a fully warping-based architecture with high efficiency and strong generalization.
Takeaways & Limitations
WAFT-Stereo falls behind leading methods on Middlebury’s averaged BP-2 across all 15 scenes, with the gap largely dominated by the illumination-different Classroom2E scene.
Abstract
from arXiv · showhide
We introduce WAFT-Stereo, a simple and effective warping-based method for stereo matching. WAFT-Stereo demonstrates that cost volumes, a common design used in many leading methods, are not necessary for strong performance and can be replaced by warping with improved efficiency. WAFT-Stereo ranks first on ETH3D (BP-0.5), Middlebury (RMSE), and KITTI (all metrics), reducing the zero-shot error by 81% on ETH3D, while being 1.8-6.7x faster than competitive methods. Code and model weights are available at https://github.com/princeton-vl/WAFT-Stereo.
1 Introduction
WAFT-Stereo replaces cost-volume-specific stereo designs with high-resolution warping and iterative updates, targeting efficiency, accuracy, and generalization. A classification module addresses the difficulty of large stereo displacements before recurrent refinement.
- Stereo matching estimates each pixel’s horizontal motion from rectified camera images, and disparity can be converted directly to depth.The task supports applications including autonomous driving and augmented reality.
- Cost volumes impose memory overhead that scales with disparity range or lookup radius, so they are typically processed at low resolution.Low resolution can hurt accuracy in highly detailed image regions.
- High-resolution warping with iterative updates can replace cost volumes while providing a simpler design with improved accuracy and efficiency.WAFT-Stereo demonstrates that strong stereo performance does not require cost-volume-specific designs.
- A classification module predicts discretized disparity bins before recurrent updates, improving large-displacement accuracy at negligible additional cost.The classification step addresses regression-only convergence difficulties for hundreds-of-pixels stereo displacements.
- 1.8–6.7× speedup over leading methods accompanies 10 FPS processing at 540p, while a smaller-backbone model reaches 21 FPS with comparable accuracy.The reported comparisons include 1.8× over S2M2-XL and 6.7× over FoundationStereo.
- 81% error reduction on ETH3D in zero-shot evaluation demonstrates strong generalization from synthetic-only training.WAFT-Stereo also reduces KITTI-2015 D1 by 9% relative to leading zero-shot methods.
2 Related Work
Related work centers on iterative deep stereo methods, cost volumes, and the challenge of obtaining real-world training data. WAFT-Stereo instead uses a fully warping-based design and achieves strong performance after synthetic-only training.
- Most modern stereo methods use deep learning, with many leading approaches iteratively refining disparity by regressing updates.
- WAFT-Stereo adds one-step classification of discretized disparity before regression-based iterative updates.Earlier methods used cost volumes for classification because normalized cost-volume values can represent matching probabilities.
- WAFT-Stereo removes reliance on cost-volume-specific network designs and uses standard architecture components in a fully warping-based framework.The design provides strong performance together with high efficiency.
- Real-world stereo annotation is expensive and technically challenging, leaving public real-world datasets three orders of magnitude smaller than synthetic datasets.Most leading methods therefore combine synthetic and real data to mitigate the sim-to-real gap.
- Cost volumes compare feature pairs across disparity candidates, whereas warping aligns target features using the current estimate without computing matching costs.Full volumes cover all candidates, while partial volumes restrict computation to a local window.
- Synthetic-only training nevertheless enables WAFT-Stereo to outperform leading approaches on public leaderboards, indicating strong generalization.
3 Method
WAFT-Stereo replaces cost-volume indexing with high-resolution feature warping and iterative refinement, adding classification before regression to handle large stereo displacements. The resulting design improves accuracy and efficiency while retaining standard architecture components.
- Iterative Refinement with Warping: Warping replaces cost-volume indexing by backward-warping right-view features with the current disparity estimate before recurrent refinement.Disparity is treated as the horizontal component of optical flow, so the optical-flow warping mechanism transfers to stereo matching.
- Iterative Refinement with Warping: Warping scales linearly with spatial resolution without depending on disparity range, enabling high-resolution processing and 10 FPS at 540p.The reported speed is 6.7× over FoundationStereo and 1.8× over S2M2-XL.
- Classification before Regression: Large stereo displacements make regression-only iterative updates harder to predict and slower to converge than typical optical-flow displacements.The paper notes that high-resolution stereo pairs can contain displacements of hundreds of pixels.
- Classification before Regression: The classification module predicts a soft disparity distribution over predefined bins, whose soft-argmax estimate initializes the first regression-stage warp.The distribution is trained with soft cross-entropy, while later updates use the Mixture-of-Laplace loss.
- Classification before Regression: Classification followed by regression produces a coarse, stable disparity estimate that later updates refine more accurately than regression alone.In the same four-iteration setting, the hybrid design improves accuracy with negligible additional cost; on ETH3D, it achieves 0.89 EPE for 40px+ disparity and 1.80 EPE for 50px+ disparity, versus 1.11 and 2.88 EPE for regression.
- Implementation: WAFT-Stereo simplifies WAFT by fine-tuning the pretrained encoder with LoRA, replacing its U-Net adaptation layer, and using ResNet blocks in recurrent updates.The ResNet replacement is reported to improve accuracy, while LoRA reduces latency.
4 Experiments
Experiments evaluate WAFT-Stereo on ETH3D, KITTI, and Middlebury, including zero-shot settings, efficiency, architectural ablations, and training-data effects. The method achieves strong benchmark accuracy and speed, while revealing sensitivity to challenging illumination, dataset bias, disparity-bin count, and regression/classification design.
- Benchmark results: WAFT-Stereo ranks first on KITTI-2012 BP-2 and KITTI-2015 D1, reducing errors by 13% and 6% over MonSter++, respectively.It achieves these results with a 2.0× speed-up using the same DepthAnythingV2-L backbone.
- Benchmark results: WAFT-Stereo achieves the best Middlebury RMSE, improving error by 5% while matching or surpassing leading methods on BP-4.It is 6.7×, 2.0×, and 1.5× faster than FoundationStereo, MonSter++, and DEFOM-Stereo, respectively.
- Limitations and trade-offs: WAFT-Stereo falls behind leading methods on averaged Middlebury BP-2 across all 15 scenes, largely because of the illumination-different Classroom2E scene.Excluding Classroom2E significantly narrows the gap, while scene-wise results place the method on the accuracy–efficiency Pareto frontier.
- Efficiency and training data: 6.7× speed-up and 5.3× fewer MACs are obtained versus FoundationStereo under its zero-shot training-data setting, with comparable performance.With the broader MonSter++ zero-shot setting, WAFT-Stereo uses fewer data while achieving significantly better performance across all metrics, with 2.0× speed-up and 2.1× fewer MACs.
- Architectural ablations: Combining classification with regression gives the best practical performance, whereas classification alone underperforms and cost-volume classification adds no noticeable improvement.The classification stage provides coarse disparity estimates that are refined by iterative regression; 40 bins are selected as the accuracy–memory trade-off.