Source-linked AI summary
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching
Lahav Lipson, Zachary Teed, Jia Deng
TL;DR
Stereo networks must estimate rectified-image disparities while managing the computational cost and limited generalization of conventional 3D cost-volume architectures. RAFT-Stereo adapts RAFT with a lightweight correlation volume and multi-level GRU updates, achieving strong cross-dataset performance and leading benchmark results, with a real-time configuration.
Problem
Conventional stereo relies heavily on computationally expensive 3D convolutions over cost volumes, limiting resolution and motivating architectures with better efficiency and generalization.
Method
RAFT-Stereo adapts RAFT for rectified stereo by using a same-height 3D correlation volume and multi-level GRUs that propagate information across multiple resolutions.
Results
RAFT-Stereo achieves state-of-the-art cross-dataset generalization, ranks first on Middlebury, and outperforms all published work on ETH3D; a modified version reaches 26 FPS with 5.91 D1 error on KITTI-resolution images.
Takeaways & Limitations
The architecture provides an accurate stereo method that can generalize from synthetic training data to real datasets and can be configured for real-time inference.
Abstract
from arXiv · showhide
We introduce RAFT-Stereo, a new deep architecture for rectified stereo based on the optical flow network RAFT. We introduce multi-level convolutional GRUs, which more efficiently propagate information across the image. A modified version of RAFT-Stereo can perform accurate real-time inference. RAFT-stereo ranks first on the Middlebury leaderboard, outperforming the next best method on 1px error by 29% and outperforms all published work on the ETH3D two-view stereo benchmark. Code is available at https://github.com/princeton-vl/RAFT-Stereo.
1. Introduction
RAFT-Stereo addresses the computational limitations of conventional stereo networks by adapting RAFT’s iterative flow framework to rectified stereo. Its multi-level update operator improves information propagation and supports strong cross-dataset performance and high-resolution inference.
- Problem: Stereo estimates a pixelwise horizontal disparity between rectified left and right images, which can be combined with calibration to recover depth.The resulting disparity supports 3D point clouds and other downstream 3D representations.
- Problem: Conventional stereo builds 3D cost volumes and processes them with 3D convolutions, creating computational costs that limit operating resolution.Specialized methods are needed for megapixel images such as those in Middlebury.
- Method: RAFT-Stereo replaces RAFT’s all-pairs 4D correlation with a same-height 3D volume and introduces multi-level GRUs operating across resolutions.The multi-level units maintain hidden states at multiple resolutions while producing a single high-resolution disparity update.
- Method: Using only 2D convolutions and a lightweight matrix-multiplication cost volume avoids the high computation and memory cost of 3D convolutions.The design can process megapixel images without resizing or patches and allows early stopping to trade accuracy for efficiency.
- Results: RAFT-Stereo shows stronger cross-dataset generalization than existing neural networks and performs well on real KITTI, ETH3D, and Middlebury datasets when trained only on synthetic data.It ranks first on Middlebury and outperforms all published work on ETH3D in the reported comparisons.
2. Related Work
Related stereo methods evolved from hand-designed matching and filtering pipelines toward end-to-end neural networks. Although 3D cost-volume architectures improved benchmark performance, their computational cost and limited generalization motivated lighter and more adaptable alternatives.
- Classical methods: Traditional stereo pipelines computed pairwise patch-matching costs and then applied optimization or filtering with priors to recover consistent disparity maps.Early work emphasized better matching costs and efficient inference algorithms.
- Deep stereo: Deep learning first improved stereo matching costs, while later work introduced end-to-end trainable networks and synthetic data for training.These developments extended the pipeline from learned matching components toward fully trainable stereo systems.
- 3D architectures: Many end-to-end stereo methods construct dense feature maps, build 3D cost volumes, filter them with 3D convolutions, and predict pointwise depth.Cost volumes may be formed through feature concatenation or correlation.
- Limitations: 3D-convolutional stereo networks achieve strong performance on datasets such as KITTI and FlyingThings3D but incur high computational cost and often generalize poorly beyond their training domain.This limits their use on datasets without ground-truth training data.
- Lightweight alternatives: Other approaches replace costly 3D components with lightweight modules, including refinement stages, depth-classification stages, and tile-based geometric guidance.These methods target efficiency while retaining stereo-specific structure.
3. Approach
RAFT-Stereo combines a correlation-based stereo pipeline with a multi-resolution GRU update operator that iteratively refines disparity. Its pyramid lookup and cross-connected hidden states improve information propagation, while a slow-fast update schedule enables substantially faster inference.
- 3.1. Architecture: RAFT-Stereo extracts features, builds a correlation pyramid, and iteratively updates a disparity field with a GRU-based operator.The update operator retrieves correlation features using the current disparity estimate and combines them with disparity and context features.
- 3.2. Correlation Pyramid: The 3D correlation volume restricts matching to feature vectors at the same image height, reflecting rectified stereo geometry.Its computation uses feature-vector similarity and can be implemented with a single matrix multiplication.
- 3.2. Correlation Pyramid: The four-level correlation pyramid pools only the disparity dimension, increasing receptive fields while retaining fine image-resolution structure.Lookup uses the current disparity and integer offsets, with linear interpolation for real-valued grid locations.
- 3.3. Multi-Level Update Operator: A multi-resolution update operator maintains cross-connected hidden states at 1/8, 1/16, and 1/32 resolutions to propagate information more effectively.The highest-resolution GRU performs correlation lookup and produces the final disparity update.
- 3.4. Slow-Fast GRU: The slow-fast schedule reduces runtime from 0.132s to 0.05s on KITTI-resolution images with 32 GRU updates, a 52% decrease.It updates the 1/16 and 1/32 hidden states several times for each 1/8-resolution update.
- 3.5. Supervision: RAFT-Stereo trains with exponentially increasing weights on the L1 disparity error across the full sequence of predictions.The supervision compares each predicted disparity with ground truth over all iterations.
4. Experiments
RAFT-Stereo achieves strong cross-dataset stereo performance, including state-of-the-art zero-shot synthetic-to-real generalization, benchmark-leading results, and a real-time configuration. Experiments also examine memory efficiency and how additional synthetic datasets affect generalization.
- Zero-Shot Generalization: State-of-the-art zero-shot performance was achieved across KITTI-2015, Middlebury, and ETH3D validation datasets after SceneFlow training.The evaluation averages results across six independent training runs after 200k steps, using thresholds of 3px, 2px, and 1px for KITTI, Middlebury, and ETH3D, respectively.
- KITTI: Second place was achieved on the KITTI-2015 leaderboard for erroneous foreground pixels among published methods.The submission was fine-tuned for 5k iterations on KITTI training data using random crops and augmentation.
- Middlebury: 4.74% bad 2px error ranked RAFT-Stereo first on the Middlebury test leaderboard, reducing error 26% versus the next best end-to-end deep learning method.Middlebury fine-tuning used 23 training images and 4,000 steps after SceneFlow pretraining.
- ETH3D: First place across every evaluation metric was achieved on the ETH3D test leaderboard among published methods.ETH3D evaluation used a SceneFlow-trained model with additional grayscale and gamma-adjusted synthetic images to simulate overexposed black-and-white imagery.
- Synthetic Datasets: Combining additional synthetic datasets improved generalization to specific validation datasets, including KITTI, ETH3D, and Middlebury.Falling Things improved KITTI and Middlebury generalization, Tartan Air generalized well to KITTI and ETH3D, and Sintel-Stereo helped ETH3D when combined with other datasets.
- Real-time Inference: 26 FPS with 5.91 D1 error enabled competitive real-time inference on 1248x384 KITTI-resolution images.The real-time configuration used Slow-Fast bi-level GRUs and a single backbone; the reported comparison was DSMNet at 6.5 D1 error.
5. Conclusions
RAFT-Stereo is a RAFT-based deep architecture for two-view stereo that uses multi-level GRUs to pass information efficiently across images. It achieves state-of-the-art cross-dataset generalization, ranks first on Middlebury, and outperforms published work on ETH3D.
- Conclusions: RAFT-Stereo extends RAFT for two-view stereo by using multi-level GRUs to efficiently pass information across the image.The conclusion identifies this architecture as the paper’s central proposal.
- Conclusions: State-of-the-art cross-dataset generalization was achieved, with first-place Middlebury performance and better results than published work on ETH3D.These conclusions summarize the reported benchmark outcomes.