DEFOM-Stereo: Depth Foundation Model Based Stereo Matching

Hualie Jiang; Zhiqiang Lou; Laiyan Ding; Rui Xu; Minglang Tan; Wenjie Jiang; Rui Huang

Source-linked AI summary

DEFOM-Stereo: Depth Foundation Model Based Stereo Matching

Hualie Jiang, Zhiqiang Lou, Laiyan Ding, Rui Xu, Minglang Tan, Wenjie Jiang, Rui Huang

arXiv:2501.09466v3Jan 16, 2025cs.CV

TL;DR

Stereo matching needs robust metric disparity estimation despite occlusion, non-texture, and difficult domain transfer. DEFOM-Stereo integrates Depth Anything V2 features and depth into recurrent stereo matching, using combined encoders, depth initialization, and scale updates. It delivers stronger zero-shot generalization and top benchmark performance across KITTI, Middlebury, and ETH3D.

Problem
Stereo matching remains challenged by occlusion, non-texture, blur, and limited zero-shot generalization, while monocular foundation-model depth lacks metric scale and shows scale inconsistency.
Method
DEFOM-Stereo combines CNN and DEFOM features, initializes recurrent disparity with DEFOM depth, and recurrently corrects scale using a scale update module.
Results
DEFOM-Stereo shows stronger zero-shot generalization, reduces Middlebury 2-pixel error by about one-third, and ranks 1st on many metrics across four leaderboards.
Takeaways & Limitations
Integrating monocular depth-foundation-model cues improves the robustness and cross-dataset performance of recurrent stereo matching.

Abstract

from arXiv · show

Stereo matching is a key technique for metric depth estimation in computer vision and robotics. Real-world challenges like occlusion and non-texture hinder accurate disparity estimation from binocular matching cues. Recently, monocular relative depth estimation has shown remarkable generalization using vision foundation models. Thus, to facilitate robust stereo matching with monocular depth cues, we incorporate a robust monocular relative depth model into the recurrent stereo-matching framework, building a new framework for depth foundation model-based stereo-matching, DEFOM-Stereo. In the feature extraction stage, we construct the combined context and matching feature encoder by integrating features from conventional CNNs and DEFOM. In the update stage, we use the depth predicted by DEFOM to initialize the recurrent disparity and introduce a scale update module to refine the disparity at the correct scale. DEFOM-Stereo is verified to have much stronger zero-shot generalization compared with SOTA methods. Moreover, DEFOM-Stereo achieves top performance on the KITTI 2012, KITTI 2015, Middlebury, and ETH3D benchmarks, ranking $1^{st}$ on many metrics. In the joint evaluation under the robust vision challenge, our model simultaneously outperforms previous models on the individual benchmarks, further demonstrating its outstanding capabilities.

1. Introduction

DEFOM-Stereo integrates monocular depth-foundation-model cues into recurrent stereo matching to address robustness and scale challenges. It combines CNN and DEFOM features, initializes disparity from predicted depth, and uses recurrent scale recovery to improve zero-shot performance.

Stereo matching still faces generalization and robustness challenges in occlusion, non-texture, blur, and high-resolution settings.
Monocular relative-depth models offer strong zero-shot generalization, but DEFOM predictions lack metric depth and exhibit scale inconsistency against disparity.
DEFOM-Stereo fuses DEFOM and CNN features in combined matching and context encoders for recurrent stereo matching.
The scale update module recurrently recovers dense disparity scales from DEFOM-initialized disparity using scale lookup over the correlation volume.
About one-third lower 2-pixel thresholding error on Middlebury was achieved in zero-shot evaluation, while the model ranked 1st on many metrics across four benchmarks.

2. Related Work

Related work covers recurrent, transformer, and domain-generalization approaches to stereo matching, alongside foundation models for relative depth. DEFOM-Stereo extends recurrent stereo matching by directly integrating foundation-model features and depth.

Learning-Based Stereo Matching: Recurrent stereo methods evolved from RAFT and refine disparity iteratively using multi-level recurrent updates and correlation structures.
Zero-shot Generalized Stereo Matching: Zero-shot stereo generalization is especially difficult for synthetic-to-real transfer because labeled real-world data are hard to obtain at scale.
Zero-shot Generalized Stereo Matching: Prior approaches improve transfer through domain-invariant features, information-theoretic constraints, or synthetic-image diversification.
Zero-shot Generalized Stereo Matching: DEFOM-Stereo differs from contemporaneous methods by replacing RAFT-Stereo’s zero disparity initialization with DEFOM depth and reforming recurrent updates.
Foundation Model for Depth Estimation: Depth Anything V2 uses DINOv2-based foundation-model representations and DPT prediction to produce accurate, efficient relative depth across scenes.

3. Methodology

DEFOM-Stereo integrates depth-foundation-model features and relative depth into RAFT-Stereo’s recurrent pipeline. It combines CNN and DEFOM representations, initializes disparity from monocular depth, and recurrently corrects scale using correlation-based updates.

Feature Extraction: DEFOM-Stereo uses two feature-extraction encoders: matching features from both images and multilevel context features from the left image.Matching features build the all-pair correlation volume, while context features initialize and condition the recurrent update operator.
Feature Extraction: A trainable DPT and convolutional alignment blocks fuse DEFOM ViT features with plain CNN features for matching and context representations.The combined feature map is formed by addition after channel alignment; the same strategy produces multiresolution combined context maps.
Correlation Construction and Lookup: The correlation pyramid repeatedly pools the last dimension, and pyramid lookup samples correlations around the current disparity and its neighbors.RAFT-Stereo’s standard lookup has a limited search range, whereas scale lookup is designed to address large-disparity scale recovery.
Delta Update: The recurrent delta update uses multilevel ConvGRUs whose finest level combines context, hidden-state, disparity-encoding, and retrieved-correlation features.The updated hidden state predicts a residual disparity correction and convex weights for full-resolution upsampling.
Monocular Depth Initialization and Scale Update: DEFOM relative depth initializes disparity after width-based scaling, while a recurrent scale-update module predicts dense scale maps to correct scale inconsistency.Scale lookup samples correlations at disparities multiplied by predefined scale factors, supplying scale-aware evidence to the update process.

4. Experiments

Experiments evaluate DEFOM-Stereo through synthetic-to-real generalization, ablations, efficiency measurements, and official benchmark comparisons. The model improves transfer across realistic datasets and achieves strong leaderboard results, while adding parameters and modest inference-time overhead.

Zero-shot generalization: DEFOM-Stereo improves zero-shot transfer from Scene Flow across KITTI, ETH3D, and Middlebury, with ViT-L reducing error rates by about 13%, 28%, and over a half versus RAFT-Stereo.On Middlebury-full, it reduces error by over 29% versus DLNR despite low-resolution pre-training.
Ablation study: Combined context or feature encoders provide about 10% improvement on Scene Flow, with clearer gains on KITTI 2012, Middlebury, and ETH3D than KITTI 2015.Combining both encoders adds little beyond the individual components.
Ablation study: Scale update with depth initialization yields around 10% improvement on Scene Flow and reduces Middlebury Bad 2.0 by over 50%, while improving generalization across four realistic datasets.The scale update addresses the scale inconsistency of DEFOM depth during disparity recovery.
Ablation study: The complete model further improves Scene Flow and Middlebury, but can slightly underperform individual component combinations on some datasets; ViT-L improves performance further.For example, ViT-S is slightly below depth initialization plus scale update on KITTI 2015.
Efficiency: The full ViT-S model increases trainable parameters by 67% and inference time by 15%, while the proposed components individually add about 10% inference time.Most inference time is spent on recurrent update iterations.
Benchmark comparisons: DEFOM-Stereo ranks first on many KITTI leaderboard metrics and ranks first in 7 of 10 Middlebury accuracy metrics when evaluated on all valid pixels.It also outperforms Selective-IGEV by 16.9% on Middlebury Bad 2.0 (all).

5. Conclusion

DEFOM-Stereo integrates Depth Anything V2 into recurrent stereo matching, combining foundation-model representations with CNN features and iterative scale correction. It achieves effectiveness in both in-domain fitting and zero-shot generation, with leading standard-benchmark performance.

DEFOM-Stereo integrates Depth Anything V2 into a recurrent stereo-matching framework.
The framework uses pre-trained foundation-model features for matching and context extraction alongside simple techniques for recurrent stereo matching.
Scale inconsistency remains across image regions in Depth Anything V2 estimates, particularly on synthetic stereo datasets.
A scale update module iteratively recovers dense scale maps to use depth estimates for recurrent disparity refinement.
DEFOM-Stereo shows significant effectiveness on in-domain fitting and zero-shot generation and achieves leading performance on standard stereo benchmarks.

A. Evaluation of Depth Anything V2

Depth Anything V2 predicts affine disparity with unknown scale and shift, so its stereo usefulness depends on alignment and within-image scale consistency. Evaluation shows larger errors and inconsistency on synthetic datasets than on realistic datasets, motivating scale correction.

Depth Anything V2 predicts affine disparity with unknown scale and shift because it is trained with scale-and-shift-invariant loss.
Least-squares alignment estimates scale and shift before computing the aligned disparity map.
Scale consistency is assessed from the standard deviation of ratios between ground-truth disparity and aligned disparity; values near 1 indicate consistency.
The evaluation covers KITTI 2015, Middlebury half resolution, ETH3D, Scene Flow, and CREStereo, using complete realistic trainsets or 200 random synthetic samples.
Synthetic Scene Flow and CREStereo produce especially large disparity errors and high ratio-map standard deviations, indicating serious within-image scale inconsistency.
Depth Anything V2 performs better on realistic datasets, while Middlebury-half error is attributed mainly to its high resolution.
Visualizations show the strongest scale inconsistency on Scene Flow, with lesser inconsistency also visible in real datasets.

B.1. Combined Encoders

The combined encoders benefit from retaining CNN features and using a new trainable DPT head. Removing CNNs substantially degrades performance, while the trainable head is generally better except on ETH3D.

The ablation compares CNN-plus-DPT encoders, DPT-only encoders, and fixed versus newly trainable DPT heads.
Removing CNNs increases Scene Flow EPE by over 35% and Middlebury-half Bad 2.0 by over 35%, across in-domain and zero-shot evaluations.
CNN features remain necessary for the proposed combined encoders.
Using a fixed DPT increases errors by about 3–5% on Scene Flow and KITTI 15 and by 23% on Middlebury-half, but reduces ETH3D error by 13%.
The authors hypothesize that fixed DPT features favor disparity ranges below 64, whereas a trainable DPT better supports larger disparities; the trainable DPT is retained overall.

B.2. Iterations of Scale Update

Scale-update iteration count has a non-monotonic effect on performance. One iteration already reduces error substantially, performance improves through eight iterations, and excessive iterations degrade results.

Increasing scale-update iterations from 0 to 1 reduces error by over 12%.
EPE continues decreasing until eight scale-update iterations, with similar performance from seven through nine iterations.
Performance drops noticeably when the number of scale-update iterations exceeds nine.
Visual comparisons evaluate DEFOM-Stereo against RAFT-Stereo, Mocha-Stereo, and Selective-IGEV across four stereo datasets, using Scene Flow-only pre-training.

D. Qualitative Comparison of RVC Models

The robust vision challenge evaluation compares DEFOM-Stereo with the strongest prior model on each individual benchmark. The model demonstrates more accurate results simultaneously across KITTI 2015, Middlebury, and ETH3D.

DEFOM-Stereo is compared with UCFNet RVC on KITTI 2015, CREStereo++ RVC on Middlebury, and LoS RVC on ETH3D.
The authors report that their model produces more accurate results simultaneously across the three benchmark comparisons.
Table 9 reports zero-shot Bad 2.0 evaluation across different areas of Middlebury-half.

E. Evaluation on Ill-Pose Regions

The paper evaluates stereo performance in ill-posed regions and on challenging reflective or transparent surfaces. It reports improvements in occluded and textureless areas, while large mirrors remain a failure case.

The Middlebury evaluation targets occluded and textureless regions, using SSIM to extract textureless areas and counting the proportions of different regions.Figure 10 visualizes examples from this evaluation.
The authors report obvious improvement in ill-posed areas on Middlebury.
The qualitative comparisons include realistic stereo datasets, Flickr1024, and robust vision challenge models across KITTI 2015, Middlebury, and ETH3D.
On Booster, DEFOM-Stereo works on reflective and transparent materials when the ill-posed factors are not too serious.
Large mirrors remain a failure case for DEFOM-Stereo, while Depth Anything V2 with a ViT-S backbone also slightly fails.
Figure 10 separates non-occluded textured, non-occluded textureless, occluded textured, and occluded textureless regions with corresponding error maps.