PaperlayerView exact revision on arXiv

Source-linked AI summary

FoundationStereo: Zero-Shot Stereo Matching

Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, Stan Birchfield

arXiv:2501.09898v4cs.CVcs.LGcs.RO

TL;DR

Stereo matching still relies on target-domain fine-tuning, limiting strong zero-shot generalization. FoundationStereo combines large-scale synthetic training with self-curation and architecture adaptations, achieving strong cross-domain zero-shot robustness without fine-tuning.

  • Problem

    Stereo matching performs strongly after per-domain fine-tuning, but strong zero-shot generalization comparable to other vision foundation models remains challenging.

  • Method

    FoundationStereo trains on 1M diverse, photorealistic synthetic stereo pairs with self-curation and adapts monocular foundation-model priors through side-tuning.

  • Results

    FoundationStereo achieves strong zero-shot generalization across various domains without fine-tuning, outperforming existing methods on in-the-wild data.

  • Takeaways & Limitations

    FoundationStereo provides an off-the-shelf stereo depth estimation model for broader practical adoption across domains.

  • Takeaways & Limitations

    Existing stereo methods remain limited by insufficient non-local context and small training datasets, with prior approaches falling short of target-domain fine-tuned performance.

Abstract

from arXiv · show

Tremendous progress has been made in deep stereo matching to excel on benchmark datasets through per-domain fine-tuning. However, achieving strong zero-shot generalization - a hallmark of foundation models in other computer vision tasks - remains challenging for stereo matching. We introduce FoundationStereo, a foundation model for stereo depth estimation designed to achieve strong zero-shot generalization. To this end, we first construct a large-scale (1M stereo pairs) synthetic training dataset featuring large diversity and high photorealism, followed by an automatic self-curation pipeline to remove ambiguous samples. We then design a number of network architecture components to enhance scalability, including a side-tuning feature backbone that adapts rich monocular priors from vision foundation models to mitigate the sim-to-real gap, and long-range context reasoning for effective cost volume filtering. Together, these components lead to strong robustness and accuracy across domains, establishing a new standard in zero-shot stereo depth estimation. Project page: https://nvlabs.github.io/FoundationStereo/

1. Introduction · 2. Related Work

FoundationStereo addresses stereo matching’s persistent zero-shot generalization gap with a large, high-fidelity synthetic dataset, self-curation, and scalable architecture components. Related work highlights cost-volume memory limits, iterative refinement, and the growing role of vision foundation models.

  • 1. Introduction: Despite near-saturated benchmarks, competitive stereo performance still typically requires fine-tuning on the target domain.The introduction contrasts this limitation with zero-shot generalization achieved in other computer vision problems through scaling laws.
  • 1. Introduction: Leading cost-volume and recurrent-refinement networks succeed after per-domain fine-tuning but struggle to gather non-local information and scale to larger datasets.Cost-volume methods use unary features and 3D CNN filtering, whereas refinement methods iteratively update disparity with recurrent modules such as GRUs.
  • 1. Introduction: Prior cross-domain methods remain less competitive than target-domain fine-tuning because of insufficient architecture, impoverished data, or both.These methods generally rely on Scene Flow, which contains only 40K annotated training image pairs, preventing an off-the-shelf solution.
  • 1. Introduction: FoundationStereo is proposed as a large stereo-depth foundation model achieving strong zero-shot generalization without per-domain fine-tuning.The model is trained on a 1M-image-pair synthetic dataset with high diversity and photorealism, alongside automatic self-curation for ambiguous samples.
  • 1. Introduction: The model’s scalable design includes a side-tuning feature backbone and long-range context reasoning, while its DT module applies full self-attention over disparity.These components are presented as part of the architecture for adapting monocular foundation-model priors and filtering cost volumes.
  • 2. Related Work: Cost-volume aggregation methods build volumes from unary features and filter them with 3D CNNs, but high memory consumption limits high-resolution application.Iterative refinement methods inspired by RAFT avoid costly 4D volume construction and filtering through recurrent updates.
  • 2. Related Work: FoundationStereo’s training dataset emphasizes the largest data volume, diverse 3D assets, randomized camera parameters, and high-fidelity rendering and spatial layouts.The related-work discussion positions these properties as distinguishing characteristics of its synthetic stereo data.
  • 2. Related Work: Vision foundation models have advanced across 2D, 3D, and multimodal alignment tasks, exemplified by CLIP’s large-scale image-text pair training.The passage introduces this broader foundation-model context within related work.

3. Approach

FoundationStereo combines monocular foundation-model priors with CNN features, hybrid cost-volume construction, and long-range filtering for robust stereo estimation. Its approach also includes iterative disparity refinement and a diverse, photorealistic synthetic training dataset targeting difficult stereo conditions.

  • Side-Tuning Adapter: STA adapts frozen DepthAnythingV2 monocular priors to stereo features while combining them with fine-grained CNN features to mitigate the sim-to-real gap.A CNN network adapts the ViT-based monocular depth estimator, synergizing CNN and ViT strengths; latent features provide geometric priors without scale ambiguity.
  • Hybrid Cost Volume Construction: The hybrid cost volume combines group-wise correlation with shifted-feature concatenation, preserving similarity measurements and monocular unary priors.The features are divided into G = 8 groups, while unary feature dimensions are reduced to 14 before concatenation to reduce memory consumption.
  • Axial-Planar Convolution Filtering: APC decouples 3D filtering into separate spatial and disparity convolutions, improving scalability for large disparities and high-resolution images.The spatial convolution uses kernel size Ks × Ks × 1, and the disparity convolution uses 1 × 1 × Kd, each followed by BatchNorm and ReLU.
  • Iterative Disparity Refinement: The model initializes disparity from the filtered cost volume and progressively refines it through iterative GRU updates to avoid local optima and accelerate convergence.Context features guide the ConvGRU hidden state and its input at each iteration with progressively refined contextual information.
  • Synthetic Training Dataset: FSD is a large-scale synthetic dataset designed around reflections, low-texture surfaces, severe occlusions, and broad camera, lighting, and object variation.It uses NVIDIA Omniverse, domain randomization, high-quality textured 3D assets, and path-tracing rendering.

4. Experiments

Experiments evaluate FoundationStereo across public synthetic and real-world stereo benchmarks, emphasizing zero-shot generalization, in-domain performance, and architectural ablations. The model consistently outperforms comparison methods, including when trained without the target domains, while fine-tuning achieves top leaderboard rankings.

  • Zero-shot generalization: FoundationStereo consistently outperforms comparison methods on four real-world datasets when trained solely on Scene Flow.The comparison uses fixed weights during evaluation and attributes the gains to adapting rich monocular priors from vision foundation models.
  • Benchmark evaluation: On Scene Flow, FoundationStereo reduces the previous best EPE from 0.41 to 0.33 under the official train/test split.Although in-domain training is not the paper’s focus, the result supports the effectiveness of the model design.
  • Benchmark evaluation: Fine-tuning on ETH3D reduces error rates by more than half and ranks FoundationStereo first on the leaderboard at submission time.The fine-tuned model also ranks first on the Middlebury leaderboard.
  • Ablation studies: Ablations compare vision foundation backbones for STA and show DepthAnythingV2 is more effective than DINOv2 for stereo matching.The study includes different model sizes of DepthAnythingV2 and DINOv2-Large; the full model does not deliberately limit size while pursuing generalization.
  • Ablation studies: Attention over the disparity dimension is more effective than full 4D cost-volume attention, providing sufficient context for better initial disparity prediction.The analysis attributes the weaker full-volume result to the extremely large and less tractable 4D cost volume.

5. Conclusion

FoundationStereo is presented as a foundation model for stereo depth estimation with strong zero-shot generalization across domains without fine-tuning. The authors also identify efficiency as a limitation, reporting 0.7 seconds per 375×1242 image on an NVIDIA A100 GPU.

  • Conclusion: FoundationStereo achieves strong zero-shot generalization across various domains without fine-tuning.It is introduced as a foundation model for stereo depth estimation.
  • Conclusion: The authors envision FoundationStereo facilitating broader adoption of stereo estimation models in practical applications.
  • Conclusion: 0.7s is required for a 375×1242 image on an NVIDIA A100 GPU, highlighting the model’s current efficiency limitation.The conclusion states that the model is not yet optimized for efficiency.

Supplementary Material · 6. ETH3D Leaderboard

At submission, the fine-tuned FoundationStereo model ranked first on the ETH3D leaderboard, outperforming published and unpublished works.

  • 6. ETH3D Leaderboard: The fine-tuned model ranked 1st on the ETH3D leaderboard at submission, outperforming both published and unpublished works.The result is shown in Fig. 6.

7. Middlebury Leaderboard

At submission, FoundationStereo’s fine-tuned model ranked first on the Middlebury leaderboard, significantly outperforming published and unpublished methods.

  • 7. Middlebury Leaderboard: At submission, the fine-tuned model ranked 1st on the Middlebury leaderboard and significantly outperformed published and unpublished works.The leaderboard result is illustrated in Fig. 8.

8. More Ablation Study on Synthetic Data

The ablation study finds that self-curation improves synthetic-data training by removing ambiguous samples, yielding better Middlebury performance at the same dataset size. The reported values are 1.15 with self-curation and 1.27 without it.

  • Effects of Self-Curation: Self-curation removes ambiguous synthetic samples that would otherwise confuse the learning process.Disabling self-curation while keeping the same data size causes a slight performance drop on Middlebury.
  • Effects of FSD for Other Methods: The section also reports an analysis of FSD’s effects on other methods.The provided passage points to Table 2 in the main paper and Table 9 for this analysis, without stating the associated results.

9. Results on Translucent Objects

FoundationStereo is evaluated zero-shot on Booster at half resolution, a challenging dataset containing specular and transparent objects, against competitive methods from the main paper’s Fig. 5.

  • Evaluation setup: The evaluation uses Booster at half resolution, whose challenging scenes contain specular and transparent objects.Quantitative and qualitative results are reported for comparison with competitive methods in the zero-shot setting.

10. More Results on Middlebury Dataset

FoundationStereo is evaluated zero-shot on Middlebury against competitive methods with public weights across varying resolutions. It significantly outperforms the baselines across all resolutions, with runtime and peak memory also reported on identical hardware.

  • Zero-shot comparison: FoundationStereo is compared zero-shot on Middlebury against competitive methods with publicly released weights.NMRF is rerun using its released weights because it did not report the evaluated Middlebury resolution.
  • Evaluation settings: At full resolution, FoundationStereo uses a maximum disparity of 320.
  • Results: FoundationStereo significantly outperforms the baselines across all Middlebury resolutions.Peak memory usage and running time are averaged across the dataset on the same hardware.

11. More Details of Synthetic Data Generation

Synthetic data generation combines high-fidelity Omniverse rendering with extensive asset, material, texture, camera, and scene-layout randomization. It mixes chaotic and realistic configurations to diversify views, disparities, appearances, and simulated environments.

  • Tooling and Assets: The dataset uses NVIDIA Omniverse RTX path tracing with 32 to 128 samples per pixel, generated across 48 NVIDIA A40 GPUs for 10 days.The asset library contains more than 5K objects collected from artist designs and 3D scanning.
  • Tooling and Assets: Appearance diversity is augmented with 12 large scene models, 16 skybox images, more than 150 materials, and 400 tiled-wrapping textures.Textures come from real-world photos and procedurally generated random patterns.
  • Camera Configuration: Each sample randomly varies the stereo baseline and camera focal length to diversify field-of-view and disparity distributions.Objects and cameras are also randomized through two alternative spawning procedures.
  • Layout Configuration: Layouts combine chaotic and realistic styles, using randomized flying-object scenes alongside structured indoor and outdoor environments.Chaotic scenes randomize lighting, textures, and materials, while realistic scenes preserve asset materials and apply more natural lighting.
  • Layout Configuration: Realistic scenes are divided into navigation, driving, and manipulation types that alter camera poses, object placement, and sampled object categories.These categories target navigation, driving, and interaction scenarios with distinct camera configurations and object selections.
Loading 2501.09898v4…