PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching

Bowen Wen, Shaurya Dewan, Stan Birchfield

arXiv:2512.11130v2cs.CVcs.RO

TL;DR

Stereo foundation models generalize strongly but are too computationally expensive for real-time use, while efficient methods often sacrifice robustness and require per-domain fine-tuning. Fast-FoundationStereo combines distillation, blockwise architecture search, structured pruning, and pseudo-labeling to accelerate the foundation-model pipeline. It bridges zero-shot generalization and real-time performance, achieving a reported state-of-the-art among real-time methods while competing favorably with computationally intensive generalizable models.

  • Problem

    Stereo foundation models offer strong zero-shot generalization but are computationally prohibitive, whereas efficient architectures sacrifice robustness and rely on costly per-domain fine-tuning.

  • Method

    Fast-FoundationStereo uses divide-and-conquer acceleration with knowledge distillation, blockwise neural architecture search, structured pruning, and automatic pseudo-labeling of in-the-wild data.

  • Results

    Fast-FoundationStereo establishes a new state-of-the-art among real-time methods and competes favorably with computationally intensive generalizable models.

  • Takeaways & Limitations

    Fast-FoundationStereo bridges zero-shot generalization and real-time performance for stereo matching.

  • Takeaways & Limitations

    Performance on translucent surfaces remains challenging because the method inherits limitations from its FoundationStereo teacher.

Abstract

from arXiv · show

Stereo foundation models achieve strong zero-shot generalization but remain computationally prohibitive for real-time applications. Efficient stereo architectures, on the other hand, sacrifice robustness for speed and require costly per-domain fine-tuning. To bridge this gap, we present Fast-FoundationStereo, a family of architectures that achieve, for the first time, strong zero-shot generalization at real-time frame rate. We employ a divide-and-conquer acceleration strategy with three components: (1) knowledge distillation to compress the hybrid backbone into a single efficient student; (2) blockwise neural architecture search for automatically discovering optimal cost filtering designs under latency budgets, reducing search complexity exponentially; and (3) structured pruning for eliminating redundancy in the iterative refinement module. Furthermore, we introduce an automatic pseudo-labeling pipeline used to curate 1.4M in-the-wild stereo pairs to supplement synthetic training data and facilitate knowledge distillation. The resulting model can run over 10x faster than FoundationStereo while closely matching its zero-shot accuracy, thus establishing a new state-of-the-art among real-time methods. Project page: https://nvlabs.github.io/Fast-FoundationStereo/

1. Introduction

Stereo research has split between computationally intensive foundation models with strong zero-shot generalization and efficient methods that struggle to generalize. Fast-FoundationStereo addresses this gap with a divide-and-conquer acceleration strategy combining distillation, blockwise search, structured pruning, and pseudo-labeled real data.

  • Motivation and approach: Foundation stereo models use computationally intensive priors and architectures, which have hindered deployment in latency-bound systems.The introduction identifies rich monocular or vision-foundation-model priors and self-attention for long-range context as sources of computational cost.
  • Motivation and approach: Efficient stereo methods achieve high frame rates but often rely on per-domain fine-tuning and therefore struggle to generalize as off-the-shelf systems.The difficulty of obtaining dense, high-quality ground truth at scale further limits their use in in-the-wild embodied-agent settings.
  • Motivation and approach: The method accelerates feature extraction, cost filtering, and disparity refinement through distillation, blockwise architecture search, and structured pruning.This divide-and-conquer design builds on FoundationStereo while assigning different acceleration mechanisms to the major components.
  • Motivation and approach: Fast-FoundationStereo targets both strong zero-shot generalization and real-time stereo inference, offering varying accuracy-speed trade-offs.The model family is presented as a response to the gap between robust but slow foundation models and fast but less generalizable methods.
  • Motivation and approach: An automatic pseudo-labeling pipeline supplements synthetic data with diverse in-the-wild stereo images for knowledge distillation.The pipeline is introduced to harness the diversity, abundance, and realism of internet-scale stereo imagery.

2. Related Work

Fast-FoundationStereo accelerates foundation-model stereo matching through divide-and-conquer designs for feature extraction, cost filtering, and disparity refinement, while retaining the teacher’s learned priors.

  • 2. Related Work: FoundationStereo combines monocular and stereo priors but is computationally intensive, whereas efficient stereo methods generally require per-domain fine-tuning for generalization.
  • 3.1. Distilling Hybrid Monocular and Stereo Priors: Fast-FoundationStereo replaces the teacher’s dual backbone modules with a single student trained by knowledge distillation from hybrid monocular and stereo features.
  • 3.3. Refinement Pruning: Structured pruning models recurrent ConvGRU dependencies and jointly preserves constrained channel relationships before retraining the refinement module.The pruning constraints include fixed disparity and upsampling outputs, recurrent hidden-state dependencies, and fixed motion-encoder inputs.
  • 3.3. Refinement Pruning: The training objective supervises iteratively refined disparities while distilling per-layer latent features, using γ = 0.9 for iteration weighting and λ = 0.1 for distillation.

4. Experiments

Fast-FoundationStereo is evaluated for zero-shot generalization, robustness, component design, pseudo-labeling, and runtime. It substantially outperforms real-time baselines while approaching computationally expensive models and achieving over 10× runtime acceleration over FoundationStereo.

  • 4.3. Zero-Shot Generalization Comparison: Fast-FoundationStereo outperforms other real-time models across public zero-shot benchmarks and achieves comparable or better results than most computationally expensive models.Table 1 groups methods by real-time feasibility; the paper reports that the model is also second-best in each column.
  • 4.3. Zero-Shot Generalization Comparison: Zero-shot qualitative evaluations span Middlebury, ETH3D, Booster, and KITTI-2015 without training on target-dataset splits.The visual comparisons include textureless, transparent, specular, illumination, depth-range, viewpoint, and indoor/outdoor challenges.
  • 4.5. Runtime Analysis: More than 10× runtime acceleration is achieved over FoundationStereo with only a modest increase in error.The runtime decomposition attributes the gain to accelerating all three essential processing steps.
  • 4.4. Framework Analysis: Blockwise architecture search consistently finds better cost-filtering candidates than random assembly under varying latency constraints.Relaxing the latency constraint improves searched candidates, while tighter constraints can severely degrade randomly assembled candidates.
  • 4.4. Framework Analysis: Retraining recovers accuracy after aggressive refinement-module pruning, indicating substantial redundancy in the original module.The pruning-ratio study evaluates accuracy on Middlebury-Q and runtime under one refinement iteration.
  • 4.4. Framework Analysis: Pseudo-labeling consistently improves generalization across competitive real-time methods, especially those previously trained only on SceneFlow.The ablation compares in-the-wild results with and without pseudo-labeled data.

5. Conclusion

The conclusion presents Fast-FoundationStereo as a bridge between zero-shot generalization and real-time performance. Its divide-and-conquer strategy addresses foundation-stereo bottlenecks while retaining robustness and competitive accuracy.

  • 5. Conclusion: Fast-FoundationStereo establishes a substantial new state of the art among real-time methods while competing favorably with computationally intensive generalizable models.The authors identify quantization as a future avenue for deployment on more resource-constrained edge devices.
  • 5. Conclusion: The divide-and-conquer acceleration strategy systematically addresses foundation-stereo computational bottlenecks without sacrificing robustness.

6. Real-time Demo

The supplemental demonstration shows real-time GPU demos using the model evaluated in the main paper’s Table 1.

  • 6. Real-time Demo: Real-time demos run on an NVIDIA 3090 GPU using the model evaluated in Table 1.

7. More Details on Cost Filtering

The cost-filtering search space combines multiple 3D convolution, deconvolution, and transformer choices across eight blocks. Blockwise distillation makes this enormous design space tractable, while direct structured pruning provides only marginal speed gains.

  • 7. More Details on Cost Filtering: Candidate cost-filtering blocks vary 3D convolution channels, kernel and stride settings, optional normalization and activation, deconvolution choices, and transformer depth and width.The transformer encoder repeats from 1 to 6 times, with feedforward dimensions of 2× or 4× and 2 or 4 attention heads.
  • 7. More Details on Cost Filtering: Direct structured pruning of the cost-filtering module yields marginal speed gains compared with blockwise architecture search.

8. Effects of Refinement Iterations

Refinement iterations improve accuracy and runtime trade-offs differently under pruning ratios of 0.6 and 0.8.

  • 8. Effects of Refinement Iterations: Under pruning ratio 0.8, increasing refinement iterations yields only marginal accuracy improvements, indicating limited benefit from further refinement.The aggressively pruned module has less capacity to benefit from iterative refinement.
  • 8. Effects of Refinement Iterations: Under pruning ratio 0.6, accuracy saturates at around 8 refinement iterations.
  • 8. Effects of Refinement Iterations: Higher pruning ratios provide greater runtime savings as refinement iterations increase because their effects accumulate across iterations.At small iteration counts, runtime differences between pruning ratios are marginal and within millisecond-scale measurement noise.
  • 8. Effects of Refinement Iterations: The model evaluated in Table 1 uses a pruning ratio of 0.6.

9. More details on Model Efficiency

Efficiency analysis shows substantial reductions in model computation and runtime, with measurements profiled across NVIDIA GPUs.

  • 9. More details on Model Efficiency: The model achieves dramatic reductions in parameters, MACs, and runtime across different NVIDIA GPU hardware.
  • 9. More details on Model Efficiency: Runtime profiling is reported on different NVIDIA GPUs to characterize hardware-dependent efficiency.

10. Efficient GWC Volume Construction

The paper replaces the original loop-based group-wise correlation construction with an optimized PyTorch implementation designed to reduce disparity-wise overhead.

  • 10. Efficient GWC Volume Construction: The original GWC volume construction iterates over disparity levels, shifting feature maps and computing one normalized dot-product slice per iteration.This produces a 5-D volume V ∈ R^B×G×D×H×W.
  • 10. Efficient GWC Volume Construction: The disparity-level Python loop incurs repeated GPU kernel launches and prevents compiler fusion across disparity operations.
  • 10. Efficient GWC Volume Construction: The optimized PyTorch variant eliminates the explicit loop by using left-padding before constructing the correlation representation.

11. Model Parameter and Memory Usage

The model combines reduced parameterization with low peak memory usage and qualitative zero-shot performance across challenging out-of-domain scenes, while retaining limitations on translucent surfaces.

  • 11. Model Parameter and Memory Usage: Peak memory usage is 0.63GB on Middlebury-Q, fitting edge devices such as NVIDIA Jetson Orin or Thor for real-time deployment.
  • 11. Model Parameter and Memory Usage: Parameter-count comparisons are provided in Table 6, while efficiency analysis covers runtime across different NVIDIA GPUs.
  • 11. Model Parameter and Memory Usage: Qualitative zero-shot results cover out-of-domain images with textureless, translucent, specular, illumination, depth-range, perspective, indoor, and outdoor challenges.
  • 11. Model Parameter and Memory Usage: The pseudo-labeling process filters unsuitable noisy internet samples and produces 1.4M stereo pairs for training.
  • 11. Model Parameter and Memory Usage: Performance on translucent surfaces remains challenging because the method inherits limitations from its FoundationStereo teacher.The paper states that datasets enriched with relevant objects can mitigate this issue.
Loading 2512.11130v2…