Source-linked AI summary
NBS: No Bias Stereo
Vage Taamazyan, Zhuowen Shen, Stefan Hinterstoisser, Alberto Dall'Olio, Agastya Kalra, Aarrushi Shandilya, Xin Li, Wenping Wang, Kartik Venkataraman
TL;DR
Stereo reconstruction commonly depends on specialized inductive biases, but their necessity for accuracy and efficiency remains challenged. NBS trains a unified Vision Transformer on massive synthetic data and reports competitive accuracy, efficient inference, and fusion of stereo and monocular cues without explicit geometric modules.
Problem
State-of-the-art stereo methods rely on strong architectural inductive biases, leaving open whether general-purpose architectures can achieve high accuracy and efficiency without them.
Method
NBS uses an end-to-end Vision Transformer with local and global attention, trained on massive synthetic data to combine inter-image matching with intra-image monocular semantics.
Results
The model ranks first on ETH3D, performs competitively across other benchmarks, and operates several times faster than top-performing methods at 1K and 2K resolutions.
Takeaways & Limitations
The results support stereo reconstruction without explicit cost volumes, 3D convolutions, or iterative refinement, while the model learns to fuse geometric matching with monocular priors.
Takeaways & Limitations
The method assumes positive disparities, requires resizing or padding to fixed spatial resolutions, and faces a quadratic self-attention memory bottleneck at larger scales.
Abstract
from arXiv · showhide
Stereo reconstruction is one of the last remaining Computer Vision tasks where all state-of-the-art methods employ a heavy architectural inductive bias. Even though it has been demonstrated that the task can be solved using general-purpose methods, it is widely believed that inductive biases in stereo are strictly necessary for both high-quality results and computational efficiency. We challenge this paradigm. In this paper, we demonstrate that both state-of-the-art accuracy and superior runtime efficiency are achievable with a model completely devoid of architectural inductive biases, relying instead on a simple, end-to-end Vision Transformer. By training on massive synthetic datasets, we show that pure data-driven learning can surpass explicitly engineered geometry. This work proves that explicit inductive biases are no longer a prerequisite for stereo matching, ultimately unlocking true scaling laws for continuous improvement in 3D reconstruction.
1 Introduction
Stereo reconstruction has increasingly relied on specialized architectural components and strong inductive biases. NBS proposes a unified Vision Transformer trained on massive synthetic data, reporting competitive accuracy, efficient inference, and broad generalization without explicit geometric modules.
- Modern stereo systems combine deep features with explicit 3D or 4D cost volumes, 3D convolutions, and iterative refinement, increasing architectural complexity.
- All methods in the top 30 of ETH3D or Middlebury rely on strong architectural inductive biases, separating stereo from general-purpose machine learning.
- NBS uses a single end-to-end Vision Transformer trained on massive synthetic corpora to jointly attend to inter-image matching and intra-image semantics.
- The model ranks first on ETH3D and achieves highly competitive performance across other evaluated benchmarks.
- The architecture omits explicit cost volumes, 3D convolutions, and iterative refinement while relying on a unified Vision Transformer.
- The paper reports emergent fusion of geometric matching and monocular priors, with robust zero-shot generalization and plausible depth under total camera occlusion.
2 Related Work
Stereo research progressed from local and semi-global matching to deep cost volumes, recurrent refinement, and transformer-based global reasoning. Recent work increasingly incorporates foundation-model representations and monocular depth priors, while broader vision moves toward unified architectures.
- Deep stereo replaced hand-crafted features with dense correlation volumes, followed by compute- and memory-intensive 3D CNN filtering.
- RAFT-Stereo reduced reliance on heavy 3D convolutions through GRU-based iterative refinement over a 2D correlation field.
- Transformer methods such as STTR, CSTR, and Uni-Match replaced explicit cost volumes with self- and cross-attention mechanisms.
- Recent stereo systems inject internet-scale monocular depth priors to address the sim-to-real gap, despite their use of global attention.
- The broader vision field has developed unified foundation-model architectures and task-agnostic representations for dense spatial and monocular geometry prediction.
3 Method
NBS uses a standard ViT encoder with local and global attention, a DPT decoder, high-resolution adaptation, diverse synthetic training data, and a multi-stage objective. The method predicts normalized disparity and combines masked L1 accuracy with multi-scale gradient matching.
- 3.1 Task: The task reconstructs a disparity map from rectified image pairs under the assumption that disparities are strictly positive.
- 3.2 Model Architecture and High-Resolution Inference: The architecture combines a standard ViT-Large encoder with local and global self-attention and a DPT decoder for fine-grained disparity prediction.
- 3.2 Model Architecture and High-Resolution Inference: The model uses higher-resolution positional embeddings and fine-tuning for fixed resolutions, including 966 × 546 and 1932 × 1330 inference settings.
- 3.2 Model Architecture and High-Resolution Inference: The decoder outputs normalized disparities between 0 and 1, which are multiplied by image width to obtain pixel disparities.
- 3.3 Training Data and Pipeline: Training uses a multi-stage pipeline and approximately 2.4 million internally generated synthetic scenes combined with diverse public datasets.
- 3.4.4 Training Objective: The objective combines a masked L1 loss for global disparity accuracy with a multi-scale gradient matching term for smoother gradients and sharper discontinuities.
4 Results and Experiments
The evaluation spans real-world, synthetic, and industrial benchmarks, testing accuracy, efficiency, monocular depth behavior, and positional-embedding resolution. The model reports state-of-the-art stereo performance across datasets, faster inference with lower memory, competitive monocular depth, and gains from denser positional embeddings.
- Benchmark Evaluation: The evaluation uses ETH3D, out-of-distribution SimpleProc data, and the repurposed XYZ-IBD industrial dataset to test generalizability.SimpleProc includes 661 scenes at 966 × 546 pixels and 530 scenes at 1932 × 1092 pixels; scenes were generated without manual filtering.
- Benchmark Evaluation: Qualitative reconstructions preserve fine details including container holes, cluttered-object shapes, and thin grill structures.Figure 3 compares these details against other methods across four rows.
- Quantitative Results: The model achieves state-of-the-art performance across all four evaluation datasets, outperforming baselines across SimpleProc bad@1.0 through bad@4.0.On SimpleProc, it reduces the severe error rate of the next-best model, S2M2, by nearly 50%.
- Efficiency: The architecture achieves state-of-the-art accuracy while reducing computational overhead, inference latency, and memory usage.With Flash-Attention and float-16 precision, inference is 5.4× faster and memory usage is reduced 2×, with marginal accuracy impact; Table 2 reports 4x faster runtime and 2.8x lower peak memory on SimpleProc-S versus FS and S2M2.
- Emergent Monocular Depth Capabilities: Under total right-camera occlusion, the model remains competitive for monocular depth while retaining state-of-the-art stereo depth performance.It ranks among the top 2 models for Absrel and δ1 across multiple datasets and produces physically plausible disparity maps.
- Ablations: Increasing the positional-embedding grid from 37 × 37 to 148 × 148 reduces bad@0.5 by 7.21 points at 966 × 546 and by 3.83 points at 280 × 518.The denser grid supports refined high-frequency spatial variations and improves precision in highly textured regions.
5 Limitations
The model has limitations involving disparity sign, fixed inference resolutions, and quadratic memory growth at extreme resolutions.
- The model predicts only positive disparities, so it cannot accurately resolve negative disparities that may occur in real-world deployments.
- Fixed-resolution fine-tuning requires input images to be resized or padded before inference, limiting arbitrary image-size and aspect-ratio flexibility.
- The standard ViT backbone creates a severe quadratic memory bottleneck during training at extreme resolutions.
- The fixed spatial-dimension requirement prevents native-scale evaluation on ultra-high-resolution datasets such as Middlebury and Booster.
6 Conclusion
The paper argues that explicit architectural inductive biases are not strictly necessary for state-of-the-art stereo reconstruction. A scaled Vision Transformer can learn both geometric matching and monocular priors through data-driven learning.
- The paper challenges the necessity of 3D convolutions, cost volumes, and iterative refinement for state-of-the-art stereo reconstruction.
- Scaling a simple, end-to-end Vision Transformer on massive synthetic data enables pure data-driven learning to match and surpass explicitly engineered geometry.
- The model achieves highly competitive accuracy across standard benchmarks without articulated stereo sub-modules.
- The model naturally fuses inter-image matching with intra-image monocular priors.
A.1 Full Evaluation Results
Across XYZ-IBD, SimpleProc-S, SimpleProc-M, and ETH3D, the method shows state-of-the-art or highly competitive performance across synthetic and real-world settings.
- The quantitative comparisons cover XYZ-IBD, SimpleProc-S, SimpleProc-M, and ETH3D using detailed benchmark tables.
- The method demonstrates state-of-the-art or highly competitive performance across all four benchmarks and across synthetic, real-world, indoor, and outdoor settings.
- XYZ-IBD results include the best EPE, bad@2 of 36.64, and bad@4 of 23.39.
- At 1932 × 1092 on SimpleProc-M, the method achieves the lowest EPE of 0.40 and nearly halves bad@4 relative to the second-best model.
- On ETH3D non-occluded pixels, the method achieves EPE of 0.09 and bad@0.5 of 0.65, nearly half FoundationStereo’s error rate.
A.2 Generating SimpleProc Evaluation Dataset
The SimpleProc evaluation dataset is generated through modified procedural rendering at specified resolutions, with stereo-camera geometry and unique scene seeds supporting reproducible evaluation.
- The dataset-generation procedure requires modifying field-of-view and pixel-count parameters for the desired render dimensions without rounding image resolution.
- An anchor rig defines the right camera, while the left rectified camera is positioned using a uniformly sampled 5 to 45 cm baseline shift.
- Both camera rigs use identity parent inverses so local camera coordinates map 1:1 to their rigs.
- Each scene is rendered with a unique seed, and the seed lists are attached in supplementary material for each resolution.
- Figure 7 provides example images of the generated evaluation dataset.
A.3 Industrial Parts Evaluation Dataset
The industrial-parts evaluation uses stereo-rectified XYZ-IBD data, while the ablation evidence compares disparity structure with and without Gradient Matching loss.
- XYZ-IBD is postprocessed for stereo and illustrated with left images, right images, and ground-truth disparity arranged in rows.
- Gradient Matching loss is associated with sharper local structural results despite stable global metrics.
- The dataset contains calibration information for constructing 3D point clouds, while disparity values require division by 2 to recover true disparity.
A.4 Ablation on Gradient Matching Loss
The ablation evaluates Gradient Matching loss with a reduced ViT-Base model, while leaderboard evidence reports first-place ETH3D performance across multiple metrics.
- Gradient Matching loss significantly enhances local structural sharpness while leaving global metrics stable.
- The ablation uses a reduced ViT-Base model trained exclusively on FSD and evaluated on the ETH3D training split.
- The method achieved first place on the ETH3D leaderboard and consistently outperformed existing approaches across multiple metrics.
- The model ranks first on ETH3D for bad@0.5 and bad@1, evaluated on both non-occluded pixels and all pixels.