PaperlayerView exact revision on arXiv

Source-linked AI summary

Lite Any Stereo V2: Faster and Stronger Efficient Zero-Shot Stereo Matching

Junpeng Jing, Ronglai Zuo, Zhelun Shen, Shangchen Zhou, Rolandos Alexandros Potamias, Stefanos Zafeiriou, Krystian Mikolajczyk, Jiankang Deng

arXiv:2606.24457v1cs.CV

TL;DR

Efficient stereo models remain less established for strong zero-shot generalization and practical edge deployment than accuracy-oriented approaches. LAS2 addresses this with deployment-focused 2D aggregation and staged distillation, achieving state-of-the-art efficient-stereo accuracy at significantly lower latency.

  • Problem

    High-performing stereo matching often depends on computationally expensive models, while efficient methods have limited evidence for strong zero-shot generalization and edge-device deployment.

  • Method

    LAS2 combines a deployment-focused 2D-only cost aggregation architecture with three-stage training using synthetic supervision, self-distillation, and real-world knowledge distillation.

  • Results

    LAS2 achieves state-of-the-art accuracy among efficient stereo methods with significantly lower latency, while LAS2-M runs 1.6× and 1.9× faster than LAS on H200 and Orin.

  • Takeaways & Limitations

    LAS2 demonstrates that efficient stereo models can deliver strong zero-shot generalization across diverse real-world scenarios while remaining practical for GPU and edge-device deployment.

  • Takeaways & Limitations

    LAS2 remains behind prior-based high-accuracy methods, while limited diverse, high-quality real-world stereo data constrains efficient stereo models’ performance ceiling.

Abstract

from arXiv · show

Recent advances in stereo matching have achieved remarkable accuracy, but often rely on large models, heavy computation, or additional foundation-model priors, making them difficult to deploy on resource-constrained platforms. In contrast, efficient stereo models offer faster inference but are commonly considered less capable of strong zero-shot generalization. In this paper, we challenge this assumption by introducing Lite Any Stereo V2 (LAS2), an ultra-fast model series designed for efficient zero-shot stereo matching. LAS2 is developed from both architecture and training perspectives. Architecturally, we revisit efficient stereo design under practical deployment settings and propose a 2D-only cost aggregation framework, optimized for real inference latency rather than theoretical MACs alone. For training, we develop a three-stage strategy that combines synthetic supervision, self-distillation, and real-world knowledge distillation. To improve the reliability of real-world pseudo supervision, we further introduce pseudo-label filtering and an error-clamping operation, enabling smoother synthetic-to-real transfer. We instantiate LAS2 as a family of models, including feed-forward variants for different efficiency budgets and an iterative variant for higher accuracy. Extensive experiments show that LAS2 achieves state-of-the-art accuracy among efficient stereo methods while maintaining significantly lower latency. Specifically, LAS2-H achieves stronger overall zero-shot performance than the iterative method Fast-FoundationStereo, with 1.8x and 2.7x faster inference on H200 and Orin, respectively. The project page, demos, and code are available at https://tomtomtommi.github.io/LiteAnyStereoV2/.

I. INTRODUCTION

Recent stereo models improve accuracy through larger computation and foundation-model priors, while efficient methods often sacrifice zero-shot generalization and practical deployment. LAS2 addresses this gap with a deployment-oriented 2D architecture and multi-stage training, achieving strong zero-shot accuracy with lower latency.

  • Motivation: Large, computation-intensive stereo models have improved benchmark accuracy but remain difficult to deploy on resource-constrained platforms.
  • Motivation: Foundation-model-based stereo systems achieve strong zero-shot generalization, but their accuracy-driven designs remain difficult to deploy efficiently.
  • Problem: Efficient stereo approaches reduce inference cost but retain an accuracy gap, encouraging domain-specific fine-tuning instead of practical off-the-shelf use.
  • Contributions: LAS2 combines a purely 2D cost aggregation framework with a three-stage strategy using synthetic supervision, self-distillation, and real-world knowledge distillation.
  • Contributions: Pseudo-label filtering and error clamping improve real-world pseudo supervision and support smoother synthetic-to-real transfer.
  • Results: LAS2 provides S, M, L, and H variants for different deployment requirements, with stronger zero-shot performance and significantly lower latency than representative baselines.

II. RELATED WORK · A. Deep Stereo Methods

Deep stereo evolved from handcrafted pipelines to end-to-end networks centered on learned cost volumes. Existing methods include feed-forward, iterative, and transformer-based approaches, with iterative refinement generally targeting higher accuracy.

  • II. RELATED WORK: The related-work discussion covers deep stereo development, zero-shot generalization, and efficient approaches.
  • A. Deep Stereo Methods: Traditional stereo pipelines combine handcrafted matching costs, aggregation, disparity selection, and post-processing.
  • A. Deep Stereo Methods: Since DispNet, end-to-end deep stereo networks have become the dominant paradigm.
  • A. Deep Stereo Methods: Most deep stereo methods construct cost volumes to encode correspondences before neural regularization infers disparities.Representations may be 3D over height, width, and disparity, or 4D with an additional feature dimension.
  • A. Deep Stereo Methods: Feed-forward methods estimate disparity in one pass through cost aggregation and disparity regression.
  • A. Deep Stereo Methods: Iterative methods repeatedly update disparities using local cost-volume lookup and recurrent refinement, leading to better accuracy.
  • A. Deep Stereo Methods: Transformer-based methods form another line of work within deep stereo networks.The supplied passage identifies transformer architectures as a separate methodological direction.

B. Zero-Shot Stereo Methods … A. Feed-forward Framework: LAS2-S/M/L

The paper reviews zero-shot and efficient stereo methods before introducing LAS2, whose feed-forward S/M/L framework uses shared-weight features, correlation, 2D cost aggregation, and full-resolution disparity estimation. LAS2 avoids heavy monocular-depth priors and scales capacity through the aggregation module.

  • B. Zero-Shot Stereo Methods: Early zero-shot stereo methods improve cross-domain robustness through domain-invariant representations, including normalization, graph filtering, shortcut learning, contrastive learning, visual transformation, and masked image modeling.Other methods also address domain shifts through architectural or training-design choices.
  • C. Efficient Stereo Matching: Efficient stereo methods reduce computation through lower-resolution prediction or specialized aggregation, but domain-focused models often sacrifice fine detail or generalize poorly beyond KITTI.Later approaches seek better efficiency–accuracy trade-offs, while many still rely on 3D cost aggregation.
  • C. Efficient Stereo Matching: Recent real-time variants reduce latency by shrinking channels, reducing iterations, replacing heavy priors, or combining architecture search, pruning, and distillation.These methods retain or adapt components from high-accuracy stereo models to improve inference speed.
  • III. METHOD: LAS2 comprises feed-forward LAS2-S/M/L variants, an iterative LAS2-H variant, and a three-stage training strategy presented across Sections III-A, III-B, and III-C.The feed-forward variants are introduced before the iterative high-accuracy model and training strategy.
  • A. Feed-forward Framework: LAS2-S/M/L: The LAS2-S/M/L feed-forward pipeline contains four stages: feature extraction, correlation, cost aggregation, and disparity estimation.This organization is shown in the left part of Fig. 3.
  • A. Feed-forward Framework: LAS2-S/M/L: LAS2 uses an ImageNet-pretrained conventional backbone with two weight-sharing encoders, avoiding the computational overhead of even the smallest Depth Anything-S variant.The encoders extract multi-scale feature pyramids, whose features are upsampled to 1/4 resolution for matching.
  • A. Feed-forward Framework: LAS2-S/M/L: The model constructs a cost volume over [0, Dmax/4] from left–right feature correlations, then replaces LAS’s hybrid 3D-2D aggregation with a 2D-only design.LAS’s 3D component jointly aggregates disparity and spatial dimensions, whereas the proposed approach focuses aggregation in the image plane.
  • A. Feed-forward Framework: LAS2-S/M/L: A U-Net-style aggregator uses three resolution levels, FasterNet-style residual blocks, transposed-convolution upsampling, skip connections, and retained attention before disparity prediction.Soft-argmax regresses disparity at reduced resolution, and convex upsampling produces the full-resolution map.

B. Iterative Framework: LAS2-H

LAS2-H is an iterative stereo framework that uses LAS2-M’s 2D cost aggregation for initial disparity estimation, enabling efficient refinement and pretrained-weight reuse. It then performs compact recurrent disparity refinement using retained group-wise and all-pairs geometry volumes, with four iterations in practice.

  • Iterative architecture: LAS2-H converts LAS2-M’s standard correlation volume into a group-wise form while preserving the input format through group averaging.This transformation is equivalent to Eq. 1, allowing LAS2-H to retain the group-wise volume for iterative geometry lookup.
  • Iterative refinement: Starting from d0 = dinit, LAS2-H progressively refines disparity using local features from the retained group-wise cost volume Cg, all-pairs correlation volume Ca, and left-image context.The compact recurrent update fuses local geometry features around the current disparity with context features at each iteration.
  • Iterative refinement: 4 iterations are used in practice, with 64-channel hidden-state and context features and two-layer Encoderg and Encoderd modules.Each iteration applies local correlation lookup over multiple pyramid levels, convex-upsamples the disparity, and produces the final prediction as Dn.

C. Training Strategy

LAS2 uses a three-stage training strategy that progresses from supervised synthetic training to self-distillation and real-world adaptation. Pseudo-label filtering and error clamping improve the reliability and stability of the synthetic-to-real transition.

  • Stage ①: Synthetic supervision: LAS2 first trains from scratch on approximately 1.8M annotated synthetic stereo pairs to establish robust stereo matching capability.The selected datasets exclude several alternatives because of annotation-quality issues or significant domain gaps.
  • Stage ②: Self-distillation: In Stage ②, self-distillation trains a student on strongly perturbed inputs against a teacher receiving clean inputs, encouraging domain-invariant feature representations.The teacher and student share the same architecture and are initialized from Stage ①.
  • Stage ②: Self-distillation: The fixed-teacher self-distillation scheme achieves the best performance among the evaluated alternatives and is therefore adopted.The alternatives include fixed teacher weights, EMA teacher updates, and directly copying student weights to the teacher.
  • Stage ③: Real-world adaptation: Stage ③ adapts the lite model using 0.5M unlabeled real-world stereo pairs with dense pseudo labels generated by FoundationStereo.The real-world Weather subset is excluded from adaptation and reserved for evaluation.
  • Stage ③: Real-world adaptation: Pseudo-label filtering combines left-right consistency, edge-aware, and sky masks, while error clamping prevents a few high-error pixels from destabilizing adaptation.Only valid pixels supervise training, and clamping focuses optimization on the majority of reliable regions for a smoother synthetic-to-real transition.

IV. EXPERIMENTS · A. Benchmarks, Metrics, and Baselines

The experiments evaluate LAS2 on five real-world stereo datasets under specified benchmark settings and compare against retrained efficient methods and the fastest deployable Fast-FoundationStereo configuration. Consistent training data and metric definitions are used to reduce comparison discrepancies.

  • A. Benchmarks, Metrics, and Baselines: Five widely used real-world stereo datasets benchmark the method’s zero-shot performance.The datasets include Middlebury, ETH3D, and three additional real-world stereo benchmarks described in the paper.
  • A. Benchmarks, Metrics, and Baselines: 15 Middlebury stereo pairs provide indoor evaluations with structured-light ground-truth disparities.Results use the half-resolution and non-occluded evaluation settings.
  • A. Benchmarks, Metrics, and Baselines: 27 ETH3D grayscale stereo pairs provide evaluations against laser-scanned ground truth.ETH3D is one of the five real-world stereo datasets used for benchmarking.
  • A. Benchmarks, Metrics, and Baselines: Benchmark configurations and metric definitions are aligned to avoid discrepancies such as occlusion masking and D1 versus Bad-3.0.The protocol explicitly addresses differences in evaluation configuration and metric definitions.
  • A. Benchmarks, Metrics, and Baselines: Efficient feed-forward baselines are retrained with official code and the same synthetic training data as LAS2.Most efficient feed-forward methods release only SceneFlow-pretrained weights, motivating consistent retraining.
  • A. Benchmarks, Metrics, and Baselines: 6 Fast-FoundationStereo variants are available, and the “20-30-48” variant with 4 iterations is selected as the fastest configuration.The largest Fast-FoundationStereo variant cannot be deployed on NVIDIA Orin because of memory constraints.

B. Implementation Details

LAS2 is implemented in PyTorch and trained across three stages on NVIDIA H200 GPUs. LAS2-H additionally initializes selected modules from LAS2-M, freezes them early, and then fine-tunes all parameters.

  • Training setup: LAS2 training runs for 200K, 50K, and 200K steps in Stages ①, ②, and ③, respectively, with total batch size 128.Training uses NVIDIA H200 GPUs.
  • LAS2-H initialization: LAS2-H initializes its feature extraction and cost aggregation modules with LAS2-M weights and freezes them during the first 100K Stage ① steps.The remaining Stage ① steps fine-tune all model parameters.
  • Optimization: The models use the AdamW optimizer for training.

C. Evaluation

LAS2 achieves strong zero-shot accuracy and robustness across public benchmarks and challenging weather while maintaining efficient inference. Qualitative comparisons further show smoother, more coherent disparity maps with sharper boundaries and finer geometric detail.

  • Zero-shot Generalization: Across four public benchmarks, LAS2 feed-forward variants consistently outperform efficient baselines, while LAS2-S offers competitive accuracy with the lowest feed-forward latency.All methods use fixed checkpoints and are not trained on target domains.
  • Zero-shot Generalization: LAS2-H delivers the strongest overall zero-shot performance among efficient iterative methods while substantially reducing latency compared with existing iterative baselines.Against Fast-FoundationStereo, it improves KITTI and Middlebury accuracy, matches ETH3D, and reduces latency on H200 and Orin.
  • Challenging Weather: LAS2-S achieves the best overall D1 and EPE with the lowest latency on the DrivingStereo weather subset.The subset includes cloudy, foggy, rainy, and sunny scenes; LAS2-M and LAS2-L also outperform previous feed-forward efficient baselines.
  • Challenging Weather: On challenging weather scenes, LAS2-H outperforms both Fast-FoundationStereo and its FoundationStereo pseudo-label teacher on the benchmark’s overall metrics.This result supports the effectiveness of pseudo-label filtering and staged training for transferring geometric priors.
  • Qualitative Results: Qualitative comparisons show LAS2-M and LAS2-H produce smoother, more spatially coherent disparities while preserving fine structures and sharper object boundaries.The improvements address noisy artifacts, broken patterns, thin structures, reflective surfaces, and large depth discontinuities in real-world scenes.

D. Ablation Study

The ablations identify FasterNet-based design choices and a three-stage training strategy as key to LAS2’s accuracy-efficiency balance and synthetic-to-real adaptation. They also show that the training recipe generalizes beyond LAS2 to other efficient stereo models.

  • Architecture Design: FasterNet provides the best cost-aggregation accuracy-efficiency trade-off, combining the lowest KITTI 2012 and Middlebury errors with the lowest Orin latency.ConvNeXt has slightly lower MACs but substantially higher latency, emphasizing measured deployment speed over theoretical computation.
  • Architecture Design: The final feature extractor uses FasterNet because it balances performance and latency with a simpler architecture.A FasterNet-plus-1×1-convolution variant reduces latency from 107 ms to 95 ms but noticeably degrades KITTI and Middlebury accuracy.
  • Training Strategy Choices: Left-right consistency filtering gives the largest overall Stage ③ improvement, while error clamping is essential for stable synthetic-to-real adaptation.Removing clamping degrades all metrics; τclamp = 5 and τclamp = 10 perform similarly, τclamp = 20 is less effective, and τclamp = 10 is selected as the balanced default.
  • Effectiveness of the Training Strategy: Knowledge distillation in Stage ② outperforms direct data augmentation overall, with fixed teacher weights producing the most stable results.Stage ③ provides the largest improvement in the full training pipeline, confirming the complementary value of synthetic pretraining, self-distillation, and real-world distillation.
  • Effectiveness of the Training Strategy: The full training pipeline substantially improves LightStereo-M and BANet-2D and clearly surpasses the previous-version training strategy, demonstrating recipe generality.Feature alignment adds no clear gain and increases training cost, while multiple teachers produce inconsistent benchmark results and increase pseudo-label generation cost.

E. Latency Analysis

Under a unified benchmarking protocol, LAS2 delivers favorable latency across desktop/server GPUs and embedded hardware. Its feed-forward variants outperform comparable methods, while LAS2-H accelerates iterative stereo inference, especially on Orin.

  • Benchmarking protocol: For fair comparison, all methods are evaluated without torch.compile at 384 × 1248 using the same protocol across desktop/server GPUs and embedded hardware.The paper notes that differing hardware, protocols, and implementation-specific acceleration make prior latency reports unreliable.
  • Feed-forward methods: LAS2-S/M/L are consistently faster than corresponding LightStereo variants across all tested GPUs and Orin power modes.The efficiency advantage holds at comparable model scales for feed-forward stereo methods.
  • Feed-forward methods: Replacing expensive 3D aggregation with the proposed 2D aggregation substantially reduces LAS2-M latency versus the previous LAS model on all platforms.The comparison directly attributes the efficiency improvement to the aggregation redesign.
  • Feed-forward methods: LAS2-S achieves the lowest latency among all feed-forward methods, supporting latency-sensitive deployment scenarios.This result is reported across the evaluated hardware and operating conditions.
  • Iterative methods: LAS2-H is consistently faster than RT-MonSter++ and Fast-FoundationStereo, with the clearest advantage on the embedded Orin platform.The model retains iterative refinement while reducing computational overhead relative to recent iterative baselines.

V. LIMITATIONS AND DISCUSSION · VI. CONCLUSION

LAS2 improves efficient zero-shot stereo matching through deployment-oriented 2D cost aggregation and staged distillation, but remains behind prior-based high-accuracy methods and is limited by insufficient high-quality real-world stereo data.

  • V. LIMITATIONS AND DISCUSSION: LAS2 still trails prior-based high-accuracy methods that use stronger monocular depth priors and larger foundation backbones.These priors and backbones provide stronger semantic and geometric representations.
  • V. LIMITATIONS AND DISCUSSION: Limited high-quality real-world stereo data constrains LAS2’s potential for further improvement.The available diverse real-world stereo data remains insufficient relative to monocular foundation-model data.
  • V. LIMITATIONS AND DISCUSSION: Pseudo-label filtering and staged training exploit unlabeled real-world stereo images despite the data bottleneck.The passage identifies these methods as effective for using unlabeled real-world stereo images.
  • VI. CONCLUSION: LAS2 is an efficient stereo model series designed for zero-shot generalization.This is the central characterization stated in the conclusion.
  • VI. CONCLUSION: Pure 2D cost aggregation substantially improves measured inference latency on GPUs and edge devices.The architecture was developed by revisiting stereo design from a practical-deployment perspective.
  • VI. CONCLUSION: A three-stage strategy combines synthetic supervision, self-distillation, and real-world knowledge distillation.The conclusion links this training strategy with LAS2’s strong generalization across deployment-oriented design choices.
Loading 2606.24457v1…