Source-linked AI summary
CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow
Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, Jérôme Revaud
TL;DR
Self-supervised pre-training has not matched its semantic-task success on dense geometric tasks such as stereo matching and optical flow. CroCo v2 improves cross-view completion with large-scale real-world pairs, relative positional embeddings, and larger transformers, reaching state-of-the-art performance on both tasks without classical task-specific designs.
Problem
Self-supervised pre-training has achieved strong semantic-task results but has not produced comparable breakthroughs for stereo matching and optical flow.
Method
CroCo v2 combines cross-view completion with scalable real-world pair collection, rotary positional embeddings, and larger encoder and decoder networks.
Results
CroCo v2 reaches state-of-the-art performance on stereo matching and optical flow with a generic ViT-based architecture that omits classical task-specific designs.
Takeaways & Limitations
Large-scale pre-training with a geometry-adapted pretext task and real-world data can support dense geometric tasks using a more universal vision architecture.
Takeaways & Limitations
Tiling-based inference may be inaccurate for extremely large disparities or flows when corresponding pixels lie outside the second-image tile.
Abstract
from arXiv · showhide
Despite impressive performance for high-level downstream tasks, self-supervised pre-training methods have not yet fully delivered on dense geometric vision tasks such as stereo matching or optical flow. The application of self-supervised concepts, such as instance discrimination or masked image modeling, to geometric tasks is an active area of research. In this work, we build on the recent cross-view completion framework, a variation of masked image modeling that leverages a second view from the same scene which makes it well suited for binocular downstream tasks. The applicability of this concept has so far been limited in at least two ways: (a) by the difficulty of collecting real-world image pairs -- in practice only synthetic data have been used -- and (b) by the lack of generalization of vanilla transformers to dense downstream tasks for which relative position is more meaningful than absolute position. We explore three avenues of improvement. First, we introduce a method to collect suitable real-world image pairs at large scale. Second, we experiment with relative positional embeddings and show that they enable vision transformers to perform substantially better. Third, we scale up vision transformer based cross-completion architectures, which is made possible by the use of large amounts of data. With these improvements, we show for the first time that state-of-the-art results on stereo matching and optical flow can be reached without using any classical task-specific techniques like correlation volume, iterative estimation, image warping or multi-scale reasoning, thus paving the way towards universal vision models.
1. Introduction
CroCo v2 addresses the limited transfer of self-supervised pre-training to stereo matching and optical flow by combining cross-view completion, large-scale real-world image pairs, relative positional embeddings, and larger transformers. The resulting generic architecture reaches state-of-the-art performance on both tasks without classical task-specific designs.
- Self-supervised methods have excelled on semantic tasks but have not produced comparable breakthroughs for stereo matching and optical flow.
- Cross-view completion reconstructs masked patches using visible patches from one image and an additional view of the same scene, requiring geometric understanding.
- Cross-view completion is limited by the difficulty of collecting same-scene image pairs at scale and by absolute positional embeddings that generalize poorly across resolutions and crops.
- CroCo v2 collects millions of diverse real-world training pairs with controlled overlap and viewpoint differences using sources including meshes, sensors, and SfM reconstructions.
- The method improves CroCo through real-world data, rotary positional embeddings, and larger encoder and decoder networks.
- CroCo-Stereo and CroCo-Flow use plain transformer components with a DPT head and reach state-of-the-art performance without cost volumes, image warping, iterative refinement, or multi-level feature pyramids.
2. Related work
Prior self-supervised approaches have mainly targeted high-level or dense semantic tasks, while CroCo introduced cross-view completion to better support geometric downstream tasks. Related work also explored relative positional embeddings and task-specific unsupervised reconstruction methods.
- Instance discrimination uses augmentation-invariant features and achieved strong results on image-level tasks, with some extensions to object detection and human pose estimation.
- CroCo adds a second view of the same scene to masked image modeling, making reconstruction depend on implicitly understanding scene geometry.
- Relative positional embeddings improved performance in several vision transformer systems, and were reported as crucial in cross attention for optical flow.
- Other self-supervised methods estimate depth, stereo, or optical flow with unsupervised reconstruction losses, whereas CroCo pre-trains a task-agnostic model for multiple downstream tasks.
3. Cross-view completion pre-training at scale
CroCo v2 scales cross-view completion pre-training by constructing diverse real-world image pairs, replacing absolute positional embeddings with RoPE, and enlarging the transformer architecture.
- Cross-view completion: CroCo extends masked image modeling to two images of the same scene, masking 90% of the first image’s tokens and reconstructing targets from both views.The two images are processed by shared-weight encoders, while a decoder combines their features.
- Collecting real-world image pairs: Suitable training pairs depict the same scene with partial overlap: too little overlap reduces geometric information, whereas too much makes reconstruction trivial.The method therefore seeks overlap that requires scene-geometry understanding rather than auto-completion or copying.
- Collecting real-world image pairs: The collection pipeline uses posed real-world images plus available meshes, point clouds, or SfM data, then generates 224×224 overlapping crops using image matching.For each pair, matching estimates corresponding crops and retains crops with consistent matches and no overlap in the first image.
- Collecting real-world image pairs: Pair selection scores visual overlap and viewpoint difference, then greedily selects high-scoring, diverse pairs while discarding redundant images.The score favors different viewpoints with substantial overlap; pairs with IoU above 0.75 are treated as too redundant.
- Positional embeddings: RoPE replaces absolute positional embeddings by making attention similarity depend on relative position differences, with separate embeddings for image x- and y-dimensions.This design is applied to two-dimensional image features.
- Scaling the architecture: The model scales both encoder and decoder capacity, using a decoder with 12 blocks, 768-dimensional features, and 12 attention heads instead of the smaller decoder.The decoder is enlarged because it combines information from the two views.
4. Application to stereo matching and flow
CroCo-Stereo and CroCo-Flow apply the cross-view-pre-trained model to disparity and flow prediction with a generic transformer architecture and probabilistic outputs.
- Architecture: CroCo-Stereo and CroCo-Flow use a plain ViT encoder, transformer decoder, and DPT head to directly predict disparity or optical flow without classical task-specific components.The architecture is described as correlation-free and contrasts with methods using cost volumes, warping, iterative refinement, or feature pyramids.
- Probabilistic prediction: The models regress Laplacian distributions whose location predicts disparity or flow and whose scale represents prediction uncertainty.The predicted scale is used as a per-pixel confidence-related quantity and adapts the penalty for errors.
- Architecture: The decoder combines self-attention over first-image tokens, cross-attention with second-image tokens, and an MLP before DPT produces the final prediction.Intermediate token features are also fed to the DPT module.
- Ablations: Table 1 evaluates CroCo changes using bad@1.0 for stereo and EPE for optical flow across Middlebury, ETH3D, SceneFlow, FlyingThings, and MPI-Sintel validation sets.The table compares clean and final renderings and distinguishes Small and Base decoder configurations.
- Probabilistic prediction: The probabilistic loss empirically improves performance and supports tiling strategies during inference through per-pixel confidence estimates.Inference samples overlapping tiles, predicts each tile pair, and merges predictions at each pixel.
5. Experiments
Experiments show that CroCo v2 improves stereo and optical-flow performance through its pre-training and reaches strong benchmark results with a generic architecture, while tiling can fail on very large disparities or flows.
- Ablations: All four CroCo v2 changes consistently improve performance, reducing Middlebury bad@1.0px from 26.3 to 15.5 and MPI-Sintel clean EPE from 2.07 to 1.43.The changes are RoPE, larger decoder, larger-scale pre-training data, and a larger encoder.
- CroCo-Stereo vs. the state of the art: CroCo-Stereo achieves the lowest average error on 6 of 15 Middlebury sequences despite omitting cost volumes, coarse-to-scale processing, and iterative refinement.A few large-disparity sequences still produce large errors under the tiling-based inference approach.
- CroCo-Stereo vs. the state of the art: CroCo-Stereo performs best on KITTI 2015 D1-all, including foreground pixels and near-best background performance.The cited result reports the best value on foreground pixels and performance within 0.01% of the best methods on background pixels.
- CroCo-Flow vs. the state of the art: CroCo-Flow ranks second on MPI-Sintel clean rendering, leads KITTI 2015 F1-all at 3.64% versus GMFlow+ at 4.49%, and reaches Spring EPE 0.50 versus 0.64.On KITTI, the advantage mainly comes from background pixels; on MPI-Sintel final rendering, performance is competitive with recent approaches.
6. Conclusion
The paper concludes that large-scale real-world pre-training with a geometry-adapted pretext task enables strong dense geometric prediction using a ViT architecture without task-specific designs. It frames this as a route toward more universal vision models, while appendix material also documents evaluation and training details.
- 6. Conclusion: Large-scale pre-training succeeds for dense geometric tasks when paired with a well-adapted pretext task and real-world data at scale.
- 6. Conclusion: CroCo v2 reaches state-of-the-art performance with a ViT-based architecture without task-specific designs such as correlation volumes or iterative estimation.
- 6. Conclusion: The results open routes toward tackling geometric problems with more universal vision models.
A. Cross-view completion examples
Qualitative cross-view completion examples on real-world Middlebury and KITTI scenes show that CroCo v2 preserves substantially more visual detail than the original CroCo model, even under heavy masking.
- A. Cross-view completion examples: CroCo v2 reconstructions preserve details such as lines, eyes, map roads, and readable text under a 90% masking ratio.
- A. Cross-view completion examples: The original CroCo reconstructions are often blurry, whereas CroCo v2 preserves more detail on real-world scenes unseen during pre-training.
- A. Cross-view completion examples: Some CroCo v2 predictions remain blurry where the relevant content is not visible in the reference image.
B.1. Impact of pre-training
Appendix ablations show that improved cross-view completion pre-training substantially benefits downstream stereo and optical-flow finetuning compared with random initialization and provides a binocular architecture unavailable directly from common alternatives.
- B.1. Impact of pre-training: Pre-training reduces FlyingThings final-rendering EPE from 10.57 to 2.45 pixels and Middlebury bad@1.0px from 43.4% to 15.5%.
- B.1. Impact of pre-training: Cross-view completion naturally includes a binocular decoder or architecture, unlike the other pre-training strategies discussed.
- B.1. Impact of pre-training: The comparison includes MAE pre-trained on ImageNet and the original CroCo pre-trained only on synthetic data.
B.2. Smaller training data
CroCo-Flow remains competitive when trained only on FlyingChairs and FlyingThings, outperforming RAFT and GMFlow before their iterative refinement procedures.
- CroCo-Flow performs better than RAFT and GMFlow before their iterative refinement procedures on the MPI-Sintel training set.The comparison uses training only on FlyingChairs and FlyingThings.
B.3. Runtime and tiling
The models run quickly per tile, but full-resolution tiled inference trades higher accuracy for rapidly increasing runtime as tile overlap grows.
- A single 704×352 tile runs in the order of a few tens of milliseconds on an NVIDIA A100 GPU.
- The models use one order of magnitude more parameters than most existing stereo and flow methods.The authors did not study parameter reduction or claim superiority at a fixed computational budget.
- Higher tile overlap improves stereo and flow performance, but the number of tiles—and therefore runtime—can rapidly increase.
- Overlap around 0.5 or 0.7 offers performance close to 0.9 while keeping the tile count reasonable.
B.4. Laplacian-based loss
The Laplacian-based formulation supplies both predictions and uncertainty estimates, which improve benchmark performance and support uncertainty-aware merging of overlapping tiles.
- A Laplacian distribution models disparity or flow through its location parameter and represents uncertainty with its scale parameter.The reported uncertainty is the logarithm of the predicted Laplacian scale.
- Lower predicted uncertainty corresponds to lower disparity errors, while higher uncertainty corresponds to higher errors.This correlation is observed in both the visualization and pixel-ordering statistics.
- 95% of pixels have an error below 1 in the reported uncertainty analysis.
- The Laplacian loss outperforms the L1 loss on all stereo and flow benchmarks.Its predicted scale also enables uncertainty-aware merging of overlapping tiles.
- A secondary shifted tile reduces Middlebury v3 validation bad@1.0 from 17.1% to 12.0%, but doubles the number of tiles processed.The strategy replaces predictions beyond 200px from the original tile.
C. Training details
CroCo-Stereo training uses paired crops and dataset-specific training-data configurations, while flow training draws samples probabilistically from available datasets.
- CroCo-Stereo is trained for 32 epochs with batches of six paired 704×352 crops.Training uses AdamW, weight decay 0.05, a cosine schedule, one warm-up epoch, and learning rate 3.10^-5.
- The stereo training-data overview specifies ablation train/validation splits and the number of training pairs, with repeated ETH3D and Middlebury pairs per epoch.
- Flow training samples images randomly from available datasets according to the percentages listed in the probability column.The overview also reports the number of remaining training pairs and images sampled per epoch.