Source-linked AI summary
CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion
Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, Jérôme Revaud
TL;DR
Existing MIM methods learn strong semantic representations, but this paper targets 3D and lower-level geometric tasks. CroCo masks one view and reconstructs it using a second view of the same scene, achieving stronger geometric transfer than classical auto-completion while remaining competitive on binocular tasks. Its main scope limits are dependence on same-scene image pairs and weaker high-level semantic performance.
Problem
Existing MIM pre-training is strong for high-level semantics, while representations that transfer broadly to 3D vision and lower-level geometric tasks remain a target.
Method
CroCo masks patches in one image and reconstructs them from its visible content and a second unmasked image depicting the same scene from another viewpoint.
Results
CroCo significantly outperforms MAE on depth prediction and Taskonomy while transferring directly to monocular and binocular tasks, including optical flow and relative pose estimation.
Takeaways & Limitations
Cross-view completion learns representations better suited to 3D vision than classical MIM auto-completion and works with a generic architecture on binocular tasks.
Takeaways & Limitations
CroCo is less tailored to high-level semantic tasks and requires pairs of images depicting the same scene.
Abstract
from arXiv · showhide
Masked Image Modeling (MIM) has recently been established as a potent pre-training paradigm. A pretext task is constructed by masking patches in an input image, and this masked content is then predicted by a neural network using visible patches as sole input. This pre-training leads to state-of-the-art performance when finetuned for high-level semantic tasks, e.g. image classification and object detection. In this paper we instead seek to learn representations that transfer well to a wide variety of 3D vision and lower-level geometric downstream tasks, such as depth prediction or optical flow estimation. Inspired by MIM, we propose an unsupervised representation learning task trained from pairs of images showing the same scene from different viewpoints. More precisely, we propose the pretext task of cross-view completion where the first input image is partially masked, and this masked content has to be reconstructed from the visible content and the second image. In single-view MIM, the masked content often cannot be inferred precisely from the visible portion only, so the model learns to act as a prior influenced by high-level semantics. In contrast, this ambiguity can be resolved with cross-view completion from the second unmasked image, on the condition that the model is able to understand the spatial relationship between the two images. Our experiments show that our pretext task leads to significantly improved performance for monocular 3D vision downstream tasks such as depth estimation. In addition, our model can be directly applied to binocular downstream tasks like optical flow or relative camera pose estimation, for which we obtain competitive results without bells and whistles, i.e., using a generic architecture without any task-specific design.
1 Introduction
CroCo introduces cross-view completion to learn 3D geometry from unlabeled image pairs, reconstructing masked content in one view using visible content and a second viewpoint. The approach targets transfer to geometric downstream tasks and reports improved performance over existing MIM models.
- Cross-view completion: CroCo masks parts of one image and reconstructs them using its visible patches plus a second image of the same scene from another viewpoint.The task resolves ambiguity by requiring reasoning about scene geometry and the spatial relationship between views.
- Motivation and evaluation: CroCo is designed to learn representations for 3D geometry from unlabeled data rather than primarily for high-level semantic recognition.The paper evaluates depth estimation and dense 2D and 3D regression tasks, while noting lower ImageNet classification performance than established MIM models.
- Model overview: The first image’s visible patches and the reference image are encoded, then a decoder uses both representations to predict the hidden patches.The two images share the same encoder weights, while the decoder includes cross-attention layers.
- Main findings: High masking ratios, including 90%, provide the best reported pre-training performance, and CroCo outperforms existing MIM models on evaluated geometric tasks.The comparison includes models pre-trained on the same synthetic data or on ImageNet.
2 Related work
Self-supervised vision methods learn transferable features from unlabeled data by constructing surrogate prediction tasks. Prior work includes augmentation-invariant instance discrimination, masked image prediction, and approaches designed to preserve dense local information for per-pixel tasks.
- Self-supervised learning: Self-supervised learning removes or transforms information to create surrogate labels, then learns visual features that support later supervised downstream tasks.Examples include recovering color, orientation, or patch ordering.
- Instance discrimination: Instance discrimination pulls augmented views of the same image instance together while pushing representations of different instances apart.Its objective is invariance to selected data augmentations.
- Masked image modeling: Masked image prediction reconstructs masked pixels, discrete tokens, or deep features and has achieved strong results across downstream tasks.These methods are motivated by BERT and commonly use Vision Transformers.
- Dense downstream tasks: Global-image self-supervision can transfer poorly to dense per-pixel prediction, motivating methods that learn contrastive representations locally.Dense downstream tasks include semantic segmentation.
- Geometric supervision: Other dense-task methods obtain supervision through view synthesis with photometric consistency or by matching RGB-D visual features to 3D geometric representations.These approaches use explicit geometric or multi-view signals.
3 Cross-view Completion Pre-training
CroCo pre-training masks most patches in one image and reconstructs them from the remaining patches and a second view encoded by a shared-weight network. Its transformer decoder integrates the two token sets, and training uses reconstruction losses on masked RGB patches.
- Cross-view completion: CroCo splits two same-scene views into non-overlapping patch tokens and typically discards 90% of the first image’s tokens.The masking ratio is defined as the fraction of first-image patches randomly masked.
- Decoder: The decoder reconstructs the first image by conditioning encoded visible tokens, padded masked-token representations, and reference-view tokens.This design lets the decoder use information from both viewpoints to predict hidden patches.
- Encoder: A shared-weight Siamese encoder independently processes visible tokens from the first image and all tokens from the reference image.The encoder is implemented with a Vision Transformer.
- Decoder architectures: CrossBlock alternates self-attention and cross-attention, whereas CatBlock concatenates both token sets before applying self-attention.These are the two decoder architectures evaluated in the paper.
- Training objective: Training predicts 16 × 16 RGB patches with 768 output values and minimizes reconstruction error over masked tokens.The paper also evaluates a variant that normalizes each target patch using its pixel mean and standard deviation.
- Pre-training data: The pre-training dataset contains 1,821,391 synthetic image pairs rendered from indoor 3D scenes with Habitat.Pairs are sampled from camera viewpoints with co-visibility greater than 50%.
- Transfer: CroCo can support monocular tasks with the encoder alone and binocular tasks using both the pre-trained encoder and decoder.This enables transfer to dense or non-dense downstream tasks.
4 Experimental results
CroCo’s experiments test masking, target normalization, decoder design, training profiles, image-pair construction, and transfer across monocular and binocular tasks. Cross-view pre-training improves 3D vision performance, while real viewpoint pairs and suitable co-visibility are important for effective geometric learning.
- Ablations: A 90% masking ratio gives the best performance across semantic segmentation, NYUv2 depth estimation, and Taskonomy.The authors attribute the high optimum to information supplied by the reference image during cross-view completion.
- Ablations: Target normalization consistently improves performance on all evaluated tasks, so CroCo uses normalized targets thereafter.The comparison is reported in Table 1 at a 90% masking ratio.
- Ablations: CrossBlock and CatBlock perform similarly, with CatBlock slightly better on segmentation and depth while CrossBlock is better on Taskonomy.CatBlock has fewer parameters but more FLOPs; the authors select CrossBlock for subsequent experiments.
- Ablations: Using the pre-trained decoder during monocular fine-tuning raises ADE performance from 40.6 to 41.0 and NYUv2 performance from 86.1 to 88.1.It also improves 6 of 8 Taskonomy tasks, but significantly increases computational cost.
- Ablations: CroCo significantly outperforms MAE on depth prediction and Taskonomy after Habitat pre-training, although both models perform similarly on semantic segmentation.The 3D-task advantage supports cross-view completion over pure auto-completion for geometric transfer.
- Ablations: Real viewpoint pairs outperform geometrically transformed pairs, while approximately 0.5 co-visibility produces the best pre-trained models.Synthetic transforms can let the model fit the transformation directly instead of reasoning about scene geometry.
- Comparison to the state of the art: On downstream comparisons, CroCo leads on NYUv2 and most Taskonomy tasks, reduces Sintel optical-flow error by almost 2 pixels versus the best MAE model, and outperforms all relative-pose methods except CamNet.CroCo achieves 85.6% Acc@1.25 on NYUv2, ranks first on 6 of 8 Taskonomy tasks, and is competitive on binocular tasks despite its generic architecture.
5 Discussion
CroCo introduces cross-view completion for 3D-oriented representation learning and transfers to monocular and binocular tasks. The approach uses same-scene image pairs, synthetic renderings in this work, and appears less suited to high-level semantic tasks.
- CroCo introduces cross-view completion as a pre-training task tailored to 3D vision downstream tasks.
- The learned representations transfer to both monocular and binocular tasks.
- The model appears less tailored to high-level semantic tasks, possibly because of the synthetic-rendering dataset choice.
- The method requires same-scene image pairs, and this work uses synthetic renderings only.
B Details on the decoder architectures
The decoder architectures use transformer attention to combine tokens from the masked and reference views. The appendix also specifies the optimization and masking settings used during pre-training.
- Transformer blocks: The transformer blocks use attention projections with learnable parameters, followed by a residual MLP output transformation.
- Decoder architectures: The decoder architectures comprise CrossBlock and CatBlock variants for combining tokens from the two views.
- CrossBlock: CrossBlock applies self-attention and cross-attention sequentially to tokens from the two views.
- CatBlock: CatBlock concatenates the two token sets and adds learned embeddings identifying the input views.
- Pre-training settings: Pre-training uses AdamW for 400 epochs with batch size 256, 90% masked tokens, and 224 × 224 inputs.
C.2 Ablation on overlaps between pre-training pairs
The overlap ablation varies co-visibility in otherwise matched training sets and finds that pairs near 0.5 co-visibility perform best on most 3D-related Taskonomy tasks.
- Pair construction: The main experiments retain pairs above a 0.5 co-visibility threshold and encourage large viewpoint changes.
- Definitions: Co-visibility is defined as the minimum of the two directional visibility ratios between a pair of views.
- Experimental setup: 700,000-pair training sets with different co-visibility distributions were compared while keeping other pre-training parameters fixed.
- Results: Pairs with co-visibility close to 0.5 yielded better performance on most 3D-related Taskonomy tasks than pairs with greater or lower co-visibility.
- Results: Figure 9 plots Taskonomy performance against co-visibility settings, with lower values indicating better performance.
D.2 Leveraging the decoder for monocular tasks
For monocular downstream tasks, CroCo can use its encoder, decoder, and DPT module in different combinations. Combining decoder features with DPT improves performance, whereas removing DPT or freezing the backbone degrades it.
- Default setup: The default monocular setup appends a DPT module to the CroCo encoder and discards the decoder during finetuning.
- Encoder-only variant: Removing the DPT module significantly degrades performance across all evaluated tasks.
- Decoder variant: The decoder can replace DPT because it was trained to produce dense predictions, although it does not fuse features across multiple depths.
- Decoder and DPT: Using the decoder together with DPT produces consistent performance gains but increases running cost.
- Frozen backbones: Freezing the backbone degrades performance compared with finetuning it.
D.3 Visualization of monocular depth prediction and Taskonomy results
This section presents qualitative depth and Taskonomy visualizations, alongside absolute pose regression experiments comparing backbones and pre-training methods under different training-set sizes.
- D.3 Visualization of monocular depth prediction and Taskonomy results: Figure 10 shows NYUv2 validation inputs, predicted depths, and depth predictions overlaid on the inputs.The visualization presents the input image followed by the depth prediction and its overlay.
- D.3 Visualization of monocular depth prediction and Taskonomy results: Figure 11 presents predictions and ground truth for each of Taskonomy’s 8 dense regression tasks.The top row contains images, while predictions and ground truth are shown for the eight tasks.
- D.4 Application to absolute pose regression from a single image: The pose-regression loss combines position and rotation terms whose contributions are balanced by learned weights β and γ.The rotation is represented through the logarithmic form of a unit quaternion, with quaternion signs restricted to one hemisphere.
- D.4 Application to absolute pose regression from a single image: CroCo achieves similar full-data absolute pose performance to most compared models, while MultiMAE consistently underperforms.The comparison uses median errors averaged over seven scenes across different training-set ratios.
- D.4 Application to absolute pose regression from a single image: With only 5%, 10%, or 20% of the training data, CroCo pre-training outperforms the pre-trained ResNet34 baseline and the MAE/MultiMAE models.The original AtLoc ResNet34 model is unable to learn from these small data fractions, whereas CroCo performs better than the alternative pre-training methods.
E Further details and results on the binocular downstream tasks
CroCo transfers to binocular optical flow, relative pose, and stereo matching using simple task heads, achieving competitive or improved results without task-specific architectural designs.
- E.1 Optical flow: Optical flow uses the unchanged pre-training architecture with a simple head producing two flow channels, trained with MSE on AutoFlow and evaluated by AEPE on MPI-Sintel.The evaluation covers clean and final renderings of the MPI-Sintel training split.
- E.1 Optical flow: CroCo flow estimation correctly handles varied and fast motion, but becomes slightly inaccurate under occlusions and blurry on extremely fine-grained motion.The model was finetuned on 40,000 images without elaborate data augmentation.
- E.2 Relative pose regression: Relative pose prediction replaces CroCo’s decoder head with a projection, multilayer perceptron, and differentiable Procrustes layer that outputs rotation R and translation t.The MLP produces a 12D output reshaped into an affine transformation before orthonormalization.
- E.3 Stereo image matching: Stereo matching replaces the linear prediction head and predicts pixel-wise disparity from two rectified images.The model is evaluated using the 3-pixel 5% discrepancy on Virtual KITTI.
- E.3 Stereo image matching: CroCo pre-training achieves significantly better stereo results than MAE across all 10 VKITTI variants.The variants cover weather conditions and camera orientations, and evaluation uses average 3-pixel error.
- E.3 Stereo image matching: CroCo stereo finetuning is competitive with state-of-the-art methods without spatial pyramid pooling, hierarchical neural search, or explicit neighbor-relationship designs.Unlike the compared methods, CroCo is finetuned directly on VKITTI without SceneFlow pre-training.
F Compute resources, code and dataset assets
This section reports computational cost and catalogs the code, pre-trained models, datasets, and licenses used across the experiments.
- F.1 Floating point operations (FLOPs): Most FLOPs arise from matrix-matrix multiplications in attention and feed-forward layers.The standard ViT estimate sums the attention and feed-forward contributions across encoder blocks.
- F.1 Floating point operations (FLOPs): For ViT-Base/16, the FLOPs formula gives 34.7 GFLOPs, close to the 35.3 GFLOPs obtained when all other operations are included.The calculation uses 12 blocks, 142 tokens, and embedding dimension 768.
- F.2 Compute resources: Pre-training CroCo for 400 epochs takes about 64 GPU-days on NVIDIA V100, or approximately two weeks on a 4-GPU server.This is the reported training-resource requirement for the CroCo model.
- F.3 Code and dataset assets: The paper lists code and pre-trained models for CroCo-related baselines including MAE, MultiMAE, and DINO.Licensing information is provided for these assets and the experimental datasets.
G Further visual examples
Additional visualizations show cross-view reconstructions on unseen scenes and compare CroCo against reference ablations and MAE.
- G Further visual examples: Cross-view completion generally reconstructs masked regions well, except where the content is not visible in the reference image.The comparison replaces the reference image with random uniform noise to assess the role of the second view.
- G Further visual examples: CroCo produces more detailed normalized-patch reconstructions than MAE on unseen scenes.The displayed reconstructions are un-normalized using target-image patch statistics for visualization.