PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Cross-View Completion: Self-Supervised Pre-Training via Reconstruction Error Comparison

Thibaut Loiseau, Guillaume Bourmaud, Vincent Lepetit

arXiv:2609.01530v1cs.CV

TL;DR

Cross-view completion becomes implicitly monocular when masked regions are not co-visible in the reference view. Gekko compares cross-view and MAE reconstruction errors, jointly predicts their relative improvement, and under matched settings consistently outperforms CroCo on cross-view 3D tasks while supporting raw-video training.

  • Problem

    Cross-view completion provides little binocular information for non-co-visible patches, limiting its training signal on image pairs with low overlap.

  • Method

    Gekko jointly performs cross-view completion, masked autoencoding, and per-pixel prediction of relative reconstruction improvement without ground-truth 3D labels.

  • Results

    Under identical architectures and training data, Gekko consistently outperforms CroCo across relative pose estimation, correspondence estimation, and pointmap regression.

  • Takeaways & Limitations

    Gekko supplies a binocular signal for all masked regions, detects co-visibility on unseen scenes, and can be trained from raw videos while matching curated-data models.

  • Takeaways & Limitations

    Gekko costs more to pre-train than CroCo, with approximately 2.4× higher GPU-hours for matched Base pre-training runs.

Abstract

from arXiv · show

Self-supervised pre-training via cross-view completion learns strong features for 3D vision from co-visible regions of image pairs. However, the reference view provides little information for reconstructing non-co-visible patches, implicitly yielding a monocular training signal in these regions. We introduce Gekko, which turns this limitation into a useful signal. The relative improvement of the cross-view reconstruction error over a masked-autoencoder error is a self-supervised proxy for co-visibility: large improvements indicate co-visible regions, negligible ones non-co-visible areas. Gekko is a network, trained from scratch, that jointly performs cross-view completion, masked autoencoding, and per-pixel prediction of this relative improvement, providing an additional binocular signal for all masked regions without any ground-truth 3D annotation. Under identical architectures and training data, Gekko consistently outperforms CroCo on zero-shot correspondence estimation, relative pose estimation, and pointmap regression, with up to 6 times higher accuracy at the strictest relative-pose threshold and a 22% drop in end-point error on ETH3D. The extra channel it learns is itself a strong co-visibility detector on unseen scenes, and Gekko's frozen features outperform released cross-view backbones of comparable or larger size. It can also be trained directly from raw videos with a simple stride-based curriculum, removing the cumbersome 3D preprocessing prior methods require while matching models trained on curated data. Code and pre-trained models are publicly available.

1 Introduction

Gekko addresses the monocular limitation of cross-view completion by predicting relative improvement between cross-view and masked-autoencoder reconstruction. It adds this binocular signal while preserving CroCo’s architecture and improves cross-view 3D evaluation, including raw-video training.

  • Cross-view completion becomes implicitly monocular when masked target patches are not co-visible in the reference image.
  • Gekko uses relative improvement between cross-view and MAE reconstruction errors as a self-supervised co-visibility proxy without ground-truth 3D labels.
  • Gekko jointly performs cross-view completion, masked autoencoding, and relative-improvement prediction with one additional output channel.Its encoder and cross-attention decoder remain byte-for-byte identical to CroCo.
  • Under identical architectures and training data, Gekko outperforms CroCo across relative pose estimation, correspondence estimation, and pointmap regression.Reported gains include up to 6× higher accuracy at the strictest relative-pose threshold and a 22% drop in ETH3D end-point error.
  • Gekko can use raw videos with a stride-based curriculum, matching curated-data models while removing prior 3D preprocessing requirements.Its predicted channel also detects co-visibility on unseen scenes, and its frozen features outperform comparable or larger released cross-view backbones.

2 Related work

The paper situates Gekko between single-image self-supervised learning and cross-view image-pair methods, addressing their shared monocular signal in non-co-visible regions without relying on ground-truth overlap.

  • Masked image modeling and self-distillation learn visual features from single images without manual labels.
  • CroCo extends masked image modeling to image pairs, while later cross-view methods build on this framework for 3D reconstruction and matching.
  • Cross-view completion methods share an implicitly monocular training signal where regions are not co-visible across views.Gekko explicitly models relative improvement over monocular reconstruction to provide an additional binocular signal.
  • Co-visibility masks are commonly derived from depth maps and camera poses to select training pairs or learn matchability scores.

3 Background and motivation

CroCo and MAE provide complementary reconstruction baselines for comparing cross-view benefit. Their relative improvement is more informative for co-visibility than CroCo’s error alone, though it remains an approximation.

  • CroCo reconstructs masked target patches using the target image and a reference image, whereas MAE reconstructs them from the masked target alone.
  • CroCo benefits from reference information in co-visible regions but degrades where the reference contributes little.
  • CroCo’s reconstruction error alone is unreliable for predicting co-visibility because its co-visible and non-co-visible distributions largely overlap.
  • 3.2 Relative improvement of CroCo over MAE: The relative improvement C(p) compares MAE and CroCo reconstruction errors, approaching 1 for co-visible regions and 0 for non-co-visible regions.The comparison uses MAE as a reference-independent reconstruction baseline.
  • 3.2 Relative improvement of CroCo over MAE: Relative improvement reaches AP 0.74 for co-visibility prediction, compared with AP 0.57 for CroCo’s error alone.
  • 3.2 Relative improvement of CroCo over MAE: C(p) approximates co-visibility rather than perfectly classifying it because low MAE error can make the ratio uninformative, including in repeated or self-similar regions.

4 Method

Gekko turns cross-view reconstruction improvement into a self-supervised co-visibility signal and jointly learns completion, masked autoencoding, and relative-improvement prediction without 3D annotations. Its learned maps correlate with ground-truth co-visibility labels.

  • Gekko uses relative reconstruction improvement as pseudo co-visibility labels to improve CroCo pre-training without 3D annotations.
  • The cross-view pass reconstructs the target from its masked patches and a reference image, while the masked-autoencoder pass uses the same mask without the reference image.
  • Gekko jointly performs cross-view completion, masked autoencoding, and prediction of the relative improvement between their reconstruction errors.
  • Gekko adds a fourth pixel-decoder channel for predicting the relative improvement map while retaining CroCo’s ViT encoder and cross-attention decoder.
  • Down-weighting pixels with low masked-autoencoder loss improves performance and produces sharper co-visibility maps.
  • Gekko’s predicted relative-improvement maps correlate strongly with ground-truth co-visibility labels, though they are less sharp in uniform regions.

5 Experiments

Experiments evaluate Gekko under matched and curriculum-based settings across correspondence, pose, pointmap, and co-visibility tasks, while also examining scaling, ablations, cost, and scope limits.

  • Zero-shot correspondence estimation: 22% lower AEPE for encoder features and 21% for decoder features on ETH3D over CroCo in zero-shot correspondence estimation.Features are evaluated immediately after pre-training without fine-tuning; all models are Base and pre-trained on ScanNet-50.
  • Relative metric pose estimation: Nearly 6× higher strict-threshold accuracy than CroCo-B on ScanNet-1500 relative metric pose estimation, with the gap widening on ScanNet-all.On ScanNet-all, Gekko-B reaches 43.7% versus 6.6% for CroCo-B.
  • Relative metric pose estimation: 35.9% versus 7.5% at 10°/0.25m on ScanNet-all gives Gekko-L a 4.8× improvement over CroCo-L, showing gains scale to larger architectures.Both Large models use the same architecture, data, batch size, and 100k-step schedule.
  • Curriculum pre-training: 31.2% versus 29.1% at 10°/0.25m shows raw-video Gekko matches its curated-data counterpart under stride-based curriculum training.The raw-video setting uses no 3D annotation or overlap-based preprocessing, while the exact training pairs differ.
  • Pointmap regression: 10% lower in-domain and 5–7% lower out-of-domain Chamfer error in pointmap regression show gains across ScanNet, DL3DV, and ETH3D.The evaluation uses a DPT head on frozen features, with models pre-trained on DL3DV using curriculum and fine-tuned on ScanNet-all.
  • Co-visibility classification: 0.763 AP versus 0.576 for CroCo demonstrates that Gekko’s predicted channel detects co-visibility on ScanNet-1500.The channel also reaches 0.691 balanced accuracy at one global threshold and performs relatively better on the lowest-overlap tertile.
  • Ablation studies: Adding MAE without relative-improvement prediction barely changes results, supporting the relative-improvement loss as the source of the ablation gain.The relative formulation outperforms the absolute one, and down-weighting low-MAE pixels is reported to improve results by 18.0 points.
  • Training cost: Gekko’s extra full-resolution prediction pass roughly doubles activation memory, halves per-device batch size, and raises matched Base pre-training cost to approximately 2.4× CroCo.The additional cost is confined to pre-training; fine-tuning and inference cost what CroCo costs.

6 Conclusion

Gekko addresses the monocular signal in non-co-visible regions by predicting relative cross-view improvement, adding binocular supervision without ground-truth 3D labels. It outperforms CroCo on evaluated cross-view tasks and can train directly from raw video.

  • 6 Conclusion: Gekko uses relative improvement between cross-view and MAE reconstruction errors as a self-supervised co-visibility proxy.Large improvement indicates co-visible regions, while near-zero improvement indicates non-co-visible regions.
  • 6 Conclusion: Gekko jointly performs cross-view completion, masked autoencoding, and relative improvement prediction with one extra output channel.The predicted signal applies to all masked regions without ground-truth depth or pose supervision.
  • 6 Conclusion: Gekko consistently outperforms CroCo on the evaluated cross-view tasks under matched architecture and data.The evaluations cover correspondence estimation, relative pose estimation, and pointmap regression.
  • 6 Conclusion: Gekko trains directly from raw video with a stride curriculum, removing the 3D preprocessing required by prior work.The conclusion states that raw-video training matches models trained on curated data.

C Training details

The training setup follows CroCo’s ViT-based framework, using fixed pre-training and task-specific fine-tuning configurations. Raw reconstruction behavior differs by visibility: CroCo is sharper in co-visible regions and blurred elsewhere.

  • Pre-training: All models are trained from scratch for 100k steps with batch size 768, learning rate 1.5 × 10^-4, warmup, cosine decay, 90% masking, and normalized patches.Images are resized to a maximum dimension of 512 pixels while maintaining aspect ratio.
  • Fine-tuning: Relative metric pose estimation adds an MLP head over frozen decoder features and trains it for 20k steps.Fine-tuning uses a homoscedastic loss and the all-overlap variant.
  • Fine-tuning: Pointmap regression fine-tunes a DPT head over frozen features using a confidence-aware regression loss on ScanNet-all.It uses the same batch size and learning rate as pose estimation.
  • Architectures: The Base model uses a 12-layer ViT encoder and 8-layer transformer decoder, while the Large model uses 24 and 12 layers respectively.Base and Large embedding sizes are 768/512 and 1024/768 for encoder/decoder components.
  • Qualitative behavior: CroCo reconstructions are sharper in co-visible regions and blurred in non-co-visible regions.The reference image supplies useful warped information only where the views overlap.

F Detailed ablation analysis

Ablations show that relative improvement prediction, rather than MAE joint training alone, drives Gekko’s gains, while formulation and normalization choices materially affect performance.

  • Impact of the relative improvement loss: 20.9% versus 18.5% at 10°/1m shows that adding MAE without relative improvement yields only marginal improvement over CroCo.The ablation identifies relative improvement prediction as the key contribution.
  • Confidence prediction vs. relative improvement: 45.8% at 10°/1m on ScanNet-50 falls to 11.2% on ScanNet-all for confidence-aware prediction.Confidence maps also lack the sharp co-visibility boundaries achieved by relative improvement.
  • Relative vs. absolute improvement: 39.4% versus 57.6% on ScanNet-all shows that absolute improvement underperforms the relative formulation.The relative formulation produces sharper maps that better capture co-visibility boundaries.
  • Impact of patch normalization: 25.1% versus 39.4% on ScanNet-all shows a significant performance drop without target patch normalization.Removing normalization makes predictions noisier in uniform regions and destabilizes relative improvement estimates.
  • Loss formulation: 25.7% to 43.7% at 10°/.25m on ScanNet-all results from multiplying the prediction by the denominator instead of directly regressing the ratio.This is the final loss formulation described in Eq. 8.

G Ablation study: technical details

The technical ablations define alternative losses and evaluate co-visibility prediction on raw-video mixtures and held-out benchmarks. Protocol details clarify masking, labels, thresholds, and comparisons with released backbones.

  • Loss definitions: The relative-improvement loss predicts the ratio-based co-visibility signal, while alternative losses reweight it, predict absolute improvement, or estimate precision.The confidence-aware formulation expects precision to be higher in co-visible regions and lower in non-co-visible regions.
  • Raw-video training: Raw-video mix training samples frame pairs by stride without structure-from-motion, depth, camera pose, or overlap estimates.The mix uses twelve video sources spanning real and synthetic indoor, outdoor, and object-centric data.
  • Evaluation protocol: Co-visibility evaluation reads the fourth output channel directly without post-processing or test-time augmentation.Ground truth uses reprojection and depth agreement within 5%, excluding pixels landing in reference-depth holes.
  • Evaluation protocol: The evaluation covers all 1500 ScanNet-1500 pairs and 558 retrieval pairs from 7-Scenes, using threshold-free average precision and ROC-AUC.Balanced accuracy uses one fitted global threshold per dataset and score.
  • Evaluation caveats: Restricting co-visibility evaluation to SIFT keypoints raises every score and improves raw signals more than the prediction.The reported prediction change is 0.763 → 0.790, compared with 0.576 → 0.587 for ℓCroCo.
  • Released backbones: The self-supervised released-backbone comparison is not data-matched, and VGGT is a supervised ∼1B reference with overlapping training data.Matched architecture-and-data comparisons remain the controlled evidence for objective comparisons.

K Data scaling on the raw-video mix

Data scaling experiments test whether Gekko’s relative-improvement signal remains beneficial as the raw-video training mix grows. Gekko leads across the evaluated data fractions, although fixed-step pose performance can dip when larger datasets are undertrained.

  • Experimental control: The experiment varies only the fraction of mix scenes, while sources, sampling weights, resolution, batch size, step count, and fine-tuning budgets remain matched.The comparison uses nested, seeded, scene-level subsets from 1% to 100%.
  • Data scaling: Gekko leads at every evaluated data scale, with strictest-threshold pose gains ranging from +7.4 to +24.1 points.The matched Base models use nested scene-level subsets from 1% to 100% of the mix, with identical sampling, optimization, and fine-tuning budgets.
  • Data scaling: Above 50% data, both methods dip on pose at the fixed 100k-step budget, indicating undertraining rather than harm from additional data.Extending the best mix model to 200k steps recovers pose scores of 39.4/56.1/61.1 and a pointmap Overall of 0.092.
  • Data composition: On an indoor benchmark, composition matters more than size: the indoor-only mix gives Gekko 39.4/59.2/65.5 versus 34.4/51.5/56.5 for the full mix.

L Generality across architectures

Gekko’s advantage is not confined to CroCo’s original encoder–decoder design. Replacing either architectural half preserves a positive advantage over matched CroCo controls.

  • Experimental design: The architecture substitution study pre-trains six Base models from scratch on ScanNet-50 for 100k steps while varying architecture and holding other settings fixed.This tests whether the gain could be an artifact of CroCo’s specific encoder–decoder.
  • Evaluation: Table 10 reports matched Base-model comparisons on ScanNet-1500 pose and ScanNet pointmap Chamfer Overall after architectural substitutions.
  • Architecture substitutions: Gekko’s advantage survives both decoder and encoder substitutions, with gains of +17.2 and +18.9 points versus +23.6 for the paper’s architecture.The joint self-attention decoder removes cross-attention, while the DINOv3 variant freezes an off-the-shelf monocular encoder.
  • Architecture substitutions: CroCo remains nearly unchanged across the three architectures, moving from 5.0 to 5.7, whereas Gekko changes substantially.

M Single-image probes

Single-image probes show that Gekko’s binocular objective does not provide a general advantage on monocular tasks, while broader evaluations distinguish frozen-feature transfer from full-network fine-tuning and test representation consistency across domains.

  • Single-image probes: The authors do not expect the binocular relative-improvement signal to help single-image tasks, and report that it does not hurt them.This scope is stated for monocular depth on NYUv2 and semantic segmentation on ADE20K under a frozen-backbone protocol.
  • Single-image probes: Gekko and CroCo tie on single-image ImageNet-1K classification, with 61.3 top-1 for Gekko-Bmix versus 61.0 for its matched CroCo-B control.On Taskonomy, Gekko-Bmix wins three dense-task columns but loses five to the much larger released CroCo v2-L and MuM-L.
  • Evaluation protocol: Fine-tuning the whole network measures initialization value rather than the frozen-backbone property reported by the main paper, and can rank models differently.
  • Full-network fine-tuning: On MPI-Sintel optical flow, Gekko-Bmix obtains AEPE clean/final of 2.02/2.72 versus 2.21/2.59 for matched CroCo-B, while released CroCo v2-L leads.
  • Full-network fine-tuning: With full-network fine-tuning on 7-Scenes, Gekko-Bmix reaches 6.16 cm/1.90° versus 7.70 cm/2.08° for matched CroCo-B, while released ViT-L models remain ahead.The reported improvements are 20% on position and 9% on orientation; the authors do not claim a win over the released larger models.
  • Pointmap regression: Table 12 reports Gekko outperforming CroCo across pointmap benchmarks, including out-of-domain evaluation on DL3DV and ETH3D.
  • Representation visualization: PCA visualizations compare encoder-token components as RGB across both views, with consistent colors indicating similar features for corresponding 3D points.The out-of-domain visualization includes DL3DV, Cambridge Landmarks, ETH3D, 7-Scenes, and ScanNet; Gekko retains matching colors across viewpoint changes.
  • Ablation visualization: Figure 7 shows that the relative improvement loss produces sharper co-visibility maps than alternative ablation configurations.
Loading 2609.01530v1…