PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Cross-View Completion: Self-Supervised Pre-Training via Reconstruction Error Comparison

Thibaut Loiseau, Guillaume Bourmaud, Vincent Lepetit

arXiv:2609.01530v1cs.CV

TL;DR

Cross-view completion becomes implicitly monocular when masked regions are not co-visible in the reference view. Gekko compares cross-view and MAE reconstruction errors, jointly predicts their relative improvement, and under matched settings consistently outperforms CroCo on cross-view 3D tasks while supporting raw-video training.

  • Problem

    Cross-view completion provides little binocular information for non-co-visible patches, limiting its training signal on image pairs with low overlap.

    Source: §1 · p. 2

  • Method

    Gekko jointly performs cross-view completion, masked autoencoding, and per-pixel prediction of relative reconstruction improvement without ground-truth 3D labels.

    Source: §1 · p. 2

  • Results

    Under identical architectures and training data, Gekko consistently outperforms CroCo across relative pose estimation, correspondence estimation, and pointmap regression.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Gekko supplies a binocular signal for all masked regions, detects co-visibility on unseen scenes, and can be trained from raw videos while matching curated-data models.

    Source: §1 · p. 2

    Gekko costs more to pre-train than CroCo, with approximately 2.4× higher GPU-hours for matched Base pre-training runs.

    Source: §5.9 · p. 8

Abstract

from arXiv · show

Self-supervised pre-training via cross-view completion learns strong features for 3D vision from co-visible regions of image pairs. However, the reference view provides little information for reconstructing non-co-visible patches, implicitly yielding a monocular training signal in these regions. We introduce Gekko, which turns this limitation into a useful signal. The relative improvement of the cross-view reconstruction error over a masked-autoencoder error is a self-supervised proxy for co-visibility: large improvements indicate co-visible regions, negligible ones non-co-visible areas. Gekko is a network, trained from scratch, that jointly performs cross-view completion, masked autoencoding, and per-pixel prediction of this relative improvement, providing an additional binocular signal for all masked regions without any ground-truth 3D annotation. Under identical architectures and training data, Gekko consistently outperforms CroCo on zero-shot correspondence estimation, relative pose estimation, and pointmap regression, with up to 6 times higher accuracy at the strictest relative-pose threshold and a 22% drop in end-point error on ETH3D. The extra channel it learns is itself a strong co-visibility detector on unseen scenes, and Gekko's frozen features outperform released cross-view backbones of comparable or larger size. It can also be trained directly from raw videos with a simple stride-based curriculum, removing the cumbersome 3D preprocessing prior methods require while matching models trained on curated data. Code and pre-trained models are publicly available.

1 Introduction

Gekko addresses the monocular limitation of cross-view completion by predicting relative improvement between cross-view and masked-autoencoder reconstruction. It adds this binocular signal while preserving CroCo’s architecture and improves cross-view 3D evaluation, including raw-video training.

  • Cross-view completion becomes implicitly monocular when masked target patches are not co-visible in the reference image.Source: §1 · p. 2
  • Gekko uses relative improvement between cross-view and MAE reconstruction errors as a self-supervised co-visibility proxy without ground-truth 3D labels.Source: §1 · p. 2
  • Gekko jointly performs cross-view completion, masked autoencoding, and relative-improvement prediction with one additional output channel.Its encoder and cross-attention decoder remain byte-for-byte identical to CroCo.Source: §1 · p. 2
  • Under identical architectures and training data, Gekko outperforms CroCo across relative pose estimation, correspondence estimation, and pointmap regression.Reported gains include up to 6× higher accuracy at the strictest relative-pose threshold and a 22% drop in ETH3D end-point error.Source: §1 · p. 2
  • Gekko can use raw videos with a stride-based curriculum, matching curated-data models while removing prior 3D preprocessing requirements.Its predicted channel also detects co-visibility on unseen scenes, and its frozen features outperform comparable or larger released cross-view backbones.Source: §1 · p. 2

2 Related work

The paper situates Gekko between single-image self-supervised learning and cross-view image-pair methods, addressing their shared monocular signal in non-co-visible regions without relying on ground-truth overlap.

  • Masked image modeling and self-distillation learn visual features from single images without manual labels.Source: §2 · p. 3
  • CroCo extends masked image modeling to image pairs, while later cross-view methods build on this framework for 3D reconstruction and matching.Source: §2 · p. 3
  • Cross-view completion methods share an implicitly monocular training signal where regions are not co-visible across views.Gekko explicitly models relative improvement over monocular reconstruction to provide an additional binocular signal.Source: §2 · p. 3
  • Co-visibility masks are commonly derived from depth maps and camera poses to select training pairs or learn matchability scores.Source: §2 · p. 3

3 Background and motivation

CroCo and MAE provide complementary reconstruction baselines for comparing cross-view benefit. Their relative improvement is more informative for co-visibility than CroCo’s error alone, though it remains an approximation.

  • CroCo reconstructs masked target patches using the target image and a reference image, whereas MAE reconstructs them from the masked target alone.Source: §3.1 · p. 3
  • CroCo benefits from reference information in co-visible regions but degrades where the reference contributes little.Source: §3.1 · p. 3
  • CroCo’s reconstruction error alone is unreliable for predicting co-visibility because its co-visible and non-co-visible distributions largely overlap.Source: §3.2 · p. 4

3.2 Relative improvement of CroCo over MAE

  • The relative improvement C(p) compares MAE and CroCo reconstruction errors, approaching 1 for co-visible regions and 0 for non-co-visible regions.The comparison uses MAE as a reference-independent reconstruction baseline.Source: §3.2 · p. 4
  • Relative improvement reaches AP 0.74 for co-visibility prediction, compared with AP 0.57 for CroCo’s error alone.Source: §3.2 · p. 4, Figure 2 · p. 4
  • C(p) approximates co-visibility rather than perfectly classifying it because low MAE error can make the ratio uninformative, including in repeated or self-similar regions.Source: §3.2 · p. 4

4 Method

Gekko turns cross-view reconstruction improvement into a self-supervised co-visibility signal and jointly learns completion, masked autoencoding, and relative-improvement prediction without 3D annotations. Its learned maps correlate with ground-truth co-visibility labels.

  • Gekko uses relative reconstruction improvement as pseudo co-visibility labels to improve CroCo pre-training without 3D annotations.Source: §4 · pp. 4–5
  • The cross-view pass reconstructs the target from its masked patches and a reference image, while the masked-autoencoder pass uses the same mask without the reference image.Source: §4.1 · p. 5
  • Gekko jointly performs cross-view completion, masked autoencoding, and prediction of the relative improvement between their reconstruction errors.Source: §4.1 · p. 5
  • Gekko adds a fourth pixel-decoder channel for predicting the relative improvement map while retaining CroCo’s ViT encoder and cross-attention decoder.Source: §4.2 · p. 5
  • Down-weighting pixels with low masked-autoencoder loss improves performance and produces sharper co-visibility maps.Source: §4.3 · p. 5
  • Gekko’s predicted relative-improvement maps correlate strongly with ground-truth co-visibility labels, though they are less sharp in uniform regions.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate Gekko under matched and curriculum-based settings across correspondence, pose, pointmap, and co-visibility tasks, while also examining scaling, ablations, cost, and scope limits.

Zero-shot correspondence estimation

  • 22% lower AEPE for encoder features and 21% for decoder features on ETH3D over CroCo in zero-shot correspondence estimation.Features are evaluated immediately after pre-training without fine-tuning; all models are Base and pre-trained on ScanNet-50.Source: §5.2 · p. 6, Table 1 · p. 6

Relative metric pose estimation

  • Nearly 6× higher strict-threshold accuracy than CroCo-B on ScanNet-1500 relative metric pose estimation, with the gap widening on ScanNet-all.On ScanNet-all, Gekko-B reaches 43.7% versus 6.6% for CroCo-B.Source: §5.3 · p. 7, Table 3 · p. 7
  • 35.9% versus 7.5% at 10°/0.25m on ScanNet-all gives Gekko-L a 4.8× improvement over CroCo-L, showing gains scale to larger architectures.Both Large models use the same architecture, data, batch size, and 100k-step schedule.Source: §5.3 · p. 7

Curriculum pre-training

  • 31.2% versus 29.1% at 10°/0.25m shows raw-video Gekko matches its curated-data counterpart under stride-based curriculum training.The raw-video setting uses no 3D annotation or overlap-based preprocessing, while the exact training pairs differ.Source: Table 2 · p. 6, §5.4 · p. 7

Pointmap regression

  • 10% lower in-domain and 5–7% lower out-of-domain Chamfer error in pointmap regression show gains across ScanNet, DL3DV, and ETH3D.The evaluation uses a DPT head on frozen features, with models pre-trained on DL3DV using curriculum and fine-tuned on ScanNet-all.Source: §5.5 · p. 7

Co-visibility classification

  • 0.763 AP versus 0.576 for CroCo demonstrates that Gekko’s predicted channel detects co-visibility on ScanNet-1500.The channel also reaches 0.691 balanced accuracy at one global threshold and performs relatively better on the lowest-overlap tertile.Source: §5.6 · p. 8

Ablation studies

  • Adding MAE without relative-improvement prediction barely changes results, supporting the relative-improvement loss as the source of the ablation gain.The relative formulation outperforms the absolute one, and down-weighting low-MAE pixels is reported to improve results by 18.0 points.Source: §5.8 · p. 8

Training cost

  • Gekko’s extra full-resolution prediction pass roughly doubles activation memory, halves per-device batch size, and raises matched Base pre-training cost to approximately 2.4× CroCo.The additional cost is confined to pre-training; fine-tuning and inference cost what CroCo costs.Source: §5.9 · pp. 8–9

6 Conclusion

Gekko addresses the monocular signal in non-co-visible regions by predicting relative cross-view improvement, adding binocular supervision without ground-truth 3D labels. It outperforms CroCo on evaluated cross-view tasks and can train directly from raw video.

6 Conclusion

  • Gekko uses relative improvement between cross-view and MAE reconstruction errors as a self-supervised co-visibility proxy.Large improvement indicates co-visible regions, while near-zero improvement indicates non-co-visible regions.Source: §6 · p. 10
  • Gekko jointly performs cross-view completion, masked autoencoding, and relative improvement prediction with one extra output channel.The predicted signal applies to all masked regions without ground-truth depth or pose supervision.Source: §6 · p. 10
  • Gekko consistently outperforms CroCo on the evaluated cross-view tasks under matched architecture and data.The evaluations cover correspondence estimation, relative pose estimation, and pointmap regression.Source: §6 · p. 10
  • Gekko trains directly from raw video with a stride curriculum, removing the 3D preprocessing required by prior work.The conclusion states that raw-video training matches models trained on curated data.Source: §6 · p. 10

C Training details

The training setup follows CroCo’s ViT-based framework, using fixed pre-training and task-specific fine-tuning configurations. Raw reconstruction behavior differs by visibility: CroCo is sharper in co-visible regions and blurred elsewhere.

Pre-training

  • All models are trained from scratch for 100k steps with batch size 768, learning rate 1.5 × 10^-4, warmup, cosine decay, 90% masking, and normalized patches.Images are resized to a maximum dimension of 512 pixels while maintaining aspect ratio.Source: C Training details · p. 16

Fine-tuning

  • Relative metric pose estimation adds an MLP head over frozen decoder features and trains it for 20k steps.Fine-tuning uses a homoscedastic loss and the all-overlap variant.Source: C Training details · p. 16
  • Pointmap regression fine-tunes a DPT head over frozen features using a confidence-aware regression loss on ScanNet-all.It uses the same batch size and learning rate as pose estimation.Source: C Training details · p. 16

Architectures

  • The Base model uses a 12-layer ViT encoder and 8-layer transformer decoder, while the Large model uses 24 and 12 layers respectively.Base and Large embedding sizes are 768/512 and 1024/768 for encoder/decoder components.Source: D Architecture details · p. 17

Qualitative behavior

  • CroCo reconstructions are sharper in co-visible regions and blurred in non-co-visible regions.The reference image supplies useful warped information only where the views overlap.Source: Figure 4 · p. 17

F Detailed ablation analysis

Ablations show that relative improvement prediction, rather than MAE joint training alone, drives Gekko’s gains, while formulation and normalization choices materially affect performance.

Impact of the relative improvement loss

  • 20.9% versus 18.5% at 10°/1m shows that adding MAE without relative improvement yields only marginal improvement over CroCo.The ablation identifies relative improvement prediction as the key contribution.Source: F Detailed ablation analysis · p. 17

Confidence prediction vs. relative improvement

  • 45.8% at 10°/1m on ScanNet-50 falls to 11.2% on ScanNet-all for confidence-aware prediction.Confidence maps also lack the sharp co-visibility boundaries achieved by relative improvement.Source: F Detailed ablation analysis · p. 17

Relative vs. absolute improvement

  • 39.4% versus 57.6% on ScanNet-all shows that absolute improvement underperforms the relative formulation.The relative formulation produces sharper maps that better capture co-visibility boundaries.Source: F Detailed ablation analysis · p. 18

Impact of patch normalization

  • 25.1% versus 39.4% on ScanNet-all shows a significant performance drop without target patch normalization.Removing normalization makes predictions noisier in uniform regions and destabilizes relative improvement estimates.Source: F Detailed ablation analysis · p. 18

Loss formulation

  • 25.7% to 43.7% at 10°/.25m on ScanNet-all results from multiplying the prediction by the denominator instead of directly regressing the ratio.This is the final loss formulation described in Eq. 8.Source: F Detailed ablation analysis · p. 18

G Ablation study: technical details

The technical ablations define alternative losses and evaluate co-visibility prediction on raw-video mixtures and held-out benchmarks. Protocol details clarify masking, labels, thresholds, and comparisons with released backbones.

Loss definitions

  • The relative-improvement loss predicts the ratio-based co-visibility signal, while alternative losses reweight it, predict absolute improvement, or estimate precision.The confidence-aware formulation expects precision to be higher in co-visible regions and lower in non-co-visible regions.Source: G Ablation study: technical details · p. 18

Raw-video training

  • Raw-video mix training samples frame pairs by stride without structure-from-motion, depth, camera pose, or overlap estimates.The mix uses twelve video sources spanning real and synthetic indoor, outdoor, and object-centric data.Source: H The 12-source raw-video mix · p. 18

Evaluation protocol

  • Co-visibility evaluation reads the fourth output channel directly without post-processing or test-time augmentation.Ground truth uses reprojection and depth agreement within 5%, excluding pixels landing in reference-depth holes.Source: I Co-visibility probing protocol · p. 19
  • The evaluation covers all 1500 ScanNet-1500 pairs and 558 retrieval pairs from 7-Scenes, using threshold-free average precision and ROC-AUC.Balanced accuracy uses one fitted global threshold per dataset and score.Source: I Co-visibility probing protocol · p. 19

Evaluation caveats

  • Restricting co-visibility evaluation to SIFT keypoints raises every score and improves raw signals more than the prediction.The reported prediction change is 0.763 → 0.790, compared with 0.576 → 0.587 for ℓCroCo.Source: I Co-visibility probing protocol · p. 19

Released backbones

  • The self-supervised released-backbone comparison is not data-matched, and VGGT is a supervised ∼1B reference with overlapping training data.Matched architecture-and-data comparisons remain the controlled evidence for objective comparisons.Source: J Probing protocol for Table 7 · p. 19

K Data scaling on the raw-video mix

Data scaling experiments test whether Gekko’s relative-improvement signal remains beneficial as the raw-video training mix grows. Gekko leads across the evaluated data fractions, although fixed-step pose performance can dip when larger datasets are undertrained.

Experimental control

  • The experiment varies only the fraction of mix scenes, while sources, sampling weights, resolution, batch size, step count, and fine-tuning budgets remain matched.The comparison uses nested, seeded, scene-level subsets from 1% to 100%.Source: K Data scaling on the raw-video mix · p. 19

Data scaling

  • Gekko leads at every evaluated data scale, with strictest-threshold pose gains ranging from +7.4 to +24.1 points.The matched Base models use nested scene-level subsets from 1% to 100% of the mix, with identical sampling, optimization, and fine-tuning budgets.Source: K Data scaling on the raw-video mix · p. 19
  • Above 50% data, both methods dip on pose at the fixed 100k-step budget, indicating undertraining rather than harm from additional data.Extending the best mix model to 200k steps recovers pose scores of 39.4/56.1/61.1 and a pointmap Overall of 0.092.Source: K Data scaling on the raw-video mix · p. 19

Data composition

L Generality across architectures

Gekko’s advantage is not confined to CroCo’s original encoder–decoder design. Replacing either architectural half preserves a positive advantage over matched CroCo controls.

Experimental design

  • The architecture substitution study pre-trains six Base models from scratch on ScanNet-50 for 100k steps while varying architecture and holding other settings fixed.This tests whether the gain could be an artifact of CroCo’s specific encoder–decoder.Source: L Generality across architectures · p. 20

Evaluation

  • Table 10 reports matched Base-model comparisons on ScanNet-1500 pose and ScanNet pointmap Chamfer Overall after architectural substitutions.Source: Table 10 · p. 20

Architecture substitutions

  • Gekko’s advantage survives both decoder and encoder substitutions, with gains of +17.2 and +18.9 points versus +23.6 for the paper’s architecture.The joint self-attention decoder removes cross-attention, while the DINOv3 variant freezes an off-the-shelf monocular encoder.Source: L Generality across architectures · p. 20
  • CroCo remains nearly unchanged across the three architectures, moving from 5.0 to 5.7, whereas Gekko changes substantially.Source: L Generality across architectures · p. 20

M Single-image probes

Single-image probes show that Gekko’s binocular objective does not provide a general advantage on monocular tasks, while broader evaluations distinguish frozen-feature transfer from full-network fine-tuning and test representation consistency across domains.

Single-image probes

  • The authors do not expect the binocular relative-improvement signal to help single-image tasks, and report that it does not hurt them.This scope is stated for monocular depth on NYUv2 and semantic segmentation on ADE20K under a frozen-backbone protocol.Source: M Single-image probes · p. 20
  • Gekko and CroCo tie on single-image ImageNet-1K classification, with 61.3 top-1 for Gekko-Bmix versus 61.0 for its matched CroCo-B control.On Taskonomy, Gekko-Bmix wins three dense-task columns but loses five to the much larger released CroCo v2-L and MuM-L.Source: M Single-image probes · p. 20

Evaluation protocol

  • Fine-tuning the whole network measures initialization value rather than the frozen-backbone property reported by the main paper, and can rank models differently.Source: N Full-network fine-tuning · p. 21

Full-network fine-tuning

  • On MPI-Sintel optical flow, Gekko-Bmix obtains AEPE clean/final of 2.02/2.72 versus 2.21/2.59 for matched CroCo-B, while released CroCo v2-L leads.Source: N Full-network fine-tuning · p. 21
  • With full-network fine-tuning on 7-Scenes, Gekko-Bmix reaches 6.16 cm/1.90° versus 7.70 cm/2.08° for matched CroCo-B, while released ViT-L models remain ahead.The reported improvements are 20% on position and 9% on orientation; the authors do not claim a win over the released larger models.Source: N Full-network fine-tuning · p. 21

Pointmap regression

  • Table 12 reports Gekko outperforming CroCo across pointmap benchmarks, including out-of-domain evaluation on DL3DV and ETH3D.Source: Table 12 · p. 21

Representation visualization

Ablation visualization

  • Figure 7 shows that the relative improvement loss produces sharper co-visibility maps than alternative ablation configurations.Source: Figure 7 · p. 24

Related papers

Loading 2609.01530v1…