PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Multi-View Geometry Through Self-Distillation

David Nordström, Thibaut Loiseau, Vincent Lepetit, Michael Felsberg, Guillaume Bourmaud, Fredrik Kahl

arXiv:2609.39227v1cs.CVcs.AI

TL;DR

Multi-view SSL approaches use RGB reconstruction that entangles appearance with geometry, motivating the question of transferring information across views without reconstruction. Poincar3 learns from unlabeled image sequences through multi-view self-distillation, combining image-level and masked-patch objectives with additional teacher views, without RGB reconstruction or 3D supervision. Poincar3 substantially improves over state-of-the-art single- and multi-view SSL approaches, including DINOv3, MuM, and Muskie, across 3D tasks such as relative pose, point-cloud, and correspondence estimation.

  • Problem

    Multi-view SSL approaches use RGB reconstruction that entangles appearance with geometry, motivating the question of transferring information across views without reconstruction.

    Source: §1 · pp. 1–2

  • Method

    Poincar3 learns from unlabeled image sequences through multi-view self-distillation, combining image-level and masked-patch objectives with additional teacher views, without RGB reconstruction or 3D supervision.

    Source: §1 · p. 2

  • Results

    Poincar3 substantially improves over state-of-the-art single- and multi-view SSL approaches, including DINOv3, MuM, and Muskie, across 3D tasks such as relative pose, point-cloud, and correspondence estimation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Poincar3 learns strong representations from unlabeled image sequences, with multi-view geometry emerging without labels as shown by zero-shot correspondences.

    Source: §1 · p. 2

    Training is fragile to changes in EMA coefficient, weight decay, and learning-rate scaling.

    Source: §5 · p. 9

Abstract

from arXiv · show

Over a century ago, Henri Poincar\'e argued that a motionless observer cannot acquire the notion of space. Yet, most visual representation learning methods operate on individual images, while those that leverage multiple views rely on RGB reconstruction, entangling geometry with appearance. We propose Poincar3, a self-supervised method that learns representations from multiple views through self-distillation instead of RGB reconstruction. We combine masked patch and image-level distillation with a teacher that observes additional views, enabling training from scratch without explicit 3D supervision. Poincar3 outperforms both previous single and multi-view self-supervised approaches such as DINOv3, MuM, and Muskie on correspondence estimation, camera pose estimation, and 3D reconstruction. Using a lightweight Poincar\'e adapter, we also find that our learned features encode camera motion more accurately than existing self-supervised representations.

1 INTRODUCTION

The introduction asks whether representations can transfer information across views without RGB reconstruction, which entangles appearance with geometry. Poincar3 uses multi-view self-distillation and reports broad gains on 3D tasks, with geometric capabilities emerging without labels.

1 INTRODUCTION

  • RGB reconstruction across views entangles appearance with geometry, while downstream 3D tasks benefit from representations robust to appearance variation.The introduction asks whether information can be transferred across views without RGB reconstruction.Source: §1 · pp. 1–2
  • Poincar3 substantially improves over DINOv3, MuM, and Muskie across relative pose, point-cloud, and correspondence estimation.Figure 1 presents the main results across multi-view geometric tasks.Source: Figure 1 · p. 1, §1 · p. 2
  • Poincar3 trains a multi-view teacher–student objective from scratch without 3D supervision or RGB reconstruction.The method is presented as a new self-supervised objective for learning from unlabeled image sequences.Source: §1 · p. 2
  • Stable multi-view self-distillation relies on image-level distillation, additional teacher views, and retaining full images rather than local and global crops.Source: §1 · p. 2
  • The learned representations exhibit zero-shot multi-view correspondence and lightweight SE(3) decodability via a Poincaré adapter, despite no correspondence or 3D supervision.Source: §1 · p. 2

2 RELATED WORK

Related work traces self-supervised vision from teacher–student and masked-image objectives to methods designed for 3D tasks. The paper distinguishes its from-scratch self-distillation approach from reconstruction-based and supervised-pretraining pipelines.

2 RELATED WORK

  • Vision SSL developed through hand-crafted pretext tasks, clustering, contrastive learning, and EMA teacher–student frameworks, with DINOv3 combining EMA methods and masked image modeling.Source: §2 · pp. 2–3
  • CroCo, MuM, and Muskie use masked-image reconstruction across views, while An et al. (2025) showed these models can perform zero-shot correspondence estimation.Source: §2 · p. 3
  • Unlike VGGT-Ω's pipeline, which requires supervised 3D pretraining before SSL post-training, Poincar3 learns from scratch and is competitive with supervised 3D features for zero-shot correspondence.Source: §2 · p. 3

3 METHOD

Poincar3 learns dense, 3D-aware features from image sequences through teacher–student self-distillation rather than RGB reconstruction. Its method combines masked-patch and image-level objectives with additional teacher views, a multi-view transformer, and self-supervised training on image sequences.

3.1 NOTATIONS

  • Dense patch features represent each image as N patch vectors of dimension C, while the backbone also produces a C-dimensional global representation for each image.The features are intended to capture scene 3D structure for downstream 3D vision tasks.Source: §3.1 · p. 4

3.2 MULTI-VIEW SELF-DISTILLATION

  • Poincar3 replaces RGB reconstruction with multi-view self-distillation, addressing the poor performance of a naive multi-view extension of single-image self-distillation.The method aligns masked patch predictions and image-level representations; the teacher also receives extra views that provide geometric context.Source: §3.2 · pp. 4–5
  • The student sees masked images and learns from teacher predictions on masked patches, while an image-level objective additionally aligns student and teacher global representations.The teacher observes full images; the additional T frames are excluded from loss computations and supply geometric context through multi-view attention. Sinkhorn-Knopp and KoLeo regularization are used to prevent representational collapse, with α = 0.5 and β = 0.1.Source: §3.2 · p. 5

3.3 NETWORK ARCHITECTURE

  • The model uses a ViT-L image encoder and a multi-view transformer decoder that propagates information across views through alternating frame-wise and global attention.The default decoder has 12 layers and hidden dimension 1024; the model has around 650M trainable parameters.Source: §3.3 · p. 5

3.4 TRAINING

  • Poincar3 is trained from scratch on unlabeled image sequences, with sequence lengths sampled from 2 to 24 views and 400k AdamW training steps.Training data includes internet-video collections and 3D-annotated datasets; downstream performance improves steadily during training, and features become increasingly consistent across views compared to RGB reconstruction.Source: §3.4 · p. 6

4 RESULTS

Poincar3 is evaluated on feed-forward reconstruction, zero-shot multi-view correspondence, camera-motion decoding, and design ablations. The results report stronger performance on geometric tasks, including improved correspondence with additional training data.

4.1 FEED-FORWARD 3D RECONSTRUCTION

  • Poincar3 substantially improves feed-forward reconstruction over existing models, especially for camera pose; its weights reach 65.0%+ AUC@30° on RE10K after 10K finetuning steps.The reported finetuning uses a low learning rate.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 MULTI-VIEW CORRESPONDENCE ESTIMATION

  • The attention map achieves a top-accuracy of 94.9, and Poincar3 is more accurate than other SSL methods and feed-forward reconstruction models on multi-view correspondence.The evaluation samples sequences of eight images and reports the best-performing layer for fair comparison.Source: §4.2 · pp. 6–7, Table 2 · p. 8

4.3 EMERGENT SE(3) STRUCTURE

  • Poincar3 features yield the strongest camera-motion results with a lightweight Poincaré adapter, outperforming features obtained from one view.The adapter is fitted per scene on 20 held-out ScanNet++ scenes.Source: §4.3 · pp. 7–8, Table 3 · p. 9

4.4 ABLATION STUDIES

  • Objective V outperforms multi-view RGB reconstruction, while the VGGT-Ω(SSL) MSE loss does not work from scratch as self-supervision.The supervision comparison keeps architecture fixed and uses a four-H200, one-day compute budget.Source: §4.4 · pp. 8–9, Table 4 · p. 9, Figure 10 · p. 9
  • Using all available data raises attention-map PCK@50 from 87.5 to 94.9 on ScanNet and from 78.5 to 86.5 on NAVI.The comparison is against training on only 3D-labeled data.Source: §4.4 · p. 9

5 LIMITATIONS

The authors note a trade-off in task emphasis and constraints in compute and training robustness. Frame selection within videos is also handcrafted.

5 LIMITATIONS

  • The approach prioritizes geometric over semantic performance, and the authors lack resources to match DINOv3’s model size, using around 100× less compute.The paper still reports significant outperformance of previous approaches.Source: §5 · p. 9
  • Representations degrade when EMA coefficient or weight decay deviates from DINOv3 settings, and adapting learning rate to batch size requires following prior work.The authors identify mitigating this fragility as future work.Source: §5 · p. 9
  • Frame selection within video sequences remains handcrafted; automating it is identified as future work.Source: §5 · p. 9

6 CONCLUSIONS

Poincar3 improves 3D-task performance over prior self-supervised methods, while the authors report instability with alternative objectives and greater sensitivity to low-quality data. The paper’s comparisons focus on selected self-supervised and supervised baselines.

6 CONCLUSIONS

  • Poincar3 improves substantially over MuM and Muskie on 3D vision tasks and achieves higher performance than DINOv3 using 100× less compute.The authors conclude that it learns strong 3D representations from scratch without explicit 3D supervision or pixel reconstruction.Source: §6 · p. 10

A THINGS THAT DID NOT WORK

  • BYOL-style alternatives degraded during training, and neither normalization choices nor more sophisticated regularization stabilized them.The authors report that feature maps quickly degraded and qualitative results were unsatisfactory.Source: A THINGS THAT DID NOT WORK · p. 16
  • Self-distillation appeared more sensitive to low-quality data samples than RGB reconstruction, motivating careful data curation.The authors suggest this may explain the DINO team’s investment in elaborate curation pipelines.Source: A THINGS THAT DID NOT WORK · p. 16

B BASELINES

  • Comparisons center on DINOv3, MuM, and Muskie; CroCo is omitted as already inferior to MuM and Muskie, while GenCeption and Alligat0r are excluded for stated method constraints.Supervised 3D methods are included where the authors consider comparisons fair, but omitted from task-specific feed-forward reconstruction comparisons.Source: B BASELINES · p. 16

C ADDITIONAL EXPERIMENTS

The additional experiments visualize learned features and compare two-view matching performance. Poincar3 leads the reported zero-shot and linear-probe matching evaluations.

C.1 QUALITATIVE

  • PCA visualizations encode the three principal components of patch features as RGB across sequences from the same scene.The authors also visualize Poincaré-adapter alignment with ground-truth camera trajectories and additional multi-view correspondence tracks.Source: §C.1 · p. 16, Figure 12 · p. 17

C.2 TWO-VIEW MATCHING

  • Poincar3 outperforms all other methods on both zero-shot matching and linear probing on ScanNet-1500.The reported evaluation uses a lightweight protocol with linear probes and zero-shot nearest neighbors in feature space.Source: §C.2 · p. 16, Table 5 · p. 17

C.3 SEMANTIC TASKS

Without ImageNet training, Poincar3 outperforms other multi-view self-supervised models on the reported semantic tasks, while its semantic performance is lower than DINOv3 under the stated training changes. Its correspondence performance remains strong across network depths.

C.3 SEMANTIC TASKS

  • Without ImageNet training, Poincar3 outperforms other multi-view SSL models on ImageNet-1K classification and ADE20K semantic segmentation.Removing local crops and training on substantially more geometric data degrades performance relative to DINOv3; the comparison uses final-layer outputs for all models, while MuM is evaluated after its encoder.Source: §C.3 · p. 17, Table 6 · p. 18

C.4 LAYER-WISE MULTI-VIEW CORRESPONDENCE ESTIMATION

  • Poincar3 outperforms baselines at every tested depth, with multi-view geometry retained throughout its network while supervised methods collapse at later layers.The evaluation measures multi-view correspondence accuracy across network depths.Source: §C.4 · p. 18

D IMPLEMENTATION DETAILS

The implementation combines a ViT-L/16 backbone with multi-view self-distillation and a teacher that sees additional frames. Training uses a mixture dominated by internet video scenes, with no overlap between training and evaluation scenes.

D.1 HYPER-PARAMETERS

  • Poincar3 uses a ViT-L/16 backbone and a 12-block multi-view decoder alternating full cross-frame attention with register-bottleneck attention at blocks 2, 6, and 9.The decoder’s register-bottleneck attention uses 16 register tokens; the backbone is initialized from scratch.Source: §D.1 · p. 18
  • Separate DINO/iBOT heads distill patch and global tokens, with Sinkhorn–Knopp-normalized teacher targets and a warmed-up teacher temperature and EMA decay.Each head uses 65,536 prototypes; the final layer of both heads is frozen for the first 2k steps.Source: §D.1 · p. 18
  • Poincar3’s multi-view correspondence performance exceeds baselines at different depths, and the figure describes geometry as encoded throughout its network.The figure caption contrasts this with supervised methods whose performance collapses at later layers.Source: Figure 14 · p. 19
  • Student sequences contain 2–24 frames, while the teacher receives 0–12 additional unseen frames; loss is computed only on frames both networks observe.Patches are masked independently per frame, and batches are dynamically assembled to a fixed budget of 64 frames per GPU.Source: §D.1 · p. 19
  • Training uses AdamW with a fixed 2 × 10−4 learning rate, 10k-step warmup, 0.04 weight decay, gradient clipping at 1.0, and bf16 mixed precision.Source: §D.1 · p. 19

D.2 DATA

  • Most training scenes come from internet videos, and none overlap with evaluation scenes.The authors describe internet videos as a rich source of data mined through self-supervision.Source: §D.2 · p. 19

D.3 TRAINING DYNAMICS

Training-dynamics plots track six evaluation metrics and DINOv3 initialization effects; stability metrics indicate that teacher features become more discriminative rather than collapsing.

D.3 TRAINING DYNAMICS

  • Teacher features become more discriminative during training rather than collapsing, as indicated by standard deviation and effective rank.Source: §D.3 · p. 19
  • Figure 15 tracks six evaluation metrics and the effect of initializing the encoder from DINOv3.Source: §D.3 · p. 19
  • Figure 16 visualizes loss-term evolution and the stability metrics tracked during training.Source: §D.3 · p. 19

E.1 FEED-FORWARD RECONSTRUCTION

This section describes feed-forward reconstruction training and evaluation, zero-shot correspondence testing, emergent camera-motion structure, and feature-correlation visualizations.

E.1 FEED-FORWARD RECONSTRUCTION

  • Table 7 separates internet-video datasets with noisy or no annotations from annotated 3D datasets, weighting each by its probability of being sampled.Source: Table 7 · p. 20

E.2 MULTI-VIEW CORRESPONDENCE ESTIMATION

  • Figure 15 compares multi-view correspondence metrics for scratch training versus DINOv3 initialization; evaluation tracks 100 points across eight covisible views using feature or attention matches and 2D PCK and 3D accuracy.Source: Figure 15 · p. 20, §E.2 · pp. 21–22

E.1 FEED-FORWARD RECONSTRUCTION

  • Feed-forward models use camera and depth heads over a backbone, trained on posed, depth-annotated multi-view data with batches of 2–24 frames.Source: §E.1 · p. 20
  • The three protocols compare heads-only training with a frozen backbone, a frozen backbone plus adapter, and full finetuning, which excludes MuM and Muskie as architecturally unfair comparisons.The adapter adds cross-view capacity; full finetuning may use feature distillation against a frozen VGGT-Ω teacher.Source: §E.1 · pp. 20–21
  • Training combines camera and depth losses; the depth loss uses confidence weighting, plain regression, and multiscale gradient matching on valid pixels after dropping the worst 2%.Depth and camera translations are rescaled per batch because monocular absolute scale is unobservable.Source: §E.1 · p. 21
  • Pose evaluation averages AUC@30 across scenes and four test sets, while point-cloud quality is evaluated on five-view DTU and ETH3D sequences.Predicted point clouds are aligned to ground truth before accuracy, completion, and normal consistency are measured.Source: §E.1 · p. 21

E.3 EMERGENT SE(3) STRUCTURE

  • Per-scene Poincaré adapters are fit on 20 held-out ScanNet++ scenes using chronological 80/20 frame splits; results include clipped mean R2 and fractions of scenes above zero and 0.3.Source: §E.3 · p. 22

E.4 VISUALIZING FEATURE CORRELATIONS

  • Feature-correlation visualizations rescale each backbone row independently, clarifying within-backbone dispersion while sacrificing absolute cross-row intensity comparisons.Source: §E.4 · p. 22

Related papers

Loading 2609.39227v1…