PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

Basavaraj Sunagad, Artur Jesslen, Adam Kortylewski

arXiv:2609.28187v1cs.CV

TL;DR

The roles of DINO-style training components in producing semantic representations have not been cleanly isolated. Through controlled retraining and evaluation, the paper finds that same-instance global-view alignment anchors semantic structure, while correspondence metrics better track dense downstream performance than classification probes.

  • Problem

    DINO-style recipes combine multiple components, but their individual contributions to semantic representation quality, rather than classification accuracy, have not been cleanly isolated.

    Source: §1 · p. 1

  • Method

    The authors systematically ablate crop configurations, patch-level prediction, and crop scale, evaluating classification, correspondence, and dense downstream tasks.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Same-instance global-view alignment anchors semantic structure; iBOT refines it by 10.5 PCK points, while local crops benefit classification but not correspondence.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Semantic and geometric correspondence metrics track downstream performance more faithfully than classification accuracy and should complement it in SSL evaluation.

    Source: §1 · p. 3, §6 · p. 9

    The conclusions are specific to DINO-style teacher–student SSL; whether the same inductive bias governs other SSL families remains open.

    Source: §5 · p. 9

Abstract

from arXiv · show

Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, yet the mechanisms underlying this behavior remain unclear. We present a systematic empirical dissection of the DINO family and show that semantic representations arise primarily from enforcing consistency between geometrically distinct global views of the same image instance. This instance-specific global alignment acts as the semantic anchor of DINO-style learning. Across controlled retraining experiments evaluated on semantic correspondence and a diverse suite of 2D and 3D downstream tasks, we find that patch-level masking objectives enhance semantics only when trained jointly with this global alignment, indicating that the iBOT objective refines and densifies existing semantic structure rather than creating it independently. In contrast, local-to-global view alignment does not substantially improve semantic qualities at fixed compute beyond a purely global alignment. Beyond training design, we revisit how semantic representation quality should be evaluated: while classification accuracy is the standard validation score, semantic correspondence provides a complementary axis that more reliably predicts downstream task performance. Together, these findings provide a functional decomposition of DINO-style learning and represent an important step toward understanding how semantic representations emerge in self-supervised vision models.

1 Introduction

DINO-style recipes combine several objectives and view configurations, but their individual roles in semantic representation quality have not been cleanly isolated. Controlled experiments identify same-instance global-view alignment as the semantic anchor, while correspondence metrics complement classification when evaluating dense-task utility.

1 Introduction

  • The unresolved question is whether global-global agreement creates semantic structure, or whether local-to-global alignment and patch-level prediction create or refine it.DINO-style recipes combine multi-crop views, masked patch prediction, augmentation design, and momentum distillation, whose individual effects on semantic quality had not been cleanly isolated.Source: §1 · pp. 1–2
  • Two global crops of the same image already yield part-consistent structure; DINOv1 with local crops closely matches this minimal setup, while adding iBOT yields richer structure.Figure 1 reports PCK@.1 of 31.2 for DINO minimal, 32.0 for DINOv1, and 42.5 for DINOv2.Source: Figure 1 · p. 2
  • The study isolates recipe components through controlled pretraining ablations and evaluates classification, correspondence, and dense tasks spanning depth, pose, tracking, and segmentation.The evaluation includes SPair-71k and NAVI correspondence alongside downstream tasks on NYU, ImageNet3D, TAPVid, and ADE20K.Source: §1 · p. 2
  • Semantic and geometric correspondence track dense downstream performance more faithfully than kNN or linear probes, making them complementary evaluation axes alongside classification.Classification and dense correspondence have opposing crop-scale optima: s=0.40 versus s=0.60–0.72.Source: §1 · p. 3
  • Same-instance global-view alignment is sufficient for strong semantic patch features; local crops and iBOT refine, but do not create, this structure.Removing geometric displacement or same-instance identity collapses semantic correspondence.Source: §1 · pp. 2–3

2 Related Work

Prior work leaves open which parts of DINO-style recipes create semantic structure and which refine representations or improve other measures. This section frames correspondence as a diagnostic for that question.

Self-supervised learning and the two-view paradigm

  • Two-view agreement is a starting point, but the roles of geometric displacement, same-instance identity, and global contrastive loss remain to be distinguished.Prior methods report competitive representations without negative pairs, while redundancy-reduction methods offer another route to avoiding collapse.Source: §2 · p. 3

Multi-crop and local-global view strategy

  • Local-to-global alignment had not been isolated as a source of semantic quality, and removing local crops leaves correspondence largely unchanged.The ablation challenges the assumption that local-to-global matching drives semantic emergence.Source: §2 · p. 3

Patch-level objectives and masked image modeling

  • iBOT refines rather than creates semantic structure: removing it degrades correspondence, while iBOT alone yields near-zero correspondence.Token-level supervision is often interpreted as the primary driver of dense semantic correspondence.Source: §2 · p. 3

Simplifying self-supervised learning pipelines

  • Recipe-simplification work asks which components can be removed without accuracy loss; this paper instead diagnoses which components create semantic structure.The questions are orthogonal because a component may be replaceable yet remain the causal origin of a capability.Source: §2 · pp. 3–4

Probing learned representations

  • Correspondence has mostly served as a post-hoc capability measure, while this work uses it to analyze training components and finds it predicts dense downstream performance more reliably than classification proxies.The diagnostic probes are SPair-71k [22] and NAVI.Source: §2 · p. 4

3 Probing Semantic Emergence in DINO-Family SSL

The paper describes DINO-family objectives, then varies losses and crop configurations to isolate their contributions. It evaluates frozen global features with classification probes and patch features with zero-shot correspondence metrics.

3.1 DINO-Family Pretraining

  • DINOv2 combines global-view [CLS] alignment with iBOT masked patch-level refinement, whose masked student features match unmasked teacher features.DINOv1 uses an EMA teacher and student; the teacher processes global views, while the student processes global and local views.Source: §3.1 · p. 4

3.2 Interventions

  • The objective comparison includes full DINOv2 (D+I), DINO only (D), and iBOT only (I), while view interventions vary global-crop count, local crops, and crop pairing.Crop pairing includes photometric-only views and cross-instance views; separate global-to-global sweeps vary random or fixed crop scale, and photometric augmentations are held fixed.Source: §3.2 · pp. 4–5

3.3 Evaluation Protocol

  • Classification uses frozen [CLS] features for ImageNet-1k kNN and linear-probe top-1; correspondence uses frozen patch features for SPair-71k PCK@0.1 and NAVI R@0.05m.The correspondence metrics are also evaluated as proxies for dense downstream performance.Source: §3.3 · p. 5

4 Experiments

The experiments separate classification quality from semantic correspondence: two same-instance global views provide the key alignment signal, while local crops and iBOT refine different aspects. Crop-scale results further show that correspondence aligns more closely with dense downstream performance than classification accuracy.

4.1 Model Size

  • Semantic correspondence nearly doubles from ViT-S/16 to ViT-L/14 (24.8→47.3), while ViT-g/14 does not improve over ViT-L/14 on correspondence.ViT-L/16 was selected for the ablations as the best-corresponding patch-16 model at a favorable compute-to-performance trade-off.Source: §4.1 · p. 5

4 Experiments

  • The experiments use a low-compute ViT-L/16 setup to make more than 20 ablation configurations feasible.The authors note that these configurations would be computationally prohibitive at full DINOv2 data scale.Source: §4.1 · p. 5

4.2 Intervention Ladder: Progressive Component Analysis

  • Removing iBOT reduces semantic correspondence from 42.5 to 32.0 without collapse, while removing local crops lowers classification but leaves correspondence nearly unchanged.Adding four global crops partially recovers the classification gap, with essentially unchanged correspondence.Source: §4.2 · p. 6

4.3 Necessity Tests: What Makes Global-View Contrast Work?

  • Necessity tests show that semantic emergence requires two geometrically displaced global views of the same image; iBOT alone cannot bootstrap semantic structure.Removing global-to-global contrast, spatial displacement, or same-instance identity each severely degrades semantic correspondence.Source: §4.3 · p. 7

4.4 Crop Scale Analysis: Classification and Correspondence Have Opposing Optima

  • With random scale, semantic correspondence peaks at smin = 0.60, while kNN peaks at smin = 0.24 and geometric correspondence at smin = 0.32.As smin reaches 0.60, kNN has already dropped 4.5 points from its peak.Source: §4.4 · p. 8
  • At fixed scale, classification peaks at s = 0.40, while correspondence and dense downstream tasks peak at s ≥0.60.The fixed-scale experiment provides the cleaner signal; correspondence metrics also better predict dense-task performance than classification accuracy.Source: §4.4 · p. 8, Figure 3 · p. 8, Figure 4 · p. 9

5 Discussion and Limitations

Same-instance geometric alignment is the semantic anchor of DINO-style learning, and the DINO loss alone recovers much of the full recipe’s correspondence quality. The conclusions are bounded by scale, model-family, and training-dynamics questions that remain open.

5 Discussion and Limitations

  • The DINO loss alone recovers 75% of semantic PCK and 93% of geometric R@.05 relative to the full DINOv2 recipe.Same-instance geometric view alignment is identified as the operative inductive bias for semantic emergence.Source: §5 · p. 9
  • Whether component contributions shift with larger data and longer training, and whether the same inductive bias holds beyond DINO-style teacher–student SSL, remains open.The analysis also excludes DINOv3, which exceeds the fixed compute budget, and does not ablate batch size or self-distillation dynamics.Source: §5 · p. 9

6 Conclusion

The conclusion identifies same-instance global-view alignment as necessary and sufficient for semantic structure to emerge in DINO-style SSL. It also distinguishes refinement from creation and highlights correspondence metrics as a more faithful indicator of downstream performance than classification accuracy.

6 Conclusion

  • Same-instance global-view alignment is the semantic anchor, sufficient to produce strong semantic patch features and necessary for their emergence.Necessity tests find that both geometric displacement and same-instance identity are required.Source: §6 · p. 9
  • iBOT refines existing semantic structure by 10.5 PCK points, while local crops benefit classification but not correspondence.The conclusion reports that local crops add view diversity without improving correspondence.Source: §6 · p. 9
  • Semantic and geometric correspondence metrics on SPair-71k and NAVI track downstream performance more faithfully than classification accuracy.The paper advocates routinely including these metrics in SSL evaluation.Source: §6 · p. 9

Supplementary Material

The supplementary material provides implementation and evaluation details, qualitative visualizations, and bootstrap confidence intervals. Its model-size comparison shows that larger ViT-g/14 training does not outperform ViT-L/14 on semantic correspondence, while ViT-L/16 offers better throughput with a modest correspondence cost.

Supplementary Material

  • The supplement contains implementation and training details, downstream-task definitions, qualitative feature visualizations, and bootstrap confidence intervals for correlation analysis.These materials are organized in Sections A–D, respectively.Source: Supplementary Material · p. 13
  • After 3× more iterations, ViT-g/14 does not surpass ViT-L/14 on semantic correspondence and gains no geometric correspondence beyond 124k iterations.The figure tracks kNN classification, semantic correspondence, and geometric correspondence across checkpoints.Source: Figure A1 · p. 14
  • ViT-L/16 improves throughput with a modest ~5% semantic-correspondence cost compared with ViT-L/14.The figure identifies ViT-L/16 as the best compute-to-performance choice.Source: Figure A1 · p. 14

A DINOv2 training

The experiments use controlled ImageNet-1k training and a ViT-L/16 primary ablation backbone. The model-size study finds no correspondence advantage from longer ViT-g/14 training and selects ViT-L/16 for subsequent experiments because of its throughput and modest semantic-correspondence trade-off.

A.1 Implementation details

  • The primary ablation backbone is ViT-L/16 with patch size 16 and global crop size 224; the model-size study also trains ViT-S/16, ViT-B/16, ViT-L/14, and ViT-g/14.Models use a Vision Transformer trained with the DINOv2 codebase in the low-compute regime.Source: §A.1 · p. 14
  • All models are trained exclusively on ImageNet-1k to isolate architecture and augmentation effects from data curation and scale.This differs from the full DINOv2 recipe, which uses a curated 142M-image dataset.Source: §A.1 · p. 14
  • Experiments use batch size 1024 and cosine learning-rate schedules with warmup; component ablations and model-size studies train for 100 epochs, except ViT-g/14 at 300.The two-global-crop ablations use a 4×10^-3 base learning rate, while four-global-crop and crop-scale sweeps use 2×10^-3.Source: §A.1 · p. 14

A.2 Model size

  • After 374k versus 124k iterations, ViT-g/14 reaches ~47.2% versus ViT-L/14’s 47.3% on SPair and gains no further NAVI improvement beyond 124k.The figure follows checkpoints across ImageNet-1k kNN classification, SPair-71k semantic correspondence, and NAVI geometric correspondence.Source: §A.2 · pp. 14–15
  • ViT-L/16 yields better throughput than ViT-L/14 with a ~5% reduction in SPair performance (42% versus 47%), so it is used for later experiments.Its larger patch size reduces tokens per image.Source: §A.2 · p. 15

B Downstream task details

The downstream evaluation uses frozen-backbone features across classification, correspondence, geometry, temporal tracking, and semantic segmentation, with task-specific probes or zero-shot matching.

B.1 Classification

  • Classification uses frozen [CLS] features, evaluated by kNN over ImageNet-1k training images and by a linear probe trained for 100 epochs.Both report Top-1 validation accuracy; the kNN protocol uses k=10 and cosine similarity without fine-tuning.Source: §B.1 · p. 15

B.2 Correspondence

  • Correspondence is evaluated by zero-shot cosine-similarity matching of frozen patch features, using semantic keypoints in SPair-71k and 3D reprojection correspondences in NAVI.The reported metrics are PCK@0.1 for semantic keypoints and R@0.05 m for geometric correspondence.Source: §B.2 · p. 15

B.3 Geometry

  • Geometry evaluation uses frozen patch features for NYU Depth v2 linear probing and frozen backbone features for ImageNet3D pose classification.Depth uses a DPT-style decoder on features from the last three transformer blocks and reports RMSE; pose heads predict azimuth, elevation, and in-plane rotation, reporting Acc@π/6.Source: §B.3 · p. 15
  • The intervention table covers ViT-L/16 models trained on ImageNet-1k for 125k iterations; crop-scale tables compare random minimum scales with fixed equal crop scales.The tables report all metrics, with RMSE lower-is-better; footnotes specify photometric variants of one crop and crops drawn from two same-class images.Source: Table A1 · p. 16, Table A2 · p. 16, §B.3 · p. 16

B.4 Temporal

  • Temporal tracking uses zero-shot cost-volume matching of frozen patch features across TAP-Vid DAVIS frames, with low-capacity convolutional branches containing 5.5K parameters.δvis_avg is the average fraction of visible points within five pixel-distance thresholds, from 1 to 16 pixels, evaluated only on visible points.Source: §B.4 · p. 16

B.5 Semantic Segmentation

  • Semantic segmentation is evaluated on ADE20K with a linear probe on frozen final-layer patch features, reporting validation mIoU.A single linear head is trained on spatial patch features, and predictions are bilinearly upsampled to input resolution.Source: §B.5 · p. 16

C Qualitative analysis

The qualitative analysis finds that geometrically distinct global crops of the same image produce part-consistent semantic structure, while local or cross-instance views do not establish it independently.

Qualitative visualization

  • The maps visualize patch-feature PC1–3 as RGB, with PCA fitted jointly over foreground tokens within each category so consistent colors indicate part correspondence.Backgrounds are masked to black.Source: C Qualitative analysis · p. 17, Figure A2 · p. 18

Views must come from the same image instance

  • Replacing same-image views with global crops from different instances produces fragmented maps without stable part groupings.The comparison indicates that global views must retain the same image instance for consistent part structure to emerge.Source: C Qualitative analysis · p. 17, Figure A2 · p. 18

Local crops do not drive semantic structure

  • Local crops add little correspondence structure to the two-global-crop baseline, while local crops alone and iBOT alone fail to sustain coherent spatial structure.Adding iBOT to the DINOv1 recipe yields richer part structure, indicating refinement of organization already present in DINO minimal.Source: C Qualitative analysis · p. 17, Figure A2 · p. 18

Same-instance global crops drive semantics

  • Two geometrically distinct global crops of the same image already yield spatially coherent, part-consistent features; additional global crops add little visible change.The PCA maps show corresponding parts sharing relative hues across instances, and DINO 4G is nearly indistinguishable from DINO minimal.Source: C Qualitative analysis · p. 17, Figure A2 · p. 18

D Bootstrap confidence intervals for correlation analysis

The analysis estimates bootstrap confidence intervals for proxy-to-downstream-task correlations and finds correspondence metrics more strongly correlated with dense-task performance than classification proxies.

Bootstrap confidence intervals

  • Semantic and geometric correspondence metrics have higher mean correlations with dense downstream tasks than kNN or linear probes across all three experimental settings, especially the fixed-scale sweep.Figure A3 reports 95% intervals from 10,000 non-parametric bootstrap resamples; depth RMSE is sign-flipped so higher values consistently indicate better performance.Source: D Bootstrap confidence intervals for… · p. 18, Figure A3 · p. 19

Related papers

Loading 2609.28187v1…