The Locality Trade-off in Masked Patch Self-Distillation

1 Introduction

• Masked patch self-distillation predicts teacher-generated patch targets, raising the question of whether it preserves local evidence beyond semantic consistency.The targets are learned representations rather than raw pixels.Source: §1 · p. 2
• In ViT-B, iBOT broadens similarity responses over buildings and mountains, while ViT-L shows more spatially selective responses to windows and rugged cliffs.Semantic masks can leave these within-region structural differences unresolved.Source: Figure 1 · p. 1, §1 · p. 2
• For ViT-S/B, iBOT substantially weakens patch correspondence while leaving appearance decodability largely unchanged or slightly improved; for ViT-L/H, correspondence is preserved or enhanced as deep-layer appearance decodability declines.The results indicate that scale changes which local properties are affected, rather than uniformly removing the locality cost.Source: §1 · p. 2
• Gram anchoring improves correspondence with a DINO-based anchor in ViT-B, but improves deep appearance decodability without consistently improving correspondence in ViT-L.The effects depend on both the anchor representations and model scale.Source: §1 · pp. 2–3
• Two annotation-free diagnostics separate linearly recoverable pixel and texture evidence from spatial distinguishability and matching across views.They address properties that segmentation evaluation and similarity visualizations alone cannot resolve.Source: §1 · p. 2

2 Background, Related Work, and Notation

• The section introduces the class-token joint-embedding architecture and masked patch self-distillation as background, with further related work deferred to Appendix H.The supplied passage does not specify additional background details.Source: §2 · p. 3

2.1 Class-Token Joint-Embedding Self-Distillation

• DINO matches student predictions across image views to stop-gradient teacher targets, with teacher parameters updated by exponential moving average.For controlled DINO versus DINO+iBOT comparisons, teacher targets use Sinkhorn–Knopp normalization and temperature-controlled sharpening.Source: §2.1 · p. 3

2.2 Masked Patch Self-Distillation: iBOT

• iBOT trains the student to predict teacher-generated patch distributions at masked positions, using the unmasked crop as teacher input and matching spatial positions.The loss is evaluated only at masked positions; Figure 2 compares downstream effects across four backbone scales, with DAVIS tracking effects reversing from degradation in smaller models to modest gains in larger ones.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 The Downstream Signature of iBOT’s Locality Cost

• The comparison uses DINO and DINO+iBOT across ViT-S/16, ViT-B/16, ViT-L/16, and ViT-H/16, pretrained on ImageNet-1K for 300 epochs with matched settings within each scale.DAVIS label propagation tests cross-frame patch matching and instance consistency; Sintel scores measure training-set fitting.Source: §3 · p. 4
• Across all scales, DINO+iBOT improves classification, segmentation, and detection, while DINO more often leads on Copydays and ROxford retrieval.RParis generally benefits from iBOT, so the two retrieval benchmarks are reported separately.Source: §3 · pp. 4–5
• iBOT reduces DAVIS performance on ViT-S/B but modestly improves it on ViT-L/H, while patch effective rank decreases at every scale.Reduced spectral diversity therefore does not necessarily mean weaker matching.Source: §3 · p. 5
• With ViT-B/16, iBOT is associated with identity confusion and weaker track recovery after occlusion; these failures are alleviated with ViT-L/16.The examples involve visually similar instances of the same category, such as overlapping dogs.Source: §3 · p. 5, Figure 3 · p. 5

4 Locating Lost Evidence via Annotation-Free Diagnostics

• Two annotation-free diagnostics separate changes in recoverable pixel and texture evidence from changes in cross-view spatial matching under iBOT.Local Appearance Decodability measures the former; Local-to-Global Correspondence measures whether patches remain distinguishable and matchable across views.Source: §4 · p. 5

4.1 Local Appearance Decodability

• For ViT-S/B, iBOT largely preserves appearance decodability, with final-layer RGB-mean and HOG scores improving for both backbones.Modest gains or losses vary by descriptor and layer, so these models show no systematic loss of measured linearly recoverable appearance.Source: §4.1 · p. 6
• For ViT-L/H, all six descriptor scores decrease at the final block, with particularly pronounced HOG and LBP losses for ViT-H.Selected shallow or intermediate descriptors, including HOG and RGB mean, improve slightly; the appearance cost concentrates in deeper representations.Source: §4.1 · p. 6

4.2 Local-to-Global Correspondence

• At the final block and crop area ratio 0.1, iBOT lowers patch matching accuracy by approximately 11.8 points for ViT-S and 16.9 for ViT-B, but raises it by approximately 3.3 for ViT-L and 4.0 for ViT-H.For larger models, patch matching accuracy and region IoU improve across sampled layers; geometric metrics generally improve or remain comparable, with small exceptions.Source: §4.2 · p. 7
• In smaller models, correspondence often declines late in training, while larger models generally improve or stabilize; crop size and matching criterion also affect the comparison.For ViT-S/B, matching and localization deficits persist across evaluated crop ratios, whereas the ViT-L patch-matching advantage narrows and slightly reverses at ratio 0.5.Source: §4.2 · pp. 7–8
• Smaller models retain broadly comparable appearance decodability while losing spatial matching, whereas larger models improve matching despite reduced deep-layer appearance decodability.Thus, preserving measured appearance evidence does not guarantee effective correspondence, and better correspondence does not require preserving all measured appearance attributes.Source: §4.2 · p. 8

5 How Gram Anchoring Changes Local Representations

• Gram anchoring constrains pairwise patch similarities, and the comparison tests standard anchoring against DINO-Gram, which differs in its initial teacher anchor.Standard anchoring starts from an epoch-250 DINO+iBOT checkpoint; DINO-Gram starts from DINO, with regularization beginning at epoch 260.Source: §5 · p. 8

5.1 Downstream Gains Do Not Imply Uniform Locality Recovery

• Downstream effects depend on scale and teacher: for ViT-B/16, DINO-Gram raises DAVIS J&F from 55.5 to 61.4, while standard Gram anchoring raises it to 57.6.For ViT-L/16, standard Gram anchoring benefits segmentation and detection but reduces DAVIS; DINO-Gram improves DAVIS and copy detection while sacrificing some segmentation and detection performance.Source: §5.1 · p. 9

5.2 Teacher Choice Affects Appearance and Correspondence Differently

• For ViT-B/16, final-layer patch matching accuracy rises from 24.8% to 30.1% with standard Gram anchoring and to 40.8% with DINO-Gram, while appearance changes are small or mixed.The corresponding DAVIS gains indicate the correspondence improvement extends beyond the diagnostic.Source: §5.2 · p. 9
• For ViT-L/16, DINO-Gram improves all six final-layer appearance descriptors, but neither Gram variant consistently improves correspondence.DINO-Gram reduces patch matching accuracy and region IoU despite a small increase in RANSAC box IoU; all six descriptor scores remain below DINO without iBOT.Source: §5.2 · p. 9

5.3 What Does Gram Anchoring Preserve?

• The authors hypothesize that an anchor’s usefulness depends on which property its teacher preserves; DINO-Gram increases patch effective rank at both scales but improves patch matching only for ViT-B/16.Within-view similarity preservation does not directly enforce cross-view feature alignment, and the proposed mechanisms remain hypotheses.Source: §5.3 · p. 10

6 Conclusion

• Locality has no single measure: correspondence and recoverable appearance can respond differently, and scaling can improve one while weakening the other.Semantic transfer and visually coherent feature maps alone do not establish that representations retain all local cues needed downstream.Source: §6 · p. 10

A Class-Token Joint-Embedding Objectives

• DINO matches student predictions to stop-gradient teacher targets across views, with teacher parameters updated by exponential moving average.Teacher targets use Sinkhorn–Knopp normalization; the student objective averages cross-entropy over images and valid cross-view pairs.Source: A Class-Token Joint-Embedding Objectives · p. 15

B.1 Capabilities Probed by the Downstream Benchmarks

• The benchmarks probe overlapping capabilities: semantic recognition and localization, instance retrieval, transformed-image identity, tracking correspondence, optical flow, and patch-spectrum diversity.Segmentation and detection do not establish pixel or texture preservation; the Sintel flow head is fitted and evaluated on the same videos, making it a training-set readout diagnostic.Source: §B.1 · pp. 15–16

B.2 Experimental Settings; B.2.1 Pretraining

• The four backbones share patch size, registers, LayerScale, and an EMA teacher; paired DINO and DINO+iBOT runs differ by adding the masked-patch objective.Both use the DINO class-token objective, while the iBOT loss weight is fixed at 1.0.Source: §B.2.1 · pp. 16–17

B.2.1 Pretraining

• Adding iBOT increases peak memory and training time at every backbone scale; the eight pretraining runs total approximately 3,964 H200 GPU-hours.Patch-level prediction over 131,072 prototypes contributes substantially to overhead; ViT-H uses half the batch size of the other backbones.Source: §B.2.1 · p. 17, Table 1 · p. 17

• Frozen teacher checkpoints are evaluated on eight downstream benchmarks, with RParis and ROxford counted separately.ImageNet k-NN uses class-token features, while retrieval uses concatenated class-token and generalized-mean-pooled patch-token descriptors.Source: §B.2.2 · p. 18
• Frozen-backbone protocols assess segmentation, detection, copy detection, tracking, and optical flow through task-specific heads or descriptors.DAVIS propagates masks by patch similarity; the Sintel flow head is evaluated on the same training videos used for fitting, not a held-out split.Source: §B.2.2 · p. 18
• Patch effective rank summarizes the singular-value spectrum of each image’s patch-token matrix, with larger values indicating a more evenly spread spectrum.It is a complementary spectral statistic, not a direct measure of information loss or correspondence; the reported mean uses 10,000 randomly sampled ImageNet training images.Source: §B.2.2 · pp. 18–19

B.2.3 Experimental Settings for Gram Anchoring

• Gram anchoring is tested with ViT-B/16 and ViT-L/16 against DINO+iBOT, using either a DINO+iBOT or DINO checkpoint to initialize the Gram teacher.The Gram loss activates after epoch 260, its coefficient rises to 2, and the teacher is updated at epochs 270, 280, and 290.Source: §B.2.3 · p. 19
• The Gram loss matches within-crop pairwise similarities between masked student patch features and unmasked Gram-teacher features, including masked student positions.Gradients pass through student features only; the Gram teacher is fixed between scheduled updates.Source: §B.2.3 · p. 19

C Additional Similarity Heatmap Visualization

• Figure 10 provides additional anchor-patch similarity heatmaps for DINO and DINO+iBOT with ViT-B/16 and ViT-L/16.Source: Figure 10 · p. 20

D Numerical Results and Trajectories of Downstream Evaluations

• At epoch 300, iBOT improves ImageNet kNN, VOC2012 segmentation, and VOC2007 detection at all scales, while retrieval, copy detection, DAVIS, and optical-flow outcomes are mixed.DAVIS J&F decreases for ViT-S/16 and ViT-B/16 but improves modestly for ViT-L/16 and ViT-H/16; at selected checkpoints, EPE increases for S/L/H and slightly decreases for B.Source: Table 2 · p. 21, D Numerical Results and Trajectories… · p. 22
• Across training, task differences change over time, and lower patch rank with iBOT does not by itself imply weaker correspondence.Figure 11 shows that the DAVIS advantage can reverse by epoch 300 and retrieval or copy-detection gaps can change sign.Source: Figure 11 · p. 20, D Numerical Results and Trajectories… · p. 22

E Additional DAVIS Video Object Tracking Visualization

• Figure 12 presents additional DAVIS video object tracking visualizations for DINO and DINO+iBOT with ViT-B/16 and ViT-L/16.Source: Figure 12 · p. 23

F Implementation and Visualization of Local Appearance Decodability

• Local Appearance Decodability trains a linear decoder on six groups of pixel-derived patch attributes and evaluates it on held-out images while keeping the backbone frozen.The attributes cover color, edge orientation, texture, spatial frequency, and raw-pixel variation; no semantic annotations or model-generated targets are used.Source: F Implementation and Visualization of… · p. 23

F.1 Image Preparation and Probe Data

• The probe uses aligned raw-image patches and frozen patch tokens, with image-level train-validation splitting to prevent patches from one image crossing splits.It samples 20,000 ImageNet-1K images and uses 196 patches per image in each split.Source: §F.1 · p. 24

F.2 Local Appearance Targets

• Six descriptor groups form a 122-dimensional appearance target, spanning color, texture, frequency energy, and raw-pixel variation.Groups are evaluated separately; HOG and LBP are computed on the full image before pooling within patches.Source: §F.2 · pp. 24–25

F.3 Fitting the Linear Decoder

• Separate ridge-regression decoders map frozen patch tokens to appearance targets, with preprocessing statistics, PCA basis, and decoder fixed during validation.The reported probes use ridge_alpha 10^-3.Source: §F.3 · pp. 25–26

F.4 Scoring and Interpretation

• Group scores average dimension-wise validation R2, giving each descriptor dimension equal weight; higher scores indicate greater linear accessibility under this probe.A lower score does not establish that information is absent, and a higher score does not imply better spatial matching.Source: §F.4 · p. 26

F.5 The Trajectories of Local Appearance Decodability

• In ViT-L/16 and ViT-H/16, reduced final-layer appearance decodability persists across descriptor groups, while ViT-S/16 and ViT-B/16 generally preserve or slightly improve it.The larger-model cost emerges during training and is concentrated in deeper layers.Source: §F.5 · p. 27

F.6 Visualization of Local Appearance Decodability

• The visualization shows RGB-mean predictions as patch colors and other descriptor-vector norms as heatmaps, but independently normalized heatmaps do not compare calibrated magnitudes across models.Quantitative comparisons use full descriptor vectors and R2.Source: §F.6 · pp. 27–28

G Implementation and Visualization of Local-to-Global Correspondence

• Local-to-global correspondence compares frozen crop and source-image patch tokens using cosine similarity, then evaluates whether matches identify the crop region and support geometric localization.The reference region comes from the crop operation, so this diagnostic requires no human annotations or learned matching head.Source: G Implementation and Visualization of… · p. 28, §G.2 · p. 29

G.1 Constructing Local Queries

• The crop protocol samples local regions at a default area ratio of 0.1 and resizes both crop and source image to 224 × 224 before feature extraction.Paired model comparisons reuse the same sampled crops.Source: §G.1 · p. 29

G.3 Matching and Localization Metrics

• Correspondence is evaluated through individual match region membership, response-map region overlap, and geometric agreement on crop position.The geometric estimate uses a RANSAC-style translation consensus with known crop dimensions rather than unconstrained affine or homography estimation.Source: §G.3 · pp. 29–30
• Patch matching accuracy counts top-1 matches anywhere inside the reference crop, not only matches at the exact corresponding position.It is interpreted alongside geometric localization metrics.Source: §G.3 · p. 30
• Region IoU selects as many high-response source patches as the reference region contains, then compares the predicted set with that region.The fixed region size avoids a similarity threshold, but crop ratio can still alter matching difficulty.Source: §G.3 · p. 30

G.4 Crop Ratios and Sampling

• Crop-ratio comparisons should be made between models at the same ratio because changing it changes available context and scale difference.Figure 15 reports paired differences across crop area ratios.Source: §G.4 · p. 31, Figure 15 · p. 31
• Correspondence results use one query-sampling run with seed 0 and no variability estimates across sampling seeds or independent pretraining runs.Small differences should not be interpreted as statistically established gains.Source: §G.4 · p. 31

G.6 Additional Visualization of Local-Global Correspondence

• Figure 17 presents complementary views of matching: source-image similarity heatmaps, individual patch matches, and estimated boxes summarizing geometric consensus.Together these views distinguish distributed responses, incorrect matches, and failures to localize a coherent region.Source: §G.5 · pp. 32–33, Figure 17 · p. 34

G.5 Visualization and Training Trajectories

• ViT-S/16 and ViT-B/16 show a sustained +iBOT decline in final-layer patch matching after an early peak, whereas ViT-L/16 and ViT-H/16 generally preserve or modestly improve correspondence.Related region-level and geometric localization metrics show deterioration in smaller models, with varying magnitude.Source: Figure 16 · p. 32, §G.5 · p. 33
• The correspondence cost seen in smaller models develops during optimization and can be alleviated at larger scales, alongside reduced deep-layer appearance decodability.Improved matching therefore does not imply greater accessibility of pixel and texture evidence.Source: §G.5 · p. 33

H. Extended Related Work

• The related-work discussion frames the study around patch supervision, prediction targets, and relational regularization, noting that DINO, iBOT, DINOv2, and DINOv3 combine global and patch-level supervision.The paper examines the patch objective through paired DINO and DINO+iBOT experiments.Source: H Extended Related Work · p. 35, §H.1 · p. 35

H.2 Prediction Targets in Masked Image Modeling

• Masked image modeling targets range from pixels and HOG features to learned tokens or teacher representations; the paper notes that target type alone does not establish which appearance evidence remains accessible.Its appearance descriptors evaluate frozen features and do not add a reconstruction objective during pretraining.Source: §H.2 · p. 35

H.3 Gram Anchoring and the Preservation of Patch Relations

• Gram anchoring matches the student’s patch Gram matrix to an earlier teacher checkpoint while allowing feature vectors themselves to change.DINOv3 introduced it to address degradation in dense feature quality during extended training.Source: §H.3 · p. 35
• In ViT-B, the DINO-based anchor substantially improves correspondence with limited and mixed appearance changes, while in ViT-L it improves deep appearance decodability without consistently improving correspondence.The observed anchoring effects depend on model scale and reference representations.Source: §H.3 · p. 35
• Within-view patch-relation matching does not explicitly constrain cross-view consistency or recoverability of pixel-derived attributes, so these properties require separate evaluation.The experiments extend relational-regularization analysis beyond aggregate dense-task performance.Source: §H.3 · p. 35

I Limitations

• Pretraining is restricted to ImageNet-1K for 300 epochs, so whether the trade-offs persist with larger datasets and longer training is unknown.The study is substantially smaller in data and compute than DINOv2 and DINOv3 regimes.Source: I Limitations · p. 36
• The analysis fixes the iBOT loss weight at 1 and does not isolate student masking from the iBOT loss, limiting conclusions to this commonly used setting.Other loss weights or schedules may change the balance among semantic learning, appearance decodability, and correspondence.Source: I Limitations · p. 36
• The diagnostics cover only linear appearance decodability and crop-based correspondence, not nonlinear information, broad viewpoint changes, deformations, occlusions, or exact pointwise matching.Geometric localization assumes known crop dimensions, and the optical-flow head is evaluated on the same Sintel videos used to fit it rather than held-out data.Source: I Limitations · p. 36
• Single pretraining and query-sampling runs leave small differences descriptive, while cross-scale comparisons are confounded and the proposed mechanisms remain hypotheses.Drop-path rates differ across ViT-S/B and ViT-L/H; ViT-H also uses a smaller batch and scaled learning rate, so cross-scale differences cannot be attributed to capacity alone.Source: I Limitations · p. 36, §2 · pp. 38, 41

12. Licenses for existing assets

• The paper reports no human-subject research, so IRB approval or equivalent review is not applicable.The N/A response also indicates that the paper does not involve crowdsourcing.Source: §12 · p. 44
• The paper states that its core methods and experiments do not use LLMs as an important, original, or non-standard component.Source: §12 · p. 44