PaperlayerView exact revision on arXiv

Source-linked AI summary

CODI: Improving Self-Supervised Vision Transformers with Cross Distillation

Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, Xiaochun Cao, Qingming Huang

arXiv:2610.30639v1cs.CV

TL;DR

Dense prediction depends on patch tokens, but self-supervised ViTs can leave low-attention patches less reliable than the class-token representation. CODI adds teacher class-token information to masked patch targets, and it outperforms aligned DINOv2 counterparts across dense and image-level evaluation settings.

  • Problem

    Self-supervised ViTs can learn strong image-level representations while patch-level representations remain unreliable for dense prediction, which requires spatially meaningful features.

    Source: §1 · pp. 1–2

  • Method

    CODI mixes a small amount of the teacher class-token distribution into each teacher patch-token target for masked patch distillation.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Across seven evaluation settings spanning dense and image-level tasks, CODI outperforms its DINOv2 counterpart under aligned pretraining settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The target mixing gives masked patches global semantic guidance without explicitly gating that correction by their [CLS] attention mass.

    Source: §1 · p. 2

    The gradient analysis is local to the [CLS] readout and treats upstream representations and other network paths as fixed; masked-patch loss may add gradients.

    Source: §3.2 · pp. 3–4

Abstract

from arXiv · show

Self-supervised Vision Transformers learn strong image-level representations, but their patch-level representations can be unreliable for dense prediction. We study this mismatch and find that the class token concentrates global semantics on a small number of representative patches, while dense prediction errors often occur on low-attention regions. We provide a local gradient analysis showing that, along the [CLS] attention readout, image-level self-distillation gradients are weighted by [CLS]-to-patch attention mass. To provide masked patches with global semantics without changing the architecture, we propose Cross Distillation (CODI), which mixes a small amount of the teacher class-token distribution into each teacher patch-token target. CODI gives every masked patch an attention-independent semantic correction while preserving the standard self-distillation architecture. Experiments on dense and image-level benchmarks show that CODI improves both dense and image-level performance under aligned pretraining settings. Code is available at https://github.com/EldercatSAM/CODI.

1 Introduction

Self-supervised ViTs need both global class-token semantics and spatially reliable patch tokens, but the [CLS] attention path delivers semantic updates unevenly. CODI adds weak class-token guidance to masked patch targets and outperforms aligned DINOv2 counterparts across seven evaluation settings.

1 Introduction

  • Across seven evaluation settings spanning semantic segmentation, depth estimation, video object segmentation, and image classification, CODI outperforms aligned DINOv2 counterparts.Figure 1 compares CODI with baselines on Pascal VOC segmentation.Source: Figure 1 · p. 1, §1 · p. 2
  • During DINOv2-style pretraining, [CLS]-to-patch attention mass falls alongside dense transfer performance, while attention focuses on a few discriminative patches.The paper calls high-attention regions that serve as evidence for the image-level readout representative patches.Source: §1 · p. 2, Figure 2 · p. 2
  • Downstream errors concentrate on low-attention, non-representative patches, although dense prediction evaluates patch tokens at every spatial location.A patch not selected by the global readout can still matter for segmentation or depth estimation.Source: §1 · p. 2
  • Along the [CLS] readout, image-level semantic gradients reach patch tokens in proportion to attention mass, weakening updates to low-attention patches.The paper identifies this attention-gated path as one explanation for the uneven allocation of semantic updates.Source: §1 · p. 2
  • CODI mixes a small amount of the teacher class-token distribution into each teacher patch target, adding global guidance without removing spatial specificity.The correction is not explicitly gated by each patch's [CLS] attention mass.Source: §1 · p. 2

2 Related Work

Prior self-supervised ViTs use contrastive or masked prediction objectives, while dense representation learning explores local alignment, object-part grouping, and masked prediction. CODI modifies masked patch targets to address representative-patch issues during pretraining.

2.1 Self-supervised Vision Transformers

  • Self-supervised ViTs mainly use contrastive methods based on view discrimination or cross-view consistency, or prediction-based methods that predict masked pixel or latent content.iBOT provides self-distillation targets for class and masked patch tokens; DINOv2 scales this framework with stronger training strategies and larger-scale data.Source: §2.1 · p. 2

2.2 Self-supervised Dense Representation Learning

  • Dense representation learning approaches include local feature alignment, object-part grouping, and masked prediction.These approaches address the need for spatially meaningful local semantics in dense prediction.Source: §2.2 · p. 3
  • Longer self-supervised training can degrade dense representations; CODI instead modifies patch targets during pretraining, without an additional fine-tuning stage.The paper compares CODI with Gram anchoring under aligned settings in Section 5.3.Source: §2.2 · p. 3

3 The Representative Patch Phenomenon

The empirical analysis finds that segmentation errors are more common on low-attention patches, while the theoretical analysis links attention mass to semantic gradient allocation along the [CLS] readout. The analysis is local to that readout path, and experiments report a strong attention-gradient correlation.

3.1 Empirical Observation

  • After 800-epoch ImageNet-1K DINOv2 pretraining, Pascal VOC segmentation errors rise monotonically as normalized [CLS]-to-patch attention falls.The study bins patches by normalized attention mass and measures segmentation error rate within each bin.Source: §3.1 · p. 3

3.2 Theoretical Justification

  • The local analysis isolates the one-step [CLS] readout under class-token loss, treating upstream representations and other network paths as fixed; masked-patch loss may add gradients.The patch-loss gradients are outside the local formulation, so the claim concerns semantic gradients transmitted through the [CLS] attention path.Source: §3.2 · pp. 3–4
  • Along the [CLS] readout, a patch's value-path gradient norm equals its attention mass times the global semantic gradient norm, and its score update also contains that attention factor.Thus low-attention patches are weakly coupled to the image-level semantic target through this path.Source: §3.2 · p. 4
  • The measured correlation between [CLS]-to-patch attention mass and DINO-loss gradient norm is Spearman 0.925.The experiment checks whether the local theory aligns with practical observations involving the projection head, multi-head attention, and registers.Source: §3.2 · p. 4

4 Methodology

CODI modifies masked-patch targets in a self-distillation framework by mixing teacher class-token semantics into patch targets. The analysis describes this as a target-level semantic correction not explicitly gated by [CLS] attention mass.

4 Methodology

  • CODI mixes a small proportion of teacher class-token semantics into patch targets, adding a correction not explicitly weighted by [CLS] attention mass.This is a target-side modification within a typical self-distillation framework.Source: §4 · p. 4

4.1 Self-Distillation Framework

  • A momentum teacher supplies class- and patch-token target distributions, while the student learns them in a shared projection-head prototype space.The teacher is updated as an exponential moving average of the student.Source: §4.1 · pp. 4–5

4.2 Cross Distillation via Target Mixing

  • For each masked patch, CODI mixes its teacher patch distribution with the teacher class-token distribution from the same view; α controls guidance, and α=0 recovers the standard iBOT loss.The two targets share prototype coordinates, so their convex combination remains a valid teacher target; image-level supervision retains the standard DINO objective.Source: §4.2 · p. 5

4.3 Theoretical Analysis

  • The analysis describes CODI’s target-level semantic correction as controlled by α and independent of the patch’s [CLS] attention mass πi.This contrasts with the standard [CLS] path, which gates the global semantic signal by attention mass.Source: §4.3 · p. 6

5 Experiments

CODI is evaluated across dense and image-level tasks, with comparisons against aligned DINOv2 and Gram anchoring. Results cover segmentation, depth, video object segmentation, classification, and ablations of the target distribution and mixing strength.

5.2 Overall Performance

  • The experiments compare CODI across seven protocols with representative methods from contrastive, masked-modeling, clustering, and dense self-supervised learning.Additional results cover unsupervised segmentation, object detection, and smaller models.Source: §5 · p. 6, §5.2 · p. 6

5.1 Pretraining Setting

  • Pretraining follows the ImageNet-1K AdamW setup with batch size 1024, patch size 14, register tokens, and α = 0.1.Training lasts 800, 400, and 200 epochs for ViT-S/14, ViT-B/14, and ViT-L/14, respectively.Source: §5.1 · p. 6

5.2.1 Evaluation on Dense Prediction Tasks

  • CODI improves average mIoU over reproduced DINOv2 by +2.7, +1.7, and +1.9 for ViT-S/14, ViT-B/14, and ViT-L/14, respectively.It also improves NYU δ1 by +1.4 for ViT-S/14 and matches or improves DINOv2 in 24 of 27 depth metric comparisons; ViT-L/14 reaches 66.0 J &F, +1.8 over original DINOv2.Source: §5.2.1 · pp. 7–8

5.2.2 Evaluation on Image-level Tasks

  • ViT-Large CODI improves Top-1 accuracy over DINOv2 by +0.8 for k-NN and +1.0 for linear classification.After full fine-tuning, CODI still improves Top-1 accuracy by +0.4; under patch occlusion it outperforms DINOv2 at every masking ratio from 0% to 90%.Source: §5.2.2 · pp. 8–9, Figure 6 · p. 8

5.3 Comparison with Gram Anchoring

  • CODI fine-tuning outperforms Gram anchoring by +1.2 mIoU, while training CODI from scratch surpasses it by +2.9 mIoU with fewer iterations.Fine-tuning does not improve classification performance; training CODI from scratch yields the best classification results.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 The Effect of Cross Distillation

  • The α = 0.1 target-mixing choice improves dense performance with [CLS] targets, whereas uniform targets reduce classification and segmentation performance.At α = 0, CODI reduces to DINOv2, and performance degrades when α is large; α = 0.1 is a strong starting point but not a universal optimum.Source: §5.4 · pp. 9–10, Table 6 · p. 9
  • Entropy-matched local targets improve VOC mIoU by +1.5 over DINOv2 but reduce k-NN by 0.3; same-image [CLS] guidance improves both metrics and yields the largest mIoU gain.Random-image [CLS] yields a smaller dense-prediction improvement and also reduces k-NN, indicating that regularization contributes while semantically aligned guidance is the dominant source of improvement.Source: Table 7 · p. 9, §5.4 · p. 10
  • CODI’s performance gain mainly comes from patches with low [CLS]-to-patch attention rank.Patches are ranked by final-layer attention mass averaged over heads, excluding register tokens.Source: §5.4 · p. 10, Figure 7 · p. 10

6 Conclusion

The conclusion attributes unreliable patch features to attention-weighted semantic gradients along the [CLS] readout. CODI mixes teacher class-token information into masked patch targets, and the reported benchmark results support its effectiveness and scalability.

6 Conclusion

  • CODI adds a small amount of teacher class-token information to masked patch targets, giving each patch supervised by the patch loss a target-level global semantic correction.The paper reports empirical evaluation across multiple benchmarks and settings, with effectiveness and scalability as its conclusion.Source: §6 · p. 10

A Detailed Theory and Proofs

The appendix isolates how the final [CLS] attention readout allocates image-level gradients across patches, then formalizes sparse attention and CODI’s attention-independent patch-target correction.

A.1 Why a Local [CLS] Readout Is Sufficient for the Argument

  • The local analysis isolates the final [CLS] attention row; gradients through its softmax-weighted readout are multiplied by attention mass, without excluding gradients from other network components.The calculation holds upstream representations, teacher targets, and other paths fixed.Source: §A.1 · p. 16, A Detailed Theory and Proofs · p. 16

A.2 Exact Gradients of the [CLS] Attention Readout

  • The proof separately differentiates the readout with respect to patch values and attention scores, and derives ratio identities for patch gradients.The exact form of the prediction head is not needed for the local calculation.Source: §A.2 · pp. 16–17

A.3 Attention Amplification of Representative Patches

  • Under the stated local sign condition, total attention mass on representative patches increases.The condition assigns positive score velocities to representative patches and nonpositive velocities to others.Source: §A.3 · pp. 17–18

A.4 Sparse Sufficiency of an Image-Level Readout

  • With a Lipschitz loss, bounded patch vectors, and a representative subset already near the best readout, attention can approximate that subset while assigning arbitrarily little mass elsewhere.This formalizes how an image-level readout can be sufficient with sparse patch evidence.Source: §A.4 · pp. 18–19

A.5 CODI Target Mixing

  • CODI mixes the teacher class-token distribution into each masked patch target, adding a patch-logit correction that does not depend on [CLS] attention mass.The analysis applies directly to the masked patch prediction head.Source: §A.5 · p. 19

A.6 Register Tokens as Softmax Competitors

  • Register tokens compete with patches in the [CLS] softmax, whereas CODI’s masked-patch correction is not reduced by attention assigned to registers.Registers can still store global or internal computation.Source: §A.6 · p. 20

B Pseudo Code

  • The pseudocode combines the usual DINO loss with a weighted CODI loss and updates the teacher by exponential moving average.It also forms mixed targets for masked patches using the mixing ratio α.Source: B Pseudo Code · p. 21

C.1 Pretraining Settings

  • Unless otherwise specified, pretraining uses ImageNet-1K; CODI otherwise keeps the protocol unchanged, mixing the teacher class-token distribution into each masked teacher patch target at a class-to-patch ratio of 0.1.The default ViT-S/14 run trains for 800 epochs.Source: §C.1 · pp. 21–22

C.2 Evaluation Settings

The evaluation settings cover image classification, dense prediction, and video object segmentation; additional COCO experiments report gains in unsupervised segmentation, detection, and instance segmentation.

C.2 Evaluation Settings

  • The evaluation suite uses frozen CLS features for classification, patch-token probes for segmentation and depth, and first-frame mask propagation for video object segmentation.Classification includes k-NN, linear probing, and full fine-tuning; dense segmentation and depth use frozen backbones.Source: §C.2 · pp. 22–23

D.1 Unsupervised Semantic Segmentation

  • +1.6 mIoU in unsupervised segmentation and +1.2 mIoU in linear probing on COCO-Stuff-27 versus DINOv2.CODI also exceeds STEGO with DINO ViT-S/8 by +1.1 mIoU in unsupervised segmentation despite using a larger patch size.Source: §D.1 · p. 23, Table 10 · p. 23

D.2 Object Detection and Instance Segmentation

  • CODI achieves the best performance across all reported COCO object-detection and instance-segmentation metrics.The passage characterizes this as a stronger pretrained initialization for these tasks.Source: §D.2 · p. 24, Table 11 · p. 24

D.3 Results on Smaller Models

CODI’s reported gains extend to a tiny model and a reference comparison with SigLIP2, while robustness tests show stronger performance under patch perturbations.

D.3 Results on Smaller Models

  • +0.3 in ImageNet classification and +3.2 mIoU in VOC segmentation for CODI versus DINOv2 with ViT-T/14.The model was pretrained on ImageNet-1K for 200 epochs.Source: §D.3 · p. 24, Table 12 · p. 24

D.4 Comparison with DINOv3 and SigLIP2

  • CODI ViT-Large outperforms SigLIP2 by +6.1 mIoU in the reported reference comparison.The models differ in architecture and pretraining data. A scratch DINOv3 ViT-S/16 scored 70.9 ImageNet k-NN accuracy and 61.3 VOC mIoU, compared with 71.1 and 61.4 for DINOv2; the authors hypothesize DINOv3 design choices need further tuning at ImageNet-1K scale.Source: §D.4 · pp. 24–25, Table 13 · p. 24

D.5 Additional Experiments on Model Robustness

  • CODI consistently improves ImageNet k-NN robustness under attention-based occlusion, random occlusion, and patch shuffling.The reported advantage grows at high random-masking ratios and remains slight but consistent as spatial order is disrupted.Source: §D.5 · p. 25, Figure 8 · p. 25

D.6 Sensitivity to the Mixing Ratio Across Pretraining Datasets

The preferred CODI mixing ratio depends on the pretraining dataset and downstream metric; α = 0.1 is supported as a strong starting point, not a universal optimum.

D.6 Sensitivity to the Mixing Ratio Across Pretraining Datasets

  • The COCO ablation uses 800 epochs; the ImageNet-22K run uses 100K iterations and excludes the ImageNet-1K subset to avoid downstream-benchmark overlap.Both evaluate ImageNet-1K k-NN classification and VOC semantic segmentation.Source: §D.6 · p. 25, Table 14 · p. 26
  • On COCO, α = 0.075 matches α = 0.05 on k-NN, but its segmentation result falls between α = 0.05 and α = 0.1; α = 0.2 lowers both metrics below DINOv2.On ImageNet-22K, α = 0.1 gives the best results on both metrics.Source: §D.6 · p. 25, Table 14 · p. 26

E Qualitative Results

The qualitative results show more coherent semantic masks, preserved spatial structure in depth predictions, and semantically aligned local matches across image changes.

Semantic Segmentation Visualizations

  • CODI produces more coherent object masks than the fragmented regions or background activations in baseline predictions.The examples use frozen-backbone linear segmentation probes.Source: E Qualitative Results · p. 26

Depth Estimation Visualizations

  • CODI preserves coherent indoor layouts and object boundaries in depth predictions.These visualizations use frozen dense representations.Source: E Qualitative Results · p. 26

Sparse Correspondence Visualizations

  • Matched points connect semantically consistent object parts across changes in instance, viewpoint, scale, and background.The examples examine whether local features remain aligned across related images.Source: E Qualitative Results · p. 26

F Limitations

CODI’s mixing ratio may be problematic in multi-object scenes, and larger ratios can reduce patch discriminability. The appendix also includes qualitative visualizations and states that the paper’s claims match its contributions and experimental scope.

F Limitations

  • In multi-object scenes, the class-token target can mix object semantics, and overly strong mixing may confuse patch targets.The authors identify α = 0.1 as a strong starting point, not a universal optimum, and leave adaptive or multi-object-aware mixing for future work.Source: F Limitations · p. 26
  • CODI’s additional examples show more complete, less fragmented semantic regions than baseline predictions.The figure compares input images and predictions from iBOT, DINOv2, and CODI.Source: Figure 9 · p. 27
  • Additional depth examples show coherent scene layouts and depth discontinuities, while correspondence examples show semantically aligned matches across image changes.The figures compare CODI with iBOT and DINOv2 for depth, and visualize matched local features for correspondence.Source: Figure 10 · p. 28, Figure 11 · p. 28

NeurIPS Paper Checklist

  • The checklist states that the abstract and introduction accurately reflect the paper’s contributions and scope, supported by theory and experiments.It identifies the representative-patch failure mode, CODI target mixing, and evaluations on dense and image-level tasks.Source: NeurIPS Paper Checklist · p. 29

2. Limitations

The checklist records limitations involving mixing sensitivity and pretraining scope, missing statistical uncertainty and compute details, no separate broader-impact discussion, and incomplete asset-license reporting.

2. Limitations

  • The authors’ limitations discussion flags possible confusion from strong class-token mixing on multi-object images, sensitivity to the mixing ratio, and limited scope of large-scale pretraining experiments.Source: §2 · p. 29
  • Results are single-run benchmarks without error bars, confidence intervals, or statistical significance tests, mainly because repeated large-scale runs are costly.The checklist says GPU types, counts, and approximate wall-clock time are reported for one ViT-S/14 800-epoch run, but memory and execution times for all runs are not fully reported.Source: §2 · p. 32
  • The paper is foundational representation-learning work and has no separate broader-impact discussion.Source: §2 · p. 33
  • The paper credits datasets, baselines, and model components but does not enumerate licenses and terms of use for all existing assets.Source: §2 · p. 34

Related papers

Loading 2610.30639v1…