PaperlayerView exact revision on arXiv

Source-linked AI summary

Emerging Properties in Self-Supervised Vision Transformers

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin

arXiv:2104.14294v2cs.CV

TL;DR

The paper asks whether self-supervised pretraining gives Vision Transformers distinctive properties beyond convnets, and develops DINO, a label-free self-distillation method. Self-supervised ViT features expose scene layout and support strong k-NN classification, while DINO reaches 80.1% top-1 on ImageNet linear evaluation with ViT-Base.

  • Problem

    The paper examines whether self-supervised pretraining can give Vision Transformers distinctive properties beyond convnets, whose benefits over convnets remain unclear.

  • Method

    The paper develops DINO, a label-free self-distillation framework that trains a student to predict a momentum-encoder teacher.

  • Results

    Self-supervised ViT features encode scene layout and achieve 78.3% top-1 ImageNet accuracy with k-NN, while DINO reaches 80.1% with ViT-Base linear evaluation.

  • Takeaways & Limitations

    Self-supervised learning can produce ViT features useful for k-NN classification and containing information relevant to scene layout.

  • Takeaways & Limitations

    The paper leaves large ViT pretraining on random uncurated images for future work.

Abstract

from arXiv · show

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

1 Facebook AI Research 2 Inria∗ 3 Sorbonne University

Without labels or supervision, a Vision Transformer trained with 8 × 8 patches learns class-specific features that produce unsupervised object segmentations through [CLS]-token self-attention.

  • Without supervision, last-layer [CLS]-token self-attention maps automatically yield class-specific features and unsupervised object segmentations.The model uses 8 × 8 patches, and the [CLS] token is not attached to any label or supervision.

1. Introduction

The paper studies whether self-supervised pretraining unlocks distinctive properties in Vision Transformer features beyond supervised ViTs and convnets. It identifies semantic structure and strong label-free classification, then develops DINO to achieve leading ImageNet performance.

  • 1. Introduction: Self-supervised ViT features explicitly encode scene layout and object boundaries, with this information directly accessible through last-block self-attention.Figure 1 demonstrates the emergence of semantic segmentation information, which is less evident in supervised ViTs and convnets according to the paper context.
  • 1. Introduction: 78.3% top-1 accuracy on ImageNet is achieved by self-supervised ViT features with a basic k-NN classifier, without finetuning, a linear classifier, or data augmentation.The paper identifies momentum encoders and multi-crop augmentation as important components for this k-NN performance.
  • 1. Introduction: Smaller ViT patches improve the quality of resulting self-supervised features, while momentum encoders and multi-crop augmentation help produce strong k-NN performance.The emergence of segmentation masks appears shared across self-supervised methods, whereas strong k-NN performance depends on combining particular components.
  • 1. Introduction: DINO is a simple label-free self-distillation framework that trains a student to predict a momentum-encoder teacher using standard cross-entropy, with centering and sharpening preventing collapse.The work is motivated by whether supervision in ViT pretraining limits their benefits, given the success of self-supervised pretraining in NLP and image methods with convnets [10] [12] [30] [33].
  • 1. Introduction: 80.1% top-1 accuracy on ImageNet linear classification with ViT-Base and small patches outperforms previous self-supervised features, while DINO matches state-of-the-art performance with ResNet-50.These results validate the synergy between DINO and ViT and confirm that DINO also works with convnets.

2. Related work

Prior self-supervised methods include instance-classification approaches that do not scale well with dataset size and feature-matching approaches such as BYOL. This work builds on BYOL [30] while framing its method as label-free Mean Teacher self-distillation [65] with identical student and teacher architectures.

  • Self-supervised learning: Instance-classification methods [12] [33] treat each image as a separate class, but explicitly discriminating among all images scales poorly with dataset size.Wu et al. use a noise contrastive estimator (NCE) to compare instances.
  • Self-supervised learning: BYOL [30] learns unsupervised features by matching representations from a network to those produced by a momentum encoder.Such methods can work without a momentum encoder, but performance drops [16] [30].
  • Self-supervised learning: The approach draws inspiration from BYOL [30] but uses a different similarity-matching loss and identical architectures for the student and teacher.It extends BYOL’s interpretation of self-supervised learning as Mean Teacher self-distillation [65] without labels.
  • Self-training and knowledge distillation: Self-training propagates a small annotation set to unlabeled instances using either hard labels [78] [79] or soft assignments [76].Soft-label approaches are often called knowledge distillation [35] and have primarily trained smaller networks to mimic larger networks for compression.

3. Approach

DINO frames self-supervised learning as knowledge distillation without a predefined teacher: a student matches momentum-teacher outputs across multi-crop views. The method uses shared architectures, teacher centering and sharpening to avoid collapse, and supports ViT or ResNet backbones.

  • DINO framework: DINO trains a student to match a teacher’s output distributions using cross-entropy, while both networks share the same architecture and the student is optimized by stochastic gradient descent.The teacher is not given a priori; it is built from past student iterations.
  • Multi-crop training: DINO applies multi-crop training with two global and several local views, sending all crops to the student but only global views to the teacher to encourage local-to-global correspondences.The basic setting uses global views at 224^2 resolution covering more than 50% of the image and local views at 96^2 covering less than 50%.
  • Teacher network: The teacher is updated from past student weights, with an exponential moving average working well, whereas directly copying student weights fails to converge.Freezing the teacher for an epoch also works surprisingly well.
  • Network architecture: The network combines a ViT [19] or ResNet [34] backbone with a 3-layer projection MLP, uses no predictor, and applies no batch normalization in ViT-based DINO.Downstream features come from the backbone, while the projection head includes hidden dimension 2048, ℓ2 normalization, and a weight-normalized K-dimensional layer.
  • Avoiding collapse: Centering and sharpening of momentum-teacher outputs stabilize DINO and avoid collapse without requiring contrastive loss, clustering constraints, predictors, or batch normalization.Centering prevents one dimension from dominating, while a low teacher temperature provides output sharpening.

4. Main Results

DINO performs strongly in standard ImageNet comparisons and reveals especially effective ViT features for retrieval, object segmentation, and downstream transfer. Larger ViTs help, but smaller patches provide a greater performance benefit without adding parameters.

  • Comparing with the same architecture: +3.5% with linear classification and +7.9% with k-NN evaluation, DINO ViTs outperform BYOL, MoCov2, and SwAV.DINO also performs on par with the state of the art for ResNet-50, validating the standard self-supervised setting.
  • Comparing across architectures: Reducing ViT patch size improves performance more than increasing architecture size, despite adding no parameters.The comparison concerns the best performance obtained across architectures and includes smaller “/8” variants.
  • Image Retrieval: DINO ViT features trained on GLDv2 outperform previously published off-the-shelf retrieval methods on landmark retrieval.Features are frozen and used directly with k-NN, with retrieval measured by mAP on revisited Oxford and Paris.
  • Probing the self-attention map: Different DINO self-attention heads focus on distinct semantic regions, including occluded or small objects, exposing image-segmentation information.The study also evaluates these features on DAVIS-2017 video instance segmentation without training or fine-tuning task-specific weights.
  • Downstream transfer: For ViT architectures, self-supervised DINO pretraining transfers better to downstream tasks than supervised ImageNet pretraining.The comparison finetunes features from the same architectures on each downstream task.

5. Ablation Study of DINO

The ablations identify momentum encoding, multi-crop augmentation, small patches, centering, and sharpening as important components of DINO for ViTs. Overall, DINO reaches 76.1% top-1 accuracy with reduced computational requirements compared with comparable convolutional self-supervised systems [30] [10].

  • Core components: Removing the momentum encoder causes DINO to fail, whereas adding stronger operations such as SK avoids collapse; with momentum, SK has little impact, highlighting the encoder’s importance.The default best combination includes the momentum encoder, multi-crop augmentation, and cross-entropy loss; Table 7 also reports BYOL [30], MoCo-v2 [15], and SwAV [10].
  • Patch size: Smaller input patches substantially improve k-NN performance without adding parameters, although 5×5 patches reduce throughput to 44 im/s.The patch-size comparison evaluates 16×16, 8×8, and 5×5 patches for ViT-S, plus 16×16 and 8×8 patches for ViT-B, after 300 training epochs.
  • Teacher network: The momentum teacher achieves 72.8% k-NN accuracy, outperforming previous-epoch, previous-iteration, and copied-student teachers, while also consistently outperforming the student during training.A teacher based on a recent student version does not converge in this setting, whereas the momentum teacher is the best-performing strategy but not the only viable option.
  • Collapse prevention: Centering and sharpening complement each other: omitting either makes the KL divergence converge to zero, indicating collapse, while applying both balances their opposing effects.Without centering, entropy converges to 0; without sharpening, it converges to −log(1/K), revealing distinct collapse modes.
  • Multi-crop training: Multi-crop training reaches 74.6% in 24 hours versus 72.5% in 46 hours without multi-crop, improving accuracy by +2% while halving training time at higher memory use.The multi-crop setting uses 2×224^2+10×96^2 crops, compared with 2×224^2 without multi-crop; the reported memory use is 15.4G versus 9.3G.
  • Overall result: 76.1% top-1 accuracy is achieved by DINO with two 8-GPU servers over 3 days, outperforming comparable convolutional self-supervised systems while requiring significantly fewer computational resources [30] [10].The result is reported for Vision Transformers and is accompanied by publicly available training code for limited-GPU settings.

6. Conclusion · Appendix

The authors conclude that self-supervised pretraining makes standard ViTs competitive with specialized convnets while yielding promising features for retrieval and scene-layout applications. They propose scaling DINO to large ViTs pretrained on random, uncurated images as future work.

  • 6. Conclusion: Self-supervised pretraining enables a standard ViT to achieve performance comparable with the best convnets designed for this setting.
  • 6. Conclusion: The quality of ViT features for k-NN classification suggests potential for image retrieval, where ViTs have already shown promising results [22].
  • 6. Conclusion: The features also contain information about scene layout that could be leveraged in future applications.
  • 6. Conclusion: Future work will test whether pretraining a large ViT with DINO on random, uncurated images can further improve visual features [28].
  • 6. Conclusion: The authors acknowledge collaborators and teams who contributed help, support, and project discussions.
  • 6. Conclusion: The work received funding from ERC grant 714381 (SOLARIS project) and ANR 3IA MIAI@Grenoble Alpes grant ANR-19-P3IA-0003.

A. Additional Results

Additional experiments show that DINO-trained ViT features outperform ResNet-50 features in both k-NN and linear evaluation, transfer competitively in low-shot learning, and improve supervised ViT training when used for pretraining.

  • k-NN classification: ViT-S accuracies are better than ResNet-50 accuracies with DINO in both linear and weighted k-NN evaluation protocols.Representations are extracted without data augmentation and evaluated on ImageNet-1k, Places205, PASCAL VOC, and Oxford-102 flowers.
  • Self-supervised ImageNet pretraining of ViT: DINO pretraining improves supervised ViT performance by +1% over random initialization.The comparison concerns supervised ViT models trained without additional data or models; the gain is not attributed to longer training.
  • Low-shot learning on ImageNet: With 1% and 10% labels, frozen DINO-trained ViT-S features are on par with state-of-the-art semi-supervised models.Performance is measured by logistic regression on frozen features without finetuning or data augmentation, comparing models with similar parameter counts and image/sec.

B. Methodology Comparison

DINO performs on par with SwAV and BYOL on ResNet-50 but substantially outperforms MoCo-v2, SwAV, and BYOL with ViTs. Ablations identify the momentum encoder and centered teacher outputs as important for performance and training stability.

  • Methodology comparison: DINO matches SwAV and BYOL with ResNet-50, then exceeds MoCo-v2, SwAV, and BYOL with ViT by 4.3% linear and 6.2% k-NN evaluation margins.The comparison uses ImageNet validation accuracy after 300 epochs for DEIT-small and ResNet-50.
  • Relation to MoCo-v2 and BYOL: Ablations comparing DINO with MoCo-v2 and BYOL vary the loss, student predictor, centering, projection-head batch normalization, and multi-crop training.DINO uses cross-entropy on sharpened softmax outputs, whereas MoCo-v2 uses InfoNCE and BYOL uses mean squared error on l2-normalized outputs.
  • Relation to SwAV: Momentum encoders are important for DINO performance and training stability, while centering alone fails without momentum and requires more advanced teacher-output operations.The SwAV comparison varies momentum encoders and teacher-output operations including centering, batch-axis softmax, and Sinkhorn-Knopp.
  • Relation to SwAV: A batch-axis softmax, equivalent to a single Sinkhorn-Knopp iteration, works competitively with SwAV by selecting each dimension’s best matches in the batch.The simplified operation removes the need for normalization beyond centering.
  • Validating our implementation: The authors’ reproductions match or exceed published ResNet-50 results, including BYOL at 72.7% versus 72.5% and MoCo-v2 at 71.1% versus 71.1%.These comparisons use the 300-epoch setting, while the cited MoCo-v2 result used 800 epochs.

C. Projection Head

The DINO projection head uses an MLP followed by ℓ2 normalization and a weight-normalized output layer, with GELU activations and no batch normalization. The ℓ2 bottleneck stabilizes deeper heads, large output dimensions improve performance, and changing GELU to ReLU has little impact.

  • C. Projection Head: DINO uses an MLP projection head with 2048-dimensional GELU hidden layers, followed by ℓ2 normalization and a weight-normalized K-dimensional fully connected layer.The final MLP layer omits GELU.
  • C. Projection Head: DINO with ViT is entirely batch-normalization-free because adding batch normalization to the projection heads has little impact.The BN-free design is reported as an advantage for achieving state-of-the-art performance without batch normalization.
  • C. Projection Head: Without the ℓ2-normalization bottleneck, DINO training fails as projection-head depth increases, whereas the bottleneck stabilizes deeper heads.The comparison uses ImageNet top-1 k-NN accuracy after 100 epochs of pre-training with ViT-S/16 and K=4096.
  • C. Projection Head: Larger output dimensionality improves performance, and the default projection-head settings are K=65536 and d=256 for the bottleneck.The ℓ2 bottleneck allows a large output dimension with only a moderate increase in total parameters.
  • C. Projection Head: Replacing GELU with ReLU in the projection head has relatively little impact on performance.GELU is retained by default for consistency with ViT activations.

D. Additional Ablations

Additional ablations show that DINO is sensitive to overly slow centering and insufficient sharpening, while longer training, momentum teachers, self-supervised attention, and more heads improve or preserve its behavior under stated conditions.

  • Online centering and sharpening: DINO avoids collapse across centering updates except when smoothing is too slow (m = 0.999), while teacher temperatures above 0.06 require warm-up.In practice, τt is warmed linearly from 0.04 to 0.07 during the first 30 epochs; without suitable sharpening, the loss converges to ln(K).
  • Longer training: 74.5 k-NN top-1 is reached after 800 epochs versus 72.8 after 300 and 70.9 after 100 for DINO ViT-S.BYOL on ViT-S instead worsened beyond 300 epochs, so comparisons report BYOL for 300 epochs and DINO for 800 epochs.
  • The teacher outperforms the student: The momentum teacher outperforms the student with both ViT and ResNet-50, supporting DINO’s interpretation as Mean Teacher [65] self-distillation.Weight averaging can produce a better model than individual iteration models, improving the student through better targets and subsequently improving the teacher.
  • Self-attention maps from supervised versus self-supervised learning: Self-supervised ViT self-attention maps consistently contain scene layout and object boundaries across different self-supervised methods.The evaluation thresholds attention maps to retain 80% of their mass and compares masks with PASCAL VOC12 validation ground truth using Jaccard similarity.
  • Impact of the number of heads in ViT-S: Increasing the number of ViT-S heads improves performance but slightly reduces inference throughput; the paper’s default DeiT-S [69] uses six heads.Throughput is measured in images per second at inference on a single V100 GPU.

E. Multi-crop

The appendix studies multi-crop training in DINO, varying crop scales and comparing its effects across self-supervised frameworks. Multi-crop is framework-dependent and can improve early BYOL transfer performance before later decline.

  • Range of scales in multi-crop: The multi-crop setup uses two 224^2 global views and six 96^2 local views, with global scales sampled from (s, 1) and local scales from (0.05, s).The non-overlapping scale ranges follow SwAV’s original design, although overlapping ranges and finer hyperparameter searches could be explored.
  • Multi-crop in different self-supervised frameworks: Multi-crop does not benefit all frameworks equally, making its effectiveness framework-dependent rather than a simple add-on [16].Comparisons use MoCo-v2 [15], BYOL [30], and SwAV [10] with ViT-S/16 after 300 training epochs, evaluating k-NN and linear probing.
  • Training BYOL with multi-crop: With BYOL and ViT-S, multi-crop initially improves transfer performance over the two-crop baseline, but performance growth slows and declines after a certain training duration.The same pattern persists across learning-rate, weight-decay, and crop-number sweeps, including 2, 4, or 6 small crops.

F. Evaluation Protocols … H. Class Representation

The evaluation freezes pretrained representations and tests them with weighted k-NN or linear classifiers, while visual analyses examine self-attention and class structure in DINO’s feature space.

  • F. Evaluation Protocols: The evaluation protocols measure frozen representations using weighted k-NN and supervised linear classification, avoiding feature fine-tuning during downstream assessment.The k-NN protocol stores training features, while linear evaluation trains a classifier on top of frozen features.
  • F.1 k-NN classification: k = 20 consistently gives the best k-NN accuracy, using αi = exp(Tix/τ) with τ = 0.07 and requiring neither hyperparameter tuning nor augmentation.The protocol compares each test representation with stored training features and can run in one downstream-data pass.
  • F.2 Linear classification: Linear evaluation trains with SGD for 100 epochs using batch size 1024, no weight decay, learning-rate sweeps, standard crops and flips, and central-crop top-1 accuracy.The projection head is removed, and the classifier is trained on frozen features.
  • F.2 Linear classification: For ViT-S, concatenating the [CLS] tokens from the last l layers is optimal at l = 4.This feature-based evaluation follows the layer-concatenation design used in BERT [18].
  • F.2 Linear classification: For ViT-B, the final layer alone is used because concatenating the last l layers provides no performance gain; pooled patch tokens are combined with the final [CLS] token.This adapts the convnet global-average-pooling evaluation design to ViT representations.
  • G. Self-Attention Visualizations: DINO self-attention is visualized on randomly selected, training-unused COCO validation images, including last-layer heads queried by the [CLS] token.The visualizations include a DINO ViT-S/8 and emphasize that the [CLS] token receives no label or supervision.
  • H. Class Representation: DINO’s t-SNE embedding of ImageNet class-average validation features recovers meaningful structure, grouping similar animal species together.Features are reduced to 30 dimensions with PCA before t-SNE using perplexity 20, learning rate 200, and 5000 iterations.
Loading 2104.14294v2…