PaperlayerView exact revision on arXiv

Source-linked AI summary

Emerging Properties in Self-Supervised Vision Transformers

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin

arXiv:2104.14294v2cs.CV

TL;DR

The paper asks whether self-supervised pretraining gives ViTs distinctive visual features compared with convnets. It introduces DINO, a label-free self-distillation approach, and reports scene-layout information in self-supervised ViT features and 80.1% top-1 on ImageNet linear evaluation with ViT-Base.

  • Problem

    The paper asks whether self-supervised pretraining gives ViTs distinctive features compared with convnets, against a backdrop of ViTs having no clear benefits or unique feature properties.

    Source: §1 · p. 1

  • Method

    DINO is a label-free self-distillation approach that trains a student to predict the output of a teacher network built from past student iterations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Self-supervised ViT features encode scene layout and object boundaries and perform well with k-NN; DINO with ViT-Base reaches 80.1% top-1 on ImageNet linear evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that self-supervised learning could be key to developing a BERT-like model based on ViT.

    Source: §6 · p. 10

    Whether pretraining a large ViT with DINO on random, uncurated images can further push visual-feature limits remains for future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

1. Introduction

The introduction argues that self-supervised pretraining may give ViTs distinctive features beyond supervised training, and presents DINO as a simple label-free method. It highlights segmentation information, strong k-NN performance, and ImageNet classification results.

1. Introduction

  • Self-supervised ViT features explicitly encode scene layout and object boundaries, accessible through last-block self-attention, unlike supervised ViTs or convnets.Source: §1 · p. 2
  • 78.3% top-1 accuracy on ImageNet with a basic k-NN classifier, without finetuning, a linear classifier, or data augmentation.Source: §1 · p. 2
  • Segmentation masks appear across self-supervised methods, while strong k-NN performance depends on momentum encoders and multi-crop augmentation; smaller ViT patches improve feature quality.Source: §1 · p. 2
  • DINO predicts a momentum teacher's output with a standard cross-entropy loss, and works with convnets and ViTs without architecture changes or adapted internal normalization.Centering and sharpening the teacher output can prevent collapse; predictors, advanced normalization, and contrastive loss add little stability or performance benefit.Source: §1 · p. 2
  • 80.1% top-1 accuracy on ImageNet linear classification with small-patch ViT-Base; DINO reaches 76.1% using two 8-GPU servers over three days.The paper also reports that DINO matches the state of the art with ResNet-50.Source: §1 · p. 2

2. Related work

The related work places the method between instance-discriminative and non-discriminative self-supervision, and extends knowledge distillation into a label-free objective with a dynamically updated teacher.

Self-supervised learning

  • Instance-discrimination methods treat images as classes [12] [33], but classification does not scale well, NCE needs many comparisons, and clustering offers an alternative [2] [8] [9] [36] [42] [74] [80] [85].NCE methods compare instances and typically require large batches [12] or memory banks [33].Source: §2 · p. 2
  • Inspired by BYOL [30], which matches features to momentum-encoder representations, the method uses a different similarity loss and identical student and teacher architectures to frame label-free learning as Mean Teacher self-distillation [65].BYOL can work without a momentum encoder, but with reduced performance [16] [30].Source: §2 · pp. 2–3

Self-training and knowledge distillation

  • Self-training propagates annotations through hard [78] [79] or soft labels [76]; the method extends distillation [35] to unlabeled data, building on the link between soft pseudo-labeling and self-training [76].Knowledge distillation was primarily designed to train smaller networks to mimic larger ones for model compression [35].Source: §2 · p. 3
  • Unlike prior combinations of self-supervision and distillation [25] [63] [13] [47], which use a fixed pretrained teacher, this method updates its teacher during training; unlike codistillation [1], its teacher is updated by averaging the student.Distillation becomes the self-supervised objective rather than a post-processing step.Source: §2 · p. 3

3. Approach

DINO trains a student to match a momentum teacher’s outputs across image crops, using centering and sharpening to prevent collapse. The section also describes the ViT architecture, training setup, and feature-evaluation protocols.

3.1. SSL with Knowledge Distillation

  • DINO trains identical student and teacher networks by minimizing cross-entropy between their output distributions across views, with the student processing all crops and the teacher only global crops.The basic multi-crop setting uses two 224 × 224 global views and several 96 × 96 local views, encouraging local-to-global correspondences.Source: §3.1 · p. 3
  • An exponential-moving-average teacher, updated with a cosine-scheduled momentum from 0.996 to 1, provides higher-quality targets and guides the student during training.The teacher performs model ensembling; copying student weights directly fails to converge, while freezing the teacher over an epoch works surprisingly well.Source: §3.1 · p. 4
  • DINO uses a backbone and a three-layer projection head; for ViTs, the student and teacher share this architecture, and the projection heads omit batch normalization.The backbone features, rather than projection-head outputs, are used for downstream tasks.Source: §3.1 · p. 4
  • Centering and sharpening the momentum teacher’s outputs balance opposing collapse tendencies, with an exponential-moving-average center and a low teacher temperature.Centering prevents one dimension from dominating but encourages uniform outputs; sharpening has the opposite effect.Source: §3.1 · p. 4

3.2. Implementation and evaluation protocols

  • The ViT implementation forms patch embeddings, adds a learnable [CLS] token, and feeds both patch and class tokens through a Transformer; the study typically uses 16 × 16 or 8 × 8 patches.Models are pretrained without labels on ImageNet using AdamW, batch size 1024 for ViT-S/16, BYOL augmentations, and multi-crop.Source: §3.2 · pp. 4–5
  • Feature quality is evaluated with frozen-feature linear classification, downstream fine-tuning, and weighted k-NN, for which 20 neighbors worked best in most runs.The k-NN protocol avoids additional hyperparameter tuning and data augmentation, and requires only one pass over the downstream dataset.Source: §3.2 · p. 5

4. Main Results

The results validate DINO on ImageNet and show that DINO-trained ViTs yield strong k-NN and retrieval performance, expose semantic structure, and transfer well to downstream tasks.

4.1. Comparing with SSL frameworks on ImageNet

  • With ViT-S, DINO beats BYOL, MoCov2, and SwAV by +3.5% in linear classification and +7.9% in k-NN, with 74.5% k-NN versus 77.0% linear accuracy.DINO performs on par with the state of the art on ResNet-50; the near-parity between k-NN and linear classification emerges with DINO ViTs, not other tested methods or ResNet-50.Source: §4.1 · p. 6
  • With 8 × 8 patches, DINO ViT-Base achieves 80.1% top-1 in linear classification and 77.4% with a k-NN classifier, using 10× fewer parameters and 1.4× faster runtime than the prior state of the art [13].Reducing patch size has a bigger impact on performance than training a larger ViT, without adding parameters, though it increases memory usage.Source: §4.1 · p. 6

4.2.1 Nearest neighbor retrieval with DINO ViT

  • Frozen DINO features outperform supervised ImageNet features on Oxford/Paris retrieval; GLDv2-pretrained features surpass published off-the-shelf descriptors, and DINO ViT features are competitive on Copydays.Retrieval uses frozen features with k-NN; Copydays uses cosine similarity on concatenated [CLS] and GeM-pooled patch-token features, with whitening.Source: §4.2.1 · p. 6, Table 3 · p. 6, Table 4 · p. 6

4.2.2 Discovering the semantic layout of scenes

  • Frozen DINO features perform competitively on DAVIS segmentation, with ViT-B/8 gaining +9.1% (J &F)m; attention heads align with semantic regions better than supervised ViT attention.The DAVIS evaluation does not train or finetune on the task; different attention heads can focus on different objects or parts, including small or occluded ones.Source: §4.2.2 · p. 7, Table 5 · p. 7, Figure 3 · p. 7

4.2.3 Transfer learning on downstream tasks

  • On downstream tasks, self-supervised DINO pretraining transfers better than same-architecture ImageNet supervision for ViTs and improves ImageNet results by +1-2%.The downstream features are finetuned following the protocol used in Touvron et al. [69].Source: §4.2.3 · p. 7

5. Ablation Study of DINO

The ablations show how DINO’s components, teacher, and training choices affect ViT feature quality, collapse prevention, and compute. They also show that DINO can train effectively with smaller batches and that multi-crop improves its accuracy–time tradeoff.

5.1. Importance of the Different Components

  • Momentum, multi-crop training, and DINO’s cross-entropy loss are important for good features; without momentum the framework fails, while adding a predictor has little impact here, unlike in BYOL [16] [30].With momentum, SK has little impact, whereas comparing variants highlights the importance of the momentum encoder.Source: Table 7 · p. 8, §5.1 · p. 8
  • Smaller patches greatly improve k-NN performance without adding parameters, but 5×5 patches reduce throughput to 44 im/s versus 180 im/s for 8×8 patches.The comparison covers ViT-S and ViT-B models trained for 300 epochs.Source: Figure 5 · p. 8, §5.1 · p. 8

5.2. Impact of the choice of Teacher Network

  • The momentum teacher outperforms the student during training and gives superior performance to a previous-epoch teacher, which remains competitive with existing frameworks.A teacher based on a recent student version does not converge in this setting; the authors interpret momentum as exponentially decayed Polyak-Ruppert averaging that guides the student.Source: Figure 6 · p. 9, §5.2 · p. 9

5.3. Avoiding collapse

  • Centering and sharpening counterbalance different collapse modes: without either operation KL converges to zero, while applying both balances their effects.Without centering, entropy converges to 0; without sharpening, it converges to −log(1/K).Source: Figure 7 · p. 9, §5.3 · p. 9

5.4. Compute requirements

  • 74.6% in 24 hours with 2×224^2+10×96^2 crops compares with 72.5% in 46 hours without multi-crop, using more memory: 15.4G versus 9.3G.The passage attributes the gain to local-to-global augmentation and says adding views yields diminishing gains for longer training.Source: Table 8 · p. 9, §5.4 · p. 10
  • 76.1 top-1 accuracy was achieved by DINO with Vision Transformers using two 8-GPU servers for 3 days, outperforming comparable-size convolutional self-supervised systems with less computation [30] [10].The passage reports this as an overall result for DINO with Vision Transformers.Source: §5.4 · p. 10

5.5. Training with small batches

  • DINO reaches high performance with batch size 128, slightly below the default batch size 1024; a batch size of 8 reaches 35.2% after 50 epochs.The batch-size-128 experiment runs on one GPU, and the authors note that smaller batches may require hyperparameter retuning.Source: Table 9 · p. 10, §5.5 · p. 10

6. Conclusion · Appendix

The conclusion presents self-supervised ViT pretraining as a route toward a BERT-like vision model, with features that may support retrieval and weakly supervised segmentation. It also proposes testing DINO on large ViTs and random uncurated images.

6. Conclusion

  • The authors argue that self-supervised learning could be key to developing a BERT-like model based on ViT.They describe this as the paper’s main result.Source: §6 · p. 10
  • Self-supervised pretraining of a standard ViT achieved performance comparable with the best convnets designed for this setting.Source: §6 · p. 10
  • The quality of self-supervised ViT features in k-NN classification may support image retrieval, where ViTs show promising results [22].Source: §6 · p. 10
  • Scene-layout information in the features may benefit weakly supervised image segmentation.Source: §6 · p. 10
  • Future work will explore whether pretraining a large ViT with DINO on random uncurated images can push the limits of visual features [28].Source: §6 · p. 10

A. Additional Results

DINO-trained ViT features perform strongly with k-NN classification and frozen-feature low-shot evaluation, while DINO pretraining also improves supervised ViT performance and narrows the gap to models using extra data or convnet distillation.

k-NN classification

  • On ImageNet 1%, DINO-pretrained ViT-S outperforms ResNet-50 by +14.1% with k-NN evaluation, a larger gap than with linear evaluation.The study evaluates frozen representations without data augmentation and reports that ViT-S also has higher accuracy than ResNet-50 with a linear classifier; DINO ViT features are particularly k-NN-friendly.Source: Table 10 · p. 13, A. Additional Results · p. 13

Self-supervised ImageNet pretraining of ViT

  • DINO pretraining improves supervised ViT performance by +1% over random initialization and narrows the gap to models pretrained on extra data or distilled from a convnet.The gain is not attributed to longer training: supervised pretraining instead of DINO does not improve performance.Source: A. Additional Results · p. 13, Table 11 · p. 13

Low-shot learning on ImageNet

  • With 1% and 10% labels, DINO ViT-S features are on par with state-of-the-art semi-supervised models using a logistic regression on frozen features, without finetuning or data augmentation.The evaluation trains a multiclass logistic regression on frozen features.Source: A. Additional Results · p. 14, Table 12 · p. 14

B. Methodology Comparison

DINO performs on par with other self-supervised frameworks on ResNet-50 but gains large margins with ViT; ablations highlight the roles of momentum encoding and multi-crop training.

Methodology comparison

  • With ViT, DINO outperforms MoCo-v2 [15], SwAV [10], and BYOL [30] by +4.3% in linear and +6.2% in k-NN evaluations; with ResNet-50, it performs on par.Table 13 reports ImageNet validation accuracy after 300 epochs of pre-training for DeiT-small and ResNet-50.Source: Table 13 · p. 14, B. Methodology Comparison · p. 14

Relation to MoCo-v2 and BYOL

  • DINO’s cross-entropy loss can be replaced with MSE, but performance changes; adding a predictor has little impact, while BYOL needs one to prevent collapse.Teacher-output centering lets BYOL avoid collapse without a predictor or batch normalization, but with a significant performance drop, likely because centering is designed to work with sharpening.Source: B. Methodology Comparison · p. 14, Table 14 · p. 15
  • Multi-crop benefits DINO and MoCo-v2, whose performance drops 2–4% without it, while adding it to BYOL does not work out of the box.The passage notes that BYOL may require further adaptation for multi-crop training.Source: B. Methodology Comparison · p. 14, Table 14 · p. 15

Relation to SwAV

  • For ViT, a momentum encoder significantly improves performance and stabilizes training; without momentum, centering alone fails and more advanced operations are required.With a momentum encoder, centering also avoids collapse and removes the need for normalization beyond centering.Source: B. Methodology Comparison · pp. 14–15, Table 15 · p. 15

Details on the Softmax(batch) variant

  • The batch-axis softmax variant of SwAV’s Sinkhorn-Knopp operation works competitively with SwAV and selects each dimension’s best matches in the batch.A single Sinkhorn iteration simplifies to the softmax(batch) variant.Source: B. Methodology Comparison · p. 15, Table 15 · p. 15

Validating our implementation

  • The reproduced BYOL result is 72.7% after 300 epochs versus [30]’s 72.5%; reproduced MoCo-v2 is 71.1% after 300 epochs versus [15]’s 71.1% after 800.The authors attribute their improvement over [15]’s implementation to a larger projection head with three layers, batch normalization, and projection dimension 256.Source: B. Methodology Comparison · p. 15

Relation to other works

  • DINO is a soft UIC variant with a momentum teacher, using centering to prevent collapse rather than UIC’s balance sampling techniques [8] [14].UIC uses outputs from the previous epoch as hard pseudo-labels for unsupervised classification.Source: B. Methodology Comparison · pp. 15–16
  • Concurrent CsMI [77] also achieves strong ImageNet k-NN performance, including with convnets, and combines a momentum network with multi-crop training; the authors leave component analysis for future work.The paper identifies this investigation as a way to clarify which components matter for good k-NN performance.Source: B. Methodology Comparison · p. 16

C. Projection Head

The projection head improves DINO accuracy, and its design choices support stable training and strong performance without batch normalization. A deep head benefits from an L2-normalization bottleneck, while larger output dimensions improve performance.

Projection head

  • The projection head uses an n-layer MLP with 2048-dimensional GELU hidden layers, followed by a GELU-free layer, L2 normalization, and a weight-normalized K-dimensional fully connected layer.Inspired by SwAV’s prototype-layer projection head [10], this design uses no batch normalization; the projection head itself follows prior self-supervised frameworks [12] and uses a weight-normalized layer [16] [61].Source: C. Projection Head · p. 16

BN-free system

  • Adding batch normalization to projection heads has little impact, so DINO with ViT uses no batch normalization anywhere.The paper notes this BN-free system avoids the training slowdown associated with batch normalization, especially when synchronized across processes.Source: C. Projection Head · p. 16

L2-normalization bottleneck

  • The L2-normalization bottleneck stabilizes DINO training as projection-head depth increases, and increasing depth improves accuracy.Without the bottleneck, training fails when head depth increases; the default head has four linear layers: three in the MLP and one after the bottleneck.Source: C. Projection Head · p. 16

Output dimension

  • Larger output dimensionality improves performance; the default is K=65536 with bottleneck dimension d=256.The L2-normalization bottleneck permits a large output dimension with a moderate increase in parameter count.Source: C. Projection Head · p. 16

GELU activations

  • Replacing GELU with ReLU in the projection head has relatively little impact on accuracy.GELU is used by default for consistency with ViT activations.Source: C. Projection Head · p. 16

D. Additional Ablations

The ablations show that longer training improves DINO ViT-S k-NN accuracy, while centering and sharpening have collapse-sensitive settings. They also examine teacher–student performance, self-attention, and ViT head count.

Online centering

  • Centering convergence is robust across smoothing settings, but the update is too slow and the model collapses at m = 0.999.Source: D. Additional Ablations · p. 17

Sharpening

  • Teacher temperature below 0.06 is required to avoid collapse; above 0.06, loss converges to ln(K), unless temperature is warmed linearly from 0.04 to 0.07 over 30 epochs.As temperature approaches zero, sharpening becomes argmax and yields one-hot hard distributions.Source: D. Additional Ablations · p. 17

Longer training

  • k-NN top-1 rises from 70.9 at 100 epochs to 72.8 at 300 and 74.5 at 800 for DINO ViT-S.BYOL on ViT-S performed worse beyond 300 epochs, so comparisons report BYOL at 300 epochs and SwAV, MoCo-v2, and DINO at 800 epochs.Source: D. Additional Ablations · p. 17

The teacher outperforms the student

  • The momentum teacher outperforms the student with ViT and ResNet-50, supporting a Mean Teacher [65] self-distillation view in which student representations improve and the teacher then improves from student weights.Weight averaging can produce a better model than individual iteration models.Source: D. Additional Ablations · p. 17

Self-attention maps from supervised versus self-supervised learning

  • With self-attention masks retaining 80% of map mass, self-supervised ViT maps across methods show scene layout and object boundaries on PASCAL VOC12 validation images.The masks are evaluated against ground-truth segmentation using Jaccard similarity.Source: D. Additional Ablations · p. 17

Impact of the number of heads in ViT-S

E. Multi-crop

Multi-crop training affects self-supervised frameworks differently, with DINO benefiting most; its scale settings and BYOL training dynamics qualify its effects.

E. Multi-crop

  • The best multi-crop scale parameter s was around 0.3 in these experiments, higher than SwAV’s 0.14.The setup used two global views and six local views with non-overlapping scale ranges; the authors note that overlapping ranges and finer searches could yield a more optimal setting.Source: E. Multi-crop · p. 18
  • DINO gains +3.4% in linear evaluation with multi-crop and leads other frameworks by a moderate 1% without it.Multi-crop does not benefit all frameworks equally, and framework rankings depend on the evaluation protocol.Source: E. Multi-crop · p. 18
  • With BYOL and ViT-S, multi-crop initially improves transfer performance, but its growth slows and later declines.The pattern persisted across learning-rate, weight-decay, and crop-count sweeps and also appeared with ResNet-50; low learning rates avoided the breakpoint but had low overall accuracy, so the authors leave the interaction for future study.Source: E. Multi-crop · p. 18

F. Evaluation Protocols … H. Class Representation

The evaluation protocols assess frozen ViT features with weighted k-NN and linear classification, while the visualizations examine DINO attention and the organization of ImageNet classes in feature space.

F.1 k-NN classification

  • The k-NN evaluation uses frozen [CLS] features and similarity-weighted voting, with k = 20 best across runs and no tuning, augmentation, or more than one downstream-data pass required.The vote weights are exp(T_ix/τ), with τ = 0.07.Source: §F.1 · p. 18

F.2 Linear classification

  • The linear evaluation trains an SGD classifier on frozen features for 100 ImageNet epochs, sweeping the learning rate and using random resized crops and horizontal flips; it reports central-crop top-1 accuracy.The projection head is removed; training uses batch size 1024 and no weight decay.Source: §F.2 · pp. 18–19
  • For ViT-S, concatenating the last four [CLS] tokens is optimal; for ViT-B, the final [CLS] token is combined with globally average-pooled patch tokens.ViT-B gains no performance from concatenating the last layers.Source: §F.2 · p. 19

G. Self-Attention Visualizations

  • DINO ViT-S/8 last-layer attention is visualized on randomly selected COCO validation images that were not used for training.The paper shows attention from several reference points, and Figure 10 examines last-layer heads queried by the [CLS] token, which has no label supervision.Source: G. Self-Attention Visualizations · p. 19, Figure 10 · p. 20

H. Class Representation

  • DINO’s ImageNet class embeddings form coherent clusters of similar species, including birds and dogs, with terriers especially grouped together.Each class is represented by the mean feature of its validation images; features are reduced to 30 dimensions with PCA before t-SNE.Source: H. Class Representation · p. 19, Figure 11 · p. 21
  • The class-embedding visualization uses t-SNE with perplexity 20, learning rate 200, and 5000 iterations on the PCA-reduced features.The inputs are average features for ImageNet validation images within each class.Source: H. Class Representation · p. 19, Figure 11 · p. 21

Related papers

Loading 2104.14294v2…