PaperlayerView exact revision on arXiv

Source-linked AI summary

SEED: Self-supervised Distillation For Visual Representation

Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, Zicheng Liu

arXiv:2101.04731v2cs.CVcs.AI

TL;DR

Small networks perform poorly with existing contrastive self-supervised learning, motivating SEED, which distills a larger self-supervised teacher’s similarity distributions into a smaller student without labels. SEED substantially improves small-model ImageNet performance, including EfficientNet-B0 and MobileNet-v3-Large linear-probe accuracy.

  • Problem

    Existing self-supervised methods work well with large networks but perform poorly on small models, which the authors conjecture reflects limited capacity to learn instance-discriminative representations.

    Source: §1 · pp. 1–2

  • Method

    SEED pretrains a large teacher self-supervised, freezes it, and trains a smaller student to mimic its similarity-score distribution over an instance queue without labeled data.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    67.6% top-1 accuracy on EfficientNet-B0 and 68.2% on MobileNet-v3-Large are achieved with SEED, versus 42.2% and 36.3% for MoCo-V2.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SEED effectively addresses the weakness of self-supervised learning for small models and achieves state-of-the-art results across various small-architecture benchmarks.

    Source: §5 · p. 9

    The paper notes that larger training sets and more fine-tuning iterations can reduce the importance of initialization, limiting how strongly pretrained weights matter in that setting.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.

1 INTRODUCTION

SEED addresses the weak performance of contrastive self-supervised learning on small networks by distilling representations from a larger self-supervised teacher without labeled data. It substantially improves small-model ImageNet linear-probe accuracy.

Motivation

  • Existing self-supervised methods mainly use large networks, while contrastive learning performs poorly on small architectures.The paper reports this limitation empirically for small networks.Source: §1 · p. 1
  • 36.3% top-1 accuracy with MobileNet-V3-Large under MoCo-V2 trails its supervised training accuracy of 75.2%.Source: §1 · pp. 1–2
  • 42.2% top-1 accuracy with EfficientNet-B0 under MoCo-V2 trails its supervised training accuracy of 77.1%.Source: §1 · p. 2

SEED

  • SEED trains a large model self-supervised as teacher, then distills its knowledge into a smaller student without labeled data.The approach differs from conventional supervised distillation because both training stages are self-supervised.Source: §1 · p. 2
  • SEED can use different advanced SSL methods for the teacher and transfer knowledge to target networks with shallower, thinner, or different architectures.Source: §1 · p. 2

Results

  • 67.6% top-1 accuracy on EfficientNet-B0 and 68.2% on MobileNet-V3 follow SEED, versus 42.2% and 36.3% for MoCo-V2.Source: §1 · p. 2

2 RELATED WORK

Related work covers contrastive self-supervised representation learning and knowledge distillation, including objectives based on logits, attention maps, feature maps, and probability consistency.

Contrastive self-supervised learning

  • Contrastive self-supervised methods estimate latent distributions by contrasting representations with random or generated negatives.Source: §2 · pp. 2–3
  • Prior contrastive approaches improve efficiency, representation consistency, and the alignment-uniformity properties of learned features.Source: §2 · pp. 2–3

Knowledge distillation

  • Knowledge distillation transfers knowledge from a large model to a smaller one through objectives such as logits, attention maps, feature maps, or probability consistency.Source: §2 · p. 3

3 METHOD

SEED transfers a frozen teacher’s self-supervised similarity structure to a student by matching teacher and student distributions over a dynamically maintained instance queue. The objective uses augmented views and cross-entropy distillation.

3.1 PRELIMINARY ON KNOWLEDGE DISTILLATION

  • Knowledge distillation trains a student to mimic a teacher while combining supervised prediction loss with a distillation loss.In the supervised formulation, the distillation signal may come from softmax predictions or intermediate feature maps.Source: §3.1 · p. 3

3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION

  • SEED removes the need for labels by transferring knowledge from a self-supervised large model to a smaller model for downstream representations.Source: §3.2 · p. 3
  • The teacher is pretrained and frozen, while the student minimizes cross-entropy between teacher and student similarity-score distributions over queued instances.Source: Figure 2 · p. 4, §3.2 · p. 4
  • For an augmented image view, teacher and student encoders produce normalized feature representations used to compute similarities against the instance queue.Source: §3.2 · p. 4
  • The queue stores teacher features and is updated first-in-first-out as current-batch features are enqueued and earliest samples are removed.Source: §3.2 · p. 4
  • SEED adds the teacher embedding itself to the queue, forming D+ so the student can align with the teacher representation while contrasting unrelated features.Source: §3.2 · pp. 4–5
  • As the teacher temperature approaches zero, its similarity distribution becomes one-hot and the objective becomes similar to Info-NCE.Source: §3.2 · p. 5

4 EXPERIMENT

Experiments evaluate SEED across ImageNet classification, semi-supervised learning, transfer tasks, teacher configurations, pre-training algorithms, and distillation strategies. Across these settings, distillation generally improves small-model representations, with gains depending on teacher scale, training choices, and downstream task.

Evaluation setup

  • SEED distillation transfers self-supervised knowledge from a larger teacher to smaller student networks for downstream evaluation.Teachers are self-supervised pretrained, while students include MobileNet-v3-Large, EfficientNet-B0, and smaller ResNets.Source: §4.1 · p. 5, §4.2 · p. 5

ImageNet evaluation

  • 68.2% Top-1 accuracy is achieved by MobileNet-V3-Large with a ResNet-50×2 teacher and multi-crop distillation, versus 36.3% for its MoCo-V2 baseline.A ResNet-152 teacher reaches 61.4%, while the wider ResNet-50×2 teacher with multi-crop training reaches 68.2%.Source: §4.2 · p. 6

Semi-supervised evaluation

  • SEED also improves semi-supervised ImageNet accuracy, and stronger teachers with more parameters produce better-performing students.The evaluation uses fixed 1% and 10% annotation subsets with linear fine-tuning.Source: §4.2 · pp. 6–7, Figure 3 · p. 6

Transfer classification

  • SEED surpasses contrastive self-supervised pre-training consistently on CIFAR-10, CIFAR-100, and SUN-397 transfer classification benchmarks.These evaluations test whether improvements learned on ImageNet transfer across domains and classes.Source: §4.2 · p. 7

Detection and segmentation

  • With ResNet-152 distillation, ResNet-18 Faster R-CNN gains +0.7 AP, +1.4 AP50, and +1.6 AP75 on VOC, while COCO improvements are relatively minor.The experiments evaluate object detection on VOC and detection plus instance segmentation on COCO.Source: §4.2 · p. 7

Ablations

  • Teacher depth and width improve accuracy, but further enlargement has limited effects, suggesting the student capacity can become the limiting factor.SEED is also agnostic to teacher pre-training algorithms, and longer teacher or distillation training can provide additional gains.Source: §4.3 · p. 8

5 CONCLUSIONS

SEED addresses the weakness of self-supervised learning for small models through distillation from a larger SSL-pretrained model, achieving state-of-the-art results across small architectures.

  • SEED targets the low instance-discrimination capacity that limits self-supervised learning on smaller architectures.Source: §5 · p. 9
  • SEED learns representations by distilling knowledge from a larger self-supervised model rather than directly training the small model on unlabeled data.Source: §5 · p. 9
  • SEED achieves state-of-the-art results on various benchmarks for small architectures.Source: §5 · p. 9

A APPENDIX

The appendix provides additional details and hyperparameter settings for SEED during distillation.

  • The appendix discusses further details of SEED distillation.Source: A APPENDIX · p. 13
  • The appendix covers different hyperparameters used during distillation.Source: A APPENDIX · p. 13
  • These appendix materials supplement the description of SEED during distillation.Source: A APPENDIX · p. 13

A.1 PSEUDO-IMPLEMENTATIONS

The pseudo-implementation defines teacher and student encoders, maintains a queue of representations, computes their score distributions, and updates the queue during distillation.

  • A queue Q maintains previous representations, while the earliest B instances are removed after each update.Source: §A.1 · p. 13
  • The pseudo-implementation treats T as the teacher encoder and S as the target student encoder.Source: §A.1 · p. 13
  • Teacher and student temperatures are applied to produce soft labels before the loss is computed.Source: §A.1 · p. 13
  • The teacher operates in evaluation mode, and the current batch embedding from T is inserted into the distillation process.Source: §A.1 · p. 13
  • The implementation computes teacher and student probability score distributions from their embeddings and the queued representations.Source: §A.1 · p. 13

A.1.1 DATA AUGMENTATIONS

Teacher pre-training and distillation use a shared augmentation pipeline combining crops, color transformations, blur, grayscale conversion, and horizontal flips.

  • Both teacher pre-training and distillation adopt the same data augmentations.Source: §A.1.1 · p. 14
  • Random resized crops produce 224×224 images from randomly selected scales between 0.2 and 1.0.Source: §A.1.1 · p. 14
  • The pipeline applies color jittering, grayscale conversion, Gaussian blur, and horizontal flipping with specified probabilities.Source: §A.1.1 · p. 14

A.1.2 PRE-TRAINING AND DISTILLATION ON MOBILENET AND EFFICIENTNET

Small architectures struggle with contrastive self-supervised pre-training, motivating architectural changes and distillation-oriented evaluation for EfficientNet and MobileNet.

Motivation

  • Smaller models are hypothesized to struggle because their limited parameters cannot discriminate among large numbers of categories and data.Source: §A.1.2 · p. 14

Architectural manipulation

  • Adding a deeper projection head substantially improves representation quality and linear-evaluation performance for small architectures.The experiments add an additional linear projection head above convolutional backbones.Source: §A.1.2 · p. 14, Table 6 · p. 14

A.2 ADDITIONAL DETAILS OF EVALUATIONS

The evaluation suite covers semi-supervised classification, transfer learning, detection, segmentation, and multiple distillation-view strategies across ImageNet and other datasets.

Classification evaluation

  • Semi-supervised ImageNet evaluation trains fully connected layers using 1% and 10% class-balanced labeled subsets after distillation.The subsets contain roughly 12 and 128 images per class, respectively.Source: §A.2 · pp. 14–15

Transfer learning

  • Transfer learning evaluates frozen representations with logistic classifiers on CIFAR-10, CIFAR-100, and SUN-397.Source: §A.2 · pp. 15–16

Evaluation caveat

  • COCO segmentation annotations can exert greater influence on the backbone than VOC detection, making pre-training gains less obvious and AP differences trivial.Source: §A.2 · p. 16

Detection and segmentation

Distillation views

  • SEED compares identical-view, cross-view, and large-small view strategies, including small 96×96 patches and separate queues for small and large views.The study reports that identical views outperform the contrastive-style alternative in the cited discussion.Source: §A.3 · p. 17, Figure 6 · p. 17, Table 11 · p. 17

A.4 STRATEGIES FOR OTHER DISTILLATION METHODS

The paper compares several alternatives to SEED, including direct representation matching, clustering-based methods, and contrastive distillation objectives.

Alternative methods

  • The l2-distance strategy trains the student by minimizing squared representation distance from the teacher for an identical view.Source: §A.4 · p. 17
  • K-Means generates pseudo-class labels from offline teacher features using 4k or 16k centroids, with uniform sampling over pseudo-labels.The reported results are close when the centroid count changes.Source: §A.4 · pp. 17–18
  • Online clustering replaces frozen K-Means with in-batch, dynamically updated centroids because frozen centroids produce unsatisfying results.The frozen K-Means setup reaches 51.0% on ResNet-18 with ResNet-50 as teacher.Source: §A.4 · p. 18
  • Binary contrastive loss treats teacher-student features for the same instance as positives and random instances as negatives in an Info-NCE-like objective.Source: §A.4 · p. 18

A.5 DISCUSSIONS ON SEED

SEED combines teacher-student alignment with softened contrastive learning, simplifies optimization, and remains compatible with supervised distillation during downstream fine-tuning.

Objective

  • SEED aligns student and teacher encodings while softly contrasting the student against random samples maintained in a queue.Source: §A.5 · p. 18
  • The proposed objective is a simplified approximate upper bound of an l2-plus-cross-entropy formulation with several hyper-parameters.Direct optimization of the fuller formulation makes hyper-parameter search more difficult.Source: §A.5 · p. 18

Relation to Info-NCE

  • SEED can be viewed as a soft Info-NCE variant because negative samples receive probability-based weights rather than uniformly strict negative labels.Source: §A.6 · p. 19

Hyper-parameters

  • Higher teacher temperature can improve results on other datasets, while the optimal temperature remains dataset-specific.The paper explicitly reports this behavior for CIFAR-10.Source: §A.6 · p. 20

Compatibility with supervised distillation

  • Combining SEED pre-training with supervised distillation during fine-tuning raises CIFAR-100 ResNet-18 accuracy to 78.1%.SEED pre-training alone gives 75.4% versus 75.2%, while supervised distillation alone gives 76.0%.Source: §A.7 · p. 20, Table 12 · p. 20

Hyper-parameters

Related papers

Loading 2101.04731v2…