PaperlayerView exact revision on arXiv

Source-linked AI summary

SEED: Self-supervised Distillation For Visual Representation

Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, Zicheng Liu

arXiv:2101.04731v2cs.CVcs.AI

TL;DR

Small networks perform poorly with existing contrastive self-supervised learning, motivating SEED, which distills a larger self-supervised teacher’s similarity distributions into a smaller student without labels. SEED substantially improves small-model ImageNet performance, including EfficientNet-B0 and MobileNet-v3-Large linear-probe accuracy.

  • Problem

    Existing self-supervised methods work well with large networks but perform poorly on small models, which the authors conjecture reflects limited capacity to learn instance-discriminative representations.

  • Method

    SEED pretrains a large teacher self-supervised, freezes it, and trains a smaller student to mimic its similarity-score distribution over an instance queue without labeled data.

  • Results

    67.6% top-1 accuracy on EfficientNet-B0 and 68.2% on MobileNet-v3-Large are achieved with SEED, versus 42.2% and 36.3% for MoCo-V2.

  • Takeaways & Limitations

    SEED effectively addresses the weakness of self-supervised learning for small models and achieves state-of-the-art results across various small-architecture benchmarks.

  • Takeaways & Limitations

    The paper notes that larger training sets and more fine-tuning iterations can reduce the importance of initialization, limiting how strongly pretrained weights matter in that setting.

Abstract

from arXiv · show

This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.

1 INTRODUCTION

SEED addresses the weak performance of contrastive self-supervised learning on small networks by distilling representations from a larger self-supervised teacher without labeled data. It substantially improves small-model ImageNet linear-probe accuracy.

  • Motivation: Existing self-supervised methods mainly use large networks, while contrastive learning performs poorly on small architectures.The paper reports this limitation empirically for small networks.
  • Motivation: 36.3% top-1 accuracy with MobileNet-V3-Large under MoCo-V2 trails its supervised training accuracy of 75.2%.
  • Motivation: 42.2% top-1 accuracy with EfficientNet-B0 under MoCo-V2 trails its supervised training accuracy of 77.1%.
  • SEED: SEED trains a large model self-supervised as teacher, then distills its knowledge into a smaller student without labeled data.The approach differs from conventional supervised distillation because both training stages are self-supervised.
  • SEED: SEED can use different advanced SSL methods for the teacher and transfer knowledge to target networks with shallower, thinner, or different architectures.
  • Results: 67.6% top-1 accuracy on EfficientNet-B0 and 68.2% on MobileNet-V3 follow SEED, versus 42.2% and 36.3% for MoCo-V2.

2 RELATED WORK

Related work covers contrastive self-supervised representation learning and knowledge distillation, including objectives based on logits, attention maps, feature maps, and probability consistency.

  • Contrastive self-supervised learning: Contrastive self-supervised methods estimate latent distributions by contrasting representations with random or generated negatives.
  • Contrastive self-supervised learning: Prior contrastive approaches improve efficiency, representation consistency, and the alignment-uniformity properties of learned features.
  • Knowledge distillation: Knowledge distillation transfers knowledge from a large model to a smaller one through objectives such as logits, attention maps, feature maps, or probability consistency.

3 METHOD

SEED transfers a frozen teacher’s self-supervised similarity structure to a student by matching teacher and student distributions over a dynamically maintained instance queue. The objective uses augmented views and cross-entropy distillation.

  • 3.1 PRELIMINARY ON KNOWLEDGE DISTILLATION: Knowledge distillation trains a student to mimic a teacher while combining supervised prediction loss with a distillation loss.In the supervised formulation, the distillation signal may come from softmax predictions or intermediate feature maps.
  • 3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION: SEED removes the need for labels by transferring knowledge from a self-supervised large model to a smaller model for downstream representations.
  • 3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION: The teacher is pretrained and frozen, while the student minimizes cross-entropy between teacher and student similarity-score distributions over queued instances.
  • 3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION: For an augmented image view, teacher and student encoders produce normalized feature representations used to compute similarities against the instance queue.
  • 3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION: The queue stores teacher features and is updated first-in-first-out as current-batch features are enqueued and earliest samples are removed.
  • 3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION: SEED adds the teacher embedding itself to the queue, forming D+ so the student can align with the teacher representation while contrasting unrelated features.
  • 3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION: As the teacher temperature approaches zero, its similarity distribution becomes one-hot and the objective becomes similar to Info-NCE.

4 EXPERIMENT

Experiments evaluate SEED across ImageNet classification, semi-supervised learning, transfer tasks, teacher configurations, pre-training algorithms, and distillation strategies. Across these settings, distillation generally improves small-model representations, with gains depending on teacher scale, training choices, and downstream task.

  • Evaluation setup: SEED distillation transfers self-supervised knowledge from a larger teacher to smaller student networks for downstream evaluation.Teachers are self-supervised pretrained, while students include MobileNet-v3-Large, EfficientNet-B0, and smaller ResNets.
  • ImageNet evaluation: 68.2% Top-1 accuracy is achieved by MobileNet-V3-Large with a ResNet-50×2 teacher and multi-crop distillation, versus 36.3% for its MoCo-V2 baseline.A ResNet-152 teacher reaches 61.4%, while the wider ResNet-50×2 teacher with multi-crop training reaches 68.2%.
  • Semi-supervised evaluation: SEED also improves semi-supervised ImageNet accuracy, and stronger teachers with more parameters produce better-performing students.The evaluation uses fixed 1% and 10% annotation subsets with linear fine-tuning.
  • Transfer classification: SEED surpasses contrastive self-supervised pre-training consistently on CIFAR-10, CIFAR-100, and SUN-397 transfer classification benchmarks.These evaluations test whether improvements learned on ImageNet transfer across domains and classes.
  • Detection and segmentation: With ResNet-152 distillation, ResNet-18 Faster R-CNN gains +0.7 AP, +1.4 AP50, and +1.6 AP75 on VOC, while COCO improvements are relatively minor.The experiments evaluate object detection on VOC and detection plus instance segmentation on COCO.
  • Ablations: Teacher depth and width improve accuracy, but further enlargement has limited effects, suggesting the student capacity can become the limiting factor.SEED is also agnostic to teacher pre-training algorithms, and longer teacher or distillation training can provide additional gains.

5 CONCLUSIONS

SEED addresses the weakness of self-supervised learning for small models through distillation from a larger SSL-pretrained model, achieving state-of-the-art results across small architectures.

  • SEED targets the low instance-discrimination capacity that limits self-supervised learning on smaller architectures.
  • SEED learns representations by distilling knowledge from a larger self-supervised model rather than directly training the small model on unlabeled data.
  • SEED achieves state-of-the-art results on various benchmarks for small architectures.

A APPENDIX

The appendix provides additional details and hyperparameter settings for SEED during distillation.

  • The appendix discusses further details of SEED distillation.
  • The appendix covers different hyperparameters used during distillation.
  • These appendix materials supplement the description of SEED during distillation.

A.1 PSEUDO-IMPLEMENTATIONS

The pseudo-implementation defines teacher and student encoders, maintains a queue of representations, computes their score distributions, and updates the queue during distillation.

  • A queue Q maintains previous representations, while the earliest B instances are removed after each update.
  • The pseudo-implementation treats T as the teacher encoder and S as the target student encoder.
  • Teacher and student temperatures are applied to produce soft labels before the loss is computed.
  • The teacher operates in evaluation mode, and the current batch embedding from T is inserted into the distillation process.
  • The implementation computes teacher and student probability score distributions from their embeddings and the queued representations.

A.1.1 DATA AUGMENTATIONS

Teacher pre-training and distillation use a shared augmentation pipeline combining crops, color transformations, blur, grayscale conversion, and horizontal flips.

  • Both teacher pre-training and distillation adopt the same data augmentations.
  • Random resized crops produce 224×224 images from randomly selected scales between 0.2 and 1.0.
  • The pipeline applies color jittering, grayscale conversion, Gaussian blur, and horizontal flipping with specified probabilities.

A.1.2 PRE-TRAINING AND DISTILLATION ON MOBILENET AND EFFICIENTNET

Small architectures struggle with contrastive self-supervised pre-training, motivating architectural changes and distillation-oriented evaluation for EfficientNet and MobileNet.

  • Motivation: Smaller models are hypothesized to struggle because their limited parameters cannot discriminate among large numbers of categories and data.
  • Architectural manipulation: Adding a deeper projection head substantially improves representation quality and linear-evaluation performance for small architectures.The experiments add an additional linear projection head above convolutional backbones.

A.2 ADDITIONAL DETAILS OF EVALUATIONS

The evaluation suite covers semi-supervised classification, transfer learning, detection, segmentation, and multiple distillation-view strategies across ImageNet and other datasets.

  • Classification evaluation: Semi-supervised ImageNet evaluation trains fully connected layers using 1% and 10% class-balanced labeled subsets after distillation.The subsets contain roughly 12 and 128 images per class, respectively.
  • Transfer learning: Transfer learning evaluates frozen representations with logistic classifiers on CIFAR-10, CIFAR-100, and SUN-397.
  • Evaluation caveat: COCO segmentation annotations can exert greater influence on the backbone than VOC detection, making pre-training gains less obvious and AP differences trivial.
  • Detection and segmentation: Detection and segmentation experiments use Faster R-CNN on VOC07 and Mask R-CNN on COCO with all model parameters fine-tuned.VOC and COCO use different detector configurations and training schedules.
  • Distillation views: SEED compares identical-view, cross-view, and large-small view strategies, including small 96×96 patches and separate queues for small and large views.The study reports that identical views outperform the contrastive-style alternative in the cited discussion.

A.4 STRATEGIES FOR OTHER DISTILLATION METHODS

The paper compares several alternatives to SEED, including direct representation matching, clustering-based methods, and contrastive distillation objectives.

  • Alternative methods: The l2-distance strategy trains the student by minimizing squared representation distance from the teacher for an identical view.
  • Alternative methods: K-Means generates pseudo-class labels from offline teacher features using 4k or 16k centroids, with uniform sampling over pseudo-labels.The reported results are close when the centroid count changes.
  • Alternative methods: Online clustering replaces frozen K-Means with in-batch, dynamically updated centroids because frozen centroids produce unsatisfying results.The frozen K-Means setup reaches 51.0% on ResNet-18 with ResNet-50 as teacher.
  • Alternative methods: Binary contrastive loss treats teacher-student features for the same instance as positives and random instances as negatives in an Info-NCE-like objective.

A.5 DISCUSSIONS ON SEED

SEED combines teacher-student alignment with softened contrastive learning, simplifies optimization, and remains compatible with supervised distillation during downstream fine-tuning.

  • Objective: SEED aligns student and teacher encodings while softly contrasting the student against random samples maintained in a queue.
  • Objective: The proposed objective is a simplified approximate upper bound of an l2-plus-cross-entropy formulation with several hyper-parameters.Direct optimization of the fuller formulation makes hyper-parameter search more difficult.
  • Relation to Info-NCE: SEED can be viewed as a soft Info-NCE variant because negative samples receive probability-based weights rather than uniformly strict negative labels.
  • Hyper-parameters: Higher teacher temperature can improve results on other datasets, while the optimal temperature remains dataset-specific.The paper explicitly reports this behavior for CIFAR-10.
  • Compatibility with supervised distillation: Combining SEED pre-training with supervised distillation during fine-tuning raises CIFAR-100 ResNet-18 accuracy to 78.1%.SEED pre-training alone gives 75.4% versus 75.2%, while supervised distillation alone gives 76.0%.
  • Hyper-parameters: A queue size of 65,536 improves accuracy by +1.5 compared with 256, although gains are limited as the queue grows.
Loading 2101.04731v2…