PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation Meets Self-Supervision

Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy

arXiv:2006.07114v2cs.CV

TL;DR

Existing distillation cues are often task-specific, motivating a more general way to extract richer teacher knowledge. SSKD uses self-supervision as an auxiliary task with selective transfer, and reports state-of-the-art performance on CIFAR100 and ImageNet across varied architectures.

  • Problem

    Existing teacher representations used for distillation are task-specific and may reflect only one facet of the knowledge in the teacher network.

    Source: §1 · p. 2

  • Method

    SSKD uses contrastive self-supervision as an auxiliary task to extract richer teacher knowledge, with selective transfer to suppress noisy predictions.

    Source: §1 · p. 3, §5 · p. 15

  • Results

    SSKD reports state-of-the-art performance on CIFAR100 and ImageNet across various architectures.

    Source: §5 · p. 15

  • Takeaways & Limitations

    The authors report that SSKD works well in few-shot and noisy-label scenarios.

    Source: §5 · p. 15

    On CIFAR100, adding conventional KD to competing methods makes the results slightly better than those reported in CRD.

    Source: §4.2 · p. 11

Abstract

from arXiv · show

Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an important technique for model compression and transfer learning. Unlike previous works that exploit architecture-specific cues such as activation and attention for distillation, here we wish to explore a more general and model-agnostic approach for extracting "richer dark knowledge" from the pre-trained teacher model. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. For example, when performing contrastive learning between transformed entities, the noisy predictions of the teacher network reflect its intrinsic composition of semantic and pose information. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student. In this paper, we discuss practical ways to exploit those noisy self-supervision signals with selective transfer for distillation. We further show that self-supervision signals improve conventional distillation with substantial gains under few-shot and noisy-label scenarios. Given the richer knowledge mined from self-supervision, our knowledge distillation approach achieves state-of-the-art performance on standard benchmarks, i.e., CIFAR100 and ImageNet, under both similar-architecture and cross-architecture settings. The advantage is even more pronounced under the cross-architecture setting, where our method outperforms the state of the art CRD by an average of 2.3% in accuracy rate on CIFAR100 across six different teacher-student pairs.

1 Introduction

The paper proposes SSKD, which uses self-supervised learning as an auxiliary task to extract richer knowledge from a teacher for student distillation. It extends distillation to transformed data and self-supervision signals, aiming to improve generalization, including in few-shot and noisy-label settings.

1 Introduction

  • Conventional distillation uses softened teacher outputs and KL divergence, while the relative probabilities assigned to incorrect classes can provide informative dark knowledge.The hidden knowledge in these secondary probabilities is termed “dark knowledge.”Source: §1 · pp. 1–2
  • Existing intermediate-representation methods draw on a single task and may capture only one facet of a teacher, motivating complementary auxiliary tasks.Examples include attention maps, gram matrices [44], gradients, pre-activations, and feature distribution statistics.Source: §1 · p. 2
  • The method adds a lightweight auxiliary branch for a self-supervised pretext task, updates it while freezing the teacher backbone, and extracts its signals for distillation.Examples of pretext tasks include exemplar-based learning [9], rotation prediction, jigsaw, and contrastive learning [4] [27].Source: §1 · pp. 2–3
  • SSKD’s self-supervision predictions provide structured knowledge that improves overall distillation and helps students generalize in few-shot and noisy-label scenarios.It extends mimicking from normal data and a single classification task to transformed data and an additional self-supervision pretext task.Source: Figure 1 · p. 2, §1 · p. 3
  • Extensive experiments on CIFAR100 and ImageNet demonstrate SSKD’s effectiveness over other state-of-the-art methods.The paper also investigates different self-supervised pretext tasks and the impact of noisy self-supervised predictions.Source: §1 · p. 3
  • SSKD is model-agnostic because it transfers only last-layer outputs, giving students flexibility to learn representations suited to their own architectures.This contrasts with transferring architecture-specific features, which can be difficult across architectures such as ResNet50 and ShuffleNet.Source: §1 · p. 3

2 Related Work

Prior work distills logits and intermediate representations, while this paper defines distillation knowledge through conventional self-supervised pretext tasks. It adopts contrastive learning as its main pretext task while allowing alternatives.

Knowledge Distillation

  • Knowledge distillation transfers teacher supervision through logits or compact feature representations, with methods matching intermediate features, attention, statistics, factors, or pre-activation maps.Examples include FitNets [37], AT, FSP, FT, AB, and IRG.Source: §2 · pp. 3–4
  • Unlike KDSVD and CRD, SSKD defines knowledge via conventional self-supervised pretext tasks, performed separately in teacher and student and matched through KL-divergence.KDSVD uses teacher-derived feature-singular-vector correlation maps rather than a conventional pretext task; CRD performs contrastive learning across teacher and student to maximize mutual information.Source: §2 · p. 4

Self-Supervised Learning

  • Self-supervised learning derives labels from data through pretext tasks, including construction-, prediction-, cluster-, generation-, and contrastive-based methods.Examples include inpainting, colorization, prediction-based methods [9], cluster-based methods, generation-based methods, and contrastive methods [4] [14] [27] [32] [41].Source: §2 · p. 4
  • SSKD adopts SimCLR [4] as its main pretext task, while also allowing alternatives such as Exemplar [9], Rotation, and Jigsaw.Jigsaw++ [31] also combines self-supervision and distillation, but uses knowledge transfer to improve self-supervision, the inverse direction of SSKD.Source: §2 · p. 4

3 Methodology

SSKD uses self-supervised contrastive predictions as auxiliary teacher knowledge alongside conventional distillation. It trains a teacher self-supervision module, transfers its similarity signals to the student, and selectively limits transfer of highly erroneous predictions.

3.1 Preliminaries

  • Knowledge distillation uses softened teacher outputs to convey inter-class similarities beyond hard labels, and trains the student with cross-entropy and KL-divergence distillation losses.Temperature scaling softens the teacher’s peaky softmax distribution.Source: §3.1 · p. 5
  • Contrastive prediction matches each transformed input with its original as a positive pair and other originals as negatives, using cosine similarity to mine teacher knowledge rather than chiefly learn invariance.A two-layer MLP projection head maps representations to the latent space where the contrastive loss is applied.Source: §3.1 · pp. 5–6

3.2 Learning SSKD

  • Teacher training first optimizes the backbone and classifier on normal data, then freezes them and trains only the self-supervision module for contrastive prediction.The study uses color dropping, rotation, cropping followed by resize, and color distortion; it omits transformed inputs from classification because forcing their classification may damage learned semantics.Source: §3.2 · pp. 6–7
  • The student matches teacher and student self-supervision similarity distributions, encourages matching classifier outputs on transformed inputs, and combines these losses with cross-entropy and knowledge distillation.The self-supervision distributions are formed by applying temperature-scaled row-wise softmax to each network’s similarity matrix.Source: §3.2 · pp. 7–8

3.3 Imperfect Self-Supervised Predictions

  • Incorrect teacher predictions can encode structured knowledge, but SSKD transfers only correct predictions and the top-k% ranked incorrect predictions to suppress harmful outliers.Error is ranked by the ground-truth positive pair’s position among the teacher’s similarity scores; lower rank indicates greater error.Source: §3.3 · p. 8

4 Experiments

Experiments assess SSKD through component ablations, benchmark comparisons, and few-shot and noisy-label tests. The reported results cover standard benchmarks, learned representations, and robustness to limited or perturbed data.

4.1 Ablation Study

  • Despite teacher contrastive predictions being only around 50% accurate, mimicking them produces clustered student features; adding LT and Lss also boosts accuracy across all four tested pairs.The ablation tests contrastive self-supervision output alone and compares conventional KD, KD+LT, and full SSKD.Source: §4.1 · p. 9, Figure 3 · p. 9
  • Transferring incorrect teacher predictions helps compared with transferring only correct predictions, while removing extreme outliers yields peak performance at k = 50–75 across architectures.The paper fixes k = 75 for teacher-student comparisons in Sections 4.2 and 4.3.Source: §4.1 · p. 10, Table 1 · p. 10
  • Student accuracy under SSKD is positively correlated with the linear-evaluation performance of the incorporated self-supervision method.The comparison covers Exemplar, Rotation, Jigsaw, and Contrastive methods.Source: §4.1 · p. 10, Table 2 · p. 10

4.2 Benchmark

  • SSKD performs best on four of five similar-architecture CIFAR100 pairs, with a 0.52% average gap over the best competitors, and wins all six cross-architecture pairs by 2.14% average absolute accuracy.The cross-architecture results are presented as evidence that SSKD does not rely on architecture-specific cues.Source: §4.2 · p. 11, Table 3 · p. 11, Table 4 · p. 12
  • SSKD achieves the best ImageNet Top-1 and Top-5 error rates in the tested ResNet34-to-ResNet18 pair, and the highest linear-classification accuracies on both STL10 and TinyImageNet.The representation evaluation uses a wrn40-2 teacher and ShuffleNetV1 student trained on CIFAR100.Source: §4.2 · p. 12, Table 5 · p. 12, Table 6 · p. 13
  • SSKD makes the student more similar to its teacher: it has the best KL-divergence and CKA results on CIFAR100, and wins all but STL10 KL-divergence among the reported out-of-distribution comparisons.The similarity analysis compares VGG13 and VGG8 on CIFAR100, STL10, and SVHN.Source: §4.2 · p. 13, Table 7 · p. 13

4.3 Further Analysis

  • At 25% of CIFAR100 training data, SSKD improves accuracy by about 7% over all competing methods; with 50% noisy labels, its accuracy drops only 0.45%.SSKD performs best at every tested data proportion and noise ratio; FT and CRD drop dramatically as label noise increases.Source: §4.3 · pp. 14–15, Figure 4 · p. 14

5 Conclusion

SSKD combines contrastive self-supervision with knowledge distillation to extract richer teacher knowledge while selectively suppressing noise. Experiments on CIFAR100 and ImageNet report state-of-the-art performance, with further analysis showing benefits in few-shot and noisy-label scenarios.

5 Conclusion

  • SSKD achieves state-of-the-art performance in experiments on CIFAR100 and ImageNet across various architectures.The experiments examine the method on both datasets using different architectures.Source: §5 · p. 15
  • SSKD combines contrastive prediction as an auxiliary self-supervision task with knowledge distillation to extract richer knowledge from the teacher.The paper presents this as the first framework combining self-supervision with knowledge distillation.Source: §5 · p. 15
  • A selective transfer strategy suppresses noise in the teacher's knowledge.Source: §5 · p. 15
  • Further analysis shows that SSKD makes the student more similar to the teacher and works well in few-shot and noisy-label scenarios.Source: §5 · p. 15

6 Appendix

The appendix documents the network backbones, image processing, self-supervision implementations, and training settings used in the experiments. Its correlation visualization reports that SSKD most closely matches teacher classifier-weight correlations among the compared methods.

6.1 Network Architectures

  • Experiments use CIFAR-style ResNet, ResNet, WideResNet, MobileNet, VGG, and ShuffleNet backbones, with architecture variants specified for the datasets.The appendix defines CIFAR-style ResNet block channels and wider variants, ImageNet-style ResNet bottlenecks, WideResNet depth and width, and a 0.5 MobileNet width multiplier; VGG and ShuffleNet are adapted to CIFAR100.Source: §6.1 · p. 19

6.2 Data Processing

  • CIFAR100 images use padded random crops, horizontal flips, and dataset-channel normalization; self-supervision uses color dropping, rotation, cropping with resizing, and color jitter.ImageNet images are randomly cropped and resized to 224×224, flipped, and channel-normalized; the specified crop scales range from 0.08 to 1.0 and aspect ratios from 3/4 to 4/3.Source: §6.2 · p. 19

6.3 Implementations of different self-supervision tasks

  • The appendix describes Exemplar, Jigsaw, and Rotation self-supervision as auxiliary classifiers whose teacher logits are transferred to the student.Exemplar treats dataset instances as classes; Jigsaw uses four patches and 24 permutation classes; Rotation uses a four-way angle classifier. Contrastive is also evaluated, with implementation details given in the main paper.Source: §6.3 · pp. 19–20

6.4 Training Hyperparameters

  • CIFAR100 competing methods are rerun using CRD’s implementation, while ImageNet comparison results are copied from CRD.SSKD is trained for 240 epochs on CIFAR100 and 100 epochs on ImageNet; ImageNet uses eight GPUs with total batch size 256, versus batch size 64 on CIFAR100.Source: §6.4 · p. 20

6.5 Visualization of Correlation Difference

  • SSKD has the smallest classifier-weight correlation difference on both teacher–student pairs, indicating the closest match to the teachers’ correlation structures.The comparison includes a vanilla student, AB, CRD, and SSKD.Source: Figure 5 · p. 21, §6.5 · p. 21

Related papers

Loading 2006.07114v2…