PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation Meets Self-Supervision

Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy

arXiv:2006.07114v2cs.CV

TL;DR

Existing distillation methods often transfer task-specific or architecture-specific teacher knowledge, motivating a more general way to extract richer information. This paper introduces SSKD, which uses self-supervised signals from transformed data with selective transfer, and reports state-of-the-art performance across CIFAR100 and ImageNet, including few-shot, noisy-label, and cross-architecture settings.

  • Problem

    Existing distillation approaches commonly rely on task-specific or architecture-specific teacher representations, limiting the extraction of the teacher’s richer knowledge.

  • Method

    SSKD uses self-supervised pretext tasks on transformed data as an auxiliary distillation signal and applies selective transfer to suppress noisy teacher predictions.

  • Results

    SSKD achieves state-of-the-art performance on CIFAR100 and ImageNet across similar- and cross-architecture settings, while improving generalization in few-shot and noisy-label scenarios.

  • Takeaways & Limitations

    Self-supervised predictions provide richer structured knowledge that can improve distillation and make student networks more similar to teachers across varied architectures and training conditions.

  • Takeaways & Limitations

    The reported CIFAR100 results include conventional KD with competing methods, making them slightly better than the results reported in CRD.

Abstract

from arXiv · show

Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an important technique for model compression and transfer learning. Unlike previous works that exploit architecture-specific cues such as activation and attention for distillation, here we wish to explore a more general and model-agnostic approach for extracting "richer dark knowledge" from the pre-trained teacher model. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. For example, when performing contrastive learning between transformed entities, the noisy predictions of the teacher network reflect its intrinsic composition of semantic and pose information. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student. In this paper, we discuss practical ways to exploit those noisy self-supervision signals with selective transfer for distillation. We further show that self-supervision signals improve conventional distillation with substantial gains under few-shot and noisy-label scenarios. Given the richer knowledge mined from self-supervision, our knowledge distillation approach achieves state-of-the-art performance on standard benchmarks, i.e., CIFAR100 and ImageNet, under both similar-architecture and cross-architecture settings. The advantage is even more pronounced under the cross-architecture setting, where our method outperforms the state of the art CRD by an average of 2.3% in accuracy rate on CIFAR100 across six different teacher-student pairs.

1 Introduction

Knowledge distillation traditionally transfers task-specific teacher outputs or representations, while SSKD uses self-supervised signals from transformed data to extract richer, more structured knowledge. The approach is model-agnostic and improves distillation, including in few-shot and noisy-label settings.

  • Motivation: Traditional distillation transfers soft teacher outputs, whose secondary probabilities encode dark knowledge beyond hard labels.The student is trained by matching teacher and student output distributions using KL divergence.
  • SSKD: SSKD extends mimicking from normal data and classification outputs to transformed data and an additional self-supervision pretext task.The example uses contrastive learning on transformed images, constructing positive and negative pairs through image transformations.
  • Motivation: Existing representation-based distillation methods mainly derive knowledge from a single classification task, making the transferred information highly task-specific.Examples include attention maps, Gram matrices, gradients, pre-activations, and feature-distribution statistics.
  • SSKD: SSKD treats self-supervised learning as an auxiliary task to extract more rounded knowledge from the teacher.The teacher can use a lightweight auxiliary branch trained with its backbone frozen, producing self-supervised signals for distillation.
  • Advantages: SSKD transfers structured self-supervision knowledge while remaining model-agnostic because it transfers only the teacher’s last-layer outputs.This allows the student to search for intermediate representations suited to its own architecture and supports cross-architecture distillation.
  • Advantages: SSKD improves overall distillation performance and helps students generalize better under few-shot and noisy-label scenarios.The paper attributes these benefits to richer structured knowledge from the teacher’s self-supervision predictions.

2 Related Work

Related work has distilled teacher outputs, intermediate representations, and sample similarities, while differing in how self-supervision is defined and used. SSKD distinguishes itself by defining transferred knowledge through conventional self-supervised pretext tasks.

  • Knowledge Distillation: Knowledge distillation methods have transferred classifier outputs, intermediate feature maps, attention, Gram matrices, compressed factors, pre-activations, and sample similarities.These approaches use losses or transformations tailored to particular teacher representations or relationships.
  • Self-Supervised Distillation: KDSVD calls its method self-supervised knowledge distillation but uses teacher-derived correlation maps rather than a conventional self-supervised pretext task.Its self-supervised labels come from the teacher instead of from a separate pretext task.
  • Self-Supervised Distillation: CRD also combines self-supervision with distillation, but differs from SSKD in the purpose of self-supervision and the implementation of contrastive learning.The distinction concerns how the contrastive task is performed and what role it serves in distillation.
  • Self-Supervised Distillation: SSKD is presented as the first study to define distillation knowledge through self-supervised pretext tasks.The framework investigates different pretext tasks and the effects of noisy self-supervised predictions.

3 Methodology

SSKD uses self-supervised contrastive prediction as an auxiliary distillation signal, extending teacher–student matching beyond ordinary classification outputs. The student learns from transformed inputs and selectively transfers structured, potentially noisy teacher predictions alongside standard classification losses.

  • 3.1 Preliminaries: Contrastive prediction maximizes agreement between each data point and its transformed version while separating mismatched pairs in latent space.The method uses transformed inputs, cosine similarity, and positive/negative pair matching to learn transformation-related signals.
  • 3.2 Learning SSKD: SSKD adds a self-supervised module to teacher and student networks, using a backbone, classifier, and task-specific module to produce auxiliary outputs.In this framework, the self-supervised module uses a 2-layer MLP and similarity computation for contrastive prediction.
  • 3.2 Learning SSKD: The teacher is trained in two stages: classification trains its backbone and classifier, then contrastive prediction fine-tunes only the self-supervised module with the backbone fixed.The second stage is efficient because it updates only the small auxiliary head and adapts it to existing backbone features.
  • 3.2 Learning SSKD: During student training, teacher and student outputs are matched on normal classification data, transformed-data classification outputs, and self-supervised prediction distributions.The final objective combines cross-entropy, conventional distillation, self-supervised distillation, and transformed-data distillation terms with balancing weights.
  • 3.3 Imperfect Self-Supervised Predictions: SSKD transfers inaccurate but structured teacher predictions because their relative probabilities can encode useful knowledge beyond hard labels.The teacher’s imperfect contrastive predictions arise partly because its backbone is not fine-tuned for the auxiliary task, yet these predictions remain the core distillation signal.
  • 3.3 Imperfect Self-Supervised Predictions: Selective transfer retains all correct predictions and only the top-k% least erroneous incorrect predictions, reducing the influence of extremely incorrect teacher outputs.Error level is defined by the rank of the corresponding ground-truth positive pair, with lower rank indicating greater error.

4 Experiments

Experiments evaluate SSKD through ablations, robustness tests, and benchmark comparisons on CIFAR100, ImageNet, STL10, and TinyImageNet. Results show benefits from self-supervision signals across few-shot, noisy-label, similar-architecture, and cross-architecture settings.

  • Ablation study: Around 50% contrastive-prediction accuracy still yields highly clustered student features when the student mimics the teacher’s self-supervision output.This supports the claim that inaccurate self-supervision predictions transfer meaningful structured knowledge.
  • Ablation study: Self-supervision components improve accuracy across all four tested teacher-student pairs, while selective transfer of noisy predictions produces a rise-and-fall performance pattern.Incorrect predictions help compared with transferring only correct predictions, but removing extreme outliers can improve performance.
  • Benchmark comparison: SSKD performs best in four of five similar-architecture pairs and all six cross-architecture pairs, with average gains of 0.52% and 2.14%, respectively.The cross-architecture results are consistent with SSKD’s use of final-layer outputs rather than architecture-specific intermediate cues.
  • Benchmark comparison: SSKD achieves the best Top-1 and Top-5 error-rate performance for the evaluated ImageNet ResNet34-to-ResNet18 pair.This experiment demonstrates scalability to a large-scale dataset.
  • Few-shot and noisy-label analysis: SSKD outperforms competing methods across all tested noise ratios, with accuracy dropping only 0.45% as label noise increases from 0% to 50%.The method is also strongest when training samples are reduced, including an approximately 7% absolute accuracy improvement at 25% of reserved samples.

5 Conclusion

SSKD combines self-supervision with knowledge distillation to extract richer teacher knowledge while suppressing noisy signals through selective transfer. Experiments on CIFAR100 and ImageNet show strong performance across architectures and challenging data settings.

  • SSKD combines contrastive prediction as an auxiliary task with knowledge distillation to extract richer knowledge from the teacher network.A selective transfer strategy suppresses noise in the teacher knowledge.
  • SSKD achieves state-of-the-art performance on CIFAR100 and ImageNet across various teacher and student architectures.
  • SSKD makes students more similar to teachers and performs well under few-shot and noisy-label scenarios.

6 Appendix

The appendix describes datasets, augmentation, self-supervision implementations, training settings, and analyses showing that SSKD captures teacher correlation structures across teacher-student pairs.

  • Data and augmentation: CIFAR100 uses padded 32×32 crops, random horizontal flips, and channel normalization; ImageNet uses random resized crops, flips, and normalization.
  • Self-supervision tasks: The ablation compares Contrastive, Exemplar, Jigsaw, and Rotation self-supervision tasks combined with knowledge distillation.
  • Self-supervision tasks: Exemplar treats each dataset instance as a separate class, while Jigsaw predicts one of 24 permutation patterns for four image patches.
  • Training configuration: Competing-method losses combine cross-entropy, conventional distillation, and method-specific losses, while SSKD uses specified temperatures and loss weights.
  • Analysis: SSKD produces the smallest classifier-weight correlation-matrix difference on both teacher-student pairs, indicating the best capture of teacher correlation structure.
Loading 2006.07114v2…