PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Representation Distillation

Yonglong Tian, Dilip Krishnan, Phillip Isola

arXiv:1910.10699v3cs.LGcs.CVstat.ML

TL;DR

Knowledge distillation often transfers outputs rather than the structural information in neural representations, limiting its suitability for representation transfer. The paper introduces Contrastive Representation Distillation, which uses contrastive learning to align corresponding teacher and student representations, and reports consistent improvements over existing distillation objectives across multiple transfer tasks.

  • Problem

    Standard knowledge distillation minimizes output KL divergence, which does not capture structural dependencies in representations and is undefined for some cross-modal transfers.

  • Method

    CRD uses a contrastive objective that brings teacher and student representations of the same input closer and separates representations from different inputs.

  • Results

    CRD consistently outperforms other distillation objectives across model compression, cross-modal transfer, and ensemble distillation, with a reported 57% average relative improvement over original KD.

  • Takeaways & Limitations

    Contrastive representation distillation provides a practical objective for transferring knowledge across varied network, modality, and ensemble settings.

Abstract

from arXiv · show

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation. Code: http://github.com/HobbitLong/RepDistiller.

1 INTRODUCTION

The paper argues that standard knowledge distillation misses structural dependencies in teacher representations and proposes contrastive learning to transfer more information across several distillation settings.

  • Knowledge distillation traditionally minimizes KL divergence between teacher and student outputs, but this is undefined when transferring non-probabilistic representations across modalities.Cross-modal examples include transferring image representations to sound or depth processing networks.
  • Standard KD treats output dimensions as conditionally independent, making it insufficient for transferring dependencies between representation dimensions.The paper compares this limitation to blurry image-generation results caused by independence assumptions in an L2 objective.
  • The proposed contrastive objective captures correlations and higher-order dependencies by bringing teacher and student representations of the same input together while separating representations of different inputs.The objective is applied to model compression, cross-modal transfer, and ensemble distillation.
  • The objective maximizes a lower bound on mutual information between teacher and student representations, aiming to transfer more information than conditionally independent class probabilities alone.The authors conjecture that correlations among class probabilities can also regularize learning in class-probability distillation.
  • 57% average relative improvement over original KD is reported for CRD across benchmarking of 12 recent distillation methods.The paper reports that CRD outperforms all other methods in the benchmark, while original KD performs second best.

2 RELATED WORK

Related work includes output-logit matching, attention and feature-based transfer, and contrastive representation-learning objectives. The paper positions its objective as a representation-based contrastive criterion related to InfoNCE and NCE.

  • Early knowledge distillation methods transferred knowledge by matching output logits, with Hinton et al. introducing temperature in the softmax output.
  • Attention transfer matches response patterns in teacher and student feature maps but requires feature maps with the same spatial resolution.This requirement constrains teacher and student networks to have very similar architectures.
  • FitNets and related methods guide student learning through regressions or other criteria applied to intermediate representations.The related-work discussion notes that weighted regression can improve performance over unweighted regression.
  • The paper's contrastive objective uses the same criterion as CMC but derives it from a different perspective and proves it is a lower bound on mutual information.The objective is also related to InfoNCE and NCE, which are distinct from adversarial learning.

3 METHOD

The method adapts contrastive learning to distill representations by distinguishing matched teacher–student pairs from mismatched pairs. It maximizes a mutual-information lower bound through a learned critic, enabling joint student and critic optimization across several transfer settings.

  • Transfer tasks: The framework is applied to model compression, cross-modal transfer, and ensemble distillation, with pairwise teacher–student losses summed for ensembles.Cross-modal transfer matches teacher and student features using the contrastive loss, while ensemble distillation defines one contrastive loss per teacher.
  • Contrastive objective: Contrastive learning brings student and teacher representations of the same input closer while separating representations from different randomly chosen inputs.The framework uses congruent pairs from the joint distribution and incongruent pairs from the product of marginals.
  • Contrastive objective: The method models matched pairs as C = 1 and mismatched pairs as C = 0, with one congruent pair for every N incongruent pairs.This yields q(C = 1) = 1/(N + 1) and q(C = 0) = N/(N + 1).
  • Critic learning: Maximizing the critic-based objective increases a lower bound on mutual information between teacher and student representations.The optimal critic estimates q(C = 1|T, S), whose expectation lower-bounds I(T; S).
  • Critic learning: The critic estimates whether a teacher–student embedding pair is matched, and its log-likelihood objective is a binary classification problem.The critic h maps teacher and student embeddings to a value in [0, 1].
  • Optimization: The student and critic can be optimized jointly, so the formulation does not require the critic to be optimized perfectly.The resulting student representation still optimizes a weaker mutual-information lower bound for any critic.
  • Implementation: The practical critic uses transformed, L-2-normalized teacher and student embeddings, with temperature τ controlling concentration, and differs from InfoNCE despite sharing a mutual-information lower-bound motivation.A memory buffer supplies many negative samples without requiring a very large batch size.

4 EXPERIMENTS

CRD is evaluated across model compression, cross-modal transfer, ensemble distillation, and representation transfer, consistently outperforming competing distillation objectives. The experiments also examine correlation matching, scalability, hyperparameters, and computational overhead.

  • Experimental scope: CRD is evaluated on model compression, cross-modal knowledge transfer, and ensemble distillation tasks.The study uses CIFAR-100, ImageNet, STL-10, TinyImageNet, and NYU-Depth V2 settings.
  • Model compression: CRD consistently outperforms other distillation objectives on CIFAR-100, including KD, with a 57% average relative improvement over KD.CRD is reported as the only method that always outperforms KD across the evaluated CIFAR-100 comparisons.
  • Model compression: CRD captures teacher–student logit correlation structure more closely than vanilla training, attention transfer, or KL-divergence distillation.The smaller correlation-matrix differences under CRD are accompanied by reduced error rates.
  • Model compression: CRD narrows the ResNet-34-to-ResNet-18 top-1 accuracy gap on ImageNet by 1.42%, compared with 0.95% for attention transfer.The reported 1.42% reduction corresponds to a 50% relative improvement over attention transfer’s gap reduction.
  • Transfer: CRD transfers representations effectively across datasets and modalities, with CRD+KD improving over the teacher by 3.6% on STL-10 and 4.1% on TinyImageNet.In the depth segmentation task, CRD significantly outperforms the other evaluated distillation methods on average pixel accuracy and mean IoU.
  • Ablations: The contrastive objective outperforms InfoNCE in 4 of 5 teacher–student combinations, while increasing negatives improves performance until gains become negligible beyond N = 4096.The reported error-rate difference between N = 4096 and N = 16384 is less than 0.1%.
  • Hyperparameters: Temperatures between 0.05 and 0.2 generally work well on CIFAR-100, but the optimal temperature may vary across datasets.The experiments use τ = 0.1 generally and τ = 0.07 for ImageNet.
  • Computation overhead: On ImageNet with ResNet-18, CRD adds 260 MFLOPs, about 12% of the original 2 GFLOPs, without a significant observed training-time difference.The measured throughput is 1.75 versus 1.67 epochs/hour on two Titan-V GPUs, and the feature memory bank uses around 600MB.

5 CONCLUSION

The paper presents contrastive objectives as a practical approach to neural-network distillation, applying them across model compression, cross-modal transfer, and ensemble distillation. Across these tasks, the contrastive objective outperforms other distillation objectives and consistently surpasses knowledge distillation.

  • The paper develops a neural-network distillation technique based on contrastive objectives used for representation learning.
  • The technique is evaluated for model compression, cross-modal transfer, and ensemble distillation.
  • The contrastive objective outperforms other distillation objectives by significant margins across these tasks.
  • CRD is the only evaluated distillation objective that consistently outperforms knowledge distillation across a wide variety of knowledge-transfer tasks.

6 APPENDIX

The appendix identifies other state-of-the-art distillation methods from the literature. It lists representative approaches including similarity-preserving, probabilistic, activation-boundary, factor-transfer, flow-based, and neuron-selectivity methods.

  • The appendix compares the paper with other state-of-the-art methods from the literature.

3. Attention Transfer (AT) (Zagoruyko & Komodakis, 2016a)

Similarity-Preserving Knowledge Distillation is identified as one of the related methods considered in the paper's literature comparison.

  • Similarity-Preserving Knowledge Distillation is listed as the fourth related method.
  • The method is abbreviated as SP.
  • SP is attributed to Tung and Mori (2019).

6. Variational information distillation for knowledge transfer (VID) (Ahn et al., 2019)

The appendix lists related distillation methods, implementation settings, and supplementary experiments. These experiments examine correlation discrepancy, combinations with KD, representation transferability, mutual training, and variability across runs.

  • Related methods: The appendix lists related methods including SP, PKT, AB, FT, FSP, and NST.
  • Implementation settings: The contrastive objective uses β = 0.8, with β values from 0.5 to 1.5 generally working reasonably well.
  • Supplementary results: CRD minimizes the correlation discrepancy between student and teacher networks more effectively than the other evaluated objectives at the logit layer.
  • Supplementary results: Combining CRD with KD or PKT further improves performance over the single CRD objective.
  • Supplementary results: CRD achieves the best transferring accuracy in 7 out of 10 evaluated settings.
  • Additional experiments: The appendix also reports experiments with simultaneous teacher-student training and standard deviations across CIFAR-100 runs.
Loading 1910.10699v3…