PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation from A Stronger Teacher

Tao Huang, Shan You, Fei Wang, Chen Qian, Chang Xu

arXiv:2205.10536v3cs.CVcs.AIcs.LG

TL;DR

Stronger teachers can create large prediction discrepancies that make exact KL-divergence matching ineffective. DIST instead matches teacher–student prediction relations at inter-class and intra-class levels, achieving strong performance across benchmark tasks, while leaving more advanced relation metrics as future work.

  • Problem

    Existing KD methods are mainly studied with baseline teachers and strategies, while the effects of stronger teachers and their larger prediction discrepancies remain insufficiently investigated.

  • Method

    DIST replaces exact KL matching with Pearson-correlation relational matching for inter-class prediction distributions and intra-class class-wise probabilities.

  • Results

    DIST achieves state-of-the-art performance consistently across image classification, object detection, and semantic segmentation, including 82.3% accuracy on Swin-T and a 1% improvement over KD.

  • Takeaways & Limitations

    DIST provides a simple, efficient, and practical way to distill knowledge from stronger teachers with nearly the same training cost as vanilla KD.

  • Takeaways & Limitations

    The paper leaves more advanced and delicate relation metrics beyond cosine similarity as future work.

Abstract

from arXiv · show

Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD .

1 Introduction

Knowledge distillation helps compact students learn from larger teachers, but stronger teachers can make vanilla KD fail because prediction discrepancies become severe. DIST relaxes exact prediction matching by preserving teacher–student prediction relations and achieves strong results across tasks and settings.

  • Motivation: Knowledge distillation boosts efficient student models by transferring knowledge from larger teachers during training.The motivation is deployment under computational and memory constraints.
  • Stronger teachers: Stronger teachers can arise from larger models or advanced training strategies, yet vanilla KD may underperform standalone student training.Examples include label smoothing and data augmentation.
  • Stronger teachers: Larger teacher–student prediction discrepancies make exact KL-divergence matching difficult and can disturb student training.The discrepancy also grows when stronger training strategies are used.
  • DIST: DIST preserves relative prediction preferences through correlation-based inter-class and intra-class relational matching.It replaces exact probabilistic matching with relations between prediction distributions and class-wise probabilities across instances.
  • Results: 72.07% accuracy is achieved by DIST on ImageNet with ResNet-18, while Swin-T reaches 82.3% and improves KD by 1%.The experiments cover image classification, object detection, and semantic segmentation.

2 Revisiting Prediction Match of KD

Vanilla KD minimizes prediction discrepancies between teacher and student, commonly using KL divergence and optionally ground-truth classification loss. Stronger training strategies enlarge teacher–student discrepancies, making exact matching demanding and potentially misaligned with classification training.

  • Vanilla KD: Vanilla KD transfers knowledge by minimizing the discrepancy between teacher and student prediction scores.The teacher provides soft targets to guide the student.
  • Vanilla KD: KL divergence measures the prediction mismatch, with temperature τ controlling the softness of the logits.The logits are converted into probabilistic prediction vectors before matching.
  • Vanilla KD: KD commonly combines its loss with ground-truth classification loss using balancing factors α and β.The classification term is usually cross-entropy between student predictions and ground-truth labels.
  • Stronger strategies: Training strategy B2 raises standalone accuracy over B1, including 73.4% versus 69.8% for ResNet-18.The stronger strategy is used to examine how teacher–student discrepancy changes.
  • Catastrophic discrepancy: Stronger strategies enlarge teacher–student discrepancies and intensify misalignment between KD and classification losses.ResNet-18 outputs change less than ResNet-50 outputs under the stronger strategy, reflecting the student’s limited representational capacity.
  • Catastrophic discrepancy: Exact KL matching becomes overly demanding when teacher–student discrepancies are large, motivating a relaxed prediction-matching approach.The paper frames exact equality as the minimum-loss condition for KL divergence.

3 DIST: Distillation from A Stronger Teacher

DIST relaxes exact prediction matching by preserving teacher–student relations through correlation, using both inter-class and intra-class relations. Its loss combines classification with these two relational objectives, while positive linear transformations provide scale- and shift-invariant matching.

  • Relaxed match with relations: Positive linear transformations make relation matching invariant to separate prediction scale and shift changes.The transformations use constants satisfying m1 × m2 > 0, enabling Pearson-based matching.
  • Relaxed match with relations: Pearson correlation is used as the relation metric because it supports the required scale- and shift-invariant relaxed match.The coefficient is defined using covariance, means, and standard deviations of the compared vectors.
  • Relaxed match with relations: DIST replaces exact prediction matching with correlation to preserve the teacher’s relative prediction preferences.The approach relaxes KL-style matching while retaining the semantic relations within prediction vectors.
  • Better distillation with intra-relations: Intra-class distillation transfers correlations among instances’ predicted probabilities for each class, complementing row-wise inter-class relations.These scores represent instance-specific similarities to a class and can also reflect informative intra-class semantic variance.
  • Overall training objective: The overall training loss combines classification, inter-class knowledge distillation, and intra-class knowledge distillation terms with balancing factors α, β, and γ.The relational loss gives the student adaptive freedom in matching the teacher’s output.

4 Experiments

Experiments evaluate DIST across architectures, model sizes, training strategies, and vision tasks. DIST consistently improves over vanilla KD and often surpasses specialized distillation methods.

  • Image Classification: DIST achieves the strongest ImageNet baseline performance while operating only on model outputs with computational cost similar to vanilla KD.Compared methods may require memory banks or additional convolutions for feature alignment.
  • Image Classification: As teacher model size increases, DIST improves while ResNet-18 students trained with vanilla KD perform worse.The gains over KD become more significant for larger teachers.
  • Image Classification: With strong ImageNet training strategies, DIST performs best across similar and dissimilar student architectures.These experiments include ResNet-18, ResNet-34, MobileNetV2, and EfficientNet-B0 students.
  • Image Classification: A Swin-L teacher improves ResNet-50 and Swin-T students by 1.7% and 1.0%, respectively.The experiments include stronger models and strategies based on Swin-Transformer architectures.
  • Object Detection: On COCO, DIST significantly outperforms vanilla KD, and combining DIST with feature mimicry surpasses state-of-the-art detection distillation methods.DIST is applied as additional supervision on final class predictions.
  • Semantic Segmentation: On Cityscapes, DIST outperforms CIRKD by 1.58% on PSPNet-R18 using supervision from class predictions.The result supports relation modeling for semantic segmentation.
  • Ablation Studies: Inter-class and intra-class relations each outperform vanilla KD, and combining them further improves performance.The ablation uses ImageNet with ResNet-18 students and ResNet-34 teachers.
  • Ablation Studies: With intra-class relation only, Pearson correlation reaches 71.55% versus 70.61% for KL divergence.Adding intra-class relation to vanilla KD improves accuracy from 71.21% to 71.62%.

5 Conclusion

The paper introduces DIST for distillation from stronger teachers and reports broad effectiveness across benchmark tasks. Its relation-based loss relaxes exact KL matching and can outperform task-specific state-of-the-art methods.

  • Conclusion: DIST replaces exact KL matching with a relation-based loss to address catastrophic prediction discrepancy between students and stronger teachers.The method is presented as simple and effective for stronger teachers.
  • Conclusion: Extensive experiments report superiority across benchmark tasks, including object detection and semantic segmentation.DIST can outperform state-of-the-art methods designed specifically for those tasks.

Checklist

The checklist records affirmative documentation of contributions, limitations, ethics review, reproducibility materials, and asset attribution, while human-subject items are marked not applicable.

  • Disclosure: The paper states that it describes its contributions, scope, and limitations, with limitations referenced in the Appendix.It also reports discussing potential negative societal impacts.
  • Reproducibility: The paper reports that code, data, training details, logs, and instructions needed to reproduce experiments are provided or released.It also reports standard deviations on CIFAR-100 and documentation of compute resources.
  • Assets: Existing assets are reported as cited, licensed, and accompanied by relevant attribution.The checklist marks these asset-documentation items affirmatively.
  • Ethics: Consent, personally identifiable information, offensive content, and human-subject procedures are marked not applicable where relevant.Crowdsourcing and participant-risk documentation are likewise marked not applicable.

A.1 Implementation of DIST

DIST uses only student and teacher output logits, making its implementation simple and easy to integrate. Figure 4 presents the PyTorch implementation.

  • Implementation: DIST takes only the output logits of the student and teacher models as inputs.The method therefore uses simple inputs and computations and is easy to implement.
  • Implementation: Figure 4 shows the PyTorch implementation of DIST.The figure documents the code-level realization of the method.

A.2 Related work

Prior KD work often transfers instance-level relations or matches intermediate features, whereas DIST focuses on prediction relations and jointly models inter-class and intra-class structure.

  • A.2 Related work: Earlier KD methods transfer relations between teacher and student instances, aiming to preserve semantic similarity structures in the learned representations.These methods encourage same-class instances to remain close and different-class instances to remain farther apart.
  • A.2 Related work: DIST instead transfers relations within probabilistic predictions rather than intermediate features, relaxing KL matching for stronger-teacher settings.It uses prediction-based inter-class and intra-class relations simultaneously.
  • A.2 Related work: DIST proposes inter-class relations across classes within each instance and intra-class relations across instances for each class.This distinguishes its relational target from prior methods that measure relations between teacher and student instances individually.

A.3 More ablation studies

Ablations examine correlation choices and their relation to training strategies, showing why Pearson correlation is preferred for DIST, particularly with label smoothing.

  • Correlation analysis: DIST achieves higher inter-class and intra-class teacher–student correlations than plain classification loss and KD across Pearson, Spearman, and Kendall’s Tau measures.The comparison uses ResNet-18 students and ResNet-34 teachers trained on ImageNet with strategy B1.
  • Correlation metric: Pearson correlation is selected as DIST’s relation metric, while cosine similarity is evaluated as an alternative with the same matching-function form.Both metrics evaluate relations between teacher and student predictions.
  • Correlation metric: Pearson correlation achieves higher accuracies than KD and cosine-based DIST, especially when teacher and student use label smoothing.Its centering provides shift-invariance, which can be more robust to distribution changes caused by label smoothing.

A.4 Comparisons of training speed

DIST is evaluated for computational speed and matching-function landscapes, while the discussion identifies a task boundary and an environmental trade-off.

  • A.4 Comparisons of training speed: DIST has almost the same highest training speed as vanilla KD and outperforms feature-based KD methods in average batches per second.The speed comparison trains a ResNet-18 student with a ResNet-34 teacher on ImageNet using strategy B1 across eight NVIDIA V100 GPUs.
  • A.5 Landscapes of matching functions: The landscape analysis compares MSE, KL divergence, cosine distance, and Pearson distance by varying one student logit while fixing the remaining logits.Loss values are zero when student and teacher logits are identical, and the landscapes characterize sensitivity to distribution shifts.
  • A.5 Landscapes of matching functions: PCC in DIST has sharper gradient curves in non-optimal regions and the largest area around optima in the gradient landscapes.These observations come from the visualized gradient landscapes for the compared matching functions.
  • A.6 Discussion: DIST is less effective for binary classification because only two classes provide limited inter-class relation information.The stated limitation applies despite consistent improvements reported for multi-class classification tasks.
  • A.6 Discussion: Evaluating DIST consumes considerable computing resources and may increase carbon emissions, although deploying compact models could save production energy.The paper identifies both the environmental concern and the potential energy-saving benefit.
Loading 2205.10536v3…