PaperlayerView exact revision on arXiv

Source-linked AI summary

From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels

Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, Yu Li

arXiv:2303.13005v2cs.CV

TL;DR

Knowledge distillation needs effective soft labels, while self-KD needs them without an external teacher. The paper introduces NKD and USKD to normalize non-target supervision and generate customized target and non-target labels, achieving strong results across teacher-based and teacher-free settings. These methods extend efficient self-KD to both CNN and ViT models.

  • Problem

    KD and self-KD require effective soft labels, but self-KD must generate them without a real teacher and existing methods have limited model coverage or added overhead.

  • Method

    The paper decomposes KD loss into target and normalized non-target components for NKD, and uses smoothed student targets plus Zipf-distributed feature-derived non-target labels for USKD.

  • Results

    NKD and USKD achieve state-of-the-art performance, with USKD improving ImageNet Top-1 accuracy by 1.17% for MobileNet and supporting both CNN and ViT models.

  • Takeaways & Limitations

    The unified formulation makes teacher-based and self-knowledge distillation usable with customized soft labels and little additional computation.

Abstract

from arXiv · show

Knowledge Distillation (KD) uses the teacher's prediction logits as soft labels to guide the student, while self-KD does not need a real teacher to require the soft labels. This work unifies the formulations of the two tasks by decomposing and reorganizing the generic KD loss into a Normalized KD (NKD) loss and customized soft labels for both target class (image's category) and non-target classes named Universal Self-Knowledge Distillation (USKD). We decompose the KD loss and find the non-target loss from it forces the student's non-target logits to match the teacher's, but the sum of the two non-target logits is different, preventing them from being identical. NKD normalizes the non-target logits to equalize their sum. It can be generally used for KD and self-KD to better use the soft labels for distillation loss. USKD generates customized soft labels for both target and non-target classes without a teacher. It smooths the target logit of the student as the soft target label and uses the rank of the intermediate feature to generate the soft non-target labels with Zipf's law. For KD with teachers, our NKD achieves state-of-the-art performance on CIFAR-100 and ImageNet datasets, boosting the ImageNet Top-1 accuracy of ResNet18 from 69.90% to 71.96% with a ResNet-34 teacher. For self-KD without teachers, USKD is the first self-KD method that can be effectively applied to both CNN and ViT models with negligible additional time and memory cost, resulting in new state-of-the-art results, such as 1.17% and 0.55% accuracy gains on ImageNet for MobileNet and DeiT-Tiny, respectively. Our codes are available at https://github.com/yzd-v/cls_KD.

1. Introduction

The paper unifies knowledge distillation and self-knowledge distillation through normalized non-target logits and customized soft labels. NKD improves teacher-based distillation, while USKD provides efficient self-KD for CNN and ViT models.

  • 1. Introduction: NKD decomposes KD loss into target and non-target components, then normalizes non-target logits to improve use of teacher soft labels.The non-target component matches internal distributions while normalization addresses differences in their sums.
  • 1. Introduction: USKD creates customized target and non-target soft labels without a real teacher for self-KD.It smooths student target outputs and derives non-target ranks from weakly supervised intermediate features with Zipf’s law.
  • 1. Introduction: USKD applies to both CNN and ViT models with almost negligible additional time and resources.The method requires only an extra linear layer for weak supervision.
  • 1. Introduction: NKD and USKD are evaluated on CIFAR-100 and ImageNet, with additional evidence from COCO detection experiments.The authors report state-of-the-art performance for the proposed methods.

2. Related work

Related work spans teacher-based logit and feature distillation and self-KD methods that generate soft labels with auxiliary structures, contrastive learning, or manual distributions. Existing self-KD methods differ in computational overhead and model coverage, motivating a more general low-cost approach.

  • 2. Related work: Knowledge distillation transfers teacher output information through hard and soft labels while preserving the student network.Prior work improves soft-label use through weighting, feature alignment, decoupled losses, and alternative similarity measures.
  • 2. Related work: Feature-based distillation transfers intermediate semantics, attention, margins, relations, or contrastive information rather than only logits.Examples include FitNet, attention transfer, OFD, RKD, and CRD.
  • 2. Related work: Self-KD removes the external teacher by using auxiliary branches, pairwise alignments, online models, distorted inputs, image mixtures, or feature-derived labels.These approaches use diverse mechanisms to generate supervision from the student or training data.
  • 2. Related work: Low-overhead self-KD methods include label smoothing, Tf-KD, and Zipf’s LS, which manually or algorithmically construct soft labels with limited extra computation.The cited methods distribute non-target values uniformly, use high-temperature manual logits, or derive logits from feature-map outputs.

3. Method

The method decomposes KD into target and non-target losses, normalizes non-target logits for teacher-based KD, and extends the formulation to teacher-free customized soft labels in USKD.

  • 3.1. Normalized KD for Better Using Soft Labels: Normalizing teacher and student non-target logits equalizes their sums, yielding NKD to better align non-target distributions.The original non-target loss can match logits while differing sums prevent identical distributions.
  • 3.1. Normalized KD for Better Using Soft Labels: NKD combines the original classification loss with its normalized distillation loss for teacher-based and self-KD settings.The combined objective is expressed as Lall = Lori + Lnkd.
  • 3.2. Universal Self-Knowledge Distillation: USKD replaces teacher labels with customized target and non-target labels, enabling teacher-free distillation across different model types.Its target label is derived from the student, while non-target labels are generated from intermediate-feature information.
  • 3.2.1. Soft Target Label: The soft target label squares and smooths the student’s target probability, stabilizing labels across training samples and accommodating CNN-like and ViT-like models.Batch-wise smoothing addresses the student prediction changes during training.
  • 3.2.2. Soft Non-target Labels: Weak supervision on intermediate features produces a weak logit whose normalized combination with the final logit determines non-target ranks.The procedure uses CNN stage-2 features or a ViT middle-layer classification token, then balances both logits for ranking.
  • 3.2.2. Soft Non-target Labels: Zipf’s law supplies the non-target distribution, which is sorted by the obtained ranks to form self-KD labels.The resulting labels are used in the NKD non-target loss.
  • 3.2.3. Overall for Self-KD: USKD trains with the original loss, weighted target and non-target losses, and a weak-logit loss.The objective is Lall = Lori + α · Ltarget + β · Lnon + Lweak.

4. Experiments

Experiments evaluate NKD with teacher-student pairs and USKD without teachers across CIFAR-100, ImageNet, CNNs, and ViT-like models. The methods outperform prior approaches while USKD adds negligible training time.

  • Datasets and Details: The experiments use accuracy on CIFAR-100 and ImageNet, with 50k/10k CIFAR-100 and 1.2 million/50k ImageNet train/validation images.Training uses the same student settings with and without distillation, aside from method-specific hyper-parameters.
  • KD with Teachers: NKD outperforms previous distillation methods on ImageNet across homogeneous ResNet34–ResNet18 and heterogeneous ResNet50–MobileNet pairs.The experiments also evaluate multiple architectures on CIFAR-100 and report remarkable accuracy gains.
  • Universal Self-KD without Teachers: USKD surpasses prior self-KD methods across CIFAR-100 and ImageNet, gaining 1.17% and 0.89% ImageNet Top-1 accuracy for MobileNet and ResNet-18.Compared methods include label smoothing, Tf-KD, and Zipf’s LS.
  • Universal Self-KD for More Models: USKD improves MobileNetV2, ShuffleNetV2, and ResNet-101 while training MobileNet-V2 for an epoch takes 10.18 minutes, only 0.01 minutes above direct training.The results report considerable improvements with negligible extra time consumption.
  • Universal Self-KD without Teachers: USKD generalizes to ViT-like models, producing a 0.55% Top-1 accuracy gain for DeiT-Tiny and gains across RegNet, DeiT, and Swin-Transformer.An extra linear layer connected to a ViT middle layer supplies the weak logit.

5. Analysis

Analysis shows that normalized non-target loss improves NKD, while USKD benefits from combining target and non-target information. Additional analyses examine customized labels, smoothing, ranking, and downstream detection.

  • 5.1. Effects of Normalizing the Non-target Logits: NKD’s normalized non-target loss produces greater gains than KD’s non-target loss, while combining target and non-target knowledge improves the student by 2.06% Top-1 accuracy.Using target loss alone increases accuracy by 1.16% in the ResNet34-teacher, ResNet18-student setting.
  • 5.2. Difference between Our NKD and DKD: NKD normalizes student and teacher non-target logits because their differing sums prevent direct matching, and this modification outperforms DKD.The analysis presents normalization as a slight modification to the decomposed KD loss.
  • 5.3. Effects of USKD’s Target and Non-target Loss: Combining target and non-target distillation yields 70.38% MobileNet accuracy, exceeding either component alone.The ablation evaluates target and non-target information on MobileNet and RegNetX-1.6GF.
  • Downstream Detection: Pretraining with USKD is also evaluated for COCO object detection using Mask R-CNN, extending analysis beyond classification.The reported detection evaluation uses AP box and ARbox on COCO val2017.
  • Customized Soft Labels: USKD assigns the target class a smoothed student-derived value and distributes larger non-target values to classes similar to the target.Figure 3 visualizes the target and top-3 non-target customized soft-label values; non-target values follow Zipf’s law.
  • Rank of Soft Non-target Labels: Combining normalized weak and final logits provides the rank used for soft non-target labels, while alternative rank constructions also improve accuracy.The rank analysis compares weak-logit, final-logit, and combined normalized or unnormalized variants.

6. Conclusion

The paper introduces NKD by decomposing KD loss and normalizing non-target logits, then extends the formulation to teacher-free USKD with customized soft labels.

  • NKD decomposes KD loss into target and non-target components and normalizes non-target logits to improve distillation.
  • USKD extends the formulation to CNN-like and ViT-like models using customized target and non-target soft labels without a real teacher.It squares and smooths the student target logit, then uses weak supervision and Zipf’s law for non-target labels.
  • Temperature is an important NKD hyper-parameter, especially on easier datasets, and the paper uses λ = 1 on ImageNet.

B. NKD for ViT-liked Models

NKD also improves DeiT vision transformers and outperforms classical KD for ViT distillation.

  • 84.96% Top-1 accuracy is achieved by DeiT-Base with NKD, 3.20% higher than the baseline.
  • NKD outperforms classical KD for ViT distillation, supporting the effectiveness of its modified KD formulation.

C. Square for Soft Target Label

The square operation in USKD’s target-label construction enlarges differences between samples and improves self-distillation accuracy.

  • 70.18% top-1 accuracy is achieved by MobileNet with the squared target distillation loss, versus 70.04% without squaring.
  • Squaring the student’s target output enlarges differences between samples’ target values within a training batch.

D. Sensitivity Study of USKD’s Parameters

USKD is relatively insensitive to α across a broad range and to β below 0.1, while excessive β can reduce its improvement.

  • α adjusts the target-loss scale, whereas β balances the non-target loss in the sensitivity experiments.
  • 70.68% is the worst accuracy as α varies from 0.6 to 1.4, only 0.13% below the highest accuracy.
  • USKD remains insensitive to β when β < 0.1, but performance improvement may be affected when β reaches 0.14.

F. Extension to Regression Tasks

NKD extends to object detection, surpassing KD and DKD on COCO with a reported 1.3 mAP gain for Faster RCNN.

  • 1.3 mAP gains are reported for Faster RCNN, with NKD surpassing KD and DKD on COCO detection.The comparison uses a Faster RCNN ResNet-101 teacher and Faster RCNN ResNet-50 student.

G. Analysis on the Coefficient.

The coefficient analysis distinguishes NKD from DKD by excluding the non-target term and evaluates these choices separately for KD and self-KD.

  • NKD excludes T2, using only T1, whereas DKD uses T1+T2 in the analyzed distillation settings.T1 is −Ttlog(St), while T2 is −(1 −Tt)log(1 −St).
  • The formula difference is the term −(1 −Tt)log(1 −St), giving NKD a zero coefficient for log(1− St).DKD assigns this logarithmic term coefficient −(1 −Tt).
  • Separate ImageNet experiments analyze the coefficient choices for KD and the soft-label choices for self-KD.The KD analysis is reported in Table 14, while the self-KD analysis is reported in Table 15.
Loading 2303.13005v2…