PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Knowledge Distillation

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang

arXiv:2203.08679v2cs.CVcs.AI

TL;DR

Feature-based distillation is effective but costly, while the potential of logit distillation remains underexplored. The paper decomposes KD into TCKD and NCKD, identifies coupling problems, and introduces DKD, which achieves strong results across classification and detection tasks with better training efficiency. DKD also has a scope limitation on object detection because logits-based methods cannot transfer localization knowledge.

  • Problem

    Feature-based distillation is widely used but adds training computation and storage, while the significance and potential of lower-cost logit distillation remain limited and underexplored.

  • Method

    The paper reformulates classical KD into target-class TCKD and non-target-class NCKD, then decouples their weighting and removes NCKD’s negative dependence on teacher confidence in DKD.

  • Results

    DKD achieves significant improvements across CIFAR-100, ImageNet, and MS-COCO image-classification and object-detection tasks, with better training efficiency and feature transferability than feature-based methods.

  • Takeaways & Limitations

    NCKD is a prominent source of logit-distillation effectiveness, while DKD enables TCKD and NCKD to contribute more efficiently and flexibly.

  • Takeaways & Limitations

    DKD does not outperform state-of-the-art feature-based methods on object detection because logits-based methods cannot transfer localization knowledge.

Abstract

from arXiv · show

State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the "difficulty" of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megvii-research/mdistiller.

1. Introduction

Knowledge distillation reduces the cost of deploying lightweight models, but logit distillation has received less attention than feature-based methods despite its lower training overhead. The paper decomposes classical KD into TCKD and NCKD, analyzes their distinct roles, and proposes DKD to decouple their limitations.

  • Classical KD couples NCKD to teacher confidence and couples TCKD with NCKD, suppressing NCKD and limiting independent weighting of the two parts.
  • Knowledge distillation transfers knowledge from a heavy teacher to a light student, improving the student without extra inference costs.
  • Feature-based distillation often performs better than logit-based methods but introduces extra computational and storage costs during training.
  • Classical KD can be reformulated into TCKD, a binary target-versus-nontarget prediction, and NCKD, a multicategory prediction among non-target classes.
  • TCKD transfers information about training-sample difficulty, whereas NCKD captures knowledge among non-target logits and can match or exceed classical KD alone.
  • DKD replaces the confidence-dependent NCKD coefficient with a constant and introduces separate weights for TCKD and NCKD.
  • DKD achieves state-of-the-art performance across various tasks while improving training efficiency and feature transferability relative to feature-based distillation methods.

2. Related work

Related work established knowledge distillation as teacher-to-student transfer through soft predictions, while later research emphasized feature-based methods. The paper focuses on understanding and revitalizing logit distillation under its lower training overhead.

  • Knowledge distillation trains a smaller student network under guidance from a larger teacher network using soft labels.
  • Logit-distillation research has explored mutual learning, teacher assistants, and regularization or optimization strategies.
  • Feature-based methods transfer representations or inter-sample correlations from teachers to students, generally achieving higher performance than logit-based methods.
  • Feature-based distillation also requires considerably higher computational and storage costs than logit-based distillation.
  • This paper analyzes what limits logit-based methods and seeks to revitalize logit distillation.

3. Rethinking Knowledge Distillation

The paper reformulates classical knowledge distillation into target-class and non-target-class components, then studies their distinct effects and limitations. It finds that NCKD is especially important but can be suppressed by the coupled formulation, motivating a more flexible decoupling.

  • 3.1. Reformulating KD: Classical KD is reformulated as TCKD plus a teacher-confidence-weighted NCKD term.TCKD matches binary target-versus-non-target probabilities, while NCKD matches probabilities among non-target classes.
  • 3.2. Effects of TCKD and NCKD: Singly applying TCKD can be unhelpful or harmful, whereas NCKD produces comparable or better distillation performance.Reported TCKD changes include a 2.30% drop on WRN-16-2 and a 3.87% drop on ResNet8×4; NCKD reaches 1.76% versus 1.13% for classical KD on ResNet8×4.
  • 3.2. Effects of TCKD and NCKD: NCKD captures non-target-class knowledge and can match or outperform classical KD when applied alone.The results identify non-target logits as a prominent source of the dark knowledge transferred by logit distillation.
  • 3.2. Effects of TCKD and NCKD: TCKD transfers knowledge about training-sample difficulty, with larger gains on more challenging data such as strongly augmented samples.Experiments vary augmentation strength, label noise, and task difficulty to test this interpretation.
  • 3.2. Effects of TCKD and NCKD: Classical KD couples NCKD to teacher confidence and couples the weights of TCKD and NCKD, suppressing well-predicted samples and limiting independent balancing.Well-predicted samples can provide more performance gain, yet their NCKD loss weight is reduced by high teacher confidence.
  • 3.3. Decoupled Knowledge Distillation: DKD decouples the two components and assigns separate hyper-parameters α and β to TCKD and NCKD.This design addresses both the confidence-dependent suppression of NCKD and the inability to adjust the two contributions independently.

4. Experiments

Experiments evaluate DKD on image classification, object detection, training efficiency, teacher selection, feature transferability, and visual representations. Across these settings, DKD improves classification results, complements feature distillation for detection, and offers a favorable performance–cost trade-off.

  • Ablation: α and β: 74.79% versus 73.63% follows decoupling NCKD from teacher confidence, while 76.32% follows independently weighting NCKD and TCKD.The ablation uses ResNet32×4 as teacher and ResNet8×4 as student.
  • CIFAR-100 image classification: DKD consistently improves CIFAR-100 accuracy over baseline and classical KD across same- and different-architecture teacher–student pairs.Reported gains are 1–2% for same-series pairs and 2–3% for different-series pairs.
  • ImageNet image classification: DKD achieves significant ImageNet improvements and exceeds the reported state-of-the-art feature-distillation results for the evaluated teacher–student pairs.Experiments report top-1 and top-5 validation accuracy with ResNet-34/ResNet-18 and ResNet-50/MobileNet-V1 pairs.
  • MS-COCO object detection: Combining DKD with ReviewKD produces new state-of-the-art MS-COCO detection results, although DKD alone only modestly surpasses classical KD.DKD further boosts AP metrics when added to the feature-based ReviewKD method.
  • Training efficiency: DKD provides the best performance–training-cost trade-off because it retains nearly classical-KD computational complexity and adds no parameters.Feature-based methods require extra training time and storage, whereas DKD is reformulated from classical KD.
  • Extensions: DKD alleviates the problem that larger teachers are not always better, transfers features with strong downstream performance, and produces more separable representations than KD.The paper evaluates teacher scaling, transfer to STL-10 and Tiny-ImageNet, and t-SNE and logit-correlation visualizations.

5. Discussion and Conclusion

The paper reframes logit distillation through TCKD and NCKD, identifies limitations in classical KD’s coupling, and proposes DKD to improve knowledge transfer across tasks. It also acknowledges that DKD does not surpass feature-based methods for object detection because logits cannot transfer localization knowledge.

  • The paper interprets logit distillation by reformulating classical KD into target-class knowledge distillation (TCKD) and non-target-class knowledge distillation (NCKD).
  • Classical KD’s coupled formulation suppresses NCKD effectiveness and restricts the flexibility to balance TCKD and NCKD.
  • DKD decouples the two components, achieving significant improvements on CIFAR-100, ImageNet, and MS-COCO for classification and detection.
  • DKD also demonstrates improved training efficiency and feature transferability.
  • DKD does not outperform feature-based methods on object detection because logits cannot transfer localization knowledge.

A.1. Details about the reformulation in Sec 3.1

The supplementary derivation decomposes the KL divergence between teacher and student predictions into a binary target-versus-non-target term and a normalized non-target-class term.

  • The teacher and student distributions are split into the target class and all non-target classes.
  • The non-target probabilities are normalized using ˆp_i = p_i/p_t.
  • The resulting expression separates a binary KL term from a non-target-class KL term weighted by the teacher’s non-target probability mass.
  • The derivation starts from the KL divergence between teacher and student prediction distributions.
  • Equation (12) presents the reformulated KD loss as KL(b^T||b^S) + (1-p_t^T)KL(ˆp^T||ˆp^S).

A.2. Implementation: Experiments in Sec 4

The experiments use established training settings across CIFAR-100, ImageNet, and MS-COCO, with DKD applied under task- and architecture-specific configurations.

  • CIFAR-100: CIFAR-100 models are trained for 240 epochs with SGD, temperature 4, α = 1.0, and scheduled learning-rate reductions.
  • ImageNet: ImageNet models are trained for 100 epochs with batch size 512, temperature 1, and α = 0.5.
  • MS-COCO: MS-COCO experiments use Faster R-CNN with FPN, train students with a 1x scheduler, and apply DKD on the R-CNN head.
  • MS-COCO: For MS-COCO, α = 1.0, β = 0.25, and temperature = 1 are used across experiments.
  • Compared-method results are taken from original papers or reproduced by previous works.

A.3. Guidance for tuning β

The supplementary analysis explores whether β should increase with teacher confidence, using CIFAR-100 accuracy and teacher-logit gaps to guide tuning.

  • The authors hypothesize that more confident teachers make NCKD more valuable and therefore require larger β.
  • An excessively large β can increase non-target-logit gradients and harm student prediction correctness.
  • The reported tables provide β values for different teachers and accuracy results under those settings.
  • CIFAR-100 experiments vary β across teachers and report top-1 accuracy (%) together with the averaged gap z_t − z_max.
  • The best β may be positively proportional to the teacher-logit gap, offering tuning guidance and a direction for further research.

A.4. Implementation: Experiments in Sec 3.2

The experiments use equal weighting for KD-related losses and test TCKD under stronger augmentation and noisy labels to examine sample-difficulty transfer.

  • Scope: The implementation details cover experiments reported in Sec 3.2 of the manuscript.These details include loss weighting, stronger augmentation, and noisy-label settings.
  • Basic settings: KD and CE loss terms are each weighted 1.0, replacing the default 0.1CE + 0.9KD setting.The equal weighting also applies when using KD, TCKD, or NCKD individually.
  • Strong augmentation: AutoAugment is added after default CIFAR augmentation, with teacher and student trained using the same policy.The policy combines random crop and horizontal flip with CIFAR AutoAugment.
  • Noisy labels: Symmetric label noise is used to test whether TCKD conveys training-sample difficulty.The teacher is trained on noisy data, and the student is distilled on the same data using the teacher’s best epoch.

A.5. Explanation about why TCKD brings performance drop in Table 1

TCKD alone can reduce accuracy because high-temperature gradients raise non-target logits without supplying class-similarity information; the drop increases with temperature.

  • Explanation: TCKD alone can cause performance drops because high-temperature gradients increase non-target logits and may harm student prediction correctness.Without NCKD, class-similarity or dark-knowledge information is unavailable.
  • Table A.3: Table A.3 reports accuracy (%) across temperatures when only TCKD is applied to a WRN-16-2 student distilled from WRN-40-2.The table varies temperature while keeping the teacher and student architectures fixed.
  • Temperature effect: 73.26% is the vanilla-training baseline approached by TCKD when T=1, while larger temperatures produce larger performance drops.The reported relationship between temperature and performance drop is positive.

A.6. How to employ DKD on detectors

DKD is applied to the R-CNN head of Faster R-CNN using proposal labels as targets and teacher logits from corresponding proposals. Training efficiency includes per-batch computation and extra memory or parameters, while feature transferability is tested by linear probing.

  • Detector integration: DKD is applied only to the R-CNN head of Faster R-CNN, using proposal-assigned labels as target classes.Proposals with IoU below 0.5 are assigned the background target class.
  • Detector integration: Teacher logits are computed for the same proposals, preserving proposal locations while obtaining features from the teacher backbone.The student network supplies the detector being distilled, and the teacher provides corresponding R-CNN prediction logits.
  • Training efficiency: Per-batch training time sums data processing, forward and backward computation, and memory updating; extra parameters and dictionary memory are also reported.The accounting includes learnable modules and non-parameter memory such as contrast memory.
  • Feature transferability: Feature transferability is evaluated by freezing a distilled WRN-16-2 feature extractor and training linear FC classifiers on STL-10 and Tiny-ImageNet.The extractor was distilled from a WRN-40-2 teacher and uses the final global-average-pooling feature.
Loading 2203.08679v2…