PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias

Jianhua Zhang, Yi Gao, Ruyu Liu, Xu Cheng, Houxiang Zhang, Shengyong Chen

arXiv:2412.09874v1cs.CV

TL;DR

Teacher bias can mislead students in knowledge distillation, which commonly transfers teacher knowledge without separating correct from incorrect predictions. The paper filters and rectifies bias and dynamically shifts learning from easier to harder tasks. Experiments report state-of-the-art results and student performance exceeding the teacher in most scenarios.

Abstract

from arXiv · show

Knowledge distillation (KD) is a model compression technique that transfers knowledge from a large teacher model to a smaller student model to enhance its performance. Existing methods often assume that the student model is inherently inferior to the teacher model. However, we identify that the fundamental issue affecting student performance is the bias transferred by the teacher. Current KD frameworks transmit both right and wrong knowledge, introducing bias that misleads the student model. To address this issue, we propose a novel strategy to rectify bias and greatly improve the student model's performance. Our strategy involves three steps: First, we differentiate knowledge and design a bias elimination method to filter out biases, retaining only the right knowledge for the student model to learn. Next, we propose a bias rectification method to rectify the teacher model's wrong predictions, fundamentally addressing bias interference. The student model learns from both the right knowledge and the rectified biases, greatly improving its prediction accuracy. Additionally, we introduce a dynamic learning approach with a loss function that updates weights dynamically, allowing the student model to quickly learn right knowledge-based easy tasks initially and tackle hard tasks corresponding to biases later, greatly enhancing the student model's learning efficiency. To the best of our knowledge, this is the first strategy enabling the student model to surpass the teacher model. Experiments demonstrate that our strategy, as a plug-and-play module, is versatile across various mainstream KD frameworks. We will release our code after the paper is accepted.

Introduction

Knowledge distillation transfers knowledge from a pretrained teacher to a smaller student, but teacher errors can bias student learning. The proposed strategy filters and rectifies that bias and schedules learning from easier to harder tasks.

Introduction

  • KD transfers knowledge from a pretrained teacher to a smaller student, aiming to approach teacher performance while compressing the model.The paper notes applications where efficiency and accuracy are both needed, including mobile and embedded devices.Source: Introduction · p. 1
  • Teacher predictions that disagree with true labels mislead students, motivating a distinction between right knowledge and bias.KD transfers teacher predictions through KL loss alongside true labels through cross-entropy loss.Source: Introduction · pp. 1–2
  • The strategy eliminates bias to retain right knowledge, then rectifies wrong predictions so students can learn from both right knowledge and corrected bias.Weighted adjustment converts bias into right knowledge.Source: Introduction · p. 2
  • Dynamic learning introduces easy tasks based on right knowledge first and harder bias-related tasks later to improve learning efficiency.The strategy uses an improved loss function to optimize training.Source: Introduction · p. 2
  • The authors report that the strategy outperforms state-of-the-art KD methods and enables students to surpass teachers, while functioning as a plug-and-play module.They report evaluation across image classification and object detection.Source: Introduction · p. 2

Related Work

Prior KD work transfers teacher knowledge through logits or intermediate features, often without assessing whether that knowledge is correct. The section also notes that filtering wrongly predicted samples can discard useful features without fully eliminating teacher bias.

Logits-based KD

  • Logits-based KD transfers teacher soft labels, but methods generally overlook mismatches between teacher predictions and true labels, which can mislead students.The section describes class-distance loss, decoupled logits probabilities, and dynamically adjusted temperature as extensions.Source: Related Work · p. 2

Features-based KD

  • Features-based KD transfers intermediate teacher features, where hidden bias can affect semantic extraction and mislead student predictions.FitNet aligns intermediate features with a small regression.Source: Related Work · p. 2
  • Removing intermediate features from wrongly predicted samples can discard useful information because right samples may contain bias and wrong samples may contain right features.The section characterizes this approach as an unreasonable association between wrong predictions and intermediate features.Source: Related Work · p. 2
  • Adversarial defenses can mitigate biased intermediate features but do not effectively eliminate bias in the teacher model.Source: Related Work · p. 3

Methodology

The method separates teacher knowledge into correct and biased signals, filters out bias, then rectifies wrong predictions so students can learn from them. A dynamic loss shifts training from right-knowledge easy tasks toward rectified hard tasks.

Definitions

  • Right knowledge is teacher probability aligned with true labels, while biased knowledge is misaligned; tasks with right predictions are easy and those with wrong predictions are hard for the student.These definitions connect knowledge type to task difficulty from the student's perspective.Source: Definitions · p. 3

Rethinking Knowledge Transmission

  • When teacher predictions match labels, KL and CE objectives guide the student toward compatible predictions; when they disagree, matching the teacher can mislead the student despite label supervision.The paper treats knowledge from correct teacher predictions as right knowledge and knowledge from incorrect predictions as bias.Source: Rethinking Knowledge Transmission · p. 3

Eliminating Biased Knowledge from Teacher

  • A mask marks teacher predictions that match the labels as right knowledge and mismatches as biased knowledge, allowing training updates to use the selected knowledge.The inverted mask selects biased knowledge; the original mask selects right knowledge.Source: Eliminating Biased Knowledge from… · p. 3

Rectifying Biased Knowledge from Teacher

  • To recover hard-task learning lost through elimination, the method rectifies wrong teacher predictions while preserving other-class probabilities and making the final distribution align with labels.The rectified probabilities are adjusted to sum to one without changing the probabilities for other classes.Source: Rectifying Biased Knowledge from Teacher · pp. 3–4

Dynamic Learning Approach

  • The dynamic loss emphasizes easy tasks early and increases focus on hard tasks later, using KL constraints to match the student to right and rectified teacher distributions.The schedule adjusts γ across training iterations to shift learning focus.Source: Dynamic Learning Approach · pp. 4–5

Experiment

Experiments span CIFAR-10, CIFAR-100, ImageNet 1K, and MS-COCO. The method surpasses teachers in reported classification and detection comparisons, while ablations and training analyses assess its modules and learning behavior.

Experiment

  • Experiments cover CIFAR-10, CIFAR-100, and ImageNet 1K classification, plus MS-COCO object detection.Source: Experiment · p. 5

Comparison with SOTA KD methods

Ablation Analysis

  • On CIFAR-100, either bias module alone surpasses regular KD, and combining both yields the best results across same- and different-structure models.The ablation uses ResNet-50 as teacher and ResNet-18 and MobileNet-V2 as students.Source: Ablation Analysis · pp. 6–7, Table 5 · p. 7
  • The method reaches the highest final accuracy within the same training period, reaches student accuracy in fewer epochs, and reduces training time cost by 25%.Figure 4 plots ResNet-34 accuracy on CIFAR-100 validation against training epochs.Source: Figure 4 · p. 7, Ablation Analysis · p. 7
  • The method consistently improves feature-based and logit-based KD, with its combination with DKD achieving the best results.The reported improvement is less significant for feature-based methods, where misleading intermediate-layer information is challenging to eliminate.Source: Table 6 · p. 7, Ablation Analysis · p. 7

Conclusion

The proposed KD framework addresses biased knowledge transfer by eliminating and correcting teacher bias. Across four datasets and eight teacher–student model sets, experiments report state-of-the-art results, teacher-beating performance in most scenarios, and compatibility with existing KD methods.

  • The framework targets biased knowledge transfer and uses knowledge correction to enhance students.Source: Conclusion · p. 7
  • Across four widely used datasets and eight teacher–student model sets, experiments achieve state-of-the-art results and surpass teachers in most scenarios.Source: Conclusion · p. 7
  • The framework is compatible with existing KD methods and enhances their effectiveness.Source: Conclusion · p. 7

Related papers

Loading 2412.09874v1…