Source-linked AI summary
Confidence-Aware Multi-Teacher Knowledge Distillation
Hailin Zhang, Defang Chen, Can Wang
TL;DR
Multi-teacher distillation combines diverse teacher predictions, but label-free weighting can mislead students when predictions are low quality. CA-MKD uses ground-truth labels to assign sample-wise confidence weights to predictions and intermediate features, and outperforms competitors across various teacher-student architectures.
Problem
Existing multi-teacher methods combine predictions with fixed or other label-free weights, which may mislead student training when teacher predictions are low quality.
Source: §1 · p. 1Method
CA-MKD uses ground-truth labels to estimate sample-wise confidence and weight teacher predictions and intermediate features during knowledge transfer.
Source: §1 · p. 1Results
CA-MKD outperforms competitors across various teacher-student architectures; on CIFAR-100, it achieves 0.81% average improvement over EBKD.
Source: §5 · p. 4, §4.1 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Knowledge distillation is initially introduced to utilize additional supervision from a single teacher model for the student model training. To boost the student performance, some recent variants attempt to exploit diverse knowledge sources from multiple teachers. However, existing studies mainly integrate knowledge from diverse sources by averaging over multiple teacher predictions or combining them using other various label-free strategies, which may mislead student in the presence of low-quality teacher predictions. To tackle this problem, we propose Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD), which adaptively assigns sample-wise reliability for each teacher prediction with the help of ground-truth labels, with those teacher predictions close to one-hot labels assigned large weights. Besides, CA-MKD incorporates intermediate layers to stable the knowledge transfer process. Extensive experiments show that our CA-MKD consistently outperforms all compared state-of-the-art methods across various teacher-student architectures.
1. INTRODUCTION
Knowledge distillation transfers knowledge from a pre-trained teacher to a lightweight student, and multi-teacher methods combine diverse predictions. CA-MKD uses ground-truth labels to weight teacher predictions by sample-wise confidence and extends this weighting to intermediate features.
1. INTRODUCTION
- Knowledge distillation transfers a pre-trained teacher’s softmax outputs to improve a lightweight student, with later methods also transferring intermediate-layer representations.The paper frames this transfer as a model-compression technique for models too resource-intensive for embedded devices.Source: §1 · p. 1
- Multi-teacher methods combine predictions using fixed weights or label-free schemes, which can mislead students when teacher predictions are low quality.Fixed weights do not distinguish high- from low-quality teachers, and averaging can lead the student away from the correct direction when most predictions are biased.Source: §1 · p. 1
- CA-MKD estimates each teacher prediction’s sample-wise confidence from its cross-entropy loss against ground-truth labels, then uses that confidence for adaptive integration.The authors describe this as enabling learning from a relatively correct direction compared with label-free weighting strategies.Source: §1 · p. 1
- CA-MKD outperforms competitors across eight teacher-student architecture combinations by adaptively weighting predictions and intermediate features.The method uses confidence-aware weights to keep poor teacher predictions from dominating knowledge transfer.Source: §1 · p. 1
2. RELATED WORK
Related work covers distillation from a single teacher, online knowledge sharing among students, and multi-teacher methods that combine pre-trained teachers’ predictions. CA-MKD differs from label-free multi-teacher strategies by using ground-truth labels to assess teacher predictions.
Knowledge Distillation
- Vanilla distillation transfers a teacher’s softened outputs through KL-divergence minimization, while intermediate-layer methods transfer teacher representations.Online methods instead train multiple students together and encourage them to learn from each other; CA-MKD distills from multiple pre-trained teachers.Source: §2 · p. 2
Multi-teacher Knowledge Distillation
- Multi-teacher methods combine pre-trained teachers’ predictions using fixed weights, entropy, latent factors, or gradient-space multi-objective optimization.These approaches integrate knowledge from multiple teachers rather than relying on a single teacher.Source: §2 · p. 2
- Label-free weighting can mislead student training: entropy favors sharp predictions even when wrong, while optimization can favor majority opinion and be misled by noisy data.CA-MKD instead evaluates teacher predictions using ground-truth labels.Source: §2 · p. 2
3. METHODOLOGY
CA-MKD weights teacher predictions using sample-wise confidence derived from ground-truth labels, and extends weighted knowledge transfer to intermediate features. Its overall loss combines these distillation objectives with standard cross-entropy.
3. METHODOLOGY
- CA-MKD uses labeled data and temperature-scaled softmax predictions to define its training notation and model outputs.The method denotes the labeled set as D, with N samples and K teachers.Source: §3 · p. 2
3.1. The Loss of Teacher Predictions
- Teacher predictions closer to ground-truth labels receive larger sample-wise weights, and the weighted predictions are aggregated for distillation.Weights reflect cross-entropy loss; unlike entropy-only weighting, this avoids assigning high weight merely to a sharp but incorrect prediction.Source: §3.1 · p. 2
3.2. The Loss of Intermediate Teacher Features
- CA-MKD extends distillation to intermediate features using weighted L2 matching, with a function aligning student and teacher feature dimensions.Only output features of the last block are used to limit computational cost.Source: §3.2 · pp. 2–3
- Intermediate-layer aggregation weighted by prediction confidence is reported as more effective in the ablation study.The intermediate weights also represent teacher-classifier discriminability in the student feature space.Source: §3.2 · p. 3
3.3. The Overall Loss Function
- The overall CA-MKD loss adds standard ground-truth cross-entropy to the two distillation losses, with hyperparameters balancing their effects.The two distillation objectives are prediction-based and intermediate-feature-based.Source: §3.3 · p. 3
4. EXPERIMENT
Experiments evaluate CA-MKD on CIFAR-100 across teacher–student architectures, including same- and mixed-architecture teachers, teacher-count comparisons, and ablations. Results report improvements over competing methods and show how teacher quality, architecture diversity, and intermediate-layer weighting relate to student accuracy.
4. EXPERIMENT
- On CIFAR-100, experiments cover eight teacher–student combinations; baselines include AVER, FitNet-MKD, EBKD, and AEKD, with three teachers by default.Source: §4 · p. 3
4.1. Results on the Same Teacher Architectures
- CA-MKD beats all competitors across architectures, averaging 0.81% improvement over EBKD and reaching a 1.66% absolute accuracy improvement in the best case.Source: §4.1 · p. 4
4.2. Results on the Different Teacher Architectures
- For VGG8, three teachers with different architectures yield higher student accuracy than three ResNet32x4 teachers, which the authors attribute to greater knowledge diversity.Source: §4.2 · p. 4, Table 3 · p. 4
- Figure 3 verifies that ResNet20x4 and ResNet32x4 usually receive larger weights than ResNet8x4, while confident ResNet8x4 predictions still receive dynamic weights.Source: §4.2 · p. 4, Figure 3 · p. 4
4.3. Impact of the Teacher Number
- On ResNet56 & MobileNetV2, CA-MKD accuracy rises as teachers are added and surpasses competitors using three teachers, even when those competitors use more.Source: §4.3 · p. 4, Figure 4 · p. 4
4.4. Ablation Study
- Simple teacher averaging causes a 1.67% accuracy drop; removing Equation (7) or reusing last-layer weights for intermediate features also lowers accuracy.The ablations support treating teachers according to quality and using a separate weighting approach for intermediate layers.Source: §4.4 · p. 4, Table 4 · p. 4
5. CONCLUSION
CA-MKD uses label-guided confidence weighting for teacher predictions and intermediate features in multi-teacher distillation. The paper reports that it outperforms competitors across varied teacher-student architectures.
5. CONCLUSION
- CA-MKD applies confidence-aware weighting to predictions and intermediate features for multi-teacher knowledge distillation.Teacher confidence is based on closeness to ground-truth labels for each training sample.Source: §5 · p. 4
- Label guidance lets CA-MKD integrate diverse knowledge from multiple teachers for student training.Source: §5 · p. 4
- CA-MKD outperforms competitors across various teacher-student architectures in extensive empirical results.Source: §5 · p. 4
Appendix
The appendix explains how teacher similarity in the student feature space informs knowledge transfer, then reports additional-dataset comparisons for CA-MKD.
1.1. The Detailed Description of wk
- Teacher-1 is more suitable than teacher-2 because its decision surface performs better in the student feature space.The appendix describes inter as a similarity measure for teacher-classifier discriminability in the student feature space.Source: §1.1 · p. 6, Figure 5 · p. 6
- Moving the student feature toward teacher-1’s feature can preserve classification with little or no classifier adjustment, unlike moving it toward teacher-2’s.The latter may make classification harder and disrupt student-classifier training, slowing convergence.Source: §1.1 · p. 6
2.2. Additional Dataset experiments
- CA-MKD consistently surpasses competitors on Dogs and Tinyimagenet in the additional-dataset experiments.Tables 5 and 6 report top-1 test accuracy comparisons with multi-teacher and single-teacher distillation methods, respectively.Source: §2.2 · p. 6, Table 5 · p. 6, Table 6 · p. 6
- Tinyimagenet uses CIFAR-100 hyperparameters, while fine-grained, higher-resolution Dogs images require a different training procedure following prior work.Source: §2.2 · p. 6