Source-linked AI summary
Adaptive Multi-Teacher Multi-level Knowledge Distillation
Yuang Liu, Wei Zhang, Jun Wang
TL;DR
Knowledge distillation must help lightweight students learn from teachers without overlooking the potential value and differing relevance of multiple teachers. AMTML-KD addresses this by adaptively weighting teachers for soft-target integration and gathering intermediate hints from multiple teachers, with experiments reporting state-of-the-art image-classification performance and effective components.
Problem
Prior distillation methods often use one teacher or treat multiple teachers equally, leaving instance-specific teacher importance insufficiently addressed.
Method
AMTML-KD uses latent representations to learn instance-level teacher weights for integrated soft-targets and a multi-group hint strategy for intermediate-level knowledge.
Results
AMTML-KD achieves state-of-the-art performance on image classification, while experiments verify the effectiveness of its key components.
Takeaways & Limitations
A student can learn high-level and intermediate-level knowledge from multiple teachers through adaptive teacher integration.
Abstract
from arXiv · showhide
Knowledge distillation~(KD) is an effective learning paradigm for improving the performance of lightweight student networks by utilizing additional supervision knowledge distilled from teacher networks. Most pioneering studies either learn from only a single teacher in their distillation learning methods, neglecting the potential that a student can learn from multiple teachers simultaneously, or simply treat each teacher to be equally important, unable to reveal the different importance of teachers for specific examples. To bridge this gap, we propose a novel adaptive multi-teacher multi-level knowledge distillation learning framework~(AMTML-KD), which consists two novel insights: (i) associating each teacher with a latent representation to adaptively learn instance-level teacher importance weights which are leveraged for acquiring integrated soft-targets~(high-level knowledge) and (ii) enabling the intermediate-level hints~(intermediate-level knowledge) to be gathered from multiple teachers by the proposed multi-group hint strategy. As such, a student model can learn multi-level knowledge from multiple teachers through AMTML-KD. Extensive results on publicly available datasets demonstrate the proposed learning framework ensures student to achieve improved performance than strong competitors.
1. Introduction
Large models and ensembles improve performance but demand resources and can be slow, motivating knowledge distillation from teachers to smaller, faster students. AMTML-KD extends this paradigm by adaptively combining multi-level knowledge from multiple teachers, with experiments reporting improved effectiveness.
- Motivation: Large deep models and ensembles increase model size, while their computational and memory demands restrict deployment on limited-capability devices.Complex models can also be slow when applications require high-speed execution.
- Motivation: Knowledge distillation trains a smaller, faster student from a larger teacher while retaining high classification performance.Original KD transfers soft-targets, whereas FitNet transfers intermediate representations.
- Research gap: Prior work mainly distills from one teacher, while multi-teacher methods either treat teachers equally or require manually tuned importance weights.These approaches do not automatically discriminate teacher importance for individual instances.
- Proposed framework: AMTML-KD adaptively learns instance-level teacher weights for integrated soft-targets and uses a multi-group hint strategy to transfer intermediate knowledge from multiple teachers.The framework combines high-level soft-target knowledge with intermediate-level hint knowledge.
- Experiments: Extensive experiments on several publicly available datasets verify AMTML-KD’s effectiveness and the benefits of its main components.The supplied contribution statement reports experimental validation without specifying numerical results.
2. Related work
Related work covers model compression and multiple forms of distillation knowledge, but relatively few studies address learning from multiple teachers. Existing multi-teacher approaches include noise-based regularization, multi-task or multi-domain settings, manual weighting, and equal fusion.
- Model compression: Model compression methods use trained large models or teacher-generated labels to help learn smaller models, with distillation combining ground-truth and teacher labels.Privilege learning is described as a related strategy using information available only during training.
- Knowledge forms: Distillation research transfers soft-targets, intermediate representations, explicit rules, and structural relationships as different forms of knowledge.FitNet and attention transfer use intermediate representations, while relational knowledge distillation models structural knowledge.
- Multi-teacher distillation: Only a few studies consider multiple teachers, including noise-based regularization and teacher assignments based on source domains or tasks.These settings differ from multiple teachers serving one task in one domain.
- Limitations of prior work: Manual teacher weighting is costly and non-automatic, whereas equal soft-target averaging cannot capture different teacher importance for each data instance.The cited approaches therefore lack adaptive discrimination among teachers for a general single-task, single-domain setting.
3. Proposed learning framework
AMTML-KD adaptively combines high-level soft-target knowledge and intermediate-level hints from multiple teachers. It learns instance-level teacher importance through latent teacher representations and assigns teacher-specific guidance across student layer groups.
- Overview: AMTML-KD addresses adaptive multi-level knowledge distillation from multiple teachers by combining soft-target and intermediate-level knowledge.The framework is motivated by the differing usefulness of teachers for different problems and examples.
- Overview: For each image, teacher networks produce soft targets and intermediate representations, while an adapter learns instance-level importance weights for integrating soft targets.The integrated soft targets support both standard distillation and structural-relation losses.
- Adaptive teacher importance: Each teacher is represented by a latent factor θ_t, and its image-specific score γ_t,i is computed from an element-wise interaction with the image representation.The image representation is extracted from the student’s last convolutional layer and max-pooled to match the teacher-factor space.
- Adaptive teacher importance: The framework weights teacher soft targets rather than averaging them, with larger normalized weights indicating greater teacher importance for an image.The weighted integrated soft target is used as fused high-level knowledge for student training.
- High-level knowledge distillation: An angle-based loss transfers structural relations among examples, complementing the standard distillation loss based on integrated soft targets.The angle metric operates on triplets of examples, and Huber loss provides robust regression.
- Intermediate-level knowledge: The multi-group hint strategy assigns each teacher to a group of student layers and uses mapped teacher features to transfer intermediate-level knowledge.Teachers with better performance are assigned to higher-layer student groups by default because this strategy performs slightly better than alternatives.
4. Experimental setup
Experiments evaluate AMTML-KD against representative single-teacher, multi-teacher, and multi-student distillation baselines on image-classification datasets. The setup uses standard datasets, tuned hyperparameters, and student–teacher model comparisons summarized in tables.
- Baselines: The baseline set includes OKD, FitNet, RKD, AvgMKD, DML, and related single- and multiple-teacher knowledge-distillation methods.The comparisons cover standard and state-of-the-art approaches, including deep mutual learning with multiple students.
- Baselines: Baseline notation records the teacher or student count, such as OKD(X), DML(nS), and AvgMKD(mT).The same notation principle applies to other multiple-teacher approaches, including AMTML-KD.
- Implementation details: All methods are tuned on validation datasets; the default settings include temperature 5.0, λ = 0.7, α = 1, β = 2, and mini-batches of 128.Optimization uses SGD, with the learning rate initialized at 0.1 and training performed on a single GTX 1080Ti GPU.
5. Experimental results
Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet evaluate AMTML-KD across students, teacher counts, resource costs, and ablations. AMTML-KD consistently improves over comparison methods, while adaptive weighting and multi-group hints contribute to performance.
- Comparison with baselines: 93.64% top-1 accuracy improves CIFAR-10 performance over AvgMKD(3T)'s 93.01%.The comparison is reported for AMTML-KD(3T) against the previous state-of-the-art AvgMKD(3T).
- Comparison with baselines: AMTML-KD outperforms AvgMKD and DML across different student networks on CIFAR datasets.The approach is reported to have potential for wide application across student architectures.
- Effect of the number of teachers: More teachers improve all student networks, but gains become smaller as the teacher count increases beyond three.The gap between two and three teachers is larger than the gap between three and five teachers.
- Resource cost: In multi-teacher training, AMTML-KD requires only slightly more training time and GPU memory, whereas DML consumes more resources by updating all student networks simultaneously.The additional AMTML-KD parameters from FitNets and the adapter are described as negligible compared with the large models.
- Results on Tiny-ImageNet: On Tiny-ImageNet, AMTML-KD achieves the best result and remains robust across datasets, while multi-teacher guidance outperforms single-teacher models.AMTML-KD also boosts all three students over AvgMKD on Tiny-ImageNet.
- Ablation study: Ablations show benefits from adaptive weighting, structural knowledge, and intermediate-level knowledge transfer.Removing RKD reduces accuracy by about 0.2%, while the adapter, HT loss, and adaptive angle loss each serve distinct roles.
- Ablation study: Weight visualizations indicate that adaptive allocation can shield the student from erroneous predictions by some teachers.The examples include CIFAR-10 and MNIST visualizations.
6. Conclusion
AMTML-KD enables students to learn high- and intermediate-level knowledge from multiple teachers through adaptive weighting and multi-group hints. Image-classification experiments report state-of-the-art performance and support the effectiveness of its key components.
- AMTML-KD learns instance-specific importance weights to integrate soft-targets from multiple teachers for high-level knowledge transfer.
- Its multi-group hint strategy gathers intermediate-level knowledge from multiple teachers.
- Image-classification experiments demonstrate state-of-the-art performance for AMTML-KD.
- The experiments verify the effectiveness of AMTML-KD’s key components.