Source-linked AI summary
Densely Guided Knowledge Distillation using Multiple Teacher Assistants
Wonchul Son, Jaemin Na, Junyong Choi, Wonjun Hwang
TL;DR
The paper addresses poor student learning when teacher and student capacities differ substantially. It proposes densely guided distillation with multiple teacher assistants and stochastic teaching, reporting significant accuracy improvements across datasets and architectures.
Problem
Knowledge distillation can fail when the student capacity is too low to mimic the teacher, especially when their model-size gap is large.
Method
Densely guided distillation trains progressively smaller teacher assistants using the teacher and all previously learned larger assistants, while stochastic teaching randomly removes trainer connections during student training.
Results
The proposed method achieves significant accuracy improvements over well-known knowledge-distillation methods across various datasets and network architectures.
Takeaways & Limitations
Using multiple trainers gives the low-capacity student more opportunities to learn correct knowledge and helps alleviate error avalanche in deep assistant paths.
Takeaways & Limitations
Sequential assistant distillation can accumulate incorrect knowledge, and a low-capacity student may be unable to overcome these accumulated errors when the teacher–student gap is large.
Abstract
from arXiv · showhide
With the success of deep neural networks, knowledge distillation which guides the learning of a small student network from a large teacher network is being actively studied for model compression and transfer learning. However, few studies have been performed to resolve the poor learning issue of the student network when the student and teacher model sizes significantly differ. In this paper, we propose a densely guided knowledge distillation using multiple teacher assistants that gradually decreases the model size to efficiently bridge the large gap between the teacher and student networks. To stimulate more efficient learning of the student network, we guide each teacher assistant to every other smaller teacher assistants iteratively. Specifically, when teaching a smaller teacher assistant at the next step, the existing larger teacher assistants from the previous step are used as well as the teacher network. Moreover, we design stochastic teaching where, for each mini-batch, a teacher or teacher assistants are randomly dropped. This acts as a regularizer to improve the efficiency of teaching of the student network. Thus, the student can always learn salient distilled knowledge from the multiple sources. We verified the effectiveness of the proposed method for a classification task using CIFAR-10, CIFAR-100, and ImageNet. We also achieved significant performance improvements with various backbone architectures such as ResNet, WideResNet, and VGG.
1. Introduction
Knowledge distillation struggles when teacher and student capacities differ greatly, motivating densely guided distillation with multiple teacher assistants and stochastic teaching. The proposed method addresses error accumulation while improving student learning across architectures and datasets.
- Motivation: Knowledge distillation transfers teacher outputs to less-parameterized students, but large capacity gaps can cause poor student learning.Soft logits can guide students more effectively than hard class labels.
- Limitations of prior work: TAKD bridges large teacher–student gaps with intermediate assistants, but sequential teaching can create an error avalanche as errors propagate across assistants.An error introduced by one assistant can be repeatedly transferred to lower-capacity assistants.
- Proposed method: DGKD guides each smaller assistant using the teacher and all previously learned higher-level assistants rather than a single preceding assistant.Dense guidance preserves previously distilled knowledge and provides multiple educators closer to the student.
- Proposed method: Stochastic DGKD randomly removes trainer connections during student training, producing varying trainer ensembles that act as regularization.The method randomly removes a fraction of guided knowledge for each training iteration.
- Evaluation: Experiments report significant accuracy improvements over well-known distillation methods across datasets and backbone architectures.The paper evaluates the approach through extensive experiments on varied datasets and network architectures.
2. Related Work
Related work establishes knowledge distillation as a model-compression strategy and explores teacher-free and self-distillation alternatives. For large teacher–student gaps, assistant-based methods help bridge capacities, while DGKD extends them by using all generated assistants.
- Knowledge distillation: Knowledge distillation compresses knowledge from a deeper or larger teacher into a computationally efficient student network.The teacher’s knowledge is transferred so the student can mimic the teacher.
- Alternative distillation settings: Related approaches transfer knowledge among peers or students without relying on a single teacher model.Examples include self-distillation and collaborative learning among arbitrary students.
- Alternative distillation settings: Contrastive and few-sample approaches seek broader knowledge or reduced training cost beyond traditional distillation procedures.These methods use self-supervision or align and merge compressed teacher–student models with additional layers.
- Large capacity gaps: When teacher and student capacities differ substantially, direct distillation may fail, motivating intermediate teacher assistants and gradual mimicry.TAKD sequentially deploys assistants to improve learning across a large capacity gap.
- Large capacity gaps: DGKD differs by densely guiding the student with all assistants generated between teacher and student rather than relying on a single teaching path.Using all intermediate models is intended to help avoid error avalanche.
3. Densely Guided Knowledge Distillation using Teacher Assistants
The paper proposes densely guided knowledge distillation with multiple teacher assistants to bridge large teacher–student size gaps. Teacher assistants are trained and used collectively, while stochastic teacher dropping regularizes student training.
- Background: The conventional distillation objective combines cross-entropy supervision with a temperature-scaled KL-divergence loss.The temperature controls signal softening, while λ balances the two losses.
- Motivation: Large teacher–student size gaps can make direct knowledge distillation ineffective, motivating intermediate teacher assistants.TAKD improves learning by sequentially using intermediate-sized networks, but assistant models have less knowledge than the teacher.
- Densely Guided Knowledge Distillation: DGKD trains multiple teacher assistants sequentially and guides each smaller assistant using the teacher and previously learned larger assistants.This densely connected teaching pattern extends beyond sequentially transferring knowledge only from the immediately preceding assistant.
- Densely Guided Knowledge Distillation: The student combines distilled knowledge from the teacher and all previously learned assistants, covering logit distributions across model sizes.The combined knowledge is intended to improve learning efficiency despite a large teacher–student gap.
- Stochastic DGKD: Stochastic DGKD randomly removes teacher-to-student knowledge connections during training iterations to regularize learning.Teacher or assistant knowledge is dropped according to Bernoulli survival variables, with dropping applied when teaching the student.
4. Experimental Setting
The experiments evaluate distillation methods on CIFAR-10, CIFAR-100, and ImageNet using several backbone architectures. The baseline setup uses plain CNN teacher, assistant, and student models with progressively fewer layers.
- Datasets: Experiments use CIFAR-10, CIFAR-100, and ImageNet as image-classification benchmarks.CIFAR images are 32×32 with 10 or 100 classes, while ImageNet has 1,000 classes.
- Networks: The study evaluates plain CNN, ResNet, WideResNet, and VGG architectures.The baseline plain CNN follows a VGG-like design.
- Networks: The baseline plain CNN uses a 10-layer teacher, 8-, 6-, and 4-layer assistants, and a 2-layer student.Additional assistant configurations use 9, 7, 5, and 3 layers by removing the last convolutional layer from corresponding networks.
- Implementation Details: Training uses PyTorch with random cropping, horizontal flipping, stochastic gradient descent, Nesterov momentum, and weight decay.The reported classification experiments also tune the stochastic DGKD survival probability alongside λ and τ.
5. Result and Discussion
DGKD consistently improves student performance over TAKD and other distillation or ensemble baselines, especially when teacher–student gaps are large. Its dense guidance and stochastic teaching also reduce error accumulation and improve results as the distillation path deepens.
- Comparison with TAKD: DGKD achieves better results than TAKD across the reported ablation cases, including a 3.78% improvement for the plain CNN on CIFAR-100.The comparison uses the T10 → A8 → A6 → A4 → S2 path.
- Comparison with TAKD: On ImageNet, DGKD achieves over 1% better accuracy than Hinton’s KD and the best performance among the compared methods for ResNet-34 to ResNet-18 distillation.The experiment uses ResNet-34 as teacher, ResNet-26 as a single assistant, and ResNet-18 as student.
- Classifier Ensemble: Ensembles of independently trained teachers outperform KD but fail to overcome large teacher–student gaps, whereas TA-based methods, including DGKD, solve this problem successfully.The ensemble comparison uses plain CNNs on CIFAR-100.
- Error Avalanche Problem: TAKD shows higher neighboring-model error overlap than DGKD in every tested case, with the difference increasing closer to the student.The experiment follows T10 → A8 → A6 → A4 → S2 and evaluates error overlap between neighboring models.
- Knowledge Distillation Path: With a maximum-length path, TAKD reaches 44.07% accuracy at n = 7, whereas DGKD improves with path depth and reaches 49.56%.DGKD is reported as approximately 5% better than TAKD as the path lengthens.
- Stochastic Teaching: Stochastic DGKD improves over DGKD and TAKD, achieving gains of 1.23% and 5.01%, respectively, on CIFAR-100 with a two-layer plain CNN student.The reported stochastic setting uses survival probability p = 0.75.
6. Conclusion
The paper proposes densely guided knowledge distillation with multiple teacher assistants to improve low-capacity student networks across large teacher–student gaps. It combines guidance from the teacher and all higher-level assistants with stochastic knowledge removal, and reports state-of-the-art performance among well-known distillation methods.
- DGKD uses the teacher and whole assistant hierarchy to provide the student more opportunities to learn correct knowledge.
- Deeper assistant paths can accumulate incorrect knowledge recursively, creating an error avalanche that a low-capacity student may struggle to avoid.
- Stochastic learning randomly abandons teacher or assistant knowledge during training to improve student learning efficiency.
- The proposed method achieves state-of-the-art performance among the well-known distillation methods.