Source-linked AI summary
Online Knowledge Distillation with Diverse Peers
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, Chun Chen
TL;DR
The paper addresses the rapid homogenization of peers in teacher-free online distillation, where simple group aggregation can reduce the diversity needed for effective knowledge transfer. OKDDip uses attention-weighted first-level distillation among auxiliary peers and second-level distillation into a group leader. Experiments report consistently better generalization than state-of-the-art online methods and classic teacher-guided KD without increased training or inference cost or complexity.
Problem
Teacher-free group distillation uses peer-derived targets, but simple aggregation can quickly homogenize group members and limit within-group knowledge transfer.
Method
OKDDip performs two-level distillation: attention-based, individually weighted targets train diverse auxiliary peers, whose ensemble is then distilled into an inference group leader.
Results
OKDDip consistently generalized better than state-of-the-art online approaches and classic teacher-guided KD without increasing training or inference cost or complexity.
Takeaways & Limitations
Diversity-maintained group distillation provides a teacher-free deployment option in which auxiliary peers transfer their ensemble knowledge to an ultimate group leader.
Takeaways & Limitations
Experiments use a fixed group-size setup in which group-based methods generally have four students, while OKDDip aggregates one fewer peer because one model is reserved as group leader.
Abstract
from arXiv · showhide
Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this work, we propose Online Knowledge Distillation with Diverse peers (OKDDip), which performs two-level distillation during training with multiple auxiliary peers and one group leader. In the first-level distillation, each auxiliary peer holds an individual set of aggregation weights generated with an attention-based mechanism to derive its own targets from predictions of other auxiliary peers. Learning from distinct target distributions helps to boost peer diversity for effectiveness of group-based distillation. The second-level distillation is performed to transfer the knowledge in the ensemble of auxiliary peers further to the group leader, i.e., the model used for inference. Experimental results show that the proposed framework consistently gives better performance than state-of-the-art approaches without sacrificing training or inference complexity, demonstrating the effectiveness of the proposed two-level distillation framework.
Introduction
Online distillation avoids pre-training a powerful teacher by training student groups from peer-derived soft targets, but simple aggregation can homogenize peers. OKDDip preserves peer diversity through attention-based targets and transfers their ensemble to a group leader for inference.
- Motivation: Teacher-guided distillation transfers soft predictions to smaller models, whose deployment is motivated by the computational demands of over-parameterized networks.Soft predictions carry richer information than hard labels for optimizing small networks.
- Online distillation: Online group-based distillation jointly trains multiple students using ground-truth labels and aggregated intermediate peer predictions, eliminating the need for a pre-trained large teacher.This unified approach reduces training cost and pipeline complexity relative to vanilla two-stage KD.
- Problem: Simple aggregation can quickly homogenize peers, reducing the effectiveness of group distillation when peer prediction quality differs.OKDDip addresses this challenge with multiple auxiliary peers and one group leader.
- OKDDip: The ensemble of diverse auxiliary peers is distilled into a group leader, which serves as the inference model while auxiliary peers are not retained for inference.The provided passage states that OKDDip uses a group leader for inference and that the framework maintains efficient inference.
- OKDDip: Each auxiliary peer uses attention-based, individually learned aggregation weights to derive distinct target distributions from other peers and maintain diversity.The asymmetric weights allow higher-quality peers to receive greater importance than under simple aggregation.
- Results: Across CIFAR-10, CIFAR-100, and ImageNet-2012, OKDDip consistently generalized better than state-of-the-art online methods and classic teacher-guided KD without increased cost or complexity.The study also observed larger peer diversity and stronger ensembles than competing approaches.
Related Work
Knowledge distillation compresses teacher knowledge into smaller models, while online variants replace a pre-trained teacher with peer prediction exchange. The paper highlights that equal or shared targets can reduce student diversity, and describes self-attention as a mechanism for weighted interaction among elements.
- Knowledge Distillation: Knowledge distillation steers a smaller student toward a large teacher's predictions, using soft class information to support robust generalization.Soft predictions contain fine-grained class information beyond hard ground-truth labels.
- Online Knowledge Distillation: Online knowledge distillation jointly trains student models from peers' predictions instead of using a pre-trained teacher, with designs differing in target construction and parameter sharing.Some approaches use separate networks, while others share early blocks to reduce training cost.
- Online Knowledge Distillation: Equal peer weighting or forcing all students to learn from identical targets hurts student diversity and limits within-group knowledge transfer.This limitation motivates target-construction mechanisms that distinguish peer contributions.
- Self-Attention: Self-attention captures global dependencies by mapping each position to query, key, and value vectors and weighting neighboring values according to query-key matches.The mechanism produces an output by averaging neighbors' values with different learned weights.
Peers
OKDDip uses two-level online distillation to preserve diversity among auxiliary peers and transfer their ensemble knowledge to a deployable group leader. Attention-based, asymmetric aggregation produces personalized targets, while the combined objective balances group sharing with independent learning.
- Two-level distillation: OKDDip trains m−1 auxiliary peers with individual group-derived targets, then distills their ensemble knowledge into the mth group leader for deployment.The auxiliary peers and group leader use the same network architecture; only the leader is retained after training.
- Attention-based targets: Each auxiliary peer uses attention-based weights over group members to generate a distinct target distribution from their predictions.The weights are learned from projected peer features and normalized embedded Gaussian distances.
- Attention-based targets: Asymmetric weights let a well-performing peer reduce influence from a poorly performing peer while allowing the less optimized peer to learn strongly from the better one.This directional weighting differs from treating all peers equally and is intended to suppress negative contributions.
- Loss and diversity: The first-level mechanism increases independence between peers’ targets, helping prevent rapid diversity degradation during group-based distillation.The paper reports that the attention-based properties enable OKDDip to outperform state-of-the-art approaches and simplified variants.
- Loss and diversity: The first-level distillation loss can be approximated by mean squared error on zero-meaned logits and balances group sharing with independent learning.The independent-learning extreme has αaa = 1 and αab = 0 for b ≠ a, while the total objective also includes ground-truth and second-level distillation terms.
Experiments
Experiments evaluate OKDDip across datasets, architectures, model organizations, peer diversity, ensembles, ablations, group sizes, and teacher-guided extensions. Across these settings, OKDDip maintains greater peer diversity and generally achieves lower error rates without increasing cost or complexity.
- Experimental setup: Evaluations cover CIFAR-10, CIFAR-100, and ImageNet-2012 using multiple network architectures and comparisons with online, independent, baseline, and teacher-guided methods.Results include network-based and branch-based student implementations, with four students in most group-based comparisons.
- Classification performance: OKDDip achieved lower error rates than all compared approaches for both network-based and branch-based student models.In the best cases, it outperformed Baseline and DML by 17% and 20%, and Ind, CL-ILR, and ONE by 16%, 8%, and 9%, respectively.
- Peer diversity: Peer diversity under OKDDip remained significantly larger than under CL-ILR and ONE across training for both evaluated ResNet architectures.Its diversity approached independently trained students, treated as an upper bound, while CL-ILR and ONE showed sharper declines before later learning-rate changes.
- Ensemble performance: OKDDip produced a more effective ensemble than competing group methods and even lower error rates than ensembles of individually trained students.The authors attribute this result to learning distinct targets for auxiliary peers while transferring the resulting group knowledge to the group leader.
- Ablation study: Ablations increased error rates by 2.61% without self-attention, 0.72% with mean aggregation, 0.42% without asymmetric weights, and 2.16% without two-level distillation.The self-attention ablation using an entropy term reduced performance by 1.08%.
Conclusion
OKDDip uses two-level online distillation to preserve peer diversity and transfer enhanced group knowledge to a group leader. It reportedly outperforms state-of-the-art online approaches without additional training or inference cost.
- Conclusion: OKDDip maintains diversity among auxiliary peers by deriving distinct target distributions with attention-based aggregation weights.The auxiliary peers are discarded after training, while their diversity-enhanced knowledge is transferred to the group leader.
- Conclusion: A second distillation level transfers diversity-enhanced group knowledge from auxiliary peers to the group leader used for inference.
- Conclusion: OKDDip outperforms state-of-the-art online knowledge-distillation approaches without additional training or inference cost.
Supplementary Material
The supplementary material reports additional experiments covering ensemble predictions and peer diversity comparisons. It states that these results show observations similar to those in the main submission.
- Supplementary Material: The supplementary material reports additional experimental results with observations similar to those in the main submission.
- Supplementary Material: Ensemble results cover network-based students on CIFAR-10 and CIFAR-100, plus branch-based students on CIFAR-10.
- Supplementary Material: Table 10 compares OKDDip+KD with Baseline, classic KD, and teacher-free OKDDip using ensemble classification error rates with an additional teacher.
- Supplementary Material: Figure 4 compares peer diversity across four approaches for WRN-20-8 on CIFAR-10 and CIFAR-100, with independently trained students serving as an upper-bound reference.