Source-linked AI summary
Knowledge Distillation by On-the-Fly Native Ensemble
Xu Lan, Xiatian Zhu, Shaogang Gong
TL;DR
Knowledge distillation seeks compact, generalisable models, but offline methods require pre-trained teachers and online methods lack a strong teacher. ONE trains a single multi-branch network while constructing an ensemble teacher on-the-fly, and experiments show stronger generalisation than alternative methods across four benchmarks with lower training cost.
Problem
Offline distillation requires a pre-trained teacher and multi-phase training, while online distillation lacks a strong teacher and can require costly multiple-network training.
Method
ONE trains a single multi-branch target network while simultaneously constructing an on-the-fly native ensemble teacher for one-phase knowledge transfer.
Results
ONE produces more generalisable target models than offline and peer-distillation alternatives across CIFAR10, CIFAR100, SVHN, and ImageNet, while taking the least model training cost.
Takeaways & Limitations
ONE improves both small and large network learning with less computational cost by constructing a high-capacity online teacher during training.
Takeaways & Limitations
The formulation assumes labelled samples with ground-truth class labels for its supervised classification objective.
Abstract
from arXiv · showhide
Knowledge distillation is effective to train small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a highcapacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation. Specifically, ONE trains only a single multi-branch network while simultaneously establishing a strong teacher on-the- fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.
1 Introduction
Knowledge distillation addresses the need for compact yet discriminative models, but offline methods require pre-trained teachers while online methods trade away teacher capacity and efficiency. ONE uses a single multi-branch network to form an on-the-fly teacher and achieves stronger generalisation with lower training cost across four benchmarks.
- Knowledge distillation trains smaller student models using supervision from high-capacity teachers or ensembles.
- Online distillation combines teacher and student training into one phase by having peer student models learn from each other.
- Existing online methods provide limited peer information, increase computational costs by training multiple students, and require asynchronous model updates.
- ONE constructs a multi-branch target network, forms a native ensemble teacher on-the-fly, and distils its predictions into each branch during training.The trained model is converted back to the original single-branch architecture for testing, avoiding additional test-time cost.
- Extensive experiments on CIFAR10, CIFAR100, SVHN, and ImageNet show that ONE trains more generalisable target models in one phase than offline and peer-distillation alternatives.
2 Related Work
Prior work uses pre-trained teachers or peer students for knowledge transfer, while ONE addresses online distillation's missing strong teacher and computational burden. The method is architecture-independent and learns an online teacher and target network together in one phase.
- Knowledge Distillation: Knowledge distillation transfers classification probabilities, feature representations, or inter-layer flow from larger or more powerful networks to smaller target models.
- Knowledge Distillation: Offline distillation generally requires at least two training phases, whereas deep mutual learning performs one-phase online teaching between peer students.
- Online Distillation: Existing online distillation lacks a strong teacher, limiting knowledge discovery and transfer while retaining the computational expense of training multiple networks.
- Multi-branch Architectures: ONE is a meta-network learning algorithm designed to improve any given target network rather than introduce a new multi-branch building block.
3 Knowledge Distillation by On-the-Fly Native Ensemble
ONE reformulates online distillation as a single multi-branch network that builds a teacher ensemble during training and distils its knowledge back to the branches. The design shares low-level layers, supports single-model deployment without extra test-time cost, and targets improved generalisation with lower training complexity.
- Knowledge Distillation: Each branch learns from both its ground-truth label and the on-the-fly teacher through cross-entropy and distillation losses.The teacher’s knowledge is computed for each mini-batch and distilled back to all branches in a closed-loop process.
- On-the-Fly Native Ensemble: ONE adds m auxiliary branches sharing low-level layers with the target network, then uses a gate to ensemble all branches into a teacher.Each branch has an individual classifier, while the gate learns branch importance to produce the teacher logits.
- Design Merits: ONE’s multi-branch design aims to create a strong teacher without training multiple full networks, regularise simultaneous learning, and avoid asynchronous updates.The branches share most layers, reducing training cost relative to separately trained networks.
- Knowledge Distillation: Higher temperature values produce softer branch and teacher probability distributions for knowledge transfer.The method defines soft targets for each branch and the teacher using temperature T.
- Model Training and Deployment: The target network and ONE teacher are trained simultaneously in one phase, with distillation performed in every mini-batch throughout training.The single multi-branch network uses standard incremental batch-wise optimisation rather than asynchronous updates among multiple networks.
- Model Training and Deployment: After training, auxiliary branches can be removed to recover the original target architecture, or retained for the optional ONE-E ensemble deployment.Removing the branches leaves single-model deployment with no increased test-time cost.
4 Experiments
Across CIFAR10/100, SVHN, and ImageNet, ONE improves generalisation across network models while reducing training cost relative to alternative distillation and ensembling methods. Ablations and robustness tests attribute these gains to online teacher transfer, multi-branch regularisation, and wider local minima.
- Experimental setup: ONE is evaluated on CIFAR10, CIFAR100, SVHN, and ImageNet using top-1/top-5 error rates and FLOPs for training and test cost.ONE reports average branch performance with standard deviation.
- Results on CIFAR and SVHN: All evaluated networks benefit from ONE, with larger gains for small models, while individual branches achieve similar performance.The authors interpret similar branch performance as evidence of agreement and knowledge exchange through the ONE teacher.
- Results on ImageNet: On ImageNet, ONE produces more effective training and more generalisable models than vanilla SGD, supporting application to large-scale image classification.The comparison uses the 1000-class ImageNet benchmark.
- Comparison with distillation methods: ONE outperforms offline KD and online DML in error rates, with the least training cost and the same test cost as the alternatives.KD uses ResNet-110 as teacher and ResNet-32 as student; DML and ONE are evaluated with ResNet-32 or ResNet-110 targets.
- Comparison with ensembling methods: ONE-E achieves the best test error and lowest test cost among compared ensembling methods, at the second-lowest training cost.Snapshot Ensemble has the least training cost but the highest deployment cost and unsatisfactory generalisation performance.
- Comparison with ensembling methods: Compared with 2-Net Ensemble, ONE improves error rate, training cost, and test cost; compared with 3-Net Ensemble, it approaches generalisation with greater efficiency.The comparison also distinguishes ONE without branch ensemble from ensemble deployment.
- Component analysis: Removing online distillation increases test error by 3.11% from 21.62% to 24.73%, while the ONE teacher error falls from 21.84% to 21.03% through closed-loop distillation.Without distillation, the multi-branch design still generalises better than vanilla training; adding branches also improves ResNet-32 generalisation over 31.18% error.
- Robustness analysis: ONE finds the widest local minimum among ONE, DML, and vanilla solutions under parameter perturbations, while retaining lower training errors than DML.The robustness test perturbs solutions as θ*(d,v)=θ*+d·v across magnitudes d∈[0,5] and five random directions.
5 Conclusion
ONE improves learning for both small and large networks with less computational cost, and its benefits are especially pronounced for smaller networks. It therefore suits low-memory and fast-execution scenarios.
- ONE improves deep network learning through online knowledge distillation in a one-stage training procedure.
- ONE enables more discriminative learning of both small and large networks with less computational cost than conventional offline alternatives.
- Smaller networks obtain more performance gains from ONE, supporting low-memory and fast-execution scenarios.