Source-linked AI summary
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, Kaisheng Ma
TL;DR
Convolutional neural networks face accuracy and response-time challenges in demanding applications, while traditional distillation requires separately designed teacher models. The paper proposes self distillation within one network, reporting a 2.65% average accuracy boost and depth-scalable inference for resource-limited devices.
Problem
Existing approaches face accuracy–resource trade-offs, and traditional distillation requires substantial effort to design and train suitable teacher models.
Method
Self distillation divides a network into sections and transfers knowledge from deeper portions to shallower classifiers within the same network.
Results
2.65% accuracy boost is obtained on average across the reported convolutional-network experiments, varying from 0.61% in ResNeXt to 4.07% in VGG19.
Takeaways & Limitations
Self distillation provides one network executable at different depths, enabling adaptive accuracy-efficiency trade-offs on resource-limited edge devices.
Takeaways & Limitations
Continuing conventional training of the deepest classifiers after self-distillation convergence further boosted performance by 0.3% to 0.7%, while hyper-parameter adjustment was not thoroughly investigated.
Abstract
from arXiv · showhide
Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time. In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation - a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices.Our codes will be released on github soon.
IIISCT
The supplied passage contains an email address associated with the paper.
- The passage lists the email address jean.chenjingwei@hisilicom.
- No research content is stated in the passage.
- The passage does not describe a method, result, or conclusion.
1. Introduction
The paper proposes self distillation to improve CNN accuracy without increasing response-time costs, addressing accuracy and latency demands in error-intolerant applications. Experiments report average gains of 2.65% and depth-scalable inference from one network.
- CNNs must improve prediction accuracy while shortening response time in applications such as automated driving and medical image analysis.
- Traditional distillation requires training an over-parameterized teacher before transferring knowledge to a compact student, creating efficiency and teacher-design challenges.
- Self distillation uses a one-step framework that trains the compact model directly instead of separately training a large teacher and distilling from it.
- 2.65% average accuracy boost is obtained, ranging from 0.61% in ResNeXt to 4.07% in VGG19.
- A single neural network can execute at different depths, enabling adaptive accuracy-efficiency trade-offs on resource-limited edge devices.
- Experiments cover five kinds of convolutional neural networks and two kinds of datasets to evaluate generalization.
2. Related Work
Related work covers knowledge distillation, adaptive computation, and deep supervision. The paper distinguishes self distillation by using teacher and student models from the same CNN and training shallow classifiers through distillation.
- Knowledge Distillation: Knowledge distillation compresses models by training compact students to approximate over-parameterized teachers.
- Self Distillation Architecture: The proposed architecture divides a ResNet into four depth-based sections and adds bottleneck and fully connected layers after each section to form multiple classifiers.
- Self Distillation Architecture: Each classifier can operate independently, providing different accuracy and response-time options.
- Knowledge Distillation: Traditional knowledge distillation transfers knowledge between separate teacher and student models, whereas self distillation derives both from the same CNN.
- Adaptive Computation: Adaptive-computation methods reduce execution by selectively skipping layers, channels, or less important input pixels.
- Deep Supervision: Self distillation differs from deeply supervised networks because shallow classifiers receive distillation supervision rather than labels alone.
3. Self Distillation
Self distillation divides a convolutional network into sections with multiple classifiers and trains shallow classifiers using labels, deepest-classifier outputs, and feature hints. Training-only additions support comparisons with traditional distillation and deeply supervised networks while preserving inference flexibility.
- Framework construction: The framework divides the target network into shallow sections and adds a classifier to each section, including bottleneck and fully connected layers.For example, ResNet50 is divided into four sections according to ResBlocks.
- Inference flexibility: The added training layers do not affect inference, enabling depth-wise scalable inference for resource-limited edge devices.The framework can therefore support runtime time-accuracy tradeoffs through adaptive depth.
- Training objectives: Each shallow classifier receives cross-entropy supervision from dataset labels, while the deepest classifier is supervised only by labels.The deepest classifier’s balancing coefficients for the other supervision sources are zero.
- Training objectives: Kullback-Leibler divergence trains shallow classifiers to approximate the deepest classifier’s softmax outputs.The deepest classifier supplies the distillation target through its softmax output.
- Comparative results: Self distillation outperforms traditional distillation methods under matched computation and storage, and exceeds deep supervision for every ResNet classifier.Shallow classifiers benefit more from self distillation than from deep supervision.
- Training objectives: Feature-map hints use L2 distance and bottleneck layers to align shallow and deepest-classifier representations.Additional layers align feature-map sizes, and the bottleneck architecture is used instead of a convolutional alignment layer.
4. Experiments
Experiments across five convolutional neural networks and two datasets evaluate self distillation, including its accuracy gains, comparisons with other training methods, and scalable inference behavior.
- Self distillation improved accuracy by 2.65% on CIFAR100 and 2.02% on ImageNet on average.
- 4.05% improvement was observed for ResNet101, compared with 2.58% for ResNet18.
- Self distillation outperformed directly trained student networks and most traditional distillation methods at matched computation and storage.
- 4.6X acceleration in training time was achieved because self distillation uses network subsections as teachers and students instead of an extra teacher model.
- Self distillation outperformed deep supervision for every ResNet classifier, with greater benefits for shallow classifiers.
- 3.16X acceleration was achieved with classifier 2/4 at an accuracy loss of 3.3%, while the deepest-three-classifier ensemble improved accuracy by 0.67% with a 0.05% computation penalty.
5. Discussion and Future Works
The discussion attributes self distillation’s performance gains to flatter minima, stronger gradients, and more discriminating features. It also identifies unresolved questions about hyper-parameter tuning and whether the learned flat minimum is ideal.
- Flat minima: Self distillation can help models converge to flat minima associated with generalization.In noise experiments, two 18-layer ResNets trained with and without self distillation were compared after Gaussian perturbations.
- Vanishing gradients: Self distillation produces larger gradient magnitudes, especially in the first and second ResBlocks, than conventional training.The comparison used two 18-layer ResNets and measured mean gradient magnitude in each convolutional layer.
- Discriminating features: Deeper classifiers generate more concentrated and stable feature clusters than shallow classifiers.Figure 6 compares classifiers in a WideResNet trained on CIFAR100; Table 6 uses SSE/SSB to assess cluster density and discrimination.
- Discriminating features: More discriminating feature maps are associated with higher model accuracy.The discussion links decreasing SSE/SSB across deeper classifiers with clearer class separation and higher accuracy.
- Future works: Automatic adjustment of λ and α remains unexplored because limited computation prevented a thorough investigation.Future work may investigate learning-rate-decay- or momentum-inspired adjustment methods.
- Future works: Continuing conventional training after self distillation can further improve the deepest classifier by 0.3% to 0.7%, but shallow classifiers may prevent final convergence.The authors leave this improvement out of the paper’s tables and suggest alternating training methods.
6. Conclusion
The paper concludes that self distillation improves convolutional-network performance without requiring an extra teacher model. It also frames the technique as enabling runtime depth selection while clarifying that its primary purpose is performance improvement rather than compression or acceleration.
- Conclusion: Self distillation improves convolutional-network performance without requiring an extra teacher model.The technique is compared with deeply supervised networks and previous distillation methods.
- Conclusion: Self distillation provides a single network executable at different depths for runtime accuracy-efficiency trade-offs.This design targets resource-limited edge devices.
- Scope: Self distillation is primarily a training technique for boosting performance, not a method for compressing or accelerating models.The authors distinguish knowledge transfer within one model from transfer among different models.