Source-linked AI summary
Self-supervised Knowledge Distillation Using Singular Value Decomposition
Seung Hyun Lee, Dae Ha Kim, Byung Cheol Song
TL;DR
Existing teacher-student distillation transfers insufficient knowledge, supports limited network structures, and loses transferred knowledge during later training. This paper uses SVD- and RBF-based knowledge distillation with self-supervised learning, and reports that the resulting student can outperform the teacher and a state-of-the-art distillation baseline.
Problem
Existing teacher-student distillation extracts limited information, supports limited structures, and loses transferred knowledge as subsequent training progresses.
Method
The method compresses and transfers feature-map knowledge using SVD and RBF, while self-supervised multitask training continuously transfers knowledge from the teacher.
Results
A student using one-fifth of the teacher's computation became 1.79% better than the teacher-student baseline, while the method generalized across VGG, MobileNet, and ResNext.
Takeaways & Limitations
Combining the proposed distillation technique with the training mechanism improved performance over the student alone and enabled continued teacher-knowledge transfer during learning.
Takeaways & Limitations
The paper identifies limited network structures, low knowledge quality, and rapid knowledge loss in existing transfer techniques as scope boundaries motivating its method.
Abstract
from arXiv · showhide
To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To improve the quality of the transferred knowledge from T-DNN, we propose a new knowledge distillation using singular value decomposition (SVD). In addition, we define a knowledge transfer as a self-supervised task and suggest a way to continuously receive information from T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of the T-DNN can be up to 1.1\% better than the T-DNN in terms of classification accuracy. Also assuming the same computational cost, our S-DNN outperforms the S-DNN driven by the state-of-the-art distillation with a performance advantage of 1.79\%. code is available on https://github.com/sseung0703/SSKD\_SVD.
1 Introduction
The paper targets the computational burden of DNNs and the limited transfer quality and scope of existing teacher-student distillation. It combines SVD/RBF knowledge compression with self-supervised learning to improve transfer and retain teacher information during training.
- 1 Introduction: Existing teacher-student methods extract limited teacher information, support restricted network structures, and lose transferred knowledge as the student learns its main task.These limitations motivate richer transfer and a mechanism that keeps teacher knowledge active beyond parameter initialization.
- 1 Introduction: SVD compresses teacher knowledge while RBF analyzes correlations between compressed feature maps, enabling more efficient transfer across spatial resolutions.The approach uses SVD for knowledge compression and RBF for feature-map correlation analysis.
- 1 Introduction: Self-supervised learning treats knowledge transfer as a training task so the student can continuously receive teacher information rather than only initialize its parameters.The paper also motivates self-supervision as a source of additional regularization.
- 1 Introduction: The method generalizes across VGG, MobileNet, and ResNext, with a student using 1/5 of the teacher’s computation outperforming the teacher by 1.79%.The reported experiments also state that self-supervised learning further improves student performance by up to 1.2%.
2 Related Works
Related work frames knowledge distillation as transferring information from a complex teacher to a smaller student, while prior approaches use staged initialization or feature correlations. The paper builds on SVD, RBF, and self-supervised learning to make transfer more informative and efficient.
- 2 Related Works: Knowledge transfer moves information from a complex teacher DNN to a smaller student DNN to improve the student’s performance.Earlier methods commonly use transferred knowledge to obtain better student initialization before retraining the main task.
- 2 Related Works: Prior feature-based distillation represents knowledge through correlations between teacher and student feature maps, often using a Gram matrix over selected layer groups.This approach differs from transferring layer parameters directly.
- 2 Related Works: SVD is used to reduce feature-map dimensions and extract important information, while RBF remaps features by distance for kernelized analysis and robust handling of noisy data.The proposed method combines these operations to extract core feature-map knowledge and compute correlations between feature maps.
- 2 Related Works: The paper proposes self-supervised learning as a more efficient transfer approach than using knowledge transfer only for parameter initialization.Self-supervised learning generates labels and learns them within the training process.
3 Method
The method distills correlations between teacher and student feature maps by compressing them with truncated SVD and computing robust representations with Gaussian RBF. It then jointly trains the main and transfer tasks so teacher knowledge is continuously retained during student learning.
- 3.1 Proposed Distillation Module: The distillation module converts teacher and student front-end and back-end feature maps into correlation-based distillation feature vectors, supporting different feature-map sizes.It senses two layer-module endpoints and produces DFVs from their feature-map relationship.
- 3.1 Proposed Distillation Module: Truncated SVD compresses feature maps by retaining rank-k components, using V and Σ to preserve global information and energy with reduced memory and computation.The resulting compressed representations are designed to fit smaller student networks while minimizing information loss.
- 3.1 Proposed Distillation Module: Post-processing normalizes teacher singular values and combines them with teacher singular vectors to form compressed feature vectors learned in decreasing importance order.This addresses ordering and sign ambiguities that can occur among singular vectors with similar information.
- 3.1 Proposed Distillation Module: Gaussian RBF computes correlations between compressed feature vectors because the singular-vector-derived feature information can be noisy.The resulting correlation is used as the knowledge transferred from the teacher to the student.
- 3.2 Training Mechanism: Self-supervised joint training adds a transfer loss to the main-task loss, allowing teacher knowledge to be continuously transferred instead of disappearing after initialization.The method uses teacher-generated knowledge as labels and reports that joint multi-task learning can further improve student performance.
4 Experimental Results
Experiments evaluate the proposed distillation across network structures, feature-map resolutions, training mechanisms, labeled-data availability, and transferred-feature counts. The method consistently improves compact S-DNN performance, including results that surpass the teacher in one setting.
- Small Network Enhancement: The proposed method outperformed across VGG, MobileNet, and ResNext, improving over S-DNN by 3.68%, about 2%, and 1.43%, respectively.It also exceeded T-DNN alone by 0.61% for VGG and 0.3% for MobileNet.
- Small Network Enhancement: 6.98% improvement was obtained for an S-DNN using about 0.03 times T-DNN FLOPS, even when feature-map spatial resolution was reduced by strided convolution.The comparison used only the proposed method because knowledge transfer using was impossible for this structure.
- Training Mechanism: Up to 1.1% higher accuracy than T-DNN and up to 1% higher than were achieved with an S-DNN costing 1/5 as much as T-DNN.The combined method also improved performance by about 4.17% over S-DNN alone.
- Training Mechanism: The training mechanism improved performance by 0.35% for and 0.49% when combined with the proposed distillation technique.It continuously regularizes S-DNN while reducing the required learning epochs by half compared with the conventional two-stage structure.
- Number of DFVs: Performance generally improved with transferred DFVs, but excessive DFVs caused accuracy to rise and then drop for pooling-based S-DNNs.Stride-based S-DNNs showed a slight increase, so the number of DFVs should be selected according to network structure and available cost.
5 Conclusion and Future Work
The paper proposes SVD- and RBF-based knowledge transfer without network-structure limitations, with self-supervised multi-task learning to preserve teacher knowledge during training. It reports improved VGG accuracy and identifies semi-supervised learning as future work.
- About 4.96% accuracy improvement was achieved with the proposed method, compared with 3.17% for the referenced method on VGG.
- SVD and RBF define richer knowledge transfer without structural limitations on the network.
- Self-supervised learning with multi-task learning continuously receives teacher knowledge during training and may provide additional performance enhancement.
- Future work will extend the self-supervised knowledge-transfer concept toward a semi-supervised learning scheme.