Source-linked AI summary
Paraphrasing Complex Network: Network Compression via Factor Transfer
Jangho Kim, SeongUk Park, Nojun Kwak
TL;DR
High-performance DNNs require substantial computation and storage, limiting their use in resource-constrained embedded systems. The paper proposes factor transfer, using a paraphraser and translator to make teacher knowledge more understandable to the student, and reports enhanced student-network performance, including a 1.34% reduction in ResNet-18 Top-1 error on ImageNet.
Problem
High-performance DNNs require substantial computational power and storage, making them difficult to deploy in resource-constrained embedded systems.
Method
Factor transfer uses an unsupervised paraphraser to extract teacher factors and a convolutional translator to extract and match student factors.
Results
The proposed approach enhances student-network performance and consistently outperforms AT and KD across the evaluated network types.
Takeaways & Limitations
Matching paraphrased factors helps relieve differences between teacher and student networks while translating teacher knowledge for the student.
Takeaways & Limitations
Factor transfer requires training a paraphraser and additional paraphraser parameters; teacher factors are extracted from the last group of feature maps.
Abstract
from arXiv · showhide
Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.
1 Introduction
The paper targets resource-intensive DNNs by proposing factor transfer, which paraphrases teacher knowledge into factors and translates them for the student. It reports improved student performance over conventional knowledge-transfer methods.
- Motivation: High-performance DNNs require substantial computation and storage, limiting their use in resource-constrained embedded systems.
- Problem: Conventional knowledge-transfer methods directly transfer softened teacher distributions or attention maps, despite structural and channel differences between teacher and student networks.
- Proposed Method: Factor transfer introduces teacher and student factors to provide transportable information rather than directly matching network outputs.
- Proposed Method: The method uses an unsupervised paraphraser to extract meaningful teacher factors and a convolutional translator on the student side to learn them.
- Results: Experiments show that factor transfer improves student-network performance over conventional knowledge-transfer methods.
2 Related Works
Related work compresses DNNs through pruning, quantization, efficient architectures, and knowledge transfer. Prior transfer methods include softened class distributions, intermediate representations, and attention-based signals, while convolutional autoencoders motivate unsupervised factor extraction.
- Compression Methods: DNN compression methods include network pruning, weight quantization, efficient small architectures, and knowledge transfer.
- Knowledge Transfer: Knowledge distillation transfers teacher information through softened class distributions, while FSP and FitNet transfer intermediate feature representations.
- Knowledge Transfer: FitNet directly mimics teacher activation maps, which can be problematic when teacher and student capacities differ.
- Knowledge Transfer: Attention transfer uses attention maps derived from feature maps as an alternative knowledge-transfer signal.
- Unsupervised Feature Extraction: The paraphraser resembles a convolutional autoencoder trained with reconstruction loss to learn compact, meaningful features from teacher feature maps.
3 Proposed Method
The method extracts teacher factors with an unsupervised paraphraser, then trains a student-side translator to produce matching student factors. Factor transfer combines factor matching with classification training so the student can absorb the teacher’s knowledge through rephrased features.
- Teacher Factor Extraction: Factor Transfer first trains a paraphraser with reconstruction loss to extract teacher factors from teacher feature maps.The paraphraser produces factors from the teacher network’s feature maps before student training.
- Teacher Factor Extraction: The paraphraser maintains spatial dimensions while adjusting factor channels because it operates on sufficiently downsampled feature maps from the teacher’s last group.The last group is selected because its trained features are described as containing enough task information.
- Factor Transfer with Translator: A translator inserted after the student’s last group rephrases student feature maps and is trained jointly with the student network.The translator acts as a buffer between the student and the teacher’s output rather than requiring direct output matching.
- Factor Transfer with Translator: Student training combines classification loss and factor-transfer loss, weighted by β, in an end-to-end training procedure.The classification term is cross entropy between ground-truth labels and the student’s softmax output.
- Factor Transfer with Translator: The translator outputs student factors that mimic teacher factors, with matching factor dimensions and l2 normalization.The reported experiments use l1 factor-transfer loss, while the difference from l2 loss is described as minor.
4 Experiments
The experiments evaluate factor transfer across image-classification and object-detection settings, architectures, and knowledge-transfer combinations. Factor transfer consistently outperforms AT and KD across tested network types, improves ImageNet and Faster-RCNN performance, while some hybrid and grouped variants show limitations.
- CIFAR-10 results: FT consistently achieved better performance than AT and KD across CIFAR-10 network types.The comparison used the best FT result among experiments with k values from 0.5 to 4.
- CIFAR-10 results: AT+KD outperformed standalone AT or KD in every tested CIFAR-10 case, while adding KD to FT reduced FT’s benefit in some cases.The results indicate that the two hybrid combinations behaved differently across the tested settings.
- Ablation and component analysis: The grouped FT loss produced 7.01% classification error, 0.06% higher than the single FT loss, so experiments used a single FT loss.The spatial-dimension-preserving paraphraser also outperformed CAE- and RAE-based paraphrasers.
- Transfer representation: FT produced better accuracy improvement than full-activation transfer, supporting paraphrased information over directly transferring full activation maps.This comparison used FitNets-style hints and full feature maps.
5 Conclusion
The paper proposes factor transfer, which uses paraphrased teacher information and a translator to make teacher knowledge more understandable and transferable to the student. It reports effectiveness across image-classification datasets and beyond classification, while requiring extra training and temporary parameters.
- Factor transfer extracts paraphrased teacher information as factors using a paraphraser.
- A student-side translator helps the student understand teacher factors by mimicking them.
- The method can relieve inherent differences between teacher and student networks during knowledge transfer.
- Factor transfer requires training a paraphraser and additional paraphraser and translator parameters, but these parameters are unnecessary after student training.The paraphraser's training converges very fast.
- Experiments showed factor transfer was effective across various image-classification datasets and applicable to domains beyond classification.
Appendix A Implementation details
The implementation uses compact convolutional paraphrasers, dataset-specific training choices, and factor visualizations to examine training behavior and task-specific representations.
- The paraphraser uses three convolutional and three transposed-convolutional layers with 3 × 3 kernels, stride 1, padding 1, batch normalization, and leaky-ReLU rate 0.1.Spatial height and width are preserved; the second convolution changes the number of output feature maps.
- For most datasets, the paraphraser omits batch normalization, while CIFAR-10 retains it; using l2 loss without batch normalization requires learning-rate tuning to avoid exploding gradients.
- On CIFAR-10 and CIFAR-100, paraphraser training lasted at most 30 epochs from learning rate 0.1 because validation loss converged within a few epochs.Training too long slightly diminished student-network performance.
- Using last-group feature maps lets the paraphraser extract task-specific teacher factors because higher-level features are more class-specific.
- The student used learning rate 0.1 with decays at 32,000 and 48,000 iterations, stopping at 64,000 iterations, with SGD and mini-batch size 128.Weight decay was 5 × 10^-4 and momentum was 0.9 on one Titan Xp.
- The paper consistently used l1 FT loss because its performance difference from l2 loss was minor.
Appendix B Details of Convolutional autoencoders
The appendix describes convolutional autoencoder variants used alongside the paraphraser, including a six-layer CAE and an l1-regularized RAE.
- The convolutional autoencoder has three convolutional and three transposed-convolutional layers using 4 × 4 kernels, stride 2, padding 1, batch normalization, and leaky-ReLU rate 0.1.Convolutions reduce spatial dimensions and double the number of channels.
- The regularized autoencoder uses the paraphraser architecture with paraphrase rate k = 2.
- The regularized autoencoder applies an l1 penalty to teacher factors extracted by the paraphraser.The stated motivation is that l1 regularization produces sparse coefficients and can be robust to irrelevant features.
Appendix C Training Curves on Datasets
Training curves compare AT, KD, FT, and a basic student on CIFAR-10 and CIFAR-100, and report Top-1 and Top-5 error curves for ResNet-18 on ImageNet.
- Figure 6 compares training curves for AT, KD, FT, and the basic student on CIFAR-10 and CIFAR-100.
- Figure 7 presents ResNet-18 training curves on ImageNet using Top-1 and Top-5 error.Top-1 error is failure to predict the correct answer; Top-5 error is failure to include it among the five highest softmax classes.