Source-linked AI summary
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi
TL;DR
Knowledge transfer across different network architectures remains challenging, and prior response-based methods do not directly target the activation boundaries that shape hidden-space partitions. The paper distills those boundaries with an activation-based loss and a differentiable approximation. Across varied experiments, the method is reported to outperform current state-of-the-art approaches, including ImageNet pre-training in many transfer-learning cases.
Problem
Knowledge transfer between different network architectures is challenging, while activation-boundary information has not been considered in the knowledge-transfer literature.
Method
The method transfers hidden-neuron activation status using an activation transfer loss and a differentiable hinge-like alternative that can be minimized by gradient descent.
Results
The proposed method outperforms current state-of-the-art methods across various knowledge-transfer experiments and performs better than ImageNet pre-training in many transfer-learning cases.
Takeaways & Limitations
Accurately transferring hidden-neuron activation boundaries can enhance knowledge-transfer performance within the evaluated settings.
Takeaways & Limitations
The method is described under the assumption that teacher and student hidden layers have the same size.
Abstract
from arXiv · showhide
An activation boundary for a neuron refers to a separating hyperplane that determines whether the neuron is activated or deactivated. It has been long considered in neural networks that the activations of neurons, rather than their exact output values, play the most important role in forming classification friendly partitions of the hidden feature space. However, as far as we know, this aspect of neural networks has not been considered in the literature of knowledge transfer. In this paper, we propose a knowledge transfer method via distillation of activation boundaries formed by hidden neurons. For the distillation, we propose an activation transfer loss that has the minimum value when the boundaries generated by the student coincide with those by the teacher. Since the activation transfer loss is not differentiable, we design a piecewise differentiable loss approximating the activation transfer loss. By the proposed method, the student learns a separating boundary between activation region and deactivation region formed by each neuron in the teacher. Through the experiments in various aspects of knowledge transfer, it is verified that the proposed method outperforms the current state-of-the-art.
Introduction
Knowledge transfer aims to improve a smaller student network using a previously trained teacher, but transferring knowledge across different architectures remains challenging. This paper transfers hidden-neuron activation boundaries rather than response magnitudes, using differentiable training losses and evaluating the approach across multiple experiments.
- Introduction: Knowledge transfer accelerates and improves a new student network by transferring knowledge from a previously learned teacher, especially when the architectures differ.Identical architectures permit parameter copying, whereas transfer between different architectures remains incompletely solved.
- Introduction: Hidden-layer responses contain substantial information, but their high dimensionality and non-linearity make perfect transfer difficult.
- Introduction: Activation boundaries are separating hyperplanes whose combinations form neural-network partitions and contribute to classification decision boundaries.This motivates transferring activation-boundary information between teacher and student networks.
- Introduction: The proposed method transfers whether hidden neurons are activated, using an activation transfer loss and a differentiable hinge-like approximation that can be optimized by gradient descent.The activation transfer loss ignores response magnitude, while the alternative loss enables gradient-based training.
- Introduction: Experiments compare the method with state-of-the-art algorithms across multiple knowledge-transfer settings, including transfer learning and further analyses.
Related Work
Prior knowledge-transfer methods mainly distill hidden-layer responses, often reducing spatial or channel information to handle low-level layers. These reductions can discard information because both dimensions matter.
- Related Work: ReLU activation divides the input space with activation boundaries, producing piecewise-linear partitions whose placement contributes to network complexity.
- Related Work: Response-transfer methods include full hidden-neuron matching, channel-correlation transfer, and spatial-attention transfer.The latter two approaches reduce spatial or channel information, respectively, to support low-level hidden-layer transfer.
- Related Work: Reducing hidden-response dimensions can lose information because both channel and spatial information are important.
Method
The method transfers hidden-neuron activation patterns and boundaries rather than response magnitudes, using a differentiable approximation suitable for student initialization and differing architectures.
- Transfer scheme: Knowledge transfer is performed during student initialization, after which classification training uses the cross-entropy loss independently.The paper distinguishes integrated training from initialization-based transfer and analyzes the transfer loss separately.
- Motivation: Mean square error between ReLU responses can leave activation boundaries different because it emphasizes large response differences over weak-versus-zero responses.This motivates focusing on whether neurons are activated rather than matching response magnitudes.
- Activation transfer loss: The activation transfer loss assigns a constant penalty when teacher and student activations differ, weighting activation states equally regardless of response magnitude.It aims to preserve teacher neuron activations and thereby transfer activation boundaries accurately.
- Differentiable approximation: Because the activation transfer loss is discrete, the alternative loss uses a hinge-loss-like, piecewise differentiable penalty that can be minimized by gradient descent.The alternative loss penalizes mismatched activations while using a margin for training stability.
- Architecture differences: When teacher and student layer widths differ, a connector function r: R^N → R^M converts the student response vector to the teacher's size during initialization.The connector and student are trained simultaneously, but the connector is discarded for classification.
- Convolutional extension: For convolutional layers, the connector is shared across spatial locations and implemented as a 1×1 convolution, requiring matching spatial sizes.Each spatial location is treated as an M-dimensional neuron-response vector.
Experiments
Experiments evaluate the proposed activation-boundary transfer method across learning speed, limited data, network compression, transfer learning, layer-wise transfer, activation similarity, and margin sensitivity. The method is reported to outperform competing approaches across these settings, with particularly strong results for short training, small datasets, and compressed networks.
- Learning speed: The proposed method outperformed other algorithms across all tested learning-speed settings, with larger performance gaps at shorter training durations.WRN22-4 was used as teacher and WRN16-2 as student; the teacher error rate was 4.51%.
- Small training data: The proposed method achieved the best performance for every tested training-data ratio, becoming more effective as the available data decreased.The experiments used WRN22-4 as teacher and WRN16-2 as student, with data ratios down to 0.1%.
- Network compression: The proposed method achieved the best performance across network-size configurations, while all transfer methods were more vulnerable to depth than channel compression.Its advantage was especially large for the “Tiny network” configuration.
- Transfer learning: On MIT scenes and CUB 2011 transfer learning, the proposed method outperformed other methods in all reported cases, with larger gains as training samples decreased.Classifier performance was reported as test-set accuracy, and the datasets used ImageNet pre-training or random initialization as relevant comparisons.
- Layer-wise analysis: In layer-wise analysis, the proposed method achieved the best performance in every tested transfer configuration, unlike other methods whose effectiveness varied by layer.The analysis compared low-, mid-, and high-level representatives and multiple-layer transfer.
- Additional analysis: The alternative loss was evaluated against l1, l0.5, and l2 losses for activation similarity, while margin ablations found little performance difference once the margin was sufficiently large.Activation similarity was measured by the percentage of neurons sharing teacher and student activation states.
Conclusion
The paper concludes that transferring hidden-neuron activation boundaries enhances knowledge transfer and accurately transfers those boundaries. The method outperforms current state-of-the-art approaches, including ImageNet pre-training in many transfer-learning cases.
- Transferring activation boundaries can greatly enhance knowledge-transfer performance.
- The proposed method accurately transfers the activation boundaries of hidden neurons.
- The proposed method achieves much higher performance than current state-of-the-art methods.
- In many transfer-learning cases, the method performs better than ImageNet pre-training.
- The results suggest that activation boundaries are powerful for knowledge transfer and may alter established neural-network practices.