Source-linked AI summary
Similarity-Preserving Knowledge Distillation
Frederick Tung, Greg Mori
TL;DR
The paper addresses how to transfer teacher knowledge without requiring the student to mimic the teacher’s representation space. It introduces similarity-preserving distillation, which preserves pairwise activation similarities and shows potential across three public datasets, including transfer learning under domain shift.
Problem
Knowledge distillation needs losses that capture and transfer teacher knowledge when direct representation imitation or access to privileged source-domain data is restricted.
Method
Similarity-preserving distillation trains the student to preserve pairwise similarities between activation patterns produced by input images in the teacher network.
Results
Experiments on three public datasets show potential for improving student training outcomes over data supervision alone, complementing traditional distillation methods, and addressing domain shift in transfer learning.
Takeaways & Limitations
Pairwise activation similarities provide a supervisory signal that can transfer useful teacher knowledge without requiring the student to mimic the teacher’s representation space.
Takeaways & Limitations
Applications of similarity-preserving distillation to semi-supervised and omni-supervised learning remain future work.
Abstract
from arXiv · showhide
Knowledge distillation is a widely applicable technique for training a student neural network under the guidance of a trained teacher network. For example, in neural network compression, a high-capacity teacher is distilled to train a compact student; in privileged learning, a teacher trained with privileged data is distilled to train a student without access to that data. The distillation loss determines how a teacher's knowledge is captured and transferred to the student. In this paper, we propose a new form of knowledge distillation loss that is inspired by the observation that semantically similar inputs tend to elicit similar activation patterns in a trained network. Similarity-preserving knowledge distillation guides the training of a student network such that input pairs that produce similar (dissimilar) activations in the teacher network produce similar (dissimilar) activations in the student network. In contrast to previous distillation methods, the student is not required to mimic the representation space of the teacher, but rather to preserve the pairwise similarities in its own representation space. Experiments on three public datasets demonstrate the potential of our approach.
1. Introduction
Knowledge distillation transfers teacher knowledge to student networks, but the best way to define the distillation loss remains open. The paper introduces similarity-preserving distillation, which transfers pairwise activation similarities rather than requiring the student to mimic the teacher’s representation space.
- Motivation: Resource constraints and data-privilege or privacy issues motivate transferring knowledge from trained networks to deployable students.Mobile and autonomous systems have limited memory and power, while transfer learning may restrict access to source-domain training data.
- Background: Knowledge distillation supervises a student with an additional loss that captures some aspect of a trained teacher’s richer signal.The technique was motivated by neural-network compression and has also been applied to privileged learning, adversarial defense, and noisy-data settings.
- Open question: How to define a distillation loss that best captures teacher knowledge remains an open question.Prior approaches transfer softened class scores, intermediate-layer hints, or inter-layer feature transformations.
- Proposed approach: Similarity-preserving distillation trains the student to preserve pairwise similarities among activation patterns elicited by input mini-batches.For a batch of b images, the method forms b × b similarity matrices from student and teacher activation maps and defines the distillation loss on those matrices.
- Motivation: The method is motivated by activation patterns that are consistent within object categories and distinctive across different categories.Figure 2 visualizes channel-wise average activations from the last convolutional layer of a WideResNet-16-2 teacher on CIFAR-10 test images.
- Evidence: Experiments on three public datasets show the approach’s potential for improving student training outcomes and complementing traditional distillation methods.The paper presents these findings as experimental validation of similarity-preserving knowledge distillation.
2. Method
Knowledge distillation trains a computationally cheaper student under teacher supervision, but the best way to capture teacher knowledge remains open. The proposed method transfers pairwise activation similarities, allowing the student to preserve relational structure without mimicking the teacher’s representation space.
- Motivation: Knowledge distillation adds teacher supervision to train a cheaper student network, beyond ordinary ground-truth labels.The student may be shallower, thinner, or use cheaper operations.
- Prior approaches: Traditional distillation transfers softened class scores, while other methods match intermediate features or inter-layer transformations.These approaches supervise the student by mimicking selected aspects of the teacher’s representation or computation.
- Similarity-preserving distillation: Similarity-preserving distillation uses pairwise activation similarities within each mini-batch as the teacher-derived supervisory signal.Semantically similar inputs tend to produce similar activation patterns, motivating preservation of these relationships.
- Similarity-preserving distillation: Teacher and student activation maps may differ in channels and spatial dimensions, with corresponding layers selected by depth or block position.The method compares activations from layers at the same depth or at the ends of corresponding blocks.
- Similarity-preserving distillation: The method forms L2-normalized activation outer products and penalizes their differences across selected teacher–student layer pairs.The loss sums mean element-wise squared differences between the resulting similarity matrices, using the Frobenius norm.
- Training objective: The total student-training loss combines cross-entropy with the similarity-preserving loss, weighted by balancing hyperparameter γ.Figure 3 illustrates class-structured activation-similarity matrices and the objective of making the student’s matrices closer to the teacher’s.
- Difference from previous approaches: Unlike prior representation-matching methods, the student need only preserve teacher pairwise similarities, making the behavior unchanged by rotation of the teacher representation space.The student is not required to express the teacher’s representation space directly.
3. Experiments
Experiments on CIFAR-10, transfer learning, and CINIC-10 evaluate similarity-preserving distillation across student-teacher architectures and show consistent improvements, favorable baseline comparisons, and complementarity with attention transfer.
- CIFAR-10: On CIFAR-10, similarity-preserving distillation reduced median error by 0.5 to 1.2 absolute percentage points, or 7% to 14% relative, versus conventional training without extra parameters or operations.The gains held across tested pairs with matching or differing depth and width.
- CIFAR-10: Similarity-preserving distillation achieved the lowest error in four of five CIFAR-10 student-teacher cases against traditional softened-score and attention-transfer baselines.This result supports using activation similarities across images as a supervisory signal.
- CIFAR-10: LSP and performance were correlated across CIFAR-10 experiments using students trained with traditional knowledge distillation, attention transfer, and similarity-preserving distillation.The comparison covered three student-teacher pairings.
- CIFAR-10: A 5× compression distilled a 11.0M-parameter WideResNet-16-8 teacher into a 2.2M-parameter WideResNet-40-2 student with only 0.3% accuracy loss.The experiment used off-the-shelf PyTorch without specialized hardware or software.
- Design choices: Using activations only from final convolution layers outperformed using activations at the ends of all WideResNet blocks, while post-softmax similarities performed worse.The authors associate deeper activations with more specialized and distinctive semantic features.
- Transfer learning combining distillation with fine-tuning: In describable-textures transfer learning, distillation reduced error by 1.0% to 1.3% absolute across tested MobileNet and MobileNetV2 pairs versus fine-tuning alone.Similarity-preserving distillation also outperformed spatial attention transfer on 19 of 30 MobileNet splits and 24 of 30 MobileNetV2 splits.
4. Related Work
Related work spans knowledge-distillation paradigms, network-compression methods, and alternative student–teacher training designs. These approaches differ in how they capture teacher knowledge, alter architectures, or structure training.
- Knowledge distillation: Knowledge distillation captures and transfers knowledge from trained teacher networks to student networks, while related work also explores alternative training paradigms.
- Alternative training paradigms: Alternative distillation systems add a discriminator or train a native ensemble teacher jointly with multiple student branches.
- Experimental comparisons: Additional experiments compare students and teachers from different architecture families on CIFAR-10 using median error over five runs.
- Network compression: Compression methods include weight pruning, which removes unimportant weights and induces structured or unstructured sparsity in network connectivity.
- Network compression: State-of-the-art compression can reduce network size by an order of magnitude, but practical acceleration often requires specialized software, hardware, simulators, or measurements.
5. Conclusion
The paper proposes similarity-preserving knowledge distillation, which transfers teacher knowledge by preserving pairwise similarities in the student representation space. It reports benefits for student training, transfer learning under domain shift, and complementarity with attention transfer, while identifying unlabeled-data settings as future work.
- Similarity-preserving distillation preserves pairwise similarities in the student representation space instead of mimicking the teacher’s representation space.
- Experiments demonstrate improved student training outcomes over data supervision alone and a robust solution for transfer learning under domain shift.
- The method complements attention transfer and captures teacher knowledge not fully encoded in spatial attention maps.
- Future work will explore semi-supervised and omni-supervised settings using auxiliary unlabeled images for the distillation loss.