Source-linked AI summary
Relational Knowledge Distillation
Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho
TL;DR
Knowledge distillation traditionally trains smaller students to mimic individual teacher outputs, but the paper asks whether relations among examples better represent transferable knowledge. It introduces relational knowledge distillation with distance-wise and angle-wise losses, and finds improved students across tasks, including metric-learning students that can outperform teachers. The approach is most effective within the training domain and is not suitable alone when individual output values are crucial, such as classification softmax outputs.
Problem
Traditional knowledge distillation transfers individual teacher outputs, motivating a method that instead captures relations among data examples and learned representations.
Method
Relational knowledge distillation trains students to match teacher relational structures using distance-wise and angle-wise losses, with relational potentials supporting higher-order properties.
Results
RKD significantly improves student performance across metric learning, image classification, and few-shot learning, with metric-learning students sometimes outperforming their teachers.
Takeaways & Limitations
RKD provides a general framework for transferring structural relations and can complement conventional or task-specific distillation methods.
Takeaways & Limitations
RKD can sacrifice generalization to domains different from the training domain and should be combined with individual-output or task-specific losses when individual values are crucial.
Abstract
from arXiv · showhide
Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.
POSTECH
The passage identifies Yan Lu as affiliated with Microsoft Research.
- Yan Lu is affiliated with Microsoft Research.
1. Introduction
The paper addresses the computational cost of large neural networks by proposing relational knowledge distillation, which transfers structural relations among learned representations rather than individual outputs. RKD uses distance-wise and angle-wise losses and improves student models across multiple tasks.
- Large state-of-the-art neural networks impose substantial inference computation and memory costs.
- Conventional knowledge distillation transfers knowledge by training students to match teacher outputs for individual examples.
- RKD transfers relations among learned representations, treating structural information in the embedding space as central knowledge.
- RKD instantiates this idea with distance-wise second-order and angle-wise third-order distillation losses.
- Experiments on metric learning, image classification, and few-shot learning significantly improve student-model performance.
2. Related Work
Related work covers model compression, self-distillation, broader learning settings, and task-specific distillation methods. RKD extends relational knowledge transfer beyond prior metric-learning-specific approaches into a general framework evaluated across tasks.
- Early knowledge-distillation work compressed ensembles or larger models into smaller neural networks for model compression and accuracy improvement.
- Self-distillation studies show that students with identical architectures can improve over teachers, often across generations using teacher outputs.
- Knowledge distillation has also been extended to unsupervised, semi-supervised, multitask, and omni-supervised learning settings.
- Task-specific distillation methods address object detection, face compression, image retrieval, and person re-identification.
- Prior metric-learning work transfers image similarities through a rank loss, whereas RKD introduces a general relational framework applicable to multiple tasks.
3. Our Approach
Relational knowledge distillation (RKD) transfers structural relations among examples rather than individual teacher outputs. It formulates relational potentials over tuples and instantiates them with distance-wise and angle-wise losses, optionally combined with task-specific or conventional distillation losses.
- 3.1. Conventional knowledge distillation: Conventional knowledge distillation (IKD) trains the student to match individual teacher outputs using a loss that penalizes their differences.Examples include pre-softmax outputs with temperature-scaled softmax and Kullback–Leibler divergence, or hidden activations with squared Euclidean distance and a dimension-matching linear map.
- 3.2. Relational knowledge distillation: RKD computes a relational potential for each tuple of examples and trains the student to reproduce the teacher’s relational structure.The potential measures relational energy, enabling transfer of higher-order properties that can remain invariant to lower-order properties and output-dimension differences.
- 3.2. Relational knowledge distillation: RKD generalizes IKD because its objective reduces to conventional KD when the relation is unary (N = 1) and the potential function ψ is the identity.The relational potential and loss compare teacher and student representations through the selected relation rather than directly matching outputs.
- 3.2.1. Distance-wise distillation loss: Distance-wise distillation measures pairwise Euclidean distances and penalizes differences between teacher and student distance structures.Distances are normalized by the mini-batch average, which is useful when teacher and student distance scales differ; the authors observed more stable and faster convergence with this normalization.
- 3.2.2. Angle-wise distillation loss: Angle-wise distillation measures angles formed by triplets and penalizes angular differences using the Huber loss.The authors report that this higher-order relation often allows faster convergence and better performance, while potentially giving the student more flexibility during training.
- 3.2.3. Training with RKD: RKD losses can be used alone or with task-specific losses, with λ_KD balancing the task and distillation objectives.When multiple KD losses are used, each receives its own balancing factor; RKD is not adequate alone when individual output values are crucial, such as softmax outputs for classification.
4. Experiments
Experiments evaluate RKD across metric learning, classification, and few-shot learning, with detailed metric-learning studies comparing student models, baselines, self-distillation, and state-of-the-art methods. RKD improves student performance, can exceed teacher performance, complements other distillation methods, and strongly adapts to the training domain at the cost of cross-domain generalization.
- RKD is evaluated on metric learning, classification, and few-shot learning, with comparisons against FitNet, Attention, HKD, and DarkRank.The distance-wise, angle-wise, and combined variants are denoted RKD-D, RKD-A, and RKD-DA.
- Metric learning: Recall@1 measures the average fraction of test queries whose top K retrieved neighbors include the query’s category.
- Metric learning: RKD significantly improves student retrieval performance over direct Triplet training and outperforms DarkRank, with relative Recall@1 gains reaching 27.7 on CUB-200-2011 and 45.5 on Cars 196.RKD-DA benefits from training without ℓ2 normalization, and a ResNet18-128 student reaches 82.50 on Cars 196 versus 77.17 for the ResNet50-512 teacher.
- Self-distillation: Self-distilled models with the same architecture as their teachers consistently outperform initial Triplet-trained teachers, but performance does not improve beyond the second generation.
- Metric learning: State-of-the-art comparisons show RKD-DA is best on CUB-200-2011 across backbones and best among ResNet50 methods on all benchmark datasets, while ranking second on two GoogLeNet benchmarks.
- Discussion: Training RKD-DA on Cars 196 yields much lower Recall@1 on other domains than on the training domain, unlike the Triplet teacher.The reported pattern indicates stronger training-domain adaptation accompanied by reduced cross-domain generalization.
- Image classification: RKD-DA outperforms all compared configurations on CIFAR-100 and Tiny ImageNet when combined with HKD, indicating complementarity with other distillation methods.
5. Conclusion
RKD transfers knowledge through mutual relations among data examples and demonstrates effective knowledge transfer across tasks and benchmarks. In metric learning, it can enable smaller students to outperform larger teachers while supporting further exploration of higher-order relational losses.
- RKD effectively transfers knowledge using mutual relations of data examples across different tasks and benchmarks.
- In metric learning, RKD enables smaller students to outperform their larger teachers.
- The RKD framework supports task-specific losses with high-order potentials beyond the distance-wise and angle-wise instances used here.