PaperlayerView exact revision on arXiv

Source-linked AI summary

Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching

Mingi Ji, Byeongho Heo, Sungrae Park

arXiv:2102.02973v1cs.LG

TL;DR

Knowledge distillation needs effective links between teacher and student features, but manual selection can miss similarities and limit use of teacher knowledge. AFD learns similarities with an attention-based meta-network and applies them across all feature pairs, achieving better or comparable performance with more efficient computation across compression and transfer tasks.

  • Problem

    Manual teacher–student feature links may force incorrect intermediate processes, select only a few possible links, and leave effective intermediate-feature linking unresolved.

  • Method

    AFD uses an attention-based meta-network to learn teacher–student feature similarities and control distillation intensities for all possible feature pairs during student training.

  • Results

    AFD shows better or comparable performance across multiple model-compression and transfer-learning tasks with more efficient computation than the previous feature-linking approach.

  • Takeaways & Limitations

    AFD adaptively transfers teacher knowledge to multiple student-layer levels and adjusts feature levels across different teacher and student architectural styles.

Abstract

from arXiv · show

Knowledge distillation extracts general knowledge from a pre-trained teacher network and provides guidance to a target student network. Most studies manually tie intermediate features of the teacher and student, and transfer knowledge through pre-defined links. However, manual selection often constructs ineffective links that limit the improvement from the distillation. There has been an attempt to address the problem, but it is still challenging to identify effective links under practical scenarios. In this paper, we introduce an effective and efficient feature distillation method utilizing all the feature levels of the teacher without manually selecting the links. Specifically, our method utilizes an attention-based meta-network that learns relative similarities between features, and applies identified similarities to control distillation intensities of all possible pairs. As a result, our method determines competent links more efficiently than the previous approach and provides better performance on model compression and transfer learning tasks. Further qualitative analyses and ablative studies describe how our method contributes to better distillation. The implementation code is available at github.com/clovaai/attention-feature-distillation.

Introduction

Knowledge distillation transfers teacher knowledge to smaller student networks, but manually selected feature links may be ineffective and underuse teacher knowledge. AFD learns similarities across all teacher–student feature pairs with attention, improving link selection and performance across compression and transfer tasks.

  • Motivation: Knowledge distillation transfers knowledge from a pre-trained teacher network to a smaller student, supporting model compression and cross-domain transfer learning.The teacher provides both the solution and intermediate process of a problem.
  • Problem: Manual feature links can force incorrect intermediate processes and use only a few of the teacher–student pairs.These links do not account for teacher–student feature similarity.
  • Method: AFD uses an attention-based meta-network to identify feature similarities and control distillation intensity for all possible teacher–student pairs.The method is presented as an attention-based feature linking approach.
  • Novelty: Compared with L2T, AFD considers feature granularity and avoids the expensive inner-loop procedure required for classification-loss-based link learning.AFD learns from feature similarities, whereas L2T uses information from individual pairs and requires Hessian computation.
  • Results: AFD achieves better or comparable results across multiple tasks with more efficient computation, including image classification and domain-transfer experiments.Experiments cover CIFAR-100, tinyImageNet, ImageNet, CUB200, MIT67, Stanford40, and Stanford Dogs.

Related Work

Prior distillation work refines teacher knowledge or transfers relationships, but the problem of linking intermediate teacher and student features remains. AFD addresses this linking problem alongside L2T, using different link-identification properties.

  • Prior Approaches: Earlier methods transferred output distributions, intermediate representations, refined features, low-dimensional representations, or relational knowledge.Other approaches also modified distillation regularization to help students accept more teacher knowledge.
  • Open Problem: Intermediate-feature linking remains unresolved despite advances in refining and propagating teacher knowledge.The paper positions AFD as a solution to this problem, alongside L2T.

Attention-based Feature Distillation

AFD identifies similarities across all teacher–student feature pairs with an attention-based meta-network, then uses those similarities to weight feature-distance distillation. The student and attention network are trained jointly using the resulting distillation loss alongside classification loss.

  • Attention-based Feature Distillation: AFD evaluates all S × T teacher–student feature combinations and transfers teacher knowledge according to their identified similarities.Teacher and student candidates may differ in feature-map size and channel dimension.
  • Attention-based Feature Distillation: The attention similarities weight a distance computed from channel-wise averaged features, controlling the intensity of knowledge transfer for each feature pair.The meta-network uses globally pooled features for similarity estimation and channel-wise averaged features for distance calculation.
  • Attention-based Feature Distillation: Global average-pooled features generate queries and keys whose attention values represent relations between teacher and student candidates.Teacher features generate queries, while student features generate keys; feature-specific transition weights account for different feature levels.
  • Attention-based Feature Distillation: The feature-distance term compares normalized, channel-wise pooled teacher features with resized student features across teacher and student levels.Student features are up-sampled or down-sampled to match teacher feature-map sizes before distance computation.
  • Attention-based Feature Distillation: The total objective combines classification loss with the AFD loss, and the student and attention-based network are trained simultaneously.The trade-off parameter β controls the impact of the distillation term, while AFD is trained without Hessian computations tied to classification loss.

Experiments

Experiments evaluate AFD across model compression, large-scale datasets, transfer learning, feature-link behavior, sensitivity, and ablations. AFD generally improves or matches competing methods while adapting links to architecture and training conditions.

  • Experimental setup: AFD is evaluated on model compression using CIFAR-100, tinyImageNet, and ImageNet, plus transfer learning across four target-domain datasets.The experiments compare manually linked methods, L2T, and AFD under multiple architecture and task settings.
  • Model compression: On CIFAR-100, manually linked distillation baselines outperform vanilla students, and AFD improves over ATT in the reported model-compression comparisons.The experiments include same-style and different-style teacher–student architectures; the supplied results text does not provide the table values.
  • Model compression: AFD outperforms other baseline distillation methods on tinyImageNet and ImageNet, while L2T is omitted because updating its meta-network is too time-consuming.AFD also performs consistently better than ATT, which uses the same distillation loss with manually selected feature pairs.
  • Qualitative studies on feature attention: AFD changes and converges its feature links during training, adapting link patterns to same-style, different-style, and identical architectures.Observed patterns include low- and mid-level teacher links to low-level student features, smoothly connected mid-level features across different styles, and ordered links for identical architectures.
  • Qualitative studies on feature attention: AFD spreads teacher high-level features across multiple student levels while preserving feature order, unlike ATT’s ordered manual links and L2T’s switched-level links.The ablation study finds AFD performs best when trained jointly, and pretrained AFD links improve performance even when only the links are changed.

Conclusion

The paper proposes attention-based distillation that adaptively links teacher features to multiple student layers during training. It reports benefits for model compression and transfer learning, including architectural robustness and better performance than baselines.

  • Conclusion: AFD adaptively transfers teacher knowledge to multiple student layers through attention-map links learned during student training.The method identifies links between teacher and student features and transfers knowledge through those links.
  • Conclusion: AFD is learned simultaneously with student training, avoiding the additional inner-loop procedure required by the previous feature-linking method.This is presented as an efficiency advantage over the previous approach.
  • Conclusion: AFD adjusts student feature levels regardless of teacher and student architectural styles and provides better performance than baseline methods.The conclusion reports this result from further analysis.

Appendix A. Changes made since NeurIPS Submission

The appendix records revisions addressing reviewer feedback, including sensitivity analysis, metric and pooling comparisons, and supplementary fine-tuning results. It also reports an additional CRD–AFD experiment that improved accuracy over AFD alone.

  • Appendix A. Changes made since NeurIPS Submission: The revision adds sensitivity analysis for AFD’s hyperparameter β and discusses its experimental results.This analysis was added in the Sensitivity Analysis section.
  • Appendix A. Changes made since NeurIPS Submission: The revision analyzes distance metrics and channel-wise pooling methods in equation 3 to identify an optimal choice.The details are described in the corresponding Distance Metric and Pooling Methods section.
  • Appendix A. Changes made since NeurIPS Submission: 0.44pp accuracy improvement over the AFD-only model was obtained by combining CRD and AFD, although the result was omitted as orthogonal to the paper’s main work.The combined experiment was conducted but not added to the paper.
  • Appendix A. Changes made since NeurIPS Submission: Fine-tuning results for transfer learning were added to supplementary Appendix C, while the paper reports results under the L2T experimental conditions.The authors considered maintaining the L2T conditions more important for the main-paper comparison.

Appendix B. Implementation Details of Attention-based Feature Distillation

The implementation uses fixed query, key, and positional-encoding dimensions, average pooling for student-feature scaling, specified activations, architecture-dependent kernels and strides, and Xavier initialization.

  • Appendix B. Implementation Details of Attention-based Feature Distillation: The query, key, and positional-encoding dimension d is set to 128 for all experiments.The authors report that results change little with the choice of d.
  • Appendix B. Implementation Details of Attention-based Feature Distillation: Average pooling is used to scale the student feature hS_s.The implementation details specify average pooling for this scaling operation.
  • Appendix B. Implementation Details of Attention-based Feature Distillation: The query uses the identity activation and the key uses ReLU.Kernel size and stride are determined from the student and teacher feature sizes.
  • Appendix B. Implementation Details of Attention-based Feature Distillation: All AFD-network parameters are initialized with Xavier initialization.The implementation code is stated to be open-sourced.

Transfer Learning

The appendix supplements transfer-learning experiments with fine-tuning using ImageNet-pre-trained networks. AFD performs best among the compared distillation methods, although fine-tuning reduces method differences and remains partly similar to model compression.

  • Transfer Learning: The supplementary experiment fine-tunes networks pre-trained on ImageNet for the transfer-learning task.Table 8 reports accuracy for a ResNet34-to-ResNet18 transfer on MIT67.
  • Transfer Learning: AFD shows the best performance among the evaluated distillation methods with fine-tuned networks.The passage states that all distillation methods remain valid with fine-tuned transfer-learning networks.
  • Transfer Learning: Fine-tuning reduces the performance differences attributable to distillation methods compared with training from scratch.The appendix notes that fine-tuning is somewhat similar to model compression and calls for more research on their differences.

Appendix D. Additional Results

Additional ablations evaluate teacher–student linking methods across architectural styles, while qualitative analyses examine links produced for varied architecture pairs. AFD performs best when teacher and student architectural styles differ, and converged attention-based links guide feature-link selection.

  • Ablation studies: AFD shows the best performance when teacher and student architectural styles differ.The comparison covers various linking methods and architectural styles.
  • Ablation studies: Table 9 reports ablation studies on selecting teacher candidates and linking them to student features.All experiments are repeated 5 times.
  • Qualitative studies: Figure 6 compares manual links with converged attention-based links across various architecture pairs.The resulting links provide guidance for determining feature links between different architecture pairs.
Loading 2102.02973v1…