PaperlayerView exact revision on arXiv

Source-linked AI summary

Class Attention Transfer Based Knowledge Distillation

Ziyao Guo, Haonan Yan, Hui Li, Xiaodong Lin

arXiv:2304.12777v1cs.CV

TL;DR

Prior knowledge distillation methods perform well but leave unclear how transferred knowledge improves student networks. This paper proposes CAT-KD, which transfers class activation maps to strengthen class-discriminative-region identification, and reports state-of-the-art performance on multiple benchmarks with high interpretability.

  • Problem

    Prior knowledge distillation methods provide limited explanation of how transferred knowledge improves the student network, especially for attention transfer.

  • Method

    CAT-KD transfers class activation maps to improve the student network’s capacity to identify class-discriminative input regions.

  • Results

    CAT-KD achieves state-of-the-art performance on multiple benchmarks while having high interpretability.

  • Takeaways & Limitations

    Transferring CAMs can obtain and enhance the class-discriminative-region identification capacity considered critical for CNN classification, while supporting interpretable distillation.

  • Takeaways & Limitations

    CAT transfers CAMs for all categories and normalizes them so category information is not released to the trained model.

Abstract

from arXiv · show

Previous knowledge distillation methods have shown their impressive performance on model compression tasks, however, it is hard to explain how the knowledge they transferred helps to improve the performance of the student network. In this work, we focus on proposing a knowledge distillation method that has both high interpretability and competitive performance. We first revisit the structure of mainstream CNN models and reveal that possessing the capacity of identifying class discriminative regions of input is critical for CNN to perform classification. Furthermore, we demonstrate that this capacity can be obtained and enhanced by transferring class activation maps. Based on our findings, we propose class attention transfer based knowledge distillation (CAT-KD). Different from previous KD methods, we explore and present several properties of the knowledge transferred by our method, which not only improve the interpretability of CAT-KD but also contribute to a better understanding of CNN. While having high interpretability, CAT-KD achieves state-of-the-art performance on multiple benchmarks. Code is available at: https://github.com/GzyAftermath/CAT-KD.

1. Introduction

The paper proposes CAT-KD to make knowledge distillation more interpretable by transferring class attention, while showing that class-discriminative-region identification is central to classification and that CAT-KD performs competitively.

  • Knowledge distillation transfers information from larger teacher networks to smaller students, but prior logit- and feature-based methods provide limited explanation of how transferred knowledge improves performance.
  • Class activation maps identify input regions that are discriminative for specific categories, enabling attention transfer to tell students where to focus during classification.
  • CAT-KD applies class attention transfer to knowledge distillation, targeting improved student performance through stronger class-discriminative-region identification.
  • Class attention transfer demonstrates that identifying class-discriminative regions can be obtained and enhanced by transferring CAMs.
  • Experiments reveal properties of CAM transfer that improve CAT-KD’s interpretability and contribute to understanding CNNs.
  • CAT-KD achieves state-of-the-art performance on multiple benchmarks while retaining high interpretability.

2. Background

The background reviews knowledge distillation and attention-based transfer, emphasizing that prior attention maps lack an established classification role and explanatory account for their effectiveness.

  • Knowledge distillation improves smaller student networks by transferring knowledge from larger teacher networks through logits, features, or attention.
  • Prior attention transfer defines attention as a spatial map of where the model focuses, but does not explain its classification role or effectiveness.
  • Class activation maps use high-level features and fully connected-layer parameters to highlight class-discriminative input regions for specific categories.
  • The paper addresses the unresolved role of attention in classification by proposing an interpretable attention-based distillation method with competitive performance.

3. Our Method

The method converts CNN classification into class activation map generation followed by spatial averaging, showing that class-discriminative localization can be transferred to students through CAT and CAT-KD.

  • 3.1. Revisit the structure of CNN: Mainstream CNNs can be converted so class logits equal the global average activation of corresponding class activation maps.The conversion uses a 1×1 convolutional representation of the fully connected layer without changing prediction scores.
  • 3.2. Class Attention Transfer: Class attention transfer trains a model to mimic normalized teacher CAMs, without exposing category labels or logits.The CAT objective averages squared distances between normalized, optionally downsampled teacher and student CAMs across categories.
  • 3.2. Class Attention Transfer: CAT experiments show that transferring CAMs can obtain and enhance the capacity to identify class-discriminative regions of input.The experiments also find that CAMs from all categories contain beneficial information.
  • 3.2. Class Attention Transfer: Smaller transferred CAMs perform better, and their critical information lies in high-activation regions rather than specific activation values.These findings characterize which aspects of CAM transfer are useful for CAT.
  • 3.3. CAT-KD: CAT-KD combines cross-entropy and CAT losses, using β to balance the two objectives while improving student localization capacity.The method explains the transferred knowledge through its effect on identifying class-discriminative regions.

4. Experiments

Experiments show that transferring CAMs can train models to identify class-discriminative regions, with benefits across CAT analyses and CAT-KD benchmarks. CAT-KD remains competitive across CIFAR-100, ImageNet, transferability, data-efficiency, and computational-cost evaluations.

  • Exploration of CAT: CAM transfer trains models with high classification accuracy, indicating that it conveys capacity to identify class-discriminative regions.The experiments use CAT without category information such as ground-truth labels or logits.
  • Exploration of CAT: CAMs from all categories provide beneficial information, including categories selected by low prediction scores.Higher-scoring categories bring more improvement, but lower-scoring categories also help CAT.
  • Exploration of CAT: CAT enables classification of categories absent from the reduced training set when CAMs for all classes are transferred.Accuracy is evaluated on the complete test set and on a subset containing classes excluded from training.
  • Exploration of CAT: Smaller, pooled CAMs perform better because pooling reduces inter-model localization bias and expands highlighted regions toward broader discriminative areas.The authors generally pool transferred CAMs to 2×2 during CAT and CAT-KD.
  • Exploration of CAT: Binarizing CAMs causes an accuracy drop of less than one percent, showing that spatial location matters more than exact CAM values.The binarization retains regions above the average-value threshold as activated.
  • Evaluation of CAT-KD: On CIFAR-100, CAT-KD outperforms attention-based AT by 1.07% ∼ 12.78% and matches or exceeds feature-based distillation in the reported comparisons.Feature-based methods require additional networks and multiple-layer information.
  • Evaluation of CAT-KD: CAT-KD outperforms most KD methods on ImageNet despite performance being restricted by weak teacher networks.The ImageNet evaluation reports both top-1 and top-5 accuracy.
  • Efficiency: CAT-KD is minimally affected by reduced training data, while feature-based methods require substantially more computational resources.These findings come from data-reduction and training-cost comparisons.

5. Conclusion

The paper concludes that CAT-KD combines interpretability with competitive performance by transferring CAMs to build class-discriminative localization capacity. Its CAM-transfer properties also support broader understanding of CNNs.

  • Conclusion: CAT-KD combines high interpretability with competitive performance while transferring CAMs to improve class-discriminative localization capacity.The paper presents this capacity as critical for CNN classification.
  • Conclusion: The reported CAM-transfer properties are intended to support future research on CNN interpretability and knowledge distillation.The conclusion frames these properties as contributions to understanding CNNs.

A.1. Cross-entropy loss and CAT loss

The appendix compares cross-entropy training with CAM transfer and shows that CAT’s effectiveness depends on the relative accuracy of the CAM-producing teacher.

  • Cross-entropy loss and CAT loss: Class-discriminative localization capacity can be obtained either through cross-entropy training or by transferring CAMs.Raw training provides only hard labels, whereas CAT transfers class-discriminative spatial information.
  • Cross-entropy loss and CAT loss: When the CAM producer is stronger than the trained model, CAM-only transfer can outperform training from scratch.When the producer is weaker, CAM transfer is less effective than using cross-entropy loss alone.

A.2. Guidance for balancing CE loss and CAT loss

The CAT and cross-entropy losses must be balanced according to the relative strength of teacher and student networks. Transferred CAMs help most when the teacher is substantially stronger.

  • Guidance for balancing CE loss and CAT loss: The factor β balances CAT loss and cross-entropy loss in CAT-KD.The appendix presents β as the tuning factor governing the combination of the two losses.
  • Guidance for balancing CE loss and CAT loss: Transferred CAMs provide more improvement when the teacher is much stronger than the student, but may be less beneficial otherwise.This guidance is supported by evaluations using different teachers and β values.

A.3. Normalization in CAT-KD

Normalization of transferred CAMs is unnecessary in CAT-KD overall, but its effect depends on whether teacher and student architectures differ. It helps with different architectures and harms with similar ones.

  • Normalization benefits CAT-KD when teacher and student architectures differ, according to the reported comparison.The comparison is reported in Table 14.
  • Normalization harms CAT-KD when teacher and student architectures are similar.The comparison is reported in Table 13.
  • CAT-KD does not require l2 normalization of transferred CAMs, unlike the CAT procedure described in the passage.For CAT, l2 normalization prevents category-indicating information from being released to the trained model.

A.4. Extensions

The extensions examine alternative CAM generation and coefficient choices. CAT can use CAMs generated by GradCAM, while equally weighting all category CAMs avoids extra implementation complexity without reported benefit from preferential weighting.

  • Transfer CAMs generated by other methods: CAT remains evaluated when transferred CAMs are generated by GradCAM rather than the paper’s original CAM-generation approach.The experiment uses CIFAR-10 and tests whether CAT remains effective with generalized CAM generation.
  • Transfer CAMs generated by other methods: CAMs generated by alternative methods correctly highlight target-class discriminative regions but consume substantially more resources.The passage attributes this property to the alternative CAM-generation methods considered.
  • Coefficients in CAT loss: Preferentially transferring CAMs for higher-scoring categories brings little benefit to CAT and CAT-KD while increasing implementation complexity and cost.The paper therefore does not prioritize categories using their prediction scores.
  • Coefficients in CAT loss: CAT and CAT-KD transfer CAMs from all categories with equal coefficient 1/k.This design is specified in Eqn (5).

A.5. More implementation details.

The paper specifies consistent CAM pooling, data-reduction protocols, visualization settings, hardware, and training details across its experiments. These details include dataset-specific CAM resolutions and fixed evaluation configurations.

  • General settings: Transferred CAMs are pooled to 2×2 by default during CAT and CAT-KD.Experiments in Section 4.2 instead use batch size 128 while retaining the other Section 4.1 settings.
  • Hardware setup: CIFAR-100 experiments use one RTX 2080 Ti GPU, whereas ImageNet experiments use four RTX 2080 Ti GPUs.The server also includes an Intel Xeon Silver 4214 CPU and 128GB RAM.
  • Visualization: All paper visualizations are generated by ResNet50, which achieves 76.16% accuracy on ImageNet.
  • CAM resolution: CAM resolution is 8×8 for most CIFAR-100 models and 7×7 for ImageNet, with several CIFAR-100 architectures using lower resolutions.ShuffleNet, ResNet50, and VGG use 4×4, while MobileNet uses 2×2 on CIFAR-100.
  • Figure 4 settings: For Figure 4, the reduced training set contains only n categories, specifically the first n categories in CIFAR-100’s default order.
  • Table-specific details: Table 3 binarizes transferred CAMs before normalization.
  • Table-specific details: The Table 4 teacher is a TrivialAugment-trained ResNet32×4 with 81.36% CIFAR-100 validation accuracy, while compared methods use author-released code and unchanged hyperparameters.
  • Figure 7 settings: Figure 7 reduces each category’s training data by the same proportion and measures per-epoch training time on one RTX 2080 Ti GPU.The reduced CIFAR-100 data follows the default category order.
Loading 2304.12777v1…