PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Layer Distillation with Semantic Calibration

Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Yan Feng, Chun Chen

arXiv:2012.03236v2cs.CVcs.AIcs.LG

TL;DR

Feature-map distillation is limited by manually fixed teacher–student layer associations despite differing layer semantics and network depths. SemCKD learns soft cross-layer associations with attention, transfers information from multiple teacher layers, and is reported to outperform compared state-of-the-art approaches, although selected hand-crafted associations can still be better in some cases.

  • Problem

    Feature-map distillation lacks reliable layer associations because teacher and student layers can have different abstraction levels, making manually specified pairs potentially harmful.

  • Method

    SemCKD uses attention to learn soft associations that connect each student layer with multiple semantically related teacher layers, while projecting feature maps to aligned spatial dimensions.

  • Results

    SemCKD consistently outperforms compared state-of-the-art approaches across extensive experiments, and SemCKDτ further widens the lead.

  • Takeaways & Limitations

    Learned multi-layer associations provide matched-semantic supervision and make the approach applicable across different tasks and network architectures.

  • Takeaways & Limitations

    SemCKD can be inferior to the best hand-crafted layer association in some cases, and its theoretical analysis includes an identity-transformation assumption for MLP(·).

Abstract

from arXiv · show

Knowledge distillation is a technique to enhance the generalization ability of a student model by exploiting outputs from a teacher model. Recently, feature-map based variants explore knowledge transfer between manually assigned teacher-student pairs in intermediate layers for further improvement. However, layer semantics may vary in different neural networks and semantic mismatch in manual layer associations will lead to performance degeneration due to negative regularization. To address this issue, we propose Semantic Calibration for cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple teacher layers rather than a specific intermediate layer for appropriate cross-layer supervision. We further provide theoretical analysis of the association weights and conduct extensive experiments to demonstrate the effectiveness of our approach. Code is avaliable at \url{https://github.com/DefangChen/SemCKD}.

1 INTRODUCTION

Knowledge distillation improves lightweight students using teacher outputs, while feature-map methods face semantic mismatch from manually assigned cross-layer associations. SemCKD learns soft associations with attention, supports heterogeneous architectures, and is reported to generalize better across extensive benchmark settings.

  • Motivation: Knowledge distillation improves lightweight-model generalization by training students against predictions from powerful pretrained teachers.Teacher outputs provide supervision beyond discrete labels while preserving model simplicity.
  • Challenge: Feature-map distillation exploits hierarchical intermediate representations, but differing teacher–student abstraction levels make fixed layer assignments vulnerable to semantic mismatch and negative regularization.The mismatch can deteriorate student performance during training.
  • Proposed approach: SemCKD uses attention to learn soft layer associations, binding each student layer to multiple semantically related teacher target layers instead of one fixed layer.Student feature maps are projected to the target layers’ spatial dimensions before the distillation loss is calculated.
  • Scope: SemCKD is applicable to heterogeneous teacher–student architectures and aims to improve feature-map distillation effectiveness.The contribution explicitly targets settings using different architectures for teacher and student models.
  • Evidence: Extensive experiments on standard benchmarks and varied network settings report that SemCKD consistently generalizes better than state-of-the-art approaches.The paper also provides a theoretical connection between association weights and the Orthogonal Procrustes problem.

2 RELATED WORK

Knowledge distillation research has expanded from prediction matching to intermediate feature representations, but most feature-map methods retain hand-crafted layer associations. SemCKD instead uses attention-based cross-layer association and is compatible with feature-embedding distillation.

  • Knowledge Distillation: Knowledge distillation is a hardware-friendly model-compression method for improving lightweight students using soft targets from cumbersome teachers.It addresses deployment constraints posed by complex models with large parameter counts.
  • Feature-Map Distillation: Feature-map distillation transfers hidden responses, spatial attention, mutual information, or similarity-based representations after matching teacher–student feature dimensions.These methods typically add convolutional or pooling transformations before calculating the distillation loss.
  • Feature-Map Distillation: Existing feature-map approaches generally use hand-crafted layer associations, which may discard extra layers or force suboptimal same-depth matches when network depths differ.A better student layer association may involve different or multiple teacher layers.
  • Related Methods: The closest related approaches associate channels or groups within specified layer pairs, whereas SemCKD learns associations at a more general cross-layer level.This distinguishes SemCKD’s scope from channel- or group-wise association methods.
  • Feature Embeddings: SemCKD is compatible with state-of-the-art feature-embedding distillation for further performance improvement.Feature embeddings preserve structural information while being more tractable than high-dimensional feature maps.

3 SEMANTIC CALIBRATION FOR DISTILLATION

This section reviews vanilla and feature-map distillation, then introduces SemCKD, which learns soft cross-layer associations to calibrate semantic supervision. The approach aggregates multi-layer teacher guidance after dimensional alignment and connects its weights to the Orthogonal Procrustes problem.

  • 3.1 Notations and Vanilla Knowledge Distillation: Vanilla KD trains a lightweight student to match predictions from a pre-trained teacher, supplementing labels with softened class information.The temperature T controls the softening effect, while the KL-divergence term is combined with cross-entropy.
  • 3.2 Feature-Map Distillation: Existing feature-map methods add intermediate-layer alignment losses, commonly using transformed feature maps and MSE distance.Their associations are typically one-pair selections or one-to-one matches between teacher and student layers.
  • 3.2 Feature-Map Distillation: Fixed layer associations can discard extra layers or force suboptimal matches when teacher and student depths or semantics differ.Such mismatches may produce negative regularization and deteriorate student performance.
  • 3.3 Semantic Calibration Formulation: SemCKD learns attention-based soft associations so each student layer can distill from multiple semantic-related teacher layers.The learned association set includes every student-teacher layer pair, with weights normalized across teacher layers for each student layer.
  • 3.3 Semantic Calibration Formulation: SemCKD projects student feature maps to each target layer’s spatial dimensions, leaves teacher maps unchanged, and aggregates pairwise MSE losses using learned weights.FitNet is recovered as a special case when one association weight is 1 and all others are 0.
  • 3.4 Theoretical Insights: The paper links attention allocation to the Orthogonal Procrustes problem and interprets the resulting objective as measuring inherent semantic similarity between feature maps.This interpretation provides the stated rationale for semantic calibration.

4 EXPERIMENTS

The experiments evaluate SemCKD across diverse datasets, tasks, teacher-student architectures, and distillation baselines using Top-1 accuracy and relative-improvement metrics.

  • Experimental Settings: Experiments cover image classification, transfer learning, few-shot learning, and noisy-label learning on four datasets.The datasets are CIFAR-100, STL-10, Tiny-ImageNet, and ImageNet.
  • Experimental Settings: Evaluation uses diverse homogeneous and heterogeneous teacher-student network combinations, including VGG, ResNet, wide-ResNet, MobileNetV2, and ShuffleNet variants.Network notation specifies depth and width or filter-scaling factors.
  • Compared Approaches: The comparison includes logits, feature-map, and feature-embedding distillation approaches, with vanilla KD and methods such as FitNet, AT, SP, VID, HKD, MGD, PKT, RKD, IRG, CC, and CRD.These approaches transfer knowledge from different network positions or representations.
  • Implementation: Experiments are implemented in PyTorch using stochastic gradient descent with Nesterov momentum across one NVIDIA TITAN X-Pascal GPU and eight NVIDIA RTX 2080Ti GPUs.For CIFAR-100, models train for 240 epochs with scheduled learning-rate reductions.
  • Evaluation Metrics: Top-1 test accuracy, Relative Improvement, and Average Relative Improvement quantify performance and SemCKD’s additional gains over existing feature-map distillation.RI compares SemCKD’s gain over a feature-map method with that method’s gain over a regularly trained student.
  • Evaluation Metrics: The Semantic Mismatch score approximates an otherwise intractable nuclear-norm calculation using average Euclidean distances between similarity matrices of associated layer pairs.The score uses similarity matrices for student and target layers and association weights for compared approaches.

4.2 Comparison of Different Feature-Map Distillation

On CIFAR-100, SemCKD is compared with feature-map distillation methods across twelve network combinations. It generally achieves higher accuracy and relative improvement, while other methods can fall below vanilla KD under semantic mismatch.

  • Comparison Results: SemCKD consistently achieves higher CIFAR-100 accuracy than state-of-the-art feature-map distillation approaches across twelve network combinations.The combinations include two homogeneous and ten heterogeneous teacher-student settings.
  • Comparison Results: 60.03% average relative improvement over the compared feature-map approaches is reported across all evaluated combinations.The best case is a 242% RI over AT for the “VGG-8 & ResNet-32x4” setting.
  • Comparison Results: 45.82% relative improvement over HKD is reported overall, although RI is only 3.93% for “ShuffleNetV2 & ResNet-32x4” and 9.98% for “MobileNetV2 & WRN-40-2”.HKD uses a teacher-auxiliary-student paradigm.
  • Semantic Mismatch: Compared approaches do not consistently outperform vanilla KD, with failures especially common for one-pair selection, more candidate student layers, and MobileNetV2 students.FitNet fails in 7/12 cases and MGD in 8/12 cases; several settings show failures for most compared methods.
  • Semantic Mismatch: The reported failures are attributed to semantic mismatch among associated layer pairs, motivating SemCKD’s semantic calibration strategy.The passage connects the mismatch problem with degraded comparative performance.

4.3 Semantic Calibration Analysis

The analysis examines negative regularization from manual layer associations and compares semantic matching across distillation approaches. It finds that SemCKD’s learned associations generally avoid severe mismatches, while visualizations show more class-discriminative attention.

  • Negative regularization: Manual teacher-student layer associations can produce extremely poor student performance when semantic gaps are large.The analysis reports severe failures for several specified layer pairs across four model combinations.
  • Negative regularization: One-to-one layer association is suboptimal because useful supervision may come from a teacher layer at a different depth.Examples include Student Layer-1 with Target Layer-2 and Student Layer-3 with Target Layer-4.
  • Negative regularization: SemCKD performs reasonably well across many layer associations, but can remain inferior to the best hand-crafted association in some cases.The authors identify these cases as evidence that the association strategy still has room for refinement.
  • Semantic matching: The study evaluates semantic matching with SM-score and CKA across training epochs, alongside Grad-CAM visualizations on ImageNet.SM-scores and CKA values are averaged over the last 10 epochs, while Grad-CAM highlights prediction-relevant regions.
  • Semantic matching: Grad-CAM shows SemCKD concentrating class-discriminative regions similarly to the teacher, whereas compared approaches often attend to surrounding or incorrect regions.The examples include bathtub, sweatshirt, and goose images.

4.4 Ablation Study

The ablation study tests attention allocation and the components used to compute it. Removing adaptive allocation, projection, nonlinear transformations, similarity matrices, or instance-specific weights reduces performance.

  • Attention allocation: Equal attention allocation lowers accuracy by 2.33%, from 75.27% to 72.94%, and increases variance by 0.74%, from 0.13% to 0.87%.This ablation replaces personalized attention across multiple target layers with equal weights.
  • Projection: Removing feature-map projection reduces performance by 2.76%.The ablation replaces feature-map MSE with loss on projected pairwise-similarity value vectors.
  • Attention components: Replacing the two-layer nonlinear MLP with a linear transformation causes a 2.49% performance drop.The reported result is attributed to the usefulness of MLP(·) for alleviating noise and sparseness.
  • Attention components: Removing similarity matrices causes a 0.52% performance decline and would also increase memory cost when feature maps have larger spatial dimensions.The similarity matrices support subsequent attention calculation while avoiding direct feature-map query and key construction.
  • Instance-specific allocation: Sharing association weights across all training instances incurs a 0.40% accuracy drop.The full method instead learns independent attention allocation across teacher-student layer pairs for each instance.

4.5 Softening Attention

The experiments study how softening the attention distribution affects SemCKD across teacher-student combinations. Moderate softness usually improves accuracy, whereas overly sharp attention degrades performance.

  • Softening attention: Softening attention improves SemCKD in most tested cases by a considerable margin.The experiments vary softness τ across many teacher-student combinations.
  • Softness selection: At τ = 4, accuracy increases by 0.47% for VGG-8 & ResNet-32x4 and by 0.86% for MobileNetV2 & WRN-40-2.These are absolute accuracy boosts relative to original SemCKD.
  • Softness selection: When τ is less than 1, sharpened attention causes significant performance degeneration.The overall target direction becomes strongly influenced by a single component.
  • Softness selection: τ = 2 or 4 is reported as a satisfying initial choice for a given teacher-student combination.The recommendation follows the observed behavior of softened attention across combinations.
  • Pre-activation distillation: SemCKDτ+Pre slightly outperforms SemCKDτ but is distinctly inferior to original SemCKD in the reported comparison.Pre-activation distillation moves the distillation position before the ReLU operation.

4.6 Generalization to Different Scenarios

The paper evaluates SemCKD beyond standard training through representation transfer, few-shot learning, and noisy-label learning. SemCKD outperforms the compared approaches in these settings and remains relatively robust to reduced or perturbed training data.

  • Transfer Learning: SemCKD outperforms the second-best method, FitNet, by 2.76% absolute accuracy in one transfer-learning setting.Representations learned on CIFAR-100 are evaluated with a newly trained linear classifier on STL-10 or Tiny-ImageNet.
  • Transfer Learning: SemCKD improves student-model transfer performance by 3.25% on average despite teacher representations transferring worst in most settings.The teacher representations transfer worst in 3/4 settings discussed by the authors.
  • Few-shot and noisy-label learning: SemCKD is evaluated in few-shot and noisy-label learning scenarios.The study samples 25%, 50%, and 75% of training images and perturbs 10% through 50% of labels.
  • Few-shot and noisy-label learning: The experiments indicate that SemCKD makes better use of training data and is relatively robust to noisy perturbation.This is the paper’s stated conclusion for the two nonstandard learning scenarios.
  • Few-shot and noisy-label learning: SemCKD consistently outperforms compared approaches and regular training across the few-shot and noisy-label settings.The improvement becomes larger as the number of available training images decreases, according to the reported experiments.

4.7 Extension to Feature-Embedding Distillation

SemCKD extends feature-map distillation with feature-embedding distillation and remains compatible with CRD. Combining the two approaches further improves performance across all evaluated cases.

  • SemCKD is evaluated alongside several feature-embedding distillation approaches on twelve network combinations.The comparison uses the same combinations as Tables 2 and 3.
  • Adding CRD loss to SemCKD further boosts performance in all cases.The combined method requires no hyper-parameter tuning in the reported experiment.

4.8 Sensitivity Analysis

SemCKD performs best across the tested hyper-parameter settings and remains stable over a broad range of β values. Its default setting also exceeds CRD by 1.73% absolute accuracy.

  • Figure 7 reports β sensitivity for VGG-8 and ResNet-32x4 on CIFAR-100.
  • Vanilla KD uses β = 0, producing a horizontal line in Figure 7.CRD uses a separate search range from 0.5 to 1.5.
  • SemCKD achieves the best results in all tested cases and surpasses CRD by about 1.73% absolute accuracy at the default setting.The comparison includes KD, CRD, and SemCKD.
  • Performance remains stable when SemCKD’s β exceeds 400.The tested SemCKD range is 100 to 1100 in increments of 100.

5 CONCLUSION

SemCKD addresses performance degeneration from manually specified layer pairs by learning soft associations across semantically matched layers. Experiments report consistent gains, compatibility with feature-embedding distillation, and applicability across tasks and architectures.

  • SemCKD uses attention-based association learning to transfer knowledge in a matched semantic space instead of relying on manually specified layer pairs.The paper connects the learned association weights to the Orthogonal Procrustes problem.
  • Extensive experiments show that SemCKD consistently outperforms compared state-of-the-art approaches.The softening attention variant SemCKDτ further widens the lead.
  • SemCKD is reported as applicable to different tasks and network architectures and highly compatible with feature-embedding distillation.
Loading 2012.03236v2…