PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation

Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, Xiaodan Liang

arXiv:2202.03680v1cs.CVcs.AI

TL;DR

Knowledge distillation transfers learned representations from larger teacher networks to smaller students, but prior methods overlook inter-channel correlation and the balance of feature diversity and homology. ICKD distills that correlation, with a grid-level extension for dense prediction, and reports state-of-the-art results across image classification and semantic segmentation, including ResNet18 beyond 72% Top-1 accuracy on ImageNet.

  • Problem

    Prior knowledge-distillation methods overlook inter-channel correlation, leaving the teacher’s feature-space distribution and balance of diversity and homology insufficiently represented.

    Source: §1 · p. 1

  • Method

    ICKD trains the student to match the teacher’s inter-channel correlation, adding grid-level correlation distillation for dense prediction.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    ICKD reports state-of-the-art results across image classification and semantic segmentation, including ResNet18 beyond 72% Top-1 accuracy on ImageNet.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The ICC matrix is invariant to feature spatial dimensions, and grid-level ICC extends the approach to semantic segmentation.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Knowledge Distillation has shown very promising abil-ity in transferring learned representation from the largermodel (teacher) to the smaller one (student).Despitemany efforts, prior methods ignore the important role ofretaining inter-channel correlation of features, leading tothe lack of capturing intrinsic distribution of the featurespace and sufficient diversity properties of features in theteacher network.To solve the issue, we propose thenovel Inter-Channel Correlation for Knowledge Distillation(ICKD), with which the diversity and homology of the fea-ture space of the student network can align with that ofthe teacher network. The correlation between these twochannels is interpreted as diversity if they are irrelevantto each other, otherwise homology. Then the student isrequired to mimic the correlation within its own embed-ding space. In addition, we introduce the grid-level inter-channel correlation, making it capable of dense predictiontasks. Extensive experiments on two vision tasks, includ-ing ImageNet classification and Pascal VOC segmentation,demonstrate the superiority of our ICKD, which consis-tently outperforms many existing methods, advancing thestate-of-the-art in the fields of Knowledge Distillation. Toour knowledge, we are the first method based on knowl-edge distillation boosts ResNet18 beyond 72% Top-1 ac-curacy on ImageNet classification. Code is available at:https://github.com/ADLab-AutoDrive/ICKD.

1. Introduction

The introduction argues that knowledge distillation should preserve the teacher’s balance of feature diversity and homology, not only transfer instance-level information. ICKD matches inter-channel correlations, with a grid-level extension for dense prediction, and reports strong results on classification and segmentation.

1. Introduction

  • Large networks offer greater learning capacity, but their energy and computation costs make deployment on mobile systems difficult.Reducing model size while retaining comparable performance is presented as a way to bridge the gap between small and large networks.Source: §1 · p. 1
  • Knowledge distillation transfers teacher knowledge to a student, but instance-level approaches may neglect structural information in the feature space.Earlier relational approaches focus on correlations between feature maps, whereas ICKD focuses on inter-channel correlation.Source: §1 · pp. 1–2
  • Prior methods overlook the coexistence of feature diversity and homology, although both reflect feature richness; ICKD distills their inter-channel correlation instead of matching whole feature maps.The introduction notes that forcing a smaller student to mimic the teacher’s whole feature map is not viable given their differing learning capacities.Source: §1 · p. 2
  • ICKD reports ResNet18 beyond 72% Top-1 accuracy on ImageNet and a 3% mIoU improvement over the baseline on Pascal VOC.The introduction also reports that the method consistently outperforms state-of-the-art methods across knowledge-transfer tasks.Source: §1 · p. 2
  • Inter-channel correlation is invariant to spatial dimensions and serves as an indicator of feature diversity and homology for student representation learning.The paper evaluates channel correlation using the inner product.Source: §1 · p. 2
  • Grid-level inter-channel correlation extends ICKD to dense prediction tasks such as semantic segmentation.Dividing feature maps into patches makes distillation more controllable and preserves region-specific spatial information.Source: §1 · p. 2

2. Related Works

Related work covers output-logit distillation, structural relations among examples, and intermediate-layer feature matching. ICKD differs from Gram-matrix approaches by measuring relations between channels rather than pixel positions.

2. Related Works

  • Knowledge distillation matches student and teacher logits using KL-divergence, while intermediate-layer methods match feature maps between guided and hint layers.The cited discussion also describes attention compression, semantic calibration, and feature-similarity weighting for layer matching.Source: §2 · pp. 2–3
  • Structural distillation methods transfer feature-space relations, including instance similarity, kernel correlation, triplet angles, and mutual information.These approaches are cited as ways to transfer structural information from teacher to student.Source: §2 · p. 2
  • Some segmentation methods transfer pixel affinities, pairwise similarity, holistic knowledge, or pixel-to-prototype variation.The cited methods include auto-encoder feature matching and adversarial learning.Source: §2 · p. 3
  • Unlike Gram-matrix approaches that measure relations between pixel positions, ICKD measures correlation between channels.The discussion identifies Huang and Wang [13] and Yim et al. as related Gram-matrix approaches.Source: §2 · p. 3

3. Method

ICKD aligns student and teacher inter-channel correlations to match feature diversity and homology, then extends this matching over local grids for segmentation.

3.1. Preliminary

  • Traditional knowledge distillation minimizes KL divergence between softmaxed teacher and student outputs, using temperature τ on penultimate-layer logits.The teacher and student feature embeddings may have different channel and spatial dimensions.Source: §3.1 · p. 3

3.2. Formulation

  • ICKD measures channel relationships with inner products: each scalar contributes to an ICC matrix, which is spatial-size invariant and matched between teacher and student with L2 distance.A 1 × 1 convolution and batch-normalization projection aligns mismatched student channel counts to the teacher.Source: §3.2 · p. 4

3.3. Grid-Level Inter-Channel Correlation

  • For large segmentation maps, long-vector channel products may be noise-vulnerable; ICKD-S divides features into grids, matches paired ICCs with MSE, and retains local spatial information.Each grid represents a patch-level semantic region, and its ICC matrix is computed individually.Source: §3.3 · p. 4, Figure 3 · p. 4
  • ICKD-C outperforms all other methods in Table 1's Cifar-100 Top-1 accuracy comparison.The table compares methods against traditional KD and also reports ICKD-C without LKL.Source: Table 1 · p. 5
  • The segmentation objective uses supervised loss LSeg, which the authors note could be replaced for other tasks but do not study here.α is the weight factor for LSeg in the stated objective.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate ICKD on Cifar-100 and ImageNet classification and Pascal VOC segmentation, across architectures and teacher choices. Results include improvements over competing methods and show that heavier teachers do not consistently improve student performance.

4.1. Datasets

  • Experiments cover Cifar-100 and ImageNet classification and Pascal VOC segmentation, with the Pascal VOC training set augmented by coarse annotations.Pascal VOC evaluation uses mIoU; ImageNet and Cifar-100 use classification accuracy.Source: §4 · p. 5, §4.1 · p. 5

4.2. Implementation Details

  • Classification distills pre-global-pooling features, while segmentation distills features from DeepLabV3+'s last batch-normalization layer.ImageNet training uses AdamW; Cifar-100 and segmentation use SGD.Source: §4.2 · p. 5

4.3. Image Classification

  • Across tested architecture mismatches, ICKD-C outperformed competing methods, including when VGG13 and MobileNetV2 had different feature-map sizes.The Cifar-100 comparisons include VGG, ResNet, MobileNetV2, ShuffleNetV1, and ShuffleNetV2.Source: §4.3 · p. 6, Figure 4 · p. 6

4.3. Image Classification; 4.4. Semantic Segmentation

4.4. Semantic Segmentation

  • On Pascal VOC, ResNet101-teacher distillation into ResNet18 and MobileNetV2 students uses DeepLabV3+ and raises student performance from 72.07% to 75.01%.Source: §4.4 · pp. 6–7, Table 3 · p. 7

4.5. The potential of a better teacher

  • Heavier teachers did not consistently improve ResNet18, which reached Top-1 accuracies of 72.31% with ResNet101 and 72.19% with ResNet34.The authors note that ResNet152's larger channel-count mismatch may explain its limited student performance.Source: §4.5 · p. 7

4.6. Ablation Study

  • Grid-level ICC raised ResNet18 performance from 72.07% to 75.01% with 32×32 patches, although finer grids were not uniformly better.Without LKL, ICKD-C reached 71.59% Top-1 on ImageNet and was robust to β2; the 1×1 adapter gave minor gains in most cases.Source: §4.6 · p. 8

5. Conclusion

The work introduces inter-channel correlation (ICC) distillation, including grid-level ICC for semantic segmentation, and reports state-of-the-art results in image classification and semantic segmentation. It describes ICC computation as invariant to feature spatial dimensions and capable of distilling generic knowledge across network architectures.

5. Conclusion

  • Grid-level inter-channel correlation extends the method to semantic segmentation, a task that prior work largely overlooked in this respect.The conclusion highlights dense prediction as an application beyond image classification.Source: §5 · p. 8
  • The method uses inter-channel correlation to mimic the teacher network’s feature diversity.Source: §5 · p. 8
  • The method achieves state-of-the-art results in image classification and semantic segmentation.The ICC matrix is invariant to feature spatial dimensions and can distill generic knowledge across different network architectures.Source: §5 · p. 8

Related papers

Loading 2202.03680v1…