Source-linked AI summary
Correlation Congruence for Knowledge Distillation
Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, Zhaoning Zhang
TL;DR
Knowledge distillation commonly matches teacher and student responses instance by instance, while correlations among multiple instances are also valuable but often ignored. CCKD transfers both types of information using correlation congruence and a Taylor-series-based generalized kernel method, with experiments reporting improved distillation performance across four representative tasks.
Problem
Knowledge distillation frameworks often impose instance-level congruence while neglecting correlations among multiple instances, despite their value for knowledge transfer.
Method
CCKD transfers instance information and inter-instance correlation information, using a generalized Taylor-series kernel method and mini-batch training strategies.
Results
CCKD significantly promotes student-network performance across CIFAR-100, ImageNet-1K, person re-identification, and face recognition tasks.
Takeaways & Limitations
CCKD provides a distillation framework that can be deployed in teacher-student methods such as standard KD and hint-based learning.
Abstract
from arXiv · showhide
Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information, but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and achieves state-of-the-art accuracy compared with other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hint-based learning methods.
1. Introduction
Knowledge distillation helps transfer behavior from large networks to smaller ones, but instance-level matching can miss relationships among examples. CCKD adds correlation congruence to transfer these relationships alongside instance information.
- Motivation: Large DNNs often outperform smaller networks but are difficult to deploy on resource-limited embedded systems.The motivation is to reduce computation and memory while narrowing the performance gap between minor and large networks.
- Motivation: Knowledge distillation improves small-network performance by matching a high-capacity teacher’s output behavior for each input instance.The framework commonly uses a strong congruence constraint, such as minimizing Kullback-Leibler divergence between teacher and student outputs.
- Motivation: Instance-only congruence can produce student embeddings whose inter-instance correlations differ from the teacher’s, weakening intra-class cohesiveness.Teacher embeddings typically bring intra-class instances together and separate inter-class instances, whereas instance-constrained students may lack this characteristic.
- Contribution: CCKD transfers both instance information and correlation information between instances through a new correlation-congruence distillation framework.The method can be trained with mini-batches and requires teacher and student embedding spaces to have the same dimension.
- Contribution: A general kernel-based method and different mini-batch sampler strategies are introduced to better capture and transfer correlation knowledge.The paper evaluates correlation metrics and explores sampler strategies for mini-batch training.
- Evaluation: CCKD improves distillation performance across CIFAR-100, ImageNet-1K, person re-identification, and face recognition experiments.The reported evaluation combines empirical experiments and ablation studies across four representative tasks and different networks.
2. Related Work
Prior work compresses networks through architectural, pruning, factorization, and distillation techniques, including output, intermediate-feature, attention, and collaborative approaches. CCKD extends this line by treating embedded-space correlations between instances as transferable knowledge.
- Model Compression and Acceleration: Model compression seeks networks with fewer computation and parameter costs while maintaining high performance.Examples include lightweight architectures, depthwise separable convolution, pointwise group convolution, channel shuffle, and network pruning.
- Knowledge Distillation: Knowledge distillation transfers information from large networks to small networks using teacher outputs that encode fine-grained structure among data.Subsequent methods also transfer intermediate outputs, spatial attention maps, gradients, or feature Gram matrices.
- Knowledge Distillation: Collaborative distillation trains peer or multiple networks in parallel so they share knowledge through distillation losses.Deep mutual learning and online distillation differ from offline teacher-student training by using collaboratively trained networks.
- Knowledge Distillation: Adversarial distillation methods model teacher-student transfer by learning loss functions or discovering adversarial samples near decision boundaries.These approaches combine generative adversarial networks with distillation or use adversarial samples to support boundary discovery.
- Correlation Congruence: CCKD adds correlations between instances in embedded space to the instance knowledge transferred during knowledge distillation.This positions correlation congruence as an additional transfer target beyond conventional instance-level information.
3. Correlation Congruence Knowledge Distillation
CCKD augments conventional instance-level distillation with correlation congruence between samples in embedded feature space. It uses kernel-based correlation modeling and structured mini-batch sampling to transfer these relationships during training.
- Motivation: Instance-only distillation can produce student embeddings with weaker intra-class cohesion and inter-class separation than the teacher.The capacity gap makes it difficult for the student to learn an identical mapping function from the teacher.
- Framework: CCKD combines prediction-level instance congruence with correlation congruence between teacher and student instances.The framework uses KL divergence for prediction alignment and Euclidean distance for aligning teacher and student correlations.
- Correlation modeling: Each correlation-matrix element represents the relationship between two instances in embedding space, using a selectable metric ϕ.The paper allows ϕ to be any correlation metric and introduces multiple choices for capturing instance relationships.
- Generalized kernel-based correlation: Gaussian RBF is presented as more flexible for nonlinear instance relationships than naive MMD and bilinear pooling, with correlations approximated by a P-order Taylor series.The kernel matrix encodes pairwise correlations between features, while γ is tunable.
- Mini-batch sampler: Uniform random mini-batches can bias intra-class correlation estimates, motivating class-uniform and superclass-uniform sampling strategies.CUR samples a fixed number of examples per class; SUR clusters teacher features into superclasses and samples by those groups, offering greater tolerance for label imbalance.
- Efficiency: Mini-batch correlation computation has O(pbd^2) time complexity and O(b^2 + d^2) extra space complexity.Here b is batch size, d is feature dimension, and p is the Taylor-series order.
4. Experiments
Experiments evaluate CCKD across classification, person re-identification, and face recognition, with ablations examining correlation metrics, kernel order, sampling, and embedding structure. CCKD improves over conventional baselines and KD across these settings.
- Classification Results on CIFAR-100: CCKD reaches 72.4% and 70.2% top-1 accuracy on CIFAR-100 with ResNet-20 and ResNet-14, respectively.These results surpass CE by 4.0% and 3.8%, and KD by 1.6% and 1.9%.
- Person Re-Identification on MSMT17: CCKD improves original KD on MSMT17 by 3.1% in rank-1 accuracy and 2.4% in mAP.The comparison includes other state-of-the-art KD-based methods, with distillation methods except DML trained without ImageNet-1K pretraining.
- Face Recognition Results on Megaface: 86.29% Rank-1 identification with 1M distractors exceeds L2-mimic by 3.28% and improves over Adv by 2.16%.ArcFace alone reaches 65.91%, while teacher-guided L2-mimic reaches 83.01%.
- Ablation Studies: Gaussian RBF performs best among MMD, Bilinear Pool, and Gaussian RBF, so the main experiments use a 2-order Taylor approximation.All three correlation metrics surpass original KD; 3-order expansion outperforms 1- and 2-order expansions.
- Embedding Analysis: CCKD produces higher intra-class cosine similarity than KD, while inter-class similarity differs little between the methods.The visualization indicates greater intra-class cohesion in the CCKD embedding space.
5. Conclusions
CCKD transfers both instance information and inter-instance correlation, using a Taylor-series kernel method to capture correlation more effectively. Experiments across four representative tasks show that it significantly improves student-network performance.
- CCKD transfers both instance information and correlation information between instances during knowledge distillation.
- A generalized method based on Taylor series expansion of a kernel function is proposed to better capture inter-instance correlation.
- Two new mini-batch sampler strategies are proposed to further improve CCKD.
- Extensive experiments on four representative tasks show that CCKD significantly promotes student-network performance.