Source-linked AI summary
Heterogeneous Knowledge Distillation using Information Flow Modeling
Nikolaos Passalis, Maria Tzelepi, Anastasios Tefas
TL;DR
Existing KD methods often overlook critical learning phases and struggle with heterogeneous teacher–student architectures. This paper models teacher information flow, uses critical-learning-aware supervision, and trains an auxiliary teacher aligned with the student; experiments demonstrate effectiveness across several tasks, including a reported mAP (c) gain of over 2.5% over the next-best method. A limitation is that flow divergence can reuse one teacher layer for multiple student layers when processing differs substantially.
Problem
Existing KD methods overlook neural networks’ distinct learning phases and often cannot effectively match intermediate layers across heterogeneous architectures.
Method
The method models teacher information flow, applies critical-learning-aware KD, and trains an auxiliary teacher close to the student architecture for layer matching.
Results
The method is effective across several tasks and increases mAP (c) by over 2.5% relative to the next-best method in one reported evaluation.
Takeaways & Limitations
Information-flow modeling with an auxiliary teacher and phase-specific supervision supports lightweight students with architectures that differ significantly from the teacher.
Takeaways & Limitations
When teacher and student process information very differently, flow divergence may match one teacher layer to multiple student layers, reducing information-flow granularity.
Abstract
from arXiv · showhide
Knowledge Distillation (KD) methods are capable of transferring the knowledge encoded in a large and complex teacher into a smaller and faster student. Early methods were usually limited to transferring the knowledge only between the last layers of the networks, while latter approaches were capable of performing multi-layer KD, further increasing the accuracy of the student. However, despite their improved performance, these methods still suffer from several limitations that restrict both their efficiency and flexibility. First, existing KD methods typically ignore that neural networks undergo through different learning phases during the training process, which often requires different types of supervision for each one. Furthermore, existing multi-layer KD methods are usually unable to effectively handle networks with significantly different architectures (heterogeneous KD). In this paper we propose a novel KD method that works by modeling the information flow through the various layers of the teacher model and then train a student model to mimic this information flow. The proposed method is capable of overcoming the aforementioned limitations by using an appropriate supervision scheme during the different phases of the training process, as well as by designing and training an appropriate auxiliary teacher model that acts as a proxy model capable of "explaining" the way the teacher works to the student. The effectiveness of the proposed method is demonstrated using four image datasets and several different evaluation setups.
1. Introduction
The paper identifies two limitations of existing knowledge distillation: they overlook phase-specific supervision and struggle with heterogeneous architectures. It proposes information-flow modeling, an auxiliary teacher, and critical-learning-aware supervision to train students that mimic teacher pathways.
- Motivation: Knowledge distillation transfers knowledge from a large teacher to a smaller student, while intermediate-layer methods provide richer guidance than final-layer transfer.
- Motivation: Neural networks form critical connections during early training, but existing KD methods generally ignore these distinct learning phases and their differing supervision needs.
- Proposed approach: The proposed method models teacher information flow and uses critical-learning-aware supervision so students form connections that mimic the teacher’s pathways.
- Motivation: Heterogeneous multi-layer KD remains difficult because teacher and student architectures often lack an obvious layer-to-layer correspondence, making mismatches harmful.
- Proposed approach: An auxiliary teacher closer to the student architecture explains the larger teacher’s processing and enables more effective layer matching across heterogeneous networks.
- Evaluation: The method is evaluated across metric learning, classification, handcrafted-feature imitation, and representation learning tasks for lightweight and embedded settings.
2. Related Work
Prior work expanded KD from final-layer transfer to multi-layer guidance, but commonly assumes compatible architectures. The proposed method addresses heterogeneous KD through an auxiliary network aligned with the student while exploiting knowledge from the teacher.
- Existing approaches: Knowledge-transfer methods commonly use teacher-generated soft labels to train smaller student networks.
- Existing approaches: Multi-layer methods such as hints, FSP, attention transfer, and singular-value decomposition provide intermediate guidance but usually target architecturally compatible networks.
- Proposed approach: The proposed method performs heterogeneous multi-layer KD by designing and training an auxiliary network and using knowledge encoded in its earlier layers.
- Proposed approach: The auxiliary network is close to the student architecture, enabling one-to-one intermediate-layer matching regardless of the teacher architecture.
3. Proposed Method
The method models information flow as layerwise mutual-information progressions and trains the student to match the teacher through a divergence measure that supports heterogeneous architectures. An auxiliary teacher aligned with the student architecture and phase-aware weighting address layer-matching and training-stage limitations.
- Information Flow Modeling: The method defines each network’s information flow as mutual information between every layer representation and the teacher’s training targets.The teacher and student are represented by separate information-flow vectors across their respective layer counts.
- Information Flow Modeling: The student is trained by minimizing squared differences between paired elements of the teacher and student information-flow vectors.The layer mapping selects teacher layers to minimize divergence for corresponding student layers.
- Tractable Divergence Measures: Quadratic mutual-information matching uses kernel-based neighbor probabilities to preserve feature-space geometry without requiring training labels.Cosine and T-student kernels are combined through a divergence objective, with kernel choice affecting task performance and robustness.
- Auxiliary Networks: When architectures differ substantially, the same teacher layer may map to multiple student layers, reducing information-flow granularity and destabilizing distillation.This limitation arises because the mapping uses estimated mutual information rather than each model’s actual learning capacity.
- Auxiliary Networks: An auxiliary teacher matching the student architecture acts as a proxy for the original teacher and enables direct layer correspondence.The proposed auxiliary network uses the student architecture with twice as many neurons or convolutional filters per layer.
- Training Objective: The auxiliary-teacher loss weights intermediate-layer matching through α_i, while stochastic gradient descent updates the student parameters.The intermediate-layer loss is computed using the auxiliary teacher, and α_i can be dynamically selected during training.
4. Experimental Evaluation
The proposed method is evaluated across metric learning, classification, distribution shift, and handcrafted-feature cloning, with ablations examining supervision timing and auxiliary-network size. It consistently outperforms competitive KD approaches, while results show that appropriate layer matching and auxiliary-model complexity are important.
- Evaluation setup: The method is evaluated on four datasets and compared with four competitive KD methods across multiple evaluation setups.The datasets are CIFAR-10, STL-10, CUB-200, and SUN Attribute; competitors include neural network distillation, hint-based transfer, PKT, and MKT.
- Metric learning: In CIFAR-10 metric learning, multi-layer transfer usually provides small to no retrieval gains for competing methods, whereas the proposed method increases mAP (c) by over 2.5% relative to PKT-H.The proposed method also performs better overall without additional supervision.
- Distribution shift and classification: Under STL-10 distribution shift and on CUB-200, the proposed method outperforms the evaluated alternatives across retrieval and classification metrics.On CUB-200 classification, accuracy increases by over 1.5% relative to the next-best method.
- Handcrafted-feature cloning: HoG cloning experiments show that the method can transfer knowledge from a handcrafted feature extractor and outperform the other evaluated approaches.This setup targets neural implementations of handcrafted features for embedded devices and domains with limited annotated data.
- Ablation studies: The proposed weighting scheme converges faster than single-layer PKT and multi-layer PKTH, supporting supervision that accounts for critical learning periods.The ablation study evaluates whether emphasizing intermediate layers during the critical period improves training.
- Ablation studies: Student accuracy improves only when auxiliary-teacher layers are correctly matched, and auxiliary models closest in complexity to the student produce the best performance.CNN-1-A is best for CNN-1 students, while the weaker CNN-1 teacher is best for CNN-1-L students; KD efficiency declines as student complexity increases.
5. Conclusions
The paper presents a heterogeneous KD method that models teacher information flow and uses an auxiliary teacher plus critical-learning-aware supervision to help lightweight students mimic it.
- The method models information flow through the teacher’s layers to train a student to reproduce those information-flow paths.
- An auxiliary teacher is designed and trained to support knowledge transfer when student and teacher architectures differ substantially.
- Critical-learning-aware KD ensures critical connections form so the student can effectively mimic the auxiliary teacher’s information-flow paths.
A.1. Datasets and Evaluation Setups
The method was evaluated on four image datasets using dataset-specific training and retrieval setups. Image preprocessing included dataset-dependent resizing, cropping, flipping, and padding.
- Evaluation used CIFAR-10, STL-10, CUB-200, and SUN Attribute.
- For CIFAR-10 retrieval, the training split formed the database, while the test set provided queries for measuring representation performance.
- CIFAR-10 and STL-10 retained their original 32 × 32 images and used random horizontal flipping and random cropping with 4-pixel padding during training.
- CUB-200 images were resized to 256 × 256 before random 224×224 crops, alongside a similar augmentation protocol.
A.2. Network Architectures
The experiments used three architecture families matched to dataset groups, with auxiliary teachers marked by the “-A” suffix. Auxiliary teachers were trained through PKT from either a ResNet-18 teacher or handcrafted features.
- CNN-1 was used for CIFAR-10 and STL-10, CNN-2 for CUB-200, and CNN-3 for SUN Attribute.
- Classification experiments added a fully connected layer with NC neurons.
- The “-A” suffix identifies models used as auxiliary teachers in the architecture configurations.
- Auxiliary teachers were trained with PKT using a ResNet-18 teacher’s penultimate-layer knowledge for CIFAR-10, STL-10, and CUB-200.
- For SUN Attribute, the auxiliary teacher was trained with PKT from handcrafted features.
A.3. Training Hyper-parameters
Training used Adam with dataset-specific schedules and settings. Additional contrastive supervision was combined with the KD loss under a specified weighting for CIFAR-10 experiments.
- All experiments used the Adam optimizer with default training hyper-parameters.
- CIFAR-10 experiments ran for 50 epochs with a 0.001 learning rate and batches of 128 samples across evaluated methods.
- CUB-200 training used 100 epochs, with a 0.001 learning rate for the first 50 and 0.0001 for the subsequent 50 epochs.
- SUN Attribute training ran for 20 epochs, with the decay factor γ set to 0.6 because of the shorter schedule.
- Contrastive CIFAR-10 experiments used margin 1, weighting the contrastive loss by 0.1 before combining it with the KD loss.