Source-linked AI summary
Feature-map-level Online Adversarial Knowledge Distillation
Inseop Chung, SeongUk Park, Jangho Kim, Nojun Kwak
TL;DR
Existing online distillation methods use class-probability information but do not transfer richer feature-map information, whose moving targets complicate online learning. The paper proposes AFD, combining mutual logit distillation with adversarial feature-map distribution matching and cyclic peer learning. Experiments report higher performance than direct alignment and existing online methods across network architectures, especially for small–large network pairs.
Problem
Existing online distillation methods use logits but not richer feature-map information, while co-trained networks provide no static feature-map target.
Method
AFD combines mutual logit distillation with adversarial discriminators that make networks learn one another’s feature-map distributions, using cyclic peer learning for multiple networks.
Results
AFD achieved higher performance than existing online methods and direct alignment across architectures, with especially strong improvement for small–large network pairs.
Takeaways & Limitations
Adversarial feature-map distribution transfer is suitable for online distillation across architectures ranging from very small to large networks.
Abstract
from arXiv · showhide
Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distribution. We show that our method performs better than the conventional direct alignment method such as L1 and is more suitable for online distillation. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.
1. Introduction
Deep and wide networks deliver high performance but require substantial computation and memory, motivating compact-network methods. The paper identifies a gap in online distillation: existing methods use logits but not feature-map information, while feature-map targets continually change.
- Deep and wide networks are unsuitable for resource-limited environments because they require substantial computation and memory.
- Knowledge distillation improves lightweight networks by transferring class-probability knowledge from a teacher or peer networks.
- Feature maps encode image intensity and spatial correlation, providing richer information than logits for distillation.
- Online distillation methods had not used feature maps because co-trained networks provide no static target and their feature maps change during training.
- The proposed AFD transfers logits and feature-map distributions adversarially, and introduces cyclic learning for more than two networks.
2. Related work
Prior work established knowledge distillation for compact models and extended it to online peer teaching. This paper connects online distillation with adversarial training to transfer feature-map distributions between co-trained networks.
- Knowledge distillation trains a compact student using softened logits from a pretrained teacher alongside conventional label supervision.
- Online distillation removes the need for a pretrained teacher by having networks teach one another during training.
- DML mutually applies distillation losses, whereas ONE uses a gated ensemble logit because DML lacks an appropriate teacher role.
- AFD uses adversarial training so networks learn one another’s feature-map distributions in addition to exchanging logits.
3. Proposed Method
AFD combines mutual logit distillation with adversarial feature-map distillation. Discriminators distinguish networks’ feature-map distributions, while cyclic peer learning extends the approach beyond two networks with reduced discriminator cost.
- Logit-based Mutual Knowledge Distillation: AFD combines conventional cross-entropy and mutual KL-divergence losses for logit-based learning.
- Adversarial Training for Feature-map-based KD: The method uses the last convolutional feature map and trains each network to fool a discriminator that distinguishes peer and self distributions.
- Adversarial Training for Feature-map-based KD: The discriminator treats peer-network feature maps as real and the corresponding network’s feature maps as fake, while the network learns to reverse that judgment.
- Logit-based Mutual Knowledge Distillation: The logit loss uses softened probabilities, with temperature T controlling smoothness and T = 3 in every experiment.
- Cyclic Learning Framework: For more than two networks, cyclic peer learning transfers knowledge along 1 →2, 2 →3, through K →1, avoiding the cost of pairwise discriminators.
4. Experiment
Experiments on CIFAR-100 and ImageNet compare AFD with direct alignment and online distillation baselines across same, different, and three-network architectures. AFD consistently improves performance, particularly for smaller networks paired with larger ones, while adversarial feature-map distillation contributes to these gains.
- Direct alignment comparison: AFD outperforms direct feature-map alignment, whereas L1 alignment can reduce online-distillation accuracy below vanilla networks.ResNet-32 fell more than 2% from its 69.38% vanilla accuracy, and WRN-16-2 fell below its 71.07% vanilla accuracy under direct alignment.
- Ablation study: Removing adversarial feature-map distillation decreases accuracy across all three ablation settings.ResNet-32 and WRN-16-2 dropped by 0.65% and 0.52%, respectively, while WRN-16-2 increased by almost 1% with different sub-network architectures when the component was included.
- Same architecture: For same-architecture pairs, AFD achieves the highest 2 Net average and ensemble accuracy among DML, ONE, and AFD.For ResNet-20, ResNet-32, and WRN-16-2, AFD improves accuracy by more than 4% over vanilla networks, while other methods improve around 3% on average except DML on ResNet-32.
- Different architectures: Across different-architecture pairs, AFD improves both networks and raises Net1 accuracy by 1.66% over KD and 1.15% over DML on average.The ensemble accuracy is also higher by 0.61% and 0.67%, respectively.
- Different architectures: For the WRN-16-2 and WRN-28-4 pair, AFD improves the smaller network by 1.72% over KD and 1.28% over DML despite the large model-size difference.The paper reports that AFD improves both networks and that smaller networks considerably benefit from larger ones.
- Three networks: With three networks, AFD uses cyclic learning to reduce computation and memory while outperforming competing online methods on average and ensemble accuracy.AFD outperforms the compared methods across every model type, and its overall performance-gain tendency is maintained relative to two-network experiments.
- ImageNet: On ImageNet, AFD-trained ResNet-18 and ResNet-34 achieve higher performance than DML after 30 epochs.The evaluation uses 1.2M training images, 50K validation images, and 1,000 classes.
5. Analysis
The analysis compares feature-map similarity and visual activations across online distillation methods. Although L1+KD produces the most closely aligned maps, AFD transfers feature-map knowledge while preserving networks’ learned features.
- The analysis compares L1/L2 distance, cosine similarity, and Grad-CAM activations across L1+KD, DML, and AFD.The experiments use pretrained networks from earlier tables and examine both numerical similarity and spatial activation patterns.
- L1+KD yields the shortest L1/L2 distance and highest cosine similarity across all three model types.These measurements indicate the strongest direct feature-map alignment among the evaluated methods.
- AFD transfers feature-map-level knowledge indirectly through a discriminator without directly or drastically changing each network’s feature map.This allows the method to preserve features learned independently by each network.
- L1+KD produces identical activation maps for Net1 and Net2, indicating that the networks output the same feature maps.The Grad-CAM visualization highlights that both networks attend to the exact same image regions.
6. Conclusion
The paper proposes online distillation that transfers both logit and feature-map knowledge through adversarial training. Experiments support adversarial distribution transfer for online feature-map distillation, with higher performance than existing online and direct-alignment methods.
- The proposed method transfers knowledge at both the logit and feature-map levels using adversarial training.
- The paper includes a cyclic-learning framework for training three networks simultaneously.Figure 4 presents the framework schematically.
- Experiments demonstrate that adversarial training can transfer feature-map distributions during online knowledge distillation.
- The method achieves higher performance than existing online distillation and conventional direct-alignment methods.
H. Formulation of cross-entropy loss and KL divergence loss
The logit-based training objective combines cross-entropy with KL-divergence mimicry loss. Temperature-controlled softened probabilities enable networks to match peer outputs while also fitting ground-truth labels.
- The logit-based objective uses cross-entropy loss and KL-divergence mimicry loss for two networks.
- The temperature T controls probability smoothness, with T = 1 yielding the original softmax and larger T producing softer distributions.
- The mimicry loss transfers peer-network information by measuring KL distance between the networks’ softened class probabilities.The networks use softened probabilities at temperature 3.
- The combined logit-based loss trains networks to predict ground-truth labels while matching peer-network outputs.The KL term is multiplied by T^2 because soft-target gradients scale by 1/T^2.
I. Schematic of Cyclic-learning framework
The cyclic-learning framework transfers knowledge among three networks in a loop. Each network distills to the next, with transfer occurring at both logit and feature-map levels.
- The framework trains three networks simultaneously through cyclic knowledge transfer.Network Θ1 transfers to Θ2, Θ2 transfers to Θ3, and Θ3 transfers back to Θ1.
- Knowledge is transferred at both the logit and feature-map levels.At logit level, KL-divergence loss is applied between networks.
J. Grad-cam visualization
Grad-CAM visualizations compare feature attention across distillation methods for ResNet-32 and WRN-16-2. AFD preserves distinct network features while transferring knowledge at the feature-map level.
- J. Grad-cam visualization: L1+KD produces identical feature maps and highlights the same image regions for both networks.The authors interpret this as the networks becoming effectively identical and copying each other's results.
- J. Grad-cam visualization: DML and AFD produce different feature maps for Net1 and Net2, preserving network-specific features and spatial information.Both networks benefit from distillation without losing their learned representations.
- J. Grad-cam visualization: AFD improves performance over DML while retaining distinct feature maps despite transferring knowledge at the feature-map level.DML distills only logit-level knowledge, whereas AFD also transfers feature-map knowledge.
- J. Grad-cam visualization: Figures 5 and 6 show Grad-CAM visualizations for ResNet-32 and WRN-16-2 under different distillation methods.The visualizations use additional CIFAR-100 test samples.