Source-linked AI summary
Distilling Knowledge via Knowledge Review
Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia
TL;DR
Knowledge distillation has largely focused on transformations and losses between same-level teacher and student features, leaving cross-level connection paths underexplored. This paper proposes knowledge review, using multiple teacher layers to supervise one student layer with residual learning, ABF, and HCL. The framework improves compact-model performance across classification, object detection, and instance segmentation, while the authors identify stage-level outputs as sufficient for decent general results.
Problem
Prior knowledge-distillation methods mostly focus on feature transformations and loss functions between same-level teacher and student features, leaving cross-level connection paths underexplored.
Method
Knowledge review uses multiple teacher layers to guide one student layer, supported by residual learning, attention based fusion, and hierarchical context loss.
Results
The framework achieves significant improvement across classification, object detection, and instance segmentation tasks and state-of-the-art performance for many compact models.
Takeaways & Limitations
Cross-stage review provides an effective knowledge-distillation framework for compact models across multiple computer vision tasks.
Takeaways & Limitations
The current framework uses only stage outputs; future work will investigate features inside a stage and other loss functions.
Abstract
from arXiv · showhide
Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement. Code is available at https://github.com/Jia-Research-Lab/ReviewKD
1. Introduction
The paper identifies same-level teacher–student connections as a bottleneck in knowledge distillation and proposes knowledge review, which uses multi-level teacher information to guide student learning. Residual learning, ABF, and HCL support this framework, which improves compact models across computer vision tasks.
- 1. Introduction: Same-level teacher–student connections can bottleneck knowledge distillation, while changing the connection structure improves performance across many tasks.Previous methods use same-level information; the paper reports that revising this structure consistently improves whole-system performance.
- 1. Introduction: The student high-level stage can learn useful information from the teacher’s low-level features.The paper presents this as an observed capacity supported by further analysis.
- 1. Introduction: Knowledge review uses multiple teacher layers to supervise one student layer, connecting information across different network levels.The mechanism reviews shallower teacher features when guiding current student features.
- 1. Introduction: The framework combines a residual learning design with an attention based fusion module and hierarchical context loss.These components are intended to make multi-level transfer stable, efficient, and more effective.
- 1. Introduction: The method achieves state-of-the-art performance for compact models across multiple computer vision tasks.The authors report this outcome across classification, object detection, and instance segmentation.
2. Related Work
Prior distillation methods transfer knowledge through logits, intermediate features, attention, distributions, relations, contrastive objectives, or multi-stage representations. The paper emphasizes that these approaches do not investigate reviewing knowledge across stages.
- 2. Related Work: FitNet transfers one-stage intermediate features through convolution layers and measures feature differences with L2 distance.The student feature is transferred to the teacher feature’s shape before distance measurement.
- 2. Related Work: PKT, RKD, and CRD use probability distributions, example relations, and contrastive objectives to guide student learning.These methods represent alternatives to direct one-stage feature matching.
- 2. Related Work: AT, FSP, SP, and OFD transfer knowledge using attention maps, feature matrices, example similarity, or a marginal-ReLU distance.These methods incorporate multi-stage information or relationships between examples and features.
- 2. Related Work: Previous methods do not discuss reviewing knowledge, which the paper reports as effective for quickly improving system performance.The paper distinguishes knowledge review from earlier transfer strategies.
3. Our Method
The method formalizes knowledge review as cross-level distillation, where multiple teacher features guide one student feature, then develops a compact progressive residual-learning framework with ABF and HCL. The framework addresses the complexity and information mismatch of directly distilling all stages while preserving multi-level guidance.
- Review Mechanism: Knowledge review uses previous teacher features to guide the current student feature, unlike conventional same-level distillation.The review mechanism connects shallower teacher features to deeper student features and can complement multiple-layer distillation.
- Optimization: The review loss is added during training while inference remains unchanged, making the method cost-free at test time.A factor λ balances the review distillation loss against the original losses.
- Review Mechanism: Directly distilling all stages is complicated and costly because features from different stages contain substantially different information.The authors therefore progressively optimize fusion rather than using every feature pair independently.
- Residual Learning Framework: The final architecture uses residual learning so deeper student features progressively learn the teacher–student residual at corresponding reviewed levels.For example, stage-4 student features are aggregated with stage-3 student features to mimic stage-3 teacher features; the formulation reduces complexity to n pairs of distances.
- ABF and HCL: The ABF module resizes and concatenates features from different levels, generates attention maps, and adaptively aggregates the feature maps.Adaptive aggregation accounts for the diverse information and spatial focus of low- and high-level features.
- ABF and HCL: HCL uses pyramid-pooled student and teacher features to transfer compound multi-level context because global L2 distance is insufficient for that information.The method introduces hierarchical context loss alongside attention-based fusion.
4. Experiments
Experiments across classification, object detection, and instance segmentation show that knowledge review consistently improves student performance, including across differing architectures and stages.
- Classification: The method outperforms previous knowledge-distillation methods on all evaluated CIFAR-100 architectures.The comparison includes same-style teacher–student architectures and uses multilayer features with the review mechanism.
- Classification: On ImageNet, the method reduces the student–teacher gap from 2.14 to 1.70, a 20% relative performance improvement.The experiments include ResNet50-to-MobileNet and ResNet34-to-ResNet18 distillation settings.
- Object Detection: The method outperforms FGFI by a large margin on object detection, where conventional KD and FitNet provide limited gains.FGFI is described as performing better than other compared methods before the proposed method exceeds it.
- Object Detection: 3.49 and 2.43 mAP gains are obtained for ResNet18 and ResNet50 on FasterRCNN, while ResNet50-to-MobileNetV2 improves from 29.47 to 33.71.RetinaNet also gains 2.33 mAP, supporting effectiveness across two-stage and one-stage detectors.
- Instance Segmentation: Instance segmentation improves by 2.37 for ResNet18, 1.74 for ResNet50, and 3.19 for MobileNetV2.For same-style architectures, the teacher–student gap is reduced by 32% and 51% relatively for ResNet18 and ResNet50.
- Cross-Stage Analysis: Cross-stage analysis finds that teacher lower-level information helps supervise deeper student stages, whereas teacher higher-level information adversely affects training.The student baseline in the stage-transfer analysis is 69.1, and lower-level supervision is explicitly marked as helpful.
- Component Analysis: Ablations show progressively larger gains from review, residual learning, attention-based fusion, and hierarchical context loss, with their combination producing the best results.The review mechanism improves the baseline, while aggregating the additional components yields the strongest result.
5. Conclusion
The review mechanism uses multiple teacher layers to supervise one student layer and improves performance across classification, object detection, and instance segmentation. The current framework uses stage outputs, while future work will investigate features within stages and other loss functions.
- The review mechanism uses multiple teacher layers to supervise one layer in the student network.
- The method achieves significant improvement consistently across classification, object detection, and instance segmentation tasks compared with previous state-of-the-art methods.
- The framework currently uses only stage outputs, while future work will explore features inside stages and other loss functions.