Source-linked AI summary
Knowledge Distillation via the Target-aware Transformer
Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, Gang Wang
TL;DR
The paper addresses semantic mismatch in one-to-one teacher–student feature matching, where corresponding spatial locations can carry different information. It introduces target-aware one-to-all distillation with hierarchical efficiency mechanisms and reports gains across classification and segmentation benchmarks. The authors report state-of-the-art performance, including 72.41% ImageNet top-1 accuracy for ResNet18 and a 1.75% COCOStuff10k mIoU improvement.
Problem
One-to-one feature distillation assumes corresponding teacher and student locations contain the same spatial information, although architecture differences can produce semantic mismatch.
Method
A target-aware transformer uses parametric correlations to distill each teacher spatial component across student locations, with hierarchical patch-group and anchor-point distillation for large feature maps.
Results
The method achieves state-of-the-art performance across multiple vision tasks, including 72.41% ImageNet top-1 accuracy and a 1.75% COCOStuff10k mIoU gain.
Takeaways & Limitations
The framework extends feature distillation beyond strict spatial correspondence while supporting applications with large feature maps through hierarchical distillation.
Takeaways & Limitations
The study distills only the backbone’s last layer and does not investigate applications such as object detection.
Abstract
from arXiv · showhide
Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features from the teacher to the student in a one-to-one spatial matching fashion. However, people tend to overlook the fact that, due to the architecture differences, the semantic information on the same spatial location usually vary. This greatly undermines the underlying assumption of the one-to-one distillation approach. To this end, we propose a novel one-to-all spatial matching knowledge distillation approach. Specifically, we allow each pixel of the teacher feature to be distilled to all spatial locations of the student features given its similarity, which is generated from a target-aware transformer. Our approach surpasses the state-of-the-art methods by a significant margin on various computer vision benchmarks, such as ImageNet, Pascal VOC and COCOStuff10k. Code is available at https://github.com/sihaoevery/TaT.
1. Introduction
The paper identifies semantic mismatch as a weakness of one-to-one feature distillation and proposes target-aware, one-to-all matching. Hierarchical distillation reduces the method’s complexity, and experiments report state-of-the-art results across vision tasks.
- 1. Introduction: One-to-one distillation can mismatch teacher and student semantics because their receptive fields differ at the same spatial location.The teacher’s deeper architecture produces a larger receptive field and richer semantic information than the student.
- 1. Introduction: The target-aware transformer distills each teacher spatial component to all student locations using learned correlations.The correlations control feature aggregation and reconfigure the student to align with the teacher.
- 1. Introduction: Hierarchical distillation lowers computation by applying one-to-all matching within patch groups and distilling pooled anchor representations.The framework transfers local features and global dependency instead of operating directly on entire large feature maps.
- 1. Introduction: 72.41% top-1 accuracy boosts ResNet18 from 70.04% on ImageNet and surpasses prior distillation by 0.8%.On COCOStuff10k, the compact MobileNetV2 gains 1.75% mIoU over previous approaches.
- 1. Introduction: The framework achieves state-of-the-art performance against related alternatives on multiple computer vision tasks.The reported evaluations cover image classification and semantic segmentation.
2. Related Works
Prior work progressed from logits to intermediate feature-map distillation but commonly retained spatial correspondence assumptions. Related approaches relax this constraint through relational representations, while the paper also situates its framework among efficient vision transformers.
- 2. Related Works: Intermediate feature-map distillation became a stronger alternative to distilling only final-layer logits.These methods encourage similar spatial patterns between teacher and student features.
- 2. Related Works: Some methods relax strict spatial correspondence by transferring teacher relational graphs or similarity matrices to the student.This line of work addresses semantic mismatch between teacher and student pixels.
- 2. Related Works: The framework is adapted to semantic segmentation and is related to relation knowledge distillation based on similarity matrices.The paper investigates hierarchical distillation for this setting.
- 2. Related Works: Vision transformers motivate efficient patch-level and multiscale processing, including shifted-window and pyramid designs.These approaches address the computation burden associated with global or large-scale feature processing.
3. Method
The method replaces one-to-one feature matching with target-aware one-to-all distillation, then uses hierarchical grouping to reduce the resulting computational cost.
- Feature matching: The conventional feature-matching loss sums distances between teacher and student features at corresponding spatial locations.This formulation assumes that corresponding pixels have matching semantic distributions.
- Motivation: Because teacher and student architectures produce different semantic content at the same pixel, direct pixel-wise regression can yield suboptimal distillation.Teacher features typically encompass more layers and channels, giving them richer spatial semantics.
- Target-aware Transformer: One-to-all matching lets each teacher feature location guide the entire student feature through dynamically computed correlations.The Target-aware Transformer reconfigures student features before applying an L2 loss against teacher features.
- Target-aware Transformer: TaT computes normalized inner-product correlations whose entries gate how much each student component contributes to a reconfigured feature.The resulting weighted aggregation allows the student to mimic a teacher component without reconstructing it pixel by pixel.
- Hierarchical Distillation: TaT has O(H^2 · W^2) complexity on H × W feature maps, motivating patch-group and anchor-point hierarchical distillation.Patch groups retain local spatial information, while anchor points provide global representation and the two objectives are complementary.
4. Experiment
Experiments evaluate TaT across classification and semantic segmentation, with ablations examining its components, transformations, and efficiency trade-offs. The method consistently improves student performance across datasets and architectures while hierarchical distillation reduces computational cost.
- Image Classification: 72.41% top-1 accuracy is achieved by tiny ResNet18 on ImageNet, surpassing state-of-the-art knowledge distillation by 0.8%.Without LKL, the model reaches 72.07%; enabling LKL raises accuracy to 72.41%.
- Semantic Segmentation: 5.39% mIoU improvement over stand-alone training and 1.06% over ICKD is obtained for MobileNetV2 on Pascal VOC.The method surpasses previous segmentation baselines on Pascal VOC.
- Ablation Study: The semiparametric target-aware transformer performs best on ImageNet, while identity mapping for θ(·) performs better than Conv+BN on Cifar-100.The ImageNet comparison evaluates non-parametric and parametric implementations.
- Ablation Study: Increasing ϵ from 0.05 to 0.25 gives the LTaT objective a positive and stable effect on ImageNet performance.The experiment removes LKL and sets α to 0.1.
- Ablation Study: Patch-group and anchor-point distillation are complementary, and their combination achieves the best performance.Anchor-point pooling reduces computation but excessive pooling can omit informative representation and damage performance; four patches per group reaches the best reported performance in one example.
5. Conclusion
The work introduces target-aware transformation for knowledge distillation, enabling students to aggregate semantic information across spatial locations and mimic teacher features collectively.
- Target-aware transformation lets each student pixel aggregate useful semantics from the student representation before mimicking the teacher.
- The framework replaces independent partial divergences with whole-student matching of teacher knowledge.
- Hierarchical distillation extends the method to semantic segmentation through patch-group and anchor-point distillation for local features and long-range dependency.
6. Discussion
The paper reports no ethical risk from its use of public, transparent benchmarks, while identifying limited distillation depth and application coverage as future work.
- The authors report no ethical risk because the benchmarks are public and transparent.
- The method currently distills only the backbone’s last layer, leaving multi-layer distillation for future investigation.
- Object detection remains unevaluated and may require a new objective tailored to that application.
A.1 Asset Usage
The supplementary material documents the public code assets used in the work and provides a dedicated table for their usage.
- The work is built on public dataset and code assets, whose contributions are acknowledged.
- Table 11 lists the usage of code assets.
A.2.1 Comparison on COCOStuff10k
The COCOStuff10k comparison evaluates the proposed method against several approaches and reports that it outperforms the comparison methods.
- The proposed method outperforms the comparison methods on COCOStuff10k.
- Because some methods do not support COCOStuff10k, the authors re-implement them for the comparison.
- The comparison is reported using mIoU percentage on COCOStuff10k.
A.2.2 Hyperparameters on Cifar-100
On Cifar-100, the method outperforms the listed baselines under matched settings, while coefficient searches emphasize the proposed objective and visualization shows student features becoming more teacher-like.
- In 4 of 6 backbone settings, the searched coefficient ϵ exceeds α, indicating greater weight for the proposed objective than standard Cross-entropy.For VGG13→VGG8, ϵ is 8 and α is 0.1.
- Our method with KD loss surpasses all baselines again under the same settings on Cifar-100.
- A.2.4 Feature Visualization: The reconfigured student feature more closely matches the teacher feature, while the TaT map aggregates mostly neighboring semantics to enlarge receptive fields.