Source-linked AI summary
Focal and Global Knowledge Distillation for Detectors
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, Chun Yuan
TL;DR
Object-detection distillation remains limited because teacher–student features differ unevenly across foreground, background, pixels, and channels. FGD separates focal information from global pixel relations and transfers both through feature losses. Across detector types, the method achieves state-of-the-art performance, while the authors note that distillation can transfer teacher biases.
Problem
Most knowledge-distillation methods designed for image classification provide only trivial improvements for object detection, where teacher–student feature differences vary across regions and channels.
Method
FGD separates foreground and background, focuses on the teacher’s critical pixels and channels, and distills global relations between pixels using feature-based losses.
Results
FGD achieves state-of-the-art performance across two-stage, anchor-based one-stage, and anchor-free one-stage detectors in extensive COCO experiments.
Takeaways & Limitations
Because FGD operates on feature maps, it can be applied directly to diverse detection frameworks.
Takeaways & Limitations
Knowledge distillation may cause the student to inherit potential biases from the teacher.
Abstract
from arXiv · showhide
Knowledge distillation has been applied to image classification successfully. However, object detection is much more sophisticated and most knowledge distillation methods have failed on it. In this paper, we point out that in object detection, the features of the teacher and student vary greatly in different areas, especially in the foreground and background. If we distill them equally, the uneven differences between feature maps will negatively affect the distillation. Thus, we propose Focal and Global Distillation (FGD). Focal distillation separates the foreground and background, forcing the student to focus on the teacher's critical pixels and channels. Global distillation rebuilds the relation between different pixels and transfers it from teachers to students, compensating for missing global information in focal distillation. As our method only needs to calculate the loss on the feature map, FGD can be applied to various detectors. We experiment on various detectors with different backbones and the results show that the student detector achieves excellent mAP improvement. For example, ResNet-50 based RetinaNet, Faster RCNN, RepPoints and Mask RCNN with our distillation method achieve 40.7%, 42.0%, 42.0% and 42.1% mAP on COCO2017, which are 3.3, 3.6, 3.4 and 2.9 higher than the baseline, respectively. Our codes are available at https://github.com/yzd-v/FGD.
1. Introduction
Object-detection distillation is hindered by uneven teacher–student feature differences across foreground, background, pixels, and channels. FGD addresses this with focal and global distillation, and experiments report broad detector improvements.
- Knowledge distillation transfers information from a large teacher to a compact student, but classification-oriented methods yield only trivial improvements for object detection.
- Foreground–background imbalance and uneven teacher–student attention make equal distillation harmful, with joint foreground-background distillation performing worst in the reported analysis.
- Global distillation uses relations between pixels to compensate for global information missing from focal distillation.
- Feature-only losses allow FGD to apply directly to two-stage, anchor-based one-stage, and anchor-free one-stage detectors.
- FGD focuses the student on the teacher’s critical pixels and channels while also transferring relations between pixels.
- Extensive COCO experiments across detector types report state-of-the-art performance for the proposed method.
2. Related Work
Related work spans detector architectures and knowledge-distillation strategies. FGD targets limitations of prior methods by selecting valuable regions and transferring global pixel relations.
- Object detectors include two-stage, anchor-based one-stage, and anchor-free one-stage frameworks, all of which consume feature representations.
- Knowledge distillation compresses models without changing network structure and transfers teacher information through outputs or intermediate semantic features.
- Prior detector distillation methods use neck, classification-head, regression-head, RPN-sampled, fine-grained, Gaussian-mask, or performance-difference regions.
- The central challenge is selecting valuable distillation areas because prior methods may treat pixels and channels equally, distill all regions together, or omit global context.
- FGD separates images using ground-truth boxes, uses teacher attention masks to select crucial parts, and transfers global relations between pixels.
3. Method
FGD combines focal distillation, which separates foreground and background while emphasizing teacher attention, with global distillation of relations between pixels. Its feature-map-only losses make it applicable across detector architectures.
- FGD overview: FGD addresses feature-distillation differences by combining focal and global distillation, rather than treating all feature-map parts equally.Focal distillation separates foreground and background; global distillation transfers relations between different pixels.
- Focal Distillation: Focal distillation separates foreground and background with a binary mask derived from ground-truth boxes.Pixels inside ground-truth boxes receive M_i,j = 1, while other pixels receive 0.
- Focal Distillation: The scale mask balances foreground and background losses and reduces the dominance of larger targets that occupy more pixels.For overlapping targets, the smallest ground-truth box is used to calculate the scale mask.
- Focal Distillation: Teacher spatial and channel attention masks identify focal pixels and channels, while temperature T adjusts their distributions.The attention masks are computed from absolute mean feature values across pixels and channels.
- Global Distillation: Global distillation uses GcBlock to extract relations between feature-map pixels and transfers those relations from teacher to student.The inputs are feature maps from the teacher’s and student’s necks.
- Overall loss: The student is trained with the detector’s original loss plus focal and global distillation losses, computed only on neck feature maps.This feature-level formulation allows FGD to be applied to two-stage, anchor-based one-stage, and anchor-free one-stage detectors.
4. Experiments
Experiments across detector architectures, backbones, and distillation components show that FGD improves student detection and segmentation performance. Ablations indicate that foreground, background, attention masks, and global relations provide complementary benefits.
- 4.3. Main Results: FGD applies across two-stage, anchor-based one-stage, and anchor-free detectors, with feature-only losses enabling direct use across frameworks.Experiments include Faster RCNN, RetinaNet, FCOS, and Mask RCNN.
- 4.3. Main Results: FGD improves ResNet-50 students across Faster RCNN, RetinaNet, and FCOS, with RetinaNet gaining 2.3 mAP on COCO.Students can even outperform their ResNet-101 teachers in the Res101-Res50 setting.
- 4.4. More Detectors with Stronger Teachers: 40.7 mAP versus 39.7 mAP shows that a RetinaNet ResNet-50 student benefits more from a ResNeXt101 than a ResNet101 teacher.The comparisons associate stronger teacher backbones with better student features.
- 4.4. More Detectors with Stronger Teachers: Mask RCNN gains 2.9 Boundingbox AP and 2.4 Mask AP, demonstrating effectiveness for both object detection and instance segmentation.The focal distillation uses bounding-box labels in this experiment.
- 4.6.1 Sensitivity study of different losses: Combining focal and global losses yields 40.4 mAP and 56.7 mAR, indicating complementary benefits across target sizes.Focal loss benefits large targets more, while global loss benefits small and medium targets more.
- 4.6.2 Sensitivity study of focal distillation: Foreground distillation reduces false negatives, whereas background distillation reduces false-positive predictions; both areas are crucial but serve different functions.Distilling either foreground or background alone produces significant improvements.
- 4.6.2 Sensitivity study of focal distillation: The spatial attention mask brings 2.6 mAP and 2.2 mAR gains, and combining spatial and channel masks achieves the best result.Each attention mask improves performance individually.
- 4.6.3 Sensitivity study of global distillation: GcBlock-based global distillation brings a 3.1 mAP improvement, while both tested relation methods provide effective global information.The comparison uses Faster RCNN with global distillation only.
5. Conclusion
FGD transfers both crucial-part attention and global relations from teacher to student detectors. Feature-based distillation applies across detector families, while the authors identify head improvement as future work.
- 5. Conclusion: FGD guides students to match both the teacher’s crucial parts and global relations.The method combines focal and global distillation.
- 5. Conclusion: Because FGD is feature-based, it applies readily to two-stage, anchor-based one-stage, and anchor-free one-stage detectors.The authors describe the method as simple and efficient based on extensive experiments.
- 5. Conclusion: The authors consider their understanding of how to obtain a better detection head preliminary and leave it for future work.This is the paper’s stated future-work boundary.
A. Experiments on recent stronger models
FGD remains effective when evaluated with stronger recent detector models. The reported experiments cover GFL, SOLO, and YOLOX.
- A. Experiments on recent stronger models: FGD brings excellent mAP improvement to stronger recent models, including GFL, SOLO, and YOLOX.The distillation settings and results are reported in Table 8.
B. Experiments on CityScapes
FGD also improves student performance on Cityscapes. The evaluation uses Faster RCNN with a ResNet-101 teacher and ResNet-50 student.
- B. Experiments on CityScapes: FGD brings excellent AP improvement to a FasterRCNN Res101-50 student-teacher setting on Cityscapes.The results are reported in Table 9.
C. Comparison with more methods
FGD significantly surpasses three other distillation methods in a comparison using Faster RCNN-Res101 to distill Faster RCNN-Res50 on COCO.
- C. Comparison with more methods: FGD significantly surpasses three other distillation methods on COCO.The comparison uses Faster RCNN-Res101 to distill Faster RCNN-Res50.
D. Sensitivity study of hyper-parameters
The sensitivity study examines FGD's hyperparameters and reports that the balancing parameters α, β, γ, and λ do not substantially affect the method.
- D. Sensitivity study of hyper-parameters: FGD has five hyperparameters, including T and the balancing parameters α, β, γ, and λ.The study of T appears in Table 7, while α, β, γ, and λ balance the losses.
- D. Sensitivity study of hyper-parameters: FGD is not sensitive to the balancing parameters α, β, γ, and λ.This finding is reported in the sensitivity study shown in Figure 7.
E. Limitations about distillation
The paper identifies potential bias inheritance as a limitation of knowledge distillation: transferring information from teacher to student may also transfer the teacher's biases.
- E. Limitations about distillation: Knowledge distillation may cause the student model to inherit potential biases from the teacher.This follows from the method's goal of transferring information from teacher to student.