Source-linked AI summary
General Instance Distillation for Object Detection
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, Erjin Zhou
TL;DR
Object-detection distillation needs better generalization across frameworks and less dependence on GT-defined positive and negative instances. GID adaptively selects discriminative general instances and distills feature-, relation-, and response-based knowledge. Across detection frameworks, it improves students substantially, including a 2.9 absolute mAP gain for ResNet-50 RetinaNet on COCO.
Problem
Existing detection distillation methods have weak generalization across detection frameworks, rely heavily on GT, and overlook relation information between instances.
Method
GID adaptively selects discriminative general instances from teacher–student differences and distills feature-based, relation-based, and response-based knowledge.
Results
2.9 absolute mAP gain: the ResNet-50 RetinaNet student improves with GID on COCO, and GID applies across varied detection frameworks.
Takeaways & Limitations
GID can improve modern detection frameworks without manually fixing positive–negative proportions or restricting distillation to GT-related regions.
Abstract
from arXiv · showhide
In recent years, knowledge distillation has been proved to be an effective solution for model compression. This approach can make lightweight student models acquire the knowledge extracted from cumbersome teacher models. However, previous distillation methods of detection have weak generalization for different detection frameworks and rely heavily on ground truth (GT), ignoring the valuable relation information between instances. Thus, we propose a novel distillation method for detection tasks based on discriminative instances without considering the positive or negative distinguished by GT, which is called general instance distillation (GID). Our approach contains a general instance selection module (GISM) to make full use of feature-based, relation-based and response-based knowledge for distillation. Extensive results demonstrate that the student model achieves significant AP improvement and even outperforms the teacher in various detection frameworks. Specifically, RetinaNet with ResNet-50 achieves 39.1% in mAP with GID on COCO dataset, which surpasses the baseline 36.2% by 2.9%, and even better than the ResNet-101 based teacher model with 38.1% AP.
1. Introduction
Object detection needs lightweight models that balance accuracy and efficiency, but existing distillation methods struggle with detection's instance imbalance, GT dependence, and framework diversity. GID addresses these issues by selecting discriminative general instances and distilling feature-, relation-, and response-based knowledge.
- High-precision detection models are cumbersome, creating a practical need to trade off accuracy against efficiency.
- Knowledge distillation transfers knowledge from large models to small models to improve performance while enabling model compression.
- Detection distillation is difficult because positive and negative instances are extremely unbalanced, while GT-focused methods can overlook informative background regions.
- GID automatically selects discriminative general instances from teacher–student differences without requiring a fixed positive-to-negative ratio or only GT-related regions.
- GID combines feature-based, relation-based, and response-based knowledge for distillation.
- The method is evaluated on MSCOCO and PASCAL VOC across one-stage, two-stage, and anchor-free detection methods, achieving state-of-the-art performance.
2. Related Work
Object detection spans two-stage, one-stage, anchor-based, and anchor-free frameworks, while knowledge distillation transfers several forms of teacher knowledge to smaller models. Prior detection distillation manually balances GT-defined instances; GID instead uses student–teacher information gaps and instance relations.
- Two-stage detectors use region proposals and refinement for high accuracy, whereas one-stage detectors directly predict classification and localization from feature maps.
- Anchor-based detectors rely on predefined boxes and substantial computation, while anchor-free methods avoid anchor-related tuning and calculation.
- Knowledge distillation transfers teacher knowledge to improve small models and includes response-based, feature-based, and relation-based forms.
- Detection distillation must address extreme foreground–background imbalance, leading prior methods to underweight background or sample regions using RPN and GT information.
- GID selects discriminative patches using the information gap between student and teacher rather than GT labels or a manually fixed positive-to-negative ratio.
- GID additionally extracts relation-based knowledge among selected patches, which prior detection distillation methods did not exploit.
3. General Instance Distillation
GID selects discriminative instances using teacher–student prediction differences, then distills feature-based, relation-based, and response-based knowledge through a framework designed for varied detectors.
- 3.1. General Instance Selection Module: GISM selects discriminative instances from teacher and student predictions, including informative patches that may lie near objects or in background regions.The module uses predictions from both models rather than relying only on ground-truth-related regions.
- 3.1. General Instance Selection Module: GI scores are computed from the L1 distance between corresponding classification scores, while GI boxes use the regression box associated with the higher score.For one-stage detectors, scores are classification outputs; for two-stage detectors, they are RPN objectness scores.
- 3.1. General Instance Selection Module: NMS removes highly overlapping candidates, and only the top K GI instances are retained for distillation in each image.The NMS IoU threshold is 0.3.
- 3.2. Feature-based Distillation: ROIAlign resizes GI features from different-sized regions to a common size before feature-based distillation, treating targets equally.Student features are linearly adapted to the teacher feature dimension.
- 3.3. Relation-based Distillation: Relation-based distillation transfers distances among selected GI features, using Euclidean distance to measure instance relevance and smooth L1 loss for transfer.The method exploits correlation among discriminative instances in the same scene.
- 3.4. Response-based Distillation: GI assignments generate detector-specific masks for distilling classification and regression responses, while a general framework accommodates differing detection-head outputs.For two-stage detectors, the method distills RPN outputs instead of the detector head outputs.
4. Experiments
Experiments evaluate GID across detection datasets and training settings, using fixed hyper-parameters and schedules for distillation.
- 4. Experiments: Experiments evaluate effectiveness and robustness across detection frameworks, heterogeneous backbones, and few-class detection on COCO and Pascal VOC.The Pascal VOC setup uses VOC 2007 and VOC 2012 trainval images and VOC 2007 test images; COCO uses the default 120k-image training split.
- 4. Experiments: All experiments use K = 10 and fixed loss-balancing hyper-parameters, with 24 training epochs on COCO and 17.4 epochs on VOC unless specified.The listed values are λ1 = 5 × 10^-4, λ2 = 40, λ3 = 1, α = 0.1, and β = 1.
4.1. Different detection frameworks
GID is evaluated across RetinaNet, FCOS, and Faster R-CNN, using consistent feature- and relation-based distillation while adapting response-based loss to each detector. Across COCO experiments, students gain substantially and can approach or surpass teacher performance.
- Framework coverage: GID is evaluated on three detection frameworks: RetinaNet, FCOS, and Faster R-CNN.These include anchor-based one-stage, anchor-free one-stage, and two-stage detectors.
- Framework-specific design: Feature-based and relation-based distillation remain the same across frameworks, while response-based losses adapt to each model’s target definition.The detection frameworks use different target definitions, requiring different response-based distillation loss forms.
- Pascal VOC results: GID outperforms previous state-of-the-art detection distillation methods on Pascal VOC and can surpass the teacher model.The comparison uses ResNet-50 students and ResNet-101 teachers.
- COCO results: 2.9 absolute mAP is gained by the ResNet-50 RetinaNet student on COCO, recovering the performance drop from its shallower backbone.The method also produces a further APs gain over other feature-based methods by treating instances equally.
4.2. Heterogeneous network backbones
GID generalizes to heterogeneous student and teacher backbones rather than requiring matching architectures. A MobileNet-V2 RetinaNet student gains 2.5 absolute mAP when distilled from a ResNet-101 teacher.
- Heterogeneous backbones: GID is tested with heterogeneous network backbones instead of homogeneous ResNet backbones.The student uses MobileNet-V2 while the teacher uses ResNet-101.
- Performance: 2.5 absolute mAP is gained by the MobileNet-V2 RetinaNet student despite the student and teacher using different basic network modules.The teacher is a ResNet-101 based RetinaNet.
4.3. Distillation with fewer classes
GID remains effective when distillation targets fewer classes, adaptively selecting informative areas and using multiple forms of teacher knowledge. On COCO with only the person class, the student still exceeds the teacher by a large margin.
- Few-class setting: GID adaptively selects highly informative areas for distillation in few-class detection.It is designed to use all kinds of knowledge from the teacher model.
- Evaluation setting: The few-class evaluation uses the COCO dataset with only the Person class.The corresponding results are reported in Table 4.
- Results: The student model still exceeds the teacher model by a large margin in the few-class setting.This result is reported for the COCO evaluation with only the Person class.
4.4. Analysis
The analysis shows that GISM selects informative general instances across training stages and instance types, while combining instance categories and knowledge components yields the strongest distillation performance.
- Visualization of General Instances: GISM selects key characteristic patches, informative positives, and discriminative background instances for distillation.Examples include athlete shoes and clothes, selected positive regions, and background areas such as an oven-like machine or racket shadow.
- Performance gain from General Instance: General instances are categorized using IoU and intersection over proposals, separating positive, semi-positive, and negative instances.Positive instances have IoU > 0.5; semi-positive instances have IoU ≤ 0.5 and IoP > 0.7; negative instances have IoU ≤ 0.5 and IoP < 0.3.
- Performance gain from General Instance: Combining positive, semi-positive, and negative instances achieves the best performance, while negative-only distillation remains beneficial.The results indicate that useful background information can be selected while detrimental knowledge is filtered.
- Performance gain from General Instance: As training progresses, the proportion of positive distillation instances decreases, so later improvements mainly come from discriminative background areas and characteristic patches.Figure 5 tracks the three instance types over training iterations.
- Performance gain from various knowledge: Feature-based and response-based knowledge each improve performance by about 1.7 mAP, while combining all knowledge types adds another 1.2 mAP over the best single component.The combined loss benefits from unique information across feature-based, response-based, and relation-based knowledge.
- Hyperparameter analysis: Increasing top K produces significant mAP gains even with five distilled general instances, with performance stable and optimal for K from 10 to 100.K = 0 means no general instances are selected and no distillation loss is applied.
5. Conclusion
The paper proposes GID to adaptively select discriminative instances between teacher and student for distillation. It combines multiple knowledge types and applies across modern detection frameworks.
- GID adaptively selects the most discriminative instances between teacher and student for distillation.
- GID improves modern detection frameworks using feature-based, relation-based, and response-based knowledge.
- Ablation results show that some ground-truth instances can harm performance, while selected background instances can help.