PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation

Gang Li, Xiang Li, Yujie Wang, Shanshan Zhang, Yichao Wu, Ding Liang

arXiv:2112.04840v1cs.CV

TL;DR

Object-detection distillation must account for teacher–student differences in candidate-box rankings and in feature versus prediction responses. The paper proposes Rank Mimicking and Prediction-guided Feature Imitation to address these differences. On MS COCO, RetinaNet reaches 40.4% AP, 3.5% above its baseline, and the paper reports effectiveness on COCO and VOC.

  • Problem

    The paper identifies a gap in understanding teacher–student behavior differences in detection distillation, including divergent candidate-box rankings and feature differences that do not align with prediction differences.

    Source: Introduction · p. 2

  • Method

    Rank Mimicking transfers teacher anchor-rank distributions, while Prediction-guided Feature Imitation uses position-wise prediction differences to guide feature imitation.

    Source: Introduction · p. 2

  • Results

    40.4% AP on MS COCO, 3.5% above the baseline; experiments on COCO and VOC are reported to validate the method’s effectiveness and generalization.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Rank distributions consistently outperform traditional soft-label distillation, and PFI connects feature imitation with prediction accuracy.

    Source: Conclusion · p. 7

    The method lags behind GID on APS; addressing scale variance in pixel-wise distillation is left for future research.

    Source: Comparison with State-of-the-art Methods · p. 7

Abstract

from arXiv · show

Knowledge Distillation (KD) is a widely-used technology to inherit information from cumbersome teacher models to compact student models, consequently realizing model compression and acceleration. Compared with image classification, object detection is a more complex task, and designing specific KD methods for object detection is non-trivial. In this work, we elaborately study the behaviour difference between the teacher and student detection models, and obtain two intriguing observations: First, the teacher and student rank their detected candidate boxes quite differently, which results in their precision discrepancy. Second, there is a considerable gap between the feature response differences and prediction differences between teacher and student, indicating that equally imitating all the feature maps of the teacher is the sub-optimal choice for improving the student's accuracy. Based on the two observations, we propose Rank Mimicking (RM) and Prediction-guided Feature Imitation (PFI) for distilling one-stage detectors, respectively. RM takes the rank of candidate boxes from teachers as a new form of knowledge to distill, which consistently outperforms the traditional soft label distillation. PFI attempts to correlate feature differences with prediction differences, making feature imitation directly help to improve the student's accuracy. On MS COCO and PASCAL VOC benchmarks, extensive experiments are conducted on various detectors with different backbones to validate the effectiveness of our method. Specifically, RetinaNet with ResNet50 achieves 40.4% mAP in MS COCO, which is 3.5% higher than its baseline, and also outperforms previous KD methods.

Introduction

The paper studies teacher–student behavior differences in object-detection knowledge distillation, identifying divergent candidate-box rankings and mismatched feature and prediction differences as motivations for Rank Mimicking and Prediction-guided Feature Imitation.

Introduction

  • Knowledge distillation transfers knowledge from cumbersome teachers to lightweight students, while prior detection methods had not deeply investigated teacher–student behavior differences.The paper frames this as an alternative to relying only on more efficient detector designs.Source: Introduction · p. 1
  • For hard examples, students may select different, inferior detection anchors than teachers, motivating Rank Mimicking to transfer teacher anchor-rank distributions.On easy examples, both models can produce precise boxes from the same anchors.Source: Introduction · p. 2, Figure 2 · p. 2
  • Feature differences can remain large where prediction differences are small, so PFI uses position-wise prediction differences to weight feature imitation.The paper notes that large feature differences can otherwise produce unexpected or wasted distillation gradients.Source: Introduction · p. 2
  • 40.4% AP on MS COCO, 3.5% above the baseline, with the method also outperforming previous state-of-the-art methods.The experiments also cover PASCAL VOC.Source: Introduction · p. 2

Related Work

The related work reviews detector architectures and prior detection-distillation methods, then contrasts those approaches with the paper’s use of prediction-guided feature imitation and anchor-rank distillation.

Object Detection

  • One-stage detectors are the focus because the paper describes them as more efficient, practical, and easier to deploy than two-stage detectors.The section reviews Faster R-CNN and Cascade R-CNN as two-stage examples and Focal Loss, FCOS, ATSS, and GFL among one-stage developments.Source: Object Detection · pp. 2–3

Knowledge Distillation for Object Detection

  • Prior detection distillation methods use proposal features, soft labels, selected regions, foreground/background weighting, or instance-wise distillation.The reviewed approaches include Mimicking, Chen et al., FGFI, deFeat, and GID.Source: Knowledge Distillation for Object… · p. 3
  • Unlike prior methods, PFI imitates whole feature maps without hand-crafted region selection or ground-truth foreground/background partitioning, while RM mimics anchor-rank distributions rather than soft labels.The section states that PFI has no hyperparameters to tune and that anchor-rank imitation is more effective than traditional soft labels.Source: Knowledge Distillation for Object… · p. 3

Methods

The methods combine classic prediction and feature distillation with Rank Mimicking, which transfers teacher anchor rankings, and Prediction-guided Feature Imitation, which weights feature imitation using prediction differences.

Classic Knowledge Distillation

  • Classic KD uses cross-entropy to align student class probabilities with one-hot labels and KL divergence to align predictions with teacher soft labels.Teacher class probabilities provide structural information between categories.Source: Classic Knowledge Distillation · pp. 3–4
  • The method adopts FPN feature imitation, following prior empirical findings that FPN features are better to imitate than backbone features.Intermediate feature imitation is described as accelerating training convergence and improving student performance.Source: Classic Knowledge Distillation · p. 4

Rank Mimicking

  • The authors conjecture that soft labels bring negligible object-detection gains partly because they lose category relationships and scores also rank boxes for NMS.Some detectors use joint scores for class probability and localization quality.Source: Rank Mimicking · p. 4
  • RM matches teacher and student rank distributions instance-wise, applying softmax to target-class scores over positive anchors and minimizing their KL divergence.Teacher rankings capture relationships across positive anchors for each instance, unlike traditional soft labels applied locally.Source: Rank Mimicking · pp. 4–5

Prediction-guided Feature Imitation

  • PFI uses position-wise prediction differences to guide feature imitation toward regions with larger differences between teacher and student predictions.The authors report that feature differences can be large where prediction differences are small, producing unexpected gradients; PFI uses prediction differences to calibrate imitation.Source: Prediction-guided Feature Imitation · p. 5

Overall loss function

  • For object detection, the student is trained end-to-end; α and β balance each loss on a common scale, with default values α = 4 and β = 1.5.Source: Overall loss function · p. 5

Experiments

Experiments evaluate detection performance on MS COCO and PASCAL VOC using their respective standard metrics. The supplied setup also specifies training schedules and hardware.

Datasets and Metrics

  • COCO uses trainval135k (115K images) for training and minival (5K images) for validation; PASCAL VOC uses VOC 2007/2012 trainval splits and the VOC 2007 test split.COCO performance is measured by AP; VOC-style mAP uses an IoU threshold of 0.5.Source: Datasets and Metrics · p. 5

Implementation Details

  • Experiments use eight GTX 1080Ti GPUs and SGD with a batch size of two images per GPU.COCO ablations use a 12-epoch schedule, state-of-the-art comparisons use 24 epochs, and VOC models train for 18 epochs.Source: Implementation Details · pp. 5–6

Ablation Study

Ablations find that Rank Mimicking and Prediction-guided Feature Imitation improve detector performance across architectures and teacher–student pairs. The methods complement one another, and both ranking targets and prediction-guided imitation outperform their alternatives in the reported comparisons.

Effectiveness of each component

  • 2.9%: combining RM and PFI improves RetinaNet and makes up 69% of the teacher–student performance gap.Both components improve results across four detectors; RM gains 1.4% on RetinaNet, while PFI gains at least 2.1% on each detector.Source: Ablation Study · p. 6

Comparison with state-of-the-art methods

  • The COCO results compare the proposed RM and PFI with state-of-the-art methods; the VOC results compare against state-of-the-art methods.The supplied passages give table scopes but no specific VOC comparison values.Source: Table 6 · p. 7, Table 7 · p. 7

Comparison with traditional soft label distillation

  • RM consistently surpasses soft-label distillation on three detectors, while soft labels bring only around 0.2%–0.5% AP improvements.Across teacher–student pairs, RM gains more than 1%; the methods also work with weak mobileNet-V2 students.Source: Ablation Study · pp. 6–7

Comparing two types of rank distribution

  • 33.3% AP: mimicking predicted box-quality ranks surpasses the baseline by 1.4%, and combining both ranking targets performs best.Mimicking classification-score ranks alone brings 1.0% AP gains over the baseline.Source: Ablation Study · p. 7

Comparison with other feature imitation masks

  • PFI surpasses the compared hand-crafted feature-imitation masks by at least 0.5% AP.Imitating only positive or only negative samples is inferior to imitating whole feature maps; the comparison includes features inside ground-truth boxes.Source: Ablation Study · p. 7

Comparison with State-of-the-art Methods

On COCO and VOC, the method outperforms the compared state-of-the-art distillation methods in the reported results. Its COCO comparison also identifies a scale-related shortfall against GID on APS.

COCO comparison

  • 40.4% AP: the method surpasses DeFeat’s 39.7% AP by 0.7% and previous state-of-the-art methods by at least 0.5% on two COCO teacher–student pairs.The comparison reports that it surpasses FitNet, FGFI, and GID by 2.2%, 1.0%, and 0.5%, respectively.Source: Comparison with State-of-the-art Methods · p. 7

PASCAL VOC comparison

Scope boundary

  • The method lags behind GID on APS; addressing scale variance in pixel-wise distillation is left for future research.The authors note that small objects receive fewer supervisions in pixel-wise distillation.Source: Comparison with State-of-the-art Methods · p. 7

Conclusion

The paper proposes RM and PFI for object-detection distillation based on observed teacher–student behavior differences. Experiments on COCO and VOC validate their effectiveness and generalization.

Conclusion

  • The paper proposes Rank Mimicking and Prediction-guided Feature Imitation for object-detection distillation.The methods are motivated by studying behavioral differences between teacher and student models.Source: Conclusion · p. 7
  • RM uses rank distribution as a distillation target and consistently outperforms traditional soft-label distillation.Source: Conclusion · p. 7
  • PFI correlates feature imitation with prediction accuracy to support efficient feature imitation.Experiments on COCO and VOC validate the method’s effectiveness and generalization.Source: Conclusion · p. 7

Qualitative Results

On hard examples, students produce imprecise boxes from different anchors than teachers; after the proposed distillation, student rank distributions approach the teacher’s and boxes become as precise.

  • After the proposed distillation, student rank distributions approach the teacher’s, and students predict boxes as precise as the teacher’s.Source: Qualitative Results · p. 9
  • On hard examples, students fail to predict precise boxes.Source: Qualitative Results · p. 9
  • On these examples, student and teacher final boxes come from different anchors, indicating distinct rank distributions.Source: Qualitative Results · p. 9

Related papers

Loading 2112.04840v1…