PaperlayerView exact revision on arXiv

Source-linked AI summary

VarifocalNet: An IoU-aware Dense Object Detector

Haoyang Zhang, Ying Wang, Feras Dayoub, Niko Sünderhauf

arXiv:2008.13367v2cs.CV

TL;DR

Dense detectors need a reliable way to rank candidate detections because classification and separately combined localization scores are imperfect. The paper learns an IoU-aware classification score using Varifocal Loss and star-shaped features, builds VFNet, and reports consistent gains over strong baselines on COCO. Its best reported model reaches 55.1 AP on COCO test-dev.

  • Problem

    Dense detectors lack a reliable ranking score because classification scores and combinations with predicted localization scores do not consistently reflect detection quality.

  • Method

    The paper learns IACS, trains it with asymmetric Varifocal Loss, and uses star-shaped box features for IACS prediction and bounding-box refinement in VFNet.

  • Results

    ∼2.0 AP improvement over a strong baseline is reported across backbones, and VFNet-X-1200 reaches 55.1 AP on COCO test-dev.

  • Takeaways & Limitations

    IACS provides the ranking criterion used by VFNet to exploit high-quality detections within dense candidate pools.

Abstract

from arXiv · show

Accurately ranking the vast number of candidate detections is crucial for dense object detectors to achieve high performance. Prior work uses the classification score or a combination of classification and predicted localization scores to rank candidates. However, neither option results in a reliable ranking, thus degrading detection performance. In this paper, we propose to learn an Iou-aware Classification Score (IACS) as a joint representation of object presence confidence and localization accuracy. We show that dense object detectors can achieve a more accurate ranking of candidate detections based on the IACS. We design a new loss function, named Varifocal Loss, to train a dense object detector to predict the IACS, and propose a new star-shaped bounding box feature representation for IACS prediction and bounding box refinement. Combining these two new components and a bounding box refinement branch, we build an IoU-aware dense object detector based on the FCOS+ATSS architecture, that we call VarifocalNet or VFNet for short. Extensive experiments on MS COCO show that our VFNet consistently surpasses the strong baseline by $\sim$2.0 AP with different backbones. Our best model VFNet-X-1200 with Res2Net-101-DCN achieves a single-model single-scale AP of 55.1 on COCO test-dev, which is state-of-the-art among various object detectors.Code is available at https://github.com/hyz-xmaster/VarifocalNet .

1. Introduction

Dense detectors must rank many candidate boxes, but classification scores and multiplied localization scores do not reliably reflect localization quality. VFNet addresses this by learning an IoU-aware classification score with Varifocal Loss and star-shaped box features, achieving consistently stronger COCO performance.

  • Classification scores can misrepresent localization accuracy, causing accurately localized detections with low scores to be removed by NMS.
  • Multiplying classification and separately predicted localization scores can produce a worse ranking basis and adds computation through an extra branch.
  • IACS jointly represents object-class presence and the localization accuracy of a generated bounding box.
  • VFNet combines IACS ranking, Varifocal Loss, star-shaped box features, and bounding-box refinement within an FCOS+ATSS detector.
  • Varifocal Loss down-weights negative examples while up-weighting high-quality positives to focus training on prime detections.
  • Star-shaped features use nine sampling points to capture box geometry and nearby context for IACS prediction and efficient box refinement.
  • ∼2.0 AP improvement over a strong baseline is reported across backbones, while VFNet-X-1200 reaches 55.1 AP on COCO test-dev.

2. Related Work

Prior detectors use anchors, point-based representations, or separate localization-ranking scores, each balancing accuracy against efficiency. VFNet instead uses a fixed nine-point star representation and asymmetric Varifocal Loss, while extending related joint-quality scoring with box refinement.

  • Object Detection: Object detectors include anchor-based two-stage, multi-stage, and one-stage methods, alongside anchor-free key-point and point-based approaches.
  • Detection Ranking Measures: IoU-aware and centerness-based methods combine localization estimates with classification scores for ranking, but the resulting products can remain imperfect.
  • Detection Ranking Measures: Predicting only IACS avoids an additional localization network and the potentially worse ranking basis from multiplying imperfect scores.
  • Encoding the Bounding Box: Dense detectors generally use efficient point features, while learned semantic-point representations add localization difficulty and computation burden.
  • Encoding the Bounding Box: VFNet’s nine fixed sampling points encode box geometry and spatial context while remaining simple and efficient.
  • Generalized Focal Loss: Unlike GFL’s equal weighting of positive and negative examples, Varifocal Loss weights them asymmetrically and VFNet additionally refines boxes.

3. Motivation

The FCOS+ATSS analysis tests how oracle classification, localization, and centerness signals affect ranking. It shows that IoU-aware classification is the most promising selection measure because high-quality boxes already exist among candidates.

  • FCOS+ATSS Analysis: FCOS+ATSS predicts classification, box distances, and centerness, with centerness multiplied by classification for NMS ranking.
  • Oracle Evaluation: The oracle experiment replaces predicted signals with ground-truth values before NMS and evaluates AP on COCO val2017.
  • Oracle Evaluation: 39.2 AP is achieved by original FCOS+ATSS, while ground-truth centerness reaches only about 2.0 AP higher and ground-truth IoU centerness reaches 43.5 AP.
  • Oracle Evaluation: 56.1 AP is achieved with ground-truth boxes even without centerness, while ground-truth classification scores reach 43.1 AP without versus 58.1 AP with centerness.
  • Oracle Evaluation: 74.7 AP is achieved when the ground-truth class score is replaced by ground-truth IoU without centerness, identifying IACS as the most promising selection measure.

4. VarifocalNet

VFNet learns an IoU-aware classification score and combines it with star-shaped box features and refinement in an FCOS-based dense detector. Its training uses Varifocal Loss and a loss function supervising classification, localization, and refined boxes.

  • Architecture: VFNet removes the centerness branch and adds IACS prediction, star-shaped box features, and bounding box refinement to FCOS+ATSS.The resulting detector is called VarifocalNet or VFNet.
  • IACS: IACS assigns the ground-truth class the predicted-box IoU and assigns zero to other classes.This score jointly represents object presence confidence and localization accuracy.
  • Varifocal Loss: Varifocal Loss trains continuous IACS targets with asymmetric weighting: negative examples are down-weighted, while high-quality positive examples receive greater weight.For foreground points, the ground-truth class target is the generated box's IoU; background targets are zero for all classes.
  • Star-shaped representation: The star-shaped representation samples nine fixed points around a box to capture its geometry and nearby context for IACS prediction.The representation uses deformable convolution and supports efficient box refinement.
  • Bounding Box Refinement: Bounding box refinement learns four distance scaling factors that transform an initial box into a refined box closer to ground truth.The initial box is encoded by (l’, t’, r’, b’), and the refined distances are (∆l×l’, ∆t×t’, ∆r×r’, ∆b×b’).
  • Architecture: VFNet’s head uses separate localization and IACS subnetworks over FPN feature maps, with localization producing initial boxes and subsequent refinements.The IACS branch outputs one jointly confidence-and-localization score per class at each spatial location.
  • Loss Function and Inference: VFNet training combines IACS prediction with GIoU losses for initial and refined bounding boxes, using foreground-point normalization and balance weights.The localization target weights use ground-truth IoU for foreground points and zero otherwise; λ0 and λ1 are set to 1.5 and 2.0.

5. Experiments

Experiments on MS COCO evaluate VFNet’s loss, component contributions, state-of-the-art performance, efficiency, extensions, and generality across dense detectors. The proposed modules consistently improve detection performance, with VFNet-X-1200 reaching 55.1 AP.

  • Experimental Setup: VFNet’s experiments use MS COCO 2017, with ablations on val2017 and detector comparisons on test-dev using standard COCO-style AP.Ablations use ResNet-50 and a 1x schedule; test-dev comparisons use varied backbones, 2x training, and multi-scale training.
  • Varifocal Loss: 41.6 AP is achieved with α = 0.75 and γ = 2.0, while similar results above 41.2 AP show robustness across tested hyper-parameters.The selected values are used in subsequent experiments.
  • Varifocal Loss: 0.4 AP is lost without q weighting, decreasing performance from 41.6 AP to 41.2 AP.The comparison evaluates the optimal α and γ setting with and without weighting positive examples by target q.
  • Individual Component Contribution: 39.0 AP with focal loss rises to 40.1 AP with Varifocal Loss, then to 40.7 AP with star-shaped features and 41.6 AP after bounding-box refinement.The ablation attributes gains to the three proposed modules.
  • State-of-the-Art Comparison: ∼2.0 AP separates VFNet from the strong ATSS baseline across backbones, including 46.0 AP versus 43.6 AP with ResNet-101.With Res2Net-101-DCN, VFNet reaches 51.3 AP in the reported comparison.
  • Efficiency: 44.8 AP is achieved at 19.3 FPS, with only small additional computation overhead compared with ATSS.Speed is measured on an Nvidia V100 GPU under a comparison limited to VFNet and ATSS.
  • Generality and Superiority of Varifocal Loss: Varifocal Loss improves RetinaNet, FoveaBox, and ATSS by 0.9 AP and RepPoints by 1.4 AP, outperforming GFL in all tested cases.The loss is also reported to retain a consistent advantage over focal loss and GFL when applied within VFNet.

6. Conclusion

The paper proposes learning an IoU-aware Classification Score to rank detections and develops VarifocalNet to exploit it. On MS COCO, the reported experiments verify the method’s effectiveness and state-of-the-art performance.

  • 6. Conclusion: The paper learns an IoU-aware Classification Score for ranking detections and develops VarifocalNet around that score.The detector combines Varifocal Loss, star-shaped bounding-box features, and bounding-box refinement.
  • 6. Conclusion: Experiments on MS COCO verify the effectiveness of the proposed methods and report state-of-the-art performance among various object detectors.
Loading 2008.13367v2…