PaperlayerView exact revision on arXiv

Source-linked AI summary

VarifocalNet: An IoU-aware Dense Object Detector

Haoyang Zhang, Ying Wang, Feras Dayoub, Niko Sünderhauf

arXiv:2008.13367v2cs.CV

TL;DR

Dense detectors need a reliable way to rank candidate detections because classification and separately combined localization scores are imperfect. The paper learns an IoU-aware classification score using Varifocal Loss and star-shaped features, builds VFNet, and reports consistent gains over strong baselines on COCO. Its best reported model reaches 55.1 AP on COCO test-dev.

  • Problem

    Dense detectors lack a reliable ranking score because classification scores and combinations with predicted localization scores do not consistently reflect detection quality.

    Source: §1 · p. 1

  • Method

    The paper learns IACS, trains it with asymmetric Varifocal Loss, and uses star-shaped box features for IACS prediction and bounding-box refinement in VFNet.

    Source: §1 · p. 2, §2 · p. 3, §4.1 · p. 4

  • Results

    ∼2.0 AP improvement over a strong baseline is reported across backbones, and VFNet-X-1200 reaches 55.1 AP on COCO test-dev.

    Source: §1 · p. 2

  • Takeaways & Limitations

    IACS provides the ranking criterion used by VFNet to exploit high-quality detections within dense candidate pools.

    Source: §3 · p. 3, §6 · p. 8

Abstract

from arXiv · show

Accurately ranking the vast number of candidate detections is crucial for dense object detectors to achieve high performance. Prior work uses the classification score or a combination of classification and predicted localization scores to rank candidates. However, neither option results in a reliable ranking, thus degrading detection performance. In this paper, we propose to learn an Iou-aware Classification Score (IACS) as a joint representation of object presence confidence and localization accuracy. We show that dense object detectors can achieve a more accurate ranking of candidate detections based on the IACS. We design a new loss function, named Varifocal Loss, to train a dense object detector to predict the IACS, and propose a new star-shaped bounding box feature representation for IACS prediction and bounding box refinement. Combining these two new components and a bounding box refinement branch, we build an IoU-aware dense object detector based on the FCOS+ATSS architecture, that we call VarifocalNet or VFNet for short. Extensive experiments on MS COCO show that our VFNet consistently surpasses the strong baseline by $\sim$2.0 AP with different backbones. Our best model VFNet-X-1200 with Res2Net-101-DCN achieves a single-model single-scale AP of 55.1 on COCO test-dev, which is state-of-the-art among various object detectors.Code is available at https://github.com/hyz-xmaster/VarifocalNet .

1. Introduction

Dense detectors must rank many candidate boxes, but classification scores and multiplied localization scores do not reliably reflect localization quality. VFNet addresses this by learning an IoU-aware classification score with Varifocal Loss and star-shaped box features, achieving consistently stronger COCO performance.

  • Classification scores can misrepresent localization accuracy, causing accurately localized detections with low scores to be removed by NMS.Source: §1 · p. 1
  • Multiplying classification and separately predicted localization scores can produce a worse ranking basis and adds computation through an extra branch.Source: §1 · p. 1
  • IACS jointly represents object-class presence and the localization accuracy of a generated bounding box.Source: §1 · p. 2
  • VFNet combines IACS ranking, Varifocal Loss, star-shaped box features, and bounding-box refinement within an FCOS+ATSS detector.Source: §1 · p. 2
  • Varifocal Loss down-weights negative examples while up-weighting high-quality positives to focus training on prime detections.Source: §1 · p. 2
  • Star-shaped features use nine sampling points to capture box geometry and nearby context for IACS prediction and efficient box refinement.Source: §1 · p. 2
  • ∼2.0 AP improvement over a strong baseline is reported across backbones, while VFNet-X-1200 reaches 55.1 AP on COCO test-dev.Source: §1 · p. 2

2. Related Work

Prior detectors use anchors, point-based representations, or separate localization-ranking scores, each balancing accuracy against efficiency. VFNet instead uses a fixed nine-point star representation and asymmetric Varifocal Loss, while extending related joint-quality scoring with box refinement.

Object Detection

  • Object detectors include anchor-based two-stage, multi-stage, and one-stage methods, alongside anchor-free key-point and point-based approaches.Source: §2 · p. 2

Detection Ranking Measures

  • IoU-aware and centerness-based methods combine localization estimates with classification scores for ranking, but the resulting products can remain imperfect.Source: §2 · p. 2
  • Predicting only IACS avoids an additional localization network and the potentially worse ranking basis from multiplying imperfect scores.Source: §2 · p. 2

Encoding the Bounding Box

  • Dense detectors generally use efficient point features, while learned semantic-point representations add localization difficulty and computation burden.Source: §2 · p. 2
  • VFNet’s nine fixed sampling points encode box geometry and spatial context while remaining simple and efficient.Source: §2 · p. 3

Generalized Focal Loss

  • Unlike GFL’s equal weighting of positive and negative examples, Varifocal Loss weights them asymmetrically and VFNet additionally refines boxes.Source: §2 · p. 3

3. Motivation

The FCOS+ATSS analysis tests how oracle classification, localization, and centerness signals affect ranking. It shows that IoU-aware classification is the most promising selection measure because high-quality boxes already exist among candidates.

FCOS+ATSS Analysis

  • FCOS+ATSS predicts classification, box distances, and centerness, with centerness multiplied by classification for NMS ranking.Source: §3 · p. 3

Oracle Evaluation

  • The oracle experiment replaces predicted signals with ground-truth values before NMS and evaluates AP on COCO val2017.Source: §3 · p. 3
  • 39.2 AP is achieved by original FCOS+ATSS, while ground-truth centerness reaches only about 2.0 AP higher and ground-truth IoU centerness reaches 43.5 AP.Source: §3 · p. 3
  • 56.1 AP is achieved with ground-truth boxes even without centerness, while ground-truth classification scores reach 43.1 AP without versus 58.1 AP with centerness.Source: §3 · p. 3
  • 74.7 AP is achieved when the ground-truth class score is replaced by ground-truth IoU without centerness, identifying IACS as the most promising selection measure.Source: §3 · p. 3

4. VarifocalNet

VFNet learns an IoU-aware classification score and combines it with star-shaped box features and refinement in an FCOS-based dense detector. Its training uses Varifocal Loss and a loss function supervising classification, localization, and refined boxes.

Architecture

  • VFNet removes the centerness branch and adds IACS prediction, star-shaped box features, and bounding box refinement to FCOS+ATSS.The resulting detector is called VarifocalNet or VFNet.Source: §4 · p. 4, §4.5 · p. 5

IACS

  • IACS assigns the ground-truth class the predicted-box IoU and assigns zero to other classes.This score jointly represents object presence confidence and localization accuracy.Source: §4.1 · p. 4, §4.5 · p. 5

Varifocal Loss

  • Varifocal Loss trains continuous IACS targets with asymmetric weighting: negative examples are down-weighted, while high-quality positive examples receive greater weight.For foreground points, the ground-truth class target is the generated box's IoU; background targets are zero for all classes.Source: §4.2 · p. 4

Star-shaped representation

  • The star-shaped representation samples nine fixed points around a box to capture its geometry and nearby context for IACS prediction.The representation uses deformable convolution and supports efficient box refinement.Source: §4.3 · p. 4, Figure 3 · p. 5

Bounding Box Refinement

  • Bounding box refinement learns four distance scaling factors that transform an initial box into a refined box closer to ground truth.The initial box is encoded by (l’, t’, r’, b’), and the refined distances are (∆l×l’, ∆t×t’, ∆r×r’, ∆b×b’).Source: §4.4 · pp. 4–5

Architecture

  • VFNet’s head uses separate localization and IACS subnetworks over FPN feature maps, with localization producing initial boxes and subsequent refinements.The IACS branch outputs one jointly confidence-and-localization score per class at each spatial location.Source: §4.5 · p. 5, Figure 3 · p. 5

Loss Function and Inference

  • VFNet training combines IACS prediction with GIoU losses for initial and refined bounding boxes, using foreground-point normalization and balance weights.The localization target weights use ground-truth IoU for foreground points and zero otherwise; λ0 and λ1 are set to 1.5 and 2.0.Source: §4.6 · p. 5

5. Experiments

Experiments on MS COCO evaluate VFNet’s loss, component contributions, state-of-the-art performance, efficiency, extensions, and generality across dense detectors. The proposed modules consistently improve detection performance, with VFNet-X-1200 reaching 55.1 AP.

Experimental Setup

  • VFNet’s experiments use MS COCO 2017, with ablations on val2017 and detector comparisons on test-dev using standard COCO-style AP.Ablations use ResNet-50 and a 1x schedule; test-dev comparisons use varied backbones, 2x training, and multi-scale training.Source: §5 · p. 6

Varifocal Loss

  • 41.6 AP is achieved with α = 0.75 and γ = 2.0, while similar results above 41.2 AP show robustness across tested hyper-parameters.The selected values are used in subsequent experiments.Source: §5.1.1 · p. 6
  • 0.4 AP is lost without q weighting, decreasing performance from 41.6 AP to 41.2 AP.The comparison evaluates the optimal α and γ setting with and without weighting positive examples by target q.Source: §5.1.1 · p. 6

Individual Component Contribution

  • 39.0 AP with focal loss rises to 40.1 AP with Varifocal Loss, then to 40.7 AP with star-shaped features and 41.6 AP after bounding-box refinement.The ablation attributes gains to the three proposed modules.Source: §5.1.2 · p. 6

State-of-the-Art Comparison

  • ∼2.0 AP separates VFNet from the strong ATSS baseline across backbones, including 46.0 AP versus 43.6 AP with ResNet-101.With Res2Net-101-DCN, VFNet reaches 51.3 AP in the reported comparison.Source: Table 4 · p. 7, §5.2 · p. 8

Efficiency

  • 44.8 AP is achieved at 19.3 FPS, with only small additional computation overhead compared with ATSS.Speed is measured on an Nvidia V100 GPU under a comparison limited to VFNet and ATSS.Source: §5.2 · p. 8

Generality and Superiority of Varifocal Loss

  • Varifocal Loss improves RetinaNet, FoveaBox, and ATSS by 0.9 AP and RepPoints by 1.4 AP, outperforming GFL in all tested cases.The loss is also reported to retain a consistent advantage over focal loss and GFL when applied within VFNet.Source: §5.4 · p. 8

6. Conclusion

The paper proposes learning an IoU-aware Classification Score to rank detections and develops VarifocalNet to exploit it. On MS COCO, the reported experiments verify the method’s effectiveness and state-of-the-art performance.

6. Conclusion

  • The paper learns an IoU-aware Classification Score for ranking detections and develops VarifocalNet around that score.The detector combines Varifocal Loss, star-shaped bounding-box features, and bounding-box refinement.Source: §6 · p. 8
  • Experiments on MS COCO verify the effectiveness of the proposed methods and report state-of-the-art performance among various object detectors.Source: §6 · p. 8

Related papers

Loading 2008.13367v2…