Source-linked AI summary
IoU-aware Single-stage Object Detector for Accurate Localization
Shengkai Wu, Xiaoping Li, Xinggang Wang
TL;DR
Single-stage detectors can assign classification scores that poorly reflect localization accuracy, reducing detection quality. The paper adds IoU prediction and combines it with classification confidence, improving results across COCO and PASCAL VOC while revealing substantial room to improve IoU prediction.
Problem
Low correlation between classification scores and localization accuracy harms single-stage detector performance because classification and localization are trained with independent objectives.
Method
An IoU prediction head is jointly trained with classification and regression, and predicted IoU is multiplied by classification score for NMS and AP ranking.
Results
The method substantially improves localization accuracy on COCO and PASCAL VOC, including COCO AP gains of 1.7% ∼1.9% and AP75 gains of 2.2% ∼2.5%.
Takeaways & Limitations
IoU-aware confidence alleviates the mismatch between detection confidence and localization accuracy in single-stage detectors.
Takeaways & Limitations
Predicted IoU remains far less accurate than ground-truth IoU, leaving substantial potential for further performance improvement.
Abstract
from arXiv · showhide
Due to the simpleness and high efficiency, single-stage object detectors have been widely applied in many computer vision applications . However, the low correlation between the classification score and localization accuracy of the predicted detections has severely hurt the localization accuracy of models. In this paper, IoU-aware single-stage object detector is proposed to solve this problem. Specifically, IoU-aware single-stage object detector predicts the IoU for each detected box. Then the classification score and predicted IoU are multiplied to compute the final detection confidence, which is more correlated with the localization accuracy. The detection confidence is then used as the input of the subsequent NMS and COCO AP computation, which will substantially improve the localization accuracy of models. Sufficient experiments on COCO and PASCAL VOC datasets demonstrate the effectiveness of IoU-aware single-stage object detector on improving model's localization accuracy. Without whistles and bells, the proposed method can substantially improve AP by $1.7\%\sim1.9\%$ and AP75 by $2.2\%\sim2.5\%$ on COCO \textit{test-dev}. On PASCAL VOC, the proposed method can substantially improve AP by $2.9\%\sim4.4\%$ and AP80, AP90 by $4.6\%\sim10.2\%$. Code is available here: {https://github.com/ShengkaiWu/IoU-aware-single-stage-object-detector}.
1. Introduction
Single-stage detectors are efficient but suffer when classification scores poorly reflect localization accuracy, harming NMS selection and AP ranking. The paper proposes an IoU-aware detector to address this mismatch while preserving efficiency.
- Motivation: Low correlation between classification score and localization accuracy severely hurts single-stage detectors.Independent classification and localization objectives leave classification unaware of localization quality.
- Motivation: High-scoring low-IoU detections can rank before lower-scoring high-IoU detections, reducing AP at high IoU thresholds.The paper gives the ranking C2, B2, A2 as worse than A2, B2, C2.
- Motivation: High-scoring but poorly localized detections can suppress accurate boxes during NMS.Figure 1 illustrates inaccurate boxes A2, B2, and C2 suppressing accurate boxes A1, B1, and C1.
- Contribution: The proposed detector predicts IoU for each regressed anchor and multiplies it with the classification score for NMS and AP ranking.The jointly trained IoU prediction head makes detection confidence more correlated with localization accuracy.
- Contribution: The method is presented as simple and efficient, improving localization accuracy without sacrificing efficiency.The paper also reports extensive experiments and analyses of IoU prediction and its performance bound.
2. Related Work
Related work addresses the relationship between confidence and localization, more accurate localization, and anchor-free single-stage detection. The proposed method differs by using an IoU prediction head in an anchor-based detector.
- Correlation between classification score and localization accuracy: Prior methods improve confidence localization alignment by predicting IoU or related quality measures and using them in ranking.Examples include Fitness NMS and MS R-CNN, which combine predicted IoU with classification confidence.
- Accurate object localization: Other approaches improve localization through iterative or multi-step box refinement and increasing IoU thresholds.The paper discusses Multi-region detector, Cascade R-CNN, and RefineDet in this context.
- Anchor-free single-stage object detectors: Anchor-free detectors directly predict boxes and object confidences at feature-map pixels without predefined anchors.DenseBox is cited as an example using four distances from each pixel to box boundaries.
- Anchor-free single-stage object detectors: The IoU-aware detector predicts each detection’s IoU to suppress poorly localized detections but remains anchor-based.This distinguishes it from anchor-free single-stage detectors.
3. Method
The detector modifies RetinaNet by adding a lightweight IoU prediction head alongside regression. It trains this head with the other tasks and uses predicted IoU with classification scores to produce inference confidence.
- IoU-aware single-stage object detector: The architecture retains RetinaNet’s backbone and FPN while adding a parallel IoU prediction head to the regression branch.The classification, regression, and IoU heads each use a single 3*3 convolution layer.
- IoU-aware single-stage object detector: The IoU prediction head uses sigmoid activation so predicted IoU lies in [0, 1].This design adds negligible computational burden according to the paper.
- Training: Training uses focal loss for classification, smooth L1 loss for regression, and BCE for IoU prediction on positive examples.The target IoU is computed between each regressed positive box and its corresponding ground-truth box.
- Inference: At inference, final confidence combines classification score and predicted IoU before subsequent NMS ranking.Parameter α controls the relative contribution of the two factors.
4. Experiments
Experiments on COCO and PASCAL VOC evaluate IoU-aware RetinaNet across losses, confidence computation, gradient propagation, and the gap between predicted and ground-truth IoU. The method improves high-IoU localization metrics, while predicted-IoU accuracy remains an improvement opportunity.
- Main Results: COCO test-dev experiments show AP gains of 1.7% ∼1.9% and AP75 gains of 2.2% ∼2.5% over baselines across backbones.AP50 decreases by 0.7% ∼2.3%, indicating that the principal improvement is at higher localization quality.
- IoU Prediction Loss: Binary cross-entropy loss produces better performance than L2 loss for training the IoU prediction head and is used thereafter.The comparison is conducted with detection confidence computed by directly multiplying classification score and predicted IoU.
- Detection Confidence Computation: With α=0.4 or 0.5, AP reaches 35.5%, improving 1.1% over baseline, while AP70 ∼AP90 increase by 1.3% ∼2.6%.Lower α increases the contribution of predicted IoU and improves performance at higher IoU thresholds.
- Ablation Studies: Computing the gradient of LIoU with respect to predicted IoU improves AP by 0.6% and makes the regression head more powerful for accurate localization.At α=0.4, AP, AP80, and AP90 improve by 1.7%, 3.7%, and 5.4%, respectively, over baseline.
- Ablation Studies: On PASCAL VOC, computing the LIoU gradient improves AP by 2.9% ∼4.4% and AP80, AP90 by 4.6% ∼10.2%.The conclusions are consistent with COCO experiments, supporting the method’s generalization to another dataset.
- Discussions: Ground-truth IoU produces stronger confidence-localization correlation than predicted IoU, revealing substantial room to improve IoU prediction accuracy.The predicted-IoU detector slightly alleviates the mismatch, whereas ground-truth IoU makes confidence strongly correlated with localization accuracy.
5. Conclusions
The paper shows that low correlation between classification score and localization accuracy hurts single-stage detectors, and introduces IoU-aware scoring to address this mismatch. Experiments show improved performance, especially localization accuracy, while a gap between predicted and ground-truth IoU remains.
- Low correlation between classification score and localization accuracy severely hurts single-stage detectors’ localization accuracy.
- The IoU-aware detector adds an IoU prediction head and multiplies predicted IoU by classification score to rank detections during NMS and AP computation.
- Experiments on MS COCO and PASCAL VOC show substantial performance improvements, especially in localization accuracy.
- A large gap between predicted IoU and ground-truth IoU substantially limits the method’s performance and motivates future research.