Source-linked AI summary
Acquisition of Localization Confidence for Accurate Object Detection
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, Yuning Jiang
TL;DR
Object detectors lack localization confidence, so classification-based NMS and repeated regression can mishandle box quality. IoU-Net predicts each detected box’s IoU with matched ground truth and uses that prediction for NMS and optimization-based refinement. Experiments on MS-COCO demonstrate effectiveness and compatibility with multiple detection models.
Problem
Object detectors lack localization confidence despite using classification probabilities as confidence, allowing accurately localized boxes to be suppressed and repeated regression to become non-monotonic.
Method
IoU-Net predicts the IoU between each detected bounding box and its matched ground-truth box, using predicted IoU for NMS ranking and as the objective of gradient-based refinement.
Results
IoU-guided NMS outperforms traditional NMS and Soft-NMS, while IoU-Net improves localization across several state-of-the-art detectors on MS-COCO.
Takeaways & Limitations
Localization-aware confidence can preserve accurately localized detections and support interpretable, optimization-based refinement compatible with existing CNN detectors.
Takeaways & Limitations
IoU estimation becomes less accurate for lower-IoU boxes, which degrades downstream refinement and suppression; sampling more such boxes during training only partially helps.
Abstract
from arXiv · showhide
Modern CNN-based object detectors rely on bounding box regression and non-maximum suppression to localize objects. While the probabilities for class labels naturally reflect classification confidence, localization confidence is absent. This makes properly localized bounding boxes degenerate during iterative regression or even suppressed during NMS. In the paper we propose IoU-Net learning to predict the IoU between each detected bounding box and the matched ground-truth. The network acquires this confidence of localization, which improves the NMS procedure by preserving accurately localized bounding boxes. Furthermore, an optimization-based bounding box refinement method is proposed, where the predicted IoU is formulated as the objective. Extensive experiments on the MS-COCO dataset show the effectiveness of IoU-Net, as well as its compatibility with and adaptivity to several state-of-the-art object detectors.
1 Introduction
Modern object detectors provide classification confidence but lack a corresponding localization confidence, creating problems for suppression and iterative refinement. IoU-Net predicts each box’s IoU with matched ground truth and uses it to guide NMS and optimization-based refinement.
- Object detectors naturally obtain classification confidence from class probabilities but lack an explicit confidence measure for localization accuracy.
- Misaligned classification confidence can cause accurately localized bounding boxes to be suppressed during traditional NMS.
- Iterative bounding box regression can exhibit non-monotonic behavior and degrade localization when applied repeatedly.
- IoU-Net predicts the IoU between detected boxes and matched ground-truth boxes, providing a localization criterion analogous to classification confidence.
- IoU-guided NMS ranks boxes using predicted IoU, while optimization-based refinement maximizes predicted IoU and is compatible with various CNN-based detectors.
2 Delving into object localization
The section examines how classification confidence misaligns with localization accuracy and how iterative regression can become non-monotonic. IoU-guided NMS and optimization-based refinement address these issues by using localization-aware signals.
- 2 Delving into object localization: A standard FPN detector study on MS-COCO investigates misaligned confidence and non-monotonic bounding box regression.
- Misaligned classification and localization accuracy: Pearson correlation between classification confidence and localization IoU is only 0.217 for detected boxes with IoU > 0.5.
- Misaligned classification and localization accuracy: Traditional NMS can suppress more than half of detected bounding boxes with IoU > 0.9 because it ranks boxes by classification confidence.
- Non-monotonic bounding box regression: Optimization-based refinement further improves AP by 0.8% after multiple iterations, whereas regression-based refinement slightly reduces AP.
- Non-monotonic bounding box regression: Iterative bounding box regression improves AP initially but later becomes non-monotonic, complicating localization control and interpretation.
3 IoU-Net
IoU-Net predicts localization confidence as IoU and uses it to guide both suppression and bounding-box refinement. Its standalone predictor integrates with existing RoI-based detectors and FPN-based architectures.
- 3.4 Integrating IoU-Net into object detectors: The IoU predictor operates in parallel with the R-CNN branch on FPN features and uses Precise RoI Pooling, forming a standalone module compatible with existing RoI-based detectors.The architecture replaces the original RoI Pooling layer and can be integrated into detectors such as FPN.
- 3.1 Learning to predict IoU: IoU-Net predicts the IoU between each detected bounding box and its matched ground-truth box as a localization-confidence estimate.The predictor is trained using randomized transformations of ground-truth boxes, retaining candidates with IoU at least Ωtrain = 0.5.
- 3.2 IoU-guided NMS: IoU-guided NMS ranks detected boxes by predicted IoU rather than classification confidence and updates classification scores for overlapping boxes.The procedure selects the box with the highest predicted IoU, then applies a clustering-like score update to boxes whose overlap exceeds Ωnms.
- 3.2 IoU-guided NMS: Quantitative experiments report that IoU-guided NMS outperforms traditional NMS and Soft-NMS while improving several state-of-the-art object detectors.The reported benefit is attributed to resolving the misalignment between classification confidence and localization accuracy.
- 3.3 Bounding box refinement as an optimization procedure: The refinement objective replaces conventional coordinate-distance criteria with a criterion based on overlap between transformed detections and target boxes.Traditional formulations use smooth-L1 coordinate distance or −ln(IoU), whereas IoU-Net directly estimates IoU(boxdet, boxgt).
- 3.3 Bounding box refinement as an optimization procedure: IoU-Net supports optimization-based bounding-box refinement by using predicted IoU as the objective during inference.The refinement iteratively updates boxes using the gradient of the predicted IoU and stops when improvement is small or localization degenerates beyond a tolerance.
3. PrRoI Pooling
Precise RoI Pooling avoids coordinate quantization and provides continuous gradients for optimization-based bounding box refinement. The IoU predictor integrates with FPN-based detection pipelines alongside the R-CNN branch.
- Bounding Box Refinement: Gradient ascent uses predicted IoU as the optimization objective to iteratively refine bounding box coordinates.The predicted IoU also serves as an interpretable localization-confidence indicator.
- Precise RoI Pooling: Precise RoI Pooling avoids coordinate quantization and has a continuous gradient with respect to bounding box coordinates.It uses bilinear interpolation to treat the discrete feature map as continuous at arbitrary coordinates.
- Precise RoI Pooling: PrRoI Pooling computes pooling over a bin using a two-order integral of the continuous feature map.The bin is specified by continuous top-left and bottom-right coordinates.
- Precise RoI Pooling: Unlike RoI Align’s fixed four sampled points, PrRoI Pooling directly computes the integral and adapts to the bin size.Traditional RoI Pooling first quantizes continuous coordinates, while RoI Align samples N = 4 points.
- Joint Training: The IoU predictor integrates into standard FPN pipelines for end-to-end training and inference, operating in parallel with the R-CNN branch.The two branches use the same visual feature from the backbone, with Precise RoI Pooling replacing the original RoI Pooling layer.
4 Experiments
Experiments on MS-COCO show that IoU-guided NMS preserves better-localized boxes, while IoU-based refinement improves multiple CNN detectors with modest overhead. Joint IoU prediction further improves AP, though estimation errors reduce performance at lower IoU.
- 4.1 IoU-guided NMS: IoU-guided NMS improves localization-sensitive detection performance over baselines, especially at high matching IoU thresholds such as AP90.The method improves results by improving localization rather than merely preserving more boxes.
- 4.1 IoU-guided NMS: At matching IoU Ωtest = 0.9, recall is 18.7% after traditional NMS, 28.9% with IoU-NMS, and 39.7% with the No-NMS upper bound.The gap between IoU-guided NMS and other methods increases as localization accuracy requirements become stricter.
- 4.2 Optimization-based bounding box refinement: Optimization-based box refinement is compatible with most CNN-based object detectors and further improves their localization accuracy.For Cascade R-CNN, refinement increases AP90 by 2.8% and overall AP by 0.8%.
- 4.3 Joint training: The final ResNet101-FPN system reaches 40.6% AP versus a 38.5% baseline, an improvement of 2.1%.The experiments report improved detection performance with tolerable computation overhead.
- 4.3 Joint training: IoU estimation is less accurate for lower-IoU boxes, which degrades downstream refinement and suppression and contributes to inferior AP50 results.Sampling more low-IoU boxes during training partially alleviates this issue.
5 Conclusion
The paper introduces IoU-Net to acquire localization confidence by predicting box–ground-truth IoU. This supports IoU-guided NMS and optimization-based refinement, with MS-COCO experiments demonstrating effectiveness across detection models.
- 5 Conclusion: IoU-Net predicts IoU with matched ground-truth boxes to provide localization confidence for detected boxes.The predicted confidence supports accurate object localization.
- 5 Conclusion: IoU-guided NMS uses predicted IoU to prevent accurately localized boxes from being suppressed.The method addresses misalignment between classification and localization confidence.
- 5 Conclusion: IoU-Net can be integrated into a broad set of detection models to improve localization accuracy.Experiments on MS-COCO demonstrate its effectiveness and potential in practical applications.
- 5 Conclusion: The paper formulates bounding box refinement as an optimization problem and reports that the proposed solution surpasses regression-based methods.The conclusion presents this optimization view as an additional perspective on box refinement.