Source-linked AI summary
Multivariate Confidence Calibration for Object Detection
Fabian Küppers, Jan Kronenberger, Amirhossein Shantia, Anselm Haselhoff
TL;DR
Object-detector confidence calibration has largely ignored regression information, even though calibration depends on box location and scale. The paper introduces a post-processing framework that integrates classification and regression outputs, measures miscalibration with D-ECE, and reports improved calibration over traditional methods.
Problem
Object-detection calibration methods have not adequately captured confidence dependence on bounding-box location and scale, despite the importance of reliable estimates in safety-critical applications.
Method
The framework performs box-sensitive multivariate post-processing calibration by integrating detector confidence with regression information and defining D-ECE using box properties.
Results
The proposed framework outperforms state-of-the-art calibration methods and provides confidence estimates across different locations and scales.
Takeaways & Limitations
Calibration should account for regression outputs because box-sensitive evaluation reveals detector properties that traditional metrics do not reveal.
Takeaways & Limitations
D-ECE values with different dimensionalities or bin counts cannot be compared directly, so evaluation should use at least the calibration mapping’s dimensionality.
Abstract
from arXiv · showhide
Unbiased confidence estimates of neural networks are crucial especially for safety-critical applications. Many methods have been developed to calibrate biased confidence estimates. Though there is a variety of methods for classification, the field of object detection has not been addressed yet. Therefore, we present a novel framework to measure and calibrate biased (or miscalibrated) confidence estimates of object detection methods. The main difference to related work in the field of classifier calibration is that we also use additional information of the regression output of an object detector for calibration. Our approach allows, for the first time, to obtain calibrated confidence estimates with respect to image location and box scale. In addition, we propose a new measure to evaluate miscalibration of object detectors. Finally, we show that our developed methods outperform state-of-the-art calibration models for the task of object detection and provides reliable confidence estimates across different locations and scales.
1. Introduction
Object detectors can be miscalibrated in ways that depend on bounding-box location and scale, limiting confidence reliability in safety-critical settings. The paper proposes box-sensitive calibration that incorporates regression information alongside classification confidence.
- Calibration error increases near image boundaries and also depends on object width and height, so confidence reliability varies with box properties.These effects are evaluated using D-ECE on COCO detections.
- Standard calibration methods use detection confidence alone, whereas the proposed approach also incorporates location and scale from the regression branch.This extends calibration from scalar confidence to box-sensitive or multivariate calibration.
- Reliable calibration is especially important for safety-critical applications because boundary-localized objects can otherwise receive untrustworthy confidence scores.The paper gives pedestrian detection in road traffic as an example.
- The framework measures and calibrates detectors as a post-processing method without modifying detector training objectives.It is intended to remain agnostic to the detection method and calibration algorithm.
- The study evaluates the framework with pretrained SSD, Faster R-CNN, and R-FCN object detectors.
2. Related Work
Prior work transfers classifier calibration methods to object detection, but these approaches generally ignore dependencies between confidence and regression outputs. Related methods address calibration or localization separately rather than explicitly modeling both together.
- Classifier calibration methods such as histogram binning, temperature scaling, logistic calibration, and beta calibration can be transferred to object detection as post-processing.
- Transferred classifier methods may fail to exploit dependencies between estimated confidence and the detector’s regression output.
- Monte-Carlo-dropout regularization calibrates classifiers during training by using uncertainty to control an additional regularization term.
- IoU-Net and GIoU improve localization-related outputs, but IoU-Net focuses on regression quality and adds a separate location confidence alongside categorical confidence.
- Temperature scaling with an auxiliary output and a modified ECE were proposed for object detection, but neither explicitly captures regression-output properties.
3. Methods for Confidence Calibration
The framework calibrates object-detector confidence after training by integrating classification confidence with bounding-box information in a black-box, multivariate calibration map. It supports independent and dependent distributional models, while exposing trade-offs in modeling correlations and parameter growth.
- 3.1. Object Detectors: Object-detector calibration uses precision as a black-box surrogate for accuracy because recall would require incorporating undetected ground-truth objects without confidence scores.A confidence of 0.9 should correspond to approximately 90 correct detections among 100 predictions under the chosen IoU criterion.
- 3.2. Multivariate Calibration Framework: The framework combines detector confidence with bounding-box regression output to produce box-sensitive calibration without modifying detector training.The calibration map uses confidence and estimated box information, and is agnostic to detector architecture and calibration algorithm.
- 3.2. Multivariate Calibration Framework: The general calibration map applies a logistic function to a combined input containing confidence and box variables, with the logit interpretable through posterior odds or likelihood ratios.Multivariate density functions allow the calibration model to include bounding-box information in the conditional distributions.
- 3.3. Conditional Independent Calibration: Conditional independent calibration simplifies the joint model by combining univariate densities, but it neglects dependencies and correlations among confidence, location, and scale.Independent logistic and beta variants extend familiar classifier-calibration methods to each box quantity separately.
- 3.4. Conditional Dependent Calibration: Dependent logistic calibration uses multivariate normal distributions for correct and incorrect predictions, allowing correlations among calibration variables to be modeled.Its parameters include class-conditional mean vectors and covariance matrices, requiring 2(K^2 + K) + 1 optimized parameters.
- 3.4. Conditional Dependent Calibration: The dependent formulation captures positive correlations but neglects negative correlations, while the parameter count can grow exponentially with the number of dimensions.The number of modeled components must therefore be selected carefully for each dimension.
4. Measuring Miscalibration of Detectors
Standard ECE can miss location- and box-dependent detector miscalibration, so the paper defines D-ECE by incorporating confidence and bounding-box properties into the evaluation.
- Standard ECE may rank global calibration better even when location-dependent calibration improves regional confidence behavior.Because standard ECE bins confidence independently of box properties, it cannot capture spatially varying precision.
- D-ECE measures expected deviation between observed precision and confidence conditional on detector box properties.Its value depends on IoU and the number of additional box variables included.
- Multidimensional D-ECE partitions confidence and each box-property dimension into equally spaced bins, producing Q^K total bins.Each bin compares average precision with average confidence.
- Calibration methods using different box information should be compared with D-ECE of matching dimensionality rather than standard ECE.Different dimensions or bin counts change sample migration and make scores directly incomparable.
5. Experiments
Experiments on pretrained detectors and COCO evaluate calibration across confidence, location, scale, and full box information. Histogram binning is strongest for confidence-only calibration, while multivariate logistic and beta calibration perform better when box variables are jointly used.
- 5.1. Evaluation Protocol: The evaluation uses pretrained detectors on a random split of COCO validation 2017, with non-max suppression probability threshold 0.3 and IoU threshold 0.6.The COCO test set was unavailable because its annotations were not public.
- 5.2. Results: Multivariate logistic and beta calibration outperform histogram binning when calibration uses all box information, especially under dependent box variables.Table 1 reports D-ECE scores across confidence-only, position, scale, and full-dimensional settings.
- 5.2. Results: Histogram binning achieves the lowest error for global confidence-only calibration and fair results when using either position or scale.Its advantage decreases as calibration dimensionality increases.
- 5.2. Results: Using more bins improves resolution but reduces samples per bin, making D-ECE estimates less accurate in sparse regions.Figure 4 specifically warns that low relative width and high relative height have too few samples for reliable estimates.
6. Conclusion
The paper extends object-detector calibration by incorporating regression outputs, especially location and scale, into both measurement and calibration. This framework outperforms state-of-the-art calibration methods and provides additional insight into detector miscalibration.
- Figure 5 shows that dependent logistic calibration can substantially change confidence estimates for detections sharing confidence but differing in position and scale.The brown detection changes slightly, while the remaining detections undergo major changes.
- Including bounding-box information enables the framework to outperform state-of-the-art calibration methods for object detection.The additional information comes from the detector’s regression output.
- Location- and scale-dependent calibration is needed because traditional confidence calibration does not reveal miscalibration tied to box properties.The framework measures miscalibration with respect to the box regression output.
- Multidimensional histogram binning performs well for low-dimensional calibration mappings, whereas multivariate logistic or beta calibration performs better for high-dimensional mappings with fewer parameters.The method choice depends on the dimensionality of the calibration mapping.