PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection

Dimity Miller, Feras Dayoub, Michael Milford, Niko Sünderhauf

arXiv:1809.06006v3cs.CV

TL;DR

Object-detection uncertainty estimates require sampled bounding boxes to be associated, and the influence of affinity and clustering choices had not been investigated in depth. The paper compares affinity–clustering combinations and finds that the right choice can improve uncertainty effectiveness and detection performance.

  • Problem

    The influence of affinity measures and clustering techniques on associating sampled boxes had not been analyzed, despite the need for reliable uncertainty in open-set conditions.

    Source: §III · p. 2

  • Method

    The study compares three spatial and two semantic affinity measures combined with four clustering methods for MC Dropout with SSD.

    Source: §I · p. 1, §VII · p. 6

  • Results

    The correct affinity–clustering combinations can improve classification and spatial uncertainty effectiveness as well as object-detection performance.

    Source: §VII · p. 6

  • Takeaways & Limitations

    Analyzing affinity and clustering choices is an important step toward robust visual object detectors for open-set robotic applications.

    Source: §III · p. 2

Abstract

from arXiv · show

There has been a recent emergence of sampling-based techniques for estimating epistemic uncertainty in deep neural networks. While these methods can be applied to classification or semantic segmentation tasks by simply averaging samples, this is not the case for object detection, where detection sample bounding boxes must be accurately associated and merged. A weak merging strategy can significantly degrade the performance of the detector and yield an unreliable uncertainty measure. This paper provides the first in-depth investigation of the effect of different association and merging strategies. We compare different combinations of three spatial and two semantic affinity measures with four clustering methods for MC Dropout with a Single Shot Multi-Box Detector. Our results show that the correct choice of affinity-clustering combination can greatly improve the effectiveness of the classification and spatial uncertainty estimation and the resulting object detection performance. We base our evaluation on a new mix of datasets that emulate near open-set conditions (semantically similar unknown classes), distant open-set conditions (semantically dissimilar unknown classes) and the common closed-set conditions (only known classes).

I. INTRODUCTION

Open-set deployment exposes uncertainty and robustness challenges for CNNs, while object detection makes sampling-based uncertainty especially difficult because detections must be associated and merged. The paper investigates these choices and introduces an evaluation protocol and strategy for improving uncertainty estimation and detection performance.

I. INTRODUCTION

  • CNNs are described as unable to measure their lack of knowledge, while open-set testing with unseen classes is associated with decreased performance.Source: §I · p. 1
  • Object detection samples cannot simply be averaged: their bounding boxes must be correctly associated and merged to estimate uncertainty across multiple objects.Source: §I · p. 1
  • Earlier sampling-based object-detection work did not assess how association and merging choices affect detection performance or uncertainty effectiveness.Source: §I · p. 1
  • The paper investigates affinity measures and clustering techniques for associating bounding-box samples with MC Dropout and SSD.Source: §I · p. 1
  • The authors establish an evaluation protocol for spatial and classification uncertainty and propose a strategy intended to improve uncertainty measures, particularly for robotics.Source: §I · p. 1

II. RELATED WORK

The related work describes object-detection architectures and uncertainty approaches, then identifies a gap: prior open-set detection studies had not evaluated how affinity and clustering choices affect performance.

II. RELATED WORK

  • SSD and YOLO, unify detection and proposal generation in one network branch; this paper uses SSD.Source: §II · p. 2
  • Closed-set deployment assumes known classes match training, whereas robots operating in open-set environments encounter objects absent from training data.Source: §II · p. 2
  • Open-set uncertainty approaches include calibration, Bayesian deep learning with MC Dropout or MC Batch Normalization, and Deep Ensembles.Source: §II · p. 2
  • Sampling-based object detection requires affinity-based clustering, and earlier open-set detection studies had not analyzed how affinity and clustering choices affect performance.Source: §II · p. 2

III. SAMPLING-BASED OBJECT DETECTION

Sampling-based detection groups detections from multiple passes into object observations, then analyzes how affinity and clustering choices affect detector performance.

III. SAMPLING-BASED OBJECT DETECTION

  • Detections from multiple samples are grouped by affinity, clustered, and converted into observations by averaging bounding boxes and softmax-score distributions for groups of at least two detections.Source: §III · p. 2
  • The analysis focuses on how affinity measures and clustering techniques affect overall object-detection performance, a relationship the paper says had not yet been analyzed.Source: §III · p. 2

IV. EVALUATED AFFINITY AND CLUSTERING METHODS

  • The evaluated approaches compare combinations of spatial and semantic affinity measures with clustering methods.Source: §IV · p. 2

A. Affinity Measures

The paper compares spatial and semantic affinities and clustering methods, including BSAS variants, to associate sampled detections.

A. Affinity Measures

  • Spatial affinity compares bounding-box location and shape using IoU, PAC, or EAC, while semantic affinity uses matching winning labels or KL divergence between score distributions.Source: A. Affinity Measures · p. 3

B. Clustering Techniques

  • The paper distinguishes clustering methods by intra-sample exclusivity, which prevents detections from the same sample being assigned to one observation.Source: B. Clustering Techniques · p. 3

1) Basic Sequential Algorithmic Scheme (BSAS):

3) Hungarian Method:

The paper evaluates Hungarian and HDBSCAN clustering for associating sampled detections, alongside the experimental setup for measuring uncertainty with MC Dropout and SSD.

3) Hungarian Method:

  • Hungarian clustering initializes clusters from the first sample, assigns later detections through a cost matrix, and creates clusters for unassigned detections when detections outnumber clusters.Spatial costs can use negated IoU, PAC, or EAC; semantic affinity can add a KL cost or prohibit different winning class labels.Source: 3) Hungarian Method · p. 3

4) Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN):

  • HDBSCAN clusters spatial inputs without requiring an estimated cluster count and is described as robust to parameter selection, varying densities, and noisy data.The tested inputs were bounding-box centroids, top-left corners, or distances from corners to image boundaries.Source: 4) Hierarchical Density-Based Spatial… · p. 3

A. Experimental Setup

  • Classification uncertainty is the entropy of the final softmax distribution, while spatial uncertainty is the total variance of box coordinates within a detection cluster.Detections above threshold δ are rejected and those at or below δ are accepted; correct detections should be accepted and incorrect detections rejected.Source: A. Experimental Setup · p. 3
  • The comparisons include SSD without sampling and BSAS IoU 0.95 baselines, plus BSAS, exclusive BSAS, Hungarian, and HDBSCAN variations.BSAS variants vary thresholds and affinity combinations, Hungarian varies spatial and semantic affinities, and HDBSCAN varies its spatial input representation.Source: A. Experimental Setup · p. 3
  • The experiments use MC Dropout with SSD, fine-tuning dropout layers on the final two VGG16 convolutional layers at probability 0.5 and collecting 20 forward-pass samples.SSD was trained on PASCAL VOC 2007 and 2012; detections were retained only when their label matched the winning softmax class.Source: A. Experimental Setup · p. 4

B. Performance Metrics

The section distinguishes object-detection performance from uncertainty effectiveness, using mAP and UE as the two overall metrics.

B. Performance Metrics

  • mAP measures correct classification and localization of known objects at IoU ≥0.5 while minimizing incorrect detections and their confidence scores.A perfect mAP score is 100%.Source: 1) Mean Average Precision (mAP) · p. 4, B. Performance Metrics · p. 4
  • Uncertainty Error (UE) measures whether uncertainty accepts correct detections and rejects incorrect ones, including closed-set errors and/or open-set errors.At threshold δ, UE is the probability of incorrectly accepting or rejecting a detection, with the two error proportions weighted equally.Source: 2) Uncertainty Error (UE) · p. 4, B. Performance Metrics · p. 4
  • Minimum UE gives the ideal threshold for separating correct from incorrect detections and the highest uncertainty effectiveness achievable by a detector.Perfect UE is 0%, with all correct detections accepted and all incorrect detections rejected.Source: 2) Uncertainty Error (UE) · p. 4

3) Area Under the ROC curve (AUROC):

AUROC and AUPR evaluate how well uncertainty separates correct from incorrect detections; AUPR distinguishes whether accepted or rejected detections are treated as positive.

3) Area Under the ROC curve (AUROC):

  • AUROC represents the probability that a correct detection has lower uncertainty than an incorrect detection, with 100% as a perfect score.A true positive is an accepted correct detection, and a false positive is an accepted incorrect detection.Source: 3) Area Under the ROC curve (AUROC) · p. 4

4) Area Under the Precision Recall curve (AUPR):

  • AUPR-In treats accepted detections as positive and measures accepting correct detections while minimizing accepted incorrect ones; AUPR-Out treats rejected detections as positive and measures rejecting incorrect ones.For robotics, the passage identifies AUPR-Out as more informative because removing incorrect detections is often more important.Source: 4) Area Under the Precision Recall… · p. 4

5) IoU with ground truth objects (GT-IoU):

The evaluation covers detection spatial accuracy across closed-set and near open-set datasets, including unknown classes visually and semantically similar to training classes.

5) IoU with ground truth objects (GT-IoU):

  • GT-IoU measures spatial accuracy as the maximum IoU between a detection and a ground-truth object of the same class.The best result is ground-truth IoU of 1 for all detections.Source: 5) IoU with ground truth objects… · p. 4

C. Datasets

  • The affinity-clustering combinations are evaluated across datasets representing closed-set and open-set conditions.Source: C. Datasets · p. 4

1) Closed-Set Data:

  • The closed-set evaluation uses 4,952 PASCAL VOC 2007 test images, whose dataset was used to train the detector.Source: 1) Closed-Set Data · p. 4

2) Near Open-Set Data:

  • The near open-set evaluation uses 918 COCO 2017 validation images without any of the 20 known PASCAL VOC classes, representing visually and semantically similar unknown classes.Source: 2) Near Open-Set Data · p. 4

3) Distant Open-Set Data:

On distant open-set data, the paper reports that affinity and clustering choices affect both uncertainty effectiveness and object-detection performance. BSAS with IoU & SL at a 0.95 threshold performs best among the tested approaches.

3) Distant Open-Set Data:

  • The distant open-set dataset contains 2,899 underwater images with marine life and artefacts, none belonging to the detector’s known classes.The scenes differ visually and contextually from the training data.Source: 3) Distant Open-Set Data · p. 4

A. Summary of Findings

  • 3.3% difference in minimum uncertainty error and 5.3% difference in mAP separate the tested affinity-clustering variations.All tested clustering methods improve uncertainty effectiveness over the standard detector, but a poor choice can lower object-detection performance below it.Source: A. Summary of Findings · pp. 4–5
  • BSAS with IoU & SL at a 0.95 threshold achieves the lowest UE while improving mAP over the standard detector.The paper also reports lower UE than most other clustering methods at the same mAP.Source: A. Summary of Findings · p. 5, Figure 3 · p. 5
  • Combining spatial affinity with Same Label generally improves uncertainty effectiveness and object detection, especially for BSAS, which can otherwise cluster distinct high-overlap objects.KL typically offers little improvement and can perform worse than spatial affinity alone; the authors suggest MC Dropout variation may overwhelm this semantic measure.Source: A. Summary of Findings · p. 5

B. Further Review of Performance

BSAS IoU 0.95 & SL has the strongest AUPR-Out results across open-set datasets, while near-open-set testing yields smaller uncertainty gains than distant-open-set testing. Clustering choices also trade off spatial-uncertainty behavior and closed-set performance.

B. Further Review of Performance

  • HDBSCAN Corner reduces uncertainty error by 3.5% and increases AUPR-Out by 9% on distant-open-set data, versus 0.8% and 2.3%, respectively, on near-open-set data.The paper cautions that distant-open-set uncertainty appears higher quality than performance under near-open-set conditions relevant to robotics.Source: B. Further Review of Performance · p. 6
  • BSAS IoU 0.95 & SL achieves the best AUPR-Out on open-set datasets and a 4% increase over BSAS excl. IoU 0.9 & SL in near-open-set conditions.AUPR-Out emphasizes rejecting incorrect detections, even if some correct detections are rejected.Source: B. Further Review of Performance · p. 6
  • Intra-sample-exclusive clustering performs best on closed-set data, where BSAS IoU 0.95 & SL can miscluster same-sample detections and raise uncertainty on correct detections.Despite this issue, BSAS IoU 0.95 & SL still outperforms the standard detector.Source: B. Further Review of Performance · p. 6
  • BSAS methods separate spatial uncertainty for accurate and inaccurate observations, while threshold-independent HDBSCAN and Hungarian methods can give accurate observations high uncertainty when outliers raise cluster variance.BSAS can still assign low uncertainty to an inaccurate observation, but its mean spatial uncertainty differs by accuracy; Fig. 5 compares pixel variance for observations with IoU ≥0.7 versus ≤0.3.Source: B. Further Review of Performance · p. 6, Figure 5 · p. 6

VII. CONCLUSIONS AND FUTURE WORK

The paper investigates how affinity measures and clustering techniques associate sampled bounding boxes for object-detection uncertainty estimation. It finds that choosing the right combination can improve classification and spatial uncertainty estimation and resulting detection performance.

VII. CONCLUSIONS AND FUTURE WORK

  • Choosing the right affinity-clustering combination can improve classification and spatial uncertainty estimation and the resulting object-detection performance.The study compares three spatial and two semantic affinity measures with four clustering methods.Source: §VII · p. 6

Related papers

Loading 1809.06006v3…