Source-linked AI summary
Dropout Sampling for Robust Object Detection in Open-Set Conditions
Dimity Miller, Lachlan Nicholson, Feras Dayoub, Niko Sünderhauf
TL;DR
Open-set object detection is difficult because robots encounter classes absent from training, while conventional evaluations largely assume closed-set conditions. The paper extends Dropout Sampling to object detection and extracts uncertainty from repeated sampled detections. Bayesian SSD improves precision and recall relative to vanilla SSD, including 12.3% higher recall at equal precision and 15.1% higher precision at equal recall.
Problem
Open-set object detection remains challenging because real-world robots encounter new object classes that were not seen during training.
Source: §I · p. 1Method
The paper extends Dropout Sampling to object detection and extracts label uncertainty from repeated sampled detections.
Source: §I · p. 1, B. Object Detection with Dropout… · p. 3, C. Partitioning Detections into… · p. 3Results
12.3% higher recall at the vanilla SSD precision of 32.8%, and 15.1% higher precision at the vanilla SSD recall of 16.5%.
Source: B. SceneNet RGB-D · p. 5Takeaways & Limitations
Bayesian SSD improves object-detection precision and recall while reducing open-set error under open-set conditions.
Source: A. Summary · p. 5, B. SceneNet RGB-D · p. 5
Abstract
from arXiv · showhide
Dropout Variational Inference, or Dropout Sampling, has been recently proposed as an approximation technique for Bayesian Deep Learning and evaluated for image classification and regression tasks. This paper investigates the utility of Dropout Sampling for object detection for the first time. We demonstrate how label uncertainty can be extracted from a state-of-the-art object detection system via Dropout Sampling. We evaluate this approach on a large synthetic dataset of 30,000 images, and a real-world dataset captured by a mobile robot in a versatile campus environment. We show that this uncertainty can be utilized to increase object detection performance under the open-set conditions that are typically encountered in robotic vision. A Dropout Sampling network is shown to achieve a 12.3% increase in recall (for the same precision score as a standard network) and a 15.1% increase in precision (for the same recall score as the standard network).
I. INTRODUCTION
Open-set object detection remains a major challenge because robots encounter object classes absent from training, unlike conventional closed-set evaluations. The paper extends Dropout Sampling to object detection and evaluates whether its uncertainty estimates improve performance in such conditions.
- Open-set conditions expose detectors to new object classes that were not seen during training.Source: §I · p. 1
- Closed-set benchmarks typically use training and testing data containing the same object classes.Source: §I · p. 1
- Reliable open-set detection matters for mobile-robot deployment because unreliable predictions can have serious repercussions.Source: §I · p. 1
- Dropout Sampling estimates prediction uncertainty and had previously been applied to image classification and regression, but not object detection.Source: §I · p. 1
- The paper evaluates Dropout Sampling for object detection on synthetic and real-world datasets to improve performance under open-set conditions.Source: §I · p. 1
II. RELATED WORK
Related work establishes the contrast between closed-set detector success and open-set failures, where unknown objects may receive confident known-class labels. Existing responses emphasize uncertainty-based rejection, while Bayesian approaches face practical costs.
- State-of-the-art detectors perform well in closed-set settings but lose performance on images outside their development datasets.Source: A. Visual Object Detection · p. 2
- Open-set evaluation includes novel classes during testing that were absent during training.Source: B. Open-set Object Detection · p. 2
- Closed-set-trained networks may classify unknown objects as known classes with high confidence.Source: B. Open-set Object Detection · p. 2
- Open-set methods commonly estimate prediction uncertainty to identify and reject unknown classes.Source: B. Open-set Object Detection · p. 2
C. Bayesian Deep Learning
Dropout Variational Inference makes Bayesian uncertainty estimation tractable by enabling dropout during testing and averaging predictions from multiple sampled networks. The resulting class-probability distribution supports entropy-based uncertainty measurement.
- Dropout Variational Inference approximates Bayesian neural networks while remaining computationally feasible.Source: C. Bayesian Deep Learning · p. 2
- Dropout Sampling enables dropout during testing so multiple forward passes sample network weights from an approximate posterior.Source: A. Dropout Sampling for Classification… · p. 2
- The estimated class probability is obtained by averaging the Softmax scores from the sampled forward passes.Source: A. Dropout Sampling for Classification… · p. 2
- The resulting probability vector q represents probabilities across all class labels.Source: A. Dropout Sampling for Classification… · p. 2
- Entropy H(q) measures label uncertainty, increasing for uniform class probabilities and decreasing for peaked distributions.Source: A. Dropout Sampling for Classification… · p. 2
B. Object Detection with Dropout Sampling
The paper adapts Dropout Sampling from single-image classification to object detection, where each sampled pass produces multiple localized detections with class-score vectors. Overlapping detections from repeated passes are grouped into observations for later processing.
- Object detection requires bounding boxes and label distributions for multiple objects rather than one image-level label distribution.Source: B. Object Detection with Dropout… · pp. 2–3
- The paper applies Dropout Sampling to the learned weights of a detection network such as SSD.Source: B. Object Detection with Dropout… · p. 3
- Dropout remains enabled during testing so each forward pass approximately samples a different network from the weight posterior.Source: B. Object Detection with Dropout… · p. 3
C. Partitioning Detections into Observations
Dropout Sampling aggregates detections from multiple stochastic forward passes into spatially coherent observations, then averages their class-score vectors to estimate label probabilities and uncertainty.
C. Partitioning Detections into Observations
- Each sampled forward pass produces detections containing bounding-box coordinates and a Softmax score vector; multiple passes form a larger detection set.Source: C. Partitioning Detections into… · p. 3
- Detections with high mutual IoU are partitioned into observations using a Union-Find data structure.Source: C. Partitioning Detections into… · p. 3
- The empirically selected IoU threshold of 0.95 conservatively yields several observations per object and avoids erroneous grouping in cluttered scenes.Lower thresholds, such as 0.8, tend to merge detections from different ground-truth objects.Source: C. Partitioning Detections into… · p. 3
- For each observation, the detector approximates class-probability vector qi by averaging the Softmax score vectors of its constituent detections.Source: D. Extracting Label Probabilities and… · p. 3
- The resulting class-label probability follows a Categorical distribution parameterized by qi across k classes.Source: D. Extracting Label Probabilities and… · p. 3
- Entropy of qi measures label uncertainty: uniform class probabilities produce high entropy, whereas a concentrated distribution produces low entropy.Source: D. Extracting Label Probabilities and… · p. 3
E. Extracting Location Probability and Spatial Uncertainty
Bounding-box distributions are estimated by averaging coordinates across detections in each observation, while label entropy supports rejection of uncertain unknown-object detections.
E. Extracting Location Probability and Spatial Uncertainty
- The mean bounding-box vector for each observation is obtained by averaging the bounding-box vectors of its constituent detections.Source: E. Extracting Location Probability and… · p. 3
- Bounding-box coordinate uncertainty is represented by the covariance matrix over the constituent bounding-box vectors.The paper does not use this spatial-uncertainty expression, although it identifies object-based SLAM as a possible future application.Source: E. Extracting Location Probability and… · p. 3
- A threshold on entropy H(qi) can identify and reject detections likely to correspond to unknown object classes.Source: F. Using Dropout Sampling to Improve… · p. 3
- Using H(qi) is expected to be superior to H(s) because qi is considered a better approximation to the true class-probability distribution than a single Softmax vector s.Source: F. Using Dropout Sampling to Improve… · p. 3
- The paper hypothesizes that Dropout variational inference improves object-detection performance under open-set conditions relative to a non-Bayesian detection network.Source: F. Using Dropout Sampling to Improve… · p. 3
- A single SSD forward pass yields 9 detections, while 42 Dropout Sampling passes yield 393 detections grouped into 29 observations.Source: Figure 2 · p. 4
IV. EVALUATION METRICS
The evaluation uses open-set error, precision, and recall, with detections matched to known objects using IoU and filtered according to entropy-based uncertainty.
IV. EVALUATION METRICS
- The evaluation measures open-set error, precision, and recall to assess unknown-object robustness and known-object detection and classification.The ideal values are 0 open-set error, 100% precision, and 100% recall.Source: §IV · p. 4
A. Precision and Recall
- Precision and recall are computed by categorizing observations as true positives or false positives, while missed known objects are false negatives.Source: A. Precision and Recall · p. 4
- A detection is rejected when entropy H(qi) exceeds threshold θ, identifying high-uncertainty observations likely to be unknown objects.Source: A. Precision and Recall · p. 4
- For detections passing the entropy test, an observation is a true positive when its winning label matches a ground-truth object matched at IoU ≥0.5; otherwise it is false positive.Source: A. Precision and Recall · p. 4
- An accepted observation with no IoU ≥0.5 ground-truth match is false positive when its winning class is not unknown.Source: A. Precision and Recall · p. 4
- Every known ground-truth object without an associated observation at IoU ≥0.5 is counted as a false negative.Source: A. Precision and Recall · p. 4
- Precision equals TP/(TP+FP), recall equals TP/(TP+FN), and both are combined in the F1 score.Source: A. Precision and Recall · p. 4
B. Absolute Open-Set Error
Absolute open-set error counts accepted detections on unknown objects that are not labeled unknown. The evaluation uses synthetic and real-world robotic datasets to assess this error.
- Absolute open-set error counts observations passing the Entropy test that fall on unknown objects and lack an ’unknown’ winning label.Unknown objects have no overlapping ground-truth object with IoU ≥0.5 and a known true class label.Source: B. Absolute Open-Set Error · p. 4
- In the ideal case, all observations are known objects from the training set, producing an open-set error of 0.Source: B. Absolute Open-Set Error · p. 4
- The evaluation uses SceneNet RGBD, a large rendered-scene dataset, and QUT Campus, a real-world mobile-robot dataset.QUT Campus covers varied indoor and outdoor campus environments, with detections evaluated by manual visual inspection.Source: C. Datasets Used in the Evaluation · p. 4
D. Evaluation Protocol and Compared Object Detectors
The evaluation compares vanilla SSD, entropy-thresholded SSD, and Bayesian SSD using Dropout Sampling. Networks are tested across entropy thresholds and large synthetic and real-world evaluation sets.
- The study compares vanilla SSD, entropy-thresholded SSD using H(s), and Bayesian SSD using H(q) from averaged Softmax scores.Bayesian SSD varies the number of forward passes and minimum detections required per observation.Source: D. Evaluation Protocol and Compared… · p. 5
- Bayesian SSD uses Dropout Sampling and entropy of averaged Softmax scores to estimate uncertainty and reject detections.Source: D. Evaluation Protocol and Compared… · p. 5
- The evaluation varies entropy threshold θ from 0.1 to 2.5 and tests 30,000 SceneNet RGB-D images plus 75 QUT Campus images.All networks were fine-tuned on COCO; QUT images span 11 scenes.Source: D. Evaluation Protocol and Compared… · p. 5
- The experiments confirmed that Bayesian SSD improved precision and recall while reducing open-set error under open-set conditions.Source: A. Summary · p. 5
- Results examine both datasets and the effects of forward-pass counts and minimum detections per observation.Source: A. Summary · p. 5
B. SceneNet RGB-D
On SceneNet RGB-D, Bayesian SSD improves precision-recall performance and reduces open-set error relative to vanilla SSD. Forward-pass and detection requirements affect the performance-computation trade-off.
B. SceneNet RGB-D
- 12.3% higher recall at the vanilla SSD precision of 32.8% and 15.1% higher precision at the vanilla SSD recall of 16.5% are achieved by Bayesian SSD.Bayesian SSD also outperforms entropy-thresholded SSD overall and reaches higher maximum recall.Source: B. SceneNet RGB-D · p. 5
- Bayesian SSD reduces open-set error versus vanilla SSD while retaining the reference F1 score, or improves F1 at the reference error level.Source: B. SceneNet RGB-D · p. 5
C. Forward Passes
- 10 forward passes maintain the vanilla SSD reference F1 score while reducing open-set error comparably to larger pass counts.At least 20 passes are needed to maximize F1 at the reference open-set error.Source: C. Forward Passes · p. 5
- One forward pass takes 0.05 seconds with the current model, motivating fewer passes when detection requirements permit.Future computation could sample only stochastic post-dropout layers.Source: C. Forward Passes · p. 6
D. Minimum Detection
- Requiring at least 3 detections per observation marginally lowers open-set error for each F1 score but reduces maximum F1.The authors therefore favor a low minimum-detection requirement, if any.Source: D. Minimum Detection · p. 6
E. Real World Dataset
On the QUT Campus dataset, Bayesian SSD reduced total error while retaining more true detections than vanilla SSD, and an image example shows uncertainty filtering open-set errors without discarding the true detection. Future work targets spatial uncertainty for object pose estimation.
QUT Campus dataset
- Bayesian SSD reduced total error by 21 at the vanilla SSD reference point with no entropy thresholding.The total error includes open-set errors and incorrect classifications of known objects.Source: E. Real World Dataset · p. 6
- For the same total error, Bayesian SSD achieved 363 more true detections than vanilla SSD.The authors note that multiple detections per object may contribute, while the result may also partially reflect superior recall.Source: E. Real World Dataset · p. 6
Qualitative comparison
- At entropy threshold 0.64, Bayesian SSD preserved a true person detection while eliminating all open-set error in the example image.Vanilla SSD thresholding discarded all true detections while retaining most open-set error.Source: E. Real World Dataset · p. 6, Figure 7 · p. 7
Conclusions
- Dropout Sampling was shown to provide better label uncertainty information that helps improve open-set object detection performance.The conclusion frames this as verification of the paper’s central hypothesis for mobile-robot conditions.Source: §VI · p. 7
Future work
- Future work could exploit spatial uncertainty in bounding-box covariance for object-based SLAM and improved 6-DOF object pose estimates.The proposed direction concerns covariance information for groups of detections.Source: §VI · p. 7