PaperlayerView exact revision on arXiv

Source-linked AI summary

Dropout Sampling for Robust Object Detection in Open-Set Conditions

Dimity Miller, Lachlan Nicholson, Feras Dayoub, Niko Sünderhauf

arXiv:1710.06677v2cs.CV

TL;DR

Open-set object detection is difficult because robots encounter classes absent from training, while conventional evaluations largely assume closed-set conditions. The paper extends Dropout Sampling to object detection and extracts uncertainty from repeated sampled detections. Bayesian SSD improves precision and recall relative to vanilla SSD, including 12.3% higher recall at equal precision and 15.1% higher precision at equal recall.

  • Problem

    Open-set object detection remains challenging because real-world robots encounter new object classes that were not seen during training.

  • Method

    The paper extends Dropout Sampling to object detection and extracts label uncertainty from repeated sampled detections.

  • Results

    12.3% higher recall at the vanilla SSD precision of 32.8%, and 15.1% higher precision at the vanilla SSD recall of 16.5%.

  • Takeaways & Limitations

    Bayesian SSD improves object-detection precision and recall while reducing open-set error under open-set conditions.

Abstract

from arXiv · show

Dropout Variational Inference, or Dropout Sampling, has been recently proposed as an approximation technique for Bayesian Deep Learning and evaluated for image classification and regression tasks. This paper investigates the utility of Dropout Sampling for object detection for the first time. We demonstrate how label uncertainty can be extracted from a state-of-the-art object detection system via Dropout Sampling. We evaluate this approach on a large synthetic dataset of 30,000 images, and a real-world dataset captured by a mobile robot in a versatile campus environment. We show that this uncertainty can be utilized to increase object detection performance under the open-set conditions that are typically encountered in robotic vision. A Dropout Sampling network is shown to achieve a 12.3% increase in recall (for the same precision score as a standard network) and a 15.1% increase in precision (for the same recall score as the standard network).

I. INTRODUCTION

Open-set object detection remains a major challenge because robots encounter object classes absent from training, unlike conventional closed-set evaluations. The paper extends Dropout Sampling to object detection and evaluates whether its uncertainty estimates improve performance in such conditions.

  • Open-set conditions expose detectors to new object classes that were not seen during training.
  • Closed-set benchmarks typically use training and testing data containing the same object classes.
  • Reliable open-set detection matters for mobile-robot deployment because unreliable predictions can have serious repercussions.
  • Dropout Sampling estimates prediction uncertainty and had previously been applied to image classification and regression, but not object detection.
  • The paper evaluates Dropout Sampling for object detection on synthetic and real-world datasets to improve performance under open-set conditions.

II. RELATED WORK

Related work establishes the contrast between closed-set detector success and open-set failures, where unknown objects may receive confident known-class labels. Existing responses emphasize uncertainty-based rejection, while Bayesian approaches face practical costs.

  • State-of-the-art detectors perform well in closed-set settings but lose performance on images outside their development datasets.
  • Open-set evaluation includes novel classes during testing that were absent during training.
  • Closed-set-trained networks may classify unknown objects as known classes with high confidence.
  • Open-set methods commonly estimate prediction uncertainty to identify and reject unknown classes.

C. Bayesian Deep Learning

Dropout Variational Inference makes Bayesian uncertainty estimation tractable by enabling dropout during testing and averaging predictions from multiple sampled networks. The resulting class-probability distribution supports entropy-based uncertainty measurement.

  • Dropout Variational Inference approximates Bayesian neural networks while remaining computationally feasible.
  • Dropout Sampling enables dropout during testing so multiple forward passes sample network weights from an approximate posterior.
  • The estimated class probability is obtained by averaging the Softmax scores from the sampled forward passes.
  • The resulting probability vector q represents probabilities across all class labels.
  • Entropy H(q) measures label uncertainty, increasing for uniform class probabilities and decreasing for peaked distributions.

B. Object Detection with Dropout Sampling

The paper adapts Dropout Sampling from single-image classification to object detection, where each sampled pass produces multiple localized detections with class-score vectors. Overlapping detections from repeated passes are grouped into observations for later processing.

  • Object detection requires bounding boxes and label distributions for multiple objects rather than one image-level label distribution.
  • The paper applies Dropout Sampling to the learned weights of a detection network such as SSD.
  • Dropout remains enabled during testing so each forward pass approximately samples a different network from the weight posterior.

C. Partitioning Detections into Observations

Dropout Sampling aggregates detections from multiple stochastic forward passes into spatially coherent observations, then averages their class-score vectors to estimate label probabilities and uncertainty.

  • C. Partitioning Detections into Observations: Each sampled forward pass produces detections containing bounding-box coordinates and a Softmax score vector; multiple passes form a larger detection set.
  • C. Partitioning Detections into Observations: Detections with high mutual IoU are partitioned into observations using a Union-Find data structure.
  • C. Partitioning Detections into Observations: The empirically selected IoU threshold of 0.95 conservatively yields several observations per object and avoids erroneous grouping in cluttered scenes.Lower thresholds, such as 0.8, tend to merge detections from different ground-truth objects.
  • C. Partitioning Detections into Observations: For each observation, the detector approximates class-probability vector qi by averaging the Softmax score vectors of its constituent detections.
  • C. Partitioning Detections into Observations: The resulting class-label probability follows a Categorical distribution parameterized by qi across k classes.
  • C. Partitioning Detections into Observations: Entropy of qi measures label uncertainty: uniform class probabilities produce high entropy, whereas a concentrated distribution produces low entropy.

E. Extracting Location Probability and Spatial Uncertainty

Bounding-box distributions are estimated by averaging coordinates across detections in each observation, while label entropy supports rejection of uncertain unknown-object detections.

  • E. Extracting Location Probability and Spatial Uncertainty: The mean bounding-box vector for each observation is obtained by averaging the bounding-box vectors of its constituent detections.
  • E. Extracting Location Probability and Spatial Uncertainty: Bounding-box coordinate uncertainty is represented by the covariance matrix over the constituent bounding-box vectors.The paper does not use this spatial-uncertainty expression, although it identifies object-based SLAM as a possible future application.
  • E. Extracting Location Probability and Spatial Uncertainty: A threshold on entropy H(qi) can identify and reject detections likely to correspond to unknown object classes.
  • E. Extracting Location Probability and Spatial Uncertainty: Using H(qi) is expected to be superior to H(s) because qi is considered a better approximation to the true class-probability distribution than a single Softmax vector s.
  • E. Extracting Location Probability and Spatial Uncertainty: The paper hypothesizes that Dropout variational inference improves object-detection performance under open-set conditions relative to a non-Bayesian detection network.
  • E. Extracting Location Probability and Spatial Uncertainty: A single SSD forward pass yields 9 detections, while 42 Dropout Sampling passes yield 393 detections grouped into 29 observations.

IV. EVALUATION METRICS

The evaluation uses open-set error, precision, and recall, with detections matched to known objects using IoU and filtered according to entropy-based uncertainty.

  • IV. EVALUATION METRICS: The evaluation measures open-set error, precision, and recall to assess unknown-object robustness and known-object detection and classification.The ideal values are 0 open-set error, 100% precision, and 100% recall.
  • A. Precision and Recall: Precision and recall are computed by categorizing observations as true positives or false positives, while missed known objects are false negatives.
  • A. Precision and Recall: A detection is rejected when entropy H(qi) exceeds threshold θ, identifying high-uncertainty observations likely to be unknown objects.
  • A. Precision and Recall: For detections passing the entropy test, an observation is a true positive when its winning label matches a ground-truth object matched at IoU ≥0.5; otherwise it is false positive.
  • A. Precision and Recall: An accepted observation with no IoU ≥0.5 ground-truth match is false positive when its winning class is not unknown.
  • A. Precision and Recall: Every known ground-truth object without an associated observation at IoU ≥0.5 is counted as a false negative.
  • A. Precision and Recall: Precision equals TP/(TP+FP), recall equals TP/(TP+FN), and both are combined in the F1 score.

B. Absolute Open-Set Error

Absolute open-set error counts accepted detections on unknown objects that are not labeled unknown. The evaluation uses synthetic and real-world robotic datasets to assess this error.

  • Absolute open-set error counts observations passing the Entropy test that fall on unknown objects and lack an ’unknown’ winning label.Unknown objects have no overlapping ground-truth object with IoU ≥0.5 and a known true class label.
  • In the ideal case, all observations are known objects from the training set, producing an open-set error of 0.
  • The evaluation uses SceneNet RGBD, a large rendered-scene dataset, and QUT Campus, a real-world mobile-robot dataset.QUT Campus covers varied indoor and outdoor campus environments, with detections evaluated by manual visual inspection.

D. Evaluation Protocol and Compared Object Detectors

The evaluation compares vanilla SSD, entropy-thresholded SSD, and Bayesian SSD using Dropout Sampling. Networks are tested across entropy thresholds and large synthetic and real-world evaluation sets.

  • The study compares vanilla SSD, entropy-thresholded SSD using H(s), and Bayesian SSD using H(q) from averaged Softmax scores.Bayesian SSD varies the number of forward passes and minimum detections required per observation.
  • Bayesian SSD uses Dropout Sampling and entropy of averaged Softmax scores to estimate uncertainty and reject detections.
  • The evaluation varies entropy threshold θ from 0.1 to 2.5 and tests 30,000 SceneNet RGB-D images plus 75 QUT Campus images.All networks were fine-tuned on COCO; QUT images span 11 scenes.
  • The experiments confirmed that Bayesian SSD improved precision and recall while reducing open-set error under open-set conditions.
  • Results examine both datasets and the effects of forward-pass counts and minimum detections per observation.

B. SceneNet RGB-D

On SceneNet RGB-D, Bayesian SSD improves precision-recall performance and reduces open-set error relative to vanilla SSD. Forward-pass and detection requirements affect the performance-computation trade-off.

  • B. SceneNet RGB-D: 12.3% higher recall at the vanilla SSD precision of 32.8% and 15.1% higher precision at the vanilla SSD recall of 16.5% are achieved by Bayesian SSD.Bayesian SSD also outperforms entropy-thresholded SSD overall and reaches higher maximum recall.
  • B. SceneNet RGB-D: Bayesian SSD reduces open-set error versus vanilla SSD while retaining the reference F1 score, or improves F1 at the reference error level.
  • C. Forward Passes: 10 forward passes maintain the vanilla SSD reference F1 score while reducing open-set error comparably to larger pass counts.At least 20 passes are needed to maximize F1 at the reference open-set error.
  • C. Forward Passes: One forward pass takes 0.05 seconds with the current model, motivating fewer passes when detection requirements permit.Future computation could sample only stochastic post-dropout layers.
  • D. Minimum Detection: Requiring at least 3 detections per observation marginally lowers open-set error for each F1 score but reduces maximum F1.The authors therefore favor a low minimum-detection requirement, if any.

E. Real World Dataset

On the QUT Campus dataset, Bayesian SSD reduced total error while retaining more true detections than vanilla SSD, and an image example shows uncertainty filtering open-set errors without discarding the true detection. Future work targets spatial uncertainty for object pose estimation.

  • QUT Campus dataset: Bayesian SSD reduced total error by 21 at the vanilla SSD reference point with no entropy thresholding.The total error includes open-set errors and incorrect classifications of known objects.
  • QUT Campus dataset: For the same total error, Bayesian SSD achieved 363 more true detections than vanilla SSD.The authors note that multiple detections per object may contribute, while the result may also partially reflect superior recall.
  • Qualitative comparison: At entropy threshold 0.64, Bayesian SSD preserved a true person detection while eliminating all open-set error in the example image.Vanilla SSD thresholding discarded all true detections while retaining most open-set error.
  • Conclusions: Dropout Sampling was shown to provide better label uncertainty information that helps improve open-set object detection performance.The conclusion frames this as verification of the paper’s central hypothesis for mobile-robot conditions.
  • Future work: Future work could exploit spatial uncertainty in bounding-box covariance for object-based SLAM and improved 6-DOF object pose estimates.The proposed direction concerns covariance information for groups of detections.
Loading 1710.06677v2…