Discriminative out-of-distribution detection for semantic segmentation

1 INTRODUCTION

• Closed-world segmentation assumes predictions over predetermined classes, while real driving scenes include unfamiliar objects, unusual configurations, and environmental or acquisition shifts.These shifts can make models rely on dataset-specific variation rather than the visual stimulus itself.Source: §1 · p. 1
• Richer datasets and knowledge transfer are alternatives, but Vistas remains incomplete for WildDash and transfer may not suit models intended for direct use across environments.Vistas also requires considerable computational resources.Source: §1 · pp. 1–2
• Epistemic uncertainty arises when a model encounters data not seen during training, unlike aleatoric uncertainty associated with volatility or ambiguity in the data.The paper relates epistemic uncertainty to the probability that the model operates on an out-of-distribution sample.Source: §1 · p. 2
• Entropy-based OOD detection methods cannot separate epistemic from aleatoric uncertainty and achieve poor precision in the authors’ segmentation experiments.Source: §1 · p. 2
• The proposed separate OOD model discriminates the primary training dataset from a much larger background dataset, with ILSVRC serving as an approximation of visual-world diversity.The authors state that this formulation is insensitive to task-related aleatoric uncertainty and performs significantly better than approaches coupling OOD detection to the primary model.Source: §1 · p. 2

2 RELATED WORK

• Prediction-entropy approaches, including max-softmax and ODIN, estimate OOD status from model outputs, but do not distinguish epistemic from aleatoric uncertainty in dense prediction.Semantic borders, distant or underexposed regions, and ambiguous labels can yield false-positive OOD responses and reduce AP.Source: §2 · p. 2
• A separate uncertainty or confidence head learns from hard examples present in training, so it can detect aleatoric uncertainty but not epistemic uncertainty absent from the data.Source: §2 · pp. 2–3
• The mutual-information approach of Smith & Gal (2018) estimates epistemic uncertainty using MC dropout, but the authors report poor OOD accuracy and say further work is needed.They suggest inaccurate approximation of parameter sampling by MC dropout may explain the outcome.Source: §2 · p. 3
• Ensembles can recover uncertainty more accurately than MC dropout, but keeping many models in GPU memory during evaluation makes them unsuitable for real-time dense prediction systems.Source: §2 · p. 3
• Joint training with a foreign dataset signals OOD when predictions favor a class foreign to the evaluation dataset, such as a bathtub in an intelligent-vehicle image.Source: §2 · p. 3
• The proposed alternative casts OOD detection as supervised discrimination between the primary training dataset and a background distribution approximated with ILSVRC.This requires addressing overlap between primary-dataset content and ILSVRC images, while training the OOD model on more data than the primary model.Source: §2 · p. 3

3 THE PROPOSED DISCRIMINATIVE OOD DETECTION APPROACH

• The approach replaces difficult image-density modeling with supervised outlier detection, using the primary training dataset as in-distribution data and ILSVRC as a next-best background approximation.The authors note that no existing public dataset meets their requirement for the full diversity of natural images.Source: §3 · p. 3
• The method exploits ILSVRC object bounding boxes, which are present in approximately half of its images, to address training-data contention.Source: §3 · p. 4
• Training balances ID and OOD pixels to address overlapping ILSVRC classes and primary-dataset objects appearing in ILSVRC backgrounds.The resulting bias toward the ID dataset makes car pixels more likely to be classified as inliers, since cars are common in road-driving images but sparse across ILSVRC images.Source: §3 · p. 4
• The fully convolutional OOD model is trained with cross-entropy and produces dense two-class logits, with OOD detection performed by thresholding the inlier probability.Source: §3 · p. 4

4.5 EVALUATION

• The study evaluates dense OOD detection on natural images using road-driving datasets for ID and WildDash test, with average precision measuring separation without a chosen threshold.Evaluation includes all negative images and a measure that excludes five altered valid images treated as ambiguous.Source: §4 · p. 4, §4.5 · p. 6

4.1 ADAPTING IMAGE-WIDE APPROACHES FOR DENSE PREDICTION

• The experiments adapt max-softmax, ODIN, and trained confidence for dense prediction, while comparing them with a dedicated detector and the ROB runner-up model.Dense ODIN perturbs the input to increase each pixel's winning-class probability and provides a modest improvement over max-softmax.Source: §4.1 · p. 4, §4.5 · p. 6

4.6 RESULTS ON WILDDASH TEST

• Max-softmax separates ID from OOD pixels poorly on WildDash, even at low recall, and its responses concentrate on semantic borders, small objects, and distant scene details.Training the primary model with WildDash validation doubles max-softmax performance, but both approaches remain poor.Source: §4.6 · pp. 6–7, Table 1 · p. 7, Figure 1 · p. 7
• The augmented primary model with trained confidence improves segmentation mIoU over Cityscapes-only training, but its confidence is the worst OOD predictor among the evaluated approaches.The authors suggest the confidence head is confident by default and needs to learn which pixels should be uncertain.Source: §4.6 · p. 7, Table 2 · p. 8
• ROB training on indoor and road-driving datasets substantially raises average precision over road-only approaches, but it recognizes many altered negative-image pixels as ID.It also makes errors on OOD images, including ants and people located in OOD contexts.Source: §4.6 · p. 8, Table 3 · p. 8
• The discriminative detectors trained on Vistas or Cityscapes plus WildDash validation outperform the other reported approaches, with the WildDash-validation model achieving the best results.Both recognize pixels in artificially altered images as ID, and ID classes in OOD contexts can still be classified as OOD.Source: §4.6 · pp. 8–9, Table 4 · p. 9

4.7 RESULTS ON OTHER DATSETS

• Vistas-trained discriminative detection generalizes well to PASCAL and Cityscapes, while a Cityscapes-trained model classifies most Vistas pixels as OOD.The authors interpret this as evidence that Cityscapes is inappropriate for training generic road-scene models.Source: §4.7 · p. 9

• Graceful degradation amid unforeseen scenery is described as crucial for real-world computer vision, which should detect such situations to avoid disasters and fear of technology.Source: §5 · p. 10
• The OOD model identifies outliers by their greater similarity to a general dataset such as ILSVRC than to the primary model’s training data.It is trained with substantially more data than the primary model.Source: §5 · p. 10
• The proposed method substantially improves OOD-detection AP over all previous approaches and distinguishes negative WildDash images from positive ones.Source: §5 · p. 10
• Future work will use these results to guide annotation effort and further develop approaches to recognizing epistemic uncertainty in images and video.Source: §5 · p. 10
• In the WildDash examples, red encodes pixel-level OOD confidence, interpretable as epistemic uncertainty, across the original image and two training-distribution settings.Source: Figure 5 · p. 10

APPENDIX A DENSE OOD DETECTION IN IMAGES WITH MIXED CONTENT

• WildDash provides image-level OOD labels, so it cannot test unfamiliar-object detection within familiar scenes; the authors propose six datasets for this setting.Source: Appendix A · p. 13
• An improved training procedure is proposed to help the discriminative OOD model predict OOD-object borders accurately, and is used to train a model for evaluation.Source: Appendix A · p. 13
• The experiments compare AP performance across several OOD models and datasets.Source: Appendix A · p. 13

A.1 TEST DATASETS

• Three datasets contain both ID and OOD pixels for evaluating detection approaches, while the other three are control datasets testing responses to pasted ID content.Source: §A.1 · p. 13
• Two datasets paste Pascal VOC 2007 animal objects into Vistas validation images; Vistas training is reserved for OOD training.Source: §A.1 · p. 13
• The final dataset selects Vistas images with many pixels labeled “ground animal”; it is closest to unexpected-object encounters while driving, but contains relatively few images.Source: §A.1 · p. 13

A.1.1 PASCAL TO VISTAS 10%

• The procedure selects 369 large Pascal objects using pixel-level segmentation groundtruth and pastes one into a randomly chosen Vistas validation image for each object.Source: §A.1.1 · p. 13
• Each pasted object is resized to cover at least 10% of the image pixels and placed at a random location.Source: §A.1.1 · p. 13
• The resulting dataset contains 369 combined images, with Pascal animals selected from images containing segmentation groundtruth.Source: §A.1.1 · p. 13

A.1.2 PASCAL TO VISTAS 1%

• 31 images contain unresized Pascal instances pasted into Vistas validation images, with each object covering at least 1% of the image.The smaller patches make this dataset more difficult than the previous one.Source: §A.1.2 · p. 13

A.1.3 CITY TO CITY

• 288 images pair a Cityscapes validation instance with a different Cityscapes validation image, requiring the instance to cover at least 0.5% of the image.Performance indicates whether detection responds to imaging-condition differences between the source instance and host image.Source: §A.1.3 · p. 13
• In Vistas images with Pascal objects covering at least 10%, the A.2-trained model detects the OOD shape, while whole-image-trained discriminative detection misses the patches.The ROB model detects patch positions; max-softmax highlights borders.Source: Figure 6 · p. 14
• For Vistas images with Pascal objects covering at least 1%, the A.2-trained model detects small pasted objects, while ROB fails to locate them.The objects are pasted into a similar background.Source: Figure 7 · p. 14

A.1.4 VISTAS TO CITY

• 1,543 images pair a Vistas validation instance covering at least 0.5% of the image with a Cityscapes validation image.Performance indicates whether detection responds to camera characteristics of the patch rather than real OOD pixels.Source: §A.1.4 · p. 14
• None of the models accurately detect pasted Cityscapes instances covering at least 0.5% of Cityscapes images; the fourth model reacts to some pasted borders.No preprocessing is performed on these patches.Source: Figure 8 · p. 15
• On Cityscapes images with pasted Vistas instances, the fourth model struggles on atypical images and detects pasted-patch borders.The instances cover at least 0.5% of the host image.Source: Figure 9 · p. 15

A.1.5 SELF TO SELF

• In the self-to-self control, 1,873 Vistas images contain an instance pasted elsewhere in the same image; none of the models detect the pasted patches.This control tests response to objects at unusual scene locations.Source: §A.1.5 · p. 15, Figure 10 · p. 16

A.2 TRAINING DATASET

• The training change removes ground-animal images from Vistas training and uses 544,546 ILSVRC images as whole-image OOD examples or pasted-box OOD regions in Vistas ID images.Pasted boxes are resized to 5% of the Vistas image; resulting images are resized and cropped to 512×512.Source: §A.2 · pp. 15–16

A.3 MODEL AND TRAINING

• The same fully convolutional discriminative OOD model is trained using the procedure from Section 4.4 with the A.2 dataset.Source: §A.3 · p. 16

A.4 EXPERIMENTAL RESULTS

• Max-softmax highlights object borders, while ROB and whole-image-trained detection miss OOD patches in many examples; the control-set AP suggests some PascalToVistas success reflects pasted interventions.The figures compare original images, max-softmax, ROB, whole-image-trained detection, and A.2-trained detection; red encodes OOD confidence.Source: §A.4 · p. 17

A.4 EXPERIMENTAL RESULTS; A.5 CONCLUSION

• The A.2-trained discriminative model delivers the best overall performance, detects small pasted objects and genuine Vistas animals, and outperforms prior approaches by a wide margin.It sometimes flags pasted-patch borders and misclassifies parts of Cityscapes images; the authors caution that PascalToVistas results may be optimistic.Source: §A.4 · p. 17, §A.5 · p. 18

A.1.6 VISTAS ANIMALS

• VistasAnimals is the most difficult dataset for all models, but the A.2-trained discriminative model performs best.Only eight images contain qualifying ground-animal instances.Source: §A.1.6 · p. 15, §A.4 · p. 17