PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation

Robin Chan, Matthias Rottmann, Hanno Gottschalk

arXiv:2012.06575v2cs.CVcs.LG

TL;DR

Closed-set semantic segmentation needs to detect localized OoD objects in open-world, safety-critical settings. The paper retrains models to maximize entropy on COCO-based OoD proxies and applies logistic-regression meta classification to filter false predictions. Across disjoint OoD datasets, the combined approach improves detection while preserving semantic segmentation performance with only marginal sacrifice.

  • Problem

    Closed-set segmentation models need reliable localized detection of objects outside their semantic space to support open-world functional safety.

  • Method

    The method combines OoD retraining that maximizes softmax entropy on proxy OoD samples with a lightweight meta classifier using hand-crafted softmax and geometry features.

  • Results

    The method consistently improves OoD detection on disjoint LostAndFound and Fishyscapes data, outperforming baseline entropy thresholding and other tested methods.

  • Takeaways & Limitations

    Combining OoD training and meta classification offers a lightweight monitoring approach for safety-critical deep-learning systems.

  • Takeaways & Limitations

    Future work is needed to use OoD training for retrieving OoD objects and assessing whether new concepts should be learned.

Abstract

from arXiv · show

Deep neural networks (DNNs) for the semantic segmentation of images are usually trained to operate on a predefined closed set of object classes. This is in contrast to the "open world" setting where DNNs are envisioned to be deployed to. From a functional safety point of view, the ability to detect so-called "out-of-distribution" (OoD) samples, i.e., objects outside of a DNN's semantic space, is crucial for many applications such as automated driving. A natural baseline approach to OoD detection is to threshold on the pixel-wise softmax entropy. We present a two-step procedure that significantly improves that approach. Firstly, we utilize samples from the COCO dataset as OoD proxy and introduce a second training objective to maximize the softmax entropy on these samples. Starting from pretrained semantic segmentation networks we re-train a number of DNNs on different in-distribution datasets and consistently observe improved OoD detection performance when evaluating on completely disjoint OoD datasets. Secondly, we perform a transparent post-processing step to discard false positive OoD samples by so-called "meta classification". To this end, we apply linear models to a set of hand-crafted metrics derived from the DNN's softmax probabilities. In our experiments we consistently observe a clear additional gain in OoD detection performance, cutting down the number of detection errors by up to 52% when comparing the best baseline with our results. We achieve this improvement sacrificing only marginally in original segmentation performance. Therefore, our method contributes to safer DNNs with more reliable overall system performance.

1. Introduction

Semantic segmentation models trained on closed class sets must detect unknown objects in open-world settings, especially to support functional safety and localized anomaly detection. The paper proposes entropy maximization during retraining followed by meta classification to improve OoD monitoring.

  • Motivation: Closed-set segmentation models may encounter objects outside their training semantic space in real-world deployment.The paper frames these objects as out-of-distribution samples in open-world settings.
  • Motivation: Reliable OoD detection is a missing functional-safety prerequisite because it can trigger a fallback policy when models leave their proper domain.
  • Motivation: Localized anomaly segmentation is needed because everyday images may contain many objects while only some are out-of-distribution.
  • Approach: The first step retrains segmentation CNNs to produce high uncertainty on annotated OoD objects by maximizing softmax entropy with a modified multi-objective loss.
  • Evaluation: COCO images serve as OoD proxies during retraining, while LostAndFound and Fishyscapes provide disjoint evaluation data for pixel-wise OoD detection.
  • Contributions: The proposed combination of OoD training and logistic-regression meta classification improves detection while remaining lightweight and suitable for safety-critical monitoring.

2. Related Work

Prior semantic-segmentation OoD methods include uncertainty estimation, autoencoders, and training modifications, but computational cost, dataset specificity, and interpretability motivate a lightweight alternative. The paper combines retraining with a linear post-processing model without changing the CNN architecture.

  • Existing approaches: Earlier OoD methods often target entire images rather than localized out-of-distribution objects in semantic segmentation.
  • Uncertainty estimation: Bayesian and approximate uncertainty methods can require multiple models or forward passes, increasing computation relative to single-inference approaches.
  • Proposed positioning: The proposed retraining unites semantic segmentation and OoD detection in one model without modifying the underlying CNN architecture.
  • Training proxies: Training on unlabeled Cityscapes objects can be too dataset-specific because the available objects lack diversity for other OoD datasets.
  • Autoencoders: Autoencoder approaches avoid requiring OoD data but add auxiliary modeling and rely on reconstruction quality for detection.
  • Design rationale: Instead of deep auxiliary models, the paper uses a lightweight linear model, favoring computational efficiency and interpretability for safe real-time segmentation.

3. Entropy based OoD Detection

The method trains segmentation networks to produce high softmax entropy on OoD inputs, then thresholds normalized pixel entropy to form OoD object predictions. Connected components above the threshold define predicted OoD segments.

  • 3.1. Training for high Entropy on OoD Samples: The training method improves OoD pixel detection through spatial softmax-entropy heatmaps.
  • 3.1. Training for high Entropy on OoD Samples: Softmax entropy is computed from the model’s class-probability vector for an input, with image-pixel indexing omitted for exposition.
  • 3.1. Training for high Entropy on OoD Samples: The objective combines in-distribution labeled examples with out-distribution unlabeled examples.
  • 3.1. Training for high Entropy on OoD Samples: For in-distribution samples the loss is cross-entropy, whereas for OoD samples it uses the negative log-likelihood averaged over all classes.
  • 3.1. Training for high Entropy on OoD Samples: Minimizing the OoD loss maximizes softmax entropy, reaching equality when class probabilities are uniformly distributed.
  • 3.1. Training for high Entropy on OoD Samples: A weighting parameter λ controls the balance between expected in-distribution and out-distribution losses.
  • 3.2. OoD Object Prediction in Semantic Segmentation via Entropy Thresholding: At each pixel, normalized entropy is thresholded to identify OoD pixels, and neighboring threshold-satisfying pixels form OoD segment predictions.

4. Meta Classifier in Semantic Segmentation

Entropy boosting increases sensitivity to OoD objects but can also create false predictions. A segment-level meta classifier uses softmax-derived uncertainty and geometry features to filter those predictions.

  • Motivation: Entropy boosting can produce false OoD predictions because high entropy is not exclusive to OoD samples.
  • Evaluation criterion: The evaluation treats a predicted OoD segment as a true positive when its IoU with a ground-truth OoD object exceeds zero.
  • Evaluation criterion: The study treats every pixel as a potential road hazard to minimize overlooked OoD objects rather than requiring a higher IoU threshold.
  • Meta classification: The meta classifier is a binary model that removes false positives using aggregated softmax-derived uncertainty measures from connected components.
  • Meta classification: Features include entropy, variation ratio, probability margin, segment geometry, and within-segment variances.
  • Meta classification: The hand-crafted metrics form a feature dataset whose rows are predicted segments and whose columns are metrics.

5. Setup of Experiments

The experiments use Cityscapes for in-distribution segmentation, filtered COCO images as an OoD-training proxy, and LostAndFound and Fishyscapes for evaluation.

  • Cityscapes supplies 2,975 pixel-annotated urban street-scene training images for the original segmentation task.
  • DeepLabv3+ with a WideResNet38 backbone, pretrained by Nvidia, serves as the baseline model.
  • LostAndFound and Fishyscapes provide disjoint street-scene test images containing unexpected objects for pixel-wise OoD evaluation.
  • The COCO OoD proxy contains 46,751 filtered images whose instances are absent from Cityscapes and whose dimensions are at least 480 pixels.
  • Each OoD-training epoch randomly includes 297 of the 46,751 COCO images mixed with the Cityscapes training data.

6. Pixel-wise Evaluation

Pixel-wise entropy evaluation shows that OoD training improves separation of in-distribution and OoD pixels on both test datasets while largely preserving Cityscapes segmentation performance.

  • 6.1. Separability by means of Area Under Curve: OoD training improves entropy-based separability on LostAndFound and Fishyscapes, with higher true-positive rates across fixed false-positive rates than the baseline.
  • 6.1. Separability by means of Area Under Curve: An AUROC of 0.99 on Fishyscapes after OoD training outperforms all other tested methods.
  • 6.1. Separability by means of Area Under Curve: The AUPRC increases by 0.30 to 0.76 on LostAndFound and from 0.28 to 0.81 on Fishyscapes after OoD training.
  • 6.2. Original Task Performance: The baseline reaches 0.90 Cityscapes validation mIoU, while entropy-trained models reach 0.89 and remain within 1 percentage point for thresholds t = 0.3, ..., 1.0.
  • 6.2. Original Task Performance: At t = 0.2, Cityscapes mIoU falls to 0.05 for the baseline and 0.03 for the retrained model, motivating thresholds of at least t = 0.3.

7. Segment-wise Evaluation

Segment-wise meta classification removes false-positive OoD predictions using lightweight models based on segment-level softmax-derived features, improving detection while monitoring road-pixel misses.

  • Evaluation metrics: The F1-score summarizes segment-wise false-positive and false-negative rates, while road miss rate ε monitors loss in original performance.
  • Meta classification: Logistic-regression meta classifiers use aggregated dispersion and geometry metrics from predicted segments to classify OoD predictions as true or false positives.
  • Threshold effects: Lower entropy thresholds feed more predicted OoD objects to the meta classifiers and improve false-positive identification reliability.
  • Results: At t = 0.3, OoD training with meta classification achieves the best OoD detection approach under the reported experiments.
  • Results: The combined method reduces total errors by 52%, from 1,242 to 598, and overlooked OoD objects by 70%, from 1,084 to 308, with a road miss rate of 0.06%.

8. Conclusion & Outlook

The work combines OoD retraining that improves entropy-based detection with meta classification to remove false OoD predictions while preserving semantic segmentation performance. The authors also identify future OoD-object retrieval as an outlook.

  • Meta classifiers based on lightweight logistic regressions reliably classify entire LostAndFound OoD segments.Their efficiency becomes more pronounced after entropy maximization increases OoD-prediction sensitivity.
  • Future work will use OoD training to retrieve OoD objects and assess their importance and whether a new concept should be learned.
  • The method maximizes softmax entropy on out-distribution inputs by minimizing an out-distribution loss over the softmax probabilities.The loss is equivalent to entropy maximization by Jensen’s inequality, with equality for uniform class probabilities.

B. Separability by means of Data Distribution

The analysis evaluates how entropy distributions separate in-distribution and OoD pixels and builds segment-level meta classifiers from softmax-derived uncertainty, geometry, and neighborhood metrics. Variance features have a particularly strong impact on meta-classifier performance.

  • Violin plots summarize entropy distributions for in-distribution and OoD pixels using medians, interquartile ranges, full distributions, and relative pixel-frequency densities.
  • Baseline LostAndFound OoD pixels have higher median entropy than road pixels, but substantial overlap prevents perfect threshold-based separation.The positive-class median is 0.29 versus 0.02 for the negative class, while the positive 10th percentile is 0.04.
  • After OoD training, LostAndFound positive pixels concentrate at higher entropy values while the negative distribution changes only slightly, improving separability.
  • Variance features measure deviations from segment-level score means and have a great impact on OoD meta-classifier performance.
  • The meta-classification dataset feeds a logistic regression that predicts whether an OoD segment intersects the ground truth without using ground-truth segmentation as input.
  • Figures 10 and 11 compare baseline and Cityscapes-void-retrained separability for Cityscapes and LostAndFound pixels, respectively.

D. OoD Training with Cityscapes void Class

Cityscapes void objects can improve entropy-based detection of similar unlabeled objects, but the retraining is too dataset-specific to generalize reliably to LostAndFound OoD objects. Segmentation quality changes only marginally in the shown scenes.

  • Cityscapes void OoD training uses unlabeled objects ignored during Cityscapes training as the out-distribution proxy for entropy maximization.The best reported DeepLabv3+ setting used 8 epochs and out-distribution loss weight λ = 0.05.
  • The visual comparison shows marginal changes in original segmentation performance, while entropy inside Cityscapes unlabeled objects is clearly boosted.
  • The retrained model clearly improves identification of unseen unlabeled objects in the Cityscapes validation dataset.
  • Cityscapes void retraining decreases LostAndFound OoD detection performance because it boosts entropy for many in-distribution pixels as well as OoD pixels.The OoD detection AUROC decreases by 5 percentage points to 0.88.
  • On LostAndFound, many in-distribution pixels also receive boosted entropy, harming separability and making Cityscapes void training unsuitable for detecting other OoD objects.

E. OoD Training for DualGCNNet

OoD training with COCO proxy images improves DualGCNNet’s pixel-wise OoD separation on LostAndFound and Fishyscapes, while entropy-based detection and meta classification jointly reduce missed and false-positive predictions with little segmentation loss.

  • OoD training results: OoD training with COCO proxy images improves DualGCNNet’s OoD detection on both LostAndFound and Fishyscapes.The model is weaker and more lightweight than DeepLabv3+, and the evaluation uses entropy thresholding against the original model.
  • OoD training results: 9 percentage points: Fishyscapes AUROC increases to 0.94 after OoD training, matching the resulting LostAndFound AUROC.LostAndFound AUROC rises by 3 percentage points, while Fishyscapes rises by 9 percentage points.
  • OoD training results: 20 percentage points: Fishyscapes PR AUC improves to 0.38, while LostAndFound PR AUC improves by 15 percentage points to 0.51.These post-training PR AUC scores exceed those of the plain DeepLabv3+ baseline.
  • Entropy behavior: OoD training improves separation partly by increasing entropy on OoD pixels and decreasing entropy on in-distribution pixels.The violin distributions retain similar shapes across much of the in-distribution data, while entropy heatmaps show high-entropy pixels becoming more concentrated inside OoD objects.
  • Object-level detection: At t = 0.7, OoD training significantly reduces overlooked LostAndFound objects across all four road-hazard classes.The classes are humans, standard objects, emotional hazards, and random hazards.
  • Segmentation trade-off: 76.45% AUPRC: DeepLabv3+ peaks during OoD training, while Cityscapes mIoU loses less than 1 percentage point from its 90.30% baseline.The reported comparison uses LostAndFound test AUPRC and Cityscapes validation mIoU over training epochs.
  • Meta classification: Combining OoD training with meta classification yields the fewest false-positive OoD predictions in the illustrated scenes.Meta classification removes false predictions produced after the entropy boost, including small segments under the paper’s one-pixel detection definition.

J. Meta Classification Feature Analysis

The paper analyzes transparent linear meta classifiers and compares OoD detection methods by feature influence and inference time. OoD training makes entropy more influential and remains substantially faster than most uncertainty-based alternatives.

  • Feature-selection procedure: LARS selects the linear meta-classification model by adding predictors according to their correlation with responses under an L1 coefficient penalty.The procedure starts with all coefficients at zero and adds one active predictor at each step.
  • Meta-classifier performance: At t = 0.3, average precision is 98.84 for the plain model and 99.53 after OoD training.The threshold was selected because it yielded the best linear models for both meta-classification cases.
  • Feature influence: After OoD training, entropy E becomes the most important meta-classification metric.The analysis reports that hand-crafted features become active later and therefore have greater correlation and impact on the response.
  • Interpretability: Linear meta classifiers expose the contribution of each metric while providing a lightweight monitoring method for safer and more transparent applications.The paper uses logistic regressions and LARS to analyze the correlations of their hand-crafted metrics.
  • Detection pipeline: The two-step procedure combines entropy thresholding with meta classification to remove incorrect OoD object predictions from segmentation outputs.The visual comparisons include plain-model thresholding, OoD-trained thresholding, and OoD-trained predictions with meta classifiers.
  • Compared OoD scores: The method evaluates pixel-wise OoD scores using alternatives including maximum softmax probability, ODIN, Mahalanobis distance, and Monte Carlo dropout variance.Monte Carlo dropout uses the sum of class-wise empirical variances, while Mahalanobis scoring assumes class-conditional Gaussian latent features.
  • Inference time: 70 seconds: Mahalanobis distance is slower than the proposed OoD training approach, while ODIN differs by 17 seconds under the reported comparison.The comparison uses the same input and segmentation architecture for all methods.
  • Seed analysis: Across eight COCO-subsetting seeds, the paper reports averaged performance scores and standard deviations for LostAndFound, Fishyscapes, and Cityscapes.Each run uses 10 OoD-training epochs with DeepLabV3+ and λ = 0.9.
Loading 2012.06575v2…