Source-linked AI summary
Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation
Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, Alan Yuille
TL;DR
Reliable semantic segmentation requires detecting both incorrect predictions and out-of-distribution objects, especially in safety-critical applications. SynthCP synthesizes an image from a predicted segmentation and compares it with the input, improving results across three datasets, including 6% AUPR-Error on Cityscapes, 7% Pearson correlation on pancreatic tumor segmentation in MSD, and 20% AUPR on StreetHazards.
Problem
The paper studies the limited attention given to detecting and locating failures and anomalies in semantic segmentation, a need arising in safety-critical applications.
Method
SynthCP uses a semantic-to-image conditional GAN trained on label-image pairs, then compares the reconstructed image with the input through a task-specific comparison module.
Results
The method achieves state-of-the-art performance across three datasets, with improvements of 6% AUPR-Error, 7% Pearson correlation, and 20% AUPR on the reported tasks.
Takeaways & Limitations
A unified synthesize-then-compare framework can address both failure detection and anomaly segmentation for semantic segmentation.
Takeaways & Limitations
Simple image similarity such as ℓ1 distance is inaccurate because synthesis optimization does not guarantee that the reconstructed image matches the original image style.
Abstract
from arXiv · showhide
The ability to detect failures and anomalies are fundamental requirements for building reliable systems for computer vision applications, especially safety-critical applications of semantic segmentation, such as autonomous driving and medical image analysis. In this paper, we systematically study failure and anomaly detection for semantic segmentation and propose a unified framework, consisting of two modules, to address these two related problems. The first module is an image synthesis module, which generates a synthesized image from a segmentation layout map, and the second is a comparison module, which computes the difference between the synthesized image and the input image. We validate our framework on three challenging datasets and improve the state-of-the-arts by large margins, \emph{i.e.}, 6% AUPR-Error on Cityscapes, 7% Pearson correlation on pancreatic tumor segmentation in MSD and 20% AUPR on StreetHazards anomaly segmentation.
1 Introduction
The paper addresses failure detection and anomaly segmentation for semantic segmentation, motivated by safety-critical applications and the need for pixel-level analysis. SynthCP synthesizes an image from a predicted segmentation and compares it with the input, achieving state-of-the-art results across three datasets.
- Motivation: Safety-critical applications require systems to detect wrong segmentation predictions and identify out-of-distribution cases that may cause failures.The motivation includes autonomous driving, medical diagnosis, and nuclear power plant monitoring.
- Problem: Failure detection must locate segmentation errors, while anomaly segmentation identifies anomalous objects or regions at pixel level.The framework targets both image-level and pixel-level failure analysis.
- Problem: Existing image-classification strategies provide limited support for structured segmentation outputs because they do not account for semantic layout.Semantic segmentation requires more detailed error or confidence maps than single-label classification.
- Approach: SynthCP reconstructs an input image from its predicted segmentation using image synthesis, then compares the reconstruction with the input to detect failures and anomalies.The approach is motivated by the expected correspondence between segmentation quality and synthesized-image similarity.
- Results: 6% AUPR-Error, 7% Pearson correlation, and 20% AUPR improvements over state-of-the-art methods were reported on Cityscapes, pancreatic tumor segmentation in MSD, and StreetHazards, respectively.The results cover pixel-level error prediction, DSC prediction, and anomaly segmentation.
2 Related Work
Prior work largely addressed confidence, quality, and OOD detection for image classification or used reconstruction-based methods for anomaly segmentation. SynthCP instead uses semantic-to-image synthesis to exploit the structured relationship between segmentation layouts and images.
- Existing detection methods: Earlier failure and anomaly detection methods mainly focused on classifying small images rather than semantic segmentation.Segmentation introduces structured outputs and pixel-level detection requirements.
- Quality assessment: Segmentation quality assessment estimates overall segmentation quality without ground-truth labels, using Bayesian models, image-segmentation pairs, or quality-regression heads.These approaches provide alternatives for triggering alarms when segmentation models fail.
- OOD detection: Anomaly segmentation is pixel-level OOD detection that segments anomalous regions, unlike image-level OOD detection that classifies whole images.StreetHazards contains 250 anomalous object types and more than 6k high-resolution images.
- OOD detection: Autoencoder reconstruction methods struggle with complex street scenes and may not generate in-distribution images for OOD regions.Prior work also found MSP competitive with autoencoder and Bayesian approaches on StreetHazards.
- Synthesis-based approach: SynthCP uses a conditional GAN trained for semantic-to-image translation, reversing semantic segmentation to reconstruct images from semantic masks.This design motivates detecting segmentation failures through differences between original and synthesized images.
3 Methodology
SynthCP reconstructs an image from a predicted segmentation layout, then compares reconstruction with the input to detect failures and anomalies at task-appropriate resolutions.
- General Framework: SynthCP uses an image synthesis module to reconstruct the input from a predicted segmentation map, followed by a comparison module.The synthesis module is modeled with a semantic-to-image conditional GAN trained on label-image pairs.
- General Framework: The comparison module estimates task-specific confidence measures because reconstruction style differences make simple pixel similarity unreliable.The framework avoids relying directly on ℓ1 distance between the input and synthesized images.
- Failure Detection: Failure detection predicts both per-class IoU and a pixel-level error map to assess and localize segmentation failures.A light-weight siamese network uses shared-weight encoding and separate heads for these outputs.
- Anomaly Segmentation: Anomaly segmentation outputs a pixel-level confidence map for anomalous objects, using differences between the input and a reconstruction containing only in-distribution objects.The comparison function uses cosine distance between intermediate segmentation-model features, with MSP post-processing to reduce false positives.
- Conceptual Explanation: The framework interprets close reconstructions as evidence of correct segmentation and distant reconstructions as indicators of failures or out-of-distribution cases.This interpretation links reconstruction quality to the reliability of the predicted segmentation layout.
4 Experiments
SynthCP is evaluated for pixel- and image-level failure detection on Cityscapes and pancreatic tumor segmentation, plus anomaly segmentation on StreetHazards. Across these settings, synthesis-based comparison generally outperforms or complements existing methods, with post-processing affecting anomaly metrics.
- Cityscapes: SynthCP evaluates pixel-level error maps and image-level per-class IoU prediction using standard failure-detection metrics.Pixel-level evaluation uses error-map metrics, while image-level evaluation uses MAE, STD, Pearson correlation, and Spearman correlation.
- Cityscapes: SynthCP consistently outperforms other methods for image-level failure detection on Cityscapes, while Direct Prediction performs worse without synthesized images.The comparison includes MCDropout, VAE alarm, MSP, TCP, and Direct Prediction.
- Cityscapes: SynthCP achieves state-of-the-art pixel-level failure detection on Cityscapes, especially for AP-Error, with gains attributed to the image synthesis module over Direct Prediction.The method produces slightly more false positives than Direct Prediction, potentially because some correctly segmented regions are synthesized poorly.
- Cityscapes: 53.12 AUPR-Error and 12.91 MAE are obtained when a failure detector trained on Deeplab-v2 masks is tested on FCN8 masks.These results are comparable to those from a model trained on FCN8 segmentation outputs, supporting generalization across unseen segmentation models.
- Pancreatic tumor segmentation: SynthCP is comparable to VAE alarm on pancreatic tumor segmentation, while their ensemble improves all four metrics, including approximately 7% Pearson and 10% Spearman correlation gains.SynthCP captures label-to-image information that is complementary to the shape-based VAE approach.
- StreetHazards anomaly segmentation: 9.3% AUPR improves on MSP+CRF’s 6.5% for StreetHazards anomaly segmentation, and thresholding at MSP positions above 0.999 yields state-of-the-art performance across three metrics.Without post-processing, AUPR is higher but FPR95 and AUROC degrade because of additional false positives.
5 Conclusions
SynthCP provides a unified framework for detecting failures and anomalies in semantic segmentation. It combines image synthesis with comparison to achieve state-of-the-art performance on three challenging datasets.
- SynthCP unifies failure detection and anomaly detection for semantic segmentation.The framework consists of an image synthesis module and a comparison module.
- The framework achieves state-of-the-art performance on three challenging datasets.