PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel-wise Anomaly Detection in Complex Driving Scenes

Giancarlo Di Biase, Hermann Blum, Roland Siegwart, Cesar Cadena

arXiv:2103.05445v1cs.CV

TL;DR

Semantic segmentation models often fail to localize anomalous instances, limiting their use in safety-critical driving scenes. This paper combines uncertainty maps with re-synthesis-based image comparison in a wrapper framework for trained segmentation networks. The approach reports state-of-the-art anomaly detection on Fishyscapes while preserving segmentation quality.

  • Problem

    Semantic segmentation models can fail to localize anomalous instances, limiting deployment in safety-critical applications such as autonomous driving.

  • Method

    The framework combines uncertainty measurements with re-synthesis and guides a dissimilarity network to compare input images with images generated from predicted semantic maps.

  • Results

    The approach significantly outperforms re-synthesis and uncertainty-based methods on Fishyscapes and is the best overall method across datasets.

  • Takeaways & Limitations

    The framework can be used with already trained segmentation networks without constraining segmentation accuracy or requiring retraining.

  • Takeaways & Limitations

    The training procedure covers only one anomaly scenario, uses ground-truth semantic maps for training but predicted maps at inference, and cannot evaluate Street Hazards because it lacks required instance labels and a void class.

Abstract

from arXiv · show

The inability of state-of-the-art semantic segmentation methods to detect anomaly instances hinders them from being deployed in safety-critical and complex applications, such as autonomous driving. Recent approaches have focused on either leveraging segmentation uncertainty to identify anomalous areas or re-synthesizing the image from the semantic label map to find dissimilarities with the input image. In this work, we demonstrate that these two methodologies contain complementary information and can be combined to produce robust predictions for anomaly segmentation. We present a pixel-wise anomaly detection framework that uses uncertainty maps to improve over existing re-synthesis methods in finding dissimilarities between the input and generated images. Our approach works as a general framework around already trained segmentation networks, which ensures anomaly detection without compromising segmentation accuracy, while significantly outperforming all similar methods. Top-2 performance across a range of different anomaly datasets shows the robustness of our approach to handling different anomaly instances.

1. Introduction

Semantic segmentation models can fail to localize anomalous objects, limiting deployment in safety-critical driving scenarios. The paper combines uncertainty and re-synthesis information in a general framework that improves anomaly detection while preserving segmentation accuracy.

  • Semantic segmentation networks can fail on inputs outside their training distribution despite high overall accuracy.An anomalous object may be wrongly classified as road, creating a safety risk for autonomous driving.
  • Uncertainty-based methods and re-synthesis methods address anomaly detection differently but each has characteristic failure cases.Uncertainty estimates can be noisy, while re-synthesis struggles when the predicted semantic map contains noisy unknown-object regions.
  • The proposed pixel-level framework combines uncertainty and re-synthesis approaches because their information is complementary across anomaly scenarios.The framework uses uncertainty to assist detection of differences between the input and generated images.
  • The method integrates softmax entropy, softmax difference, and perceptual differences into a re-synthesis-based dissimilarity pipeline.It compares the input image with an image generated from the predicted semantic map.
  • The approach achieves state-of-the-art Fishyscapes performance while maintaining state-of-the-art segmentation accuracy across different anomaly scenarios.The authors also evaluate lighter architectures for speed in time-critical autonomous-driving settings.
  • The framework generalizes across segmentation and synthesis networks and operates as a wrapper around existing segmentation pipelines.It requires no segmentation retraining and is designed to maintain segmentation performance.

2. Related Work

Related work detects pixel-level anomalies through uncertainty estimation, outlier exposure, or image re-synthesis. The paper builds on re-synthesis by adding uncertainty information to improve discrepancy-based anomaly detection in complex scenes.

  • Pixel-wise anomaly localization is studied as out-of-distribution detection and anomaly segmentation, distinct from image-level OoD classification.
  • Uncertainty methods assume anomalous inputs produce higher uncertainty than training data, but high uncertainty can also arise from non-anomalous input noise.
  • Softmax-based and Bayesian uncertainty methods often struggle with pixel-level localization because uncertainty concentrates at object boundaries or produces false positives and mismatches.
  • Outlier-exposure methods train segmentation networks to predict anomaly maps using auxiliary outlier data, but modifying the network for multitask prediction can reduce segmentation performance.
  • Re-synthesis methods reconstruct images from predicted semantic maps and detect anomalies through appearance differences, avoiding segmentation retraining and OoD samples.
  • The paper extends re-synthesis by feeding scene uncertainty to the discrepancy network, improving its ability to detect anomalies in complex driving scenes.

3. Methodology

The framework combines semantic re-synthesis with uncertainty measurements to detect pixel-level anomalies without changing the underlying segmentation task. It compares original, generated, semantic, and uncertainty representations through a dissimilarity module trained with synthetic anomaly examples and void-class objects.

  • Framework: The framework segments the input, synthesizes a realistic image from the semantic map, computes perceptual differences, and predicts anomalies from all resulting representations.The pipeline produces a semantic map, softmax entropy, softmax distance, synthesized image, and perceptual difference before spatial-aware anomaly prediction.
  • Segmentation Module: Softmax entropy and softmax distance quantify uncertainty in the semantic prediction, with both measures normalized to [0, 1].Softmax distance is the difference between the two largest softmax values.
  • Synthesis Module: Perceptual difference compares corresponding original and synthesized pixels through features from multiple VGG layers because semantic maps omit appearance information.The resulting feature map is normalized between [0, 1].
  • Dissimilarity Module: The dissimilarity module encodes each input, concatenates input, synthesis, and semantic features, correlates them with uncertainty features, and decodes an anomaly map.SPADE normalization injects the semantic map during decoding to preserve spatial semantic information.
  • Training Procedure: The segmentation and synthesis networks are trained only on inlier data, while the dissimilarity module uses synthetic altered maps and void-class objects to represent anomalies.Using void-class objects introduces OoD data during training, but combining both generators supports generalization to unseen objects.

4. Experiments

The experiments evaluate the framework on Fishyscapes, compare it with anomaly-detection baselines under different segmentation-accuracy constraints, and assess generalization and practical limitations.

  • Datasets: The evaluation uses Fishyscapes, comprising FS Lost & Found, FS Static, and FS Web, with quantitative results submitted on private test sets.FS Lost & Found contains 275 real images; FS Static blends Pascal VOC objects into Cityscapes images, while FS Web uses newly crawled internet objects.
  • Limitations: The framework could not be evaluated on Street Hazards because its training procedure requires instance labels and a void class unavailable in that dataset.This is a dataset-compatibility boundary rather than a reported performance result.
  • Baselines: The comparison includes uncertainty-based, outlier-exposure, and image-resynthesis methods, separating approaches that retrain segmentation networks from wrapper methods.The split measures whether anomaly detection compromises semantic segmentation accuracy, using class mIOU on Cityscapes.
  • Results: The framework outperforms previous methods that preserve segmentation performance on all three datasets, significantly improving AP and reducing FPR95 by 50% on FS L&F and FS Web.Its FPR95 is comparable to the previous best method on FS Static.
  • Results: Against all existing approaches, the model achieves the best performance on FS L&F and second-best AP on FS Static and FS Web.It trails Outlier Head by 16% on FS Static and 4% on FS Web, while maintaining state-of-the-art semantic-segmentation performance.
  • Generalization: The method is the only approach with top-2 performance across the Fishyscapes test sets, whereas competing methods perform unevenly across datasets.Dirichlet DeepLab performs strongly on FS L&F but weakly on FS Static and FS Web; Outlier Head shows the opposite pattern.

5. Discussion

The discussion attributes performance gains to combining data generation, uncertainty maps, and ensemble predictions, then examines robustness across architectures and the associated trade-offs.

  • 5.1. Ablation Study: Both the training-data generator and uncertainty maps significantly improve the framework, supporting coverage of three anomaly scenarios and complementarity between resynthesis and uncertainty.The ablation study reports results as averages and standard deviations over five random weight initializations.
  • 5.1. Ablation Study: Combining softmax entropy, softmax distance, and perceptual-difference maps with the dissimilarity output greatly reduces FPR95 on both datasets.The authors attribute this to mitigating overconfidence in the dissimilarity module.
  • 5.1. Ablation Study: Ensembling can reduce AP on FS Lost & Found while improving AP on FS Static, with the authors expecting smaller AP losses but larger FPR95 gains overall.The FS Lost & Found drop results from combining the framework with a weaker detector to reduce FPR95 for safety-critical applications.
  • 5.1. Ablation Study: The ensemble produces more consistent performance across training runs and metrics because standalone dissimilarity-module training has less influence on the combined system.The authors identify this consistency as desirable for safety-critical deployment.
  • 5.2. Framework Generalization: The dissimilarity module is a wrapper independent of the specific segmentation and synthesis architectures, provided the segmentation network outputs a softmax layer.This design allows the anomaly detector to be placed over already trained networks.
  • 5.2. Framework Generalization: Ours Light significantly outperforms Image Resynthesis++ despite using lower-performing segmentation and synthesis modules, including 83.5% versus 70.6% class mIOU on Cityscapes.The lighter pipeline uses ICNet and SPADE to prioritize speed.
  • 5.2. Framework Generalization: Anomaly-detection accuracy correlates with segmentation and synthesis quality, and the full framework outperforms Ours Light across all metrics.The authors expect future improvements in these modules to improve resynthesis-based anomaly detection.

6. Conclusion

The framework combines uncertainty and re-synthesis methods for robust pixel-level anomaly detection in complex driving scenes. It outperforms both method types on Fishyscapes while supporting already trained and lighter networks.

  • The framework combines uncertainty measurement maps with re-synthesis to guide a dissimilarity network in detecting differences between input and generated images.The uncertainty maps include softmax entropy, softmax distance, and perceptual differences.
  • It significantly outperforms both re-synthesis-based and uncertainty-based methods on the Fishyscapes benchmark.The paper describes it as the best overall method across datasets.
  • The approach imposes no constraint on the segmentation network and can use already trained state-of-the-art models.
  • The method also works with lighter segmentation and synthesis networks, supporting deployment in autonomous machines.

A.1. Implementation of Dissimilarity Network

The dissimilarity network uses multiple encoders, a fusion module, and decoder blocks to compare input, synthesized, semantic, and uncertainty features. It is trained with Adam and standard image augmentation.

  • The dissimilarity network comprises two encoding architectures, one fusion module, and three decoder blocks reused across the final architecture.The architecture is illustrated as interconnected components in Figure 3.
  • The architecture uses separate encoders for input/generated images and semantic/uncertainty maps, with shared weights only for the input and synthesized image encoder.The semantic and uncertainty encoders use different weights.
  • The fusion module concatenates input, synthesized, and semantic features, applies a 1x1 convolution, and correlates the result with uncertainty features.This produces four feature maps, one at each resolution.
  • Four decoder blocks progressively process fused multiresolution features and produce the final dissimilarity output.The blocks use convolution, SPADE normalization, transposed convolution, and a final 1x1 convolution with two filters.
  • The dissimilarity network was trained for fifty (50) epochs with Adam at a learning rate of 0.0001.The learning rate was reduced on plateau with patience of 10 epochs, alongside vertical flipping and ImageNet normalization.

A.2. Re-Implementation of Image Re-Synthesis

The re-implementation isolates the proposed contributions by using matched state-of-the-art segmentation and synthesis networks, then compares Image Resynthesis++ with the original method on anomaly datasets.

  • Image Re-synthesis predicts a semantic map, re-synthesizes it into an image, and detects anomalies by comparing the natural and synthesized images.The framework consists of segmentation, synthesis, and discrepancy networks.
  • Re-synthesis performance depends strongly on segmentation and synthesis quality because better predictions simplify anomaly detection for the discrepancy network.
  • The re-implementation replaces the original segmentation and synthesis networks with the same networks used by the proposed framework.This makes the compared methods differ only in the contributions described in Section 3.
  • Image Resynthesis++ outperforms the results reported in the original work across Lost & Found and Road Anomaly using AUC ROC.The comparison uses the same datasets and metric as the original publication.
  • The main experiments use average precision and FPR95 because ROC metrics are not well-suited to highly imbalanced anomaly-detection problems.AUC ROC was retained for comparison because it was the only metric reported in the original work.
  • Road Anomaly was excluded from the main experiments because it contains only sixty (60) images and inconsistent anomaly annotations.For example, similar rocks can be labeled differently depending on whether they are on or beside the road.

A.3. Computational Complexity

The computational-complexity evaluation measures module inference times, including the CNN required for perceptual-difference dispersion maps, across repeated runs on an NVIDIA 1080Ti GPU.

  • Inference times are averaged over one hundred (100) runs for each module of the proposed approach.
  • The running complexity includes the inference time of the CNN used to compute the perceptual difference dispersion map.
  • The evaluation uses an NVIDIA 1080Ti GPU with 11GB memory and the input resolutions specified in Section 4.1.

A.4. Ensemble with End-to-End Training

The paper compares empirically selected ensemble weights with end-to-end learned weights, finding comparable AP but substantially worse FPR95 under end-to-end training.

  • The main ensemble combines uncertainty maps and dissimilarity predictions using empirically selected weights from grid search.The uncertainty maps include softmax entropy, softmax distance, and perceptual difference.
  • The alternative end-to-end ensemble learns the combination weights during dissimilarity-network training.This design produces an end-to-end ensemble prediction.
  • End-to-end training produces AP comparable to empirically selected weights, with performance varying between FS Lost & Found and FS Static.The end-to-end ensemble performs better on FS Lost & Found, while empirical weights perform slightly better on FS Static.
  • FPR95 decreases significantly with end-to-end training compared with empirical weights.The authors relate this degradation to overconfident predictions from deep CNN modules, including the dissimilarity module.

A.5. Example Predictions

Qualitative examples show the framework detecting diverse anomaly outcomes across FS Lost & Found and FS Static, while failure cases remain for difficult scenes, background-like anomalies, and small objects.

  • The framework qualitatively outperforms Softmax Entropy and Image Re-synthesis on validation examples from FS Lost & Found and FS Static.Void-labeled pixels are excluded from the displayed predictions and anomaly benchmarks.
  • Failure cases arise in scenes differing from the urban landscape, anomalies blending with the background, and small objects covering few pixels.Softmax Entropy and Image Re-synthesis are shown as reference methods in the failure-case examples.
  • Additional examples indicate reliable detection across correctly classified, over-segmented, and background-blended anomaly outcomes.The examples cover four FS Lost & Found images followed by four FS Static images.
Loading 2103.05445v1…