PaperlayerView exact revision on arXiv

Source-linked AI summary

SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation

Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, Matthias Rottmann

arXiv:2104.14812v2cs.CV

TL;DR

Closed-set segmentation networks are poorly equipped to detect previously unseen objects, while existing anomaly benchmarks lack sufficiently reliable and diverse real-world evaluation. SegmentMeIfYouCan addresses this gap with two real-image datasets and a benchmark suite covering anomaly and road-obstacle segmentation. The datasets broaden the diversity and difficulty of both evaluation landscapes, while experiments show strong separation between specialized and non-specialized methods but continued room for improvement, especially on component-wise metrics.

  • Problem

    Closed-set segmentation DNNs are overconfident on previously unseen objects, and existing benchmarks are limited by synthetic data, inconsistent labels, or insufficient evaluation infrastructure.

  • Method

    The paper introduces SegmentMeIfYouCan with RoadAnomaly21, RoadObstacle21, a public leaderboard, and pixel-wise and component-wise evaluation metrics.

  • Results

    The datasets contain more varied object types, object sizes, and scenes than existing datasets, separating specialized anomaly or obstacle methods from other methods while leaving substantial room for improvement.

  • Takeaways & Limitations

    The benchmark provides a unified, publicly available basis for comparing anomaly-segmentation methods across compatible datasets and supports research on unknown-object and road-obstacle localization.

  • Takeaways & Limitations

    The benchmark cannot certify fitness for real-world deployment, and its datasets are still insufficient to represent every possible anomaly type.

Abstract

from arXiv · show

State-of-the-art semantic or instance segmentation deep neural networks (DNNs) are usually trained on a closed set of semantic classes. As such, they are ill-equipped to handle previously-unseen objects. However, detecting and localizing such objects is crucial for safety-critical applications such as perception for automated driving, especially if they appear on the road ahead. While some methods have tackled the tasks of anomalous or out-of-distribution object segmentation, progress remains slow, in large part due to the lack of solid benchmarks; existing datasets either consist of synthetic data, or suffer from label inconsistencies. In this paper, we bridge this gap by introducing the "SegmentMeIfYouCan" benchmark. Our benchmark addresses two tasks: Anomalous object segmentation, which considers any previously-unseen object category; and road obstacle segmentation, which focuses on any object on the road, may it be known or unknown. We provide two corresponding datasets together with a test suite performing an in-depth method analysis, considering both established pixel-wise performance metrics and recent component-wise ones, which are insensitive to object sizes. We empirically evaluate multiple state-of-the-art baseline methods, including several models specifically designed for anomaly / obstacle segmentation, on our datasets and on public ones, using our test suite. The anomaly and obstacle segmentation results show that our datasets contribute to the diversity and difficulty of both data landscapes.

1 Introduction

The paper identifies a gap in evaluating segmentation of previously unseen objects and introduces SegmentMeIfYouCan to address it with real, diverse datasets and an evaluation suite.

  • Closed-set segmentation DNNs tend to be overconfident on previously unseen objects, limiting anomaly identification from uncertainty.
  • Existing benchmarks hinder reliable evaluation through limited real-world diversity, missing benchmark infrastructure, synthetic images, or inconsistent labels.
  • SegmentMeIfYouCan provides two manually annotated real-image datasets, a public leaderboard, and an evaluation suite for method comparisons.
  • The benchmark separates strict anomaly segmentation from obstacle segmentation, which includes known and unknown objects on the road.
  • RoadAnomaly21 contains 100 images with anomalies appearing anywhere, while RoadObstacle21 contains 327 images focused on obstacles in the road region.

2 Related Work

Prior anomaly-segmentation benchmarks and methods provide useful foundations but remain limited by narrow or synthetic data, inconsistent labels, and weak comparison infrastructure.

  • Earlier anomaly-detection studies commonly separate images from different source distributions, unlike pixel-level road anomaly segmentation.
  • LostAndFound includes nine anomaly types but contains labeling mistakes and treats children and bicycles as anomalies despite their presence in Cityscapes training data.
  • Fishyscapes and CAOS supplement limited-diversity real images with synthetic data that is not representative of real-world situations.
  • The benchmark hides test labels, uses server-submitted predictions, and supplies a leaderboard to reduce overfitting and improve comparison reliability.
  • The proposed datasets emphasize diverse anomaly categories and scenes, with 26 object types in RoadAnomaly21 and 31 categories in RoadObstacle21.

3 Benchmark Description

The benchmark defines complementary anomaly and obstacle tracks, uses real annotated street scenes, and evaluates both pixel-level and component-level detection behavior.

  • Benchmark Tracks: Anomaly segmentation identifies regions containing object classes never seen during training, while obstacle segmentation identifies hazards on the road regardless of category.
  • RoadAnomaly21: RoadAnomaly21 contains 100 pixel-annotated street images with anomalous objects appearing anywhere in diverse environments.
  • RoadObstacle21: RoadObstacle21 contains 327 annotated images whose objects of interest always appear on the road ahead as realistic hazardous obstacles.
  • Labeling: Both datasets use anomaly or obstacle, not anomaly or obstacle, and void labels; regions outside the road are ignored in the obstacle track.
  • Pixel level: AuPRC emphasizes the minority anomaly class, while FPR95 measures false positives required to reach a 95% true-positive rate.
  • Component level: Component-wise metrics assess localization independent of object size and use adjusted sIoU to avoid penalizing predictions covering multiple ground-truth components.

4 Numerical Experiments

The benchmark applies thresholded anomaly masks and pixel- and component-level metrics to compare anomaly and obstacle segmentation methods across RoadAnomaly21, RoadObstacle21, and public datasets. Generative and entropy-based methods perform better than classification-based approaches, but smaller objects and component-level evaluation remain challenging.

  • Evaluation protocol: The default benchmark pipeline thresholds pixel-wise anomaly scores at δ∗ selected by the optimal pixel-wise F1-score.Components below 500 pixels for anomaly segmentation and 50 pixels for obstacle segmentation are discarded.
  • Evaluation protocol: Component-wise sIoU and PPV are averaged across ground-truth or predicted components and multiple thresholds τ.This complements pixel-wise metrics with measures less dominated by large anomalies or obstacles.
  • Results: Classification-based methods such as maximum softmax, ODIN, and Mahalanobis do not generalize well to anomaly and obstacle segmentation.Domain shift particularly degrades statistics-based methods on RoadObstacle21, where road surfaces vary.
  • Results: Generative methods are better suited to both anomaly and obstacle segmentation at pixel and component levels, while maximized softmax entropy reaches 86% AuPRC and 49% component-wise F1.The entropy result outperforms the discussed autoencoder-based methods in the reported experiments.
  • Results: Component-level evaluation exposes weaknesses more clearly than pixel-wise evaluation, and all methods struggle particularly with smaller anomalies and obstacles.The reported component-wise F1 is much lower than the optimal pixel-wise F1.
  • Cross-dataset evaluation: On the LiDAR-guided Small obstacle Segmentation dataset, the full method set performs weakly because its conditions are challenging for camera-only methods.The dataset is designed for small-obstacle detection using multiple sensors, including LiDAR.

5 Conclusion

The paper introduces a unified benchmark suite with pixel- and component-level metrics, plus two publicly available datasets for anomaly and obstacle segmentation. Experiments show that the datasets provide more varied and difficult challenges, while component-wise performance still leaves substantial room for improvement.

  • Contributions: The benchmark suite evaluates anomaly segmentation with established pixel-level and recent component-level metrics and can be applied across datasets with ground truth.It is accompanied by the RoadAnomaly21 and RoadObstacle21 datasets.
  • Datasets: RoadAnomaly21 and RoadObstacle21 use real street-scene images with pixel-level annotations and greater variability in object types and sizes than existing datasets.They target unknown-object localization and obstacle detection on the road, respectively.
  • Findings: The experiments show a distinct performance separation between methods specifically designed for anomaly or obstacle segmentation and methods that are not.The datasets therefore provide challenging evaluation settings for these capabilities.
  • Findings: Component-wise metrics remain particularly difficult, highlighting substantial room for future improvement in anomaly segmentation.This conclusion follows from the reported performance across the two datasets.

Broader Impact

The benchmark supports research toward safer autonomous-vehicle deployment but does not provide real-world safety certification. Its datasets remain limited in anomaly coverage, and benchmark overfitting remains possible.

  • Scope and limitations: The benchmark measures algorithm reliability under its specified training regime, but cannot certify fitness for real-world deployment.Its results should not be interpreted as safety guarantees.
  • Scope and limitations: The datasets add anomaly diversity, yet their scale remains insufficient to represent every possible anomaly type.
  • Scope and limitations: Methods may overfit the benchmark tasks despite hidden test labels, a risk shared with other benchmarks.
  • Availability and reproducibility: The benchmark is distributed with documentation, open licensing, persistent dataset identifiers, repositories, and multiple hosting mirrors.The anomaly and obstacle datasets have DOI identifiers, while code is hosted on GitHub.

B Datasheet for Datasets

The datasheet describes a pixel-annotated benchmark of unusual street-scene objects for evaluating anomaly segmentation methods in driving scenes. It documents sampling, composition, distribution, and important geographic, annotation, and identification boundaries.

  • Purpose: The dataset was created to evaluate and compare anomaly segmentation methods in driving scenes and support autonomous-driving method development.
  • Composition: The images are high-resolution street scenes with unusual objects and pixel-level annotations; anomaly-track objects occur naturally, while obstacle-track objects were selected and placed by the authors.
  • Composition: The anomaly track contains 100 images with 262 ground-truth components, while the obstacle track contains 327 images with 388 components.Additional validation and hard-weather images are listed separately.
  • Composition: The obstacle data cover weather and seasons but are highly biased toward European street backgrounds and selected objects from Switzerland and Germany.
  • Distribution and use: The test set is intended for testing only, with test labels withheld; validation labels are public and data are distributed through multiple mirrors.
  • Limitations and privacy: Human annotations can contain errors, and faces in anomaly-track images may be matchable to other databases because those images were already public.
  • Data acquisition: Anomaly-track images are a random sample of openly licensed web images, whereas obstacle-track images were collected from sequences at selected distances.

C More Details on Evaluation Metrics

The evaluation combines pixel-wise anomaly metrics with component-level analysis to address class imbalance and errors caused by mismatched component splitting or merging. It includes threshold-independent separability and safety-oriented false-positive assessment.

  • Pixel-wise metrics: AuPRC evaluates pixel-wise anomaly-score separability across precision–recall thresholds and is threshold independent.
  • Pixel-wise metrics: AuPRC emphasizes the minority class, making it suitable for the highly unbalanced RoadAnomaly21 and RoadObstacle21 pixel distributions.
  • Safety-oriented metric: FPR95 measures the false-positive rate required to achieve a 95% true-positive rate, with void-region predictions excluded from false positives.For RoadObstacle21, evaluation is restricted to the road area.
  • Component-wise metrics: Component-wise IoU can punish predictions when one component merges multiple ground-truth components or splits one ground-truth component into several predictions.
  • Component-wise metrics: Figure 5 illustrates that adjusted IoU (sIoU) can differ substantially from IoU for merged or split components.For example, one green component has IoU 68.18% versus sIoU 87.01%.

D.1 Brief Description of Methods

The benchmark evaluates diverse anomaly-score approaches built on Cityscapes-trained semantic segmentation networks, including uncertainty, density, resynthesis, classifier, and entropy-based methods. Pixel-wise scores are designed so higher values indicate greater anomaly likelihood.

  • Common setup: All evaluated methods use a Cityscapes-trained semantic segmentation DNN and produce pixel-wise anomaly scores.
  • Baseline methods: MSP, ODIN, and Mahalanobis distance adapt image-classification anomaly-detection baselines to segmentation.MSP uses maximum softmax probability, ODIN adds perturbations and temperature scaling, and Mahalanobis distance models latent-feature distributions.
  • Uncertainty methods: MC dropout and ensembles use mutual information from multiple predictive samples or models as pixel-wise anomaly scores.
  • Resynthesis methods: Image resynthesis compares the original image with a segmentation-conditioned reconstruction using a discrepancy network, while road inpainting applies this idea specifically to road patches.
  • Density-based method: Learned embedding density estimates training-feature likelihood with a normalizing flow, then upsamples negative log-likelihood scores to image resolution.
  • Entropy-based method: Maximized entropy retrains a pretrained DNN with Cityscapes and a COCO out-distribution subset, using softmax entropy to score anomalies.

E Parameter Study

The parameter study examines how threshold choices and component filtering affect component-wise evaluation, finding that the default pixel threshold is generally near-optimal while post-processing remains useful.

  • Threshold selection: δ* selects the pixel threshold that maximizes pixel-wise F1 before component-wise performance is evaluated across τ.τ determines component matching through sIoU, whereas δ converts pixel-wise anomaly scores into binary masks.
  • Threshold selection: For most methods, δ* produces component-wise F1 close to its optimum, with the largest gaps reaching 2.8 percentage points for maximized entropy and 4.1 percentage points for maximum softmax.The 4.1-point case occurs on RoadObstacle21; the 2.8-point case occurs on RoadAnomaly21.
  • Threshold selection: Except for maximum softmax on RoadObstacle21, δ* lies within 0.05 F1 of the optimal δ; that exception has a distance of 0.30.Some methods reach their optimum directly at δ*, including MC dropout, SynBoost, and the void classifier in the reported tracks.
  • Benchmark procedure: The default δ* mask-generation procedure is therefore near-optimal, but competitors may submit masks produced with more sophisticated image operations and post-processing.The benchmark explicitly encourages competitors to improve mask generation when beneficial.
  • Benchmark procedure: Default post-processing removes predicted components smaller than 500 pixels in the anomaly track and 50 pixels in the obstacle track to reduce false-positive components.Filtering is optional, and the benchmark recommends it when no more sophisticated post-processing is used.

F.1 RoadAnomaly21 & RoadObstacle21 Validation Dataset

The validation and public-dataset evaluations probe benchmark behavior across anomaly and obstacle settings, revealing that scene diversity strongly affects difficulty and camera-only methods struggle on barely visible obstacles.

  • Validation datasets: The benchmark supplies validation splits with 10 images and 16 ground-truth components for anomaly segmentation, and 30 images with 45 ground-truth objects for obstacle segmentation.These sets are intended to verify that methods run as intended with the benchmark code.
  • Validation datasets: The validation splits are not representative of test data because they contain limited road-surface and obstacle-type diversity.They share the corresponding test setup but cover only a small range of scene variation.
  • Fishyscapes LostAndFound: Fishyscapes LostAndFound shows relatively strong component-wise performance and a smaller gap between classification and anomaly-specific methods, consistent with limited environmental diversity.Its images resemble Cityscapes and anomalies are concentrated in smaller, more central regions.
  • RoadObstacle21: RoadObstacle21 distributes obstacles across more road surfaces and image locations than LostAndFound, while its 31 object types increase environment–obstacle combination diversity.The comparison datasets obtain larger image counts partly through dense video-frame sampling.
  • Small Obstacle dataset: SynBoost misses 1,100 of 1,203 components at τ = 0.25 on the LiDAR-guided Small Obstacle dataset, whose low illumination, blur, and barely visible obstacles challenge camera-only segmentation.The dataset is considered less suitable for camera-only obstacle segmentation because cameras do not capture the obstacles well.
  • CAOS BDD-Anomaly: CAOS BDD-Anomaly is excluded because its designated anomalies are not strictly unknown to Cityscapes-trained models and its labels contain mistakes.The reported mistakes include in-distribution objects such as trains and bicycles annotated as anomalies.

G Evaluation per Environment Category

Evaluation across nine RoadObstacle21 environment categories shows no method is consistently best, with road texture, weather, lighting, and surrounding context creating distinct difficulties.

  • Environment categories: RoadObstacle21 is divided into nine environment categories spanning cracked, dark, gravel, gray-asphalt, motorway, wet, brick, night, and snowstorm scenes.The categories vary road surface, weather, illumination, and surrounding environment.
  • Environment categories: No evaluated method performs consistently across all environment subsets.The per-category evaluation reports different method behavior across the data splits.
  • Environment categories: Road texture, surrounding environments, weather, and lighting add difficulty; cracks and leaves may trigger false positives, while gravel or wet surfaces may resemble obstacles.These conditions create distinct challenges across the nine scene categories.

H Evaluation for Different Component Sizes

The benchmark evaluates segmentation quality across ground-truth component sizes and finds that larger components are handled better, while small obstacles are frequently overlooked entirely.

  • RoadAnomaly21 contains 259 ground-truth components ranging from 122 to 883,319 pixels, while RoadObstacle21 contains 388 obstacles ranging from 18 to 77,435 pixels.
  • sIoU positively correlates with component size across the evaluated methods.The relationship is especially pronounced in the size-based analysis of RoadAnomaly21 and RoadObstacle21.
  • In RoadObstacle21, maximized entropy and SynBoost outperform other approaches, particularly on component-wise segmentation quality.These methods were designed for anomaly segmentation rather than specifically for obstacle segmentation.
  • Small components are disproportionately overlooked, revealing a major component-level challenge for anomaly segmentation.A component counts as detected when at least one pixel is detected, using τ = 0 to distinguish false negatives from true positives.

I Evaluation per Object Category

The benchmark adds category-, scene-, size-, and sequence-level analyses to characterize method behavior and dataset difficulty. These analyses highlight challenging vehicle anomalies, small or distant obstacles, and differences in temporal sampling between datasets.

  • Evaluation per Object Category: Anomaly-specific methods perform worse on vehicles than on other anomaly categories in RoadAnomaly21.The vehicle subset includes challenging classes such as jet ski, rickshaw, and carriage.
  • Benchmark Coverage: The benchmark reports results across RoadAnomaly21, RoadObstacle21, Fishyscapes LostAndFound, LostAndFound test-NoKnown, and LiDAR guided Small obstacle Segmentation datasets.The corresponding evaluation sets contain 16, 45, 165, 1709, and 1203 ground-truth instances or components, respectively.
  • Evaluation per Object Category: RoadAnomaly21 includes 59 animal-only images, 23 vehicle-only images, and 11 images with other anomalies; mixed animal-and-vehicle images are excluded.The excluded mixed subset contains 7 images.
  • Qualitative Comparisons: RoadObstacle21 and RoadAnomaly21 provide qualitative examples spanning large anomalies, unfamiliar tractors, small distant obstacles, and cracked road surfaces.Green contours mark anomaly or obstacle annotations, while red contours mark the road in RoadObstacle21 examples.
  • Sequence Sampling: LostAndFound test sequences can contain 18 densely sampled, nearly identical frames, whereas RoadObstacle21 includes at most 6 frames that may differ substantially in lighting.
Loading 2104.14812v2…