Source-linked AI summary
The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
Hermann Blum, Paul-Edouard Sarlin, Juan Nieto, Roland Siegwart, Cesar Cadena
TL;DR
Fishyscapes addresses whether uncertainty estimates generalize from simple classification to safety-critical, pixel-wise anomaly detection in urban-driving segmentation. The paper builds a benchmark, adapts uncertainty methods, and evaluates them on synthetic and real-world anomalies. Its results show that anomaly detection remains far from safety-critical accuracy, with supervised methods generally outperforming unsupervised ones while important adaptation challenges remain.
Problem
Prior uncertainty methods were mainly evaluated on simple classification tasks, leaving their generalization to complex pixel-wise semantic segmentation unclear despite safety needs in autonomous driving.
Method
Fishyscapes benchmarks pixel-wise uncertainty estimates for anomalous objects in urban-driving scenes using Cityscapes-based synthetic and real-world datasets.
Results
Anomaly detection remains far from solved: on every dataset, the best method used OoD data and was at least 38% better than any unsupervised method.
Takeaways & Limitations
The benchmark enables progress measurement for anomaly segmentation and scalable uncertainty estimation, while showing that current methods are not yet adequate for safety-critical applications.
Takeaways & Limitations
Adapting some methods to semantic segmentation is difficult because special retraining losses can reduce accuracy and may be unstable or fail to converge.
Abstract
from arXiv · showhide
Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for uncertainty estimation in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects in front of the vehicle. We~adapt state-of-the-art methods to recent semantic segmentation models and compare approaches based on softmax confidence, Bayesian learning, and embedding density. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art.
1. Introduction
Fishyscapes addresses the safety problem of detecting anomalous objects that segmentation models may label confidently, extending uncertainty evaluation from simple classification to pixel-wise urban-scene segmentation. It introduces a benchmark, adapts diverse methods, and finds substantial remaining performance gaps.
- Closed-world benchmarks do not reflect the uncontrollable inputs that autonomous systems encounter, where incorrect reactions can have disastrous consequences.
- Outlier objects can be interpolated with available classes at high confidence, making uncertainty information important for safe and reliable decision-making.
- Prior out-of-distribution research focused mainly on image classification and simple datasets, leaving pixel-wise uncertainty in complex segmentation architectures unassessed.
- Fishyscapes is a benchmark for detecting hazardous anomalies in driving scenes using Cityscapes-based Web and Lost & Found datasets.
- The paper adapts classification uncertainty methods to segmentation and proposes approximations for the higher complexity and computational cost of segmentation networks.
- None of the evaluated methods reaches the accuracy required for safety-critical applications, leaving anomaly detection and uncertainty estimation open problems.
2. Related Work
Prior work covers semantic segmentation benchmarks and uncertainty or out-of-distribution estimation, but existing evaluations provide limited evidence about uncertainty on diverse pixel-level anomalies in driving scenes.
- Semantic segmentation benchmarks commonly use encoder-decoder networks on datasets with dense labels for limited object classes and urban scenes.
- Existing robustness benchmarks typically rank methods by accuracy without accounting for prediction uncertainty.
- WildDash reports uncertainty for a limited set of full-image outliers, whereas Fishyscapes evaluates diverse anomalous objects.
- Earlier semantic-segmentation OoD work often assumes large OoD datasets or evaluates only a few methods on normal in-distribution data.
- Softmax scores provide a baseline for uncertainty estimation, while later methods modify inputs or calibrate scores to improve OoD detection.
- Bayesian approaches represent outputs or weights probabilistically and define uncertainty through dispersion, including epistemic uncertainty over model parameters.
- Feature-introspection methods detect discrepancies between training and OoD feature distributions using nearest-neighbor statistics or Gaussian approximations.
3. Benchmark Design
Fishyscapes evaluates uncertainty estimates through anomalous-pixel detection while addressing overfitting with hidden, dynamically changing data and complementary synthetic and real-world datasets.
- Uncertainty estimators are compared as binary anomaly detectors because ground-truth uncertainty values cannot be directly produced.
- A public fixed OoD set risks rewarding methods that overfit to known anomalies rather than general uncertainty estimation.
- The benchmark hides larger test sets, changes its synthetic dataset over time, compares synthetic results with real-world data, and records whether OoD training data was used.
- 3.1. Does the method work in an open world?: The open-world design uses diverse objects sampled dynamically, since fixed datasets cannot remain truly unexpected as methods adapt to their contents.
- 3.1. Does the method work in an open world?: Fishyscapes Web blends web-crawled objects into Cityscapes scenes, while FS Static uses Cityscapes validation images and excludes void pixels from evaluation.
- 3.1. Does the method work in an open world?: The blending process adapts color and adds transparency, motion blur, depth blur, noise, and glow effects to integrate objects into driving scenes.
- 3.1. Does the method work in an open world?: ID objects are also blended to reduce overfitting to artifacts rather than object semantics and appearance.
- 3.2. Does the method work on real images?: Synthetic generation is necessary for repeated dynamic evaluation, while FS Lost & Found supplies real-world conditions for deployment-relevant testing.
4. Evaluated Methods
The paper adapts established uncertainty and anomaly-detection methods to DeepLabv3+ semantic segmentation and introduces learned embedding density as a scalable alternative.
- Existing baselines: Fishyscapes evaluates softmax, OoD-training, Bayesian, Dirichlet, and embedding-based uncertainty methods on DeepLabv3+ segmentation.The methods were originally developed mainly for image classification or uncertainty estimation and require adaptation to pixel-wise predictions.
- Existing baselines: Bayesian DeepLab estimates epistemic uncertainty with encoder dropout and T Monte Carlo samples, while aleatoric uncertainty comes from categorical-distribution spread.Total uncertainty is represented by predictive entropy, and epistemic uncertainty is measured using mutual information.
- Existing baselines: Dirichlet DeepLab models predicted logits as Dirichlet concentration parameters, training sharp priors for in-distribution pixels and flat priors for OoD pixels.The evaluation uses pixel-wise Dirichlet distributions and measures their differential entropy.
- Embedding-based methods: kNN embedding methods estimate uncertainty from neighbor statistics, but segmentation creates dense embedding maps, lower resolution, and ambiguous class associations.The paper evaluates both class-dependent and class-independent density estimates, then upsamples feature-map densities to the input resolution.
- Learned Embedding Density: kNN density estimation is coarse and costly, motivating a normalizing-flow method that learns embedding density and avoids storing the entire training set.The flow is trained by embedding-level negative log-likelihood, maps embeddings to Gaussian-prior latent vectors, and estimates each test embedding’s NLL.
- Learned Embedding Density: Layer-specific NLLs are normalized before aggregation, with minimum and weighted-average strategies used to combine scores across layers.Normalization accounts for differing embedding distributions, dispersions, and dimensions.
5. Discussion of Results
The benchmark exposes substantial differences across anomaly-detection methods, datasets, accuracy, supervision, runtime, and adaptation difficulty. Results show that no evaluated method yet meets safety-critical accuracy requirements, while practical trade-offs remain between detection quality, segmentation accuracy, and computational cost.
- Method performance: Softmax confidence is unreliable for anomaly detection, although training with OoD data improves its performance.The OoD-trained version is not much better than Bayesian DeepLab, which does not require OoD data.
- Dataset differences: Lost & Found is substantially more challenging than the other datasets, partly because it contains many images with very small anomalous objects.Qualitative examples also show false positives and missed small anomalies.
- Benchmark reporting: Table 1 reports benchmark results, with gray columns marking the primary metric and bold or italic formatting distinguishing overall and non-OoD-trained best performance.The table restricts FS Web evaluation to datasets containing post-submission web images.
- Segmentation accuracy: Methods such as Bayesian DeepLab and Outlier Head rank among the best across datasets but can reduce segmentation accuracy by up to 10%.Methods without retraining are therefore especially relevant when segmentation accuracy matters.
- Supervision: On every dataset, the best method used OoD data and was at least 38% better than any unsupervised method.The Outlier Head and Dissimilarity Ensemble remained robust to diverse anomalies on FS Web.
- Interpretation: Uncertainty estimation and anomaly segmentation are distinct: Bayesian DeepLab can compete with some supervised methods but not with methods designed specifically for anomaly segmentation.The benchmark evaluates both anomaly segmentation and uncertainty estimates through anomaly detection.
- Inference time: Methods requiring additional processing have up to two orders of magnitude higher inference time than single-pass methods, except single-layer embedding density.Runtime measurements depend on implementation details and GPU-memory constraints.
- Method adaptation: Some methods are difficult to adapt to semantic segmentation because special retraining losses can impair accuracy and become unstable or fail to converge.Complex network structures also complicate transferring class-based embedding methods.
6. Conclusion
Fishyscapes comparisons show that standard softmax confidence is inadequate for anomaly detection, while stronger methods often trade segmentation accuracy for improved detection. OoD supervision consistently outperforms unsupervised approaches, but current methods remain insufficient for safe autonomous deployment.
- Standard classifier softmax output is a bad indicator for anomaly detection.
- Most better-performing methods require special losses that reduce semantic segmentation accuracy.
- OoD-supervised anomaly segmentation methods consistently outperform unsupervised methods, including in open-world scenarios.
- Further research is required before semantic segmentation methods can be safely deployed in autonomous cars.
Appendix
The appendix supplies supplementary evaluations and documents the proposed datasets and evaluated methods.
- The appendix provides additional experimental evaluations.
- It gives further details on the proposed datasets.
- It gives further details on the evaluated methods.
A. Misclassification Detection
The misclassification experiments evaluate uncertainty estimates on diverse driving imagery using binary-classification metrics. Softmax scores are competitive, Bayesian predictive entropy helps, but no method is significantly superior overall.
- Misclassification detection mixes aleatoric, model, and class-prior uncertainty, making failure detection important for autonomous-agent deployment.
- The evaluation combines Foggy Driving, Foggy Zurich, WildDash, and Mapillary Vistas, ignoring pixels labeled void.
- Average precision is the primary metric, while FPR95 emphasizes false positives at high recall for safety-critical applications.
- The void classifier is unsuitable for misclassification detection, and learned embedding density is not evaluated.
- Softmax score is competitive, Bayesian DeepLab favors predictive entropy, and kNN density performs similarly to other methods.
- Overall, the experiments do not identify a single method that performs significantly better than the others.
B. Details on the Methods
The method details describe the implementation context and practical adaptations used to evaluate uncertainty methods with DeepLab-based segmentation models.
- The appendix provides implementation details to support reproducibility of the evaluated methods.
- DeepLabv3+ with an Xception-71 backbone is used, while unretrained methods use the original Cityscapes-trained model.
- ODIN adaptations omit temperature scaling and input preprocessing because they provided little benefit or were infeasible for the large DeepLab model.
- The Bayesian DeepLab setup inserts dropout layers into the middle flow of an ImageNet-pretrained Xception-65 backbone.
B.4. Dirichlet DeepLab
Dirichlet DeepLab training used smoothed, scaled targets and term reweighting to achieve classifier convergence. Replacing the first loss term with Dirichlet negative log-likelihood did not converge.
- Target labels were smoothed with ϵ = 0.01 and scaled by α0 = 100 to obtain target concentrations.
- Both the first and second loss terms were downweighted by 0.1 to ensure classifier convergence.
- All but the last layer were initialized according to the implementation procedure to support convergence.
- Replacing the first term with the Dirichlet negative log-likelihood failed to produce convergent training.
B.5. kNN Embedding
The kNN embedding approach selects a single network layer and neighbor count because exhaustive multi-layer querying and large-scale fitting are computationally costly. Reported validation performance averages account for nondeterministic fitting.
- A single embedding layer was used because each layer already contains more than 10000 vectors, making multi-layer kNN queries infeasible.
- Because hnswlib fitting was nondeterministic, validation performance was averaged over 3 experiments after fitting on 1000 sampled Cityscapes images.
- Layer selection was evaluated on the FS Lost & Found validation set for kNN density and on FS Static for class-based relative density.
- The selected layers were decoder conv0 0 for kNN density and xception 71/exit flow/block2 for class-based relative kNN density.
B.6. Learned Embedding Density
Learned embedding density uses a Real-NVP normalizing flow trained on DeepLab embeddings, with layer and preprocessing choices selected by validation. Training stability depends on the chosen feature representation.
- The normalizing flow stacks 32 Real-NVP steps, each containing affine coupling, batch normalization, and a fixed random permutation.
- For stride 16, training uses 22M Cityscapes embeddings, reserves 2000 for validation and testing, and runs 200k iterations at learning rate 10^-4 with Adam.
- The learned-density layer was selected using FS Lost & Found validation AP, with decoder conv1 0 chosen after the aspp features layer failed to converge.
- Input preprocessing applies noise with magnitude ϵ = 0.25 in all experiments.