Source-linked AI summary
Benchmarking the Robustness of Semantic Segmentation Models
Christoph Kamann, Carsten Rother
TL;DR
Semantic-segmentation robustness to real-world image corruptions is important for safety-critical applications, but prior rigorous evaluations largely addressed other vision tasks. This paper benchmarks DeepLabv3+ across datasets, architectures, ablations, and more realistic corruptions using almost 400,000 generated images. It finds that robustness generally rises with model performance and is strongly affected by architectural choices, while Dense Prediction Cell can hurt corrupted-image performance.
Problem
The paper addresses limited rigorous evidence on semantic-segmentation robustness to broad, physically realistic image corruptions, an issue important in safety-critical applications.
Method
The authors evaluate DeepLabv3+ with multiple backbones and architectural ablations across Cityscapes, PASCAL VOC 2012, and ADE20K using almost 400,000 generated images and more realistic noise and blur models.
Results
Robustness generally increases with model performance, while architectural properties significantly affect robustness; Dense Prediction Cell significantly hampers performance on corrupted images.
Takeaways & Limitations
Atrous convolutions are generally recommended for robustness, while Dense Prediction Cell vulnerability should be considered in low-light and safety-critical applications.
Takeaways & Limitations
The study's observation that robustness often increases with performance has exceptions across particular backbones and datasets, and some future-work questions remain beyond scope.
Abstract
from arXiv · showhide
When designing a semantic segmentation module for a practical application, such as autonomous driving, it is crucial to understand the robustness of the module with respect to a wide range of image corruptions. While there are recent robustness studies for full-image classification, we are the first to present an exhaustive study for semantic segmentation, based on the state-of-the-art model DeepLabv3+. To increase the realism of our study, we utilize almost 400,000 images generated from Cityscapes, PASCAL VOC 2012, and ADE20K. Based on the benchmark study, we gain several new insights. Firstly, contrary to full-image classification, model robustness increases with model performance, in most cases. Secondly, some architecture properties affect robustness significantly, such as a Dense Prediction Cell, which was designed to maximize performance on clean data only.
1. Introduction
This paper presents a large-scale evaluation of semantic-segmentation robustness to realistic image corruptions, motivated by safety-critical applications and limited prior evidence beyond classification and detection. Using multiple datasets, architectures, and ablations, it finds that robustness usually increases with performance and depends substantially on architectural properties.
- Motivation: Image corruptions can substantially reduce DCNN performance, while environmental, optical, and camera factors make robustness essential for safety-critical segmentation applications.Relevant factors include illumination, weather, temperature, camera motion, and lens aberrations.
- Study scope: The study evaluates semantic-segmentation robustness by training on clean data and validating on corrupted data.The evaluation targets a broad range of real-world image corruptions.
- Study design: The authors compare architectures and perform detailed DeepLabv3+ ablations because architectural-property knowledge can inform practical systems facing known corruption types.The benchmark uses DeepLabv3+ with multiple backbones and many architectural variants.
- Findings: Atrous convolutions and long-range links generally aid robustness, whereas Dense Prediction Cell can significantly hamper corrupted-image performance despite being designed for clean-data performance.The Dense Prediction Cell comparison is illustrated in the Cityscapes ablation figure.
- Study design: The benchmark covers Cityscapes, PASCAL VOC 2012, and ADE20K, using almost 400,000 generated images and 19 corruptions across blur, noise, weather, digital, and geometric categories.Additional transformations include intensity-dependent camera noise, PSF blur, and geometric distortions.
- Findings: In most cases, DeepLabv3+ robustness increases with model performance, contrary to findings for full-image classification.The study also reports that some architectural properties significantly affect robustness.
2. Related Work
Prior robustness work largely studied full-image classification and related tasks under common corruptions, while this paper focuses on semantic segmentation and physically realistic corruptions. Earlier findings linked robustness to architecture and image-processing properties, motivating broader segmentation evaluations and controlled architectural comparisons.
- Earlier studies evaluated corruption robustness mainly for full-image classification, including color, noise, blur, rotation, and weather changes.
- ImageNet-C found higher absolute performance from AlexNet to ResNet but little change in robustness, while Multigrid and DenseNet were less noise-prone than ResNet.
- Prior work reported that robustness increases when CNNs rely more on global object shape than local texture.
- Adversarial-robustness studies included semantic segmentation, but this work instead targets physically realistic corruptions and architectural properties.
- The study combines architecture comparisons with one-property-at-a-time ablations to support more accurate evaluation of architectural effects.
3. Image Corruption Models
The benchmark evaluates semantic segmentation under diverse corruption types, extending ImageNet-C transformations with camera- and optics-inspired distortions. It also introduces an intensity-dependent noise model and spatially varying PSF blur to better reflect real-world imaging.
- The study evaluates robustness across 19 corruptions, adding intensity-dependent camera noise, PSF blur, and geometric distortions to existing transformations.
- The benchmark uses ImageNet-C corruptions spanning blur, noise, weather, and digital categories, each with five severity levels.Corruptions include motion, defocus, frosted-glass, and Gaussian blur; Gaussian, impulse, shot, and speckle noise; weather effects; and digital changes.
- Additional Image Corruptions: The proposed noise model combines chrominance and luminance noise with intensity-dependent behavior, making low-intensity pixels noisier than high-intensity pixels.
- Additional Image Corruptions: PSF blur models spatially varying optical aberrations using three generated functions whose blur increases with distance from the image center.
- Additional Image Corruptions: Geometric distortion applies several radially symmetric fourth-grade barrel distortions to both RGB images and their ground truth.
4. Models
The model study uses DeepLabv3+ as a reference architecture, varying backbones and key segmentation components through comparisons and ablations. Its design includes backbone processing, multi-scale feature extraction, long-range feature fusion, and decoding into semantic labels.
- 4.1. DeepLabv3+: DeepLabv3+ is selected because it supports multiple backbones and contains architectural properties suitable for ablation studies.Backbones include modified aligned Xception, ResNets, and MobileNet-V2 variants.
- 4.1. DeepLabv3+: The benchmark also includes FCN8s, VGG-16, ICNet, DilatedNet, ResNet-38, PSPNet, and GSCNN.
- 4.1. DeepLabv3+: DeepLabv3+ processes inputs through a backbone, a multi-scale module using ASPP or DPC, a long-range link, and a decoder producing semantic-label estimates.
- 4.1. DeepLabv3+: Atrous convolutions enlarge the kernel field of view by spacing kernel parameters, and DeepLabv3+ uses them in its backbone.
- 4.1. DeepLabv3+: ASPP extracts features at different scales with three atrous convolutions using rates 6, 12, and 18.
- 4.1. DeepLabv3+: DPC is an alternative to ASPP designed through neural architecture search to maximize clean-image performance, while the study tests whether this objective causes overfitting.
- 4.2. Ablation Study: The ablations remove atrous convolutions, the long-range link, or ASPP, with one variant replacing ASPP by DPC.
5. Experiments
Experiments show that segmentation robustness generally tracks clean-data performance, while corruption sensitivity varies substantially across corruption types and architectural choices. Across datasets and ablations, Xception-71 is strongest overall, atrous convolutions often improve robustness, and Dense Prediction Cell can sharply reduce it.
- 5.2. Benchmarking Network Backbones: Noise, snow, frost, and JPEG compression substantially reduce performance, whereas most models handle realistic PSF blur with only about a 2% mIoU decrease.
- 5.2. Benchmarking Network Backbones: DeepLabv3+ models handle geometric distortion significantly better than non-DeepLabv3+ segmentation models.
- 5.2. Benchmarking Network Backbones: Across most cases, robustness increases with clean-data performance, with rCD generally decreasing as mIoU increases for DeepLabv3+ backbones.
- 5.2. Benchmarking Network Backbones: Xception-71 is the most robust DeepLabv3+ backbone across PASCAL VOC 2012, Cityscapes, and ADE20K.
- 5.3. Ablation Study on Cityscapes: Atrous convolutions often improve robustness across blur, noise, and geometric distortion, including a 3.8% mIoU decrease for ResNet-101 without them under defocus blur.
- 5.3. Ablation Study on Cityscapes: Replacing ASPP with Dense Prediction Cell yields the highest clean-data mIoU for XC-71 but makes models vulnerable to most corruption categories.
6. Conclusion
The study evaluates semantic segmentation robustness to real-world image corruptions and derives design rules for improving robustness in practical systems.
- Atrous convolutions are generally recommended because they increase robustness against many corruptions.
- Dense Prediction Cell vulnerability to many corruptions should be considered, especially in low-light and safety-critical applications.
- ASPP is important for decent model performance, especially with digitally and geometrically distorted inputs.
- Global average pooling should always be used on PASCAL VOC 2012.
Supplemental Material
The supplement details the corruption models, their visualizations, severity calibration, and additional experimental information used in the robustness evaluation.
- The supplementary material illustrates ImageNet-C corruptions across blur, noise, digital, and weather categories at severity level three or higher.The examples include motion, defocus, frosted glass, Gaussian, impulse, shot, speckle, brightness, contrast, saturate, JPEG, snow, spatter, fog, frost, and related corruptions.
- The study augments existing transformations with an intensity-dependent camera noise model, PSF blur, and geometric distortions.Examples of the proposed corruptions are shown, and 19 corruptions are employed overall across blur, noise, weather, digital, and geometric categories.
- Proposed image corruptions: The noise model adds independent chrominance and shared luminance components to original pixel intensities in linear color space.Both components use intensity-level-dependent behavior, with low-intensity pixels noisier than high-intensity pixels; the noise variables are normally distributed and weighted by severity.
- Proposed image corruptions: PSF kernels model spatially varying optical blur whose intensity distribution becomes more spatially distributed at higher severity levels.Kernel shape also varies with the angle of incidence, corresponding to distance from the image center.
- Proposed image corruptions: Geometric distortion reflects time-varying and environmentally affected optical calibration, while image warping can introduce re-sampling artifacts.
- Signal-to-noise ratios are provided to make severity levels of image-noise corruptions mutually comparable across the three validation sets.
B. Experiments
The experiments combine benchmarks of multiple segmentation architectures with controlled DeepLabv3+ ablations to separate architecture-level robustness patterns from property-specific effects.
- The supplement evaluates removal of ASPP, atrous convolutions, and long-range links, plus replacement of ASPP with Dense Prediction Cell or global average pooling.Each ablated variant was retrained on the corresponding clean training data for Cityscapes, PASCAL VOC 2012, or ADE20K.
- DeepLabv3+ serves as the reference architecture, alongside FCN8s-VGG16, ICNet, DilatedNet, ResNet-38, PSPNet, and GSCNN.Most comparison models used publicly available checkpoints, while FCN8s-VGG16 was trained by the authors.
B.2. Experimental Details
The detailed setup covers hardware and training protocols, a 102-model retraining effort, robustness metrics, and proposed explanations for architectural effects.
- Experimental details: Models were trained using crop size 513, batch size 16, fine-tuned batch normalization, and polynomial learning rates initialized at 0.007 or 0.01.
- Experimental details: The benchmark retrained 102 models across ADE20K, Cityscapes, and PASCAL VOC 2012 using six architectural ablations and multiple backbones.ADE20K and Cityscapes each contributed 36 retrained models, while PASCAL VOC contributed 30.
- Robustness metrics: Corruption Degradation measures absolute performance loss, whereas relative Corruption Degradation additionally accounts for clean-data performance.
- Architectural properties: Dense Prediction Cell may learn fewer multi-scale representations than ASPP because its first processing layer uses a small atrous-convolution rate.This is presented as a hypothesis explaining DPC's robustness behavior.
- Architectural properties: Global average pooling improves clean-data performance on PASCAL VOC 2012 but not on Cityscapes or ADE20K, where averaging may lose important features.
- Architectural properties: Removing the long-range link can improve robustness for shallow Xception-41 models because early representations contain more pronounced image noise.This behavior was not observed for deeper ResNet-101.
B.5. Qualitative Results
Qualitative examples show that blur causes missed pixels in small, distant classes, while noise can reduce reference-model mIoU below 15%. Architectural ablations display corruption-specific vulnerabilities.
- Blur causes models to misclassify pixels belonging to classes covering small image regions, especially when those regions are far away.
- The two noisy Cityscapes predictions from the Xception-71 reference model each achieve mIoU below 15%.
- Without AC and with DPC, ablated variants are especially vulnerable to blur.
- Without AC, without ASPP, and with DPC, ablated variants are especially vulnerable to noise.
B.6. Experimental Results on Cityscapes
Cityscapes experiments compare corruption degradation across segmentation architectures and DeepLabv3+ ablations. Atrous convolutions and long-range links generally aid robustness, while several modifications produce corruption-specific weaknesses.
- FCN8s-VGG16 and DilatedNet are vulnerable to blur in Cityscapes corruption evaluations.
- Removing ASPP produces especially high geometric-distortion rCD values of 146 % for Xception-41 and 46 % for MobileNet-V2.
- AC often aids robustness against blur, noise, and geometric distortion, especially for ResNets and Xception-71.
- The ablation without ASPP and with DPC generally follows the corruption-degradation tendency observed for the corresponding CD scores.
- Removing LRL yields intensity-noise rCD values of 124 % for ResNet-101 and 118 % for MobileNet-V2, plus 155 % for geometric distortion with MobileNet-V2.
- For Xception-71, the GAP ablation has low rCD with respect to geometric distortion.
B.7. Experimental Results on PASCAL VOC 2012
On PASCAL VOC 2012, corruption effects and architectural choices vary by backbone and dataset. Removing ASPP consistently harms clean-data performance, while DPC can reduce robustness despite strong clean performance.
- Noise and weather corruptions have less impact on PASCAL VOC 2012 performance than on Cityscapes.
- Each backbone performs significantly worse without ASPP, while GAP gives every backbone its best clean-data performance.
- Removing ASPP yields geometric-distortion rCD values ranging from 24 % for ResNet-101 to 62 % for Xception-65.
- AC shows no positive effect against blur on this dataset, although it aids robustness for ResNet-101 but not ResNet-50.
- DPC’s harmful corruption effect is especially present for Xception-71, possibly because neural architecture search was performed on Xception-71.
- Removing LRL produces geometric-distortion rCD values of 144 % for Xception-41 and 127 % for ResNet-50.
B.8. Experimental Results on ADE20K
ADE20K results show strong backbone and ablation dependence across corruptions. DPC can improve clean-data performance yet reduce robustness, whereas GAP generally produces the most robust models.
- Among reference models, Xception-71 performs best for every corruption type on ADE20K.
- Removing ASPP gives rCD above 100 % for Xception-65 and Xception-71 but below 100 % for the remaining backbones.
- Removing AC slightly decreases performance for most backbones against digital and weather corruptions.
- Applying DPC often decreases robustness, especially for Xception-71, while producing the best clean-data model with that backbone.
- Removing LRL particularly affects Xception-71 under image noise.
- Applying GAP generally produces the most robust models.
- Across datasets, removing ASPP reduces clean-data mIoU by 1.2 %–12.0 % and often produces the highest corruption CD.