Source-linked AI summary
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, Wieland Brendel
TL;DR
Object detection models need to remain reliable under image distortions and changing weather, but their robustness to such conditions is limited. The paper introduces three corruption benchmarks for object detection and shows that stylizing training images substantially improves robustness across corruptions, severities, and datasets.
Problem
Object detection models can suffer severe performance losses under perceptible corruptions and unpredictable weather, while robustness to unknown distortions remains difficult to assess.
Method
The paper introduces PASCAL-C, COCO-C, and Cityscapes-C, applying 15 corruptions at five severity levels to evaluate object-detection robustness.
Results
Stylizing training images strongly improves robustness across corruption types, severity levels, and datasets, while benchmark performance predicts robustness to natural distortions.
Takeaways & Limitations
The Robust Detection Benchmark provides a standardized way to track progress toward object detectors that better withstand corrupted and natural-distortion inputs.
Abstract
from arXiv · showhide
The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance). However, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are publicly available.
1 Introduction
Object detectors are vulnerable to perceptible image corruptions and unpredictable outdoor conditions, motivating a standardized robustness benchmark. The paper proposes three corrupted detection datasets and evaluates stylized training as a general robustness improvement.
- Motivation: Object detection models can fail on visibly corrupted images, including snow, even when objects remain recognizable to humans.Augmenting training with several distortions does not guarantee robustness to previously unknown corruption types.
- Motivation: Autonomous vehicles must handle varied conditions such as fog, frost, snow, sand storms, and falling leaves, but unusual-weather data are difficult to collect and cannot cover every real-world condition.The paper notes that existing models can target some common conditions, while unforeseen conditions remain unavoidable.
- Benchmark: The paper proposes evaluating models on diverse corruption types absent from training as an approximation for performance under natural distortions such as rain, snow, fog, and day-night transitions.Testing every possible corruption is impossible, so the benchmark uses a representative diverse range.
- Benchmark: PASCAL-C, COCO-C, and Cityscapes-C provide easy-to-use object-detection benchmarks with 15 corruptions, each tested at five severity levels.The benchmark follows standardized corrupted-dataset approaches introduced for image classification.
- Findings: Standard object detection and instance segmentation models suffer severe performance impairments on corrupted images.This establishes corruption robustness as a measurable weakness across a broad model range.
- Findings: Stylizing training data produces large robustness improvements across evaluated corruption types without additional labeling costs or architectural changes.The paper makes the benchmark, corruption tools, and stylization code openly available to support future progress.
2 Methods
The Robust Detection Benchmark evaluates object-detection robustness using three corrupted datasets, standardized corruption metrics, and natural-distortion tests. It also examines stylized training data as a robustness intervention.
- Robust Detection Benchmark: The benchmark introduces PASCAL-C, COCO-C, and Cityscapes-C, each containing 15 corruptions across five severity levels.The corruption set is adapted from ImageNet-C to support images of arbitrary dimensions and is intended for evaluation rather than training augmentation.
- Robust Detection Benchmark: Corruptions are organized into noise, blur, digital, and weather groups to assess robustness against previously unseen image distortions.The benchmark follows the corruption taxonomy introduced by Hendrycks and Dietterich.
- Metrics: Mean performance under corruption (mPC) averages dataset-specific performance across 15 corruption types and five severity levels.The dataset-specific metric is AP50 for PASCAL VOC and AP for COCO and Cityscapes.
- Metrics: Relative performance under corruption (rPC) measures performance degradation on corrupted data relative to clean data.The benchmark reports clean performance, mPC, and rPC; mPC ranks models while the other measures distinguish clean-data gains from corruption generalization.
- Baseline Models: The study evaluates common detection and instance-segmentation models, including Faster R-CNN, Mask R-CNN, Cascade R-CNN, RetinaNet, and Hybrid Task Cascade.Most models use a ResNet50 with Feature Pyramid Networks backbone, with an additional ResNet101 Faster R-CNN evaluation.
- Style Transfer as Data Augmentation: Style transfer is tested by replacing each training image with a stylized version or adding stylized versions to the original dataset.The stylization uses AdaIN with α = 1 and replaces original texture with randomly selected texture information from Painter by Numbers.
3 Results
Corruptions substantially reduce object-detection performance, while stylized training improves robustness across synthetic and natural distortions. Robustness also varies with model capacity and is not simply determined by pixel-level perturbation size.
- 3.1 Image corruptions reduce model performance: 33% rPC: Faster R-CNN on Cityscapes retains substantially less corrupted-image performance than roughly 60% rPC on PASCAL VOC.The degradation also appears across tested detection models and instance-segmentation tasks.
- 3.2 Robustness increases with backbone capacity: 64.7% rPC: Hybrid Task Cascade leads under corruption, combining a stronger backbone, improved head architecture, and additional training data.It also outperforms the strongest baseline by 9% AP on clean data.
- 3.3 Training on stylized data improves robustness: Stylized training reduces corruption sensitivity, while combined standard-and-stylized training achieves the highest absolute corrupted-data performance across all three datasets.Stylized and combined training both improve relative performance under corruption; combined training preserves high clean-data performance.
- 3.5 Robustness to natural distortions is connected to synthetic corruption robustness: Combined stylized training improves generalization to rainy, snowy, nighttime, and foggy images, with relative performance increasing from about 50% to over 70% on Foggy Cityscapes.The authors report increasing gains in harder natural-distortion conditions.
- 3.6 Performance degradation does not simply scale with perturbation size: rPC has only weak positive correlations with RMSE (r = 0.45) and SSIM (r = 0.48), so pixel perturbation size does not simply predict performance degradation.Global changes such as fog can yield high RMSE while preserving local structure, whereas impulse noise can affect few pixels but strongly reduce performance.
4 Discussion
The paper proposes a Robust Detection Benchmark for tracking corruption robustness and shows that stylized-data augmentation improves corrupted-image performance with small clean-data losses.
- 4 Discussion: Object detection and instance segmentation models suffer severe performance impairments on corrupted images.The benchmark is intended to track progress on this robustness problem.
- 4 Discussion: The benchmark comprises PASCAL-C, COCO-C, and Cityscapes-C and provides evidence that synthetic-benchmark performance predicts robustness to natural distortions.The benchmark is described as easy to use and open source.
- 4 Discussion: Stylized copies of training data improve corrupted-image performance by about 16% on PASCAL, 12% on COCO, and 41% on Cityscapes, with clean-data losses of 0–2%.The approach requires no additional labelling or architectural changes.
- 4 Discussion: The authors encourage expanding the open-source benchmark with novel corruption types to support robust-model evaluation.They argue that testing against a wide variety of corruptions is necessary.
A Implementation details: Model training
The experiments use fixed multi-GPU training settings and adapt the corruption-generation code to datasets with differing image dimensions and channels.
- A Implementation details: Model training: COCO training starts at learning rate 0.01, while PASCAL VOC training starts at 0.00125, with tenfold decay during training.The cited schedules place decay after eight and eleven epochs on COCO and after nine epochs on PASCAL VOC.
- A Implementation details: Model training: The corruption code is modified so corruption functions support datasets with different image dimensions and channel configurations.This removes hard-coded assumptions from the original ImageNet-C implementation.
C BDD100k
BDD100k is partitioned by weather annotations for natural-distortion evaluation, while the appendix notes that natural-fog results are not representative because few foggy validation images exist.
- C BDD100k: BDD100k uses clear-weather training images and validation subsets annotated as clear, rainy, or snowy.Images with foggy, partly cloudy, overcast, or undefined labels are discarded to simplify the separation.
- C BDD100k: The BDD100k evaluation uses 725-image validation subsets for each selected weather condition.The passage states that these subsets are created from the validation set.
- C BDD100k: Natural-fog results could not be combined representatively with Foggy Cityscapes because the validation set contains only 13 foggy images.This is identified as a limitation of the available validation data.
D.1 Instance Segmentation Results
Instance segmentation results closely mirror object detection results: stylized training improves robustness without the expected contour-related degradation.
- Mask R-CNN and Cascade Mask R-CNN show results very similar to those for object detection.Instance segmentation has slightly lower relative performance, by 1%.
- Stylizing training images produces similar robustness trends for instance segmentation and object detection.Mask R-CNN trained on stylized datasets follows the trends observed for Faster R-CNN.
- Stylization does not cause the expected poor segmentation performance from changed object contours.The observed segmentation performance instead mirrors object detection performance when training on stylized images.
D.2 Deformable Convolutional Networks
Deformable convolutions improve corruption robustness alongside clean-image performance, with gains comparable to those from deeper backbone architectures.
- 6-7% relative-performance improvement over standard-backbone baselines occurs on corrupted images with deformable convolutions.The networks also perform better on clean images.
- Deformable convolutions replace standard convolutions with adaptive filters in the encoder’s last stages.The modification affects backbone stages c3-c5 in the reported models.
- The robustness effect of deformable convolutions appears similar to that of other backbone modifications such as deeper architectures.
Image rights & attribution
This material includes attribution for Figure 1 and captions for benchmark-result figures covering datasets and backbone configurations.
- Figure 1 is attributed to Home Box Office, Inc. (HBO).
- Tables 8 and 9 describe object detection and instance segmentation models using deformable convolutions.
- Figures 7, 8, and 9 present results for each corruption type on PASCAL-C, COCO-C, and Cityscapes-C.
- Figure 10 presents corruption-type results for Faster R-CNN with ResNet-50, ResNet-101, and ResNext-101_64x4d backbones.