PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, Wieland Brendel

arXiv:1907.07484v2cs.CVcs.LGstat.ML

TL;DR

Object detectors can lose substantial performance on corrupted images, while robustness to previously unseen distortions remains difficult to assess. The paper introduces three corruption benchmarks and finds that stylizing training images improves robustness across corruption types, severities, and datasets.

  • Problem

    Object detectors generalize poorly to previously unseen corruptions, and evaluating robustness across all possible corruption types is not feasible.

    Source: §1 · p. 2

  • Method

    The paper introduces PASCAL-C, COCO-C, and Cityscapes-C, each covering 15 corruption types at five severity levels, and evaluates stylized-training augmentation.

    Source: §1 · p. 2

  • Results

    Across corruption types, severities, and datasets, stylizing training images substantially improves robustness, while standard detectors suffer severe performance impairments on corrupted images.

    Source: §1 · p. 2, §1.1 · p. 3, §4 · p. 10

  • Takeaways & Limitations

    The benchmark supports tracking object-detection robustness and provides evidence that benchmark performance predicts performance on natural distortions.

    Source: §1.1 · p. 3, §4 · p. 10

Abstract

from arXiv · show

The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance). However, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are publicly available.

1 Introduction

The paper introduces corruption benchmarks for object detection and instance segmentation to assess robustness beyond training conditions. It reports severe performance losses on corrupted images and improved robustness from stylizing training data.

1 Introduction

  • CNNs can fail on perceptible corruptions absent from training, and training on other distortions does not ensure generalization to novel distortion types.The introduction illustrates this with Faster R-CNN failing to recognize objects under added snow despite their remaining visible to humans [Ren et al., 2015]; poor generalization to novel distortions is also reported [Geirhos et al., 2018].Source: §1 · p. 2

1.1 Contributions

  • The proposed PASCAL-C, COCO-C, and Cityscapes-C benchmarks each cover 15 distortions at five severity levels to track object-detection robustness.The paper also makes the benchmark, corruption code, and stylization code openly available.Source: §1 · p. 2, §1.1 · p. 3
  • Broad object-detection and instance-segmentation models suffer severe performance impairment on corrupted images.The benchmark evaluates diverse corruption types not used in training as an approximation for performance under natural distortions such as rain, snow, fog, and day-to-night transitions.Source: §1 · p. 2, §1.1 · p. 3
  • Stylizing training images improves robustness across evaluated corruptions, while corruption robustness scales with clean-data performance and requires no added labels or architectural changes.Improved benchmark performance also corresponds to greater robustness to real-world rain, snow, and fog.Source: §1.1 · p. 3

1.2 Related Work

  • Prior corruption-robustness work studies model vulnerability, distortion-specific preprocessing and fine-tuning, stylized training, and methods for improving benchmark performance.Studies report CNN vulnerability to blur and Gaussian noise [Dodge and Karam, 2016], poor cross-distortion transfer from fine-tuning [Geirhos et al., 2018], and increased robustness from stylized ImageNet training [Geirhos et al., 2019].Source: §1.2 · p. 3
  • Autonomous-driving robustness research includes weather modeling and datasets, adverse-condition evaluations, and rain- or snow-specific detection studies, but no general solution has emerged.Prior work includes testing detection under simulated windshield raindrops [Bernuth et al., 2018], evaluating real-world corruptions [Pei et al., 2017], and reporting performance drops under corrupted data [Bernuth et al., 2019].Source: §1.2 · pp. 3–4

2 Methods

The methods establish a corruption-robustness benchmark for object detection, define stylized-data augmentation experiments, and evaluate performance under natural weather and lighting shifts.

2.1 Robust Detection Benchmark

  • The benchmark tests object detectors on 15 corruption types at five severity levels, reserving four separate corruptions for validation and the 15 benchmark corruptions for final testing.Training should not use the benchmark corruptions, which are intended to measure robustness to previously unseen corruptions; the corruption set adapts ImageNet-C [Hendrycks and Dietterich, 2019] to arbitrary image dimensions.Source: §2.1 · p. 4
  • The benchmark comprises PASCAL-C, COCO-C, and Cityscapes-C, using the VOC2007 test, COCO 2017 validation, and Cityscapes validation splits, respectively.It reports dataset-specific clean performance (Pclean), mean performance under corruption (mPC), and relative performance under corruption (rPC); mPC ranks models, while rPC captures relative degradation from clean performance.Source: §2.1 · pp. 4–5

2.2 Style transfer as data augmentation

  • For augmentation, the authors test replacing each training image with a stylized version or adding its stylized version, using AdaIN with α = 1 and randomly selected Painter by Numbers textures.The approach transfers style-based corruption-robustness augmentation from image classification to object detection [Geirhos et al., 2019].Source: §2.2 · pp. 5–6

2.3 Natural Distortions

  • Natural-distortion experiments test the same Cityscapes models on Foggy Cityscapes’ three synthetic fog severities and report results using AP.Fog severity is specified by attenuation coefficients β = 0.005m−1, 0.01m−1, and 0.02m−1; the fog was designed to look realistic [Sakaridis et al., 2018a].Source: §2.3 · p. 6
  • BDD100k experiments train Faster R-CNN on clear-weather or daytime data, then evaluate across clear, rainy, and snowy weather or the transition from day to night.BDD100k contains 100 thousand driving-scene videos recorded across varying conditions [Yu et al., 2018].Source: §2.3 · p. 6

3 Results

Image corruptions sharply reduce object-detection performance, while stylized-data training improves robustness on synthetic corruptions and natural distortions. The results also show that robustness relates to image encoding and is not simply proportional to pixel-level perturbation size.

3.1 Image corruptions reduce model performance

  • Faster R-CNN retains roughly 60% rPC on PASCAL VOC but only 33% rPC on Cityscapes; corruption degrades performance across tested models and instance segmentation tasks.The authors note that Cityscapes contains many small objects.Source: §3.1 · p. 6, Table 1 · p. 7

3.2 Robustness increases with backbone capacity

  • On COCO, stronger backbones improve relative performance under corruption, whereas sophisticated heads alone yield roughly 50% rPC; Hybrid Task Cascade reaches a leading 64.7% rPC.The authors interpret the results as indicating that better image encoding improves robustness, while better heads cannot recover information lost when encoding is impaired.Source: §3.2 · p. 7, Table 1 · p. 7

3.3 Training on stylized data improves robustness

  • Both stylized and combined training improve relative performance under corruption, while combined training yields the highest absolute corrupted-image performance across all three datasets.Stylized-only training performs much worse on clean data; combined training preserves high clean-data performance while improving performance under corruption.Source: §3.3 · p. 8, Figure 5 · p. 8, Table 2 · p. 8

3.4 Training directly on stylized data is better than using stylized data only during pre-training

  • Stylized ImageNet pre-training improves clean-data performance and corruption robustness, but less than training the detector directly on stylized data.The comparison uses Faster R-CNN models with stylized pre-training and standard clean training sets, versus the paper’s approach of final training on stylized data.Source: §3.4 · p. 8, Table 2 · p. 8, Table 3 · p. 9

3.5 Robustness to natural distortions is connected to synthetic corruption robustness

  • Combined stylized-data training improves generalization across tested natural distortions, with gains increasing in harder conditions.It improves performance on BDD100k rainy and snowy images and at night; for Foggy Cityscapes it raises relative performance from about 50% to over 70%, against an almost 50% reduction for standard models under the strongest fog.Source: §3.5 · p. 9, Table 4 · p. 9, Table 5 · p. 10

3.6 Performance degradation does not simply scale with perturbation size

  • On PASCAL-C, rPC has only a weak positive correlation with RMSE (r = 0.45) and a weak correlation with SSIM (r = 0.48), with SSIM better capturing performance effects.Global changes such as fog or brightness can have high RMSE while leaving local structure intact, whereas impulse noise can change few pixels yet sharply harm performance.Source: §3.6 · pp. 9–10, Figure 6 · p. 10

4 Discussion

Object detection and instance segmentation degrade severely under corruption, motivating the Robust Detection Benchmark and its PASCAL-C, COCO-C, and Cityscapes-C datasets. Stylized training copies improve corrupted-image performance with small clean-data losses, while the benchmark leaves substantial room for further robustness work.

4 Discussion

  • Stylized-copy augmentation raises corrupted-image performance by about 16% on PASCAL, 12% on COCO, and 41% on Cityscapes, with clean-data losses of 0–2%.The method adds a stylized copy of training data to reduce models’ focus on textural information; it needs no additional labeling or model tuning and can be applied to any image dataset.Source: §4 · p. 10
  • Object detection and instance segmentation models suffer severe performance impairments on corrupted images, motivating the Robust Detection Benchmark’s PASCAL-C, COCO-C, and Cityscapes-C datasets.The authors report that benchmark performance predicts performance on natural distortions and that robustness corresponds to performance on clean data.Source: §4 · p. 10
  • The benchmark shows room for improvement, and which combination of architectural changes, data augmentation, or loss-function modifications will best improve robustness remains undetermined.The authors propose the benchmark to track future progress toward robust object detection models.Source: §4 · p. 10
  • The authors encourage expanding the open-source benchmark with novel corruption types, arguing that robust models require testing against a wide variety of image corruptions.They welcome new corruption types and envision the benchmark tracking progress toward object detectors that can be reliably deployed in the wild.Source: §4 · p. 10

Appendix · A Implementation details: Model training

Models use dataset-specific image scales, schedules and hardware, with stylized training implemented by exchanging or adding dataset folders. Cityscapes training follows the procedure in as closely as possible.

A Implementation details: Model training

B Corrupting arbitrary images

The benchmark code was modified to support corruptions on images with varying dimensions and channels, including non-quadratic and grayscale images. The changes also adapt Frost to large inputs and remove a non-Python dependency for motion blur.

B Corrupting arbitrary images

  • Unlike the original ImageNet-C benchmark [Hendrycks and Dietterich, 2019], the modified code applies corruptions to non-quadratic images of varying sizes, supporting PASCAL VOC and COCO.Corruption functions had previously been defined separately for datasets with hard-coded dimensions and channel counts.Source: B Corrupting arbitrary images · p. 15
  • The code supports grayscale images and resizes Frost texture images to fit the largest input dimension when necessary.Frost corruption adds crops from supplied frost images to the input.Source: B Corrupting arbitrary images · p. 15
  • Motion blur was reimplemented in Python, removing the ImageMagick dependency used by motion_blur and snow.The external dependency could not be resolved through standard Python package managers.Source: B Corrupting arbitrary images · p. 15

C BDD100k

The BDD100k evaluation trains on clear-weather images and tests on balanced clear, rainy, and snowy validation subsets; other weather labels are excluded, and the 13 foggy validation images are not representative.

C BDD100k

  • Using BDD100k weather annotations (Yu et al. [2018]), training uses all clear-weather training images, and validation is split into clear, rainy, and snowy subsets.Source: C BDD100k · p. 15
  • Each validation subset contains 725 images, the rainy category's available count; equal sizes matter because AP scores decrease with longer evaluation sequences [Gupta et al., 2019].Source: C BDD100k · p. 15
  • Images labeled foggy, partly cloudy, overcast, or undefined are excluded; only 13 foggy validation images make fog results unrepresentative.Source: C BDD100k · p. 15

D Additional Results · D.1 Instance Segmentation Results

Mask R-CNN instance-segmentation results were similar to object-detection results, with slightly lower relative performance (1%). Training Mask R-CNN on stylized datasets produced trends that mirrored Faster R-CNN object-detection performance, without the expected segmentation penalty from altered object contours.

D.1 Instance Segmentation Results

  • Mask R-CNN and Cascade Mask R-CNN had instance-segmentation results similar to object detection, with slightly lower relative performance (1%).The passage reports these results in Table 6.Source: §D.1 · p. 16
  • Mask R-CNN trained on stylized datasets showed instance-segmentation trends similar to object-detection trends.The passage says Mask R-CNN’s stylized-training performance mirrored Faster R-CNN’s object-detection performance; it reports these findings in Table 7.Source: §D.1 · p. 16
  • Despite the possibility that stylization could alter object contours and harm segmentation, the authors did not observe that effect.Instead, instance-segmentation performance mirrored object-detection performance when Mask R-CNN was trained on stylized images.Source: §D.1 · p. 16

D.2 Deformable Convolutional Networks

Deformable convolutions improve object-detection performance on clean and corrupted images compared with standard backbones, with robustness gains similar to those from deeper architectures.

D.2 Deformable Convolutional Networks

  • Deformable-convolution networks improve relative performance by 6–7% over standard-backbone baselines on both clean and corrupted images.The effect appears similar to that of other backbone modifications, including deeper architectures.Source: §D.2 · p. 16
  • Deformable convolutions replace some standard convolutions with adaptive filters in the encoder’s last stages [Dai et al., 2017, Zhu et al., 2018].The study tested their effect on corruption robustness.Source: §D.2 · p. 16

Related papers

Loading 1907.07484v2…