PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Examples Are a Natural Consequence of Test Error in Noise

Nic Ford, Justin Gilmer, Nicolas Carlini, Dogus Cubuk

arXiv:1901.10513v1cs.LGcs.CVstat.ML

TL;DR

The paper asks why image classifiers that err on corrupted images also exhibit nearby adversarial examples. Through geometric analysis and experiments, it connects the two robustness notions and reports that robustness-improving training procedures generally improve both, while corruption evaluation exposes important defense failures.

  • Problem

    Image classifiers lack human-level robustness to common corruptions while also being vulnerable to small adversarial perturbations, and the relationship between these error types requires explanation.

  • Method

    The paper analyzes adversarial and corruption robustness through the classifier’s error set, using concentration-of-measure arguments and experiments on Gaussian noise, trained models, and corruption benchmarks.

  • Results

    Adversarial examples are a natural consequence of nonzero error in corrupted image distributions, and training procedures that improve adversarial robustness also improve many types of corruption robustness.

  • Takeaways & Limitations

    Corruption robustness should be evaluated alongside adversarial robustness because varied corruptions can reveal failures that small-perturbation evaluations miss.

  • Takeaways & Limitations

    Gaussian data augmentation does not reduce Gaussian-noise test error to zero, and some defense strategies improve reported lp robustness while failing to improve Gaussian-noise performance.

Abstract

from arXiv · show

Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to small modifications of a correctly handled input. Less surprisingly, image classifiers also lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise. In this paper we provide both empirical and theoretical evidence that these are two manifestations of the same underlying phenomenon, establishing close connections between the adversarial robustness and corruption robustness research programs. This suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions. Based on our results we recommend that future adversarial defenses consider evaluating the robustness of their methods to distributional shift with benchmarks such as Imagenet-C.

1. Introduction

The paper links adversarial examples with ordinary corruption errors, arguing that both reflect limitations in model robustness. It analyzes this connection theoretically and empirically and recommends evaluating corruption robustness alongside adversarial robustness.

  • Motivation: Modern classifiers remain vulnerable to diverse corruptions and small worst-case perturbations despite strong performance on standard image classification.Reported weaknesses include Gaussian noise, blur, pixelation, brightness changes, translations, and inserted foreign objects.
  • Two robustness programs: Adversarial robustness concerns small worst-case perturbations, whereas corruption robustness concerns distributional shift.
  • Central connection: The paper argues that adversarial examples follow naturally from nonzero test error on certain corrupted image distributions.
  • Analysis: Its analysis relates observed Gaussian-noise error rates to adversarial distances without requiring unusual decision-boundary properties.
  • Robustness requirements: Improving an alternate adversarial-robustness notion requires reducing error under large additive noise to essentially zero.
  • Empirical evaluation: Experiments report that adversarially trained and Gaussian-noise-trained models improve both relevant robustness quantities, while corruption tests can expose defenses with vanishing gradients.
  • Recommendation: The paper recommends evaluating corruption robustness in addition to adversarial robustness for future defenses.

2. Related Work

Related work treats adversarial robustness and corruption robustness as distinct research directions, while theoretical results connect nonzero test error to adversarial perturbations. The paper positions its findings as compatible with accuracy trade-offs reported for adversarial training.

  • Adversarial examples: Adversarial-example research develops methods to find small classifier-fooling perturbations and defenses against them.
  • Theory: Concentration-of-measure analyses derive adversarial-robustness bounds from test error for certain distributions.
  • Accuracy trade-offs: The paper’s connection between adversarial robustness and corrupted-distribution accuracy does not contradict reports that adversarial training reduces clean-distribution accuracy.
  • Reported finding: The paper reports that improving adversarial robustness also improves corruption robustness.

3. Adversarial and Corruption Robustness

The paper formalizes both robustness notions through a classifier’s error set and distinguishes the natural image distribution from corrupted image distributions. It defines each robustness quantity as a probability over the relevant distribution or neighborhood.

  • Error set: The classifier’s error set E contains input points on which it makes incorrect predictions, and both robustness notions are functions of E.
  • Assumption: The analysis assumes corrupted points retain the clean point’s correct label.
  • Distributions: p denotes the natural image distribution, while q denotes a corrupted distribution formed by applying random corruptions to samples from p.
  • Definitions: Corruption robustness under q is the probability that a q-sample is outside E.It is written as P_x∼q[x /∈ E].
  • Definitions: Adversarial robustness is the probability that a natural sample lies farther than ϵ from the nearest error under metric d.The paper denotes the nearest-error distance by d(x,E).
  • Models and datasets: Experiments investigate naturally trained, adversarially trained, and Gaussian-data-augmented models on CIFAR-10 and ImageNet.

4. Errors in Gaussian Noise Suggest Adversarial Examples

A geometric analysis shows why Gaussian-noise errors can imply nearby adversarial examples in high-dimensional image spaces. Experiments on neural networks broadly match the linear-model relationship and show that training can increase both corruption and adversarial distances.

  • The Linear Case: As dimension increases, the ratio between nearest-error distance and typical Gaussian-sample distance shrinks.Typical Gaussian samples lie at distance σ√n, while the nearest error can remain much closer.
  • The Linear Case: For a linear model, Gaussian-noise error corresponds geometrically to the fraction of a sphere cut off by a decision-boundary plane.The nearest error is the plane’s distance from the clean point.
  • The Linear Case: A 1% Gaussian-noise error rate at σ = 0.1 implies a linear-model error at distance about 0.23.
  • High-dimensional effect: For CIFAR-10’s n = 3072 dimensions, typical samples from that noise lie about 5.54 away, making the implied error more than 20 times closer.
  • Comparing Neural Networks to the Linear Case: Neural-network experiments found relationships closely resembling Equation 1, so adversarial examples need not reflect surprisingly shaped decision boundaries.
  • Experimental method: For each test point, the study compared σ producing 1% Gaussian-noise error with a PGD-based estimate of nearest-error distance.
  • Comparing Neural Networks to the Linear Case: Figure 2 compares l2 decision-boundary distance with σ for 1% Gaussian-noise error using medians, percentile error bars, and PGD distances capped at 1.
  • Training effects: Adversarial training and Gaussian augmentation increased both σ(x0, 0.01) and d(x0,E) on average.The adversarially trained model deviated most from the linear case toward greater decision-boundary distances.

5. Concentration of Measure for Noisy Images

The paper uses Gaussian isoperimetry to connect error rates under noisy images with distances from typical noisy samples to classification errors. Empirically, observed neural-network behavior lies close to the resulting bound, linking nonzero noise error with small adversarial perturbations.

  • Gaussian Isoperimetric Inequality: Existing concentration-of-measure results bound adversarial robustness using model error rates, but typically analyze simple toy distributions.The paper compares neural networks with a corresponding Gaussian bound for real image distributions.
  • Visualizing the Error Set: Figure 3 compares clean images, Gaussian-noise errors, and targeted PGD errors in two-dimensional image-space slices, with circles marking typical Gaussian-perturbation size.The slices visualize how random-noise and adversarial errors relate geometrically.
  • Gaussian Isoperimetric Inequality: For Gaussian perturbations, small adversarial perturbations exist because nonzero test error places most noisy images near the error set.This conclusion is independent of assumptions about the model and follows from Gaussian geometry.
  • Gaussian Isoperimetric Inequality: The Gaussian isoperimetric inequality gives q(E) ≤ −σΦ^-1(µ), with equality when the error set is a half space.Here µ is the probability that a Gaussian perturbation falls in the error set.
  • Gaussian Isoperimetric Inequality: When Gaussian error is at least µ, the median distance from a noisy image to the nearest error is at most −σΦ^-1(µ).This is the same distance expression obtained for a half-space error region.
  • Comparing Neural Networks to the Isoperimetric Bound: Across five models, estimated median distances and Gaussian error probabilities were close to the optimal relationship predicted by the inequality.For CIFAR-10, adversarial training improved small-perturbation robustness primarily by improving Gaussian-noise error rates and increased the σ yielding 1% error.

6. Evaluating Corruption Robustness

The paper evaluates whether adversarial and corruption robustness improve together across Gaussian noise, Imagenet-C corruptions, and small l_p perturbations. The results support a close but corruption-dependent relationship, while also showing important evaluation and generalization limitations.

  • Relationship Between Robustness Measures: Adversarial and corruption robustness are linked, so methods designed to improve one should also be evaluated on the other.The authors investigate this relationship across models and corruption settings.
  • Corruption Benchmark: Gaussian data augmentation and adversarial training both improved overall corruption-benchmark performance, with similar aggregate results.Adversarial training helped more with blurring, while Gaussian augmentation helped more with noise corruptions.
  • Corruption Benchmark: Both methods performed much worse than the clean model on fog and contrast corruptions.The relationship between adversarial robustness and non-Gaussian corruptions was therefore more complicated.
  • Evaluation Sensitivity: Imagenet-C accuracy fell from 57% in memory to 10% on compressed images for Gaussian-5 evaluation of a Gaussian-augmented InceptionV3 model.The drop was attributed to subtle compression artifacts rather than adversarial modification.
  • Small-Perturbation Robustness: Gaussian data augmentation improved adversarial robustness as well, although adversarial training performed better on standard l_p robustness measures.The augmentation experiment was intended as evidence for the connection, not as a novel defense proposal.
  • Scope Boundary: The MNIST adversarially trained model increased small-perturbation robustness without improving noise generalization because it violated the paper’s linearity assumption.This is presented as a special case limiting the generality of the observed relationship.
  • Generalization in Noise: Gaussian augmentation at σ = .15 achieved 99.9% training accuracy but only 77.5% test accuracy in the same noise distribution.The result illustrates that obtaining zero test error under substantial Gaussian noise is nontrivial.

7. Conclusion

The paper argues that adversarial examples arise naturally from nonzero error on corrupted distributions, linking adversarial and corruption robustness. Its results support evaluating distributional shift alongside adversarial robustness because corruption tests expose failures and can identify ineffective defenses.

  • Nearby errors occur at distance scales expected from a linear model with the same corruption robustness.
  • Nonzero Gaussian-noise error logically implies small adversarial perturbations of noisy images through concentration of measure.
  • Training for adversarial robustness improves many corruption robustness types, while Gaussian-noise training moderately improves adversarial robustness.
  • Adversarial defense evaluations should report generalization to distributional shift, such as performance on common corruption benchmarks.
  • Adversarial training reduced contrast-5 accuracy to 55.3%, compared with 85.7% for the vanilla model, despite improving small-perturbation robustness.
  • Corruption testing is easier than perfect adversarial-robustness measurement, which requires solving an NP-hard problem for every test point.
  • Previously published defenses that caused vanishing gradients showed no improvement in Gaussian noise, motivating skepticism toward defenses claiming only l_p robustness.
  • Errors on corrupted images imply that models are not secure against worst-case adversarial settings until they are perfectly robust to average-case corruptions.

A. Training Details

The experiments trained CIFAR-10 Wide-ResNet-28-10 models and ImageNet Inception v3 models with Gaussian-noise augmentation alongside standard training procedures.

  • CIFAR-10 models used Wide-ResNet-28-10 with flips, horizontal shifts, crops, and independently sampled Gaussian noise.
  • CIFAR-10 training ran for 200 epochs with weight decay 5e-4, learning rate 0.1, and batch size 128.
  • ImageNet models used Inception v3 with independently sampled Gaussian noise for each image in every minibatch.
  • ImageNet training ran for 130 epochs with learning rate 1.6, batch size 4096, and weight decay 8e-5.

B. Full Corruption Robustness Results

The corruption experiments compare adversarial training and Gaussian data augmentation across CIFAR-10-C and ImageNet-C, revealing broad gains alongside important corruption-specific failures and file-format effects.

  • On CIFAR-10-C, Gaussian data augmentation outperformed adversarial training overall, while adversarial training performed better on every blurring corruption.
  • The publicly released ImageNet-C JPEG files were significantly harder than applying the same corruptions in memory.
  • Both adversarial training and Gaussian data augmentation significantly degraded performance on severe fog and contrast corruptions.
  • JPEG compression adds artifacts whose subtle effects can degrade performance on several ImageNet-C corruptions.
  • ImageNet-C tables compare corruption performance on in-memory images with performance on the compressed dataset version.
  • CIFAR-10-C evaluations average model accuracies over five corruption severities using publicly released .npy files without additional saving noise.
  • For ImageNet-C fog and contrast, Gaussian augmentation degraded performance on in-memory corruptions, while compressed-data evaluation minimized that degradation relative to a clean model.
  • For CIFAR-10-C, both adversarial training and Gaussian augmentation significantly degraded fog and contrast performance.

D. Results on MNIST

On MNIST, adversarial robustness can fail to generalize across noise distributions, unlike the broader relationship observed between small adversarial perturbations and noise robustness. The section also develops Gaussian isoperimetric intuition underlying these robustness bounds.

  • D. Results on MNIST: Adversarially trained MNIST models can generalize worse to other noise distributions than naturally trained models.The paper reports this result for the investigated model and refers to Table 7.
  • D. Results on MNIST: MNIST models can become robust to small l∞ perturbations by learning to threshold the input.Prior work observed this behavior, and the Madry et al. model appeared to do so as well.
  • E. The Gaussian Isoperimetric Inequality: The appendix defines perimeter through the limiting ϵ-boundary measure of E and notes that equality holds for a ball.For small ϵ, the difference Eϵ − E resembles a band around the perimeter.
  • E. The Gaussian Isoperimetric Inequality: The Gaussian isoperimetric inequality compares the probability mass of a measurable set with that of its ϵ-extension.Among sets with equal Gaussian probability, half spaces have the smallest possible extensions.
  • E. The Gaussian Isoperimetric Inequality: For measurable E, the inequality yields ϵ*_q(E) ≤ −α after applying the inverse standard-normal CDF.Here α(E) is defined from the Gaussian probability of E, and ϵ*_q(E) is the median distance from a random Gaussian sample to E.

F. Visualizing the Optimal Curves

The paper visualizes the theoretical relationship between worst-case l2 perturbations and Gaussian-noise generalization using optimal curves, noisy samples, and distance-controlled image changes. The visualizations indicate that substantial ImageNet l2 robustness requires exceptionally low test error under noise.

  • F. Visualizing the Optimal Curves: The visualizations compare optimal curves, samples from x + N(0, σ), and images at varying l2 distances from clean inputs.The curves correspond to the Gaussian isoperimetric optimum and Equation 1.
  • F. Visualizing the Optimal Curves: 10^-15 test error is required at σ = .6 for worst-case perturbations to exceed l2 distance 5.0.The paper presents this as evidence that significant ImageNet l2 adversarial robustness requires nearly perfect performance under heavy noise.
  • F. Visualizing the Optimal Curves: The distance-controlled visualizations minimize SSIM subject to ||x − x_adv||_2 < ϵ.Gradient descent is used to find images with reduced visual similarity at specified l2 distances.
  • F. Visualizing the Optimal Curves: Noise-trained CIFAR models still produce many Gaussian-noise errors across most test images.The error-rate histogram shows that reduced noise performance is not attributable only to a few test points.

I. A Collection of Model Errors

The paper collects visual examples of ordinary and randomly induced model errors across CIFAR and ImageNet, comparing them with adversarial errors and varying noise distributions. These figures illustrate that error geometry and visual similarity can differ substantially across slices and perturbation types.

  • I. A Collection of Model Errors: Figures 21–23 collect ImageNet validation errors and visualize Gaussian-noise severity alongside errors found by random sampling.The ImageNet examples include errors that humans would not make, despite high model accuracy.
  • I. A Collection of Model Errors: Gaussian-noise examples use σ = 0.05, where the error rate is about 1.7%.Figure 11 also marks the distance at which a typical Gaussian sample lies.
  • I. A Collection of Model Errors: Figure 12 contrasts a PGD error at distance 0.12 with an averaged Gaussian-noise error at distance 0.33.The clean image has 99.9995% probability of the correct class, while both errors are assigned the incorrect class.
  • I. A Collection of Model Errors: Some slices place random Gaussian errors near the decision boundary even when that boundary does not appear close to the clean image.Other slices show that PGD and random errors need not share the same predicted class.
  • I. A Collection of Model Errors: Figure 19 shows an ordinary test error and an adversarial perturbation assigned to the same class, elephant.The adversarial perturbation is constructed to maintain visual similarity while having a large l2 distance.
  • I. A Collection of Model Errors: Figure 20 reports image-level cumulative distributions of error rates for models trained and tested at σ = 0.1 and σ = 0.3.About 40% of test images have an error rate of at least 10^-3 on the blue curve's leftmost point.
Loading 1901.10513v1…