PaperlayerView exact revision on arXiv

Source-linked AI summary

Intriguing Properties of Adversarial Examples

Ekin D. Cubuk, Barret Zoph, Samuel S. Schoenholz, Quoc V. Le

arXiv:1711.02846v1stat.MLcs.LG

TL;DR

Adversarial examples expose a vulnerability in classifiers and a challenge to understanding their behavior. The paper analyzes logit differences and adversarial error across models, datasets, and attacks, then studies entropy regularization and architecture search. It reports universal power-law scaling and finds CIFAR10 architectures more robust to white- and black-box attacks than previous studies.

  • Problem

    Classifiers can fail on slightly perturbed images, while prior explanations emphasize high dimensionality, overfitting, or linearity rather than a shared account of adversarial examples.

    Source: §1 · p. 1

  • Method

    The paper analyzes how adversarial error relates to logit differences across models and attacks, derives their shared power-law behavior, and studies entropy regularization and reinforcement-learning architecture search.

    Source: §1 · p. 2, §3.2 · p. 7

  • Results

    Adversarial error follows power laws across studied models and datasets, while the paper’s NAS architectures are more robust to white- and black-box attacks on CIFAR10 than previous studies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that architecture matters for adversarial robustness, which correlates strongly with clean accuracy.

    Source: §5 · p. 11

    The theoretical analysis uses a mean-field assumption that logits are i.i.d. and a factorial approximation whose validity the authors say is not obvious.

    Source: §6.2.2 · p. 15

Abstract

from arXiv · show

It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples. In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear. Here we argue that the origin of adversarial examples is primarily due to an inherent uncertainty that neural networks have about their predictions. We show that the functional form of this uncertainty is independent of architecture, dataset, and training protocol; and depends only on the statistics of the logit differences of the network, which do not change significantly during training. This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation. We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD). Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness. Finally, we study the effect of network architectures on adversarial sensitivity. To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10. Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.

1 INTRODUCTION

The introduction argues that adversarial examples challenge understanding of generalization and pose security risks, then reports common power-law behavior across models and datasets. It connects that behavior to logit differences and explores entropy regularization and architecture selection as ways to improve robustness.

1 INTRODUCTION

  • Across studied models and datasets, adversarial error follows power laws, with FGSM, L2-FGSM, and PGD exponents of 0.9–1.3 and step l.l. exponents of 1.8–2.5.The pattern also holds for FGSM on MNIST with randomly shuffled labels, where the exponent is 1.2.Source: §1 · p. 2
  • At small ϵ, attack success depends on the input-logit Jacobian and network logits, while FGSM and PGD susceptibility is largely dictated by the distribution of the top-two logit difference.The authors report that this distribution has a universal form across studied models and datasets, including randomly produced data.Source: §1 · p. 2
  • Small-ϵ adversarial examples mostly change predictions to the second-largest logit, whereas error plateaus and becomes insensitive to ϵ at larger perturbations.The authors distinguish these as two mechanisms and caution that robustness techniques should account for them.Source: §1 · p. 2
  • Single-step adversarial training protects against attacks with ϵ between 0.2 and 32 but does not appreciably defend against attacks below 0.2.At ϵ = 0.2, studied ImageNet models incur 10–25% adversarial error; vanilla NASNet-A has lower error than the adversarially trained Inception models.Source: §1 · p. 2
  • The paper analytically derives common power-law tails in logit differences and relates them to the shared form of adversarial error.The introduction also reports that entropy regularization and better architectures can quantitatively improve this otherwise universal form.Source: §1 · p. 2
  • Neural architecture search finds CIFAR10 architectures more robust to white- and black-box attacks than previous studies, while adversarial accuracy correlates strongly with clean accuracy but weakly with model size.The introduction frames adversarial examples as a challenge to understanding generalization and as a security risk in real-world applications.Source: §1 · pp. 1–2

2 SURPRISING UNIVERSALITY OF ADVERSARIAL ERROR AT SMALL ϵ

Across the tested ImageNet and MNIST models, adversarial error at small perturbations follows a power law, though attack protocol changes its exponent. The section tests this pattern across architectures, shuffled-label training, and attack types.

2 SURPRISING UNIVERSALITY OF ADVERSARIAL ERROR AT SMALL ϵ

  • For ImageNet models at ϵ < 0.2, FGSM adversarial error follows a power law with exponent 0.9–1.1, including adversarially trained models.Figure 1 plots test-set adversarial error against ϵ for FGSM and step l.l. attacks.Source: §2 · p. 3, Figure 1 · p. 3
  • Step l.l. also yields power-law error, but with exponents of 1.8–2.2 on ImageNet and 1.8–2.5 on MNIST and CIFAR10.The least-likely-class attack therefore changes the observed exponent.Source: §2 · p. 3
  • Across sampled linear, fully connected, and convolutional MNIST networks, FGSM error follows a power law with exponents of 0.9–1.2.The sampled architectures included fully connected networks with 1–4 hidden layers and simple convolutional networks.Source: §2 · p. 3, Figure 2 · p. 4
  • The reported adversarial error follows a power law with exponent 0.99 up to ϵ = 1; for a shuffled-label MNIST network, training-set adversarial error has exponent 1.2 at small ϵ.The fully connected network reached perfect training accuracy before its training-set adversarial error was shown.Source: §2 · p. 4
  • On MNIST, L2-FGSM, FGSM, and PGD exponents are 1.1, 1.2, and 1.3, respectively, compared with 2.3 for step l.l.Figure 2 plots MNIST adversarial error against ϵ across architectures, shuffled-label training, and attacks.Source: §2 · p. 4, Figure 2 · p. 4

3 A MEAN-FIELD THEORY OF ADVERSARIAL PERTURBATIONS

The section models small-perturbation attacks through linearized logit changes and a mean-field account of the top-two logit gap. It finds common power-law behavior across studied models and datasets, while entropy regularization reduces the density of small margins without changing tail exponents.

3.1 LINEAR RESPONSE

  • For small perturbations, linearized logits closely match the true logit dynamics, and predicted and measured fooling thresholds increasingly agree as the threshold becomes small.The figure also compares individual thresholds with the linear prediction and shows a mean-field prediction.Source: Figure 3 · p. 5, §3.1 · p. 5
  • For attacks targeting the second-most-likely class, the mean-field account predicts small-ϵ adversarial error from the cumulative top-two-gap distribution at a network-rescaled ϵ.The mean-field threshold approximation remains strongly correlated with the estimated fooling threshold.Source: §3.1 · pp. 5–6

3.2 UNIVERSAL PROPERTIES OF THE LOGIT DIFFERENCE DISTRIBUTION

  • Across studied models and datasets, logit differences have power-law tails: the top-two gap has exponent about 0, while higher-ranked gaps have positive exponents that increase with rank.The powers are typically nonintegral for larger j, which the authors suggest may reflect correlations between logits.Source: §3.2 · p. 6
  • Many examples have small top-two logit gaps, which the authors interpret as inherent prediction uncertainty and a source of adversarial examples across simple and complex models.This pattern appears in well-studied and random datasets, so the authors argue it is not due to model depth or dataset dimensionality.Source: §3.2 · p. 6
  • Entropy regularization increases adversarial robustness across regular and step l.l.-trained MNIST models, while the permutation-invariant model loses 0.8% clean accuracy.The penalty reduces output entropy and aims to increase logit differences.Source: §3.2 · p. 7
  • With entropy regularization, margins shift larger and fewer samples have small gaps, but the tails retain the same power-law exponents and black-box robustness does not improve significantly.The plotted distributions compare permutation-invariant MNIST models trained with and without the penalty.Source: §3.2 · p. 7, Figure 6 · p. 8

4 ROLE OF NETWORK ARCHITECTURES IN ADVERSARIAL ROBUSTNESS

Neural architecture search (NAS) finds CIFAR10 architectures with improved adversarial robustness, while adversarial accuracy among child models is more strongly correlated with clean accuracy than with model size.

4 ROLE OF NETWORK ARCHITECTURES IN ADVERSARIAL ROBUSTNESS

  • The search uses child-model rewards based on validation FGSM accuracy at ϵ = 8, with separate experiments training on step l.l. or PGD examples.Selected architectures are enlarged, trained for 100 epochs under 12 hyperparameter sets, and evaluated on a held-out test set.Source: §4 · p. 8
  • At ϵ = 8, the Experiment 2 architecture reaches 17% higher adversarial accuracy on PGD examples, matches Madry et al. (2017) on white-box PGD, and is more robust to other white- and black-box attacks.Its clean accuracy is 5.9% higher than Madry et al. (2017), despite using clean examples in half of its minibatches.Source: Figure 7 · p. 9, §4 · p. 9
  • Among child models with clean accuracy above 85%, adversarial accuracy spans 22%, with a standard deviation of 2.6%.Adversarial accuracy is strongly correlated with clean accuracy but only weakly correlated with trainable parameter count.Source: §4 · p. 9, Figure 9 · p. 10
  • At ϵ = 50, under the white-box step l.l. attack, Experiment 1’s best network had 51.8% adversarial accuracy; NAS Baseline had 26.4% on adversarial images created from middle-column clean images.Source: Figure 8 · p. 10

5 CONCLUSION

The paper derives common patterns in logit differences and adversarial error across models and datasets, and reports that architecture matters for adversarial robustness, which correlates strongly with clean accuracy.

5 CONCLUSION

  • The paper theoretically derives universality in logit differences and adversarial error across machine-learning models, and finds that architecture plays an important role in adversarial robustness, which correlates strongly with clean accuracy.Source: §5 · p. 11

6 APPENDIX

The appendix presents further experiments and derivations, including a linear-response analysis and a treatment of logit-difference distributions.

6.1 FURTHER EXPERIMENTS

  • Figure 10 covers adversarial error for MNIST models with clean accuracy above 80%, while Figure 11 fits a power law with exponent 0.99 to FGSM error for a CIFAR10 ResNet.Source: Figure 10 · p. 13, Figure 11 · p. 13
  • Figure 12 compares step l.l. adversarial accuracy with and without entropy regularization for two CIFAR10 wide ResNets with 94% clean accuracy; Figure 13 shows ImageNet models' Δ1N distributions.The Figure 12 models used the same hyperparameters and were trained for 100 epochs; the regularization parameter was λ = 3.0.Source: Figure 12 · p. 14, Figure 13 · p. 14

6.2.1 LINEAR RESPONSE

  • The linear-response derivation defines the input-to-logit Jacobian and output error, then calculates how an adversarial perturbation changes the logits.Source: §6.2.1 · pp. 14–15

6.2.2 UNIVERSAL PROPERTIES OF THE LOGIT DIFFERENCE DISTRIBUTION

  • The logit-difference derivation assumes i.i.d. logits from an arbitrary distribution and defines ranked logits to derive the joint distribution of the largest and jth-largest logits.The passage notes that the resulting distribution shares qualitative similarities with real networks despite uncertainty about the factorial approximation.Source: §6.2.2 · p. 15

6.2.3 ARCHITECTURES

  • Figure 14 contrasts the best Experiment 1 architecture with the NAS Baseline and describes the searched architecture as longer and narrower than earlier high-clean-accuracy NAS architectures.Source: Figure 14 · p. 16

6.2.4 PGD ADVERSARIAL EXAMPLES

  • Figure 15 shows white-box PGD adversarial examples at ϵ = 8 that fooled the best network from Experiment 2.Source: Figure 15 · p. 17
Loading 1711.02846v1…