PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining and Harnessing Adversarial Examples

Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy

arXiv:1412.6572v3stat.MLcs.LG

TL;DR

The paper asks why models misclassify small adversarial perturbations and why these examples generalize across architectures and training sets. It attributes the vulnerability to linear behavior in high-dimensional spaces, develops a fast gradient-sign method for generating examples, and reports reduced MNIST test error through adversarial training.

  • Problem

    The paper investigates the unexplained vulnerability of models to small perturbations and the recurring transfer of adversarial examples across architectures and training sets.

  • Method

    The paper explains adversarial examples through linear models and high-dimensional dot products, then uses gradient-sign perturbations for fast adversarial example generation and training.

  • Results

    Adversarial perturbation directions generalize across clean examples, while adversarial training reduces the test error of a maxout network on MNIST.

  • Takeaways & Limitations

    Linear behavior, rather than extreme nonlinearity, provides a supported account of adversarial vulnerability and makes adversarial training practical.

  • Takeaways & Limitations

    The fast gradient-sign adversarial objective is nondifferentiable, preventing gradient descent from modeling how the adversary reacts to parameter changes.

Abstract

from arXiv · show

Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Early attempts at explaining this phenomenon focused on nonlinearity and overfitting. We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets. Moreover, this view yields a simple and fast method of generating adversarial examples. Using this approach to provide examples for adversarial training, we reduce the test set error of a maxout network on the MNIST dataset.

1 INTRODUCTION

The paper argues that adversarial examples arise from linear behavior in high-dimensional spaces rather than extreme nonlinearity, and uses this view to motivate a fast generation method. It also identifies a tension between trainability through linearity and robustness from nonlinear effects.

  • Motivation: Models with different architectures and training subsets can misclassify the same adversarial example.These examples differ only slightly from correctly classified examples drawn from the data distribution.
  • Contribution: Linear behavior in high-dimensional spaces is sufficient to cause adversarial examples, making speculative explanations involving extreme nonlinearity unnecessary.The paper uses this explanation to design a fast method for generating adversarial examples.
  • Limitation: The explanation reveals a tension between models that are easy to train because of their linearity and models that resist adversarial perturbation through nonlinear effects.More powerful optimization methods might eventually enable training substantially more nonlinear models.

2 RELATED WORK

Prior work showed that adversarial examples can be found reliably, remain visually imperceptible, transfer across classifiers, and affect even shallow linear models. Adversarial training offered regularization but was computationally impractical, while no defense retained state-of-the-art clean accuracy.

  • Box-constrained L-BFGS can reliably find adversarial examples.
  • On some datasets, adversarial examples are so close to originals that their differences are indistinguishable to humans.
  • The same adversarial example often fools classifiers with different architectures or training subsets.
  • Shallow softmax regression models are also vulnerable to adversarial examples.
  • Adversarial-example training can regularize models, but expensive constrained optimization made it impractical, and no defense preserved state-of-the-art clean accuracy.

3 THE LINEAR EXPLANATION OF ADVERSARIAL EXAMPLES

The section explains adversarial examples through linear models: bounded, feature-level perturbations can produce activation changes that grow with input dimensionality, making high-dimensional linear classifiers vulnerable. This linearity-based account is simpler than explanations invoking neural-network nonlinearity and also covers softmax regression.

  • 3 THE LINEAR EXPLANATION OF ADVERSARIAL EXAMPLES: Linear models can exhibit adversarial examples when inputs have sufficient dimensionality.The section begins by explaining the existence of adversarial examples for linear models and concludes that sufficient dimensionality is enough.
  • 3 THE LINEAR EXPLANATION OF ADVERSARIAL EXAMPLES: Limited feature precision makes small perturbations indistinguishable from the original input to a rational classifier.For digital images, 8-bit pixels discard information below 1/255 of the dynamic range.
  • 3 THE LINEAR EXPLANATION OF ADVERSARIAL EXAMPLES: Under a max-norm constraint, choosing η = sign(w) maximizes the activation increase caused by the perturbation.The activation grows by w⊤η, and the maximizing perturbation is η = sign(w).
  • 3 THE LINEAR EXPLANATION OF ADVERSARIAL EXAMPLES: ϵmn is the activation growth when w has n dimensions and average element magnitude m.The perturbation’s max norm does not grow with dimensionality, while its activation effect can grow linearly with n.
  • 3 THE LINEAR EXPLANATION OF ADVERSARIAL EXAMPLES: The linearity hypothesis is simpler than explanations based on neural networks’ supposed nonlinearity and explains softmax regression vulnerability.The passage contrasts the linearity account with prior hypotheses about highly nonlinear neural networks.

4 LINEAR PERTURBATION OF NON-LINEAR MODELS

The section argues that neural networks’ intentionally linear behavior makes them vulnerable to fast linear adversarial perturbations. It introduces the fast gradient sign method, which efficiently generates misclassified examples and can accelerate adversarial training or network analysis.

  • Motivation: Neural networks’ linear behavior motivates testing whether linear adversarial perturbations can overcome their resistance.The passage specifically identifies LSTMs, ReLUs, and maxout networks as intentionally designed to behave linearly for easier optimization.
  • Method: Linearizing the cost function yields an optimal max-norm-constrained perturbation based on the input-gradient sign.The method uses the gradient of the training cost with respect to the input.
  • Method: The fast gradient sign method computes adversarial examples efficiently using backpropagation.The required gradient can be computed efficiently, making the procedure fast and cheap.
  • Results: 99.9% error rate and 79.3% average confidence were achieved for a shallow softmax classifier on MNIST with ϵ = .25.Under the same setting, the maxout network misclassified 89.4% of adversarial examples with 97.6% average confidence.
  • Results: The method reliably causes diverse models to misclassify inputs, supporting linearity as an explanation for adversarial examples.The resulting algorithms can also speed up adversarial training or analysis of trained networks.

5 ADVERSARIAL TRAINING OF LINEAR MODELS VERSUS WEIGHT DECAY

For logistic regression, the fast gradient sign method exactly produces the most damaging perturbation in the max-norm box, enabling an analytical adversarial-training objective. Adversarial training resembles L1 regularization but can diminish as predictions become confident, while L1 weight decay increasingly overestimates achievable damage beyond binary logistic regression.

  • Logistic regression: For logistic regression, the fast gradient sign method is exact and produces the most damaging perturbation in the max-norm box.The optimal perturbation is determined by the sign of the model weights.
  • Logistic regression: The adversarial version of logistic regression follows analytically from the gradient sign, with the gradient sign equal to −sign(w).The derivation also uses w⊤sign(w) = ||w||1.
  • Relation to weight decay: Adversarial training is somewhat similar to L1 regularization, but subtracts the penalty from activation rather than adding it to training cost.The penalty can disappear when the softplus function saturates under sufficiently confident predictions.
  • Relation to weight decay: In the underfitting regime, adversarial training is not guaranteed to reduce the penalty and instead simply worsens underfitting.This limitation follows because the penalty may fail to disappear when predictions are not sufficiently confident.
  • Multiclass extension: For multiclass softmax regression, L1 weight decay becomes more pessimistic because it treats outputs as independently perturbable when one perturbation usually cannot align with all class weight vectors.The overestimation becomes even greater in deep networks with multiple hidden units.

6 ADVERSARIAL TRAINING OF DEEP NETWORKS

Adversarial training encodes resistance to adversarial examples into the learning procedure, using fast-gradient-sign perturbations as an effective regularizer. On MNIST, this reduced clean and adversarial error while improving robustness, though the method has limitations and inconsistent effects across network layers.

  • Motivation: Neural networks can represent functions resistant to adversarial perturbations, but standard supervised training does not require learning such functions.The universal approximator theorem establishes representational capacity, not that training will discover a robust function.
  • Adversarial objective: Fast-gradient-sign adversarial training combines clean and perturbed-example losses and acts as an effective regularizer.The objective uses a mixture coefficient α = 0.5 in all experiments and continually updates adversarial examples against the current model.
  • Robustness: 17.9% adversarial-example error followed adversarial training, down from 89.4% without training for the same kind of model.Adversarial examples remained transferable between the models, but the adversarially trained model was more robust.
  • Limitations: The fast-gradient-sign objective cannot account for how the adversary reacts to parameter changes because the sign function's derivative is zero or undefined.Differentiable perturbation processes, such as small rotations or scaled-gradient additions, could allow the training procedure to incorporate that reaction.
  • Layer perturbations: Results on whether to perturb inputs, hidden layers, or both were inconsistent, with unbounded hidden activations responding by becoming very large.A prior result favored hidden-layer perturbations on a sigmoidal network, whereas these experiments found different behavior for networks with unbounded hidden units.

7 DIFFERENT KINDS OF MODEL CAPACITY

Low-capacity models can still make confident predictions, but their behavior reflects a precision–recall tradeoff. RBF networks resist adversarial examples by becoming uncertain when fooled, whereas linear units generalize broadly at the cost of precision.

  • Different kinds of model capacity: Low-capacity models can make confident predictions in restricted regions while defaulting to absent or low-confidence predictions elsewhere.Such models need not be unable to make many different confident predictions; their confidence can be localized around particular regions of input space.
  • Different kinds of model capacity: 55.4% error with 1.2% confidence on adversarial MNIST examples shows that shallow RBF networks are often fooled with low confidence.Using fast gradient sign adversarial examples with ϵ = .25, the network’s average confidence on clean test examples is 60.6%.
  • Different kinds of model capacity: RBF networks are naturally immune to adversarial examples in the sense that they have low confidence when fooled, but they generalize poorly under significant transformations.Their localized responses improve precision but sacrifice recall because RBF units are not invariant to significant transformations.
  • Different kinds of model capacity: Linear and RBF units occupy different points on a precision–recall tradeoff: linear units favor recall, while RBF units favor precision.Linear units respond to every input in a direction, whereas RBF units respond only near a specific point in space.

8 WHY DO ADVERSARIAL EXAMPLES GENERALIZE?

Adversarial examples generalize because, under the linear view, they occupy broad contiguous subspaces, while models trained similarly learn stable classification weights. This helps explain transfer across architectures and disjoint training sets, with softmax agreeing more often than RBF in one test.

  • Transfer across models: Adversarial examples often transfer across models with different architectures or disjoint training sets, and those models often agree on the assigned class.The passage identifies this transfer and agreement as the central phenomenon that nonlinearity- and overfitting-based explanations struggle to account for.
  • Linear explanation: Under the linear view, adversarial examples occupy broad, contiguous regions of the one-dimensional subspace defined by the fast gradient sign method.The perturbation direction need only have a positive dot product with the cost gradient, while ϵ need only be sufficiently large.
  • Shared classification weights: Neural networks trained with current methodologies are hypothesized to resemble a linear classifier trained on the same data, whose classification weights remain stable across training subsets.This stability is attributed to machine-learning algorithms’ ability to generalize.
  • Transfer experiment: 54.6% of examples misclassified by the deep maxout network received the maxout network’s class from the shallow softmax classifier, versus 16.0% from the shallow RBF network.The experiment generated adversarial examples on the deep maxout network and evaluated them with shallow softmax and RBF networks.

9 ALTERNATIVE HYPOTHESES

The section tests alternative explanations for adversarial examples, including generative training and model-specific quirks. An MP-DBM provides a differentiable generative-model test case, while an ensemble of twelve maxout networks tests whether averaging suppresses adversarial examples.

  • Alternative hypotheses: The authors consider and refute alternative hypotheses for the existence of adversarial examples.The first proposed alternative is that generative training constrains the training process or distinguishes real from fake data.
  • Generative training: An MP-DBM tests whether generative training improves confidence on real data, achieving an 0.88% error rate on MNIST with differentiable inference.Other generative models have non-differentiable inference procedures or require additional non-generative components.
  • Model averaging: An ensemble of twelve maxout networks tests whether model-specific quirks wash out when adversarial examples are averaged across models.The networks used different random seeds for weight initialization, dropout-mask generation, and stochastic-gradient-descent minibatch selection.

10 SUMMARY AND DISCUSSION

The paper explains adversarial examples primarily through high-dimensional dot products and model linearity, including their transfer across models and clean examples. It also introduces fast generation methods, shows adversarial-training regularization, and characterizes rubbish class examples.

  • Core explanation: Adversarial examples arise from high-dimensional dot products because models are too linear rather than too nonlinear.This reframes vulnerability as a consequence of linear behavior, not primarily nonlinearity or overfitting.
  • Core explanation: Adversarial examples transfer across models because perturbations align with model weight vectors and task-trained models learn similar functions.The explanation accounts for generalization across different architectures and training sets.
  • Core explanation: The perturbation direction matters more than the specific point, so adversarial perturbations generalize across different clean examples.The paper rejects the view that adversarial examples are isolated pockets finely tiling input space.
  • Methods and training: The paper introduces a family of fast adversarial-example generation methods and shows that adversarial training can provide more regularization than dropout.Control experiments did not reproduce this effect with L1 weight decay or added noise.
  • Rubbish class examples: Rubbish class examples are ubiquitous and easily generated, while RBF networks resist them but shallow linear models do not.These observations are presented in the appendix.

A RUBBISH CLASS EXAMPLES

Rubbish-class examples are degenerate inputs that should not receive positive-class predictions, yet models built from linear parts often classify them incorrectly with high confidence. Experiments show this vulnerability in deep and shallow linear models, while an RBF network avoids it on MNIST.

  • Definition: Rubbish-class examples are degenerate inputs that humans would classify as belonging to none of the training-set categories.The goal is to avoid false positives on such inputs.
  • Mechanism: Linearity and high-dimensional spaces make models produce more extreme predictions farther from the training data, yielding high-confidence rubbish false positives.Larger input norms produce greater confidence for linear models.
  • MNIST experiments: 98.35% was the error rate of a naively trained maxout network on Gaussian rubbish examples, with 92.8% average confidence on mistakes; independent sigmoids reduced error to 68%.The independent-sigmoid model had 87.9% average confidence on mistakes.
  • MNIST experiments: 59.8% was the error rate of softmax regression on rubbish examples, whereas an RBF network achieved 0% error.Softmax regression had 70.8% average confidence on mistakes; when error was zero, that average was undefined.
  • Class distribution: False positives were highly skewed across classes, with 45.3% of MNIST maxout mistakes classified as 5s and none as 8s.The methods also produced a skewed class distribution on CIFAR-10.
  • Fooling-image generation: The paper proposes adding ϵ∇xp(y = i | x) to a Gaussian sample as a fast method for generating a fooling image targeting class i.Repeating sampling until success gives a randomized algorithm with variable runtime; on CIFAR-10, one step succeeded for frogs and trucks 100% of the time, versus 24.7% for airplanes.
Loading 1412.6572v3…