PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Evaluating the Robustness of Neural Networks

Nicholas Carlini, David Wagner

arXiv:1608.04644v2cs.CRcs.CV

TL;DR

Neural networks remain vulnerable to small adversarial input changes, limiting their use in security-critical settings. This paper introduces three attacks that defeat defensive distillation, finding adversarial examples for 100% of tested images and providing a stronger baseline for evaluating defenses.

  • Problem

    Small, often imperceptible input changes can alter neural-network classifications, limiting their use in security-critical domains.

  • Method

    The paper systematically evaluates attack approaches and develops three attacks tailored to the L0, L2, and L∞ distance metrics.

  • Results

    100% of tested images yielded adversarial examples against defensively distilled networks, while the attacks produced less total distortion than prior methods on standard unsecured models.

  • Takeaways & Limitations

    Defensive distillation provides little security benefit against these attacks, which offer a stronger baseline for evaluating candidate defenses.

  • Takeaways & Limitations

    No distance metric perfectly measures human perceptual similarity, leaving the construction and evaluation of an optimal metric for future work.

Abstract

from arXiv · show

Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input $x$ and any target classification $t$, it is possible to find a new input $x'$ that is similar to $x$ but classified as $t$. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from $95\%$ to $0.5\%$. In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with $100\%$ probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.

I. INTRODUCTION · II. BACKGROUND · A. Threat Model

The paper frames adversarial examples as a security threat to neural networks, evaluates robustness under a white-box threat model using image-distance metrics, and shows that stronger attacks defeat defensive distillation. It introduces attacks for L0, L2, and L∞ norms, reports complete success on distilled networks, and proposes transferability as an additional defense test.

  • I. INTRODUCTION: Small, often undetectable image changes can alter a neural network’s classification, creating adversarial examples that threaten security-critical applications.Examples include self-driving cars, whose decisions could be manipulated by attackers.
  • I. INTRODUCTION: Defensive distillation initially reduced existing attacks’ success probability from 95% to 0.5%, but the paper argues this did not provide substantial robustness.The defense applies to arbitrary feed-forward networks with one retraining step, yet stronger attacks later overcome it.
  • I. INTRODUCTION: The authors introduce three attacks for L0, L2, and L∞ distance metrics, finding adversarial examples for 100% of images on defensively distilled networks.They also report that their attacks produce less total distortion than prior methods on standard unsecured models.
  • I. INTRODUCTION: High-confidence adversarial examples transfer from unsecured models to defensively distilled models, so the authors propose transferability as a simple defense-evaluation test.They argue that defenses should demonstrate that they can break this transferability property.
  • I. INTRODUCTION: The study evaluates attacks on MNIST, CIFAR-10, and ImageNet, spanning digit recognition, small-image recognition, and 1000-class large-image recognition.For ImageNet, the L0 attack is described as the first published targeted attack, and Inception v3 can be fooled by changing only each pixel’s lowest-order bit.
  • II. BACKGROUND: Neural networks increasingly support security-critical decisions in self-driving cars, drones, robots, anomaly detection, malware classification, speech recognition, and NLP.This breadth makes understanding deep-learning security properties a crucial question.
  • A. Threat Model: The required distortion depends on the application domain; for images, the paper uses Lp norms as approximations of human perceptual distance.The authors emphasize that the appropriate distance metric differs across domains.
  • A. Threat Model: The adversary is assumed to have complete white-box access to the neural network’s architecture and parameters, although attacks can also transfer from substitute models trained with black-box access.Robustness is defined by how difficult it is to find adversarial examples close to the original input.

B. Neural Networks and Notation · C. Adversarial Examples

The paper formalizes neural networks as softmax classifiers operating on vectorized inputs, then studies targeted adversarial examples under multiple target-selection strategies. Its evaluations focus on best-, average-, and worst-case targeted attacks, with ImageNet approximations for efficiency.

  • B. Neural Networks and Notation: A neural network is modeled as a fixed function F(x) = y mapping x ∈ R^n to y ∈ R^m.The model parameters θ are implicit but omitted because the network is fixed.
  • B. Neural Networks and Notation: The classifier uses softmax outputs as class probabilities and assigns C(x) = arg max_i F(x)_i.Each output satisfies 0 ≤ y_i ≤ 1, and the probabilities sum to 1.
  • B. Neural Networks and Notation: The notation distinguishes the full softmax network F from Z(x) = z, whose outputs are the logits before softmax.This follows the notation of Papernot et al.
  • B. Neural Networks and Notation: The networks primarily use ReLU activations, with weights and biases forming the model parameters.Other listed activation functions include tanh, sigmoid, and ELU.
  • B. Neural Networks and Notation: Image classification is the primary evaluation domain, using grayscale vectors x ∈ R^hw and RGB vectors x ∈ R^3hw containing raw pixel values.Pixel intensities are scaled to [0, 1], and RGB images are not converted to cylindrical color representations.
  • C. Adversarial Examples: A targeted adversarial example is a close input x′ that changes a valid input x to a specified incorrect target t, satisfying C(x′) = t.Untargeted attacks instead require only C(x′) ≠ C*(x), are strictly less powerful, and are excluded from this paper.
  • C. Adversarial Examples: The evaluations choose targeted classes using average-case, best-case, and worst-case strategies.Average case samples an incorrect class uniformly; best case reports the easiest incorrect class, while worst case reports the most difficult.
  • C. Adversarial Examples: On ImageNet, best-case and worst-case attacks sample 100 random target classes from the 1,000 possible classes for efficiency.All three attack types are performed in the evaluations.

D. Distance Metrics … A. L-BFGS

The paper defines adversarial-example similarity using Lp metrics, outlines defensive distillation, and introduces attacks targeting L0, L2, and L∞ while reviewing box-constrained L-BFGS. It argues that defensive distillation does not eliminate adversarial examples and organizes the paper around attacks and defenses.

  • D. Distance Metrics: L0 distance counts altered coordinates, corresponding to the number of changed pixels in an image.For RGB images, pixels are counted as different when any color channel differs.
  • D. Distance Metrics: L2 measures Euclidean distance and can remain small despite many small changes across many pixels.L∞ instead measures the maximum coordinate change and permits changes to any number of pixels within a maximum budget.
  • D. Distance Metrics: The authors use L0, L2, and L∞ because prior work selected these metrics and defensive distillation claimed security against two of them.They construct attacks intended to outperform prior state-of-the-art methods for each metric, without judging which metric best matches human perception.
  • E. Defensive Distillation: Defensive distillation trains an initial network with softened softmax outputs, then trains a second network using the resulting soft labels.The second network is intended to match the first network while receiving additional hidden knowledge from its soft labels.
  • E. Defensive Distillation: Defensive distillation does not remove adversarial examples, consistent with the paper’s discussion of the locally-linear explanation for their existence.The paper contrasts this explanation with the alternative view that adversarial examples arise from nonlinear-network blind spots.
  • F. Organization: The paper surveys prior L2, L∞, and L0 attacks, presents attacks for the same metrics, and then examines defensive distillation and failures of existing attacks.This sequence describes the paper’s organization before the detailed attack and defense sections.
  • A. L-BFGS: Szegedy et al.’s box-constrained L-BFGS finds an image x′ that differs from x under L2 distance while receiving a different classifier label.The method is formulated as a constrained minimization problem.
  • A. L-BFGS: L-BFGS repeatedly solves a loss-based optimization problem while adjusting c through one-dimensional search to obtain an adversarial example of minimum distance.Cross-entropy is one common choice for the loss function, and bisection search is one possible method for updating c.

B. Fast Gradient Sign · C. JSMA

Fast gradient sign prioritizes speed under the L∞ metric by shifting all pixels simultaneously, while iterative refinement improves its results. JSMA instead targets L0 distance through greedy, saliency-guided pixel modifications, with variants differing in whether they use logits or softmax outputs.

  • B. Fast Gradient Sign: B. Fast Gradient Sign: Fast gradient sign is optimized for the L∞ distance metric and speed rather than minimal adversarial perturbations.It chooses a sufficiently small ϵ and changes every pixel simultaneously according to the loss gradient.
  • B. Fast Gradient Sign: B. Fast Gradient Sign: Iterative gradient sign replaces one step of size ϵ with multiple smaller steps α, clipping the result by the same ϵ.The method is a refinement of fast gradient sign.
  • B. Fast Gradient Sign: B. Fast Gradient Sign: Iterative gradient sign was found to produce superior results to fast gradient sign.
  • C. JSMA: C. JSMA: JSMA is an attack optimized under L0 distance that greedily modifies pixels one at a time to increase the target classification.A saliency map derived from ∇Z(x)l estimates each pixel’s impact on the target class.
  • C. JSMA: C. JSMA: For pixel pairs, JSMA selects changes with αpq > 0, βpq < 0, and the largest −αpq · βpq.These conditions increase the target class while decreasing the other outputs.
  • C. JSMA: C. JSMA: JSMA-Z computes its gradient from the second-to-last-layer logits Z rather than the softmax output F.The logits are used in the saliency-map calculation.
  • C. JSMA: C. JSMA: JSMA-F is the defensive-distillation modification that computes the attack using the softmax output F instead of logits Z.This variant was used when the attack was applied to defensively distilled networks.
  • C. JSMA: C. JSMA: With multiple color channels, JSMA counts each independently changed channel toward L0, so changing all three RGB channels gives an L0 norm of 3.The authors evaluate under both this channel-wise model and the alternative model because they do not consider the channel-wise threat model meaningful.

D. Deepfool · IV. EXPERIMENTAL SETUP · V. OUR APPROACH

The paper describes Deepfool, the experimental models and training setup, and an optimization-based framework for constructing adversarial examples under L0, L2, or L∞ distance metrics. The evaluation uses MNIST, CIFAR-10, and ImageNet models, including reproduced defensive-distillation architectures and a pre-trained Inception v3 network.

  • D. Deepfool: Deepfool efficiently generates closer untargeted adversarial examples than L-BFGS under the L2 distance metric.It approximates networks as linear classifiers, analytically derives a solution, and iteratively steps toward it until finding a true adversarial example.
  • D. Deepfool: Deepfool constructs adversarial examples by repeatedly linearizing the network and moving toward an analytically derived class-separating hyperplane.The search terminates when a true adversarial example is found.
  • IV. EXPERIMENTAL SETUP: 99.5% accuracy is achieved on MNIST, while CIFAR-10 reaches 80%, matching the accuracy reported in the defensive-distillation work.The authors train networks for MNIST and CIFAR-10 and use a pre-trained network for ImageNet, following the original models and training approaches.
  • IV. EXPERIMENTAL SETUP: The CIFAR-10 model overfits substantially, with training cross-entropy loss 0.05 and accuracy 98% versus validation loss 1.2 and accuracy 80%.The network is not modified with image augmentation or additional dropout, preserving the setup of the original work.
  • IV. EXPERIMENTAL SETUP: 96% top-5 accuracy is achieved by the pre-trained Inception v3 network used for ImageNet, whose inputs are 299×299×3 dimensional vectors.The ImageNet model is used instead of training a new network.
  • V. OUR APPROACH: The adversarial-example problem seeks a small perturbation δ that changes an image x’s classification while keeping x+δ valid.The perturbation minimizes D(x, x+δ), where D is one of L0, L2, or L∞.
  • V. OUR APPROACH: The authors formulate adversarial-example construction as an optimization problem and empirically identify formulations that produce the most effective attacks.Existing optimization algorithms are used to solve the selected formulations.

A. Objective Function

The attack reformulates the nonlinear classification constraint as an objective-function condition and balances it against an l_p distance term using a positive constant c. In practice, c is selected by modified binary search, with the smallest successful value usually yielding near-optimal solutions.

  • Objective-function reformulation: The nonlinear constraint C(x + δ) = t is replaced by an objective function f satisfying C(x + δ) = t if and only if f(x + δ) ≤ 0.This reformulation makes the attack problem better suited for optimization.
  • Objective-function reformulation: For c > 0, the alternative formulation is equivalent to the original problem for some c, and instantiating D with an l_p norm yields the attack optimization problem.The constant c balances the objective function with the distance term.
  • Choosing c: The smallest c producing a solution x* with f(x*) ≤ 0 is recommended because gradient descent then minimizes both terms simultaneously.This avoids optimizing only one term before the other.
  • Choosing c: 70% of solutions are within 5% of optimal and 98% are within 30% when c is the smallest value satisfying f(x*) ≤ 0.Here, optimal means the solution obtained using the best value of c.

B. Box constraints · C. Evaluation of approaches

The paper enforces valid image pixels with box-constraint methods and evaluates objective functions and encodings on MNIST and CIFAR. Objective choice dominates attack quality, while the best-performing losses are explained by how gradient behavior changes during optimization.

  • B. Box constraints: Box constraints require every modified pixel to satisfy 0 ≤ x_i + δ_i ≤ 1.The paper notes that L-BFGS-B natively supports these constraints.
  • B. Box constraints: Projected gradient descent clips coordinates after each step, whereas clipped gradient descent incorporates clipping into the objective.Projected clipping can interfere with complicated updates such as momentum; the alternative guarantees validity through the parameterization.
  • B. Box constraints: Adam is used almost exclusively because it was most effective at quickly finding adversarial examples, while all three tested solvers produced identical results.The tested solvers were standard gradient descent, momentum gradient descent, and Adam.
  • C. Evaluation of approaches: 1000 random instances were used to compare seven objective functions across three box-constraint encodings using average L2 distortion, standard deviation, and success probability.When success was below 100%, the reported mean covered successful attacks only.
  • C. Evaluation of approaches: The relative ordering of objective functions was identical on MNIST and CIFAR, so only MNIST results were reported.The evaluation measured the quality of adversarial examples found on both datasets.
  • C. Evaluation of approaches: 3× separates the best and worst objective functions, while box-constraint handling has less impact for the best minimization functions.Cross-entropy loss performed worst despite being the approach most previously suggested in the literature.
  • C. Evaluation of approaches: When c = 0, optimization does not leave the initial image, whereas large c makes early steps overly greedy and can produce suboptimal solutions.For f1 and f4, no fixed c remains useful throughout the search because the distance and loss terms do not stay approximately balanced.
  • C. Evaluation of approaches: r > .9 was obtained between logits and probabilities along interpolations to adversarial examples on the first 1,000 MNIST and CIFAR test images.The paper reports that the target logit is mostly linear along the interpolation and the target probability is logistic.

D. Discretization … B. Our L0 Attack

The attacks enforce valid discrete pixel values after continuous optimization, while the L2 and L0 methods achieve targeted adversarial examples through confidence-controlled optimization and iterative pixel selection. L2 examples are often visually indistinguishable, whereas L0 examples are more noticeable and the attack is more difficult.

  • D. Discretization: Valid images require integer pixel intensities from 0 to 255, so continuous solutions are rounded to the nearest integer before evaluation.The ith intensity becomes ⌊255(x_i + δ_i)⌉.
  • D. Discretization: Greedy search over the discrete lattice restores attack quality when rounding degrades it and never failed for any tested attack.The search changes one pixel value at a time.
  • A. Our L2 Attack: The L2 attack searches for a target-class adversarial example by optimizing a differentiable objective, with κ controlling the confidence of the resulting misclassification.The attacks set κ = 0, while the formulation permits confidence control.
  • A. Our L2 Attack: Almost all MNIST L2 attacks are visually indistinguishable from the original digit, and no CIFAR attack is visually distinguishable from its baseline image.The MNIST result covers every source-target digit pair, while the CIFAR comparison appears in the appendix.
  • VI. OUR THREE ATTACKS: Multiple random starting points near the original image reduce the risk that gradient descent becomes trapped in a local minimum.Points are sampled uniformly from a ball whose radius is the closest adversarial distance found so far.
  • B. Our L0 Attack: Because L0 is non-differentiable, the attack iteratively fixes pixels with limited classifier effect until only a minimal, though not necessarily minimum, modifiable subset remains.Each iteration invokes the L2 adversary while restricting changes to the currently allowed pixels.
  • B. Our L0 Attack: Warm-starting each restricted L2 optimization from the previous solution makes the L0 attack about as fast as a single-start L2 attack on MNIST and CIFAR.The L0 attack is substantially slower on ImageNet and is reported as significantly more effective than JSMA.
  • B. Our L0 Attack: L0 attacks on MNIST are visually noticeable, indicating greater difficulty than L2; converting a 7 into a 6 is highlighted as a particularly difficult case.The corresponding CIFAR examples are shown in the appendix.

C. Our L∞Attack · VII. ATTACK EVALUATION

The paper develops an iterative L∞ attack that avoids gradient-descent oscillation by penalizing all perturbation values exceeding a decreasing threshold. Across MNIST, CIFAR, and ImageNet, the attacks consistently find adversarial examples, outperform prior methods in distortion or success rate, and remain practical to run.

  • C. Our L∞Attack: Standard gradient descent performs poorly for L∞ optimization because the norm penalizes only the largest perturbation entry, causing oscillation between suboptimal solutions.When two entries are nearly tied for the maximum, the gradient can alternate between them and make little progress.
  • C. Our L∞Attack: The iterative L∞ attack replaces the L2 objective term with a penalty on every perturbation exceeding τ, initially setting τ to 1 and reducing it by 0.9 after successful iterations.The penalty targets all large values simultaneously, preventing the oscillation seen with direct gradient descent.
  • C. Our L∞Attack: The L∞ attack searches for the constant c by starting very low, doubling it after failure, and aborting when c exceeds a fixed threshold.Warm-starting gradient descent makes the method about as fast as the L2 attack with one starting point.
  • VII. ATTACK EVALUATION: The evaluation compares targeted attacks against prior methods on the first 1,000 MNIST and CIFAR test images and 1,000 correctly classified ImageNet images.For ImageNet, the evaluation samples 100 target classes, representing 10% of the classes.
  • VII. ATTACK EVALUATION: 100% success probability is achieved by the L0 and L2 attacks, which find adversarial examples with 2× to 10× lower distortion than the best previously published attacks.Across all three datasets, the attacks find closer adversarial examples than previous state-of-the-art methods and never fail.
  • VII. ATTACK EVALUATION: 100% success is achieved by the L∞ attacks on ImageNet for targeted L0 adversarial examples, whereas JSMA is unable to find them.The L∞ attacks are comparable in quality to prior work but have higher success rates, and performance improves as task complexity increases.
  • VII. ATTACK EVALUATION: None of the digits generated from entirely black or entirely white MNIST images are recognizable, unlike results from the earlier L0 attack.The experiments show minimum perturbations needed to produce each target digit from the two uniform starting images.
  • VII. ATTACK EVALUATION: No attack takes longer than a few minutes on any given instance, although exact runtime comparisons are avoided because implementations differ in parallelization.The authors report their attacks as 2×–10× slower than optimized JSMA for L0 and typically 10×–100× slower than prior attacks for L2 and L∞.

VIII. EVALUATING DEFENSIVE DISTILLATION

Defensive distillation applies distillation to improve neural-network robustness while keeping teacher and distilled models the same size. Its procedure trains both networks with a large temperature, then evaluates the distilled model at temperature 1.

  • Defensive distillation: Defensive distillation aims to increase neural-network robustness without reducing model size, using identical teacher and distilled networks.Its key change from ordinary distillation is applying a large distillation temperature to make predictions more confident.
  • Temperature: A temperature constant T modifies the softmax function, with increasing temperature producing a softer maximum and decreasing temperature producing a harder maximum.Equivalently, softmax(x, T) = softmax(x/T, 1).
  • Training procedure: The teacher network is trained with softmax temperature T, and then labels each training instance using the same temperature.These outputs are soft labels rather than the training set’s hard labels.
  • Training procedure: The distilled network, which has the teacher’s shape, is trained on those soft labels with temperature T and uses temperature 1 at test time.The procedure therefore uses temperature T during distilled-model training but temperature 1 for classifying new inputs.

A. Fragility of existing attacks · B. Applying Our Attacks · C. Effect of Temperature

Existing attacks fail on distilled networks because saturated softmax outputs and vanishing gradients make them fragile, whereas the paper’s attacks succeed reliably. Varying distillation temperature similarly breaks prior attacks without increasing robustness, as improved attacks find unchanged adversarial distances.

  • A. Fragility of existing attacks: Existing attacks are fragile and can fail to find adversarial examples even when such examples exist.L-BFGS and Deepfool fail because F(·)’s gradient is almost always zero, preventing use of the standard objective.
  • A. Fragility of existing attacks: 100 times larger logits make softmax outputs nearly one-hot and gradients small enough to round to zero in 32-bit floating point.For 10-class tasks, the output is approximately 1−9ϵ for the predicted class and ϵ elsewhere.
  • A. Fragility of existing attacks: Replacing the standard L-BFGS objective with a stable objective prevents failure by avoiding gradients that vanish through floating-point rounding.The passage identifies this as evidence that the choice of loss function makes the attack fragile.
  • A. Fragility of existing attacks: JSMA-F fails because large logits turn softmax into an effectively hard maximum, while JSMA-Z’s equal treatment of logit changes magnifies suboptimal choices after distillation.Fast Gradient Sign also initially fails from nearly zero gradients; dividing logits by T does not restore its effectiveness, for an unexplained reason.
  • B. Applying Our Attacks: 100% success probability is achieved by the paper’s attacks for each of the three distance metrics on defensively distilled networks.All previous attacks fail to find adversarial examples under the same evaluation.
  • B. Applying Our Attacks: Distillation adds almost no value: the paper’s L0 and L2 attacks perform slightly worse, while its L∞ attack performs approximately equally compared with undistilled networks.All three attacks still succeed with 100% success.
  • C. Effect of Temperature: 91% success at T = 1 falls to 24% at T = 5 and 0.5% at T = 100 for the original attacks on MNIST.The original work reported that increasing temperature consistently reduced attack success rate.
  • C. Effect of Temperature: ρ = −0.05 is the correlation between temperature and mean adversarial distance for the improved L2 attack, showing no temperature effect on that distance.The improved attacks therefore indicate that higher temperature causes existing attacks to fail more often rather than increasing network robustness.

D. Transferability · IX. CONCLUSION

Adversarial examples can transfer across models, enabling attacks that move from standard to defensively distilled networks. The paper concludes that robustness should be tested with powerful direct attacks and high-confidence transferability evaluations.

  • D. Transferability: Adversarial examples often transfer across models trained on different data or built with different algorithms.
  • D. Transferability: A robust defense must break transferability, or attacks found on an easy-to-attack model can target a harder-to-attack model.
  • D. Transferability: The authors break defensive distillation by transferring attacks from a standard model to a defensively distilled model.
  • D. Transferability: High-confidence adversarial examples strongly favor the target label rather than merely crossing the classification boundary.Increasing κ strengthens adversarial examples and produces higher-confidence misclassifications.
  • D. Transferability: Near-100% transfer success occurs at κ ≈20 in the baseline MNIST experiment after success rises linearly from κ = 0 to κ = 20.
  • D. Transferability: 100% transfer success between unsecured and distilled models requires κ = 40, compared with κ = 20 for the previous baseline approach.
  • IX. CONCLUSION: The paper proposes L0, L2, and L∞ attacks that defeat defensive distillation and consistently find better adversarial examples than existing approaches.
  • IX. CONCLUSION: Defenders should test secured models with powerful direct attacks, establish L2 robustness, and show that high-confidence examples fail to transfer from unsecured models.

APPENDIX

The appendix illustrates the paper’s L0, L2, and L∞ adversaries on CIFAR, each performing targeted attacks across every source/target pair. Each example uses the first dataset image associated with its label.

  • L0 adversary: The L0 adversary performs a targeted attack for every CIFAR source/target pair.Each image is the first image in the dataset with that label.
  • L2 adversary: The L2 adversary performs a targeted attack for every CIFAR source/target pair.Each image is the first image in the dataset with that label.
  • L∞ adversary: The L∞ adversary performs a targeted attack for every CIFAR source/target pair.Each image is the first image in the dataset with that label.
Loading 1608.04644v2…