The Limitations of Deep Learning in Adversarial Settings

Nicolas Papernot; Patrick McDaniel; Somesh Jha; Matt Fredrikson; Z. Berkay Celik; Ananthram Swami

Source-linked AI summary

The Limitations of Deep Learning in Adversarial Settings

Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami

arXiv:1511.07528v1Nov 24, 2015cs.CRcs.LGcs.NEstat.ML

TL;DR

Deep neural networks can be manipulated into misclassifying inputs, raising concerns for deployed classification systems. The paper formalizes adversaries and uses forward derivatives to craft targeted adversarial samples, achieving 97.10% success while modifying 4.02% of input features on average.

Problem
Deployed deep-learning classifiers create incentives for adversaries to manipulate inputs and evade or subvert classification, including in safety-relevant systems.
Method
The paper formalizes targeted adversarial perturbations and constructs input-to-output mappings using DNN Jacobians and adversarial saliency maps.
Results
97.10% adversarial success was achieved while perturbing 4.02% of input features per sample on average.
Takeaways & Limitations
Hardness and adversarial-distance metrics provide foundations for defenses based on adversarial-sample detection and improved DNN robustness.
Takeaways & Limitations
The study does not address unsupervised DNNs or cyclical recurrent neural networks.

Abstract

from arXiv · show

Deep learning takes advantage of large datasets and computationally efficient training algorithms to outperform other approaches at various machine learning tasks. However, imperfections in the training phase of deep neural networks make them vulnerable to adversarial samples: inputs crafted by adversaries with the intent of causing deep neural networks to misclassify. In this work, we formalize the space of adversaries against deep neural networks (DNNs) and introduce a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs. In an application to computer vision, we show that our algorithms can reliably produce samples correctly classified by human subjects but misclassified in specific targets by a DNN with a 97% adversarial success rate while only modifying on average 4.02% of the input features per sample. We then evaluate the vulnerability of different sample classes to adversarial perturbations by defining a hardness measure. Finally, we describe preliminary work outlining defenses against adversarial samples by defining a predictive measure of distance between a benign input and a target classification.

I. INTRODUCTION · II. TAXONOMY OF THREAT MODELS IN DEEP LEARNING · A. About Deep Neural Networks

The paper formalizes adversarial threats to deep neural networks and introduces forward-derivative algorithms that efficiently craft targeted adversarial samples. It validates these methods on LeNet and MNIST, quantifies perturbation and source-to-target hardness, and examines defenses and human perception.

I. INTRODUCTION: Deep learning’s growing use creates incentives for adversaries to manipulate DNNs into misclassifying inputs, including classifiers for content moderation and spam detection.
I. INTRODUCTION: Adversarial samples are test-time inputs crafted after training to cause a DNN to misclassify, without altering the defender’s training process.
I. INTRODUCTION: The paper formalizes adversary goals and capabilities and introduces algorithms for feedforward DNNs that directly map input perturbations to desired outputs while changing a small feature fraction.
I. INTRODUCTION: Forward derivatives characterize learned behavior, while adversarial saliency maps guide efficient exploration of perturbations across supervised and unsupervised architectures.
I. INTRODUCTION: 97.10% success was achieved in forcing any input sample into any target class while perturbing an average of 4.02% of input features per sample.Samples were generated in less than a second in the reported setup.
I. INTRODUCTION: The study validates the algorithms on a LeNet digit-recognition DNN and MNIST, measures distortion and source-to-target hardness, explores defenses, and evaluates human perception.
II. TAXONOMY OF THREAT MODELS IN DEEP LEARNING: The taxonomy section classifies deep-learning threat models by adversary goals and capabilities and positions prior work according to modeled adversary strength.
A. About Deep Neural Networks: DNNs are layered networks of neurons whose activation functions transform inputs, with weighted and biased links serving as parameters that store information.Deep learning includes supervised models inferred from labeled data and unsupervised models that learn representations from unlabeled data.

B. Adversarial Goals · C. Adversarial Capabilities

The paper defines adversarial goals by how an input corrupts classifier integrity, and organizes test-time adversaries by the information and capabilities available to them. Goals range from confidence reduction to targeted class changes, while capabilities span complete model knowledge, oracle access, and collected samples.

B. Adversarial Goals: Adversaries seek inputs X* that produce incorrect output classifications, with the nature of the incorrectness defining the adversarial goal.
B. Adversarial Goals: The four goals are confidence reduction, misclassification, targeted misclassification, and source/target misclassification.They respectively reduce confidence, change the class arbitrarily, force a specific target class, or force a source class toward a target class.
B. Adversarial Goals: Prior work produced human-indistinguishable perturbations that caused source/target misclassifications, such as making a vehicle image classify as an ostrich.
B. Adversarial Goals: Fooling images are noise-filled inputs unrecognizable to humans but labeled as recognizable objects by DNNs with high confidence.Unlike source/target attacks, these images do not have a source class and are crafted solely to perform a target classification.
C. Adversarial Capabilities: The paper considers only test-time attacks and orders adversaries by decreasing strength and increasing attack difficulty.Tampering with the training procedure is outside the paper’s scope.
C. Adversarial Capabilities: A training-data-and-architecture adversary has perfect knowledge of the classifier, including training data, training algorithms, architecture, activations, weights, and biases.
C. Adversarial Capabilities: A network-architecture adversary knows the DNN architecture and parameter values, enough to simulate the network; the paper’s algorithms assume this threat model.
C. Adversarial Capabilities: Oracle adversaries query a neural network or proxy for classifications and adaptively use input-output differences to craft adversarial samples.Sample-based adversaries instead collect input-output pairs without modifying inputs, making the data useful chiefly in very large quantities.

III. APPROACH · A. Studying a Simple Neural Network

The approach uses knowledge of a DNN’s architecture and trained parameters to craft targeted adversarial samples, illustrated through a simple AND-network. Its forward derivative identifies how small input changes can produce large output shifts and narrows the search to susceptible input regions.

III. APPROACH: III. APPROACH: The algorithm crafts samples that induce any adversarial DNN output using knowledge of architecture and weight parameters.The method targets acyclic feedforward DNNs and does not require the training dataset.
A. Studying a Simple Neural Network: A. Studying a Simple Neural Network: A low-dimensional network trained on the AND function exposes how forward derivatives guide adversarial-sample construction.The network uses inputs X = (x1, x2) and desired output F(X) = x1 ∧x2 with null input biases.
A. Studying a Simple Neural Network: A. Studying a Simple Neural Network: The adversary seeks a minimally perturbed X* = X + δX whose output changes from Y to a desired Y* ≠ Y.The perturbation is measured with a norm appropriate to the input domain; optimization, heuristics, and brute force are difficult to apply directly to deep networks.
A. Studying a Simple Neural Network: A. Studying a Simple Neural Network: The forward derivative is the Jacobian of the learned function F, and its components are computable from the adversary’s knowledge.For the one-dimensional output in this example, the Jacobian reduces to a vector.
A. Studying a Simple Neural Network: A. Studying a Simple Neural Network: A change of δx2 = 0.05 from X = (1, 0.37) to X* = (1, 0.43) changes the output from F(X) = 0.11 to F(X*) = 0.95.After rounding to match the Boolean AND function, X* = (1, 0) and F(X*) = 1, making X* adversarial.
A. Studying a Simple Neural Network: A. Studying a Simple Neural Network: Small input variations can cause extreme output changes, while regions near inputs close to 0 are less conducive to adversarial manipulation.The forward derivative is small when either input is close to 0, reducing the adversarial-sample search space.

B. Generalizing to Feedforward Deep Neural Networks · 1) Forward Derivative of a Deep Neural Network:

The paper generalizes its adversarial-sample construction method to acyclic feedforward DNNs with differentiable activations by computing input-to-output forward derivatives. These derivatives guide a saliency-based iterative process that modifies selected input features toward a target output.

B. Generalizing to Feedforward Deep Neural Networks: The generalization applies to any acyclic feedforward DNN whose neurons use differentiable activation functions.The differentiability requirement matches that imposed by back-propagation.
B. Generalizing to Feedforward Deep Neural Networks: Algorithm 1 takes benign sample X, target output Y∗, network F, maximum distortion Υ, and feature variation θ, then returns X∗ satisfying F(X∗) = Y∗.The algorithm iterates while the target output is unmet and the distortion remains below Υ.
B. Generalizing to Feedforward Deep Neural Networks: The construction repeats three steps: compute ∇F(X∗), build saliency map S, and modify the highest-ranked input feature imax by θ.The selected feature is imax = arg max_i S(X, Y∗)[i].
1) Forward Derivative of a Deep Neural Network:: The forward derivative is the Jacobian of the learned network function, differentiated with respect to input features rather than network parameters.Unlike backpropagation, the method propagates gradients forward to identify input components that strongly affect outputs.
1) Forward Derivative of a Deep Neural Network:: The derivation expresses each output-to-input derivative recursively by applying the chain rule across hidden layers.For matrix element (i, j), the target quantity is the derivative of output neuron Fj with respect to input dimension xi.
1) Forward Derivative of a Deep Neural Network:: The network representation supports fully or sparsely connected interlayers through weight matrices, with biases and layer-specific activation functions.Neuron p in layer k connects to the previous layer using weights Wk,p and bias bk,p.
1) Forward Derivative of a Deep Neural Network:: For any input X, ∇F is computed by successively differentiating layers from the input layer through the output layer.Recursive derivatives provide the remaining term needed in the output-neuron derivative expression.

2) Adversarial Saliency Maps:

Adversarial saliency maps adapt visualization-based saliency maps to identify input features whose perturbation most efficiently drives a network toward an adversary’s specified output. For classification, the map uses forward derivatives to increase a target class while decreasing competing classes, with a counterpart for decreasing features.

Map construction: Adversarial saliency maps identify which input features to perturb to effect desired network-output changes efficiently and generate broad classes of adversarial samples.They extend saliency maps originally introduced as visualization tools.
Map construction: For classifiers, adversarial saliency maps are defined around output probabilities, where the predicted class is the component with the highest probability.The map is designed according to the adversary’s problem-specific goal.
Targeted misclassification: The classifier map uses forward derivatives to increase target-class probability and decrease all other class probabilities until the target becomes the predicted class.The target class t differs from the sample’s original label, and misclassification occurs when t = arg max_j F_j(X).
Targeted misclassification: High saliency values mark features that substantially increase the target class, decrease competing classes, or both, guiding perturbations toward misclassification.The map combines derivative information across classes so feature scores can be compared.
Alternative maps: Alternative forward-derivative maps can change the distortion introduced by Algorithm 1, including a counterpart that identifies features to decrease for misclassification.The counterpart differs in the constraints imposed on forward derivatives.

3) Modifying samples:

After adversarial saliency identifies an input feature, the algorithm perturbs it to achieve the adversary’s goal. The perturbation amount is problem-specific, while the iteration limit controls the maximum distortion and number of changed features.

Modifying samples:: Adversarial saliency maps identify input features to perturb, completing the final step needed to realize the adversary’s goal.The selected feature is then modified during each iteration of Algorithm 1.
Modifying samples:: The perturbation amount θ is problem-specific and must be set according to the application.The paper discusses setting θ for computer vision in Section IV.
Modifying samples:: The maximum iteration count Υ specifies the maximum distortion allowed and limits how many features can change in crafting an adversarial sample.Υ can take any positive integer value smaller than the unspecified upper bound in the supplied passage.

IV. APPLICATION OF THE APPROACH · A. Crafting algorithm

The approach is applied to handwritten-digit recognition using LeNet, demonstrating targeted adversarial crafting from any source digit to any target digit. Algorithm 2 iteratively selects and modifies pixels using saliency maps, subject to distortion and perceptual constraints.

IV. APPLICATION OF THE APPROACH: The authors apply their forward derivative, adversarial saliency maps, and crafting algorithm to a DNN for handwritten-digit recognition.The application tests whether the theoretical framework can effectively produce samples misclassified by the network.
IV. APPLICATION OF THE APPROACH: The method successfully crafts adversarial samples from any source digit class to any specified target digit class.For example, an image classified as 0 can be crafted toward target class 7.
IV. APPLICATION OF THE APPROACH: The evaluated network uses LeNet for handwritten-digit classification, a relevant architecture because later convolutional designs build on its convolutional layers.The authors state that they have no reason to believe the method will not perform well on larger architectures.
IV. APPLICATION OF THE APPROACH: The input consists of 28×28 black-and-white images represented by 784 normalized pixel-intensity features.The network processes these features through convolutional, pooling, fully connected, and softmax output layers.
A. Crafting algorithm: Algorithm 2 iteratively modifies two pixel intensities selected by a saliency map, rebuilding and updating the map between iterations.The selected features are changed by θ while the algorithm searches for the target classification.
A. Crafting algorithm: Maximum distortion Υ limits the percentage of pixels modified and therefore determines the maximum number of iterations.Because two pixels are modified per iteration, the distortion constraint controls the search budget over the 784-pixel input.
A. Crafting algorithm: The saliency-map policy determines which input features are modified and varies with the data type and adversarial objective.The feature variation θ must be chosen consistently with the saliency map used.
A. Crafting algorithm: Human perception limits the acceptable maximum distortion, making perceptual visibility a constraint on the crafting process.The authors identify selecting suitable distortion and feature-variation parameters as an evaluation goal discussed later.

B. Crafting by increasing pixel intensities

This strategy crafts targeted MNIST adversarial samples by increasing selected pixel intensities. Using pairwise saliency maps, it succeeds for every source-target class pair while selecting features relevant to the target class.

Setup: The experiment uses 10 MNIST test samples, one from each handwritten digit class 0 through 9, to illustrate increasing-intensity crafting.The evaluation is later scaled to the entire dataset in Section V.
Setup: With θ = +1 and unlimited distortion Υ = ∞, the method tests whether each source class can reach every target class across 90 source-target pairs.For each source image, pixel intensities are increased to generate samples targeting the other nine classes.
Saliency map: The saliency heuristic selects pixel pairs whose simultaneous increase raises the target-class output while reducing the summed outputs of all other classes.Pairs are used because one pixel can compensate for a minor flaw in the other, making the heuristic easier to satisfy than single-pixel selection.
Results: The algorithm successfully crafts adversarial samples for all 90 source-target class pairs.The resulting 90 samples are arranged by source and target class, with original samples on the diagonal.
Results: Using an empty input with all pixels initially at intensity 0, the method produces one adversarial sample for each class and identifies features relevant to classification.The samples correspond to target classes 0 through 9.

C. Crafting by decreasing pixel intensities · V. EVALUATION · A. Crafting large amounts of adversarial samples

The paper decreases pixel intensities to craft adversarial samples across all source-target pairs, then evaluates reliability, distortion, and vulnerability at larger scale. Decreasing intensities achieves lower success while slightly reducing distortion.

C. Crafting by decreasing pixel intensities: Decreasing pixel intensities with θ = −1 crafts adversarial samples across all source-target class pairs.The heuristic decreases selected pixel pairs to increase the target output while reducing the outputs of other classes.
C. Crafting by decreasing pixel intensities: The resulting distortion appears harder for humans to detect than distortions produced by increasing pixel intensities.Human perception is evaluated later in Section V.
V. EVALUATION: 97.10% success rate and 4.02% average modification establish the primary validation result for reliably crafting adversarial samples.The evaluation also defines a hardness measure for identifying sample classes that are easier to exploit.
A. Crafting large amounts of adversarial samples: The large-scale experiment tests whether all legitimate MNIST samples can be exploited by running the crafting algorithm on three sets of 10,000 samples.The experiment uses distinct handwritten-digit samples.
A. Crafting large amounts of adversarial samples: Success rate τ measures target-class classification, whereas distortion measures the percentage of pixels modified from the legitimate input.Distortion is equivalently defined as the percentage of modified input features.
A. Crafting large amounts of adversarial samples: 64.7% success rate τ and 3.62% average distortion ε were obtained for 9,000 samples using the decreasing saliency map.The decreasing strategy was less successful but had slightly lower average distortion than increasing pixel intensities.
A. Crafting large amounts of adversarial samples: Removing pixels may hinder classification because it reduces information entropy and makes necessary information harder for DNNs to extract.This provides the paper’s intuitive explanation for the lower success of decreasing pixel intensities.

B. Quantifying hardness and building defense mechanisms · 1) Class pair study: · 2) Hardness measure:

The study quantifies why adversarial crafting varies across source-target class pairs and introduces hardness as a distortion- and success-rate-based measure. Class-pair success, distortion, and probability trajectories reveal that source-class confidence drives much of the observed difficulty.

B. Quantifying hardness and building defense mechanisms: 2.9% of 270,000 adversarial samples were not successfully crafted, motivating analysis of unequal sample and class-pair hardness.Reported distortion averages also obscure that samples require different distortions to be misclassified.
1) Class pair study:: 90,000 adversarial samples from the MNIST test set were used to study crafting success rates and average distortions across source-target class pairs.The samples originated from 10,000 MNIST test examples.
1) Class pair study:: The success-rate matrix identifies, for each source class, how many samples are successfully misclassified into each target class.Rows represent source-class success rates, columns represent target-class success rates, and darker shades indicate higher success rates.
1) Class pair study:: Higher average distortion corresponds to lower success rates, with class 1 associated with the highest distortions and least success rates.Pairs with higher average distortion are more likely to reach the maximum distortion and yield unsuccessful adversarial samples.
1) Class pair study:: High-distortion class pairs require more distortion to leave the source class, while the distortion needed to reach the target class remains similar after departure.This pattern correlates with some source classes being more confidently classified by the DNN than others.
2) Hardness measure:: Hardness measures the distance between target and source classes by normalizing a class pair’s average distortion relative to its success rate.The measure is intended to help defenders identify which DNN classes are most vulnerable to adversaries.
2) Hardness measure:: The practical hardness formula uses K maximum-distortion settings to generate (ε_k, τ_k) pairs and derives the result with the trapezoidal rule.Here, ε(s, t, τ) denotes average distortion for samples at success rate τ.
2) Hardness measure:: K = 9 maximum distortion values ranging from 0.3% to 38.3% were evaluated, averaging distortions and success rates over 9,000 samples per setting.Hardness values H(s, t) were computed for all source-destination class pairs, producing a matrix similar in shape to the average-distortion matrix.

3) Adversarial distance: · C. Study of human perception of adversarial samples

The paper introduces adversarial distance as a predictive measure of sample vulnerability and proposes extending it to network robustness. Human-perception experiments show that generated MNIST adversarial samples generally remain recognizable as digits, including below a 14.29% distortion threshold.

3) Adversarial distance:: Adversarial distance A(X, t) estimates the distance between sample X and target class t.The measure is intended to predict vulnerability for individual samples and class pairs before adversarial crafting.
3) Adversarial distance:: A(X, t) is the normalized number of non-zero elements in X’s adversarial saliency map from the first crafting iteration.Values closer to 1 indicate that X is more likely to be harder to misclassify into target class t.
3) Adversarial distance:: The proposed distance between classes defines an intuitive metric for a network’s robustness against adversarial perturbations.The robustness approximation can use a sufficiently large sample set representing the network’s input domain, including the training dataset.
C. Study of human perception of adversarial samples: 349 human participants evaluated original and adversarially altered MNIST samples to assess whether adversarial examples remained visually classified as their source digits.Participants were shown three samples and asked questions about digit identification and classification.
C. Study of human perception of adversarial samples: 97.4% of unaltered samples were identified as digits, and 95.3% were classified correctly in the baseline experiment.The baseline used 74 participants and 222 original MNIST samples, with each participant seeing three randomly selected samples.
C. Study of human perception of adversarial samples: ε = 14.29% distortion marked a threshold below which participants identified 95% of samples as digits and correctly classified 90%.The distortion experiments involved 184 participants, 1707 samples, and intensity increase θ = +1.
C. Study of human perception of adversarial samples: At θ = −1 and θ = +1, correct digit classification decreased to 90.5% and 90%, while digit identification remained largely unchanged.Across 5,355 samples, participants identified 96% as digits and classified 95% correctly overall.
C. Study of human perception of adversarial samples: ε ≤14.29% distortion was achievable for roughly 97% of MNIST data, enabling adversarial samples that humans would mis-interpret while retaining recognizability.The experiments were preliminary, but the authors report that the overwhelming number of generated samples remained human-recognizable.

VI. DISCUSSION · VII. RELATED WORK · VIII. CONCLUSIONS

The paper presents architecture-aware algorithms for crafting low-distortion adversarial samples, extends the discussion to defense strategies and related attacks, and identifies open problems in adversarial deep learning. Its proposed defenses center on adversarial-sample detection and improving DNN training, while detection remains unresolved and broader settings require further study.

VI. DISCUSSION: The authors introduce algorithms that systematically craft DNN-misclassified adversarial samples when the adversary knows the network architecture.The approach also applies to unsupervised architectures by targeting an output Y* rather than a class.
VI. DISCUSSION: Reducing feature distortion makes adversarial crafting easier in difficult-to-perturb domains such as malware executables.This reduction requires more expensive saliency-map computations, creating a performance cost for the attacker.
VI. DISCUSSION: The paper proposes adversarial-sample detection and improved DNN robustness as two defense classes based on hardness and adversarial-distance metrics.Detection is described as reactive, while robustness improvements target the training process.
VI. DISCUSSION: Adding adversarial samples to training can act as a regularizer, while generative adversarial networks can augment training datasets with new samples.These observations connect adversarial samples to the training process.
VII. RELATED WORK: Prior work established taxonomies of machine-learning attacks and defenses, studied adversarial classifiers, and developed DNN attacks by optimizing inputs through backpropagation gradients.The cited classifier framework did not consider DNNs, focusing instead on other binary-classification techniques.
VII. RELATED WORK: Earlier studies found that adversarial samples generated for one neural-network architecture are likely to be misclassified by differently designed networks.This cross-model effect was linked to transferable features between DNNs.
VIII. CONCLUSIONS: The paper’s conclusion highlights forward derivatives and adversarial saliency maps for identifying input features that most affect output classification.These methods support adversarial-sample construction for an adversary with architecture knowledge.
VIII. CONCLUSIONS: Adversarial-sample detection remains an open problem, and future work will examine unsupervised DNNs, cyclical recurrent networks, and additional adversarial settings.The authors argue that improving training is key to resisting adversarial samples.

APPENDIX · A. Validation setup details

The validation setup uses Theano with GPU-accelerated float32 computation to implement and train a simplified LeNet-5 network on MNIST. After 200 epochs, the network achieves 98.93% training accuracy and 99.41% test accuracy.

A. Validation setup details: Theano implements the network architecture, backpropagation training, and forward derivative computation.The package is used to simplify large-scale scientific computing.
A. Validation setup details: float32 precision enables GPU acceleration for the experiments.The computations are configured in float32 because this precision can be accelerated using graphics processors.
A. Validation setup details: The experiments use a deep neural network with simplifications to the original LeNet-5 architecture.The simplifications are suggested by the Theano Documentation.
A. Validation setup details: The network is trained on MNIST batches containing 500 samples.The MNIST dataset is used for training the network.
A. Validation setup details: 200 epochs of training use a learning parameter of η = 0.1.These settings produce the reported learned network parameters.
A. Validation setup details: 98.93% training accuracy and 99.41% test accuracy are achieved on MNIST.The reported accuracies are described as comparable to state-of-the-art accuracies.