PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Examples: Attacks and Defenses for Deep Learning

Xiaoyong Yuan, Pan He, Qile Zhu, Xiaolin Li

arXiv:1712.07107v3cs.LGcs.CRcs.CVstat.ML

TL;DR

Deep learning’s safety-critical deployment is challenged by adversarial examples: small, often imperceptible input changes can fool deep neural networks. This paper reviews attack-generation methods and defenses, organizes them with a taxonomy, examines applications, and discusses challenges and potential solutions.

  • Problem

    Adversarial examples can use small, human-imperceptible perturbations to fool deep neural networks, creating a concern for safety-critical deployment.

  • Method

    The paper reviews adversarial-example generation methods, proposes a taxonomy, investigates applications, and discusses countermeasures and challenges.

  • Results

    The review covers attack methods, applications, countermeasures, and challenges concerning adversarial-example transferability, existence, and robustness evaluation.

  • Takeaways & Limitations

    Adversarial examples are examined both as attacks against deployed models and as a means to study neural-network decision boundaries and robustness.

Abstract

from arXiv · show

With rapid progress and significant successes in a wide spectrum of applications, deep learning is being applied in many safety-critical environments. However, deep neural networks have been recently found vulnerable to well-designed input samples, called adversarial examples. Adversarial examples are imperceptible to human but can easily fool deep neural networks in the testing/deploying stage. The vulnerability to adversarial examples becomes one of the major risks for applying deep neural networks in safety-critical environments. Therefore, attacks and defenses on adversarial examples draw great attention. In this paper, we review recent findings on adversarial examples for deep neural networks, summarize the methods for generating adversarial examples, and propose a taxonomy of these methods. Under the taxonomy, applications for adversarial examples are investigated. We further elaborate on countermeasures for adversarial examples and explore the challenges and the potential solutions.

I. INTRODUCTION

Deep learning’s broad deployment, including safety-critical applications, is accompanied by vulnerability to imperceptible inputs that can fool models. This paper surveys adversarial-example attacks and countermeasures, organizes attacks taxonomically, and identifies open challenges.

  • Deep learning supports diverse applications, from vision and speech to translation, synthesis, and game playing.
  • Safety-critical deployment raises security concerns because small, human-imperceptible perturbations can fool otherwise well-performing deep learning models.
  • The review mainly examines adversarial examples for deep neural networks in image classification and object recognition, while noting other tasks for future investigation.
  • Its threat model restricts adversaries to tampering with test-time inputs after training, without modifying the trained model or dataset.
  • The paper taxonomizes attack approaches, surveys variants and applications, discusses countermeasures, and outlines challenges involving transferability, existence, and robustness evaluation.

2) Architectures of deep neural networks:

The paper situates deep-learning adversarial examples within earlier machine-learning security work and formalizes their generation as constrained perturbation of an input. It then organizes attacks by threat model, perturbation, and benchmark, including dimensions such as knowledge, specificity, and falsification.

  • 1) Main concepts in deep learning:: Earlier machine-learning attacks targeted systems such as spam filters, intrusion detection, biometric authentication, and fraud detection using handcrafted features.
  • 1) Main concepts in deep learning:: Deep-learning adversarial examples differ because attacks generally operate from raw inputs rather than requiring knowledge of feature extraction.
  • B. Adversarial Examples and Countermeasures in Machine Learning: The review focuses on deep learning and provides a detailed discussion of recent adversarial-example studies relative to earlier work.
  • B. Adversarial Examples and Countermeasures in Machine Learning: Given a trained model f and input x, an adversarial example x′ is obtained by minimizing perturbation while changing the predicted label under input constraints.
  • B. Adversarial Examples and Countermeasures in Machine Learning: The proposed taxonomy categorizes generation methods by threat model, perturbation, and benchmark.
  • A. Threat Model: Threat-model dimensions include falsification, adversary knowledge, specificity, and attack frequency, distinguishing targeted from non-targeted and white-box from black-box settings.

B. Perturbation

Adversarial examples rely on small, human-imperceptible perturbations, which can be characterized by scope, optimization limitation, and measurement. Evaluation also varies across datasets and victim models, complicating comparisons of attack effectiveness and model robustness.

  • Perturbation premise: Adversarial examples are designed to remain close to clean inputs and imperceptible to humans while degrading deep-learning performance.The paper analyzes perturbation scope, limitation, and measurement.
  • Perturbation Scope: Individual attacks generate input-specific perturbations, whereas universal attacks create one perturbation applicable across a dataset.Universal perturbations can simplify real-world deployment because they need not change when the input changes.
  • Perturbation Limitation: Optimized perturbation methods minimize perturbation during optimization, while constraint perturbation methods require only that perturbation remain sufficiently small.
  • Perturbation Measurement: ℓ0 counts changed pixels, ℓ2 measures Euclidean distance, and ℓ∞ measures the maximum pixel change.PASS is introduced as a perceptually aligned metric.
  • Evaluation: Variation in datasets and victim models creates obstacles for evaluating attacks and measuring deep-learning robustness.MNIST, CIFAR-10, and ImageNet are common evaluation datasets, while ImageNet is described as the strongest current choice among them.

IV. METHODS FOR GENERATING ADVERSARIAL EXAMPLES

The paper reviews representative methods for generating adversarial examples and organizes them under a proposed taxonomy. The methods progress from optimization-based attacks to faster gradient, momentum, targeted, and randomized variants.

  • Overview: The section presents representative adversarial-example generation methods to show attack improvements and motivate robustness investigations.Table II summarizes the methods according to the proposed taxonomy.
  • Optimization-based methods: L-BFGS generated targeted adversarial examples through an optimization problem and line search over c, but its expensive search was time-consuming and impractical.The method was later implemented with binary search to find an optimal c.
  • Gradient-based methods: Fast Gradient Sign Method performs one gradient update using the sign of the gradient at each pixel, avoiding L-BFGS’s expensive linear search.The perturbation magnitude is controlled by ϵ.
  • Gradient-based methods: Fast Gradient Value replaces the gradient sign with the raw gradient, allowing larger local differences without per-pixel constraints.
  • Attack variants: Momentum and ensembling improve the effectiveness and transferability of iterative gradient attacks, while targeted variants maximize the target-class probability.RAND-FGSM adds randomness to updates to defeat adversarial training affected by gradient masking.
  • Attack variants: OTCM is the name used for the one-step targeted attack described in the section.

C. Basic Iterative Method (BIM) and Iterative Least-Likely Class Method (ILLC)

BIM and ILLC extend FGSM to iterative, clipped updates that can generate physical-world adversarial examples. The section also reviews saliency-, boundary-, and confidence-based attacks and their trade-offs.

  • Basic Iterative Method and ILLC: BIM applies multiple smaller FGSM-like updates and clips pixel values at every iteration to limit per-pixel changes.ILLC targets the least-likely predicted class by maximizing cross-entropy loss.
  • Basic Iterative Method and ILLC: BIM successfully fooled a neural network using an image captured by a cellphone camera, while iterative methods were less resistant to phototransformation than FGSM.
  • JSMA: JSMA computes a Jacobian-based saliency map to select features or pixels for modification during each attack iteration.The attack uses output sensitivity to identify influential input features.
  • JSMA: 97% adversarial success was achieved by modifying 4.02% of input features per sample, but JSMA was very slow because of its computational cost.
  • DeepFool: DeepFool iteratively linearizes the classifier to approximate the smallest perturbation reaching a decision boundary.It produced less perturbation than FGSM and JSMA, while reducing perturbation intensity rather than the number of selected features compared with JSMA.
  • False-positive examples: False-positive adversarial examples can be unrecognizable to humans while receiving neural-network confidence of at least 99.6%.

F. CPPN EA Fool

This section covers CPPN EA false-positive attacks, C&W attacks against defensive distillation, norm-specific objectives, and gradient-free ZOO attacks. The methods differ in confidence, perturbation norm, model access, and computational cost.

  • CPPN EA Fool: CPPN EA uses evolutionary algorithms and MAP-Elites to generate human-unrecognizable adversarial examples classified with 99% confidence.Images are encoded directly or through compositional pattern-producing networks.
  • C&W’s Attack: C&W introduced targeted attacks against defensive distillation and reported effectiveness against many existing adversarial-example detection defenses.
  • C&W’s Attack: C&W formulates attacks around a distance term and an objective function g whose nonnegative value corresponds to the target classification condition.The formulation includes a confidence-control constant κ.
  • C&W’s Attack: C&W’s optimization can fail to find optimal adversarial solutions when the perturbation and classification gradients are on different scales.
  • Norm-specific attacks: The C&W variants use ℓ0, ℓ2, and ℓ∞ distance measurements, with iterative procedures tailored to each norm.The ℓ0 attack removes pixels iteratively, while the ℓ∞ attack updates a penalty threshold τ.
  • ZOO: ZOO estimates gradients and Hessians through symmetric difference quotients, enabling black-box attacks without victim-model gradients but requiring expensive queries.Experiments found performance comparable to C&W’s Attack.
  • Universal perturbations: Universal perturbations are accumulated from per-input DeepFool perturbations and can fool groups of images across popular deep-learning architectures.They can be generated from a small part of the data rather than the entire dataset.

I. Universal Perturbation

The reviewed attacks generate adversarial examples through universal, pixel-level, feature-space, perceptual, and generative objectives. These methods vary in perturbation constraints, optimization procedures, and similarity measures.

  • Universal perturbation: Universal perturbations iteratively accumulate minimal DeepFool perturbations until most samples are fooled, using only a small subset of data.The perturbation is bounded by ϵ, while δ controls the allowed failure rate.
  • Pixel-level attacks: One-pixel attacks constrain modifications to a single pixel and use differential evolution without requiring model gradients.The constraint ϵ0 = 1 makes optimization difficult.
  • Feature-space attacks: Feature Adversary targets internal-layer representations rather than output labels, producing images closer to the target in feature space.L-BFGS-B solves the constrained optimization problem.
  • Perceptual similarity: PASS combines homography alignment with regional structural similarity, and the resulting distance can replace standard ℓp distance in adversarial optimization.The similarity weights correspond to luminance, contrast, and structure.
  • Perceptual similarity: Hot/Cold generates multiple diverse adversarial examples by moving toward a hot target class and away from the cold original class.Its results were comparable to FGSM, while PASS replaced pixel distance with a human-perceptual similarity measure based on alignment and image similarity.

M. Natural GAN

Natural GAN uses generative modeling and inversion to create adversarial examples that appear more natural while fooling target models. The framework supports multiple modalities and black-box attacks.

  • Natural GAN: Natural GAN trains a WGAN generator and an inverter so adversarial noise is produced in a learned data domain.The generator maps random noise to the input domain, while the inverter maps inputs to dense internal representations.
  • Natural GAN: The method enforces a misclassification objective while optimizing adversarial examples generated through the learned generator.Its constraint requires f(G(z)) ≠ f(x).
  • Applications: Natural GAN was applied to image classification, textual entailment, and machine translation, and it does not require gradients of the original networks.This enables its use in black-box attack settings.
  • Model-based Ensembling Attack: Model-based Ensembling Attack generates examples on multiple fully known networks and tests them against a black-box model.The ensemble uses network-specific functions and weights whose sum is one.
  • Model-based Ensembling Attack: The ensemble attack generated transferable targeted images, improved black-box attack power, outperformed previous methods for non-targeted examples, and attacked Clarifai.com.The paper motivates the method by noting that targeted examples are harder to transfer across deep models.

A. Reinforcement Learning

Adversarial examples are reviewed across reinforcement learning, generative models, face recognition, and physical sign attacks. The section emphasizes task-specific objectives and constraints, with varying attack success and robustness.

  • A. Reinforcement Learning: Fast one-time attacks were used against reinforcement-learning policies because reinforcement learning has intensive computational requirements.FGSM-based attacks were evaluated on DQN, TRPO, and A3C policies.
  • A. Reinforcement Learning: An ℓ1-constrained attack succeeded in both white-box and black-box reinforcement-learning settings across four Atari 2600 games.The black-box setting provided no access to training algorithms, parameters, or hyperparameters.
  • A. Reinforcement Learning: Injecting perturbations into only a fraction of frames was sufficient to attack the A3C algorithm on Atari Pong.
  • B. Generative Modeling: Feature Adversary attacks on autoencoders and variational autoencoders found adversarial-example generation much harder than for classifiers.The variational autoencoder was slightly more robust than the deterministic autoencoder.
  • C. Face Recognition: Physical eyeglass-frame attacks restricted perturbations to the frame area and added a non-printability penalty for printed attacks.They dodged face recognition systems over 80% of the time in non-targeted attacks and achieved high targeted success depending on the target.
  • Physical attacks: Robust Physical Perturbations modified physical stop signs through overlays or attached perturbations while optimizing printability.The cited footnote reports that this method was not effective for standard YOLO and Faster RCNN detectors.

D. Object Detection

The paper extends adversarial-example generation to object detection, semantic segmentation, NLP, and malware detection. These attacks adapt objectives to structured outputs, words, files, or detector features.

  • D. Object Detection: Dense Adversary Generation treats detection and segmentation as collections of recognition targets and iteratively optimizes losses across those targets.Targets are proposals for detection and pixels for segmentation, with the objective seeking incorrect predictions.
  • E. Semantic Segmentation: Semantic-segmentation attacks can target selected classes for removal while preserving the remaining segmentation, including hiding pedestrians.Universal perturbations were shown to exist for this task.
  • F. Natural Language Processing (NLP): NLP attacks modify text by adding or deleting words, including distracting sentences for reading comprehension and minimum-word removal for sentiment classification.AddSent and AddAny successfully fooled all sixteen tested SQuAD models.
  • G. Malware Detection: Adversarial malware examples alter Android applications, PDF object structures, domain names, or program API features to evade deep-learning detectors.MalGan uses a substitute detector and transferability to attack a real black-box detector but requires knowledge of the model’s features.

VI. COUNTERMEASURES FOR ADVERSARIAL EXAMPLES

The paper organizes defenses into reactive strategies that detect or reconstruct adversarial inputs and proactive strategies that improve model robustness before attacks. Reported results show that effectiveness is attack-dependent, with some methods improving robustness but failing against stronger or altered attacks.

  • Defenses comprise reactive detection, input reconstruction, and network verification, alongside proactive distillation, adversarial retraining, and classifier robustifying.
  • Network Distillation: Network distillation uses a temperature-controlled softmax output from one network as knowledge for training another.High temperature produces softer class probabilities and reduces sensitivity to small perturbations.
  • Network Distillation: 0.5% and 5%: network distillation reduced JSMA success rates on MNIST and CIFAR-10, respectively, while improving generalization.
  • Adversarial (Re)training: Adversarial training injects generated adversarial examples into each training step and improves robustness for some attack settings.On ImageNet, training increased robustness to one-step attacks but did not help against iterative attacks.
  • Adversarial (Re)training: Adversarially trained models were more robust to white-box examples than to transferred black-box examples.Ensembling adversarial training uses examples from the trained model and external pretrained models to address transferred attacks.
  • Adversarial Detecting: Detection methods include binary auxiliary classifiers, uncertainty estimation, probability divergence, latent-space statistics, PixelCNN p-values, and predictive models.PixelCNN-based detection reported detection of FGSM, BIM, DeepFool, and C&W attacks.
  • Adversarial Detecting: Carlini and Wagner reported that several detectors failed against a slightly modified C&W attack.
  • Input Reconstruction: Reconstruction defenses transform adversarial inputs toward clean or training-distribution data, including through autoencoders and PixelCNN.PixelDefend combines reconstruction with detection and leaves an input unchanged when it is not detected as malicious.

E. Classifier Robustifying

Classifier robustifying modifies model architecture or decision behavior to improve resistance to adversarial examples. The reviewed approaches include uncertainty-aware Bayesian components, subclass voting, verification, and combined defenses, but ensemble defenses were not universally strong.

  • Classifier Robustifying: Robustifying approaches design neural-network architectures intended to prevent adversarial examples.
  • Classifier Robustifying: GPDNNs combine Bayesian classifiers and Gaussian-process uncertainty estimation to make latent representations uncertainty-aware.
  • Classifier Robustifying: GPDNNs achieved performance comparable to general DNNs while being more robust to adversarial examples.The authors characterize this behavior as the models knowing when they do not know.
  • Classifier Robustifying: Subclass voting separates classes into subclasses and ensembles their outputs to address adversarial examples that concentrate in a small subset of incorrect classes.
  • Network Verification: Network verification checks whether an input satisfies or violates a neural-network property and can detect previously unseen attacks.
  • Network Verification: Reluplex uses an SMT solver to verify ReLU networks and showed that sufficiently small perturbations contained no misclassifying adversarial example under its setting.The paper also reports that network verification is NP-complete.
  • Network Verification: DeepSafe uses Reluplex to define safe regions, including targeted robustness regions for a specified class.
  • Combined Defenses: PixelDefend and MagNet combine adversarial detection with input reconstruction in a sequential defense strategy.MagNet measures input-encoding distance and Jensen-Shannon divergence to identify suspicious inputs before reforming them.

B. The existence of Adversarial Examples

The paper presents adversarial-example existence as an unresolved question with competing explanations involving data, model capability, decision boundaries, and manifolds. It also identifies broader limits in robustness evaluation, benchmarking, application coverage, and survey scope.

  • The existence of Adversarial Examples: Whether adversarial examples are inherent to deep neural networks remains an open question with multiple proposed explanations.
  • The existence of Adversarial Examples: The data-incompletion hypothesis attributes adversarial examples to low-probability cases and limited corner-case coverage in testing data.PixelCNN results distinguished adversarial from clean-data distributions, while robust models may require substantially more training data.
  • The existence of Adversarial Examples: Model-capability explanations attribute adversarial examples variously to high-dimensional linearity, limited classifier flexibility, or sparse and discontinuous manifolds.
  • The existence of Adversarial Examples: Other work argues that incorrect decision boundaries or properties of smooth generative and sphere datasets can make small-perturbation adversarial examples unavoidable.
  • Applications and Evaluation: The survey notes that different applications use different attack methods, and no universal attack or defense method has been established across applications.
  • Robustness Evaluation: The attack-defense arms race means defenses against existing attacks can fail after new attacks or slight attack modifications appear.
  • Robustness Evaluation: Robustness evaluation needs methods that address zero-day attacks, especially for models planned for safety-critical deployment.The paper connects this need to confidence in predictions and reliance on models in the real world.
  • Benchmarking: A benchmark platform is proposed in which attackers and defenders train strategies, generate and verify examples, and evaluate defensive strategies.Existing tools benchmark attacks but lack defensive strategies and standardized parameters can hinder reproducibility.
Loading 1712.07107v3…