PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Attacks and Defences: A Survey

Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, Debdeep Mukhopadhyay

arXiv:1810.00069v1cs.LGcs.CRstat.ML

TL;DR

Deep-learning systems are vulnerable to adversarial examples that can induce misclassification, while existing defenses do not cover all attack scenarios. This survey synthesizes attacks, threat models, and countermeasures across settings, concluding that robust protection against all adversarial examples remains an open problem.

  • Problem

    Deep-learning applications are vulnerable to adversarial attacks, and prior surveys provide limited coverage across applications and attack types.

  • Method

    The paper reviews adversarial attacks, threat models, methodologies, and countermeasures across deep neural networks and selected SVM scenarios.

  • Results

    The survey finds that defenses provide robustness against specific attacks in specific settings rather than all adversarial examples.

  • Takeaways & Limitations

    Designing a robust machine-learning model against all types of adversarial examples remains an open research problem.

  • Takeaways & Limitations

    Defensive distillation and label smoothing can be bypassed by recent black-box attacks, often because adversarial examples transfer across neural-network models.

Abstract

from arXiv · show

Deep learning has emerged as a strong and efficient framework that can be applied to a broad spectrum of complex learning problems which were difficult to solve using the traditional machine learning techniques in the past. In the last few years, deep learning has advanced radically in such a way that it can surpass human-level performance on a number of tasks. As a consequence, deep learning is being extensively used in most of the recent day-to-day applications. However, security of deep learning systems are vulnerable to crafted adversarial examples, which may be imperceptible to the human eye, but can lead the model to misclassify the output. In recent times, different types of adversaries based on their threat model leverage these vulnerabilities to compromise a deep learning system where adversaries have high incentives. Hence, it is extremely important to provide robustness to deep learning algorithms against these adversaries. However, there are only a few strong countermeasures which can be used in all types of attack scenarios to design a robust deep learning system. In this paper, we attempt to provide a detailed discussion on different types of adversarial attacks with various threat models and also elaborate the efficiency and challenges of recent countermeasures against them.

1 INTRODUCTION

Deep learning has progressed from a powerful solution for difficult learning problems to a widely deployed technology whose security is threatened by adversarial examples. This survey organizes attack types, threat models, and corresponding defenses across applications.

  • Background: Deep learning uses multiple processing layers to learn hierarchical representations from experience and large datasets.Backpropagation computes each layer’s representation from the preceding layer.
  • Background: Deep learning has advanced applications including image classification, speech recognition, and language translation.
  • Security Motivation: Adversaries can manipulate legitimate inputs imperceptibly to humans while forcing trained models to produce incorrect outputs.The survey relates this vulnerability to the security and integrity of real-world applications.
  • Motivation and Contribution: The survey addresses a gap in prior work by covering adversarial attacks and countermeasures beyond the restrictive context of computer vision.It analyzes different threat models and attack scenarios with practical examples.
  • Organization: The paper reviews attack models and methodologies, including training-time poisoning, testing-time evasion, exploratory attacks, and black-box and white-box settings.It also discusses current defense strategies and concludes with their challenges.

2 TAXONOMY OF MACHINE LEARNING AND ADVERSARIAL MODEL

This section introduces machine-learning architectures and frames adversarial risk through the system’s data-processing pipeline, attack surface, attack phase, and adversary capabilities.

  • Taxonomy: The paper presents a qualitative taxonomy of terms, keywords, threat models, and adversarial capabilities.
  • Learning Models: Deep neural networks learn features from raw data through multiple hidden layers, while convolutional networks use convolution, subsampling, and fully connected layers for structured inputs.CNN pooling reduces feature-map dimensionality while retaining important information for small distortions.
  • Attack Surface: A machine-learning system can be viewed as a pipeline that collects data, transfers it digitally, processes it into an output, and takes an action.
  • Attack Surface: In an automated vehicle example, sensor images become pixel tensors, the model predicts a stop-sign probability, and the system responds by stopping the car.
  • Attack Scenarios: Evasion attacks alter malicious samples during testing, whereas poisoning attacks inject carefully designed samples during training to compromise learning.
  • Attack Scenarios: Exploratory attacks use black-box access to learn about the underlying algorithm and training-data patterns without influencing the training dataset.The adversary’s available information determines the threat model and attack vector.

Training Phase Capabilities.

Training-phase attacks target the learning process by injecting data, modifying training records, or corrupting the learning logic, with increasing direct control over model construction.

  • Training Attack Strategies: Data injection augments the training set with adversarial samples without access to the training data or learning algorithm.
  • Training Attack Strategies: Data modification changes training data directly when the adversary has full access to it but not to the learning algorithm.
  • Training Attack Strategies: Logic corruption allows an adversary to meddle with the learning algorithm and thereby control the model itself.The paper states that designing a counter-strategy becomes very difficult against such adversaries.
  • Boundary with Testing Attacks: Testing-phase attacks do not tamper with the target model but force it to produce incorrect outputs, and their effectiveness depends mainly on adversarial information about the model.

Testing Phase Capabilities.

Testing-time attacks leave the target model unchanged while inducing incorrect outputs, and the paper distinguishes white-box and black-box settings according to adversarial knowledge.

  • Formal Setup: A randomized training procedure produces model parameters θ from training data and labels, with randomness such as initialization or dropout.
  • Threat Models: The paper compares white-box and black-box adversaries by their capabilities and summarizes these threat models in a figure.
  • Threat Models: White-box attacks assume total knowledge of the classification model, including its neural-network type and other model information.

White-Box Attacks.

White-box adversaries know the training algorithm, data distribution, and trained-model parameters, enabling them to identify vulnerable feature spaces. The survey contrasts this setting with black-box attacks, which rely on limited external information or model queries.

  • White-Box Threat Model: White-box adversaries can access the training algorithm, data distribution, and trained-model parameters.This information helps identify feature spaces where the model has high error rates.
  • White-Box Threat Model: White-box attacks exploit identified vulnerabilities by altering inputs through adversarial-example crafting.The supplied passage connects model knowledge with locating vulnerable feature spaces and modifying inputs.
  • Black-Box Contrast: Black-box attacks assume no knowledge of the model and instead analyze vulnerabilities using settings or past inputs.Oracle attacks provide one example by submitting carefully crafted inputs and observing outputs.

Black-Box Attacks.

Black-box attacks vary by the information and query access available to the adversary, from surrogate-model training to restricted input-output collection. Their goals range from reducing confidence to forcing specified classifications, with attack complexity depending on capabilities and goals.

  • Black-Box Categories: Non-adaptive black-box adversaries train a local surrogate using the target training-data distribution, then transfer crafted examples to the target.The surrogate is attacked with white-box strategies before applying its inputs to the target classifier.
  • Black-Box Categories: Adaptive black-box adversaries query the target as an oracle, label selected data, train a surrogate, and use it to produce adversarial samples.The strategy is compared with a chosen-plaintext attack in cryptography.
  • Black-Box Categories: Strict black-box adversaries may collect input-output pairs but cannot modify inputs to observe output changes.They do not seek the target model’s parameters or training randomness.
  • Adversarial Goals: Adversarial goals include confidence reduction, untargeted misclassification, targeted misclassification, and source/target misclassification.These goals differ in whether the attacker lowers confidence, selects any incorrect class, or specifies a target class or source-target mapping.
  • Threat-Model Taxonomy: Figure 5 orders goal complexity horizontally, adversary strength vertically in decreasing order, and attack difficulty along the diagonal.The taxonomy covers both evasion and poisoning attacks.
  • Threat-Model Taxonomy: Tables 2 and 3 organize representative attacks by applications and threat models.The survey uses these categorizations to introduce the detailed attack discussions.

3 EXPLORATORY ATTACKS

Exploratory attacks probe learners without modifying training data, aiming to obtain information while presenting adversarial examples as legitimate test inputs. The survey discusses model inversion, model extraction, and membership inference as representative cases.

  • Exploratory Attacks: Exploratory attacks probe the learner without modifying the training set and craft test inputs that pass as legitimate.Their purpose is to gain information about the learner’s state during testing.
  • Model Inversion: Model inversion can infer sensitive features or reconstruct facial images from model access and auxiliary information.Reported settings include white-box regression and black-box machine-learning APIs.
  • Model Inversion: Deep-learning model inversion may recover prototypical examples that poorly resemble the actual data defining a class.This limitation is attributed to the rich structure of deep-learning machines.
  • Model Extraction: Model extraction attacks build local models functionally close to target models using strict black-box access.The survey reports demonstrations against BigML and Amazon Machine Learning services.
  • Membership Inference: Membership inference determines whether a data point belongs to the target model’s training distribution.Attack models are trained using outputs from shadow models on member and non-member records.

4 EVASION & POISONING ATTACKS

Evasion attacks modify inputs during testing, whereas poisoning attacks contaminate training data. The survey reviews GAN-based generation and white-box crafting procedures, including sensitivity estimation, perturbation selection, and optimization challenges.

  • Evasion and Poisoning: Evasion attacks modify malicious inputs during testing to induce false predictions, while poisoning attacks alter training inputs to obtain a desired output.The distinction is based on whether modification occurs at testing or training time.
  • Generative Adversarial Networks: GANs combine a discriminator that distinguishes real from generated samples with a generator initialized from random noise.The generator aims to produce samples resembling the training distribution.
  • Generative Adversarial Networks: GAN training maximizes discriminator correctness while training the generator to minimize that objective.Because minimizing log(1 − D(G(z))) can be ineffective early, the generator is instead trained to maximize log(D(G(z))).
  • Generative Adversarial Networks: DCGANs use convolutional architectures, strided convolutions, reshaped noise input, batch normalization, and a sigmoid output to stabilize GAN training.The supplied passage lists these architectural insights as key features.
  • Poisoning Attacks: Poisoning attacks insert, modify, or delete training points to alter a target model’s decision boundaries.Label manipulation is one poisoning strategy, including randomly perturbing labels.
  • White-Box Evasion: White-box crafting uses model-parameter access in two phases: direction-sensitivity estimation followed by perturbation selection.The process is presented as generalizable from image-classification DNNs to supervised learning algorithms.
  • White-Box Evasion: The crafted example adds a perturbation to a legitimate input and repeats both phases until the adversarial goal is satisfied.The framework seeks a perturbation that is as small as possible while achieving the attack objective.
  • White-Box Evasion: Most DNN formulations are nonlinear and non-convex, so closed-form solutions are usually difficult to obtain.The survey therefore describes approximate techniques for solving the crafting optimization problem.

Direction Sensitivity Estimation.

Adversarial examples are generated by searching for input perturbations that induce a target misclassification while remaining within the valid input domain. The survey contrasts optimization-based, gradient-based, iterative, and Jacobian-based approaches for estimating useful perturbation directions.

  • Adversarial-example generation searches for a perturbation r that makes f(x+r)=l while keeping x+r within the input domain.
  • L-BFGS can solve the non-convex optimization problem effectively, but calculating adversarial samples is computationally expensive.
  • FGSM efficiently perturbs inputs using the gradient of the cost function with respect to the neural-network input.
  • The perturbation amplitude in FGSM is controlled by ϵ, while the gradient is computed for a correctly labeled normal sample.
  • Target-class FGSM maximizes the probability of a specified class that is unlikely to be the example’s true class.
  • The Basic Iterative Method extends FGSM by generating adversarial samples through repeated updates with a small step size.
  • Jacobian-based attacks use forward derivatives to identify output sensitivity to input components, supporting source-target misclassification attacks.

Perturbation Selection.

Perturbation selection determines whether attacks modify all input dimensions or only a selected subset. These choices trade off computational efficiency and perturbation size against the cost of identifying salient features.

  • Perturb all the input dimensions: FGSM perturbs every input dimension by a small amount in the gradient-sign direction and efficiently reduces Euclidean distance to the adversarial sample.
  • Trade-offs: Perturbing all dimensions enables fast crafting of many samples but can create larger, easier-to-detect perturbations.
  • Perturb a selected input dimensions: Saliency-map methods select a limited combination of input dimensions whose perturbation is expected to contribute to the adversarial goal.
  • Trade-offs: Selecting fewer dimensions reduces perturbations but requires higher computational cost.
  • Perturb a selected input dimensions: Jacobian matrices provide the sensitivity values used to calculate saliency for learned-model input components.
  • Perturb a selected input dimensions: Salient input components are added in decreasing saliency order until the resulting sample is misclassified.
  • Black-Box Attacks: Black-box attacks can prioritize oracle queries using directions identified by a substitute model’s Jacobian.

Jacobian based Data Augmentation.

Jacobian-based augmentation expands a substitute model’s dataset using oracle-guided directions, supporting black-box transfer attacks. The section also surveys GAN, SVM, and collaborative-system attack mechanisms and objectives.

  • Jacobian based Data Augmentation: A substitute model prioritizes oracle queries along directions identified by the sign of its Jacobian, then augments the dataset with those inputs.
  • Jacobian based Data Augmentation: The update S_n+1 = {x + λ ∗sдn(J_F(x)[O(x)]) : x ∈ S_n} ∪ S_n defines the next augmented dataset.
  • Transferability of Adversarial Samples: Adversarial samples can transfer between models with different architectures, including within-technique and cross-technique transferability.
  • Transferability of Adversarial Samples: Substitute-model training reduces oracle queries and can generalize attacks to non-differentiable targets such as decision trees.
  • Transferability of Adversarial Samples: Cross-technique transferability reduces the knowledge needed to force misclassification and has been demonstrated across SVMs, decision trees, and nearest neighbors.
  • GAN based attack in Collaborative Deep Learning: GAN attacks in collaborative deep learning generate samples resembling private training data from limited access to shared model parameters.
  • GAN based attack in Collaborative Deep Learning: The GAN attack works when the adversary’s local model improves its accuracy over time, and reported experiments found it successful against comparison defenses.
  • Evasion and Poisoning attack on Support Vector Machines: SVM evasion attacks can use a surrogate classifier even when the adversary lacks knowledge of the target classifier’s decision function.

5 ADVANCES IN DEFENSE STRATEGIES

The section reviews defense strategies for adversarial attacks, emphasizing their mechanisms, effectiveness, and limitations across attack settings. It concludes that no existing defense provides robustness against all adversarial examples.

  • Challenges: Adversarial defenses are difficult to establish theoretically because crafting attacks is generally nonlinear and non-convex.The lack of suitable theoretical tools makes it difficult to prove that a defense excludes a set of adversarial examples.
  • Open Challenges: Existing defenses generally provide robustness only against specific attacks in specific settings, leaving all-purpose robustness unresolved.Model modifications may alter the original objective, incur performance overhead, or degrade clean-data accuracy.
  • Adversarial Training: Adversarial training augments model training with crafted adversarial examples to increase robustness.It aims to make the model predict the same class for legitimate and perturbed examples.
  • Adversarial Training: Adversarial training is not robust to black-box attacks and can be bypassed by a two-step attack.The two-step attack first applies random perturbations and then performs a classical attack.
  • Defensive Distillation: Defensive distillation transfers information from one network's softmax outputs to train another network with a smoother loss function.A high temperature was reported to improve distillation performance, and the resulting model can retain classification accuracy on adversarial examples.
  • Defensive Distillation: Defensive distillation and label smoothing can be avoided by black-box attacks because adversarial examples transfer across neural networks.Label smoothing uses soft targets and relaxes the need to train an additional model.
  • Other Defenses: Feature squeezing reduces representation complexity through color-depth reduction and image smoothing so adversarial perturbations disappear.The reviewed strategies also include rejecting perturbed inputs with a NULL label, while Defense-GAN depends on GAN expressiveness and training quality.
  • Basis Function Transformations: JPEG compression performed better than PCA, low-pass filtering, and soft thresholding across black-box, grey-box, and white-box attacks.These transformations were applied as preprocessing to adversarial and legitimate images and evaluated by distinguishing the two sets.

6 CONCLUSION

The conclusion characterizes adversarial learning as a serious threat to machine-learning applications in the physical world. The survey reviews attacks, defenses, and taxonomies, but finds that robust protection against all attacks remains unresolved.

  • Conclusion: Adversarial learning threatens machine-learning applications because subtle perturbations can have catastrophic consequences in security-related environments.The survey frames robustness against adversarial attacks as an immediate research need.
  • Conclusion: The survey organizes adversarial attacks, defense strategies, and related topics through a taxonomy and review of known work.It covers attacks and countermeasures while considering different threat models and scenarios.
  • Conclusion: No existing countermeasure addresses all adversarial-learning challenges, leaving considerably robust design as an open problem.The conclusion states that current defenses are not a universal solution.
Loading 1810.00069v1…