PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Attacks and Defenses in Images, Graphs and Text: A Review

Han Xu, Yao Ma, Haochen Liu, Debayan Deb, Hui Liu, Jiliang Tang, Anil K. Jain

arXiv:1909.08072v2cs.LGcs.CRstat.ML

TL;DR

DNN adversarial examples threaten reliable deployment, especially in safety-critical applications, motivating a systematic review. The paper surveys attack and defense methods across images, graphs, and text, reporting vulnerabilities and a continually shifting attack–defense landscape.

  • Problem

    Adversarial examples raise concerns about using otherwise successful DNNs in safety-critical applications across image, graph, and text domains.

  • Method

    The paper provides a systematic and comprehensive survey of state-of-the-art adversarial attack and defense algorithms for images, graphs, and text.

  • Results

    The review covers major attack and defense techniques and reports that common DNN image classifiers can be highly vulnerable to small perturbations.

  • Takeaways & Limitations

    The survey organizes the main ideas and contributions of adversarial learning to support understanding and further progress across multiple data domains.

  • Takeaways & Limitations

    The attack–defense landscape is continually changing, as current state-of-the-art attacks may be neutralized by new defenses and those defenses later circumvented.

Abstract

from arXiv · show

Deep neural networks (DNN) have achieved unprecedented success in numerous machine learning tasks in various domains. However, the existence of adversarial examples has raised concerns about applying deep learning to safety-critical applications. As a result, we have witnessed increasing interests in studying attack and defense mechanisms for DNN models on different data types, such as images, graphs and text. Thus, it is necessary to provide a systematic and comprehensive overview of the main threats of attacks and the success of corresponding countermeasures. In this survey, we review the state of the art algorithms for generating adversarial examples and the countermeasures against adversarial examples, for the three popular data types, i.e., images, graphs and text.

1. Introduction

Adversarial examples expose a safety concern for successful DNN applications, including safety-critical systems. This survey organizes attacks and defenses across image, graph, and text data.

  • DNNs have achieved strong results across image, graph, text, speech, and other machine learning applications.The review highlights near-human object recognition in images and applications in speech recognition, natural language processing, and games.
  • Safety-critical deployments require models to be accurate, stable, and reliable because recognition failures can create dangerous situations.Examples include road-sign recognition for autonomous vehicles and fraud detection using graph convolutional networks.
  • Adversarial examples are intentionally designed inputs that cause machine learning models to make mistakes while often appearing almost unchanged to people.For MNIST image classifiers, nearly all digit samples can be attacked using imperceptible perturbations.
  • Countermeasures include gradient masking, robust optimization, and adversary detection, which respectively obscure gradients, train robust classifiers, or screen inputs.The survey describes these as three broad defense categories intended to improve DNN resistance.
  • Studying attacks and defenses also helps analyze whether DNN predictive behavior aligns with human reasoning.Perceptually indistinguishable perturbations can evade DNN detection, motivating work on explaining adversarial examples.
  • The survey systematically reviews state-of-the-art adversarial attacks and defenses for images, graphs, and text.Its organization covers definitions, image attacks and defenses, explanations, graph data, and text data.

2. Definitions and Notations

This section defines the main dimensions of adversarial attacks and defenses, including attacker goals, knowledge, victim models, and security evaluation. It also distinguishes poisoning from evasion and targeted from non-targeted attacks.

  • The review frames attacks and defenses through adversary goals, available knowledge, victim models, and security evaluation.These questions distinguish attack objectives, information access, model targets, and robustness-related metrics.
  • Poisoning Attack vs Evasion Attack: Poisoning attacks insert or modify fake training samples, whereas evasion attacks leave the classifier fixed and craft samples it cannot recognize.Poisoning can degrade accuracy or alter predictions when attackers access training data; evasion targets test-time detection.
  • Targeted Attack vs Non-Targeted Attack: Targeted attacks induce a specified label, while non-targeted attacks only seek an incorrect prediction.A targeted example is disguising a fraudster as a highly credible client.
  • Adversary’s Knowledge: White-box attackers know the target network’s architecture, parameters, and gradients; black-box attackers query inputs and outputs without internal configuration.Black-box attacks are described as more practical because model designers usually do not disclose internal details.
  • Adversary’s Knowledge: Semi-white-box attacks train a generative model with white-box access and then use it to craft examples without the victim model.After training, generation can proceed in a black-box setting.
  • Victim Models: The review focuses mainly on DNN adversarial examples and surveys architectures used for image, graph, and text data.It covers fully connected networks, CNNs, GCNs, and recurrent networks such as LSTMs.
  • Model Components: Back-propagation supplies gradients with respect to parameters and inputs, and input gradients are widely used to craft adversarial examples.The input derivative represents the output response to changes in the input.
  • Security Evaluation: Robustness measures resistance for one sample, whereas adversarial risk measures resistance across the total population.Minimal perturbation norms are introduced as robustness-related quantities, with larger norms indicating greater robustness or safety.

3. Generating Adversarial Examples

This section surveys methods for generating adversarial examples, primarily for image classifiers, across attack settings and perturbation types. The reviewed methods seek small or imperceptible changes that induce incorrect predictions, while experiments reveal substantial vulnerability across models and attack formulations.

  • Scope and setup: Image attacks are studied as a standard case because perceptual similarity is intuitive and image data and classifiers are structurally simpler than other domains.The section considers fully connected networks and CNNs, with MNIST, CIFAR10, and ImageNet as common datasets.
  • Attack formulation: White-box attacks assume access to the classifier and victim sample, then synthesize a perceptually similar image that causes an incorrect prediction.The formulation typically minimizes dissimilarity, often an l_p norm, while imposing a misclassification objective.
  • Optimization-based attacks: Szegedy et al. search for a minimally distorted targeted example by balancing similarity to x against loss toward target label t, solved approximately with L-BFGS.Varying constant c identifies an example that is close to x while fooling the classifier.
  • Gradient-based attacks: FGSM generates adversarial examples in one gradient step within an ϵ-neighbor ball, making it faster than iterative attacks and useful for large-scale adversarial training.Its objective seeks a point with low loss for target label t, and one back-propagation step produces the perturbation.
  • Decision-boundary attacks: DeepFool linearizes the decision boundary and moves samples along an orthogonal perturbation until they cross into another class.For MNIST LeNet, over 90% of test samples were attacked with l∞ perturbations below 0.1 when the input range was [0, 1].
  • Evaluation and limitations: The surveyed methods include a strong benchmark attack for testing DNN safety, but exact robustness estimation with SMT solving is slow and does not scale well to large networks.Later work improved the efficiency of this ground-truth approach.
  • Alternative attack settings: Other reviewed attacks alter one pixel, exploit l1 perturbations, learn universal perturbations, or use unrestricted samples, exposing weaknesses beyond standard pixelwise perturbations.Reported findings include 63.5% of CIFAR10 test samples attacked by changing one pixel, 85.4% of ILSVRC 2012 samples attacked by one universal perturbation under ResNet-152, and defenses against l∞ and l2 attacks remaining vulnerable to Elastic-Net attacks.

3.2. Physical World Attack

Physical-world attacks extend adversarial examples beyond direct digital inputs, using stickers, printed images, and 3D objects that remain adversarial across viewpoints and environmental conditions. The section also covers black-box, GAN-based, and poisoning attacks.

  • Physical Robustness: Printed adversarial images are tested under natural transformations such as viewpoint and lighting changes.Robustness means that the crafted images remain adversarial after transformation.
  • Eykholt’s Attack on Road Signs: Stickers placed on road signs can confuse autonomous vehicles’ sign recognizers across distances and viewpoints.The attack first locates sparse perturbations with an l1 attack, then generates sticker colors using an l2 attack.
  • Athalye’s 3D Adversarial Object: 3D-printing produced an adversarial turtle whose optimized texture remains adversarial across viewpoints, distance, lighting, rotation, and backgrounds.The texture is optimized on rendered images before an instance of the object is printed.
  • Black-Box Attacks: Black-box substitute-model attacks exploit transferability by training a model that imitates the victim and attacking the substitute.The process synthesizes and augments query-labeled data, then applies attacks such as FGSM; confidence-based ZOO attacks can achieve higher success than substitute-model attacks.
  • Other Attack Settings: GAN-based attacks generate more natural, less detectable adversarial samples, while poisoning attacks insert adversarial examples into training data before model training.Poisoning attacks are frequently applied to graph neural networks because of their transductive learning procedure.

4. Countermeasures Against Adversarial Examples

The survey groups defenses against adversarial examples into gradient masking, robust optimization, and adversarial-example detection. These strategies respectively obscure attack signals, retrain classifiers for robustness, or reject suspected adversarial inputs.

  • Gradient Masking/Obfuscation: Gradient masking or obfuscation hides classifier gradients to confuse attacks that rely on gradient information.This category targets the information used by many attack algorithms.
  • Robust Optimization: Robust optimization retrains a DNN classifier so it can correctly classify subsequently generated adversarial examples.The strategy changes the classifier’s parameters through re-learning.
  • Adversarial Examples Detection: Adversarial-example detection models the distribution of benign examples and blocks inputs identified as adversarial.Detection occurs before the sample is passed to the classifier.

4.1. Gradient Masking/Obfuscation

Gradient masking and related preprocessing, randomization, and generative defenses aim to disrupt the gradients used by adversarial attacks. The survey also reports that such defenses can remain vulnerable because they confuse attackers rather than eliminate adversarial examples.

  • Gradient Masking: Gradient masking deliberately hides classifier gradients because many attack algorithms depend on them.The strategy is intended to confound adversaries by obstructing gradient-based attack mechanisms.
  • Defensive Distillation: Defensive distillation trains a smaller network on soft-label outputs produced with a temperature-scaled softmax.The procedure trains an initial network, evaluates softened scores, and trains a second network on those soft labels.
  • Defensive Distillation: At test temperature 1, distillation can make the target class score near 1 and other scores near 0, inhibiting gradient-based attacks.The cited explanation attributes this to enlarged logits after training at temperature T.
  • Preprocessing and Randomization: Non-differentiable preprocessing, including thermometer encoding and image transformations, is used to cause attacks based on input gradients to fail.Thermometer encoding discretizes each pixel value into an l-dimensional vector before classification.
  • Preprocessing and Randomization: Randomized classifiers and input transformations reduce attack success by preventing adversaries from knowing the exact model or input representation used.Examples include randomly selecting classifiers, dropping neurons, resizing images, and padding with zeros.
  • Generative Defenses: PixelDefend and Defense-GAN project potential adversarial examples onto a benign data manifold before classification.Their generative networks act as purifiers, but the resulting deep models can produce very small or irregularly large gradients.
  • Limitations: Defensive distillation and other gradient-masking defenses remain vulnerable because masking confounds adversaries without eliminating adversarial examples.Carlini and Wagner demonstrated vulnerability of defensive distillation, while Athalye et al. developed attacks that break gradient-masking strategies.

4.2. Robust Optimization

Robust optimization improves classifier resistance by training models against specified adversarial threats, including norm-bounded perturbations. The reviewed approaches include regularization, adversarial retraining, certified defenses, and efficiency-oriented variants, with trade-offs in attack coverage and computational cost.

  • Robust optimization learns model parameters that improve predictions on potential adversarial examples within a predefined adversarial space.The defense is typically designed against small l_p perturbations, especially l_∞ and l_2 attacks.
  • Regularization methods: Regularization methods constrain layer instability or partial derivatives so input distortions cause smaller changes in the model.Penalizing large layer-wise instability is described as reducing adversarial risk and increasing robustness.
  • Adversarial (re)training: FGSM adversarial training augments training with generated adversarial examples, but the resulting classifier remains vulnerable to iterative attacks.The review attributes this weakness to gradient obfuscation and extreme non-smoothness near test samples.
  • Adversarial (re)training: PGD adversarial training demonstrates robustness against single-step and iterative attacks on MNIST and CIFAR10, but costs k times natural-training complexity with k-step PGD.This computational cost makes scaling to large datasets such as ImageNet difficult.
  • Adversarial (re)training: Ensemble adversarial training uses FGSM examples crafted from other pretrained classifiers and provides robustness against single-step and black-box attacks on ImageNet.Decoupling adversarial-example generation from model training makes the method more efficient than the compared FGSM and PGD training approaches.
  • Accelerating adversarial training: Free adversarial training reuses backward-pass calculations, while YOPO reuses gradients coupled to the first layer to reduce training costs.YOPO is presented as an efficiency-oriented strategy based on the stated first-layer coupling of PGD-generated examples.

4.3. Adversarial Example Detection

Adversarial-example detection methods classify inputs as benign or adversarial before prediction, using auxiliary models, statistical differences, or prediction consistency. However, adaptive attacks have bypassed many detectors, motivating stronger evaluation.

  • Detection framework: Detection methods first distinguish benign from adversarial inputs, allowing the classifier to refuse labels for detected attacks.Detection schemes are evaluated both on correctly identifying adversarial examples and on avoiding benign-example misclassification.
  • Auxiliary models: Auxiliary-model defenses assign adversarial examples an extra class or train a binary discriminator before classification.One approach uses K + 1 labels, while another separately discriminates adversarial from benign samples.
  • Statistical tests: Statistical defenses separate adversarial and benign data using differences in principal-component weights or a Maximum Mean Discrepancy test.These methods treat adversarial examples as statistically distinguishable from natural examples or benign datasets.
  • Consistency checks: Consistency-based defenses perturb models or inputs and flag samples whose predictions change substantially under dropout, color-depth reduction, or spatial smoothing.They hypothesize that natural examples produce stable predictions under these manipulations, unlike adversarial examples.
  • Evaluation and evasion: Carlini–Wagner bypassed 10 detection methods, while stronger attacks later broke feature-squeezing defenses.These results show that detectors must be assessed against strong attacks rather than only the attacks used during development.

5. Explanations for the Existence of Adversarial Examples

The survey reviews competing explanations for adversarial examples, including low-probability generalization, decision-boundary geometry, data distribution, and transferability. Existing evidence remains unresolved, with transferability also supporting black-box attacks and defenses.

  • Explanatory hypotheses: Initial explanations attribute adversarial examples to poor generalization in low-probability data regions caused by highly complex DNN structures.The survey presents this as an original hypothesis rather than a settled account.
  • Model capacity: Linear models are also vulnerable, while experiments report that increasing model capacity can improve robustness.These findings complicate explanations based solely on deep nonlinear model complexity.
  • Decision boundaries: Adversarial examples tend to lie near decision boundaries, whose geometry has been characterized as too flat, too curved, or inflexible.The survey lists multiple geometric hypotheses without selecting among them.
  • Open questions: The survey reports no consensus on the causes of adversarial examples, alongside hypotheses that optimal robustness may be impossible or may require more data and reduced accuracy.Proposed explanations include poorly concentrated class distributions and a robustness–accuracy cost.
  • Transferability: Transferability lets examples targeting one model mislead others, making substitute-model attacks useful in black-box settings and supporting some defenses.On ImageNet, FGSM examples are reported as more transferable than BIM examples at the same perturbation intensity.

6. Graph Adversarial Examples

Graph adversarial attacks modify discrete graph structures or node features to alter node, graph, embedding, or global classification outcomes. Their non-independent, transductive setting and unclear notion of imperceptibility distinguish them from image attacks, while defenses provide moderate robustness.

  • Threat model: Attackers slightly modify graph connections or node features to induce wrong node- or graph-classification predictions.Graph attacks raise security concerns because changing one feature or connection can influence predictions for other samples.
  • Distinctive challenges: Graph attacks differ from image attacks because samples are non-independent, node-classification attacks can alter the trained transductive classifier, and graph edits are discrete.Discrete search spaces can invalidate gradient-based image attack methods.
  • Node attacks: GCN node attacks add or remove edges or flip binary node features under a limited operation budget to mislead a classifier trained on the perturbed graph.A greedy method searches influential operations on a fixed linearized substitute GCN, which transfer to the later trained classifier.
  • Stealth constraints: Graph attack constraints can preserve degree distributions and forbid feature co-occurrences absent from the original graph, while later work also preserves graph-Laplacian spectra.These constraints aim to keep perturbed graphs similar or difficult to detect.
  • Learning-based attacks: Reinforcement learning attacks select edge additions or removals, with Q-learning rewarding changes to a victim node’s prediction after a fixed number of edits.Other methods use meta-learning for global node-classification poisoning, embedding disruption, or graph rewiring for graph classification.
  • Defenses and scope: Graph neural networks remain vulnerable despite defenses such as adversarial training, which the survey describes as providing moderate robustness.Attacks target individual nodes, global node-classification performance, node embeddings, or whole-graph classification.

7. Adversarial Examples in Audio and Text Data

Audio and text models can be misled by imperceptible or discrete perturbations across character, word, phrase, sentence, and waveform levels. Reviewed attacks span white-box and black-box settings, while reading-comprehension and translation systems show substantial vulnerability.

  • Motivation: Audio and text applications use DNNs in widely deployed systems, creating a need to study adversarial examples in these domains.Examples include speech assistants and language-processing tasks.
  • Text attack setting: Text discreteness prevents direct transfer of image gradient attacks and requires perturbations at character, word, phrase, or sentence granularity.Reviewed text attacks therefore modify symbolic inputs rather than continuous pixels.
  • Audio attacks: Speech-to-text attacks add inaudible waveform perturbations that make transcription networks output a chosen target phrase under a bounded decibel constraint.The attack is adapted from the Carlini–Wagner method and limits the maximum noise level.
  • Text classification: Text-classification attacks replace influential letters, manipulate words or phrases, apply token edits in black-box settings, or preserve syntax through synonym substitution and paraphrasing.HotFlip uses gradient information, whereas DeepWordBug and TextBugger score key tokens before applying edits.
  • Paraphrase attacks: A greedy paraphrase attack is proved to achieve a 1 − 1/e approximation factor for CNN and RNN text classifiers.The method frames discrete adversarial attack as optimization over set functions.
  • Reading comprehension: AddSent reduced 16 state-of-the-art reading-comprehension models from average 75% F1 to 36% by inserting an understandable sentence resembling the question.The inserted sentence does not contradict the correct answer but confuses the machine.
  • Machine translation: Translation models are vulnerable to natural typos and misspellings as well as manually crafted letter replacements and reorderings.The reported results suggest adversarial training as a robustness improvement.

8. Adversarial Examples in Miscellaneous Tasks

Adversarial examples affect diverse tasks beyond static image classification, including face recognition, segmentation, malware detection, and reinforcement learning. Studies show that small or physical perturbations can alter model predictions and decisions across these settings.

  • 8.1. Face Recognition: Physical adversarial glasses successfully mislead face-recognition models.The glasses are optimized digitally and then realized as 3D-printed frames worn by subjects.
  • 8.2. Object Detection and Semantic Segmentation: Universal perturbations can fool semantic-segmentation models across input images.Other attacks generate perturbations that cause incorrect predictions on all output labels in segmentation or object-detection models.
  • 8.3. Video Classification: Dynamic perturbations generated with GANs can mislead video classifiers.This addresses video settings where vulnerability does not automatically follow from attacks on static image classifiers.
  • 8.4. Malware Detection: Adding fake features lets malware evade a DNN classifier that achieves 97% accuracy on 120,000 Android applications.The DREBIN dataset includes over 5,000 malware samples, making this a safety-critical detection setting.
  • 8.5. Reinforcement Learning: Changing one pixel in a reinforcement-learning state can make the policy choose an incorrect action.An FGSM-style one-step gradient attack crafts a fake state, producing large changes in decision and performance.

9. Conclusion

The survey provides a systematic, categorical, and comprehensive overview of adversarial examples and countermeasures across multiple data domains. It concludes that attacks and defenses will continue to evolve in response to one another.

  • Conclusion: The survey organizes recent adversarial-learning studies across multiple data domains in a systemic, categorical, and comprehensive overview.The works are summarized chronologically because they are released frequently in response to one another.
  • Conclusion: New defenses will likely neutralize current state-of-the-art attacks, and subsequent attacks will circumvent those defenses.The authors hope the review clarifies adversarial-learning ideas and related applications.

B. Dichotomy of Defenses

The defense taxonomy includes gradient masking, robust optimization, and detection, alongside related issues such as exploding or vanishing gradients. The listed works reflect different mechanisms for addressing adversarial vulnerability.

  • Gradient Masking: Gradient masking defenses obfuscate gradient information used by many attacks.The cited work includes Goodfellow et al. (2014b).
  • Detection: Detection-based defenses distinguish benign samples from adversarial ones before model processing.Hendrycks and Gimpel (2016) is listed in this defense context.
  • Gradient-Related Issues: Exploding or vanishing gradients are identified as a separate gradient-related issue in the defense taxonomy.The cited works include Hein and Andriushchenko (2017) and Raghunathan et al. (2018a).
  • Defense Taxonomy: Samangouei et al. (2018) appears among the works associated with the defense taxonomy.
Loading 1909.08072v2…