PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Examples: Opportunities and Challenges

Jiliang Zhang, Chen Li

arXiv:1809.04790v4cs.LGstat.ML

TL;DR

Deep neural networks are vulnerable to imperceptible adversarial examples that induce incorrect, high-confidence predictions, creating risks for security-critical applications. This survey synthesizes adversarial-example concepts, causes, attacks, defenses, and open challenges. It highlights high-dimensional linear behavior, low black-box transfer rates, and important defense limitations.

  • Problem

    Adversarial examples threaten security-critical AI applications, while their causes remain unsettled and current attacks and defenses still face challenges.

  • Method

    The paper surveys adversarial-example concepts, causes, characteristics, construction methods, evaluation results, defenses, limitations, and future challenges.

  • Results

    Most targeted attacks had transfer rates below 0.5%, while high-dimensional linear perturbations increased a class score from 5% to 88%.

  • Takeaways & Limitations

    Adversarial examples remain difficult to deploy reliably in black-box and physical settings, motivating attack methods with higher transferability and more robust defenses.

  • Takeaways & Limitations

    Adversarial defenses incur important trade-offs: adversarial training has high overhead, while defensive distillation increases design complexity and can be bypassed.

Abstract

from arXiv · show

Deep neural networks (DNNs) have shown huge superiority over humans in image recognition, speech processing, autonomous vehicles and medical diagnosis. However, recent studies indicate that DNNs are vulnerable to adversarial examples (AEs), which are designed by attackers to fool deep learning models. Different from real examples, AEs can mislead the model to predict incorrect outputs while hardly be distinguished by human eyes, therefore threaten security-critical deep-learning applications. In recent years, the generation and defense of AEs have become a research hotspot in the field of artificial intelligence (AI) security. This article reviews the latest research progress of AEs. First, we introduce the concept, cause, characteristics and evaluation metrics of AEs, then give a survey on the state-of-the-art AE generation methods with the discussion of advantages and disadvantages. After that, we review the existing defenses and discuss their limitations. Finally, future research opportunities and challenges on AEs are prospected.

I. INTRODUCTION

AI systems have achieved strong performance across applications, but their growing deployment has exposed security risks. This review surveys adversarial examples, their construction, defenses, and underlying neural-network context.

  • I. INTRODUCTION: DNNs have achieved strong results in autonomous vehicles, robotics, network security, image and speech recognition, and natural language processing.
  • I. INTRODUCTION: AI security became increasingly important as incidents involving robots and autonomous vehicles caused human injuries and death.
  • I. INTRODUCTION: Adversarial examples are a test-stage attack that tricks a machine-learning system by constructing a specific input without changing the target system.
  • I. INTRODUCTION: The survey reviews recent adversarial-example construction methods and defense techniques for deep neural networks.
  • I. INTRODUCTION: A neural network transfers inputs through weighted layers using activation functions, with weights and biases learned by back propagation.

III. ADVERSARIAL EXAMPLES

Adversarial examples are inputs with subtle perturbations that can cause high-confidence misclassification, while their causes and real-world behavior remain important challenges. The section describes their construction, evidence for high-dimensional linear causes, transferability, and instability.

  • III. ADVERSARIAL EXAMPLES: A subtle perturbation can turn a correctly classified example into an adversarial example that is perceptually indistinguishable but incorrectly classified.
  • III. ADVERSARIAL EXAMPLES: A panda image changed to a gibbon prediction, with confidence rising from 57.7% to 99.3% while humans could not distinguish the images.
  • A. Cause of Adversarial Examples: High-dimensional linear behavior can explain adversarial examples: perturbing every input dimension in one direction increased class-1 score from 5% to 88%.
  • III. ADVERSARIAL EXAMPLES: Adversarial examples can transfer across models performing the same task, allowing attacks without access to the target model’s architecture and parameters.
  • III. ADVERSARIAL EXAMPLES: Physical transformations such as translation, rotation, and lighting can remove adversarial ability, creating difficulty for real-world deployment.

C. Evaluation Metrics

Adversarial examples are evaluated by attack success, model robustness, perturbation magnitude, and perceptual similarity. These metrics capture whether attacks fool models while remaining difficult for humans to distinguish.

  • 1) Success Rate:: Success rate is the most direct evaluation criterion, but it generally decreases as perturbation magnitude increases.FGSM can require larger perturbations and suffer label leaking, whereas iterative and saliency-map methods can achieve higher success with lower or specific perturbations.
  • 2) Robustness:: Robustness measures the smallest perturbation needed to cause misclassification, averaged across inputs.The per-example quantity is the distance to the classifier’s decision boundary, and larger values indicate stronger robustness.
  • 3) Transferability:: AEs transfer across models, and transfer rate depends on model architecture, capacity, test accuracy, and perturbation magnitude.Transfer is higher among models with similar architectures, lower capacity, and higher test accuracy; within a certain range, it increases with perturbation magnitude.
  • 4) Perturbations:: SSIM compares luminance, contrast, and structure between aligned images to quantify image similarity.The three components are combined into an overall similarity measure, with α = β = γ = 1 by default.
  • 4) Perturbations:: L2-norm measures the distance between an original example and its adversarial example, with larger distances indicating greater required perturbation.The metric quantifies perturbation magnitude between the original and adversarial inputs.

5) Perceptual adversarial similarity score:

PASS quantifies how similar an adversarial image remains to its original image from a perceptual perspective. It uses SSIM-based comparisons and a homography transform, with a threshold helping distinguish excessive perturbations and guide attack optimization.

  • 5) Perceptual adversarial similarity score:: PASS quantifies the perceptual similarity between an original image and its adversarial image using the SSIM measurement system.SSIM combines luminance, contrast, and structure comparisons.
  • 5) Perceptual adversarial similarity score:: PASS incorporates a homography transform mapping the original image X to the adversarial image X′ before measuring similarity.The transform is described as a mapping from one plane to another.
  • 5) Perceptual adversarial similarity score:: An appropriate PASS threshold distinguishes adversarial examples with excessive perturbations and can guide attack-method optimization.The construction constraint combines model misclassification with a PASS threshold.

D. Adversarial Abilities and Adversarial Goals

Adversarial abilities range from full model knowledge to limited black-box interaction, while attack goals range from confidence reduction to targeted misclassification. Black-box attacks exploit transferability by constructing examples on substitute models.

  • Adversarial Abilities:: White-box attackers know the training data, model parameters, and architecture, whereas grey-box attackers know some information but not model parameters.Grey-box knowledge may include architecture, learning rate, training data, and training steps.
  • Adversarial Abilities:: Black-box attackers lack the target model’s architecture and parameters but can interact with the system and exploit adversarial-example transferability.They train an alternative model, construct adversarial examples on it, and use them against the unknown target.
  • Black-box Attack:: A practical black-box attack builds a synthetic labeled dataset from oracle queries, trains a substitute model, generates adversarial examples, and transfers them to the target.The synthetic inputs are generated, while labels are observed from the target model used as an oracle.
  • Adversarial Goals:: Adversarial goals include confidence reduction, non-targeted misclassification, targeted misclassification, and source/target misclassification.Targeted misclassification forces a specified target class, while non-targeted misclassification changes the output to any class different from the original.
  • AE Construction Methods:: L-BFGS, FGSM, and related construction methods differ in optimization difficulty, computation cost, perturbation constraints, and susceptibility to label leaking.FGSM is fast and uses an L∞-norm but is prone to label leaking; L-BFGS uses a more complicated optimization procedure.

3) IGSM

The section surveys iterative and optimization-based adversarial attacks, emphasizing perturbation control, attack success, transferability, and computational trade-offs.

  • IGSM: IGSM applies multiple smaller perturbation steps and clips each result to remain within the original image’s neighborhood.
  • IGSM: IGSM requires multiple iterations but achieves a higher AE-construction success rate than FGSM.
  • JSMA: JSMA modifies high-impact pixels using a gradient-based saliency map until reaching the target class or maximum perturbation.
  • JSMA: JSMA has high computational complexity but produces adversarial examples with high success and transfer rates.
  • DeepFool: DeepFool iteratively linearizes nonlinear classifiers to find the nearest decision boundary and construct minimally perturbed adversarial examples.
  • Universal Adversarial Perturbations: Universal perturbations fool most dataset images with one model-dependent perturbation while preserving image structure under a norm constraint.

B. Other Attack Methods

Other attack methods broaden adversarial-example construction through generative models, randomized gradients, sparse pixel changes, and black-box semantic interactions.

  • Other Attack Methods: AdvGAN uses generative adversarial networks to create targeted adversarial examples while preserving source-example distribution and perturbation diversity.
  • Other Attack Methods: RAND + FGSM first adds random noise to escape nonsmooth neighborhoods, then applies FGSM to improve attack success and avoid label leaking.
  • Other Attack Methods: Semantic-segmentation attacks generate semantics-aware adversarial examples against cloud image detectors using only black-box API interactions.

V. COMPARISON OF VARIOUS ATTACK METHODS

The comparison evaluates attack methods by access model, attack type, targeting, perceptibility, success rate, and transfer rate using reproducible experiments.

  • V. COMPARISON OF VARIOUS ATTACK METHODS: The experiments compare black-box and white-box access, targeted and non-targeted attacks, PASS, success rates, and transfer rates.
  • V. COMPARISON OF VARIOUS ATTACK METHODS: The study compares attributes of different attack methods alongside empirical attack performance.
  • V. COMPARISON OF VARIOUS ATTACK METHODS: The experiment code is made available online for reproduction.

A. Experimental Setup

The experiments use ImageNet validation images and five pretrained network architectures to compare mainstream attacks. Results show widespread white-box and gradient-based methods, with MIM strongest in the reported Inception V3 evaluation and JSMA too memory-intensive to report.

  • A. Experimental Setup: The experiments run on TensorFlow-GPU 1.6.0 and Python 3.6 using an AMD Threadripper 1920X, NVIDIA GTX 1050Ti, and 16G memory.
  • A. Experimental Setup: The dataset is ILSVRC 2012, with 1000 randomly selected validation images spanning 1000 categories at 299×299×3 resolution.
  • A. Experimental Setup: Success and transfer rates are evaluated on Inception V3, AlexNet, ResNet34, DenseNet20, and VGG19 pretrained on ImageNet.
  • Experimental Findings: Most mainstream attacks are white-box and gradient-based, while black-box AE construction is difficult; MIM has the highest reported targeted and non-targeted success rates.
  • Experimental Findings: JSMA is excluded from large-dataset success and transfer-rate results because its pixel-wise Jacobian computation runs out of memory.

D. The Transfer Rate

Transferability measures whether adversarial examples generated on one model also fool other models. Transfer depends on model characteristics and perturbation magnitude, while targeted transfer is generally low.

  • D. The Transfer Rate: Transfer rates are evaluated by applying adversarial images generated on Inception V3 to AlexNet, ResNet, DenseNet, and VGG.The evaluation uses 1000 adversarial images from each construction method.
  • D. The Transfer Rate: Within a certain perturbation range, increasing perturbation magnitude increases the transfer rate of adversarial examples.
  • D. The Transfer Rate: Most targeted attacks have transfer rates below 0.5%, indicating difficulty conducting adversarial-example attacks in real black-box scenarios.
  • Other Defense Techniques: Adversarial training improves robustness by repeatedly adding adversarial examples to training data, but standard training remains vulnerable to examples generated by other models.
  • Other Defense Techniques: Defensive distillation trains a second network using soft labels from a first network, while detector methods identify adversarial inputs from internal representations or binary codes.

C. Other Defense Techniques

MagNet combines detector networks that distinguish adversarial from normal examples with a reformer network that moves adversarial examples toward normal examples.

  • C. Other Defense Techniques: MagNet includes one or more separate detector networks and one reformer network.
  • C. Other Defense Techniques: The detector learns to distinguish normal examples from adversarial examples by approximating normal examples.
  • C. Other Defense Techniques: The reformer network moves adversarial examples toward normal examples.

D. Limitations of Defenses

Existing defenses improve robustness or detect adversarial examples, but they incur overhead, depend on attack types or model details, and remain vulnerable to broader generalization and real-world stability challenges. The survey therefore identifies transferability, perturbation-free construction, and physical-transformation modeling as key research directions.

  • Defense limitations: Adversarial training can significantly improve robustness, but generating adversarial examples during training creates high overhead and the best attack method is theoretically unclear.These limitations complicate selecting attacks that provide the strongest robustness.
  • Defense limitations: Defensive distillation reduces perturbation sensitivity with low training and testing overhead, but increases design complexity and can be bypassed through alternative objectives, gradients, or transfer attacks.The bypass strategies include attacking a fragile model and transferring adversarial examples to the distilled model.
  • Research challenges: Adversarial examples remain difficult to stabilize under physical transformations, because changes in distance, angle, blurring, rotation, scaling, or illumination can restore correct classification.This limits reliable deployment in real-world applications.
  • Research opportunities: Future work should prioritize high-transfer-rate examples, perturbation-free construction, and explicit modeling of physical transformations.These directions target black-box evaluation, model-dependent perturbation selection, and real-world adversarial stability.
  • Defense limitations: Black-box attacks are difficult to resist through architecture or parameter changes because attackers need not obtain those model details.Defensive distillation, for example, requires modifying and retraining the target classifier.
  • Defense limitations: Defense effectiveness varies across attack methods, giving adversarial training and detectors weak generalization against adversarial examples generated by different attacks.Detector performance is also highly correlated with detector type and does not itself improve model robustness.
Loading 1809.04790v4…