PaperlayerView exact revision on arXiv

Source-linked AI summary

Advances in adversarial attacks and defenses in computer vision: A survey

Naveed Akhtar, Ajmal Mian, Navid Kardan, Mubarak Shah

arXiv:2108.00401v2cs.CVcs.CRcs.CYcs.LG

TL;DR

Deep-learning vision systems face adversarial attacks that can alter predictions with imperceptible perturbations, creating concerns for security-critical applications. This survey extends an earlier review by synthesizing post-2018 peer-reviewed research, defining terminology, and organizing attacks and defenses. It finds a rapidly expanding field, with computer-vision work emphasizing black-box attacks and machine-learning work emphasizing robustification, especially adversarial training and certified defenses.

  • Problem

    Deep-learning vision models used in security-critical applications remain vulnerable to imperceptible perturbations that can manipulate predictions.

  • Method

    The survey reviews post-2018 peer-reviewed attacks and defenses, builds on an earlier survey, defines technical terminology, and organizes the literature.

  • Results

    The literature is rapidly expanding, with computer-vision sources emphasizing black-box attacks and machine-learning sources emphasizing model robustification, adversarial training, and certified defenses.

  • Takeaways & Limitations

    Adversarial-attack research has broadened beyond classification, while adversarial training and provable defenses stand out among current defense directions.

  • Takeaways & Limitations

    The survey does not directly address backdoor or Trojan attacks, including them only because of their proximity to adversarial attacks.

Abstract

from arXiv · show

Deep Learning (DL) is the most widely used tool in the contemporary field of computer vision. Its ability to accurately solve complex problems is employed in vision research to learn deep neural models for a variety of tasks, including security critical applications. However, it is now known that DL is vulnerable to adversarial attacks that can manipulate its predictions by introducing visually imperceptible perturbations in images and videos. Since the discovery of this phenomenon in 2013~[1], it has attracted significant attention of researchers from multiple sub-fields of machine intelligence. In [2], we reviewed the contributions made by the computer vision community in adversarial attacks on deep learning (and their defenses) until the advent of year 2018. Many of those contributions have inspired new directions in this area, which has matured significantly since witnessing the first generation methods. Hence, as a legacy sequel of [2], this literature review focuses on the advances in this area since 2018. To ensure authenticity, we mainly consider peer-reviewed contributions published in the prestigious sources of computer vision and machine learning research. Besides a comprehensive literature review, the article also provides concise definitions of technical terminologies for non-experts in this domain. Finally, this article discusses challenges and future outlook of this direction based on the literature reviewed herein and [2].

I. INTRODUCTION

Deep learning has become central to computer vision and is increasingly considered for safety-critical applications, but imperceptible perturbations can reliably manipulate visual-model predictions. This survey reviews the rapidly expanding attacks-and-defenses literature since 2018, emphasizing peer-reviewed work and clearer terminology.

  • I. INTRODUCTION: Deep learning underpins computer vision systems that address increasingly complex artificial-intelligence problems.
  • I. INTRODUCTION: Computer-vision systems are being considered for autonomous vehicles, facial recognition, robotics, and surveillance despite emerging security concerns.
  • I. INTRODUCTION: Imperceptible image perturbations can completely alter deep visual-model predictions, and adversarial examples now affect classification, segmentation, detection, and tracking.
  • I. INTRODUCTION: Adversarial perturbations can produce high-confidence wrong predictions, transfer across models, or be universal across images, creating security implications for deployed systems.
  • I. INTRODUCTION: Since 2018, the field has expanded rapidly, motivating a sequel survey that organizes later attack and defense methods beyond early classification-focused work.
  • I. INTRODUCTION: The survey provides definitions for evolving terminology and prioritizes recent peer-reviewed contributions from leading computer-vision and machine-learning venues.

III. ADVERSARIAL ATTACKS: THE FORMAL PROBLEM

The formal adversarial-attack problem seeks an image modification that changes a model’s output while preserving an image acceptable to humans. The conventional formulation uses norm-bounded additive perturbations, while broader formulations also cover unrestricted replacements and perceivable localized patterns.

  • III. ADVERSARIAL ATTACKS: THE FORMAL PROBLEM: An adversary seeks a perturbation ρ that changes a model’s output from ℓ to an incorrect label ˜ℓ while satisfying ||ρ||p < η.
  • III. ADVERSARIAL ATTACKS: THE FORMAL PROBLEM: The norm constraint uses the ℓp-norm of the perturbation and a predefined scalar η to model humanly imperceptible manipulation.
  • III. ADVERSARIAL ATTACKS: THE FORMAL PROBLEM: Unrestricted examples and localized perceivable patterns such as adversarial patches fall outside the conventional norm-bounded additive formulation.
  • III. ADVERSARIAL ATTACKS: THE FORMAL PROBLEM: A broader formulation requires a manipulated image ˜I to remain in S_I, the set of images humans perceive as clean or acceptable for the desired output.
  • III. ADVERSARIAL ATTACKS: THE FORMAL PROBLEM: Image-classification attacks may target a specified incorrect label or choose it randomly for a nontargeted attack, while image-specific attacks affect individual images.

IV. FIRST-GENERATION ATTACKS

First-generation adversarial attacks established core strategies for fooling deep visual models, from imperceptible image perturbations to physical-world manipulations and universal signals. These methods span gradient-based, iterative, optimization-based, saliency-based, and norm-minimizing approaches, with later work emphasizing stronger attacks and robust training.

  • First-generation attacks: Szegedy et al. showed that minuscule, imperceptible perturbations can completely alter deep visual model predictions and transfer across classifiers.Their optimization-based attack used L-BFGS to approximately solve the perturbation problem while preserving imperceptibility.
  • First-generation attacks: FGSM efficiently computes norm-bounded adversarial perturbations in one gradient-based step, prioritizing perturbation efficiency over high fooling rates.It applies the sign of the input gradient, scaled by a fixed ϵ, to form the adversarial image.
  • First-generation attacks: BIM extends FGSM iteratively and demonstrated that printed adversarial images can fool classifiers in the physical world.Kurakin et al. fooled an ImageNet inception model on a mobile device using printed adversarial images; the targeted variant is also called ILCM.
  • First-generation attacks: PGD frames adversarial robustness through robust optimization and is identified as possibly the strongest first-order attack.PGD-based adversarial training avoids FGSM’s label leaking, improves robustness against weaker first-order attacks, but is computationally expensive.
  • First-generation attacks: DeepFool, JSMA, and universal perturbations pursue alternative attack structures, respectively minimizing perturbation norms, modifying influential pixels, and fooling many images with one signal.Universal perturbations achieved approximately 80% fooling on popular ImageNet models with around 4% norm distortion, which was slightly perceivable and therefore termed quasi-imperceptible.

V. RECENT ATTACKS ON CLASSIFIERS

Recent classifier attacks build mainly on first-generation techniques, extending their core algorithms and techniques through downstream refinements.

  • V. RECENT ATTACKS ON CLASSIFIERS: Recent classifier attacks primarily refine and adapt first-generation attack techniques.The survey organizes these developments around the structure shown in Fig. 2(c).

A. Advanced gradient based attacks

Advanced gradient-based attacks extend first-generation gradient methods by improving perturbation focus, sparsity, lethality, norm reduction, computational efficiency, and camera-pipeline specificity.

  • A. Advanced gradient based attacks: Salient-region guidance focuses FGSM-like perturbations on image regions identified with super-pixel attention.The resulting perturbations are claimed to be more robust against image processing.
  • A. Advanced gradient based attacks: GreedyFool uses pixel gradients to produce sparse distortions, while GAMA adds a relaxation term to gradient-based losses.The sparse distortion is intended to reduce perceptibility, and GAMA is claimed to find better gradient-based attack solutions.
  • A. Advanced gradient based attacks: DDN decouples direction and norm in ℓ2-bounded perturbations, while trust-region methods target norm reduction and computational efficiency.Both approaches upgrade first-generation gradient-based attacks through distinct optimization changes.
  • A. Advanced gradient based attacks: Camera-pipeline-aware attacks use differential approximation to fool classifiers through images from one camera pipeline but not another.This incorporates the influence of camera image processing into attack construction.
  • A. Advanced gradient based attacks: The survey treats advanced gradient attacks as a cross-cutting category because white-box and transfer-based attacks often directly involve model gradients.Other attacks are introduced under categories better suited to their objectives or threat models.

B. Black-box attacks

Black-box attacks are presented as pragmatic because they require no or minimal knowledge of the target model, motivating recent work on query-based and transfer-based approaches.

  • B. Black-box attacks: Black-box attacks assume no or minimal target-model knowledge and are reviewed through query-based and transfer-based directions.Their practicality has made them highly popular in recent literature.

1) Query-based attacks:

Query-based attacks probe target models to construct adversarial images, generally trading query count against perturbation distortion, while transfer-based attacks use surrogate models without querying the target.

  • 1) Query-based attacks:: Query-based attacks use target-model outputs to construct minimally distorted adversarial images while preserving model fooling.Queries are commonly used to refine perturbations toward greater imperceptibility.
  • 1) Query-based attacks:: Decision-boundary methods estimate local geometry from top-1 labels and efficiently nudge images across the boundary with small ℓp-norm perturbations.Rahmati et al. exploit the smaller mean curvature near the data point to estimate a normal vector.
  • 1) Query-based attacks:: Query efficiency is improved through history-based noise customization, transferable initializations, meta-learned priors, and constrained low-frequency search spaces.CAB, PPBA, and related methods reduce or structure the query search process.
  • 1) Query-based attacks:: Latent-space and surrogate-model approaches reduce query search dimensionality, including Bayesian optimization and encoder-decoder embeddings.TREMBA searches for adversarial examples in a learned low-dimensional embedding space.
  • 1) Query-based attacks:: Decision-based attacks dominate current query-based literature, although score-based schemes also appear frequently.The survey notes multiple recent contributions in both categories.
  • 1) Query-based attacks:: Targeted transfer-based attacks generally achieve success rates below 50% while producing often perceptible perturbations.The figure reports average success rates across ImageNet models from the original papers.
  • 1) Query-based attacks:: Transfer-based attacks avoid querying the black-box model by computing perturbations on local surrogate models intended to fool remote targets.They are more popular than query-based attacks partly because avoiding queries can reduce suspicion.
  • 1) Query-based attacks:: Transferability is enhanced by salient-region guidance, internal-representation manipulation, layer-specific representations, gradient acceleration, and scale-invariant model ensembles.These methods seek perturbations that generalize across models and transformations.

2) Transfer-based attacks:

Transfer-based attacks increasingly target diverse tasks and objectives, while methods improve cross-model transferability through feature-space manipulation, adaptive optimization, and data-free surrogate training.

  • Feature-space perturbations can produce adversarial examples that transfer more effectively across models.
  • Targeted attacks improve transferability by addressing gradient-magnitude reduction and adversarial proximity to the true-class region.
  • Transferable attacks now extend to person re-identification, universal perturbations, and data-free surrogate-model training.
  • Unrestricted attacks preserve semantic meaning for humans while replacing natural images with synthetically generated adversarial images.
  • Perceptual color distance can permit larger ℓp-norm perturbations while maintaining imperceptibility.
  • Backdoor attacks embed training-time triggers, whereas model inversion reconstructs training data or its markers and raises privacy concerns.

F. Adaptive attacks

Adaptive attacks specifically target defenses, and adversarial research has broadened beyond classification to specialized attacks on diverse models, tasks, and physical detection systems.

  • F. Adaptive attacks: Adaptive attacks are designed specifically to fool defense mechanisms after multiple defenses were broken by stronger attacks.
  • F. Adaptive attacks: Shadow Attack breaks certifiable defenses by generating perturbations outside certified ℓp bounds that produce spoofed certificates.
  • Attacks beyond classification address ranking, DRAM vulnerabilities, point clouds, videos, graph networks, binarized networks, and quantized networks.
  • Object detection and tracking motivate specialized attacks, including generator-based tracker attacks and physical-world detector attacks.
  • Universal camouflage can fool object detectors in-the-wild but produces conspicuous patterns, while adversarial training can improve detector robustness.

B. Reinforcement learning

Adversarial research extends to reinforcement learning, captioning, and face recognition, where attacks exploit task-specific temporal, sequential, geometric, and physical properties.

  • B. Reinforcement learning: FGSM-like perturbations degrade trained reinforcement-learning policies, including in black-box settings where adversaries manipulate raw policy inputs.
  • B. Reinforcement learning: Adversarial policies can reliably defeat robust self-play opponents while generating apparently random behavior in high-dimensional zero-sum games.
  • Captioning attacks are more challenging than attacks on visual models alone because captions have temporal dependencies, yet recent methods successfully fool captioning frameworks.
  • Face-recognition systems remain vulnerable to adversarial attacks, including black-box decision-based attacks against popular models.
  • Face-recognition attacks include landmark manipulation, adversarial eyeglasses, patches, light projection, and other physically realizable methods.

E. Miscellaneous attacks

Miscellaneous attacks target segmentation, depth, retrieval, autonomous driving, physical detection, illumination, and model performance, demonstrating the breadth of adversarial objectives.

  • E. Miscellaneous attacks: Adversarial attacks manipulate semantic segmentation, super-resolution, monocular depth, and hashing-based image retrieval models.
  • LiDAR-based physical attacks can make vehicles undetectable to object detectors, although the computed mesh generally remains unnatural.
  • PhysGAN generates printed perturbations resilient to lighting and viewing-angle changes to fool autonomous-vehicle steering models.
  • Adversarial patches and style-transfer-based camouflage enable physical-world attacks against recognition and detection systems.
  • Adversarial illumination attacks use rolling-shutter distortions or laser beams to make captured images adversarial.
  • AdvProp uses adversarial examples during training and reports performance gains on clean images, unlike adversarial training that generally sacrifices clean accuracy for robustness.

B. The link between attacks and model interpretation

Adversarial perturbations can expose model-understood semantic concepts and support image manipulation, while research continues to debate why adversarial examples exist. Proposed explanations include linearity, geometry, invariance, biological vision, equilibrium, manifold structure, and feature-based accounts.

  • B. The link between attacks and model interpretation: ‘Attack to explain’ perturbations visualize human-defined semantic concepts as understood by deep learning perceptual models.The method iteratively expands and refines perturbations, revealing human-understandable patterns and supporting low-level vision tasks with robust classifiers.
  • B. The link between attacks and model interpretation: Group-sparse perturbations can be interpretable, but they do not explain the model itself like ‘attack to explain’.
  • B. The link between attacks and model interpretation: No consensus explains adversarial examples because many hypotheses fail to generalize and remaining explanations often conflict.The literature discusses linearity, manifold assumptions, high-dimensional geometry, input invariance, biological vision, and theoretical game-based accounts.
  • B. The link between attacks and model interpretation: Universal perturbations are linked to shared decision-boundary directions and can dominate classifier features despite low visual power.Studies report common directions across samples and strong correlation between universal perturbations and adversarial examples, alongside low correlation with clean images.

C. Adversarial examples as features & other sources

Research interprets adversarial examples as potentially pervasive non-robust features and examines computational explanations for vulnerability. Defenses largely retain the survey’s earlier categories, with adversarial training receiving extensive refinement, analysis, and task-specific adaptation.

  • C. Adversarial examples as features & other sources: Adversarial examples may reflect pervasive non-robust features that help neural models achieve higher accuracy rather than merely representing bugs.The cited work also demonstrates that robust and non-robust features can be disentangled.
  • C. Adversarial examples as features & other sources: High-dimensional vulnerability may arise from computational constraints even when identifying a robust classifier is information-theoretically possible.
  • C. Adversarial examples as features & other sources: Defenses are organized around modifying models, transforming inputs, or adding external modules, with recent work continuing mainly along these three lines.
  • C. Adversarial examples as features & other sources: Adversarial training exposes models to adversarial examples during training and is treated as a principled robust-optimization framework.Madry et al. theoretically studied and justified the framework, which subsequently attracted substantial research attention.
  • C. Adversarial examples as features & other sources: 45% robust accuracy was achieved on CIFAR10 in 6 minutes using randomly initialized FGSM training, versus 10 hours for a PGD-based counterpart with similar results.
  • C. Adversarial examples as features & other sources: Adversarial training research includes variants, theoretical analyses, self-supervision, diverse perturbation generation, distributional objectives, and robustness-aware transfer or pruning.Task-specific adaptations address physical-world attacks, while some analyses report separate normalization for clean and adversarial images as beneficial.

2) Other model modifications:

Beyond adversarial training, defenses modify losses, activations, architectures, internal modules, or regularization schemes, while other approaches add inference-time detectors and input transformations. These methods broaden the defense landscape but are generally less generic than adversarial training or depend on external components.

  • 2) Other model modifications:: Model components can be altered for robustness through specialized losses, discontinuous activations, quantization, and internal perturbation or restoration modules.Perturbation-injection modules were reported to improve robustness by 4-7% over FGSM- and PGD-based adversarial training.
  • 2) Other model modifications:: Robust architecture search uses differentiable robustness metrics or global sparse coding to seek architectures with improved resistance to attacks.
  • 2) Other model modifications:: Regularization-based defenses target model gradients, Jacobians, biological-inspired post-learning adjustment, or bit-plane consistency.
  • 2) Other model modifications:: Adversarial training can combine with other defenses, whereas many alternative model modifications are less generic.
  • 2) Other model modifications:: Add-on mechanisms for pretrained models mostly detect adversarial inputs during inference and have become slightly less popular than earlier.Examples analyze reconstruction, context inconsistency, neuron attributes, activation paths, steganographic modification probabilities, or learned detectors.
  • 2) Other model modifications:: Input transformations seek to remove perturbations before classification through JPEG compression, learned projection, GAN reconstruction, or selective denoising.Compressed adversarial images have been found to lose significant fooling ability, and transformation methods can be combined with other defenses.

D. Certified defenses

Certified defenses aim to guarantee robustness within specified perturbation bounds, but most certify only one norm at a time. Recent work expands certification across norms, prediction ranks, transformations, and attack types.

  • Scope and limitations: Most certified defenses prove robustness against only one perturbation bound, such as ℓ2 or ℓ∞, rather than multiple ℓp norms simultaneously.A few exceptions provide more generic bounds across norms.
  • Cross-norm certification: Regularization of ReLU networks can enforce robustness against ℓ1 and ℓ∞ attacks while yielding provable robustness for any ℓp norm with p ≥1.
  • Prediction and transformation robustness: Randomized smoothing has been extended to certify tight ℓ2 robustness for top-k predictions rather than only top-1 predictions.The method builds on randomized smoothing.
  • Prediction and transformation robustness: Parameterized transformations, including translations and rotations, can be incorporated into randomized smoothing to certify robustness in transformation-parameter space.One example certifies robustness with respect to rotation angle.
  • Specialized certified defenses: Randomized smoothing defenses have also been developed for patch attacks and with non-Gaussian noise distributions targeting ℓ1, ℓ2, and ℓ∞ attacks.Patch certification depends on the given image and patch size, while non-Gaussian smoothing is designed for greater efficiency across norms.
  • Specialized and combined defenses: The literature includes specialized defenses for tasks, networks, and attack types, alongside methods combining multiple defense strategies.Examples address VAEs, bit-flip attacks, Bayesian networks, tracking, open-set recognition, face recognition, and few-shot classification.

XII. CONCLUSION

The survey reviews visual-model adversarial attacks and defenses, extending its earlier survey to literature published after 2018. It also standardizes terminology and provides historical context for this research direction.

  • The survey examines adversarial attacks and defenses for deep learning models, focusing on visual models.
  • It focuses primarily on peer-reviewed papers from top-ranked computer vision and machine learning research sources.
  • The review extends the authors’ earlier survey by covering literature published after 2018.
  • It defines frequently used technical terms and discusses early contributions to provide historical context.
Loading 2108.00401v2…