PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Example Detection for DNN Models: A Review and Experimental Comparison

Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, Olivier Deforges

arXiv:2105.00203v4cs.CVcs.CR

TL;DR

Adversarial examples threaten image-classification models because small perturbations can induce misclassification, while prior reviews gave limited attention to detection. This paper surveys and categorizes test-time detection methods, experimentally evaluates eight detectors across four datasets and attack settings, and identifies weak generalization and practical trade-offs as continuing challenges.

  • Problem

    Prior reviews focused mainly on theoretical attacks and defenses, with limited attention to adversarial-example detection and experimental comparisons.

  • Method

    The paper surveys and categorizes test-time adversarial-example detectors and experimentally evaluates eight state-of-the-art algorithms across attack settings and image-classification datasets.

  • Results

    The paper reports detection rates, false-positive rates, complexity, overhead, and inference latency, while examining how adversarial-example content and CNNs affect detection.

  • Takeaways & Limitations

    The reviewed detectors lack generalization, and future work should address broader attack coverage and trade-offs among overhead, latency, and training cost.

  • Takeaways & Limitations

    Denoisers may leave noise or introduce distortion and are ineffective against L0 attacks, while reviewed detectors also compromise at least one lightweight-design factor.

Abstract

from arXiv · show

Deep learning (DL) has shown great success in many human-related tasks, which has led to its adoption in many computer vision based applications, such as security surveillance systems, autonomous vehicles and healthcare. Such safety-critical applications have to draw their path to success deployment once they have the capability to overcome safety-critical challenges. Among these challenges are the defense against or/and the detection of the adversarial examples (AEs). Adversaries can carefully craft small, often imperceptible, noise called perturbations to be added to the clean image to generate the AE. The aim of AE is to fool the DL model which makes it a potential risk for DL applications. Many test-time evasion attacks and countermeasures,i.e., defense or detection methods, are proposed in the literature. Moreover, few reviews and surveys were published and theoretically showed the taxonomy of the threats and the countermeasure methods with little focus in AE detection methods. In this paper, we focus on image classification task and attempt to provide a survey for detection methods of test-time evasion attacks on neural network classifiers. A detailed discussion for such methods is provided with experimental results for eight state-of-the-art detectors under different scenarios on four datasets. We also provide potential challenges and future perspectives for this research direction.

1 Introduction

The paper frames test-time adversarial examples as imperceptibly perturbed inputs that can fool image-classification models, then surveys and experimentally compares their detection methods.

  • Threat and motivation: Test-time evasion attacks add carefully crafted, often imperceptible perturbations to inputs, producing adversarial examples that may be misclassified by deep-learning models.Attackers may also seek false predictions, high confidence in those predictions, or transferability across models.
  • Scope: The review focuses on evasion attacks in image classification, while related adversarial vulnerabilities also occur in object detection, language, speech, cybersecurity, and medical imaging.The paper distinguishes this focus from the broader range of affected machine-learning tasks.
  • Research gap: Prior surveys emphasized theoretical attacks and countermeasures, while giving limited attention to adversarial-example detection and experimental comparisons.The paper positions its review as addressing both gaps for computer-vision image classification.
  • Contributions: The paper reviews and categorizes state-of-the-art detection methods by attack knowledge and by the technique used to distinguish clean from adversarial inputs.The stated contribution covers the detection-method taxonomy rather than defenses alone.
  • Contributions: The first experimental study tests adversarial-example detectors against white-, black-, and gray-box attacks across MNIST, CIFAR10, SVHN, and Tiny-ImageNet.The experiments are summarized in Figure 4.
  • Contributions: The paper also discusses adversarial-example content and its impact on detection methods, releases a reproducible scalable framework, and provides a benchmark website.The framework is designed to allow new detection methods to be included.

2 Related work

Related work reviewed adversarial attacks and defenses, but detection methods remained insufficiently classified and experimentally compared, motivating this paper’s broader comparison focus.

  • Prior reviews: Earlier reviews classified countermeasures by whether they modify data, the model, or add-on networks.These categories organize where the countermeasure intervenes in the learning system.
  • Countermeasure categories: Defense by design changes the model architecture or training data, whereas detection methods build precautions around the model under attack without directly interacting with it.The passage contrasts the two categories by where they operate.
  • Experimental evidence: Prior experimental work showed that ten tested detectors could be broken with new loss functions, but did not compare the detectors with one another.This finding is described as a limitation of that earlier study’s comparison scope.
  • Research gap: Existing reviews did not classify or discuss detection methods in sufficient detail, lacked experimental comparisons, and predated considerable newer detection methods.These gaps motivate a more detailed and current comparison.

3 Adversarial attacks and defense methods

This section introduces adversarial attacks against neural-network image classifiers, their threat models, and representative attack algorithms. It also distinguishes the notation and definitions used to describe these attacks.

  • Threat models: Adversarial attacks seek an input x′ within a perturbation bound that changes the model’s prediction from f(x).The adversary searches for x′ such that ||x′ − x|| < ϵ and f(x) ≠ f(x′).
  • Threat models: Threat models classify adversaries by knowledge of the victim or detector and by targeted versus untargeted objectives.White-, black-, gray-, zero-, limited-, and perfect-knowledge settings are described alongside targeted and untargeted attacks.
  • White box attacks: L-BFGS and CW formulate attack generation as optimization problems that seek misclassification while controlling perturbation or confidence.L-BFGS seeks minimum perturbation for a target label, while CW replaces the loss with an objective function involving a confidence parameter.
  • White box attacks: FGSM generates an adversarial example through a one-step gradient-sign update bounded by ϵ.The update is x′ = x + ϵ sign(∇xℓ(x, y)), with ||x′ − x||∞ < ϵ.
  • White box attacks: BIM iteratively applies FGSM, whereas PGD starts from a random perturbation within an Lp-ball and may use multiple restarts.BIM applies FGSM k times; PGD can use L1, L2, or L∞ perturbation constraints.
  • Other attacks: Other attacks modify selected pixels, internal representations, or transformations to fool the classifier.The survey describes JSMA, feature adversary attacks, adversarial transformation networks, and expectation over transformation.

4 Adversarial example detection methods

Adversarial-example detectors are presented as third-party components that reject suspected adversarial inputs while allowing clean inputs to reach the victim model. The survey organizes detectors by attack knowledge and discrimination technique, then evaluates them using accuracy, false positives, complexity, overhead, and latency.

  • Purpose and taxonomy: Detectors classify inputs as adversarial or clean, whereas defenses aim to classify clean and adversarial versions into the same class.Both approaches pursue the broader goal of defeating attacks but have different immediate objectives.
  • Purpose and taxonomy: Detectors act as third-party entities that reject adversarial inputs and let clean inputs pass to the victim model.The survey categorizes them first by knowledge of attacks and then by the technique used to distinguish inputs.
  • Evaluation criteria: Detection rate measures the fraction of successful adversarial examples identified by the detector, with higher values preferred.Successful adversarial examples are attacks that fool the learning model.
  • Evaluation criteria: False positive rate measures clean inputs incorrectly detected as adversarial, with lower values preferred.The metric divides clean inputs detected as adversarial by the total number of clean inputs.
  • Evaluation criteria: Complexity is detector training time, while overhead concerns additional architecture and parameter size.Lower overhead is preferable for platforms with limited memory and computational resources; repeated retraining also makes high complexity less appropriate when attacks or data change.
  • Evaluation criteria: Inference-time latency is the detector’s response time, and lower latency is preferred for real-time applications.The survey also ranks reported detection accuracy with up to five stars because studies use different models, datasets, and attacks.

4.1 Supervised detection

Supervised detectors use adversarial examples generated by known attacks to learn distinctions between clean and adversarial inputs. The reviewed approaches extract model, image, statistical, gradient, or reconstruction features and classify or threshold them.

  • Supervised detection trains detector D with adversarial examples generated by one or more attack algorithms because adversarial inputs are assumed to have distinguishable features.
  • Feature-based approaches monitor clean and adversarial sample behavior, then build classifiers or thresholds from the extracted features.
  • Uncertainty and softmax/logit methods use model prediction behavior, confidence, reconstructed inputs, or outputs from multiple classifiers to detect adversarial examples.
  • Other supervised detectors classify raw images, natural-scene statistics, gradients, reconstruction outputs, or intermediate-layer representations.
  • Statistical methods compare properties of clean and adversarial inputs or their distributions, including kernel density, MMD, and PCA-based representations.
  • The comparison table summarizes reported detection rates with star ranges because methods were evaluated on different models, datasets, and attacks.

4.2 Unsupervised detection

Unsupervised detectors are trained using only clean data and infer adversarial inputs from deviations in model behavior, uncertainty, distributions, or reconstructed inputs. The section also identifies practical limitations of supervised detection and denoising-based approaches.

  • Unsupervised detectors use only clean training data and infer adversarial inputs from features derived by monitoring clean-sample behavior.
  • Unsupervised methods include intermediate-layer k-NN scoring, reverse cross-entropy with class-wise density thresholds, uncertainty estimation, and deep neural rejection.
  • Other clean-data approaches detect deviations using softmax distributions, later PCA-component variance, or Gaussian mixture models of hidden states.
  • Denoisers are not guaranteed to remove all noise, may introduce distortion, and are ineffective against L0 attacks targeting only a few pixels.
  • Denoiser methods reconstruct or purify inputs and use probability density, reconstruction error, or prediction differences as detection scores.
  • Feature squeezing transforms inputs to remove unnecessary features, but an inefficient squeezer may enlarge the perturbation.

5 Experiment settings

The experiments evaluate eight publicly available detection methods across four image datasets and white-, black-, gray-box, untargeted, and robust attack settings. The study uses separately trained victim classifiers and a common testing environment.

  • Datasets: The evaluation uses MNIST, SVHN, CIFAR-10, and Tiny ImageNet, covering grayscale or RGB images from ten to 200 classes.
  • Baseline victim classifiers: Four baseline victim models are trained, including six-layer CNNs for MNIST and SVHN, an eight-layer CNN for CIFAR-10, and fine-tuned DenseNet201 for Tiny ImageNet.
  • Baseline victim classifiers: The baseline classifiers achieve 98.73% accuracy on MNIST, 94.99% on SVHN, 89.11% on CIFAR-10, and 65% on Tiny ImageNet.
  • Threat model and attacks: The threat model assumes the adversary has zero knowledge of detection methods, while attacks use available victim-model information and are generated with ART.
  • Threat model and attacks: White-box evaluation includes L0, L1, L2, and L∞ attacks, with JSMA, PGD, CW, FGSM, BIM, and DeepFool among the tested methods.
  • Threat model and attacks: Black-box attacks include PA, SA, HopSkipJump, and spatial transformations, while gray-box attacks use surrogate models with modified architectures.
  • Threat model and attacks: All attacks are untargeted, and the study also tests robust attacks designed to bypass detectors, including a method targeting LID.
  • Detector evaluation: Eight popular detectors from different categories are retested under the same environment using publicly available source code and specified configurations.

6 Results and discussions

The evaluation compares eight detectors across white-, black-, and gray-box attacks on four datasets. Detection performance varies substantially by attack type, dataset, and detector, with density- and uncertainty-based methods often struggling on high-dimensional or transferable attacks.

  • Experimental setup: The experiments report detection rates for white-, black-, and gray-box attacks across MNIST, CIFAR, SVHN, and Tiny-ImageNet.Tables cover detection rate and false positive rate for white-box attacks, detection rate for gray-box attacks, and a summary across scenarios.
  • KD+BU: KD+BU detects BIM and PGD-L∞ well on CIFAR, SVHN, and Tiny-ImageNet, but performs medium to poorly against several other attacks.It also performs well against JSMA on MNIST and SVHN but not on CIFAR and Tiny-ImageNet.
  • KD+BU: KD+BU is generally ineffective against black-box attacks, although it performs acceptably for some HopSkipJump and PA cases on MNIST and SVHN.Its performance is poor on CIFAR and Tiny-ImageNet for several black-box attacks.
  • LID: LID detects several white-box attacks effectively but is less effective on high-dimensional Tiny-ImageNet and attacks with very small perturbations.Its main limitation is high false-positive rates for some PGD and CW training attacks on datasets other than MNIST.
  • LID: LID has dataset- and attack-specific black- and gray-box performance, including about 60% performance for HopSkipJump on datasets where it is not effective.Gray-box performance is poor for some transferable BIM and PGD attacks.

6.3 NSS [112]

NSS-based detectors show strong results on some white-box settings but limited generalization to black-box attacks. Their performance is also sensitive to noisy data and the diversity of adversarial examples used during training.

  • White-box attacks: NSS detection performs well on MNIST except against CW, but performs poorly against several attacks on other datasets.The evaluation trained the detector using the whole testing data of successful attacks, unlike prior work using 1000 PGD adversarial examples.
  • White-box attacks: NSS has low false-positive rates except on Tiny-ImageNet, where noisy training images reduce performance.The cited results concern white-box evaluations summarized in the detector tables.
  • Black-box attacks: PGD-based NSS features generalize poorly to black-box attacks, with effectiveness limited to selected attack–dataset combinations.The detector is effective only against SA on MNIST and Tiny-ImageNet, and only against ST on SVHN.
  • Gray-box attacks: NSS shows comparable gray-box performance with slight improvement, suggesting that adversarial examples from different models could improve the detector.The passage specifically recommends including adversarial examples generated from different models and noisy samples as original training data.
  • Feature squeezing: Feature squeezing performs effectively on MNIST but medium to poorly on other datasets because the implemented squeezer targets small distortions.Its false-positive rate is around 5%.
  • Feature squeezing: Feature squeezing detects HopSkipJump effectively except on Tiny-ImageNet.The detector’s gray-box threshold is estimated from a random testing-data subset and requires enough data.

6.5 MagNet [67]

MagNet’s detection performance depends on attack distortion and dataset, while its denoising defense can still correctly classify some adversarial inputs. It maintains very low false-positive rates but performs poorly for many black-box attacks.

  • White-box attacks: MagNet detects high-distortion white-box adversarial examples well on MNIST and SVHN because they produce high reconstruction error.Small-distortion examples on CIFAR and Tiny-ImageNet have reconstruction errors comparable to clean samples.
  • White-box attacks: MagNet has very low false-positive rates across datasets, which is advantageous for avoiding unnecessary processing of clean images.The evaluation reports detection accuracy only, so results are not directly consistent with the original paper’s evaluation.
  • Black-box attacks: MagNet is effective against SA except on SVHN but performs poorly against other black-box attacks.Its denoising defense can re-project adversarial examples into the training-data manifold for correct classification.
  • Gray-box attacks: MagNet is treated as an unsupervised detector, with gray-box differences attributed to threshold calculation from 5000 training samples.The passage expects gray-box performance to be comparable to white-box performance.

6.7 SFAD [127]

SFAD combines multiple detection signals and performs especially well against black- and gray-box attacks. Its white-box performance is strongest on MNIST and varies across attack families on other datasets.

  • White-box attacks: SFAD combines selective uncertainty rejection, confidence probabilities, DNR, and bi-modal mismatch detection in an ensemble detector.The ensemble can be tuned to improve detection rate and false-positive rate.
  • White-box attacks: SFAD achieves the best overall evaluation on MNIST among the tested detectors, while its performance on other datasets ranges from effective to poor or medium.It performs effectively against FGSM, CW, DF, and JSMA but poorly to moderately against BIM and PGD.
  • Black-box attacks: SFAD effectively detects all tested black-box attacks.The reported strength comes from confidence-based processing of selective-classifier features using autoencoders, up/down sampling, and noise addition.
  • Gray-box attacks: SFAD performs better against gray-box attacks than white-box attacks on all datasets except PGD-L∞ on CIFAR.It generally outperforms the other detectors in gray-box detection.

6.9 Performance on high resolution dataset

The review finds that high-resolution datasets expose major scalability, false-positive, computational, and latency challenges for adversarial-example detectors. ImageNet evaluations show substantial variation across detectors and attacks, while deployment costs remain application dependent.

  • ImageNet evaluation: Few detectors were tested on ImageNet, a 14-million-image, 1000-class dataset whose images are commonly downsampled for model and computational requirements.The review frames ImageNet as a high-resolution evaluation setting.
  • Detector results: 82% average detection rate with around 14.5% FPR was reported for LID, which does not scale well on ImageNet white-box and black-box attacks.The review attributes the difficulty partly to image noise obscuring clean–adversarial boundaries.
  • Detector results: 84% detection with 6.2% FPR was reported for NSS on ImageNet against CW attacks, whereas FS reached 43% and 64% detection for FGSM and BIM, respectively.These results illustrate attack-dependent detector performance.
  • Deployment costs: NIC performed well on ImageNet but incurred 14.6% FPR on ResNet50 and up to 28% runtime overhead, while MagNet and denoiser-based methods face scaling or training-cost limitations.SFAD and DNR require multiple classifiers, making detection time consuming to train.
  • Deployment costs: Complexity, overhead, and inference latency are application dependent; parallel feature extraction can reduce latency, but FS, MagNet, and NIC can compromise real-time constraints.Unsupervised methods may also require extra models and storage, increasing deployment overhead.
  • Deployment costs: LID, NSS, and SFAD can exhibit high false-positive rates on Tiny-ImageNet, reinforcing that no detector category is uniformly preferable.The review states that false-positive performance depends on the implemented algorithm.

6.12 Content, CNN, and Detectors related discussion

The paper visualizes how image content, classifier representations, and attribution maps differ across clean and adversarial inputs. It reports dataset-dependent saliency patterns and notes that small attribution changes can make detection harder.

  • Visualization setup: The visual analysis uses Grad-CAM and back-propagation saliency to examine clean or attacked images, intermediate classifier outputs, heatmaps, and combined visualizations.Figures 5–8 organize these views across MNIST and CIFAR examples.
  • Content and saliency: MNIST saliency regions are more restricted to regions covering the image, whereas CIFAR saliency extends beyond the target object into surrounding texture.The comparison links attribution extent to dataset content.
  • Content and saliency: For CIFAR, emphasis on non-relevant surrounding regions can distract the CNN, and small perturbations can substantially alter saliency and shift the targeted prediction class.The paper presents these as consequences of broader saliency regions.
  • Detection difficulty: When an adversarial example’s important regions differ only slightly from the clean image, detector performance becomes harder to achieve.The passage states this observation without specifying a quantitative measure.

7 Challenges, future perspectives, and conclusion

The paper concludes that adversarial-example detection remains unresolved because detectors lack robust generalization and often trade off efficiency, latency, overhead, or training requirements. It recommends continued work on scalable, lightweight, and broadly generalizing methods.

  • Conclusion: The paper concludes that most detectors are not robust against unknown attacks, and that carefully designed perturbations can also defeat existing defenses and detectors.It therefore identifies robust detection and defense as unresolved research problems.
  • Supervised versus unsupervised detection: Supervised detectors generally evaluate better using labeled clean and adversarial data, but their performance is restricted to known attacks and may not generalize across attack types.Unsupervised methods rely only on clean data but face difficult hyperparameter and feature-selection challenges.
  • Future perspectives: Detector generalization must cover white-, black-, and gray-box attacks as well as counter-counter attacks, but reported results show that substantial research remains necessary.The paper defines generalization broadly rather than limiting it to one attack setting.
  • Future perspectives: Lightweight detectors should minimize overhead, inference latency, and training time, yet reviewed methods compromise at least one of these factors.The paper recommends methods that trade off these deployment factors deliberately.
  • Future perspectives: Ensemble detection is presented as a promising direction because combining techniques can improve results with a small false-positive-rate cost.SFAD is cited as an example combining selective or uncertainty, softmax-based, and bi-match predictions.
  • Future perspectives: High-resolution data such as ImageNet remains challenging because detectors may become ineffective, produce high FPR, require high overhead, or become time consuming.The paper calls for effective detectors designed specifically for high-resolution data.
  • Conclusion: The review categorizes detectors as supervised or unsupervised, subdivides them into six methodological groups, and experimentally evaluates eight algorithms across detection and deployment criteria.The evaluated criteria include detection rate, false-positive rate, complexity, overhead, and inference latency.
Loading 2105.00203v4…