PaperlayerView exact revision on arXiv

Source-linked AI summary

On the (Statistical) Detection of Adversarial Examples

Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, Patrick McDaniel

arXiv:1702.06280v2cs.CRcs.LGstat.ML

TL;DR

Adversarial examples expose ML systems to malicious distribution shifts, especially in security-critical applications. The paper tests whether statistical differences reveal these inputs and augments models with an adversarial output class. Across datasets and attack strategies, the methods detect adversarial inputs, recover some correctly, or increase the perturbation cost required for attacks.

  • Problem

    ML models are vulnerable to adversarial examples, while their mitigation remains an open problem, particularly under changing attack strategies.

  • Method

    The paper uses statistical tests to distinguish adversarial from expected data and trains an additional model output class to recognize adversarial inputs.

  • Results

    The methods detect adversarial inputs across diverse datasets and strategies, with over 80% outlier-class success on two of three datasets and increased attack perturbation when detection is weaker.

  • Takeaways & Limitations

    Statistical differences between adversarial and legitimate data support model-agnostic detection and integrated defenses against adversarial inputs.

  • Takeaways & Limitations

    Robustness varies by dataset and attack method; for some JSMA cases, the augmented model does not achieve robustness to adversarial examples.

Abstract

from arXiv · show

Machine Learning (ML) models are applied in a variety of tasks such as network intrusion detection or Malware classification. Yet, these models are vulnerable to a class of malicious inputs known as adversarial examples. These are slightly perturbed inputs that are classified incorrectly by the ML model. The mitigation of these adversarial inputs remains an open problem. As a step towards understanding adversarial examples, we show that they are not drawn from the same distribution than the original data, and can thus be detected using statistical tests. Using thus knowledge, we introduce a complimentary approach to identify specific inputs that are adversarial. Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs. We evaluate our approach on multiple adversarial example crafting methods (including the fast gradient sign and saliency map methods) with several datasets. The statistical test flags sample sets containing adversarial inputs confidently at sample sizes between 10 and 100 data points. Furthermore, our augmented model either detects adversarial examples as outliers with high accuracy (> 80%) or increases the adversary's cost - the perturbation added - by more than 150%. In this way, we show that statistical properties of adversarial examples are essential to their detection.

1 INTRODUCTION

Adversarial examples exploit distribution shifts and pose serious risks for security-critical ML systems. The paper detects these shifts statistically and augments models with an adversarial outlier class to identify individual attacks and increase attack cost.

  • Adversaries can manipulate training or test distributions, creating severe consequences for security-critical ML applications.
  • Statistical testing distinguishes adversarial examples from expected data because the two distributions differ.The approach is evaluated across diverse datasets and adversarial-example algorithms.
  • Because statistical-test confidence diminishes for small suspicious sample sets, the model is augmented with an additional output class for adversarial inputs.This integrated mechanism targets individual adversarial examples rather than only malicious sets.
  • The test confidently detects adversarial samples across MNIST, DREBIN, and MicroRNA, including inputs generated by fast gradient sign and saliency map methods.
  • The augmented model assigns adversarial examples to the outlier class with over 80% success on two of three datasets.For the third dataset, the required perturbation increases by 150%, raising attack cost in all cases.
  • The mechanisms also perform well against black-box attacks, whose inputs are more likely to be detected than directly crafted attacks targeting the defended model.

2 BACKGROUND

The paper reviews standard ML classification assumptions and adversarial test-time attacks, then introduces two-sample statistical testing for comparing data distributions.

  • Machine Learning Classifiers: ML classifiers learn mappings from feature vectors to discrete labels, often using real-valued class probabilities to select the most likely label.
  • Machine Learning Classifiers: Training and test samples are typically assumed to come from the same underlying class-conditional distributions.
  • Adversarial Machine Learning: Adversarial test-time attacks add a perturbation δ to an input so the trained model assigns a different label while preserving task-specific constraints.
  • Adversarial Machine Learning: Existing defenses include adversarial training and defensive distillation, but both may fail under gradient masking, while game-theoretic approaches are computationally expensive.
  • Statistical Hypothesis Testing: Two-sample hypothesis tests compare samples from distributions p and q by testing H0: p = q against HA: p ≠ q using a p-value and threshold α.
  • Statistical Hypothesis Testing: The selected kernel-based test uses Maximum Mean Discrepancy and bootstrapping, making distribution comparison suitable for high-dimensional data without explicit probabilities.

3 METHODOLOGY

The methodology argues that adversarial examples form distributions distinct from learned training distributions, enabling statistical detection and an augmented outlier class for individual inputs.

  • Threat Model: The threat model covers both white-box and black-box adversaries, including strong attackers with access to model architecture, parameters, or training data.
  • Statistical Properties of Adversarial Examples: Adversarial examples are generated by perturbing legitimate training-distribution examples, forming a new class-conditional adversarial distribution.
  • Statistical Properties of Adversarial Examples: Because misclassified adversarial examples do not follow the learned training distribution, statistical tests can distinguish adversarial and benign samples.
  • Statistical Properties of Adversarial Examples: 50 misclassified adversarial examples per class suffice to observe a measurable distributional difference for existing crafting algorithms.
  • Detecting Adversarial Examples: Statistical tests require batches and cannot confidently identify adversarial examples on a per-input basis.
  • Detecting Adversarial Examples: The proposed augmented classifier adds an outlier class Cout and trains the model to classify inputs outside the learned class distributions as adversarial.

4 EXPERIMENTAL SETUP

The experiments evaluate statistical and model-integrated defenses across three datasets, multiple classifiers, and several adversarial-example crafting methods. The setup spans image, malware, and medical data while varying attack properties and model architectures.

  • Experimental questions: The experiments ask whether statistical tests distinguish adversarial from legitimate distributions, whether individual examples can be detected, and whether defenses withstand adaptive attacks.The setup uses several datasets, models, and attack algorithms to address these questions comprehensively.
  • Adversarial attacks: The attacks include FGSM, JSMA, an SVM attack, and a decision-tree attack with different targeting, sparsity, and computational-cost characteristics.FGSM perturbs along the output gradient, JSMA iteratively selects features for targeted misclassification, and the SVM and decision-tree attacks follow model-specific paths.
  • Models: The study uses decision trees, linear multi-class SVMs, neural networks, and logistic regression across the datasets.Neural-network architectures vary by dataset, while the SVM uses an l2 penalty and squared hinge loss.

5 IDENTIFYING ADVERSARIAL EXAMPLES USING STATISTICAL METRICS AND TESTS

Adversarial examples produce measurable distributional differences that enable two-sample statistical detection. Across most settings, confident detection requires about 50 examples, while class-wise comparisons can reduce this requirement and expose attack-dependent exceptions.

  • Statistical distinguishability: The experiments validate that adversarial examples have statistical properties significantly different from legitimate data.The statistical test detects adversarial examples confidently even with small sample sets.
  • Distributional metrics: MMD and energy distance often increase strongly for adversarial examples, with larger FGSM perturbations producing stronger changes and sparse attacks producing subtler changes.The JSMA changes about 20 pixels on average, whereas the decision-tree attack changes about 1 pixel on MNIST.
  • Sample-size requirements: 50 adversarial examples are sufficient for confident detection in most datasets and models.Confident detection means all 200 tests on randomly sampled examples rejected H0; the test generally accepts H0 for benign data around 95% of the time.
  • Exceptions: 100 examples are required for confident JSMA detection on a neural network, while MNIST decision-tree attacks remain undetected by the test.These exceptions correspond to attacks with less distinguishable statistical properties; an SVM attack can instead be detected with 10 examples.
  • Class-wise testing: Class-wise tests reduce the required sample size for several attacks, including JSMA to 50 examples on MNIST and 10 on DREBIN.Testing against the class into which examples are misclassified is generally more effective than testing against their original class, although MNIST FGSM reverses this tendency.

6 INTEGRATING OUTLIER DETECTION IN MODELS

Because sample-level statistical tests cannot identify which individual inputs are adversarial, the paper augments classifiers with an outlier class trained on adversarial examples. This mechanism detects many attacks or makes undetected attacks more costly, with some accuracy trade-offs on benign data.

  • Individual detection: The mechanism complements statistical testing because it identifies whether a given input is adversarial rather than only testing a sample set.The need arises because statistical-test confidence diminishes with the number of examples available in the analyzed set.
  • Outlier-class training: The outlier-class model adds an output label for inputs outside the expected distribution, enabling individual adversarial-example detection.An initial model generates adversarial training examples, which are combined with original data and assigned to a shared outlier class.
  • Benign-data impact: The defense slightly reduces benign-data accuracy, from 99.2% to 98.0%–98.8% on MNIST and by 0.2% on DREBIN.On MicroRNA, logistic-regression accuracy drops from about 95.4% to 93.4% with an outlier class.
  • MNIST results: On MNIST, the outlier-class model detects or recovers a large proportion of FGSM and JSMA examples, with detection increasing as perturbation magnitude increases.Recovery means assigning an adversarial example to its original class; remaining cases are errors.

7 PREVENTING THE ARMS RACE

The statistical test becomes less reliable when adversarial inputs are diluted among benign examples, motivating an outlier-class mechanism for individual detection. The augmented model is evaluated against adaptive and black-box attacks, including attacks using different crafting algorithms.

  • Robustness of the Statistical Test: Mixtures of adversarial-example types reduce detection confidence when one type is difficult to detect.Decision-tree adversarial inputs on MNIST are given as an example of a hard-to-detect type.
  • Robustness of the Statistical Test: The statistical test is less confident when few adversarial examples are hidden among many benign inputs.Confidence improves with a higher adversarial percentage or larger sample size.
  • Robustness of the Outlier Class: Adaptive detection is measured through recovery, outlier detection, and residual error rates on attacks differing from those used for training.Table 4 defines recovered examples as returning to their original class and detected examples as assigned to the outlier class.
  • Robustness of the Outlier Class: Black-box attacks are evaluated using substitute models, including substitutes trained with and without an outlier class.The setup targets transferability-based attacks that have evaded earlier defenses.
  • Robustness of the Outlier Class: The outlier-class model is trained with one attack type and tested against another to assess generalization to adaptive strategies.On MNIST, the model trained on JSMA examples is also robust to FGSM examples, whereas the reverse provides only slight improvement.

8 DISCUSSION

The discussion finds that statistical testing detects adversarial distribution shifts before model inference, while an outlier class improves robustness to adaptive attacks. Performance depends on perturbation size, attack type, and dataset characteristics, and broad robustness remains constrained.

  • Statistical Test: Statistical tests can detect adversarial inputs in feature space before they are fed to the classifier, making the detection model-agnostic.Larger perturbations are more likely to be confidently detected.
  • Integrating Outlier Detection: Adding an outlier class improves robustness to adaptive strategies and increases the perturbation needed for successful attacks.The discussion notes that robustness to JSMA is not achieved on some datasets.
  • Integrating Outlier Detection: Confusion matrices suggest that FGSM and JSMA examples occupy different regions relative to benign data.The authors suggest FGSM lies in a different halfspace, while JSMA may lie between benign classes.
  • Integrating Outlier Detection: Training on JSMA examples hardens the model against FGSM examples, but training on FGSM does not provide the same reverse robustness.The authors relate this asymmetry to FGSM being non-targeted and less optimal than JSMA.
  • Limitations: The proposed methods lack formal security guarantees, and constructing a robust classifier without assumptions about the adversary is described as practically infeasible.The authors leave trade-offs and broader attack coverage for future work.

9 RELATED WORK

The paper distinguishes its model-agnostic feature-space statistical test and additional outlier class from defenses tied to internal neural-network activations or previously unevaluated adversarial settings. It also systematically examines diverse attacks, datasets, models, and adaptive strategies.

  • Statistical Detection: The statistical test operates directly in feature space rather than relying on internal activations of a particular neural network.This supports model-agnostic detection.
  • Outlier-Class Defense: Unlike related approaches, the paper classifies adversarial examples in a separate additional outlier class.The comparison distinguishes this design from formal conditions and modified adversarial training.
  • Related Approaches: Earlier outlier-class and open-network approaches were not evaluated and motivated in adversarial settings.The paper positions its contribution around adversarial evaluation.
  • Model Scope: The paper differs from auxiliary-network detection by applying its outlier-class mechanism to any machine-learning model.The related auxiliary network shares parameters with the original model, whereas the paper emphasizes broader applicability.
  • Evaluation Scope: Experiments systematically cover different crafting algorithms, datasets, models, and adaptive black-box strategies.The comparison highlights attacks known to be difficult for previous defenses.

10 CONCLUSION

The paper validates statistical detection of adversarial examples before model inference and augments models with an outlier class for individual detection. These mechanisms show model-agnostic malicious properties and robustness to transferability-based attacks, while small perturbations may sometimes be recovered as their original class.

  • Conclusion: Statistical tests detect adversarial examples before they enter the ML model, supporting the model-agnostic nature of their malicious properties.The conclusion presents this as empirical validation of the paper’s central hypothesis.
  • Conclusion: An additional output classifies adversarial inputs as outliers within the augmented ML model.This mechanism is presented as a complementary approach to statistical testing.
  • Conclusion: The augmented model remains robust to transferability-based attacks that are harder to defend against than gradient-based strategies.The conclusion also notes that small-perturbation examples not detected as outliers are often correctly returned to their original class.
  • Future Work: Combining the proposed mechanisms with other defenses is suggested as a possible future direction.Defensive distillation is given as an example because it may increase adversarial perturbations.
Loading 1702.06280v2…