Source-linked AI summary
Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing
Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang, Peixin Zhang
TL;DR
DNNs can be fooled by imperceptible adversarial perturbations, while many existing defenses are ineffective. This paper detects suspicious inputs at runtime by measuring label changes across mutated models and applying statistical testing. On MNIST and CIFAR10, the approach detected adversarial samples accurately and efficiently, including 96.4% and 90.6% with NAI mutants.
Problem
DNNs are vulnerable to human-imperceptible adversarial perturbations, and many proposed defenses have been found ineffective.
Method
The approach generates accurate mutated DNN models, measures an input’s label change rate, and uses statistical hypothesis testing to classify it as likely normal or adversarial.
Results
96.4% of MNIST and 90.6% of CIFAR10 adversarial samples were detected with NAI mutants, using averages of 74.1 and 86.1 mutations, respectively.
Takeaways & Limitations
Adversarial samples have significantly higher label change rates under model mutation, supporting runtime detection on MNIST and CIFAR10.
Takeaways & Limitations
The evaluation used a limited set of test subjects, and threshold-based detection produces false positives with a tradeoff between false positives and false negatives.
Abstract
from arXiv · showhide
Deep neural networks (DNN) have been shown to be useful in a wide range of applications. However, they are also known to be vulnerable to adversarial samples. By transforming a normal sample with some carefully crafted human imperceptible perturbations, even highly accurate DNN make wrong decisions. Multiple defense mechanisms have been proposed which aim to hinder the generation of such adversarial samples. However, a recent work show that most of them are ineffective. In this work, we propose an alternative approach to detect adversarial samples at runtime. Our main observation is that adversarial samples are much more sensitive than normal samples if we impose random mutations on the DNN. We thus first propose a measure of `sensitivity' and show empirically that normal samples and adversarial samples have distinguishable sensitivity. We then integrate statistical hypothesis testing and model mutation testing to check whether an input sample is likely to be normal or adversarial at runtime by measuring its sensitivity. We evaluated our approach on the MNIST and CIFAR10 datasets. The results show that our approach detects adversarial samples generated by state-of-the-art attacking methods efficiently and accurately.
I. INTRODUCTION
DNNs are vulnerable to adversarial samples, motivating runtime detection rather than relying solely on defenses or testing. The proposed approach uses mutated models and statistical testing to detect such samples efficiently and accurately.
- Motivation: Adversarial samples apply small, human-imperceptible perturbations that can cause DNNs to make incorrect decisions.This vulnerability affects systems including image classification and speech recognition, undermining safety-critical applications.
- Motivation: Data augmentation improves robustness to some extent but does not defend against unseen adversarial samples from different attack methods.
- Motivation: DNN testing metrics and testing strategies generate adversarial samples, but testing alone neither improves DNN robustness nor guarantees a well-tested model.
- Approach: The proposed runtime detector treats adversarial samples as more sensitive to DNN mutations, where mutated models are more likely to change their labels.It raises an alarm when an input is likely adversarial and can support rejection or further checking.
- Approach: The algorithm generates DNN mutants and applies statistical hypothesis testing to determine whether an input has high label change rate and is likely adversarial.
- Results: 96.4% of MNIST and 90.6% of CIFAR10 adversarial samples were detected with NAI mutants using averages of 74.1 and 86.1 mutations, respectively.The evaluation covered six groups of adversarial samples and four mutation operators.
B. Problem Definition
The paper frames runtime detection as deciding whether a deployed DNN’s input is normal or adversarial. The detector should also quantify confidence so suspicious inputs can be rejected or checked.
- Problem Definition: Existing defense mechanisms motivate an alternative because systematic evaluation found that most state-of-the-art defenses were ineffective.
- Problem Definition: Given an input x to a deployed DNN f, the problem is to decide efficiently and accurately whether f(x)=c_x or not.The first case is defined as normal and the second as adversarial.
- Problem Definition: The runtime decision should include confidence, enabling likely adversarial samples to be rejected or checked to avoid bad decisions.
III. OUR APPROACH
The approach measures how often model mutations change an input’s label, using DNN mutation testing to distinguish adversarial from normal samples. It focuses on model-level mutations that avoid retraining.
- Sensitivity Measure: Adversarial samples are hypothesized to be more sensitive than normal samples because mutated DNNs more often change their labels relative to the original model.This sensitivity is measured to differentiate the two sample types.
- DNN Mutation: Mutation testing generates multiple program mutations with operators and evaluates how many mutants are killed by a test suite.The paper adapts this idea to DNNs because traditional software mutation operators cannot be directly applied.
- DNN Mutation: The detector uses model-level operators that modify the original DNN directly, because source-level operators would require retraining mutated models from scratch.Four of the eight operators from prior DNN mutation work are adopted.
- DNN Mutation: Table I defines the DNN model mutation operators used to generate mutated models for the detection approach.
B. Evaluating Our Hypothesis
The study measures label change rate (LCR) under accurate DNN mutations and finds adversarial samples consistently more sensitive than normal samples. This supports using LCR to distinguish and potentially detect adversarial inputs.
- LCR measures how often mutated models assign an input a label different from the original model.Only mutated models retaining at least 90% of the original model’s test accuracy are kept.
- At every mutation rate, adversarial samples have significantly higher sensitivity values than normal samples.
- The LCR distance between normal and adversarial samples remains consistent across the additionally studied mutation operators.
- LCR can potentially distinguish whether an input is normal or adversarial at runtime.The proposed interpretation links higher mutation sensitivity with adversarial status.
D. The Detection Algorithm
The detection algorithm uses sequential hypothesis testing over labels produced by accurate mutated models. It stops when the observed LCR supports classifying the input as normal or adversarial within specified error bounds.
- The algorithm generates accurate mutated models and uses LCR to test mutually exclusive normal and adversarial hypotheses.
- The detector accepts inputs as adversarial or normal with Type-II or Type-I error bounded by β or α, respectively.An indifference region controlled by δ prevents immediate decisions near the threshold.
- Sequential probability ratio testing dynamically decides whether sufficient evidence exists to reject a hypothesis.The paper selects SPRT instead of fixed-size sampling for detection.
- The procedure tracks the number of mutated models that change the original label and the total number generated.
- The NAI operator inverses selected neuron activation statuses by multiplying their weights by -1.It randomly selects ⌈N × γ⌉ neurons, where γ is the mutation rate.
IV. IMPLEMENTATION AND EVALUATION
The authors implemented the approach as a self-contained Python toolkit and evaluated its accuracy and efficiency through multiple experiments.
- The approach is implemented in a self-contained toolkit with about 5k lines of Python code.
A. Experiment Settings
The evaluation uses MNIST and CIFAR10 classifiers, four mutation operators, and several adversarial-attack settings. Experiments retain accurate mutants and compare detection across generated and misclassified samples.
- Datasets and Models: MNIST and CIFAR10 use LeNet and GooglLeNet target models, with test accuracies of 98.3% and 90.5%, respectively.
- Mutated models generation: Four mutation operators are tested at rates {0.01, 0.03, 0.05} for MNIST and {0.003, 0.005, 0.007} for CIFAR10.The models contain 236 and 7914 neurons, respectively.
- Mutated models generation: For each mutation rate, the experiments generate 500 mutated models whose test accuracy is at least 90% of the seed model’s accuracy.
- Adversarial samples generation: The evaluation covers four state-of-the-art attacks from Cleverhans and Deepfool, including FGSM, JSMA, C&W, and Deepfool.
- Adversarial samples generation: Black-Box attacks use substitute models for MNIST and CIFAR10, followed by FGSM on the surrogate model.
- Adversarial samples generation: Each attack receives 1000 generation attempts, while wrongly classified testing samples are also sampled as adversarial examples.The resulting groups contain no more than 1000 generated adversarial samples per attack.
B. Evaluation Metrics
The evaluation measures how distinctly LCR separates adversarial from normal samples and how accurately the detector classifies them.
- dlcr = ςadv/ςnor measures the distance between adversarial and normal samples’ average label change rates.Larger values indicate a more significant difference.
- ROC curves evaluate LCR under model mutation across every possible classification threshold.They plot true positive rate against false positive rate.
- AUROC summarizes feature quality, with 1 representing a perfect classifier and values closer to 1 indicating better discrimination.
- Detection accuracy is the percentage of images correctly classified as normal or adversarial.Adversarial-sample accuracy equals tpr, while normal-sample accuracy equals 1 −fpr.
C. Research Questions
The research questions assess whether label change rate (LCR) under model mutation separates adversarial from normal samples, outperforms baselines, and supports accurate runtime detection at manageable cost. Experiments also examine mutation-rate and threshold effects.
- RQ1: Adversarial samples have significantly higher LCR than normal samples under model mutation.Across attack types and mutation rates, the LCR distance is mostly larger than 10 and can reach 375.
- RQ1: Increasing model mutation rates raises LCR for both sample types but reduces their separation.Mutation rates of 0.01 for MNIST and 0.003 for CIFAR10 produced the largest observed LCR distances.
- RQ2: LCR under model mutation beats two baselines in over half of the reported AUROC cases.The AUROC values are usually above 0.9; baseline 1 and baseline 2 win only 1 and 3 cases, respectively, excluding Deepfool.
- RQ3: The detection algorithm achieves high accuracy across attacks and mutation operators on MNIST and CIFAR10.For MNIST, average accuracies range from 83.9% to 96.4%; for CIFAR10, they range from 56.6% to 90.6%.
- RQ3: Increasing ρ lowers adversarial-sample detection accuracy but raises normal-sample accuracy and the number of mutations required.The authors recommend smaller ρ for applications with high safety requirements and larger ρ otherwise.
D. Threats to Validity
The evaluation uses a limited set of test subjects, and the detector has practical tradeoffs involving model accuracy, false alarms, and mutation cost. Normal samples typically require more mutations to classify.
- The experiments are based on a limited set of test subjects.
- Lower accuracy in the original or mutated models can reduce detection effectiveness and efficiency.The authors relate higher model accuracy to a larger LCR distance between adversarial and normal samples.
- The threshold-based detector produces false positives, requiring a tradeoff between false positives and false negatives through ρ selection.
- Normal samples typically need more mutations because the detector tests against ςnor and their LCR is generally lower.
V. RELATED WORKS
Related work spans adversarial sample generation, detection, robustness measurement, and testing or formal verification. These approaches address training robustness, runtime detection, robustness evidence, and safety guarantees through different mechanisms.
- Adversarial training: Adversarial training augments training data with adversarial samples to improve DNN robustness, but may overfit to particular attack strategies.
- Adversarial sample detection: Adversarial sample detectors use subnetworks, softmax-output differences, or behavior under random perturbations to identify samples a DNN may misclassify.
- Model robustness: Robustness research proposes metrics such as adversarial frequency, adversarial severity, neuron coverage, and its extensions.
- Testing and formal verification: Testing strategies include white-box, black-box, and mutation testing, while formal verification targets certain DNN safety properties.
VI. CONCLUSION
The paper proposes runtime adversarial-sample detection for DNNs by testing how input labels change under model mutations. Experiments on MNIST and CIFAR10 show accurate and efficient detection.
- The approach detects adversarial samples for deep neural networks at runtime.
- The method statistically checks an input’s label change rate under model mutations to distinguish likely normal and adversarial samples.
- Experiments on MNIST and CIFAR10 showed that the algorithm detects adversarial samples accurately and efficiently.