Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods

Nicholas Carlini; David Wagner

Source-linked AI summary

Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods

Nicholas Carlini, David Wagner

arXiv:1705.07263v2May 20, 2017cs.LGcs.CRcs.CV

TL;DR

Neural networks can misclassify inputs that remain close to natural images, while proposed detection defenses have not been thoroughly tested against adaptive attacks. The paper evaluates ten detection schemes using consistent threat models and defense-specific loss functions, finding that all can be evaded and that adversarial examples are difficult to distinguish from natural images. It concludes that no known intrinsic properties currently separate adversarial examples from regular images and recommends broader defense evaluations.

Problem
Neural networks are vulnerable to visually similar adversarial examples, and existing defenses have generally been ineffective at classifying them correctly.
Method
The paper evaluates ten proposed detection defenses under multiple threat models and constructs defense-specific loss functions for strong iterative attacks.
Results
None of the ten defenses withstands a white-box attack, and on CIFAR each can be bypassed with imperceptible adversarial examples.
Takeaways & Limitations
Adversarial examples are harder to detect than previously recognized, with no known intrinsic properties differentiating them from regular images.
Takeaways & Limitations
Security evaluations should not rely solely on MNIST, because defenses evaluated only there often fail or weaken substantially on CIFAR.

Abstract

from arXiv · show

Neural networks are known to be vulnerable to adversarial examples: inputs that are close to natural inputs but classified incorrectly. In order to better understand the space of adversarial examples, we survey ten recent proposals that are designed for detection and compare their efficacy. We show that all can be defeated by constructing new loss functions. We conclude that adversarial examples are significantly harder to detect than previously appreciated, and the properties believed to be intrinsic to adversarial examples are in fact not. Finally, we propose several simple guidelines for evaluating future proposed defenses.

1 INTRODUCTION

The paper evaluates adversarial-example detection defenses and finds that defense-specific attacks can evade all ten methods, leaving adversarial examples indistinguishable from natural images on complex datasets. These results challenge claims that adversarial examples possess intrinsic detectable properties and motivate stronger evaluation practices.

Ten detection schemes from seven papers are evaluated alongside other defenses in a consistent comparison.The study focuses on recent methods that attempt to detect adversarial examples after direct classification defenses proved ineffective.
On more complex datasets, adversarial examples remain completely indistinguishable from the original images.On simple datasets, evading detection slightly increases the distortion required.
The findings challenge the assumption that adversarial examples have intrinsic differences from natural images.The experiments are intended to separate intrinsic properties from artifacts of existing attack techniques.
The authors recommend stronger defense evaluations and caution that MNIST may have different security properties than CIFAR.They argue that evaluating defenses solely on MNIST is insufficient.
All existing detection methods can be broken in white-box and black-box settings by choosing attacker-loss functions tailored to each defense.The paper also reports that generic attacks can undermine several defenses even without targeting a specific detector.

2 BACKGROUND

This section defines adversarial examples, threat models, datasets, detection defenses, and the attack procedure used to evaluate them. It emphasizes targeted attacks, L2 robustness measurement, and the finding that detection defenses do not make CIFAR adversarial examples visually distinguishable.

Adversarial Examples: An adversarial example is close to a natural input but receives a different classification; this paper focuses exclusively on targeted examples.Targeted examples seek a chosen class t distinct from the original label, and robustness means that finding such examples is difficult.
Threat Models: The paper evaluates defenses under zero-, perfect-, and limited-knowledge threat models, differing in the adversary’s access to the detector and its parameters.The limited-knowledge setting assumes knowledge of the defense type and training procedure, but not the trained detector or exact training data.
Datasets: The evaluation uses MNIST and the more difficult CIFAR-10 dataset, with standard models achieving 99.4% and 91.5% accuracy, respectively.MNIST contains handwritten digits, whereas CIFAR-10 contains color images of ten object categories.
Detection Defenses: The surveyed defenses include neural-network classifiers, PCA-based detectors, statistical tests, and randomized or blurred input normalization.The study selects the first seven papers proposing adversarial-example detection defenses and examines ten schemes.
Evaluation Outcome: On MNIST, some defenses slightly increase the distortion required, but no defense makes CIFAR adversarial examples visually distinguishable from original images.Robustness is measured by the L2 distance to the nearest adversarial example.
Attack Procedure: The Carlini–Wagner L2 attack generates targeted examples by gradient descent over a loss balancing proximity to the natural input and successful target classification.Its loss compares the target logit with the next-most-likely class; κ controls confidence, while c balances distance and classification objectives.
Attack Procedure: The limited-knowledge attack is the most difficult adversarial setting because the attacker knows the defense type but not the detector’s parameters.This evaluation is meaningful when zero-knowledge attacks fail but tailored perfect-knowledge attacks succeed.

3 SECONDARY CLASSIFICATION BASED DETECTION

The evaluated secondary-classifier defenses can detect some non-evasive attacks, but tailored attacks evade them while preserving visually similar adversarial examples. Across MNIST and CIFAR, defense gains are limited, and weak attacks can substantially overstate robustness.

Defense designs: Three studied approaches train a second classifier or adversarial class to distinguish adversarial from natural inputs.Grosse et al. add an N + 1st adversarial class, while Gong et al. train a binary detector D.
Zero-knowledge evaluation: 98.5% detection and 98% accuracy were achieved on previously unseen MNIST attacks, but these results came from attacks that did not target the detector.Both schemes also detected targeted attacks when trained on untargeted examples.
Perfect-knowledge evaluation: 100% attack success against Grosse’s defense increased mean L2 distance from 2.05 to 2.26, only a 10% distortion increase.The attack directly targeted the secured model, so the defense did not reduce adversarial-example success.
Perfect-knowledge evaluation: 100% attack success against Gong’s defense produced mean L2 distortion of 2.31, about 10% higher than the unprotected scheme.The combined function G lets C&W jointly fool the classifier and detector.
Zero-knowledge evaluation: 81% true positive rate at 28% false positive rate contrasted with 99% detection of simpler attacks, showing that weak attacks give a biased view of robustness.The best detector evaluated on C&W attacks achieved the lower true-positive rate and higher false-positive rate.

4 PRINCIPAL COMPONENT ANALYSIS DETECTION

PCA-based detection finds differences between MNIST natural and adversarial images, but those differences arise from dataset border-pixel artifacts rather than intrinsic adversarial properties. Restricting inputs to principal components does not provide robust protection against white-box attacks or outperform unsecured convolutional networks.

4.1 Input Image PCA: PCA reproduced a difference only in later MNIST components, while the first principal components showed no difference between natural and adversarial instances.The defense detects zero-knowledge attacks when the attacker does not attempt to evade it.
4.1 Input Image PCA: MNIST’s border pixels explain the PCA separation: natural images usually have zero-valued borders, whereas typical adversarial examples have non-zero border values.Because centered 24x24 images occupy a 28x28 grid, later components concentrate on boundary pixels.
4.1 Input Image PCA: The MNIST PCA difference is a dataset artifact, not an intrinsic property of adversarial examples; CIFAR shows no detectable difference in the corresponding evaluation.This motivates caution when drawing security conclusions solely from MNIST.
4.1 Input Image PCA: White-box attacks restricted to the first k principal components evade Hendrycks’s PCA defense while leaving later components unchanged.The attack directly targets the defense’s reliance on later-component changes.
4.2 Dimensionality Reduction: With convolutional architecture held fixed, limiting inputs to principal components was no more robust than using an unsecured convolutional network.The experiments therefore reject principal-component restriction as an effective defense.

5 DISTRIBUTIONAL DETECTION

The paper evaluates distributional detectors, including MMD and kernel density estimation, and finds that defense-specific attacks can evade them, especially on complex datasets.

Maximum Mean Discrepancy: MMD tests whether natural and adversarial image sets come from the same underlying distribution.The evaluation uses a polynomial-time approximation with Fisher’s permutation test.
Maximum Mean Discrepancy: MMD fails to detect targeted adversarial examples on MNIST with 100 images and on CIFAR even using all 10,000 test samples.In both cases, it fails to reject the null hypothesis at p > 0.05.
Kernel Density Estimation: Kernel density estimation models final-hidden-layer outputs with a Gaussian mixture model and flags inputs whose likelihood falls below a threshold.The likelihood is estimated using training instances assigned to the input’s predicted label.
Kernel Density Estimation: 80% of CIFAR adversarial examples receive higher likelihood scores than their original images, making the detector ineffective there.The defense works on MNIST but not on CIFAR under the reported evaluation.
Kernel Density Estimation: 3.81 mean distortion is required to evade the kernel-density detector on MNIST, compared with 2.05 on an unsecured network.The attack initially samples only 100 training instances, fewer than 5% of the total.

6 NORMALIZATION DETECTION

Normalization-based detectors can appear effective against some attacks, but adaptive loss functions bypass them; dropout randomization raises distortion while preserving imperceptible attacks on CIFAR.

Dropout Randomization: Dropout uncertainty applies the randomized network repeatedly and classifies inputs using a threshold on output variability.The procedure uses L = 30 evaluations and returns “adversarial” when U(x) > τ.
Dropout Randomization: 96% of MNIST adversarial examples from FGS and JSMA are detected with a false-positive rate below 1% at τ = 0.02.Against C&W examples, detection is 75% on MNIST and 95% on CIFAR.
Dropout Randomization: 98% of dropout-defense attacks succeed after optimizing a loss averaged across multiple fixed dropout realizations.The revised loss targets each sampled deterministic network rather than optimizing the randomized network directly.
Dropout Randomization: 3.68 mean l2 distortion is required on MNIST versus the 2.05 baseline, while CIFAR requires 1.1 distortion, five times larger but imperceptible.The MNIST distortion approaches human perceptibility, whereas the CIFAR examples remain entirely imperceptible.
Dropout Randomization: 90% of MNIST attacks transfer to a separately trained model, rising to 98% with only a 15% distortion increase.The authors describe this defense as not completely broken, although imperceptible CIFAR attacks remain possible.
Blurring: Blurring removes 80% of low-confidence adversarial examples at mean L2 distortion 2.05, but adaptive attacks restore the original mean distance.High-confidence examples require three times more distortion to remain adversarial after blurring.

7 LESSONS

The authors identify recurring weaknesses across defenses and recommend stronger, broader, and more transparent security evaluations.

Lessons: Dropout randomization is the most effective evaluated technique, making CIFAR attacks nearly five times more difficult while remaining imperceptible.Its difficulty approaches that of constructing transferable adversarial examples.
Lessons: Kernel density estimation is effective on MNIST but completely ineffective on CIFAR, showing that dataset-specific properties do not reliably transfer.Most defenses that increased MNIST distortion increased it much less on CIFAR.
Lessons: Defenses using additional neural networks are among the least effective because adversarial examples can fool both classifier and detector.The paper reports this pattern across all evaluated defenses.
Lessons: Pixel-level defenses provide reasonable robustness against weak MNIST attacks but fail against stronger attacks.The authors argue that simple raw-pixel detectors cannot capture the meaningful features extracted by neural networks.
Recommendations: Proposed defenses should be tested with strong attacks, including defense-aware white-box attacks that jointly optimize classification and detection objectives.The recommended white-box evaluation uses a differentiable combined objective and a strong iterative attack such as C&W’s.
Recommendations: Security evaluations should report true-positive and false-positive rates, preferably true-positive rate at 1% false-positive rate or a ROC curve.Detector accuracy alone can obscure whether useful detection is achieved at an acceptable false-positive cost.
Recommendations: Evaluations limited to MNIST are insufficient because defenses often fail or weaken substantially on CIFAR and may require harder datasets later.The authors recommend multiple datasets, potentially including ImageNet as defenses improve.

8 CONCLUSION

The paper argues that secure machine learning requires evaluating defenses against informed attackers, not merely reporting benchmark accuracy. Across ten defenses, defense-specific losses enable white-box attacks that produce imperceptible CIFAR adversarial examples.

Conclusion: Higher accuracy on a single benchmark is insufficient for security because attackers may adapt to the defense.The defense must be evaluated against an attacker who knows how it works.
Conclusion: Ten proposed defenses fail against white-box attacks constructed with defense-specific loss functions and a strong iterative attack.On CIFAR, the attacks produce imperceptible adversarial examples for every defense.
Conclusion: The authors conclude that existing defenses lack thorough security evaluations and that adversarial examples are harder to detect than previously recognized.The evaluations also expand the reported possibilities for constructing adversarial examples.