The Odds are Odd: A Statistical Test for Detecting Adversarial Examples

Kevin Roth; Yannic Kilcher; Thomas Hofmann

Source-linked AI summary

The Odds are Odd: A Statistical Test for Detecting Adversarial Examples

Kevin Roth, Yannic Kilcher, Thomas Hofmann

arXiv:1902.04818v2Feb 13, 2019cs.LGstat.ML

TL;DR

The paper asks whether adversarially manipulated examples can be reliably detected and corrected beyond defenses tailored to particular attacks. It proposes noise-calibrated statistical tests using log-odds and evaluates them against strong and defense-aware attacks. The reported experiments show high detection rates and accurate correction of test-time predictions.

Problem
Adversarial training improves robustness against specific attacks but leaves open protection against more cleverly designed counter-attacks and broader perturbations.
Method
The paper uses noise-induced changes in feature representations and log-odds to construct statistical tests, with thresholds calibrated from randomly corrupted inputs and optional logistic correction classifiers.
Results
99% detection is reported for an L∞-PGD white-box attack on CIFAR10 with FPR < 1%, and 99% detection on ImageNet with FPR 1%.
Takeaways & Limitations
The method can detect adversarial examples and, with high accuracy, recover corrected test-time predictions using access to model log-odds.
Takeaways & Limitations
The method's scope remains uncertain because it is open which model families follow the adversarial-cone paradigm and which models can be certified as immunizable.

Abstract

from arXiv · show

We investigate conditions under which test statistics exist that can reliably detect examples, which have been adversarially manipulated in a white-box attack. These statistics can be easily computed and calibrated by randomly corrupting inputs. They exploit certain anomalies that adversarial attacks introduce, in particular if they follow the paradigm of choosing perturbations optimally under p-norm constraints. Access to the log-odds is the only requirement to defend models. We justify our approach empirically, but also provide conditions under which detectability via the suggested test statistics is guaranteed to be effective. In our experiments, we show that it is even possible to correct test time predictions for adversarial attacks with high accuracy.

1. Introduction

The paper asks whether models can be protected against a broader range of adversarial perturbations than adversarial training handles. It proposes statistical detection and classification tests based on regularities induced by adversarial manipulations.

Adversarial examples are often imperceptible perturbations that can cause large changes in deep-network outputs.
Adversarial training improves robustness against specific attacks but does not address more cleverly designed counter-attacks.
The proposed method measures how feature representations and log-odds change under noise, exploiting directional variation associated with adversarial inputs.
99% detection with FPR < 1% is reported for an L8-PGD white-box attack on CIFAR10, with 96% clean and 92% adversarial corrected accuracy.
On ImageNet, the method achieves 99% detection with FPR 1%.
The paper contributes a statistical test for detecting and classifying adversarial examples, links perturbations to inverse problems, and evaluates detectability experimentally.

2. Related Work

Related work frames adversarial examples as small norm-bounded perturbations found by iterative optimization and surveys defenses that detect anomalous inputs or activations. Existing approaches include statistical tests, detector subnetworks, confidence estimates, and feature squeezing.

Iterative adversarial attacks: Adversarial attacks seek Lp-norm-bounded perturbations that cause misclassification, often using iterative optimization.
Iterative adversarial attacks: Projected Gradient Descent uses projection onto a constraint set, a small step size, and a loss function, with L∞ and L2 variants.
Iterative adversarial attacks: The Carlini-Wagner attack minimizes perturbation size plus a weighted attack objective, while the coefficient c trades off proximity against attack success.
Detection: Detection defenses identify characteristic regularities in adversarial perturbations or in the network activations they induce.
Detection: Prior methods include statistical testing, activation-based detector subnetworks, low-confidence tests, and prediction comparisons after feature squeezing.
Origin: The paper states that its method works regardless of whether adversarial examples arise from model flaws, learning objectives, generalization error, or high-dimensional statistics, provided characteristic feature regularities occur.

3. Identifying and Correcting Manipulations

The method probes inputs with random noise and tests standardized changes in pairwise log-odds to detect manipulations and infer their original classes. Noise can partially undo non-robust adversarial effects, producing separable statistics used for threshold tests and correction.

The framework considers natural inputs and adversarially perturbed inputs in a multiclass probabilistic-classification setting.
It uses continuous logits and pairwise log-odds, with the predicted class used at test time and the ground-truth class available during training.
Z-score standardization is applied separately to each class pair, and the statistic measures alignment between noise-induced feature changes and classifier weight differences.
For natural inputs, log-odds toward the true class are expected to remain stable under small noise because of robustness training or regularization.
If an adversarial perturbation is not noise-robust, noise can partially reverse its effect toward the true class, yielding separable standardized log-odds distributions.
Expected perturbed log-odds are thresholded to control false detections while maximizing adversarial-example detection, and a second-level logistic classifier can improve corrected labels.
The basic correction algorithm defines a new classifier for correcting test-time predictions on suspected manipulations.

4. Feature Space Analysis

The feature-space analysis characterizes optimal adversarial manipulation along class-weight directions and connects input-space attacks to approximate pre-image problems. It further motivates ray- and cone-like structures for adversarial perturbations.

Optimal feature-space directions: In feature space, an optimal attack targeting class z lies along the weight-difference direction Δw_z = w_z − w_y* under suitable constraint sets.The result follows because orthogonal components do not improve the change in the relevant log-odds.
Optimal feature-space directions: Successful class changes require the feature perturbation to have a nonnegative inner product with the target-class direction.This follows directly from the φ-linearity of the log-odds.
Pre-image problems: Because the network is treated as a black box, realizing an optimal feature perturbation in input space is framed as finding an approximate pre-image.The desired input perturbation makes φ(x + Δx) close to φ(x) + Δφ*; direct Jacobian inversion is unstable.
Pre-image problems: The Jacobian-transpose construction provides an input-space direction that locally improves alignment with a desired feature-space direction when the stated inner-product condition holds.Taylor expansion establishes that a sufficiently small step along this direction yields a better pre-image.
Pre-image problems: Gradient-based iterative attacks are closely related to solving the feature-space pre-image problem through the Jacobian-transpose method.This connection follows from applying the chain rule to losses defined on the feature representation.
Approximate Rays and Adversarial Cones: The analysis models optimal input perturbations as approximate rays and conjectures that angularly varying adversarial directions form cone-like structures detectable by the statistical test.The ray trajectory is x + tΔx, while commutativity between adversarial perturbations and random noise supports detection of adversarial cones.

5. Experimental Results

Experiments examine feature-space, decision-boundary, neighborhood, and adversarial-cone properties across several architectures and attacks. These results support detectability through characteristic noise-induced behavior rather than a trivial boundary-distance discrepancy.

Experimental setup: The experiments benchmark detection and correction across CIFAR10 and ImageNet models, using L∞-constrained PGD as the default white-box attack.The default budgets are ε∞ = 8/255 for CIFAR10 and ε∞ = 2/255 for ImageNet.
Induced feature space perturbations: Iterative adversarial attacks induce significantly larger feature-space perturbation norms and stronger weight-difference alignment than random noise.Noise-induced alignment is also significantly larger for adversarial examples than for natural examples.
Distance to decision boundary: 0.37 versus 0.27: adversarial examples have greater mean L2 distance to the nearest decision boundary than natural examples.The mean logit cross-over occurs at t̄ = 0.43, ruling out trivial boundary-distance proximity as the explanation for detectability.
Neighborhood of adversarial examples: 0.075±0.018 for L∞-PGD and 0.088±0.019 for L2-PGD: adversarial examples lie much closer to their unperturbed counterparts than to other neighbors.This neighborhood structure supports shifts toward the original class under random noise.
Adversarial cones: Simple noise addition generally fails to recover the natural class because its probability never becomes the highest among all classes.The natural-class probability can increase under noise without changing the predicted class.
Adversarial cones: Adversarial examples form a cone-like neighborhood in which noise-induced feature variation points toward the natural class.The cone is visualized using averaged softmax predictions over hyperplanes spanned by the adversarial perturbation and orthogonal random vectors; commutativity enables detection.

5.3. Detection rates and classification accuracies

The statistical test detects adversarial samples at high rates while maintaining a false-positive rate around 1%. A second-level correction method recovers source-class predictions with accuracies close to clean performance.

Evaluation measures: Detection rates are reported for the statistical test, while corrected-prediction accuracies are reported for the correction method.Detection rate corresponds to true-positive rate on perturbed samples and false-positive rate on clean samples.
Detection performance: Nearly all adversarial samples are detected while the false-positive rate remains around 1%.The detection test and basic correction algorithm are attack agnostic.
Correction performance: Within 5%: corrected accuracies on adversarial samples approach the corresponding clean-sample test accuracies.The second-level logistic classifier successfully reclassifies almost all detected adversarial samples to their source class.

5.4. Effective strength of adversarial perturbations.

As effective adversarial strength increases, uncorrected accuracy falls while detection improves enough for corrected classification to compensate across the tested range.

Effective strength is defined as attack success after independently accepting each perturbation entry with probability q.q = 1 recovers the usual adversarial misclassification rate; weaker attacks are expected to be harder to detect.
Uncorrected classifier accuracy decreases monotonically as attack strength rises, measured by either the attack budget ϵ8 or accepted-entry fraction q.
Detection increases at a rate that lets corrected classification compensate for declining uncorrected accuracy across the tested effective-strength range.
The remainder of the paper focuses on CIFAR10 models because of computational constraints.

5.5. Comparison to Adversarial Training

The paper compares its defense with adversarially trained classifiers using test-set and white-box attack accuracies.

The comparison reports test-set and white-box attack accuracies for adversarially trained models.The best-performing classifiers are shown in Table 4.
Adversarially trained models have significantly lower accuracy on adversarial samples and a considerably larger clean-sample performance drop.

5.6. Defending against unseen attacks

The method remains capable of detecting and correcting most adversarial examples from attacks not used to train its second-level logistic classifier.

The evaluation tests whether changing from the training attack can evade correct reclassification because unseen attacks may have different log-odds statistics.
The alternative attacks are L2-constrained PGD and L2-Carlini-Wagner.
Despite only a slight performance decrease, the method detects and corrects most adversarial examples from these previously unseen attacks.

5.7. Defending against defense-aware attacks

Against attacks crafted with full knowledge of the defense, performance drops more than for unseen attacks, but detection and corrected accuracy remain high relative to the undefended network.

A defense-aware attacker optimizes perturbations for performance in expectation under the defense’s controlled random noise.
Although performance drops more than against unseen attacks, detection rates and corrected accuracies remain remarkably high compared with the undefended network.
For L∞-PGD, the method achieves 99% detection with FPR < 1% and corrected accuracies of 96% on clean and 92% on adversarial samples, versus feature squeezing’s 55% detection with FPR < 5%.On ImageNet L∞-PGD, the method achieves 99% detection with FPR 1%, compared with feature squeezing’s 64% detection with FPR 5%.
Against defense-aware L2-CW attacks, the method achieves 71.4% detection with FPR 4.5%, whereas dropout randomization has a 2% detection rate.

6. Conclusion

The paper argues that adversarial examples occupy cone-like regions in specific directions and uses this structure to detect them and recover original labels from noisy log-odds behavior.

Adversarial examples exist in cone-like regions along specific directions from corresponding natural examples.
A statistical test of log-odds robustness to noise predicts whether a sample is natural or adversarial with high accuracy.
The method can recover the original class label when adversarial manipulation changes the test-time prediction.
Further architectural research is needed to explain the underlying cause of the observed phenomenon.

7. Appendix

The appendix documents implementation choices, threshold calibration, noise-based testing, correction classifiers, and experimental visualizations for the proposed detector.

Implementation details: Thresholds τ_y,z maximize detection rate at a prespecified false-positive rate below 1%, or below 5% in Sections 5.6 and 5.7.
Implementation details: Attack strengths are set to the lowest values after the sharp increase in successful attacks, targeting successful attacks on most datapoints.
Implementation details: The detector probes neighborhoods using Uniform, Bernoulli, and Gaussian noise sources with different magnitudes, drawing 256 samples per source and magnitude.
Implementation details: For detection, each noise-source distribution is tested separately and the final adversarial decision uses majority voting.
Wall time performance: At test time, processing N noisy versions makes robust classification scale linearly with N, while the remaining method has negligible overhead.Training requires the analogous operation over the training dataset, but computed statistics can then be stored for reuse.
Prediction correction: A logistic classifier trained on standardized perturbed log-odds Z-scores can improve corrected-label accuracy when several class scores exceed the threshold similarly.