Source-linked AI summary
Detecting Adversarial Samples from Artifacts
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, Andrew B. Gardner
TL;DR
DNNs are vulnerable to deliberately crafted perturbations, raising the question of whether adversarial samples can be distinguished from normal and noisy inputs. The paper combines Bayesian uncertainty and deep-feature density estimates for attack-agnostic detection, achieving effective performance across attacks and datasets, including up to 92.6% ROC-AUC in a representative setting.
Problem
The paper asks whether DNNs can distinguish adversarial samples from normal and noisy counterparts despite adversarial perturbations being crafted to induce misclassification.
Method
The method combines Bayesian uncertainty estimates from dropout networks with density estimates in the last hidden layer's feature space.
Results
92.6% ROC-AUC is achieved by combining the two features in logistic regression, while performance is reported across varied attacks and image datasets.
Takeaways & Limitations
The two features provide complementary indicators and can be combined into an effective defense mechanism against adversarial samples.
Takeaways & Limitations
The study considers only convolutional neural networks, although the authors believe the approach may extend to other architectures.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) are powerful nonlinear architectures that are known to be robust to random perturbations of the input. However, these models are vulnerable to adversarial perturbations--small input changes crafted explicitly to fool the model. In this paper, we ask whether a DNN can distinguish adversarial samples from their normal and noisy counterparts. We investigate model confidence on adversarial samples by looking at Bayesian uncertainty estimates, available in dropout neural networks, and by performing density estimation in the subspace of deep features learned by the model. The result is a method for implicit adversarial detection that is oblivious to the attack algorithm. We evaluate this method on a variety of standard datasets including MNIST and CIFAR-10 and show that it generalizes well across different architectures and attacks. Our findings report that 85-93% ROC-AUC can be achieved on a number of standard classification tasks with a negative class that consists of both normal and noisy samples.
1. Introduction
DNNs are vulnerable to small, deliberately crafted perturbations, motivating detection of adversarial samples. The paper proposes density and Bayesian uncertainty features that achieve effective detection when combined.
- Research question: The paper investigates whether DNNs can distinguish adversarial samples from normal and noisy counterparts.The study frames detection as a response to adversarial vulnerability and ongoing efforts to design defenses.
- Motivation: Adversarial perturbations can cause high-efficacy DNNs to misclassify despite having small size.They are crafted by perturbing normal samples in directions that maximize misclassification.
- Approach: The proposed detector uses density estimates in last-hidden-layer feature space and Bayesian uncertainty estimates from dropout networks.Density estimates target points far from the data manifold, while uncertainty estimates target low-confidence regions.
- Result: 92.6% ROC-AUC is achieved by combining uncertainty and density features in a simple logistic regression classifier.The negative class contains both noisy and normal MNIST samples.
2. Background
DNN adversarial-sample research explains model vulnerability and develops defenses through robustness improvements or adversarial detection. Existing detector subnetworks can perform well but require training on generated attacks, limiting coverage of possible attack variants.
- Vulnerability: Adversarial perturbations can be imperceptible to humans while forcing high-confidence model misclassification.This contrasts with DNN robustness to random noise.
- Explanations: Proposed explanations include adversarial pockets, classifier linearity, and boundary tilting near the training-data manifold.The linearity explanation has been weakened by counterexamples.
- Defense strategies: Adversarial-defense research addresses either classifier robustness to attacks or detection of adversarial samples.These are presented as the two general categories of defense methods.
- Detection limitation: A detector subnetwork trained on generated adversarial samples may cover only a subset of possible attacks.Future attack modifications may therefore be able to overcome the system.
3. Adversarial Attacks
Adversarial attacks seek small perturbations that preserve human-recognizable content while causing model misclassification. The section reviews gradient-based, iterative, targeted, and optimization-based attacks with differing control and perturbation characteristics.
- Attack objective: An adversary seeks a minimal perturbation that makes an input misclassified while remaining correctly judged by humans.The adversarial input is formed by adding perturbation η to a normal image.
- FGSM: FGSM perturbs every input feature in the loss-gradient direction by magnitude ϵ.With small ϵ, it can fool MNIST and CIFAR-10 classifiers at high success rates.
- BIM: BIM extends FGSM by applying small adversarial perturbations iteratively within an ϵ-neighborhood of the original input.Clipping provides control over how far the sample moves past the classification boundary.
- JSMA: JSMA performs targeted misclassification by increasing the target-class output while decreasing outputs for other classes.It iteratively perturbs feature pairs selected by their adversarial saliency scores.
- Carlini–Wagner: Carlini–Wagner attacks use a shared optimization framework for L2, L0, and L∞ distance metrics.The L0 procedure reduces changed pixels by thresholding perturbations according to gradient importance.
- Attack effectiveness: 100% adversarial-sample generation success rates were reported for three attacks against distilled networks, compared with less than 1% for other techniques.The reviewed attacks were described as particularly effective against defensive distillation.
4. Artifacts of Adversarial Samples
The paper models adversarial samples as points that leave the source-class manifold while entering an incorrect classification region, then detects these artifacts using feature-space density and Bayesian uncertainty. Density tracks manifold proximity, while uncertainty addresses cases where adversarial samples remain near a class submanifold.
- Manifold intuition: Adversarial samples are modeled as off-manifold points that are classified as an incorrect class, with detectability depending on their distance from class submanifolds and decision boundaries.The paper distinguishes cases where the sample is far from the predicted-class submanifold, near it but far from the boundary, or near both.
- Density Estimation: Bandwidth selection controls density-estimate behavior: values that are too small produce spiky estimates, whereas values that are too large produce overly smooth estimates.Increasing the training-set size permits smaller bandwidths without making the estimate excessively spiky.
- Density Estimation: Kernel density estimation in the last hidden-layer feature space measures how closely a sample lies to the submanifold of its predicted class.The estimate uses training points from the predicted class and a tuned kernel bandwidth; the last hidden layer is chosen because it is expected to simplify the data manifold.
- Density Estimation: During BIM iterations, density decreases for the source class and increases for the incorrect class as the adversarial sample leaves the correct manifold and approaches the incorrect one.The density crossover is usually near the iteration where the predicted class changes.
- Bayesian Neural Network Uncertainty: Bayesian uncertainty from dropout provides a complementary signal because density estimation may fail when an adversarial sample lies very near the predicted-class submanifold.For MNIST, adversarial uncertainty distributions differ statistically from those of normal and equally perturbed noisy samples.
5. Experiments
Experiments evaluate density and uncertainty features, individually and jointly, for distinguishing adversarial samples from normal and noisy samples across datasets and attacks. The combined classifier shows strong detection performance, including ROC-AUC results reported across multiple datasets and attacks.
- Experimental setup: The evaluation covers MNIST, CIFAR-10, and SVHN using four attacks and equal-number noisy samples as a benchmark.Only correctly classified test samples are perturbed, and Figure 8 contrasts adversarial and noisy examples at equal perturbation size.
- Feature values: Adversarial samples typically increase uncertainty and decrease density estimates relative to corresponding original samples, with larger changes than random-noise perturbations.These feature shifts support using uncertainty and density as indicators for adversarial detection.
- Experimental setup: The experiment compares uncertainty, density, and combined classifiers for separating adversarial positives from jointly normal-and-noisy negatives.The combined classifier is logistic regression using uncertainty and density estimates as its two features.
- Adversarial sample classifiers: 92.6% ROC-AUC is achieved on MNIST when uncertainty and density are combined with both noisy and normal samples as the negative class.The logistic regression uses z-scored features before training.
- Adversarial sample classifiers: The combined classifier outperforms either uncertainty or density alone, indicating that the two features detect different qualities of adversarial samples.Across datasets and attacks, the combined classifier effectively detects adversarial samples; it most easily handles JSMA, BIM-A, and C&W in the individual-attack analysis.
6. Conclusions
The approach effectively detects adversarial samples using complementary density and Bayesian uncertainty features, with good performance across attacks and image datasets. The study, however, considers only convolutional neural networks.
- 90% or more ROC-AUC can be obtained in some cases when normal and noisy samples form the negative class.
- Kernel density estimates in the last hidden-layer subspace and Bayesian neural-network uncertainty estimates serve as the two detection features.
- The two features handle complementary situations and can be combined into an effective defense mechanism against adversarial samples.
- Performance is good across a wide variety of attacks and a range of image datasets.
- The work considers only convolutional neural networks, while extension to recurrent and other architectures is left for future work.