The Limitations of Adversarial Training and the Blind-Spot Attack

Huan Zhang; Hongge Chen; Zhao Song; Duane Boning; Inderjit S. Dhillon; Cho-Jui Hsieh

Source-linked AI summary

The Limitations of Adversarial Training and the Blind-Spot Attack

Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S. Dhillon, Cho-Jui Hsieh

arXiv:1901.04684v1Jan 15, 2019stat.MLcs.CRcs.CVcs.LG

TL;DR

Adversarial training can provide strong robustness near its training data, but its generalization to valid test points outside well-covered regions remains limited. The paper measures this limitation through test-point distance from the training-data manifold, introduces blind-spot attacks, and finds that high-dimensional datasets and some certified defenses are especially difficult to protect.

Problem
It is unknown whether robustness obtained on training data generalizes well to valid test points that lie in low-density regions of the empirical training distribution.
Method
The paper relates attack success to distance from the training-data manifold and constructs blind-spot attacks using transformations such as image scaling and shifting.
Results
The paper finds that adversarial-training effectiveness is highly correlated with training-data distance, while valid blind-spot images remain vulnerable to small-distortion attacks; blind-spots also occur in some certified defenses.
Takeaways & Limitations
Adversarial training faces substantial challenges on high-dimensional datasets because practical training sets cannot cover all blind-spots.
Takeaways & Limitations
The paper's divergence analysis provides only a rough characterization because the true divergence between training and test distributions in high-dimensional space is inaccessible in its setting.

Abstract

from arXiv · show

The adversarial training procedure proposed by Madry et al. (2018) is one of the most effective methods to defend against adversarial examples in deep neural networks (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks. Consequentially, an adversarial training based defense is susceptible to a new class of attacks, the "blind-spot attack", where the input images reside in "blind-spots" (low density regions) of the empirical distribution of training data but is still on the ground-truth data manifold. For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values. Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes defending on any valid test examples difficult due to the curse of dimensionality and the scarcity of training data. Additionally, we find that blind-spots also exist on provable defenses including (Wong & Kolter, 2018) and (Sinha et al., 2018) because these trainable robustness certificates can only be practically optimized on a limited set of training data.

1 INTRODUCTION

The paper examines why adversarial training generalizes unevenly, linking robustness to test-point distance from the training-data manifold and identifying blind-spot attacks in undercovered regions.

1 INTRODUCTION: Adversarial training achieves strong MNIST robustness but is substantially less effective on CIFAR-10 under simple PGD attacks.On MNIST, attacks reduce accuracy from approximately 98% to 88%; on CIFAR-10, 20-step PGD reduces it from 87% to below 50%.
1 INTRODUCTION: Test points can lie in low-probability regions of the empirical training distribution and therefore remain uncovered by adversarial training.The gap between empirical and true data distributions matters most when test examples are not well represented by training data.
1 INTRODUCTION: The paper shows that adversarial-training effectiveness correlates strongly with distance from test images to the training-data manifold.For MNIST and Fashion MNIST, most test images are close to training data; for CIFAR, robustness declines as distance increases.
1 INTRODUCTION: Blind-spot attacks target valid, human-recognizable images located far from training examples, where adversarial training provides weak robustness.The attack class concerns inputs that remain on the ground-truth data distribution while occupying low-density regions of the empirical training distribution.
1 INTRODUCTION: The results suggest that practical adversarial training cannot scale to high-dimensional datasets because finite training data cannot cover all blind-spots.The paper connects this limitation to the limited success of adversarial training on ImageNet, where many test images may be far from the empirical training distribution.

2 RELATED WORKS

Prior work developed multiple defenses against adversarial examples, but robust generalization beyond training data remains uncertain. The paper situates its analysis amid adversarial training, certified defenses, and theoretical work on sample requirements for robustness.

2.1 DEFENDING AGAINST ADVERSARIAL EXAMPLES: Adversarial examples threaten deep-learning applications, motivating defenses beyond standard model training.The related work discusses risks to applications including autonomous driving and face recognition.
2.1 DEFENDING AGAINST ADVERSARIAL EXAMPLES: Gradient masking, randomization, and earlier defense methods have been evaded or incur costs without fully defeating adaptive attackers.The cited work reports successful BPDA attacks against obfuscated-gradient defenses and residual vulnerability of randomized defenses.
2.1 DEFENDING AGAINST ADVERSARIAL EXAMPLES: Adversarial training formulates defense as min-max robust optimization and reports state-of-the-art performance on MNIST and CIFAR datasets.The procedure trains models with adversarial examples generated throughout training.
2.1 DEFENDING AGAINST ADVERSARIAL EXAMPLES: Certified defenses provide provable robustness guarantees and can achieve useful performance with computationally feasible training on medium-sized networks.The paper considers certified methods including those of Raghunathan, Sinha, and Wong and Kolter.
2.2 ANALYZING ADVERSARIAL EXAMPLES: Prior theory indicates that robust generalization may require substantially more samples than standard generalization.The related work cites results for Gaussian mixtures and extensions of PAC learning theory to adversarial settings.
2.2 ANALYZING ADVERSARIAL EXAMPLES: A central open question is whether robustness improvements on training data generalize to unseen test images.The paper notes that test robustness is commonly evaluated through certificates for norm-bounded distortions.
2.2 ANALYZING ADVERSARIAL EXAMPLES: The paper connects robust generalization to the distance between training data and test points and reports that tiny input-distribution shifts can destroy robustness.This connection extends the concern beyond exact training examples and standard test-set evaluation.

3 METHODOLOGY

The paper measures how far test examples lie from training data using deep feature embeddings and nearest neighbors, then examines dataset-level distribution differences with projected KDEs. It uses these measures to identify blind-spot inputs that remain valid yet expose weak robustness in adversarially trained models.

3.1 Measuring the distance between training dataset and a test data point: Deep feature embeddings are used because they capture image manifolds better than direct pixel distances or dimensionality-reduction methods alone.For complex datasets such as CIFAR, extracting a meaningful low-dimensional manifold directly from input space is challenging.
3.1 Measuring the distance between training dataset and a test data point: The proposed test-to-training distance averages the deep-embedding ℓp distances to a test example’s k nearest training examples.The metric is non-parametric, and correlations are similar across naturally and adversarially trained feature extractors.
3.2 Measuring the distance between training and test datasets: Dataset-level distance is estimated by applying a neural feature extractor, projecting representations with t-SNE, fitting KDEs, and numerically approximating K-L divergence.Training and test distributions are formed in the resulting low-dimensional embedding space.
3.2 Measuring the distance between training and test datasets: The KDE-based distribution comparison is only a rough characterization because the true divergence between general high-dimensional training and test distributions is inaccessible in this setting.The method uses Gaussian kernels and Scott’s rule for bandwidth selection.
3.3 The blind-spot attack: a new class of adversarial attacks: Blind-spot attacks search for valid images far from existing training examples, then find small-distortion adversarial examples despite adversarial training.The blind-spot images remain on the ground-truth data distribution and are classified correctly by the model before attack.
3.3 The blind-spot attack: a new class of adversarial attacks: For MNIST and Fashion-MNIST, scaling and shifting pixel values produces valid transformed images whose accuracy barely decreases while adversarial examples remain easy to find.The transformation changes contrast or adds a gray background before applying Carlini-Wagner attacks.
3.3 The blind-spot attack: a new class of adversarial attacks: About 50% of CIFAR-10 test images are reported as blind-spots, and data augmentation cannot enumerate all possible high-dimensional inputs because of the curse of dimensionality.The authors argue that limited training data mainly guarantees robustness near training examples.

4 EXPERIMENTS

Experiments evaluate adversarially trained models across MNIST, Fashion-MNIST, and CIFAR-10, relating attack vulnerability to distance from training data and testing scaling-and-shifting blind-spot attacks. Results show that greater distance predicts higher attack success, while visually similar transformed images retain accuracy but become substantially easier to attack.

Experimental setup: Experiments use adversarially trained models for MNIST, Fashion-MNIST, and CIFAR-10, with C&W attacks and embedding-space nearest-neighbor distances.The study also tests a scaling-and-shifting blind-spot attack on MNIST and Fashion-MNIST.
Distance and robustness: Successful attacks concentrate among test images farther from the training set, and attack success rates increase with this distance across all three datasets.The correlation is similar whether natural or adversarially trained networks provide the feature extractor.
Distributional divergence: Fashion-MNIST has the strongest defense and smallest training-test K-L divergence, whereas CIFAR has larger divergence and only limited adversarial-training success.The reported ordering places MNIST between Fashion-MNIST and CIFAR in robustness difficulty.
Distributional divergence: Adversarial training effectiveness does not depend on accuracy: Fashion-MNIST is harder to classify than MNIST but easier to robustly train because its data distribution is more concentrated.The authors associate this concentration with fewer blind-spots.
Blind-spot attack: Scaling and shifting preserves test accuracy while moving MNIST and Fashion-MNIST images into blind-spots where C&W finds adversarial examples with much smaller perturbations.The experiments use MNIST α values {1.0, 0.9, 0.8, 0.7} and Fashion-MNIST α values {1.0, 0.95, 0.9}, with β equal to zero or a small constant.
Blind-spot attack: Distance histograms change so little after transformation that blind-spot images cannot be reliably detected by this distance metric, despite a drastic robustness change.Figure 5 compares original and transformed test images using distances to the top-5 nearest training images.

5 CONCLUSION

The paper finds that adversarial-training effectiveness depends strongly on dataset characteristics and distance from the training-data distribution. It introduces blind-spot attacks and shows that scale-and-shift transformations can expose vulnerabilities, especially in high-dimensional datasets.

Adversarial-training effectiveness is highly correlated with dataset characteristics and test-point distance from the training-data distribution.
Blind-spot attacks target valid data points located far enough from the empirical training distribution to remain vulnerable despite adversarial training.
A simple scale-and-shift scheme finds blind-spot images on MNIST and Fashion-MNIST with high attack success rates.
Blind-spots make adversarial training challenging for high-dimensional datasets.

6 APPENDIX

The appendix examines how attack success relates to embedding-space distance, extends the analysis across datasets and neighborhood sizes, and visualizes blind-spot attacks against multiple robust-training methods.

Additional dataset: For GTS, attack success rates are higher when test examples are farther from the training data.The model uses the same structure and parameters as the adversarially trained CIFAR model, with ϵ = 8/255.
Blind-spot visualizations: Figures 10 and 12 visualize blind-spot attacks created by scaling and shifting MNIST and Fashion-MNIST images.The figures pair transformed inputs with adversarial examples and report perturbation distortion.
Certified defenses: Blind-spot attacks also target certified defenses by Wong and Kolter and by Sinha et al.The Sinha et al. model is evaluated with ℓ2 distortion because that is its studied threat model.