Adversarial Examples Are Not Bugs, They Are Features

Andrew Ilyas; Shibani Santurkar; Dimitris Tsipras; Logan Engstrom; Brandon Tran; Aleksander Madry

Source-linked AI summary

Adversarial Examples Are Not Bugs, They Are Features

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry

arXiv:1905.02175v4May 6, 2019stat.MLcs.CRcs.CVcs.LG

TL;DR

The paper addresses why adversarial examples are pervasive despite prior explanations not fully capturing observed behavior. It develops a framework for robust and non-robust features, demonstrates their existence in standard datasets, and rigorously connects adversarial vulnerability to data geometry and human-selected robustness.

Problem
Prior explanations of adversarial examples do not fully capture their observed behavior, leaving the reasons for their existence and pervasiveness unclear.
Method
The paper defines robust and non-robust features, disentangles them in standard datasets, and studies adversarial vulnerability through a theoretical Gaussian classification framework.
Results
The paper demonstrates that standard datasets contain predictive non-robust features and that these features alone can support good generalization.
Takeaways & Limitations
Adversarial examples are a human-centric consequence of models exploiting predictive features that are non-robust under human-selected similarity, so robustness and interpretability require human priors in training.
Takeaways & Limitations
The findings show that adversarial vulnerability arises from non-robust features but do not exclude other causes of adversarial vulnerability.

Abstract

from arXiv · show

Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

1 Introduction

The paper reframes adversarial examples as consequences of models relying on highly predictive but non-robust features, rather than merely aberrations of high-dimensional data or training noise. It supports this perspective by disentangling robust and non-robust features in standard datasets and by analyzing their geometric relationship to adversarial perturbations.

Adversarial vulnerability is presented as a direct result of model sensitivity to well-generalizing features in the data.
Standard classifiers use highly predictive signals, including imperceptible features that humans cannot understand, because training maximizes distributional accuracy.
The paper argues that reliance on non-robust features leads to adversarial perturbations that exploit this dependence.
The authors construct robustified datasets in which standard training yields good robust accuracy on the original, unmodified test set.This shows adversarial vulnerability is not necessarily tied to the standard training framework, but can also be a property of the dataset.
They also construct datasets whose inputs are nearly identical to originals but whose labels are associated only through small adversarial perturbations, while training still yields good accuracy on the original data.
A Gaussian classification task links adversarial vulnerability to differences between intrinsic data geometry and the adversary’s perturbation set.The same setting analyzes robust training and gradient misalignment with the inter-class direction.

2 The Robust Features Model

The paper formalizes robust and non-robust features by distinguishing predictive usefulness under ordinary data from usefulness that survives adversarial perturbations. Its framework explains why standard training uses useful features without distinguishing whether they are robust, while adversarial training explicitly suppresses useful but non-robust combinations.

Feature definitions: The framework defines robust features as predictive signals whose usefulness persists under a specified set of adversarial perturbations.A feature must first be ρ-useful, then remain γ-useful under perturbation to qualify as robust.
Feature definitions: Useful, non-robust features predict labels in ordinary classification but can lose or reverse their label correlation under adversarial perturbations.They are ρ-useful for some ρ bounded away from zero but are not γ-robust for any γ ≥ 0.
Training objectives: Standard training uses any ρ-useful feature because classification loss distinguishes features by predictive usefulness, not robustness.The framework therefore predicts that ordinary training will utilize useful non-robust features alongside robust ones.
Training objectives: Adversarial training accounts for perturbations in its loss and thereby prevents classifiers from learning useful but non-robust feature combinations.The adversary can exploit these features to degrade classification accuracy.
Limitations: The framework is predictive but does not fully capture the paper’s intuitive notion of non-robust features.It permits constructions combining useful robust features with useless non-robust features, whereas the experiments preclude such constructions; bridging this gap remains future work.

3 Finding Robust (and Non-Robust) Features

The experiments disentangle robust and non-robust features by constructing datasets that selectively preserve one type. Standard training on robustified data produces robust classifiers, while data whose label associations rely only on non-robust features still supports standard generalization.

Experimental design: The experiments construct robustified datasets containing primarily robust features and non-robust datasets whose useful input-label associations rely only on non-robust features.The two constructions test whether robustness depends on the dataset’s available features and whether non-robust features alone support generalization.
3.1 Disentangling robust and non-robust features: Standard training on a robustified dataset yields good robust accuracy on the original test set, showing that robustness can arise by removing non-robust features.The construction approximately preserves features used by a robust model while suppressing other useful signals.
3.1 Disentangling robust and non-robust features: Training on the non-robust dataset achieves good standard accuracy but almost no robustness, unlike training on the robustified dataset.The comparison supports a feature-based distinction between standard performance and adversarial robustness.
Conclusion: The dataset constructions support the claim that adversarial examples can arise from features in the data itself rather than exclusively from the training procedure.Filtering out non-robust features enables significantly more robust models through standard training.
3.2 Non-robust features suffice for standard classification: Non-robust features alone suffice for standard generalization, including when robust features are correlated with the wrong class in the constructed dataset.Standard training can select these features even when predictive robust features are also present.
3.3 Transferability can arise from non-robust features: Adversarial transferability is linked to shared non-robust features because independently trained models are likely to use similar brittle signals.The paper tests this by comparing architectures’ performance on the constructed dataset with transfer from a ResNet-50.

4 A Theoretical Framework for Studying (Non)-Robust Features

The paper formalizes adversarial vulnerability as a mismatch between the data-induced feature metric and the adversary’s ℓ2 metric. Its Gaussian analysis characterizes how robust training changes learned parameters, feature sensitivity, and gradient alignment.

Metric Misalignment: The Gaussian model represents learned parameters as Θ = (µ, Σ), inducing a Mahalanobis feature metric that need not align with the adversary’s ℓ2 metric.Adversarial vulnerability arises exactly from this metric misalignment.
Metric Misalignment: Theorem 1 identifies adversarial vulnerability as a consequence of misalignment between the inherent data metric and the ℓ2 metric.Small perturbations in the adversary’s metric can produce large changes under the classifier’s data-dependent metric.
Robust Learning: Robust optimization preserves the true mean, while increasing the ℓ2 perturbation budget makes the learned covariance more aligned with identity.This reduces sensitivity in some directions even when those directions remain useful for natural classification.
Gradient Interpretability: Robust training can make classifier gradients align better with the vector connecting class means under the ℓ2 inner product.Theorem 3 establishes this as a worst-case gradient-alignment property.
Robust Learning: Robust learning combines the inherent data metric with the adversary’s metric rather than simply improving quantitative classification performance.The adversary strength controls a trade-off between this enforced prior and the data-dependent objective.
Robustness and Accuracy: Robustness can conflict with accuracy in the analyzed setting, although related settings permit classifiers with perfect robustness and accuracy.The paper therefore treats the robustness–accuracy relationship as dependent on the data distribution and feature structure.

5 Related Work

The paper distinguishes its account from prior explanations that attribute adversarial examples to statistical anomalies, poor concentration, or other geometric and optimization effects. It instead treats them as well-generalizing but brittle features.

Related Work: The paper’s key distinction is that adversarial perturbations arise from well-generalizing yet brittle features, rather than statistical anomalies or poor statistical concentration.This contrasts with prior models based on finite-sample overfitting and high-dimensional phenomena.

6 Conclusion

The paper concludes that adversarial examples are a natural consequence of highly predictive, non-robust features in standard datasets. It supports this view empirically and theoretically, and argues that human priors must be encoded explicitly for robust, interpretable models.

Conclusion: The paper frames adversarial examples as a natural consequence of highly predictive but non-robust features in standard machine-learning datasets.These features generalize well while remaining brittle and incomprehensible to humans.
Conclusion: The authors support this hypothesis by disentangling robust and non-robust features and showing that non-robust features alone can support good generalization.They further analyze vulnerability, robust training, and gradient alignment in a theoretical setting.
Conclusion: The conclusion treats adversarial examples as fundamentally human-centered because robustness is evaluated under a human-selected notion of similarity.From the classifier’s perspective, non-robust features can be as important as robust ones.
Conclusion: The paper argues that robust and interpretable models require explicitly encoding human priors into the training process.Without this, explanations may be human-meaningful but not faithful while models rely on non-robust features.
Related Work: Prior explanations discussed in the paper include piecewise-linear boundaries, insufficient data, boundary tilting, noise robustness, local linearity, transferability, and dataset feature manipulation.The paper positions its non-robust-feature framework as a unifying conceptual account for several of these phenomena.

C.4 Constructing a Robust Dataset

The dataset-construction procedure uses trained standard or robust models to isolate the features relevant to each model. It then uses optimization and constrained adversarial perturbations to construct datasets emphasizing selected feature types.

Feature-Isolated Dataset Construction: The procedure constructs datasets containing features relevant only to a specified standard or robust model.It optimizes a training objective while initializing each robust-feature construction from another randomly chosen training sample.
Optimization Details: The optimization procedures use fixed-step normalized gradient updates and ℓ2-constrained projected gradient descent.The associated appendix tables specify the optimization and PGD parameters.
Robust Dataset: The robust dataset algorithm restricts the dataset to features used by a robust model.This construction is presented as a way to isolate robust features from the original data.
Non-Robust Dataset: The non-robust dataset algorithm makes input-label association depend entirely on non-robust features.The construction uses projected gradient descent to create constrained adversarial examples for dataset inputs.

D.1 Detailed evaluation of models trained on “robust” dataset

The evaluation constructs robust and non-robust datasets by distilling features from robust or standard models, then tests classifiers trained on these datasets under strong attacks.

Dataset construction: Robust datasets can be constructed from CIFAR-10 using features relevant to a robust model, rather than standard-model features.The construction can begin from random noise instead of random training images.
Dataset construction: 10k steps are used to construct the dataset from noise, compared with 1k steps when starting from another training image.
Robustness evaluation: 48.27% robust accuracy is reached after increasing PGD attack steps until accuracy plateaus.The same plateau is observed for PGD and CW-ℓ2 attacks across tested step sizes.
Robustness evaluation: Strong attacks reduce the model’s accuracy by no more than 2% across attack types and step sizes.The evaluation uses up to 2500 PGD steps and 1000 CW-ℓ2 steps.

D.3 Performance of “robust” training and test set

The experiments evaluate classifiers trained on robust and non-robust feature-based datasets across original, relabeled, shifted, and restricted-image settings, including transferability and geometric examples.

Evaluation caveat: Validation accuracy cannot determine early stopping because validation samples come from bDrand or bDdet rather than the true distribution D.Models derived from the robust model also show declining test accuracy during training.
Robust-dataset evaluation: Robust-feature datasets do not produce strongly predictive non-robust features, so classifiers trained on them largely learn the permuted labels.They achieve high accuracy on the relabeled test set rather than the original test distribution.
Distribution shift: 44% and 55% generalization are obtained on CIFAR-10.1 by classifiers trained on bDdet and bDrand, respectively.Both results demonstrate non-trivial generalization and exceed the linear-fit predictions based on CIFAR-10 performance.
Transferability: Adversarial transferability is associated with architectures that perform better on the standard test set when trained on the non-robust-feature dataset.The figure measures targeted transfer using attack success rate from a ResNet-50 to other architectures.
Geometric example: Robust training can yield a classifier that is both perfectly robust and accurate by forcing it to ignore a predictive x2 feature.The example attributes ERM vulnerability to reliance on non-robust features.

E.2 Outline and Key Results

The theoretical analysis studies adversarially robust maximum-likelihood classification for Gaussian distributions and derives results connecting perturbations, covariance learning, and classifier gradients to data geometry.

Key results: Theorem 1 expresses adversarial vulnerability as a mismatch between the data metric induced by the learned covariance and the adversary’s perturbation geometry.A fixed Lagrangian parameter C trades off negative-log-likelihood minimization and the adversarial constraint.
Key results: For fixed tr(Σ*) = k, the adversarial loss is minimized by an isotropic covariance Σ* proportional to the identity.
Key results: Robust learning preserves the true mean for sufficiently small ε and changes the learned covariance according to the perturbation budget.
Key results: Robust classifiers can have gradients whose maximum angle to the inter-class vector is smaller than that of standard classifiers.
Adversarial perturbation: The optimal perturbation is characterized explicitly using Lagrange multipliers, with λ chosen to satisfy the norm constraint.The perturbation is expressed in terms of v = x − µ.

E.3.2 Variant with Fixed Lagrangian (Theorem 1)

The fixed-Lagrangian analysis derives adversarial loss from Gaussian geometry, identifies covariance conditions minimizing that loss, and extends the argument to an expectation-constrained linear-operator adversary.

Fixed-Lagrangian formulation: The fixed-Lagrangian formulation replaces a hard norm constraint with a penalty parameter C trading off likelihood minimization and adversarial constraint.The constraint C ≥ 1/σmin(Σ*) ensures concavity.
Theorem 1: Theorem 1 gives the adversarial loss incurred by the non-robust maximum-likelihood parameters under this formulation.
Theorem 1: For fixed tr(Σ*) = k, the adversarial loss is minimized when Σ* is proportional to the identity.The proof reduces the optimum to equal eigenvalues.
Expectation-constrained variant: The optimal perturbation in the ℓ2-constrained motivating example is a linear function of v and can be written as Dv for diagonal D.
Expectation-constrained variant: In high dimensions, Gaussian norm concentration makes the expectation-constrained linear-operator problem equivalent to the hard-norm problem as d →∞.
Proof strategy: Danskin’s Theorem supplies subgradients by evaluating the inner objective at a current maximizer, while the T = Σ−1 and m = Σ−1µ parameterization makes the outer problem convex.

E.3.5 Applying Danskin’s Theorem

The derivation applies Danskin’s theorem to characterize the robust solution, yielding an implicit covariance formulation and an exact robust mean. It then analyzes the perturbation regime, showing that small perturbations produce a positive semidefinite adversarial covariance and increase variance on initially low-variance features.

Applying Danskin’s Theorem: Danskin’s theorem places the gradient at M = M∗ in the outer problem’s subdifferential, allowing the optimality condition to be set to zero.The derivation expands gradients with respect to T and m before solving the resulting equations.
Implicit robust solution: The minimax problem admits an implicit solution whose covariance formulation underlies the main results’ bounds.The implicit expression uses an adversarial perturbation M and a parameter λ dependent on Σ.
Mean and covariance: The robust maximum-likelihood estimate preserves the true mean, with µ = µ∗.The equations for µ and Σ are independent and can be solved separately.
Perturbation regime: For sufficiently small ε, the only admissible λ makes the adversarial perturbation M positive semidefinite.The proof establishes existence of ε0, and gives the sufficient condition ε < σmin(Σ∗)^2.
Perturbation regime: As ε increases within the valid regime, the robust covariance approaches the identity and assigns more variance to initially low-variance features.The robust model also adds uncertainty proportional to the square root of the initial variance.

E.3.7 Proof of main theorems

The main theorems bound the robustly learned covariance in a small-perturbation regime and use those bounds to establish improved gradient alignment. The proof combines bounds on λ with condition-number and positive-definite-matrix angle results.

Theorem 2: Robustly Learned Parameters: Theorem 2 shows that robust learning recovers the true mean and bounds the robust covariance for ε ∈ [0, ε0).The threshold ε0 is chosen so the admissible λ is unique in the relevant regime.
Proof of main theorems: The covariance bounds are derived by obtaining lower and upper bounds on λ and substituting them into the earlier implicit formulation.The proof simplifies the bounds using positive semidefiniteness and a rescaled perturbation parameter.
Theorem 3: Gradient alignment: Theorem 3 states that the maximum angle between classifier gradients and the class-connecting vector can be smaller for the robust model.The proof combines condition-number bounds with a result relating positive definite matrices to induced angles.
Theorem 3: Gradient alignment: The proof uses auxiliary lemmas controlling condition numbers and transformations by positive definite matrices.These lemmas are combined to show κ(Σ) ≤ κ(Σ∗), which supports the gradient-alignment result.