Robustness May Be at Odds with Accuracy

Dimitris Tsipras; Shibani Santurkar; Logan Engstrom; Alexander Turner; Aleksander Madry

Source-linked AI summary

Robustness May Be at Odds with Accuracy

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, Aleksander Madry

arXiv:1805.12152v5May 30, 2018stat.MLcs.CVcs.LGcs.NE

TL;DR

The paper investigates whether adversarial robustness necessarily improves standard generalization despite its additional costs. Through theoretical analysis and model-feature investigation, it finds a provable robustness–accuracy trade-off rooted in different learned representations, alongside potential perceptual benefits of robust models.

Problem
Deep models are vulnerable to small adversarial perturbations, raising whether the costly pursuit of robustness also preserves standard accuracy.
Method
The paper analyzes robustness and standard accuracy theoretically in a simple setting and examines feature representations and adversarial examples in models.
Results
A provable trade-off between standard accuracy and adversarial robustness can exist even in a simple, natural setting.
Takeaways & Limitations
Robust models learn different representations that align with salient data characteristics and human perception, producing clean feature interpolations.
Takeaways & Limitations
The analyzed task assumes no classifier can achieve both more than 99% standard and robust accuracy.

Abstract

from arXiv · show

We show that there may exist an inherent tension between the goal of adversarial robustness and that of standard generalization. Specifically, training robust models may not only be more resource-consuming, but also lead to a reduction of standard accuracy. We demonstrate that this trade-off between the standard accuracy of a model and its robustness to adversarial perturbations provably exists in a fairly simple and natural setting. These findings also corroborate a similar phenomenon observed empirically in more complex settings. Further, we argue that this phenomenon is a consequence of robust classifiers learning fundamentally different feature representations than standard classifiers. These differences, in particular, seem to result in unexpected benefits: the representations learned by robust models tend to align better with salient data characteristics and human perception.

1 Introduction

Adversarial robustness is difficult and costly, and it may conflict with standard accuracy rather than uniformly improving it. The paper argues this tension reflects different learned features, while robust models may align better with human-perceived data structure.

Small, imperceptible input perturbations can cause deep models to make highly confident but erroneous predictions.This brittleness has motivated methods for finding adversarial perturbations and training models to resist them.
Adversarially robust training can be more resource-consuming, yet it may not improve standard classification performance.The authors question whether paying these costs makes robust models preferable in every other aspect.
The paper shows that standard performance and adversarial robustness might be fundamentally at odds, with a provable trade-off even in a simple, natural setting.Robust training can still be beneficial when training data are limited.
Optimal standard and robust classifiers can learn fundamentally different features, and this difference persists even with infinite data.This motivates techniques specifically tailored to adversarially robust learning rather than relying only on classic tools.
Robust training can produce representations that align better with human perception and yield clean inter-class interpolations.The authors connect these interpolations to those found in GANs and other generative models.

2 On the Price of Adversarial Robustness

Adversarial robustness can impose computational and data costs while creating a fundamental trade-off with standard accuracy. The paper argues and proves this tension in a simple setting, while showing that robust training changes learned features in ways that may better align with human perception.

Adversarial training costs: Adversarial training requires additional computation and may require more training data, making robust model training more demanding.The method repeatedly computes perturbations during parameter updates.
Empirical trade-off: Adversarial training can help standard generalization with few samples but decrease standard accuracy as the training set grows.The reported effect is especially positive on MNIST in the low-data regime, then reverses after more samples are added.
Feature representations: The trade-off arises because robust and standard classifiers rely on fundamentally different features, including predictive non-robust features that adversaries can manipulate.The analyzed setting distinguishes a robust feature from non-robust features that are weakly correlated with the label but collectively improve standard accuracy.
Theoretical trade-off: A theoretical model proves that classifiers with high standard accuracy can have low robust accuracy against sufficiently strong ℓ∞ perturbations.For ε ≥2η, any classifier with standard accuracy at least 1 −δ has robust accuracy at most p/(1−p)δ.
Scope of the result: The tension can persist with infinite data because it is an inherent property of the data distribution rather than insufficient sampling.In the analyzed task, no classifier is both above 99% standard accuracy and above 99% robust accuracy.
Importance of adversarial training: Robust training is necessary for non-trivial adversarial accuracy in the theoretical setting, although it may reduce standard accuracy there.Standard-loss minimization can yield over 99% standard accuracy and under 1% adversarial accuracy, whereas adversarial-loss minimization yields both accuracies at p.

3 Unexpected benefits of adversarial robustness

Adversarially robust training can produce representations that align better with human perception and salient image characteristics. The evidence includes human-aligned loss gradients, class-like large-ε adversarial examples, and smooth inter-class interpolations.

Robust representations: Robust training embeds invariance to perturbations that humans are expected to ignore, shaping representations toward human perception.Robustness to all perturbations in the training set requires representations to be invariant to those perturbations.
Loss gradients: Robust-model loss gradients align with perceptually relevant features, whereas standard-model gradients appear noisy.The gradients visualize which input pixels most strongly affect loss and prediction.
Scope: The experiments are exploratory, and the proposed human-alignment approach warrants in-depth investigation.The claim depends on encoding an appropriate prior into the perturbation set.
Adversarial examples: Large-ε adversarial examples from robust models capture salient characteristics of another class and can appear to humans as samples from that class.The perturbation radius is much larger than the value used during training.
Adversarial examples: Gradient-based adversarial examples from robust models change images meaningfully, providing evidence against gradient obfuscation as the source of robustness.Following the gradient can eventually lead to images of different classes.
Inter-class interpolations: Linearly interpolating between an original image and its large-ε adversarial example can yield smooth, perceptually plausible transitions between classes.The paper compares this behavior with interpolations associated with generative models and other representation-based methods.

4 Related work

Related work has established robustness bounds and standard-versus-robust accuracy trade-offs in specific classifier families and synthetic settings. This paper studies the phenomenon through a different theoretical setting and feature distinction.

Prior theoretical work: Prior work proves robustness upper bounds and exhibits standard-versus-robust accuracy trade-offs for specific classifier families on a synthetic task.That setting also implicitly uses robust and non-robust features, but the features differ in magnitude rather than weak correlation.

5 Conclusions and future directions

The paper concludes that standard generalization and adversarially robust generalization can conflict because standard and robust models learn different features. It also identifies human-aligned representations and clean interpolations as benefits, while motivating further study of their relative costs and benefits.

Conclusions: The paper shows that standard accuracy and adversarial robustness can trade off even in simple settings because standard and robust models learn intrinsically different feature representations.This may explain the standard-accuracy drop observed with adversarial training.
Unexpected benefits: Robust models learn meaningful representations aligned with salient data characteristics, because adversarial perturbations encode a prior for human perception.Robustness to those perturbations makes classifiers invariant to input modifications humans are expected to ignore.
Future directions: Robust models yield clean feature interpolations resembling those obtained from generative models, motivating further work on the costs and benefits of standard and adversarial robustness.The paper emphasizes a possible connection between GANs and adversarial robustness.

A.1 Datasets

The experimental analysis uses MNIST, CIFAR-10, and a restricted ImageNet subset, with models ranging from a binary MNIST linear classifier to convolutional and ResNet architectures. Restricted ImageNet groups semantically similar classes into eight super-classes.

Datasets: The experiments use MNIST, CIFAR-10, and restricted ImageNet datasets.Restricted ImageNet is used because adversarial training is significantly harder for the full, challenging classification problem.
Datasets: Restricted ImageNet groups semantically similar classes into 8 super-classes for training and evaluation.The class groupings are listed in Table 1.
Models and training: MNIST uses a simple convolutional architecture, while CIFAR-10 uses a standard ResNet with four residual-layer groups.The CIFAR-10 filter sizes are (16, 16, 32, 64), with five residual units per group.
Models and training: Restricted ImageNet uses an unmodified ResNet-50, changing only the number of examples per training epoch from 1,280,000 to 76,800.The architecture is taken from the tensorpack repository.

A.3 Adversarial training

Adversarial training uses PGD-generated perturbations under specified ℓp threat budgets, but perturbations large enough to change image classes cannot be used for training. Mixing natural and adversarial examples slightly improves standard accuracy while usually decreasing robust accuracy proportionally.

Adversarial training: PGD adversarial training starts from randomly perturbed training data and considers ℓ2- or ℓ∞-bounded perturbations.The training and evaluation budgets are specified per dataset in Table 2.
Adversarial examples for large ε: Large perturbations that can change an image’s class make training against such adversaries impossible.The larger ε values used to create visible Figure 3 changes are listed in Table 3.
Mixed training: Including unperturbed examples in every batch slightly improves standard accuracy but usually decreases robust accuracy by a roughly proportional amount.The comparison uses half natural and half adversarial samples, as summarized in Table 4.

C Proof of Theorem 2.1

The proof constructs an adversary that swaps the distributions of several label-correlated features, then expresses standard and adversarial accuracy through conditional prediction probabilities. Under high standard accuracy, the resulting bound constrains adversarial accuracy.

Adversarial construction: An ε = 2η adversary shifts features x2, ..., x_d+1 so their label-conditioned distributions G+ and G− are exchanged.The same shift can change G+ to G− and vice versa.
Accuracy expressions: The quantities p++ , p−+ , p−−, and p+− encode the probability of predicting +1 for combinations of the first-feature sign and feature distribution.Standard accuracy is then written as a weighted combination of these conditional probabilities.
Accuracy expressions: Defining a = 1 − p++ + p−− and b = 1 − p−+ + p+− rewrites adversarial accuracy as 1/2((1 − p)a + pb).The definitions isolate the two conditional error combinations used in the proof.
Accuracy bound: A standard-accuracy assumption of at least 1 − δ implies pa + (1 − p)b ≤ 2δ.This inequality is used to upper-bound adversarial accuracy.
Accuracy bound: Using nonnegative probabilities and p ≥ 0.5, the proof obtains an upper bound on adversarial accuracy from the standard-accuracy constraint.The supplied derivation gives the bound in terms of δ and p.

D Proof of Theorem 2.2

The proof analyzes a soft-margin SVM whose optimal standard classifier relies heavily on a combined feature and achieves high standard accuracy, yet is highly vulnerable to an ℓ∞ adversary. Adversarial training instead removes weights from the vulnerable features.

SVM setup: The SVM objective combines a margin term with regularization, with λ tuned so the optimal solution has ℓ2-norm 1.The normalization is adopted for convenience and does not fundamentally constrain the proof.
SVM setup: Symmetry implies that an optimal solution assigns equal weights to features x_i for i = 2, ..., d + 1, allowing them to be replaced by their sum.Swapping identical features preserves the margin term, while averaging reduces the regularization loss.
Standard classifier: The learned classifier assigns more weight to the combined feature than to the first feature, so its predictions depend primarily on the combined feature.The argument uses the concentration of its normally distributed contribution.
Standard classifier: 99% standard accuracy is achieved by the soft-margin SVM learned for problem (5).With high probability, the combined-feature contribution determines the correct class independently of x1.
Adversarial vulnerability: Adversarial accuracy is at most 1% against an ℓ∞-bounded adversary with ε = 2η.The adversary shifts the combined-feature distribution so the classifier predicts the wrong class independently of x1.
Adversarially trained classifier: For ε > 2η, adversarial training produces a classifier assigning zero weight to features x_i for i ≥ 2.Removing any positive weight on these features decreases both the margin contribution and regularization term under the stated condition.
Adversarially trained classifier: The resulting classifier has standard and adversarial accuracy p for ε < 1 because the perturbation cannot change the sign of the first feature.This establishes the theorem’s robust solution in the specified setting.

E Robustness-accuracy trade-off: An empirical examination

The empirical MNIST analysis examines whether robust and non-robust features explain the theoretical accuracy trade-off. Robust classifiers discard weakly correlated pixels, whereas standard classifiers use them to increase confidence, including features that may reflect finite-sample noise.

Empirical examination: The MNIST experiment tests whether the theoretical tension between standard accuracy and adversarial robustness appears in a dataset suitable for linear classifiers.The analysis focuses on the distinction between robust and non-robust features.
Feature correlations: Under an ℓ∞ threat model, features are roughly useful to robust classifiers only when |E[yx_i]| ≥ ε.The perturbation budget therefore imposes an implicit threshold on feature-label correlations.
Feature correlations: Standard classifiers assign weights to weakly correlated pixels, while robust classifiers assign no weight below the adversary-determined correlation threshold.Figure 5 compares pixel correlations with learned weights and visualizes the resulting feature selection.
Finite-data effects: In finite-data settings, weakly correlated features can reflect noise, yet standard classifiers still exploit this apparent information by assigning them nonzero weights.The paper illustrates this with the expected imbalance from repeated unbiased coin tosses.

E.1 An alternative path to robustness?

The section considers whether robust classifiers can be trained with standard methods by exploiting differences between predictive power and adversarial vulnerability. It situates this question among theoretical and empirical work on robustness, generalization, and learned features.

E.1 An alternative path to robustness?: A feature’s predictive power can trade off against its vulnerability to adversarial perturbations, motivating standard training on selected input features.The proposed first step uses a standard linear classifier on MNIST pixels exceeding a correlation threshold.
E.1 An alternative path to robustness?: This work differs from bounds for fixed classifiers because it provides lower bounds for all classifiers learned in the statistical setting.The contrast concerns the setting analyzed rather than a particular architecture or classifier family.
E.1 An alternative path to robustness?: Unlike a prior infinite-data result where classifiers can be both robust and accurate, this setting exhibits a trade-off between standard and adversarial accuracy.The cited prior work concerns sample complexity for adversarially robust generalization, whereas the present trade-off persists in the infinite-data limit.
E.1 An alternative path to robustness?: The paper contrasts its setting with one where perfect standard accuracy is sufficient for perfect adversarial robustness.Here, adversarial training provably matters and standard and adversarial accuracy can conflict.
E.1 An alternative path to robustness?: The analysis is distinct from work connecting robustness to generalization because it studies the infinite-data limit and directly optimizes distributional loss.The cited direction concerns robustness implying generalization in a different sense.
E.1 An alternative path to robustness?: Empirical work has reported a similar accuracy–robustness trade-off across deep architectures on ImageNet.The cited observation concerns standard models rather than the paper’s statistical setting.

G Omitted figures

The omitted figures compare standard accuracy, adversarial examples, and input-gradient interpretability under standard and adversarial training. Together, they show accuracy changes with training-data size and adversary strength, while robust-model gradients align better with perceptually relevant features.

G Omitted figures: Adversarial accuracy measures the percentage of examples correctly classified after perturbation, evaluated here under an ℓ∞ threat model with ε = 0.20.The binary MNIST comparison evaluates both standard and adversarial accuracy for linear classifiers trained standardly or adversarially.
G Omitted figures: In low-data regimes, adversarial training can resemble data augmentation and sometimes improve generalization, particularly on MNIST.With sufficient training data, robust models have lower standard accuracy than the standard model.
G Omitted figures: Standard accuracy consistently decreases as the strength of the training adversary increases.The training adversary is constrained within an ℓp-ball of radius ε_train.
G Omitted figures: Large-ε adversarial examples are visualized under both ℓ∞-norm and ℓ2-norm constraints.The corresponding figures show examples bounded in each norm.
G Omitted figures: Input-loss gradients are significantly more interpretable for adversarially trained networks and align with perceptually relevant features.The visualization uses gradients with respect to input pixels for ten randomly chosen samples.