Disentangling Adversarial Robustness and Generalization

David Stutz; Matthias Hein; Bernt Schiele

Source-linked AI summary

Disentangling Adversarial Robustness and Generalization

David Stutz, Matthias Hein, Bernt Schiele

arXiv:1812.00740v2Dec 3, 2018cs.CVcs.CRcs.LGstat.ML

TL;DR

The paper addresses whether adversarial robustness and generalization are conflicting goals. It distinguishes adversarial examples by their relationship to an assumed low-dimensional data manifold and tests the resulting claims across synthetic and real datasets. The results support the possibility of models that are both robust and accurate, while showing that different model choices can have different robustness and generalization characteristics.

Problem
It is unclear whether deep networks can simultaneously achieve adversarial robustness and good generalization, with prior work proposing an inherent trade-off.
Method
The paper distinguishes regular unconstrained adversarial examples from on-manifold examples and studies them under a low-dimensional manifold assumption using synthetic and real datasets.
Results
The authors show that regular adversarial examples leave the manifold, on-manifold examples exist, on-manifold robustness is essentially generalization, and regular robustness and generalization need not conflict.
Takeaways & Limitations
Both robust and accurate models are possible, and on-manifold adversarial training can boost generalization when the manifold or relevant data invariances are available.
Takeaways & Limitations
On-manifold adversarial examples require ensuring label invariance, but the manifold lacks a notion of perceptual similarity and true posteriors are unknown.

Abstract

from arXiv · show

Obtaining deep networks that are robust against adversarial examples and generalize well is an open problem. A recent hypothesis even states that both robust and accurate models are impossible, i.e., adversarial robustness and generalization are conflicting goals. In an effort to clarify the relationship between robustness and generalization, we assume an underlying, low-dimensional data manifold and show that: 1. regular adversarial examples leave the manifold; 2. adversarial examples constrained to the manifold, i.e., on-manifold adversarial examples, exist; 3. on-manifold adversarial examples are generalization errors, and on-manifold adversarial training boosts generalization; 4. regular robustness and generalization are not necessarily contradicting goals. These assumptions imply that both robust and accurate models are possible. However, different models (architectures, training strategies etc.) can exhibit different robustness and generalization characteristics. To confirm our claims, we present extensive experiments on synthetic data (with known manifold) as well as on EMNIST, Fashion-MNIST and CelebA.

1. Introduction

The paper frames adversarial robustness and generalization as an unresolved relationship, then distinguishes regular off-manifold examples from on-manifold examples to challenge an inherent trade-off.

Adversarial robustness remains poorly understood, while existing defenses often incur computational overhead and reduced accuracy.
The manifold assumption proposes that adversarial examples leave an underlying low-dimensional data manifold, but experimental support and contrary findings remain limited.
The authors investigate whether robustness and generalization are inherently conflicting goals, amid theoretical and experimental claims supporting both a trade-off and its opposite.
The paper distinguishes regular unconstrained adversarial examples from on-manifold adversarial examples constrained to the data manifold.On-manifold examples must preserve the actual true label.
The stated contributions are that regular examples leave the manifold, on-manifold examples exist, on-manifold robustness is essentially generalization, and the two goals need not conflict.Experiments cover a synthetic dataset with a known manifold and EMNIST, Fashion-MNIST, and CelebA.

2. Related Work

Related work spans attacks, defenses, manifold-based explanations, and the disputed connection between adversarial robustness and generalization.

Attacks: Adversarial attacks include white-box methods with full model access and black-box methods with limited access to model queries.
Defenses: Proposed defenses include detection, preprocessing, dimensionality reduction, manifold projection, stochasticity, adapted architectures, ensembles, and adversarial training.Many defenses have been broken, often by considering specialized attacks.
Explanations: The manifold assumption became a widely accepted explanation that adversarial examples leave the data manifold, following earlier explanations based on rarity and network linearity.
Robustness and generalization: Prior work argues both that robustness and generalization trade off and that better generalization improves robustness, leaving their relationship contested.
Empirical context: Figure 2 compares regular and on-manifold adversarial examples across FONTS, EMNIST, Fashion-MNIST, and CelebA.FONTS has a known manifold, whereas the other datasets use VAE-GAN approximations; visual differences contrast random noise with concept changes.

3. Disentangling Adversarial Robustness and Generalization

The paper separates regular, potentially off-manifold robustness from on-manifold robustness, showing that the former need not conflict with generalization while the latter is closely tied to it.

3. Disentangling Adversarial Robustness and Generalization: Regular unconstrained adversarial examples leave the data manifold, whereas constrained on-manifold adversarial examples exist and can be computed with manifold approximations.The study uses class-specific VAE-GANs to perturb latent codes directly, rather than perturbing images and projecting them afterward.
3.4. On-Manifold Robustness is Essentially Generalization: On-manifold adversarial examples correspond to meaningful image manipulations and are treated as generalization errors because they have non-zero probability under the data distribution.This relationship is strongest on FONTS and EMNIST and weaker on F-MNIST when the learned manifold is less accurate.
3.4.1 On-Manifold Adversarial Training Boosts Generalization: On-manifold adversarial training boosts generalization, especially with few training images, while its benefit decreases as manifold approximation becomes more difficult.Identifying invariances and using adversarial transformation training recovers gains on EMNIST and F-MNIST relative to random augmentation.
3. Disentangling Adversarial Robustness and Generalization: Regular robustness is generally independent of generalization when training-data quantity varies, although different training strategies can produce different robustness-generalization characteristics.On FONTS, regular success rate is largely invariant to test error except under adversarial training.

4. Conclusion

The paper disentangles adversarial robustness from generalization by distinguishing regular attacks that leave the manifold from on-manifold attacks that relate to generalization.

Regular adversarial examples leave the underlying data manifold, while on-manifold adversarial examples can also be found using known or approximated manifolds.
Robustness against on-manifold adversarial examples is clearly related to generalization, motivating on-manifold adversarial training.
Regular robustness and generalization are not necessarily contradictory goals for any arbitrary but fixed model.
The study supports its claims with experiments on synthetic data and datasets including EMNIST, Fashion-MNIST and CelebA.

B. Experimental Setup

The supplementary experimental setup provides technical details for the synthetic FONTS dataset and the VAE-GAN implementation.

The setup details the synthetic FONTS dataset and the VAE-GAN implementation used in the experiments.

B.1. FONTS Dataset

FONTS is a controlled synthetic dataset built from transformed characters, providing explicit access to a differentiable data manifold; VAE-GANs approximate class manifolds on other datasets.

B.1. FONTS Dataset: FONTS contains randomly rotated characters “A” to “J” drawn from 1000 Google Fonts with substantial font variation.
B.1. FONTS Dataset: The dataset provides control over transformation parameters, font and class through a differentiable generative decoder.
B.1. FONTS Dataset: Class-specific VAE-GANs approximate class manifolds for FONTS, EMNIST, F-MNIST and CelebA.
B.1. FONTS Dataset: The VAE-GAN encoder models q(z|x) with a Gaussian parameterized by the image-dependent mean and variance, while λ weights reconstruction against KL and adversarial losses.
B.1. FONTS Dataset: The VAE-GAN architecture uses convolutional layers, fully connected layers and dataset- or class-specific reconstruction weights.
B.1. FONTS Dataset: Random VAE-GAN samples are often realistic on EMNIST and FONTS but also reveal discrepancies from the true data distribution, especially on FONTS.

C. Adversarial Example Distance to Manifold

The paper measures adversarial examples’ distance from known or approximated manifolds and finds that regular attacks are distinguishable from test images and on-manifold attacks.

C. Adversarial Example Distance to Manifold: The analysis compares regular and on-manifold adversarial examples by measuring their distance to the true or approximated manifold.
C. Adversarial Example Distance to Manifold: On FONTS, the closest manifold image is obtained by optimizing a latent code with the known decoder and measuring the L2 distance to the adversarial example.
C. Adversarial Example Distance to Manifold: When the true manifold is unavailable, the method locally approximates it using a subspace of 50 nearest neighbors centered at their mean.
C. Adversarial Example Distance to Manifold: The local projection solves a least-squares problem using the adversarial direction δ, rather than the adversarial example itself.
C. Adversarial Example Distance to Manifold: Regular adversarial examples are clearly distinguishable from test images and on-manifold adversarial examples, although the finding cannot be used for detection without paired test and adversarial images.
C. Adversarial Example Distance to Manifold: The distance-distribution gap between test images and on-manifold adversarial examples reflects the quality of the VAE-GAN approximation.

C.1. Intuition and Theoretical Argument

The paper argues that regular adversarial examples leave the data manifold because a well-generalized model's loss is approximately constant along it. Empirical examples and attack analyses support this geometric interpretation.

Theoretical intuition: A nearly constant loss on the manifold makes its gradient approximately orthogonal to the manifold.The gradient therefore points in a direction that efficiently increases loss away from the data manifold.
Empirical intuition: On-manifold success rate is strongly related to test error, whereas regular robustness is independent of generalization.This relationship is shown across the reported FONTS, EMNIST, and F-MNIST experiments.
Theoretical intuition: Gradient-based regular attacks consequently produce adversarial examples outside the manifold.This follows from the gradient of expected loss being orthogonal to the manifold's tangent space.
Empirical intuition: On FONTS, on-manifold adversarial examples reflect the data-generating transformations, unlike regular examples' seemingly random noise patterns.Similar observations hold on EMNIST and F-MNIST, although approximation artifacts become visible particularly on F-MNIST and CelebA.

E. L2 and Transfer Attacks

Across L2 white-box, transfer, and architectural settings, on-manifold robustness tracks test error while regular robustness is generally independent of generalization. Transfer attacks preserve this pattern despite substantially lower success rates.

Attack setup: The L2 Madry attack maximizes training loss within an L2 perturbation ball, with separate radii used for regular and on-manifold examples.The attack uses projected ADAM and clips image values to [0, 1].
L2 attacks: Regular Carlini–Wagner L2 adversarial examples also leave the manifold.Their distances to the true or approximated manifold are illustrated for FONTS and EMNIST.
L2 attacks: L2 Madry and Carlini–Wagner attacks show a clear relationship between on-manifold success rate and test error across FONTS, EMNIST, and F-MNIST.The relationship is mostly linear for L2 Madry and nonlinear for L2 Carlini–Wagner; regular success rate is roughly constant as test error varies, except for regular adversarial training.

F. Influence of Network Architecture

The relationship between on-manifold robustness and generalization extends beyond convolutional networks to multilayer perceptrons, ResNet-13, and VGG architectures.

Multilayer perceptrons: Multilayer perceptrons reproduce the claims that on-manifold robustness is related to generalization while regular robustness is mostly independent.The experiments use four hidden layers with 128 units per layer, ReLU activations, and batch normalization.
ResNet-13 and VGG: Experiments with ResNet-13 and VGG also confirm the reported robustness–generalization claims.The architectures were adapted for the 28 × 28 resolution of FONTS, EMNIST, and F-MNIST.

G. From Class Manifolds to Data Manifold

Class-agnostic VAE-GAN manifolds make label invariance difficult, limiting the effectiveness of on-manifold adversarial training when examples cross true class boundaries. Nevertheless, the robustness–generalization relationship remains visible under the broader data-manifold definition.

Manifold approximation: Class-agnostic VAE-GANs embed different classes in one latent space, making label invariance difficult to ensure.The paper therefore uses a small latent-space L∞ constraint of η = 0.1 as an attempted safeguard.
Training consequences: When label invariance cannot be ensured, on-manifold adversarial training loses effectiveness and generalization slightly decreases relative to normal training.The positive training effect diminishes when robustness is evaluated using on-data-manifold examples.
Training consequences: Despite this limitation, on-data-manifold robustness remains clearly related to generalization.This extends the observed relationship to a less strict definition of on-manifold adversarial examples.
Label invariance: On-manifold adversarial examples may cross class boundaries and change their true labels, invalidating them under the paper's definition.This limitation is illustrated for class-agnostic VAE-GANs.

H. Baselines and Adversarial Training Variants

The paper compares adversarial-training variants, random perturbation baselines, and regular versus on-manifold training across several datasets and representations. Training choices affect robustness and generalization differently depending on the adversary and task difficulty.

Baselines and variants: Regular and on-manifold adversarial training use variants differing in clean/adversarial-example proportions, optimization strength, and stopping criteria.The considered baselines also include uniformly random perturbations without optimization.
Training effects: Randomly perturbed examples are ineffective as a substitute for adversarially perturbed examples in both image space and latent space.
Training effects: For regular adversarial training, adversary strength primarily controls the robustness-generalization trade-off.The weak variant increases generalization while reducing robustness, and the effect depends on task difficulty.
Training effects: For on-manifold adversarial training, training variants have little effect: generalization changes slightly and regular robustness remains unchanged.
Experimental comparison: The experiments visualize regular and on-manifold adversarial examples for FONTS, EMNIST, F-MNIST, and CelebA, using known or approximated manifolds.

I. Definition of Adversarial Examples

The paper defines adversarial examples using the true conditional label under the data distribution, then uses this criterion to distinguish valid attacks from perturbations that change the label. This definition challenges the toy-dataset trade-off argument of Tsipras et al., while acknowledging that other definitions may differ.

Definition: An adversarial example changes the classifier’s prediction while preserving the data distribution’s most probable true label.Formally, f(˜x) ≠ y while p(y|˜x) exceeds p(y′|˜x) for every alternative label y′.
Definition: The definition is identical to the paper’s on-manifold definition, but its general applicability is left for future work.The discussion restricts toy examples to settings where the relevant probabilities are well-defined.
Illustrative example: In a binary point-mass example, shifting x = 0 with y = 1 to ˜x = ϵ changes the true label, so the perturbation is not a proper adversarial example.At x = ϵ, p(y = 1|x = ϵ) = 0 and p(y = −1|x = ϵ) = 1.
Discussion of [102]: The toy distribution of Tsipras et al. uses labels y = 1 and y = −1 with observations x ∈ {−1, 1} × R and overlapping class structure controlled by p and η.The cited setup considers p(y = 1) = p(y = −1) = 0.5, with p ≥ 0.5.
Discussion of [102]: Although Tsipras et al. derive a robustness-accuracy trade-off for an L∞ adversary with ϵ ≥ 2η, the proposed attack is invalid under this definition.The conclusion depends explicitly on the paper’s definition of proper adversarial examples; alternative definitions may yield different conclusions.