PaperlayerView exact revision on arXiv

Source-linked AI summary

Ensemble Adversarial Training: Attacks and Defenses

Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel

arXiv:1705.07204v5stat.MLcs.CRcs.LG

TL;DR

Single-step adversarial training can appear robust while remaining vulnerable to black-box and stronger attacks. The paper diagnoses this failure and introduces Ensemble Adversarial Training, which improves robustness to transferred attacks from some models.

  • Problem

    Single-step adversarial training can produce robustness to single-step attacks while leaving models vulnerable to multi-step and black-box attacks.

  • Method

    Ensemble Adversarial Training augments training data with adversarial examples crafted on other static pre-trained models, decoupling example generation from the trained model.

  • Results

    Ensemble Adversarial Training increases robustness to adversarial examples transferred from holdout models across single-step and multi-step attacks.

  • Takeaways & Limitations

    Decoupling adversarial-example generation from the trained model can improve robustness to black-box attacks from some models.

  • Takeaways & Limitations

    More elaborate transfer-based attacks later reduced the most robust ImageNet model’s accuracy to 22%.

Abstract

from arXiv · show

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with strong robustness to black-box attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks. However, subsequent work found that more elaborate black-box attacks could significantly enhance transferability and reduce the accuracy of our models.

1 INTRODUCTION

The introduction shows that single-step adversarial training can create models that appear robust to single-step attacks yet remain vulnerable to transferable and multi-step attacks. It diagnoses this failure as a degenerate minimum caused by sharp loss-surface curvature and proposes Ensemble Adversarial Training to improve black-box robustness.

  • Motivation: Adversarial examples transfer across models, enabling simple black-box attacks against deployed models.Adversarial training augments training data with adversarial examples, but scaling it successfully to ImageNet-scale tasks had previously proven difficult.
  • Motivation: Single-step training produced models robust to single-step perturbations but vulnerable to more costly multi-step attacks.The attacks were also reported to fail to reliably transfer between models, motivating a closer examination of black-box robustness.
  • Failure of Single-Step Training: Single-step adversarial training admits a degenerate global minimum in which sharp curvature near data points makes linear loss approximations unreliable.This curvature degrades attacks based on a single gradient computation and reflects overfitting in adversarially trained models.
  • Attacks and Defenses: Perturbations crafted on undefended models often transfer to single-step adversarially trained models, while R+FGSM first applies a small random step before linearizing the loss.The random step is intended to escape the non-smooth vicinity of the data point.
  • Ensemble Adversarial Training: Ensemble Adversarial Training incorporates perturbations transferred from other pre-trained models to decouple example generation from the trained model’s parameters and increase perturbation diversity.The authors train Inception v3 and Inception ResNet v2 models on ImageNet with increased robustness to transferred adversarial examples from holdout models.
  • Limitations and Subsequent Work: Subsequent transfer-based attacks using momentum, data augmentation, or residual-network skip-connection gradients substantially improved transferability and reduced the effectiveness of Ensemble Adversarial Training.Later work also scaled multi-step adversarial training to ImageNet, superseding the reported Ensemble Adversarial Training results.

2 RELATED WORK

Prior work proposed many defenses against adversarial examples, but adaptive attackers can still defeat many of them. Adversarial training appears promising, with robustness demonstrated on MNIST when training examples closely maximize loss and with black-box attacks enabled by transferability.

  • Defenses: Many proposed defenses against adversarial examples remain vulnerable to adaptive attackers.The related work cites multiple defensive techniques and adaptive attacks against them.
  • Adversarial training: Adversarial training is presented as the most promising approach for learning robust models.The cited works include adversarial-training methods developed across several studies.
  • Adversarial training: On MNIST, adversarial training produces white-box robustness when training examples closely maximize the model’s loss.Madry et al. (2017) provide this result for MNIST.
  • Adversarial training: Recent methods provide certifiable robustness for small perturbations on MNIST.The cited results come from Sinha et al. (2018), Raghunathan et al. (2018), and Kolter & Wong (2017).
  • Black-box attacks: Adversarial examples transfer between models, enabling black-box attacks, including attacks that extract surrogate models from target predictions without training data.Szegedy et al. established transferability, while Papernot et al. showed a training-data-free surrogate-extraction route.

3 THE ADVERSARIAL TRAINING FRAMEWORK

The framework defines bounded ℓ∞ adversaries and adversarial training as risk minimization over generated perturbations, comparing single-step and iterative attack algorithms. It shows that single-step training can reach degenerate minima and motivates Ensemble Adversarial Training, which uses transferred perturbations from static models to improve black-box robustness under stated assumptions.

  • Adversarial threat model: The framework considers ℓ∞-bounded adversaries that seek close, misclassified examples within budget ϵ, distinguishing white-box access from black-box partial information.Black-box security is presented as a more tractable deployment goal, although white-box security is stronger.
  • Adversarial training: Adversarial training approximates the inner maximization with generated examples and trains on both clean inputs x and adversarial inputs xadv.The framework follows adversarial ERM, with the attack approximating the inner problem and outer minimization training over the resulting examples.
  • Attack algorithms: The framework compares FGSM and Step-LL single-step attacks with iterative I-FGSM or Iter-LL, which uses projected gradient descent and induces higher error but lower transfer at fixed ϵ.FGSM and Step-LL require one gradient computation, whereas iterative attacks apply multiple updates and project onto the ℓ∞ ball.
  • Degenerate minima: Single-step adversarial training admits a degenerate global minimum where linearization poorly fits the loss, producing far-from-optimal perturbations instead of genuine robustness.The alternative optimization problem can also have a minimum robust to all ℓ∞-bounded perturbations, but the degenerate minimum makes single-step attacks perform worse overall, including on undefended models.
  • Ensemble Adversarial Training: Ensemble Adversarial Training decouples perturbation generation by augmenting training data with examples crafted on other static pre-trained models, targeting robustness to future black-box attacks.Informally, robustness transfers when a future black-box adversary is not much stronger on average than the static adversaries used during training; guarantees do not extend to arbitrary unseen adversaries.

4 EXPERIMENTS

Experiments show that adversarial training can mask weak single-step attacks while increasing black-box vulnerability, whereas Ensemble Adversarial Training improves transfer robustness at slower convergence and with limited white-box protection. Subsequent attacks nevertheless substantially reduce this robustness, despite the model’s first-place competition result.

  • Single-step attacks: 19% of optimum loss was reached by Step-LL on standard Inception v3, but only 7% on v3adv, indicating degraded linear approximations.The average gradient norm was larger for v3adv (0.17) than standard v3 (0.10), so the effect was not caused by a flatter decision surface.
  • Black-box attacks: Adversarial training improves white-box single-step robustness but increases black-box error rates, because it degrades the gradient information used by transferred attacks.Perturbations crafted on adversarially trained models also transfer less effectively to undefended models.
  • Randomized single-step attack: R+FGSM adds a small random step before linearization to escape local curvature artifacts while requiring only one gradient computation.With ϵ = 16/256 and α = ϵ/2, the random step produced stronger attacks than Step-LL for all tested models.
  • Ensemble Adversarial Training: Ensemble Adversarial Training converges after 280 epochs and 450k iterations, compared with 187 epochs and 150k iterations for v3adv, while remaining vulnerable to white-box iterative and R+Step-LL attacks.The ensemble models are slightly less accurate on clean data and more vulnerable to white-box single-step attacks.
  • Ensemble Adversarial Training: Ensemble Adversarial Training significantly improves robustness to transferred attacks, with IRv2adv-ens losing 7.4% top-1 and 3.1% top-5 accuracy against holdout-model attacks.Its strongest tested black-box attacks were FGSM; R+Step-LL and iterative attacks transferred less reliably.
  • Limitations and evaluation: 22% accuracy remained for IRv2adv-ens under Wu et al.’s strongest reported attack, despite Ensemble Adversarial Training still improving robustness.The model nevertheless finished first among 70 submissions with a 95.3% competition score and achieved 97.9% clean-test accuracy.

5 CONCLUSION AND FUTURE WORK · A THREAT MODEL: FORMAL DEFINITIONS

The paper concludes that adversarially trained models can remain vulnerable to simple black-box and white-box attacks, while Ensemble Adversarial Training improves robustness in several settings. It formalizes white-box, non-interactive black-box, and interactive black-box threat models, including their access assumptions and attack capabilities.

  • 5 CONCLUSION AND FUTURE WORK: Adversarially trained models remain vulnerable to simple black-box and white-box attacks despite strong robustness to single-step adversarial examples.The authors suggest improving adversarial training by decoupling adversarial-example generation from the target model.
  • 5 CONCLUSION AND FUTURE WORK: Ensemble Adversarial Training was resilient to generative attacks on MNIST and CIFAR10, often exceeding models trained against iterative attacks.The cited result comes from subsequent work by Xiao et al. (2018).
  • 5 CONCLUSION AND FUTURE WORK: Interactive adversaries can estimate target gradients from prediction confidences and apply R+FGSM, but this attack does not apply when queries return only predicted labels.Finite-difference gradient estimation is the stated mechanism.
  • A THREAT MODEL: FORMAL DEFINITIONS: The formal threat model represents a target model h ∈ H as an architecture from hypothesis space H, trained on samples from distribution D using randomized procedure train.The procedure takes H, X_train, Y_train, and randomness r as inputs.
  • A THREAT MODEL: FORMAL DEFINITIONS: An adversary generates X_adv within an ℓ∞ budget ϵ for test inputs and attack success is measured by the target model’s error rate on X_adv.Each adversarial example satisfies ∥x_i − x_adv_i∥∞ ≤ ϵ.
  • A THREAT MODEL: FORMAL DEFINITIONS: A white-box adversary accesses the training algorithm, architecture, training data, randomness, and target parameters, enabling arbitrary attacks.The paper characterizes white-box access to internal weights as a very strong adversarial model.
  • A THREAT MODEL: FORMAL DEFINITIONS: A non-interactive black-box adversary knows the target training procedure and architecture, samples from D, and transfers attacks from a locally trained model with potentially different architecture.It does not learn the target’s training randomness or parameters, while the paper’s attacks additionally reuse the target’s training data.
  • A THREAT MODEL: FORMAL DEFINITIONS: An interactive black-box adversary additionally queries the target model, and class-score outputs enable gradient estimation with finite differences before applying the paper’s attacks.Such access may be unavailable when the model is embedded in a larger system, and attacks can work with only a small number of samples from D.

B GENERALIZATION BOUND FOR ENSEMBLE ADVERSARIAL TRAINING

The section formalizes Ensemble Adversarial Training’s generalization to future adversaries using a domain-adaptation bound. The guarantee depends on training-sample complexity and how different the target adversary is from the train-time adversaries.

  • Assumptions: The analysis assumes training uses only adversarial examples computed on pre-trained models, while noting the results extend to clean data and examples generated by the trained model.This is a simplifying assumption for the formal statement of Theorem 1.
  • Setup: Training samples come from k adversarial distributions, each generated by perturbing data within an ℓ∞ radius ϵ, and testing uses a target adversarial distribution.The target distribution represents a future adversary A∗.
  • Discrepancy: The discrepancy disc(Atrain, A∗) measures whether differences in model robustness across training attacks resemble differences under the target attack.The discrepancy is small when the relative robustness patterns induced by train-time and target attacks are similar.
  • Generalization bound: Relative to standard supervised-learning bounds, the domain-adaptation guarantee adds discH(Atrain, A∗), supporting generalization to future adversaries that are not too different from those used in training.The additional discrepancy term captures divergence between target and source adversarial distributions.

C EXPERIMENTS ON MNIST

The MNIST experiments revisit the ImageNet findings to test whether single-step adversarial training has similar limitations beyond ImageNet. They do not aim to match the strong white-box robustness obtained through iterative-attack training.

  • Motivation: The experiments test whether limitations of single-step adversarial training observed on ImageNet extend to MNIST.MNIST is presented as a simpler task than ImageNet.
  • Motivation: Madry et al. (2017) found that training on iterative attacks provides robustness to white-box attacks with bounded ℓ∞norm.The MNIST experiments explicitly distinguish this result from their own objective.
  • Adversarial Examples: Figure 3 illustrates clean MNIST examples, inputs rotated by 20° with 5 random pixels flipped, and I-FGSM examples using ϵ = 0.3.The figure presents these examples in top, middle, and bottom rows, respectively.

C.1 A NOTE ON ℓ∞ROBUSTNESS ON MNIST

MNIST can admit a simple closed-form route to ℓ∞ robustness because its images are nearly binary, but robustness does not generalize across perturbation types. The paper therefore treats ℓ∞ robustness as an illustrative metric while motivating broader adversarial perturbation definitions and training improvements.

  • ℓ∞ robustness on MNIST: Over 80% of pixels in an average MNIST image lie in {0, 1}, while only 6% lie in [0.2, 0.8], enabling simple ℓ∞ robustness for ϵ ≤ 0.3.The authors attribute this to MNIST’s near-binary structure and a simple “closed-form” robust representation.
  • ℓ∞ robustness on MNIST: After 20° rotation, I-FGSM reduces both the rounding model’s and Madry et al.’s model’s accuracy to 65%, versus under 5% error on clean rotated inputs.This shows that ℓ∞ robustness on unmodified MNIST does not cover geometric perturbations.
  • ℓ∞ robustness on MNIST: Randomly flipping 5 pixels per image lowers both models’ accuracy to under 50%, further exposing the limits of robustness defined only by the ℓ∞ norm.The passage contrasts this with clean rotated inputs, whose error is below 5%.
  • ℓ∞ robustness on MNIST: The authors focus on ℓ∞ perturbations to study single-step adversarial training, while noting that their approach can extend to multiple perturbation metrics.They also identify enumerating all adversarial perturbation types as an open question and point to augmentation as a possible response.

C.2 RESULTS

On MNIST, adversarial training degraded single-step loss approximations across architectures, while input dropout mitigated this effect in some settings. Ensemble Adversarial Training substantially improved black-box robustness, except for architecture B.

  • Degenerate minimum: Adversarial training degraded the quality of the FGSM’s linear approximation to the loss for every tested model architecture.The approximation ratio compared FGSM loss with the loss from a strong iterative attack.
  • Degenerate minimum: Input dropout limited adversarial training’s degradation of the single-step attack, whereas removing it caused a significant decline in architecture B.Architecture B included input dropout; B* omitted it.
  • Attacks: For most models, black-box attacks outperformed white-box single-step attacks, and a preceding random step strengthened white-box single-step attacks.Model B_adv showed the opposite pattern.
  • Attacks: Input dropout helped avoid single-step degradation but significantly delayed convergence, leaving model B_adv with relatively high error on white-box FGSM examples.The authors interpret input-dropout training as comparable to training with a randomized single-step attack.
  • Ensemble Adversarial Training: Input dropout’s benefit was architecture- and dataset-specific, marginal for models A, C, and D, and inferior to Ensemble Adversarial Training.Input dropout also significantly degraded ImageNet accuracy and was therefore not used for ImageNet models.
  • Ensemble Adversarial Training: Ensemble Adversarial Training significantly increased black-box robustness for architectures other than B.Model B_adv-ens nevertheless achieved slightly better white-box and black-box robustness than B_adv.

D TRANSFERABILITY OF RANDOMIZED SINGLE-STEP PERTURBATIONS.

Randomized single-step perturbations transfer between ImageNet models at a much lower rate than deterministic Step-LL perturbations. White-box attacks consistently outperform black-box attacks under the evaluated R+Step-LL setting.

  • D TRANSFERABILITY OF RANDOMIZED SINGLE-STEP PERTURBATIONS.: R+Step-LL extends Step-LL by prepending the attack with a small random perturbation.The randomized attack is evaluated on ImageNet using ϵ = 16/256 and α = ϵ/2.
  • D TRANSFERABILITY OF RANDOMIZED SINGLE-STEP PERTURBATIONS.: Randomized R+Step-LL perturbations transfer between ImageNet models at a much lower rate than deterministic Step-LL perturbations.The comparison uses Table 9 for the randomized variant and Table 1 for the deterministic variant.
  • D TRANSFERABILITY OF RANDOMIZED SINGLE-STEP PERTURBATIONS.: White-box attacks always outperform black-box attacks in the evaluated transferability experiments.The evaluation uses 10,000 random test set samples.

E GRADIENT ALIGNED ADVERSARIAL SUBSPACES FOR THE ℓ∞NORM

The section gives an optimal ℓ∞ construction of orthogonal perturbations aligned with the signed gradient using Regular Hadamard Matrices. It proves an upper bound on alignment and matches it for admissible matrix orders, while noting limitations for general k.

  • Construction: An optimal ℓ∞ construction finds orthogonal perturbations maximally aligned with sign(g), resolving the previously open construction problem.The construction is based on Regular Hadamard Matrices.
  • Upper bound: Lemma 6 bounds the number of orthogonal {−1, 1}^d perturbations achievable at a given alignment with the signed gradient by α ≤ k−1.The bound follows from summing squared inner products and the orthogonality constraint.
  • Hadamard construction: For any k admitting a Regular Hadamard Matrix, a randomized construction achieves sign(g)⊤r_i = d · k−1/2 and E[g⊤r_i] = k−1/2 · ∥g∥1.Rows are repeated across dimensions, multiplied componentwise by sign(g), and applied after a random permutation to account for nonuniform gradient weights.
  • Limitations: The Lemma 7 bound is attainable if and only if perturbations come from Regular Hadamard Matrix rows; other combinatorial designs may provide looser bounds when no such matrix exists.Known constructions are used for k ∈ {4, 16, 36, 64, 100}.

F ILLUSTRATIONS OF GRADIENT MASKING IN ADVERSARIAL TRAINING

Adversarial training creates local curvature artifacts around data points, causing one-shot first-order attacks to generate non-adversarial perturbations. Figures 4 and 5 illustrate this effect on ImageNet’s v3adv model and MNIST’s Aadv model.

  • Gradient masking mechanism: Adversarial training introduces spurious loss curvature around data points, making first-order one-shot attacks produce non-adversarial perturbations.The paper identifies these artifacts as the mechanism behind gradient masking.
  • ImageNet illustration: Figure 4 illustrates local curvature artifacts for ImageNet’s Inception v3adv model using signed-gradient and orthogonal adversarial directions from Inception v4.The plotted samples have the form x∗= x+ϵ1 ·g +ϵ2 · g⊥, with zoomed-in right-side plots.
  • MNIST illustration: Figure 5 illustrates local curvature artifacts for MNIST’s Aadv model using its signed gradient and an orthogonal adversarial direction from model B.The plotted samples have the form x∗= x + ϵ1 · g + ϵ2 · g⊥, with zoomed-in right-side plots.
Loading 1705.07204v5…