PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast is better than free: Revisiting adversarial training

Eric Wong, Leslie Rice, J. Zico Kolter

arXiv:2001.03994v1cs.LGstat.ML

TL;DR

Adversarial training is generally considered more expensive than standard training because constructing adversarial examples requires iterative attacks. The paper tests randomized FGSM adversarial training and efficient-training techniques, finding PGD-level empirical robustness at substantially lower training times while identifying catastrophic overfitting as a failure mode.

  • Problem

    Adversarial training is typically more expensive than standard training because constructing adversarial examples with methods such as PGD adds substantial computation.

  • Method

    The paper combines FGSM adversarial training with random perturbation initialization and standard efficient-training techniques such as cyclic learning rates and mixed precision.

  • Results

    Randomized FGSM training is as effective as PGD-based training, reaching 45% robust CIFAR10 accuracy at ε = 8/255 in 6 minutes and 43% robust ImageNet top-1 accuracy at ε = 2/255 in 12 hours.

  • Takeaways & Limitations

    Empirically robust classifiers can be trained in minutes on CIFAR10 and hours on ImageNet, with times comparable to or faster than standard and free adversarial training.

  • Takeaways & Limitations

    FGSM training can undergo catastrophic overfitting, causing PGD robust accuracy to suddenly drop to 0%, although early stopping can partially salvage affected runs.

Abstract

from arXiv · show

Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD). In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice. Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost. Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy to PGD attacks with $ε=8/255$ in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at $ε=2/255$ in 12 hours, in comparison to past work based on "free" adversarial training which took 10 and 50 hours to reach the same respective thresholds. Finally, we identify a failure mode referred to as "catastrophic overfitting" which may have caused previous attempts to use FGSM adversarial training to fail. All code for reproducing the experiments in this paper as well as pretrained model weights are at https://github.com/locuslab/fast_adversarial.

1 INTRODUCTION

Adversarial training seeks models accurate on both clean and adversarially perturbed data, but generating adversarial examples makes it substantially slower than standard training. This paper shows that randomized FGSM training can match PGD-based robustness at much lower cost, including rapid CIFAR10 and ImageNet training.

  • Adversarial training aims to maintain accuracy on both ordinary and adversarially perturbed examples, especially for safety- and security-focused applications.
  • Generating adversarial examples forms most of adversarial training’s additional computation, leaving prior approaches significantly slower than standard training.
  • Randomly initialized FGSM training matches PGD-based training while being an order of magnitude more efficient.
  • 6 minutes yields a CIFAR10 classifier with 45% robust accuracy at ε = 8/255, matching previous results and outperforming reported training times of 80 and 10 hours.The 80-hour figure is for PGD-based training, while 10 hours is for free adversarial training.

2 RELATED WORK

Prior work developed multiple adversarial defenses and progressively reduced the cost of adversarial training, but strong evaluation remains essential because weak attacks can overestimate robustness. PGD adversarial training remained empirically robust, yet its computational overhead still exceeded that of standard training.

  • Adversarial defenses include preprocessing, detection, verification, provable defenses, and theoretically motivated heuristics.
  • Certified defenses have scaled to reasonably sized networks, but their guarantees do not match the empirical robustness obtained through adversarial training.
  • Weak attacks can create a misleading sense of security because stronger attacks have defeated previously proposed defenses.
  • PGD adversarial training remained empirically robust, but embedding a strong PGD adversary inside training was expensive.
  • Earlier methods reduced adversarial-training runtime but remained capable of taking hours to days, unlike top standard-training methods taking minutes or hours.

3 ADVERSARIAL TRAINING OVERVIEW

Adversarial training approximates a robust optimization problem by generating perturbations within a threat model and updating model parameters against them. PGD improves the inner maximization through repeated projected steps, whereas free training reduces overhead through minibatch replays and fewer total epochs.

  • Adversarial training seeks model parameters that remain effective against perturbations constrained by a threat model.
  • Under the ℓ∞ threat model, FGSM approximates the inner maximization with a single closed-form gradient-sign perturbation.
  • PGD uses multiple smaller FGSM steps, projecting perturbations back into the threat model whenever they leave it.
  • Random restarts can improve PGD’s local approximation of the non-convex inner maximization.
  • PGD requires O(MN) gradient computations per epoch, making it N times slower than standard training with O(M) computations.
  • Free adversarial training uses full-size FGSM steps, replays each minibatch N times, retains perturbations between minibatches, and reduces total epochs to offset replay cost.

4 FAST ADVERSARIAL TRAINING

FGSM adversarial training with non-zero initialization can match PGD-based robustness at substantially lower computational cost. Efficient-training techniques further reduce training time, while initialization and step-size choices determine whether catastrophic overfitting occurs.

  • FGSM adversarial training with random initialization is as effective as PGD-based training while being an order of magnitude more efficient.
  • 4.1 REVISITING FGSM ADVERSARIAL TRAINING: Non-zero initialization, using either the previous minibatch’s perturbation or a uniformly random perturbation, is identified as the primary driver of FGSM success.
  • 4.1 REVISITING FGSM ADVERSARIAL TRAINING: Zero initialization and boundary-generating step sizes can cause catastrophic overfitting, making models non-robust to full-strength PGD attacks.
  • 4.2 DAWNBENCH IMPROVEMENTS: Cyclic learning rates reduce the epochs needed for convergence, while mixed-precision arithmetic reduces memory use and runtime on GPUs with tensor cores.

5 EXPERIMENTS

Experiments evaluate fast adversarial training across MNIST, CIFAR10, and ImageNet, combining FGSM with efficient-training techniques and comparing empirical or verified robustness against alternative methods. FGSM achieves rapid robust training, but its success depends on design choices that can otherwise cause catastrophic overfitting.

  • Experimental setup: Experiments use MNIST, CIFAR10, and ImageNet to evaluate fast adversarial training methods and robustness.CIFAR10 uses PreAct ResNet18 on one GeForce RTX 2080ti; ImageNet uses ResNet50 on four GeForce RTX 2080tis.
  • Experimental setup: FGSM experiments use random starting points and α = 1.25ϵ, while evaluation PGD attacks use 10 random restarts and 50 iterations.The evaluation configuration strengthens the PGD adversary with random restarts.
  • Fast CIFAR10: FGSM and PGD require fewer epochs than free adversarial training when using cyclic learning rates, producing greater speedups.The minimum epoch count is identified by repeatedly training each method across a range of maximum epochs and plotting final robustness.
  • Verification limits: Exact verification is infeasible for the larger models and datasets, so robustness beyond MNIST relies on empirical evaluation.MILP verification at ϵ = 0.3 is also too large to solve for the FGSM- and PGD-trained models.
  • Fast CIFAR10: FGSM adversarial training is fastest on CIFAR10, reaching 45% robust accuracy in 6 minutes using 15 epochs.Training times are measured after applying cyclic learning rates and mixed precision; PGD and free training take comparable times.
  • Fast ImageNet: 12 hours of FGSM training produces an ImageNet classifier, using 15 epochs and costing a fraction of free adversarial training.ImageNet training combines FGSM, mixed precision, cyclic learning rates, progressive resizing, and batch-normalization regularization changes.
  • Catastrophic overfitting: Catastrophic overfitting causes PGD robust accuracy to suddenly and drastically drop to 0% under unfavorable FGSM-training choices.Early stopping based on PGD accuracy on a small training minibatch can partially recover results for some failure modes.

6 CONCLUSION

FGSM adversarial training with random initialization matches PGD-based training while reducing training cost. The paper also identifies conditions that improve consistency and suggests approximate inner optimization can sometimes suffice.

  • FGSM adversarial training with random initialization is as effective as more costly PGD adversarial training.
  • A single FGSM iteration costs twice as much as free adversarial training but converges significantly faster, especially with a cyclic learning-rate schedule.
  • The findings suggest that rough approximations to adversarial-training’s inner optimization may suffice, despite evaluation commonly using multiple PGD restarts and many steps.
  • Uniform initialization provides the greatest marginal improvement over the original R+FGSM attack, while a full step size does not help independently.
  • The paper’s FGSM training variant is more consistent across random seeds and has a much lower standard deviation than R+FGSM.

B TRAINING PARAMETERS FOR TABLE 1

This section specifies experimental training settings and illustrates the learning behavior associated with catastrophic overfitting under zero-initialization FGSM training.

  • All methods use batch size 128, SGD with momentum 0.9, weight decay 5 ∗10^-4, and averages over 3 random seeds.
  • Early stopping detects overfitting using a 5-step PGD adversary with 1 restart on one training minibatch.
  • Figure 3 plots robust test performance across FGSM step sizes for ϵ = 8/255.
  • Figure 4 plots training loss and FGSM- and PGD-induced error rates for zero-initialization FGSM training at ϵ = 8/255.

C OPTIMAL STEP SIZE FOR FGSM ADVERSARIAL TRAINING

Robust performance improves as the FGSM step size increases up to α = 10/255, after which larger steps provide no benefit or promote overfitting.

  • Robust performance increases with step size up to α = 10/255.
  • Beyond α = 10/255, larger step sizes provide no further benefit or can make the model overfit to adversarial examples.The large step size can force the model to overfit to the perturbation-region boundary.

D CATASTROPHIC OVERFITTING AND THE EFFECT OF EARLY STOPPING

Several FGSM-training variants rapidly undergo catastrophic overfitting, losing PGD robustness; monitoring PGD performance on a small training subset enables early stopping before this failure.

  • Catastrophic overfitting can transform a reasonably robust model into one with 0% PGD robust accuracy within a couple of epochs.
  • A histogram of PGD perturbations compares successfully trained and catastrophically overfitted CIFAR10 models.
  • Figure 6 reports robust test performance across step sizes when early stopping is used to avoid catastrophic overfitting.
  • Zero initialization and Tram`er et al.’s randomization can produce perturbations limited to {−ϵ, 0, ϵ}, reducing adversarial-example diversity.
  • A 5-step PGD attack on one training minibatch can quickly detect catastrophic overfitting through a sudden robust-accuracy drop.
  • Early stopping avoids catastrophic overfitting and can recover useful robust performance for larger FGSM step sizes.

E TRAINING PARAMETERS FOR FIGURE 2

The experiments use fixed optimization settings across methods and report results averaged over multiple random seeds, with schedule-specific maximum learning rates listed separately.

  • Batch size is 128 for all methods.
  • Training uses SGD with momentum 0.9 and weight decay 5 ∗10^-4.
  • Results are averaged over 3 random seeds, and maximum learning rates for the cyclic schedule are provided in Table 8.

F COMBINING FREE ADVERSARIAL TRAINING WITH DAWNBENCH IMPROVEMENTS ON IMAGENET

The section tests whether free adversarial training benefits from DAWNBench optimizations on ImageNet, while cautioning that the comparison is not fully tuned for free training.

  • Mixed precision directly speeds up free adversarial training without hurting performance.
  • The study applies cyclic learning rates, progressive resizing, and batch-norm regularization to free adversarial training, but does not comprehensively search ImageNet parameters.
  • Even with 30 epochs, giving free adversarial training the same compute time as FGSM training, performance approaches but does not recover its original level.
Loading 2001.03994v1…