Source-linked AI summary
Adversarial Training for Free!
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, Tom Goldstein
TL;DR
Strong adversarial training is robust but too costly for many large-scale problems. The paper introduces free adversarial training, which recycles the backward-pass gradients used for parameter updates to update perturbations as well. It achieves CIFAR robustness comparable to PGD training at nearly natural-training cost and reaches 40% accuracy against non-targeted PGD attacks on ImageNet using one four-GPU workstation.
Problem
Strong adversarial training remains effective against attacks, but generating adversarial examples makes it impractical for large-scale problems such as ImageNet.
Method
Free adversarial training reuses the backward pass for parameter gradients to compute input gradients and update adversarial perturbations.
Results
The method achieves CIFAR-10 and CIFAR-100 robustness comparable to conventional PGD training and reaches 40% accuracy against non-targeted PGD attacks on ImageNet.
Takeaways & Limitations
Free adversarial training places robust training within reach of organizations with modest compute resources.
Takeaways & Limitations
Increasing the minibatch replay parameter m risks higher generalization error and possible catastrophic forgetting.
Abstract
from arXiv · showhide
Adversarial training, in which a network is trained on adversarial examples, is one of the few defenses against adversarial attacks that withstands strong attacks. Unfortunately, the high cost of generating strong adversarial examples makes standard adversarial training impractical on large-scale problems like ImageNet. We present an algorithm that eliminates the overhead cost of generating adversarial examples by recycling the gradient information computed when updating model parameters. Our "free" adversarial training algorithm achieves comparable robustness to PGD adversarial training on the CIFAR-10 and CIFAR-100 datasets at negligible additional cost compared to natural training, and can be 7 to 30 times faster than other strong adversarial training methods. Using a single workstation with 4 P100 GPUs and 2 days of runtime, we can train a robust model for the large-scale ImageNet classification task that maintains 40% accuracy against PGD attacks. The code is available at https://github.com/ashafahi/free_adv_train.
1 Introduction
Adversarial training remains one of the few defenses resistant to strong attacks, but its computational cost limits use on large-scale datasets. The paper proposes reusing gradient information to make adversarial training nearly as cheap as natural training.
- Adversarial training remains among the few defenses that withstand strong adversarial attacks.
- 3–30 times longer training makes standard adversarial training impractical for large-scale problems.
- ImageNet adversarial training has generally required research laboratories with hundreds of GPUs.
- The proposed method reuses gradient information from parameter updates to reduce adversarial-training overhead.
- 40% accuracy against non-targeted PGD attacks was achieved on ImageNet using four P100 GPUs in about two days.
2 Non-targeted adversarial examples
Non-targeted attacks move an image away from its natural class, whereas targeted attacks move it into a chosen class. The section reviews gradient-based methods, emphasizing PGD’s iterative strength and computational cost.
- Non-targeted attacks seek any incorrect class, while targeted attacks seek a specific attacker-chosen class.
- ϵ bounds the adversarial manipulation, while ||.||p specifies the distance metric for the perturbation.
- FGSM constructs a non-targeted adversarial example in one iteration using the sign of the gradients.
- PGD extends BIM with uniform random initialization and is recognized as a powerful first-order attack.
- Each PGD iteration requires a complete forward and backward pass, so increasing K strengthens attacks while increasing generation time.
3 Adversarial training
Adversarial training strengthens models by training on adversarial examples, with robustness depending on attack strength. The paper focuses on min-max training with a multi-step PGD inner loop and an SGD outer loop.
- Training on FGSM or Rand+FGSM provides robustness against non-iterative attacks but not against PGD attacks.
- Multi-step PGD adversaries were proposed for adversarial training to achieve state-of-the-art robustness.
- PGD-based adversarial training remained among the few defenses that withstood strong attacks after many defenses were broken.
- K-PGD training alternates an inner PGD-K loop that constructs adversarial examples with an outer minibatch-SGD update of model parameters.
4 “Free” adversarial training
Free adversarial training reuses one backward pass to update both model parameters and image perturbations, then replays each minibatch to obtain multiple adversarial updates. The method aims to preserve natural accuracy while reducing training cost.
- 4 days of training was required for 7-PGD WideResNet training on CIFAR-10 using a Titan X GPU.
- Free training computes input and parameter gradients in the same backward pass, giving it negligible complexity overhead over natural training.
- Repeating each minibatch m times provides multiple adversarial updates while dividing epochs by m to keep total iterations constant.
- The algorithm updates θ by SGD and updates δ with the input gradient’s sign before clipping δ to [−ϵ, ϵ].
- Large replay values can sharply reduce natural validation accuracy, whereas small values have little effect.
- Increasing m risks higher generalization error and possible catastrophic forgetting because informative examples may appear early in an epoch.
5 Robust models on CIFAR-10 and 100
On CIFAR-10 and CIFAR-100, free training achieves robustness comparable to or exceeding traditional PGD adversarial training while avoiding its computational overhead.
- Free training reaches state-of-the-art robustness on CIFAR-10 and CIFAR-100 without standard PGD training overhead.
- Training each free-training CIFAR-10 model costs roughly the same as natural training because the method preserves the same number of iterations.
- 48.03% against PGD-20 attacks for “Free” training matches 47.98% for YOPO in a direct CIFAR-10 comparison.
- CIFAR-100 comparisons used PGD-2 and PGD-7 baselines costing almost 3× and roughly 7× as much computation as free training.
- On CIFAR-100, free training exceeds traditional adversarial training in both natural-image and adversarial-image accuracy.
6 Does “free” training behave like standard adversarial training?
Free training shares key behavioral properties with PGD adversarial training: interpretable gradients and a flattened, smooth loss surface without gradient masking.
- Free training exhibits interpretable gradients, a property associated with PGD adversarially trained models.
- Adversarial examples from the free model resemble the class into which they are misclassified, preserving the generative behavior observed for PGD-trained models.
- Both free and 7-PGD training produce similarly flattened loss surfaces, with the adversarial direction showing the largest cross-entropy change.
- Free training does not rely on gradient masking, because its loss surface is not rough and adversarial directions remain informative.
7 Robust ImageNet classifiers
Free training produces robust ImageNet classifiers at substantially lower computational cost than conventional adversarial training. Robustness exceeds 40% against PGD attacks for several replay settings, while larger perturbation bounds remain difficult.
- 7 Robust ImageNet classifiers: 43% robustness against PGD attacks bounded by ϵ = 2 was achieved on ImageNet using four P100 GPUs, with each ResNet-50 experiment taking below 50 hours.The algorithm targets non-targeted adversarial training.
- 7 Robust ImageNet classifiers: Over 40% accuracy against PGD attacks was achieved by free-trained ResNet-50 models with m = 4, 6, and 8, unlike the naturally trained model.PGD-100 caused no meaningful accuracy drop compared with PGD-50.
- 7 Robust ImageNet classifiers: m = 4 performs best, and free training consistently improves robust accuracy under PGD attacks for ϵ = 2 − 7.Figure 4 compares natural and free training across perturbation bounds and replay settings.
- 7 Robust ImageNet classifiers: 2-PGD adversarial training takes roughly 3.4× longer than free training while achieving only slightly better results, with an approximately 4.5% difference.The comparison uses ResNet-50 models trained against ℓ∞ ϵ = 4 attacks.
- 7 Robust ImageNet classifiers: Higher-capacity ResNet-101 and ResNet-152 models take roughly 1.7× and 2.4× longer than ResNet-50, respectively, and the higher-capacity model gains roughly 4% in accuracy and robustness.These models were free-trained with ϵ = 4.
8 Conclusions
The paper presents free adversarial training as a near-natural-training-cost approach intended to make robust models more accessible. It combines parameter and perturbation updates in one backward pass and reports robust ImageNet training on modest hardware.
- 8 Conclusions: Free training has cost nearly equal to natural training and can be combined with other defenses without a slowdown.The authors aim to put adversarial training within reach for organizations with modest compute resources.
- 8 Conclusions: The approach is presented as a way to reduce the high computation cost that limits adoption of adversarial training.Adversarial training remains one of the few effective ways to harden networks against attacks.