Source-linked AI summary
Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, Pushmeet Kohli
TL;DR
The paper asks how far adversarial training can be pushed despite slow recent progress. It systematically studies training and model-design factors, then combines them to obtain robust models that substantially exceed prior results across datasets and threat models.
Problem
The study addresses the limited recent progress in adversarial robustness and seeks to identify the current limits of adversarial training.
Method
The authors train and dissect more than 150 robust models while varying objectives, unlabeled data, model size, activations, and weight averaging.
Results
65.88% robust accuracy is obtained on CIFAR-10 against ℓ∞ perturbations of size 8/255 with additional unlabeled data, while improvements also extend to CIFAR-10 ℓ2 and CIFAR-100 ℓ∞ settings.
Takeaways & Limitations
Accumulating small training and architecture choices can substantially improve adversarial robustness and establish stronger baselines.
Takeaways & Limitations
The usefulness of larger training perturbation radii remains under-explored, and tuning the radius yields only marginal gains with classical adversarial training.
Abstract
from arXiv · showhide
Adversarial training and its variants have become de facto standards for learning robust deep neural networks. In this paper, we explore the landscape around adversarial training in a bid to uncover its limits. We systematically study the effect of different training losses, model sizes, activation functions, the addition of unlabeled data (through pseudo-labeling) and other factors on adversarial robustness. We discover that it is possible to train robust models that go well beyond state-of-the-art results by combining larger models, Swish/SiLU activations and model weight averaging. We demonstrate large improvements on CIFAR-10 and CIFAR-100 against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $8/255$ and $128/255$, respectively. In the setting with additional unlabeled data, we obtain an accuracy under attack of 65.88% against $\ell_\infty$ perturbations of size $8/255$ on CIFAR-10 (+6.35% with respect to prior art). Without additional data, we obtain an accuracy under attack of 57.20% (+3.46%). To test the generality of our findings and without any additional modifications, we obtain an accuracy under attack of 80.53% (+7.62%) against $\ell_2$ perturbations of size $128/255$ on CIFAR-10, and of 36.88% (+8.46%) against $\ell_\infty$ perturbations of size $8/255$ on CIFAR-100. All models are available at https://github.com/deepmind/deepmind-research/tree/master/adversarial_robustness.
1. Introduction
The paper investigates the limits of adversarial training through a systematic study of training objectives, data, model scale, activations, and weight averaging. Combining these factors yields robust models that substantially improve prior results across datasets and threat models.
- The study evaluates more than 150 adversarially robust models across training losses, unlabeled data, model size, and weight averaging.The analysis examines both individual factors and their combinations.
- TRADES with early stopping outperforms regular adversarial training, while larger models, Swish/SiLU activations, and weight averaging improve robustness.Weight averaging consistently boosts robustness, with gains comparable to adding TRADES in the no-additional-data setting.
- 65.88% robust accuracy is achieved on CIFAR-10 against ℓ∞ perturbations of size 8/255 with additional unlabeled data, versus 57.20% without it.The corresponding prior-art results were 59.53% and 53.74%.
- 80.53% robust accuracy is achieved on CIFAR-10 against ℓ2 perturbations of size 128/255, while CIFAR-100 reaches 36.88% against ℓ∞ perturbations of size 8/255.These results use the same settings as the CIFAR-10 ℓ∞ experiments and improve prior art in both cases.
- The authors frame these results as evidence that accumulated small improvements can substantially advance adversarial robustness and provide new baselines.They argue that understanding current limits matters because fundamentally new techniques may eventually be needed.
2. Background
The background presents adversarial examples, adversarial training, and norm-bounded evaluation as the paper’s central technical context. It emphasizes that robust-accuracy comparisons require strong, consistent attacks.
- Adversarial training feeds adversarially perturbed examples into training and has become a leading approach for learning robust neural networks.Variants modify the attack procedure, loss function, or architecture.
- Many proposed defenses were broken by stronger adversaries, so robust accuracy under different evaluation protocols is not easily comparable.This study evaluates models with AutoAttack and MultiTargeted.
- For ℓp-bounded perturbations of size ε, the threat model contains perturbations δ satisfying ||δ||p < ε.The paper specializes this set to ℓ∞ and ℓ2 threat models.
- PGD approximates the inner attack optimization by taking multiple projected gradient-ascent steps within the allowed perturbation set.The paper denotes the K-step procedure as PgdK.
3. Setup and implementation details
The experiments use Wide ResNets, SGD-based adversarial training, pseudo-labeled unlabeled data, and validation-based early stopping. Training attacks use Pgd10, while robustness is monitored with Pgd40.
- Architecture: The baseline is a 36M-parameter Wrn-28-10, while tested models reach 70 layers, width 20, and 267M parameters.The largest model is Wrn-70-16.
- Outer minimization: Training uses SGD with Nesterov momentum, multistep learning-rate decay, and global weight decay of 5 × 10^-4.The basic no-additional-data setting uses batch size 128 and 200 epochs.
- Additional data: Unlabeled samples receive pseudo-labels from a separate non-robust classifier trained on clean CIFAR-10 data.The training pipeline also applies random crops and horizontal flips.
- Inner maximization: The inner maximization uses Pgd10 with step sizes 2/255 for ℓ∞ and 15/255 for ℓ2 perturbations.Training takes approximately 1.5 hours for Wrn-28-10 without additional data and about 2 hours with it.
- Evaluation protocol: Two models are trained per hyperparameter setting, and early stopping retains parameters with the highest validation robust accuracy measured using Pgd40.The validation set is separate from the training and test sets.
4. Experiments and analysis
The experiments systematically examine adversarial-training losses, perturbation radii, unlabeled data, and model architectures, identifying settings that improve robustness while exposing evaluation and data-quality trade-offs.
- 4.2.1. Inner maximization loss: TRADES generally achieves higher robust accuracy than adversarial training across inner losses, with TRADES-XENT best in low-data and TRADES-KL best in high-data settings.The stronger AA+MT evaluation is used alongside a weaker Pgd40 margin attack.
- 4.2.1. Inner maximization loss: Margin loss during training can create gradient masking, producing larger robust-accuracy degradation under stronger attacks; cross-entropy reduces this degradation.The issue is especially prominent in AT-MARGIN and is mitigated when TRADES is used for outer minimization.
- 4.2.2. Inner maximization perturbation radius: Increasing the training perturbation radius can close adversarial training’s robustness gap to TRADES, especially in the low-data regime.The authors report that TRADES does not show similar gains with larger radii in the settings tried.
- 4.3.1. Quality and quantity: 500K regenerated unlabeled images improve robustness by +0.71% over Carmon et al., while increasing data from 200K to 500K adds +1.83% and reaching 1M reduces it by -0.23%.The results indicate a trade-off between the quantity and quality or dataset alignment of pseudo-labeled data.
- 4.3.2. Ratio of labeled-to-unlabelled data per batch: A 3:7 labeled-to-unlabeled batch ratio improves robust accuracy by +0.95% over a 1:1 ratio, while reducing unlabeled-data emphasis eventually removes the gain.The authors interpret this as evidence that the additional data is aligned with CIFAR-10 and benefits from being seen more frequently.
- 4.4. Model architecture: Model scaling improves robustness, but depth and width are not interchangeable: a similarly timed WRN-46-15 exceeds WRN-28-20 by +0.96% without and +0.66% with additional data.Both architectures increase effective parameter count, yet their robustness effects differ.
appendix also shows that the clean accuracy improves as networks become larger.
The appendix reinforces that architecture, activation choice, weight averaging, and combined design choices affect adversarial robustness, with broader evaluations across datasets and perturbation norms.
- Model weight averaging consistently improves robustness and can maintain higher accuracy across about 25 training epochs.The reported gains were +1.41% without additional data and +0.73% with it.
- Swish/SiLU produced the strongest activation-function results, improving robustness by +0.8% without and +1.13% with additional unlabeled data.Other smooth activations did not consistently correlate positively with robustness.
- The combined approach improved CIFAR-10 ℓ∞ robust accuracy over state of the art by +3.46% without and +6.35% with additional data.The evaluation used perturbations of size 8/255.
- The same hyperparameters generalized to CIFAR-10 ℓ2 and CIFAR-100 ℓ∞ settings, surpassing known state-of-the-art results in all four data regimes.The tested perturbation sizes were 128/255 for CIFAR-10 ℓ2 and 8/255 for CIFAR-100 ℓ∞.
A.1.1. Learning rate schedule
The appendix compares multistep, cosine, and exponential learning-rate schedules, finding multistep generally strongest while noting context-dependent exceptions.
- Multistep learning-rate scheduling outperformed cosine and exponential schedules in the reported implementations.The authors do not exclude the possibility that better schedules exist or that the optimum depends on architecture and attack method.
- Cosine and exponential schedules are less sensitive to early stopping and may produce less noisy results.This is presented as a potential practical advantage rather than the primary accuracy result.
- With additional unlabeled data, model weight averaging, and a Wrn-70-16, cosine scheduling performed slightly better than multistep by +0.24%.
A.1.2. Number of optimization steps
Longer adversarial-training runs were not beneficial in either data setting, consistent with robust overfitting concerns and the need to balance epochs with other hyperparameters.
- Without additional data, increasing training from 200 to 400 epochs degraded robust accuracy by -0.64%.
- With additional data, increasing training from 400 to 800 epochs degraded robust accuracy by -1.43%.
- Robust adversarial training does not generally benefit from longer training times, because robust overfitting can reduce performance.The authors recommend balancing training epochs with hyperparameters such as ℓ2 regularization.
A.1.3. ℓ2 regularization
The appendix finds that ℓ2 regularization and inner-attack strength materially affect robustness, whereas increasing TRADES’ training radius is not consistently beneficial.
- ℓ2 regularization: A weight-decay value of 5 · 10^-4 worked well with and without additional data.This value is also frequently used when training adversarially robust Wide-ResNets.
- ℓ2 regularization: Without ℓ2 regularization, robust accuracy dropped by -5.06% in the setting without additional data.
- Number of optimization steps: Stronger inner attacks improved robustness with diminishing returns: increasing steps from 4 to 8, 8 to 10, and 10 to 16 yielded +2.59%, +0.75%, and +0.51%.The reported gains apply to the setting without additional data.
- Perturbation radius: For TRADES, increasing the training perturbation radius by 1.1× or 1.2× produced inconsistent performance.The radius changes applied during training, not evaluation.
A.3.1. Ratio of labeled-to-unlabeled data per batch (continued)
The study varies the labeled-to-unlabeled data ratio and finds that giving unlabeled data slightly more weight improves robustness, especially with larger unlabeled datasets.
- Ratio of labeled-to-unlabeled data: Slightly more importance assigned to unlabeled data improves robust accuracy across the evaluated ratio settings.
- Ratio of labeled-to-unlabeled data: +1.32% robust accuracy is achieved by the optimal 3:7 labeled-to-unlabeled ratio on the 1M-image unlabeled dataset.This improves over the 1:1 ratio.
- Ratio of labeled-to-unlabeled data: +0.95% is obtained from the optimal ratio when using Carmon et al.’s smaller unlabeled dataset.
- Ratio of labeled-to-unlabeled data: Larger unlabeled datasets can provide larger improvements in robustness.
A.4.2. Data augmentation
The experiments show that augmentation methods effective for standard classification do not necessarily improve robust generalization, while label smoothing has limited or harmful effects depending on the data.
- Data augmentation: -1.98% and -1.50% robust accuracy changes accompany stronger color jittering without and with additional unlabeled data, respectively.
- Data augmentation: AutoAugment, RandAugment, and AugMix reduce robust accuracy relative to the simplest augmentation scheme.These methods were mainly tuned for ImageNet and would require further fine-tuning to be competitive.
- Data augmentation: Data augmentation schemes that perform well for standard classification tasks do not necessarily improve robust generalization.
- Label smoothing: Label smoothing on labeled data has minor effects on robustness, with no clear correlation between the smoothing factor and robust accuracy.Factors 0.02 and 0.2 seem helpful, while 0.05 or 0.1 seem detrimental in at least one data setting.
C. Additional detailed results
Additional results show that robustness depends on network size, weight averaging, activation choice, data ratios, and weight decay, reinforcing that training design choices materially affect outcomes.
- Network size: Different network sizes produce different clean and robust accuracies on CIFAR-10 with and without additional unlabeled data.
- Weight averaging: Weight averaging decay values yield different clean and robust accuracies across both data settings.
- Activation functions: Different activation functions produce different clean and robust accuracies on CIFAR-10.
- Data ratio: Varying labeled-to-unlabeled ratios changes clean and robust accuracy on CIFAR-10 and 80M-Ti.
- Weight decay: Different weight-decay settings produce different clean and robust accuracies with and without additional unlabeled data.
D. Loss landscape analysis
The paper analyzes adversarial loss landscapes by traversing from test images toward worst-found and random perturbation directions, comparing models trained with different data regimes and a prior model.
- Landscape construction: Loss landscapes are generated by varying each input from its original image toward a Pgd40 attack direction or a random Rademacher direction.The analysis uses the margin loss, where misclassification occurs below zero.
- Compared models: The comparison includes the largest model trained with additional unlabeled data, the largest model without it, and Carmon et al.’s model.
- Findings: Most examined landscapes are smooth and do not exhibit patterns of gradient obfuscation.
- Findings: The fifth test image, a dog, has quite similar landscapes across all compared models.