Efficient Adversarial Training with Transferable Adversarial Examples

1. Introduction

• Iterative attacks can strengthen adversarial examples, but each input-gradient computation adds overhead, making adversarial training nearly 100 times slower than natural training.Source: §1 · p. 1
• ATTA reuses perturbations across neighboring epochs, where adversarial examples transfer between models, to accumulate attack strength with fewer attack iterations.The paper reports this neighboring-epoch transferability as a finding and the basis for ATTA.Source: §1 · pp. 1–2
• ATTA is evaluated with MAT and TRADES on MNIST and CIFAR10, and reported to achieve comparable adversarial accuracy on CIFAR10 in about 2 hours versus MAT’s 33 hours.For MNIST, the reported comparison is 5 minutes with ATTA versus 65 minutes with MAT.Source: §1 · pp. 1–2
• The paper also reports comparative robustness on ImageNet within 97 hours using one 2080 TI GPU, and better effectiveness and efficiency than YOPO and Free.Source: §1 · p. 2
• 12.2× on MNIST and 14.1× on CIFAR10 faster than traditional adversarial methods with comparable model robustness; ATTA improves MAT adversarial accuracy by up to 7.2% on CIFAR10.Source: §1 · p. 2

2.1. Adversarial Training

• Adversarial training uses adversarial attacks as data augmentation, training the model on perturbed images that maximize loss while remaining within an allowed perturbation space.The paper frames this as a min-max optimization problem.Source: §2.1 · p. 2
• PGD-k approximates the inner maximization with k projected gradient steps, projecting each perturbation back into the allowed space S.Each step requires an input gradient computation.Source: §2.1 · p. 2
• More PGD iterations produce higher-loss examples but increase attack-generation time; fewer iterations save time at a cost to robustness.Adversarial training may take close to 100x natural-training time, with most time spent generating adversarial examples.Source: §2.1 · p. 2

2.2. Transferability of Adversarial Examples.

• Transferability means examples adversarial to one model can remain adversarial to another; similar behavior and decision boundaries motivate testing neighboring training checkpoints as substitute models.The paper distinguishes this use from black-box attacks.Source: §2.2 · p. 3

3.1. High transferability between epochs

• Neighboring-epoch models T_i have transferability metrics above 0.93 and higher transferability than independently seeded baseline models S_i.The comparison uses both error-rate and loss transferability metrics.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2. Accumulative PGD-k attack

• Accumulative PGD-k reuses the previous epoch’s perturbation, and increasing the number of epochs m yields higher attack loss at the same iteration count k.The attack begins on an earlier checkpoint and then proceeds using accumulated perturbations.Source: §3.2 · pp. 3–4, Figure 2 · p. 4, Figure 3 · p. 4
• At m = 40 on MNIST, accumulative PGD-1 matches traditional PGD-35 attack loss; at m = 10 on CIFAR10, it matches traditional PGD-12 attack loss.These comparisons are for the same attack loss evaluated in the study.Source: §3.2 · p. 4

3. Attack strength accumulation

• The authors hypothesize that repeated perturbation reuse accumulates attack strength, allowing fewer iterations to produce examples as strong as or stronger than traditional PGD-k.The section presents neighboring-epoch transferability as the basis for this approach.Source: §3.1 · p. 3, §3 · p. 3, §3.2 · p. 4

4. Adversarial training with transferable adversarial examples

• ATTA starts each epoch’s attack from a saved adversarial example transformed by connection function C, rather than from the natural image.Repeated reuse is intended to accumulate attack strength across epochs.Source: §4 · p. 4, Figure 4 · p. 5

4.1. Connection function design

• Inverse data augmentation transfers padded images so reused perturbations cover augmentation background pixels; periodic resets mitigate the effect of early perturbations.The design responds to augmentation mismatch and drastic parameter changes in early training.Source: §4.1 · p. 5

4.2. Attack Loss

• The attack loss uses a fixed benchmark label, avoiding the epoch-varying objective noted for Equation 2 and offering a smaller computational graph.The loss represents divergence of adversarial examples from the benchmark label.Source: §4.2 · p. 6

5.1. Setup

• Experiments used MNIST and CIFAR10, with a four-convolutional, three-fully-connected-layer MNIST model and WRN-34-10 on CIFAR10.The perturbation bounds were ϵ = 0.3 for MNIST and ϵ = 0.031 for CIFAR10.Source: §5.1 · p. 6

5.2.1 Training efficiency

• 12.2× on MNIST and 14.1× on CIFAR10 training efficiency were achieved with comparable robustness; MAT accuracy improved by up to 7.2%.The Figure 6 scatter plot presents adversarial accuracy against PGD-20 attack and training time, and the authors report a better efficiency–effectiveness trade-off.Source: §5.2.1 · pp. 6–7, Figure 6 · p. 7
• 96.31% adversarial accuracy in 297 seconds for MNIST MAT ATTA-1; TRADES ATTA-1 reached 96.1% in 460 seconds, about 13× faster than TRADES(PGD-40).ATTA-1 was about 2.7× faster than YOPO and achieved 96.1% versus YOPO’s 93.69%.Source: §5.2.1 · pp. 6–7, Table 3 · p. 6
• 3.89% higher adversarial accuracy with about 15.1× training efficiency for MAT ATTA-1 versus PGD-10; MAT ATTA-10 improved accuracy by 7.26% with 2.9× efficiency.TRADES ATTA-3 matched PGD-10 accuracy with 5.3× faster training; MAT ATTA was also faster and more accurate than Free and YOPO in the reported comparisons.Source: §5.2.1 · p. 7, Table 4 · p. 7

5.2.2 Defense under other attacks

• ATTA remained comparably or more robust than baselines under PGD-100, FGSM, and CW-20; ATTA-40 performed better than PGD-40 under PGD-100.Table 5 compares MAT and TRADES models under these other attacks.Source: §5.2.2 · p. 7, Table 5 · p. 7

5.2.3 ATTA with ImageNet

• 97 hours on one 2080Ti were reported for ATTA ImageNet training, versus about 50 hours on four P100s reported for Free (m = 4).The evaluation used the MAT loss and an l∞ perturbation bound of ϵ = 2/255.Source: §5.2.3 · p. 8, Table 6 · p. 8

5.3. Ablation study

• Inverse data augmentation yielded about 6% higher accuracy for MAT and TRADES ATTA-1, while Equation 3 yielded 2.48% higher PGD-20 accuracy than Equation 2 for TRADES ATTA-1.Removing data augmentation hurt natural accuracy and robustness; the authors suggest Equation 3’s greater stability helps transferability between epochs.Source: §5.3 · p. 8, Table 7 · p. 8

• TRADES [39] adds a robustness regularizer, while BAT [34] uses perturbed labels and images to generate strong attacks with fewer iterations than Madry et al. [21].ATTA does not perturb labels and instead relies on transferability.Source: §6 · p. 8
• Free recycles gradients during training, whereas ATTA reuses perturbations across epochs rather than across iterations on the same batch.The paper calls these strategies inter-epoch and inner-batch reuse, respectively.Source: §6 · p. 8
• The authors state that ATTA outperforms existing techniques and achieves better robustness in many training scenarios.They also propose transferability analysis as a possible alternative way to understand Free.Source: §6 · p. 8

7. Conclusion

• High transferability between neighboring-epoch models lets ATTA accumulate attack strength by reusing prior-epoch perturbations, generating the same or stronger examples with fewer attack iterations.The authors report that this approach shortens training while achieving comparable or better robustness.Source: §7 · pp. 8–9
• ATTA is presented as a generic method applicable to other iterative-attack-based adversarial-training methods.Source: §7 · p. 9

A. Overview

• The supplementary overview says ATTA-1 can generate perturbations similar to PGD-40 on MNIST and PGD-10 on CIFAR10.It also points to supplementary experiment setup and complete evaluation results.Source: A. Overview · p. 10

B. Experiment setup

• MNIST uses the architecture from [21] [37] [39], no data augmentation, and perturbations bounded by an l∞ ball of ϵ = 0.3.Training lasts 60 epochs; perturbations are never reset, and robustness is evaluated with PGD, M-PGD, and CW attacks.Source: B. Experiment setup · p. 10
• CIFAR10 uses Wide-ResNet-34-10 [36], an l∞ perturbation bound of ϵ = 0.031, random flips, and four-pixel padding crops.Training lasts 40 epochs, perturbations reset every 10 epochs, and evaluation uses PGD, M-PGD, and CW attacks.Source: B. Experiment setup · p. 10
• ImageNet uses ResNet-50 [10] with an l∞ perturbation bound of ϵ = 2/255 and a 1/255 attack step size.The model is trained for 20 epochs with a piecewise learning-rate schedule.Source: B. Experiment setup · pp. 10–11
• Baselines use author implementations of MAT [21], TRADES [39], YOPO [39], and Free with recommended hyperparameters; experiments run on one RTX 2080 Ti GPU.Section 3 uses MAT with PGD-10 for training and PGD-20 to calculate loss and error rate; other attacks are implemented with the Adversarial Robustness Toolbox.Source: B. Experiment setup · p. 11

C.1. Qualitative study on training images

• ATTA-1 and PGD perturbations are compared at epoch 40 on MNIST and epoch 30 on CIFAR10, with perturbations rescaled for visualization.The comparisons use PGD-40 for MNIST and PGD-10 for CIFAR10; the rescaling uses 2× the perturbation divided by the L∞ bound.Source: §C.1 · p. 11
• ATTA-1's single-iteration perturbations resemble those from PGD-40 on MNIST and PGD-10 on CIFAR10.The visualization uses the model checkpoints trained with MAT-ATTA-1.Source: §C.1 · p. 11
• With inverse data augmentation on CIFAR10, ATTA-1 perturbations appear on padded pixels, whereas PGD-10 perturbations appear on cropped pixels.Source: §C.1 · p. 11

C.2. Complete evaluation results

• ATTA achieves comparable robustness with much less training time under additional attacks, and needs less time than other methods with the same attack iterations.The authors describe this as a better trade-off between training efficiency and robustness.Source: §C.2 · p. 11
• Accumulating perturbations lets ATTA use the same number of attack iterations for higher attack strength, which the authors say helps the model converge faster.Source: §C.2 · p. 11
• The stronger-attack evaluation tests ATTA-1 trained with MAT and TRADES losses against PGD-20 using an ϵ/4 step size and 20 independent random starts.Table 9 is described as reporting accuracy against PGD-20 under different settings.Source: §C.2 · p. 11, Table 9 · p. 11

D. Discussion

• Higher adversarial accuracy can coincide with lower natural accuracy, and stronger attacks tend to produce that same accuracy pattern.The paper notes that the trade-off has been observed and explained in prior work [33] [39], and discusses robust versus highly-predictive features [13].Source: D. Discussion · pp. 11–12
• Adversarial examples remain relevant across epochs, unlike images from traditional data augmentation, which are independent between epochs.The authors suggest this finding may inspire work to improve transferability between epochs.Source: D. Discussion · p. 12
• ATTA reuses perturbations across epochs, whereas Free reuses them across iterations on the same batch; the two reuse strategies are complementary.Source: D. Discussion · p. 12
• ATTA can outperform Free without inner-batch reuse or Free's warm start between batches.The paper notes that Free's cross-batch warm start can mismatch perturbations with images and reduce effectiveness.Source: D. Discussion · p. 12