Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attacks

Motivation

• A weaker two-step attack during training can be nearly as robust as a stronger seven-step attack, motivating a search for mechanisms beyond fitting more adversarial examples.The analysis instead observes much smaller gradient magnitudes in adversarially trained models than in undefended models.Source: §1 · p. 1

Method

• BAT perturbs both input images and labels to target low loss and small gradient magnitude during training.The method uses one-step PGD for images and a gradient-based adversarial-label formulation.Source: §1 · pp. 1–2

Motivation

• The paper frames robustness as requiring both low loss and small gradient magnitude, corresponding to zero-order and first-order conditions.This hypothesis motivates perturbing both images and labels.Source: §1 · p. 1

Results

• 63.7% versus 47.2% on CIFAR10 and 59.1% versus 42.1% on SVHN under PGD100 with cross-entropy loss demonstrate BAT’s improvement over the state-of-the-art baseline.On ImageNet, BAT trains using 8 GPUs versus 53 and 128 GPUs for comparison methods, while performing better on clean images and non-targeted attacks but worse on randomly targeted attacks.Source: §1 · p. 2

Method

• Random start and most-confusing-class targeted attacks mitigate label leaking and gradient masking, with those techniques alone achieving robustness similar to multi-step adversarial training.Adding adversarial labels then produces the reported state-of-the-art improvements.Source: §1 · p. 2

2.1. Adversarial Attacks

• Attack research includes optimization-based, gradient-based, iterative, margin-based, transfer-based, zeroth-order, and decision-based methods.The cited progression includes L-BFGS [51], FGSM [19], iterative attacks [28], DeepFool [38], CW [7], and related extensions.Source: §2.1 · p. 2
• Adversarial examples have also been studied beyond image classification, including detection, segmentation, speech recognition, captioning, reinforcement learning, and physical or geometric perturbations.These works broaden the attack setting beyond additive image perturbations.Source: §2.1 · p. 2

2.2. Adversarial Defenses

• Defense approaches include detecting adversarial examples, disrupting perturbation structure, gradient regularization, nearest-neighbor methods, and adversarial training.Prior work [1] warns that gradient masking can produce a false sense of robustness against strong attacks.Source: §2.2 · p. 2
• Adversarial training is presented as the strongest defense, with multi-step adversarial training [34] described as especially effective and resistant to gradient masking.Earlier approaches include one-step training and ensemble adversarial training.Source: §2.2 · pp. 2–3

3.1. Background

• The background method solves a minimax problem whose inner maximization generates adversarial images within an ℓ∞-bounded feasible region using random-start PGD.The outer minimization trains model parameters with the resulting adversarial examples.Source: §3.1 · p. 3

3.2. Analyses

• PGD2-8 performs roughly as robustly as PGD7-2 across attack strengths, suggesting that stronger training attacks do not necessarily produce more robust models.Both use an 8.0 perturbation budget and random starts; the variants differ in training iterations and step sizes.Source: §3.2 · p. 3, Table 1 · p. 3
• Adversarially trained models have much smaller gradient magnitudes than undefended models, while correctly predicted images have smaller gradients than wrongly predicted images.The comparison covers minimum, average, and maximum gradient magnitudes across CIFAR10 image collections.Source: §3.2 · p. 3, Table 2 · p. 3

3.3. Hypothesis

• The authors hypothesize that small gradient magnitude, or locally flat loss surfaces, helps achieve robustness against gradient-based attacks.They note that a locally flat surface should remain difficult to attack regardless of attack iterations.Source: §3.3 · pp. 3–4
• The paper explicitly leaves a rigorous treatment of the relationship between gradient magnitude, loss-surface flatness, and robustness for future work.It uses gradient magnitude as a practical flatness measure rather than proving the relationship formally.Source: §3.3 · p. 4
• Gradient regularization is a straightforward alternative, but its second-order derivatives make training slow and expensive for large networks.This limitation motivates a different fast-training formulation.Source: §3.3 · p. 4

4. Formulation

• BAT simultaneously perturbs input images and labels during adversarial training.The method replaces original training data with adversarially perturbed images and labels before optimizing network parameters.Source: §4 · p. 4
• Minimizing the perturbed loss approximately combines low clean loss with small input- and label-gradient magnitudes.The formulation links the first objective to correct predictions and the gradient objectives to difficulty for gradient-based attacks.Source: §4 · p. 4
• The training optimization decomposes into separate image and label subproblems, optimizing each adversarial variable conditioned on the other.The resulting adversarial image and label are then used as training data in place of the originals.Source: §4 · p. 4

4.1. Generating Adversarial Labels

• Adversarial labels are constructed within a perturbation budget and probability simplex by reducing the ground-truth mass and redistributing it across non-ground-truth classes.The redistribution is gradient-based, while the most-confusing class receives a very small share controlled by γ.Source: §4.1 · pp. 4–5
• The label budget ϵy is selected through a heuristic constraint that keeps the ground-truth probability at least β times larger than the largest non-ground-truth probability.Its feasible range depends on β and the number of classes, while β = ∞ recovers the original one-hot label.Source: §4.1 · p. 5
• When non-ground-truth probabilities are evenly distributed, the adversarial-label construction becomes label smoothing.For CIFAR10, the paper gives examples including ϵy = 0.1 with β = 81 and ϵy = 0.9 with β = 1.Source: §4.1 · p. 5

4.2. Generating Adversarial Images

• Targeted attacks and random initialization are used to reduce label leaking and gradient masking without the time or memory costs of multi-step PGD or model ensembles.Non-targeted attacks can expose the true label through their perturbations, while gradient masking can create false robustness against strong iterative attacks.Source: §4.2 · p. 5
• Adversarial images use fast targeted attacks, selecting the most-confusing class rather than the least-likely class as the target.The method uses one-step PGD for fast training, with two-step PGD on ImageNet, and reverses the update direction for targeted loss minimization.Source: §4.2 · p. 5

5.1. Random Start and MC Targeted Attack

• Random start prevents label leaking during one-step training, and R-MC performs best against FGSM and CE7 attacks among the tested one-step models.Without random start, FGSM and LL models show false robustness against FGSM but collapse under CE7; random-start variants are more robust.Source: §5.1 · p. 6, Table 3 · p. 6
• MC targeted attack is roughly as strong as FGSM and much stronger than LL targeted attack, motivating its use for adversarial training.FGSM remains vulnerable to label leaking and gradient masking, whereas MC targeting avoids the weakness of LL targeting.Source: §5.1 · pp. 6–7, Table 4 · p. 6

5.2. CIFAR10 Dataset

• R-MC-LA significantly improves robustness over the multi-step baseline on CIFAR10, while label adversary also improves multi-step training.R-MC-LS performs well, R-MC-LA performs better, and R-MC-LA(+) can outperform Madry-LA(+).Source: §5.2 · p. 7, Table 5 · p. 7

5.2.1 Different Perturbation Budgets during Training

• Training with ε_x = 4 achieves the best CIFAR10 robustness, while ε_x = 1 and ε_x = 12 produce less robust models.Accuracy drops quickly and then plateaus as PGD steps increase; results are averaged over three independently trained models.Source: §5.2.1 · p. 7, Figure 1 · p. 7, Table 6 · p. 7
• R-MC-LA+ (ϵx = 4) achieves 61.4% accuracy against 1000-step PGD with CE loss and 59.3% with CW loss, close to its 200-step PGD results.CW attacks are more effective than CE attacks when attacking R-MC-LA models.Source: §5.2.1 · p. 8

5.2.2 Black-box Attack Evaluation

• Black-box attacks are weaker than white-box attacks, and independently trained R-MC-LA models generate stronger transfer attacks than the undefended model.The comparison is presented as evidence that the models do not suffer from gradient masking.Source: §5.2.2 · p. 8, Table 7 · p. 8

5.3. SVHN Dataset

• R-MC-LA significantly outperforms state-of-the-art results on SVHN under clean evaluation and PGD attacks of varying strength.The SVHN experiments use a 10-way classification dataset without the additional training images.Source: §5.3 · p. 8, Table 8 · p. 8

5.4. ImageNet Dataset

• On ImageNet, the method outperforms prior art [26] against CE10-rd and beats recent work [56] on clean accuracy and non-targeted attacks, but trails it on randomly targeted attacks.Training uses two-step MC targeting because one-step training at the 16-pixel budget suffers severe label leaking and gradient masking; training takes about two days on eight GPUs.Source: §5.4 · p. 8, Table 9 · p. 8

6. Conclusion

• BAT perturbs both the adversarial image and adversarial label during training, using one- or two-step MC targeted attacks and a closed-form label formula.The conclusion summarizes experiments across CIFAR10, SVHN, and ImageNet against various white-box attacks.Source: §6 · p. 8

B. Two More Experiments on CIFAR10

• Table 10 reports CIFAR10 classification accuracy against various white-box attacks using training budget ϵx = 4.Source: Table 10 · p. 11

B.1. Against Other White-box Attacks

• Our models generally outperform TRADES across white-box attacks, except against DeepFool.The comparison uses MI-FGSM, original CW, and DeepFool attacks under the stated CleverHans settings.Source: §B.1 · p. 11, Table 10 · p. 11

B.2. Effect of Number of Training Images

• Varying the number of training images per class tests the claim that adversarial training requires more data than regular training.The corresponding CIFAR10 results are reported in Table 11; Table 15? No, Table 11 is the relevant data-size comparison.Source: §B.2 · p. 11, Table 11 · p. 11

C. CIFAR100 Dataset

• R-MC-LA models outperform the state-of-the-art on clean images and FGSM, while performing comparably on multi-step attacks.The R-MC-LA9 variant distributes ϵy across the top-9 non-groundtruth classes and brings several percentage-point gains.Source: C. CIFAR100 Dataset · p. 11

D. Gradient Magnitude

• Adversarially trained models have one- or two-order-of-magnitude smaller gradient magnitude than undefended models across CIFAR10, SVHN, and ImageNet.The comparison includes Madry’s and the authors’ models, with dataset-specific training budgets.Source: D. Gradient Magnitude · pp. 11–12, Table 13 · p. 12
• Gradient magnitude distinguishes undefended from adversarially trained models but cannot confidently identify which adversarially trained model is more robust.The paper therefore compares and reports attack accuracy, while describing precise robustness indicators as an active unsolved topic.Source: D. Gradient Magnitude · p. 11

• The alternative heuristic distributes ε_y among non-groundtruth classes in proportion to their respective gradients, using v_k = ∇_{y_k}L(x, y; θ).Source: E. Another Solution to Generating… · p. 12
• The solution is an exact application of gradient ascent, providing a direct gradient-based construction for the adversarial label.Source: E. Another Solution to Generating… · p. 12
• The authors favor the main-paper heuristic because it yields a higher maximization objective and a smaller y′_MC than solution (26).The main solution allocates more share to classes with larger gradients, while keeping the most-confusing class share smaller.Source: E. Another Solution to Generating… · p. 12
• Under CIFAR10 white-box attacks, the main solution is slightly better than the supplementary solution against multi-step PGD attacks.Table 14 evaluates R-MC-LA models trained with the two adversarial-label constructions using β = 9 and ε_x = 8.Source: E. Another Solution to Generating… · p. 12, Table 14 · p. 12