PaperlayerView exact revision on arXiv

Source-linked AI summary

Bag of Tricks for Adversarial Training

Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, Jun Zhu

arXiv:2010.00467v3cs.LGcs.CVstat.ML

TL;DR

The paper investigates whether inconsistent, overlooked training settings confound comparisons among adversarial-training methods. Through CIFAR-10 ablations and re-implementations, it finds that robustness is highly sensitive to these settings and that standardized recipes can improve reported results.

  • Problem

    Recent benchmarks suggest that many proposed adversarial-training improvements are no more effective than early stopping, motivating scrutiny of inconsistent implementation details.

  • Method

    The authors evaluate overlooked training tricks and hyperparameters on CIFAR-10, then apply a baseline recipe to PGD-AT and re-implement TRADES.

  • Results

    Changing weight decay from 2 × 10−4 to 5 × 10−4 improves TRADES clean accuracy by ∼1% and AutoAttack accuracy by ∼4%.

  • Takeaways & Limitations

    Fair comparisons of adversarial-training methods require consistent implementation details and careful tuning of training settings.

  • Takeaways & Limitations

    The empirical conclusions may not generalize to other datasets or adversarial-training frameworks.

Abstract

from arXiv · show

Adversarial training (AT) is one of the most effective strategies for promoting model robustness. However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure. This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods. Surprisingly, we find that the basic settings (e.g., weight decay, training schedule, etc.) used in these methods are highly inconsistent. In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models. Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought. For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7%, which is probable to override the potential promotion induced by the proposed methods. We conclude a baseline training setting and re-implement previous defenses to achieve new state-of-the-art results. These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.

1 INTRODUCTION

The paper argues that inconsistent implementation details can obscure the effectiveness of adversarial-training methods. It evaluates overlooked training settings, proposes a baseline recipe, and shows that retraining can improve benchmark results.

  • Recent benchmarks find that early stopping PGD-AT can match gains from almost all previously proposed improvements, including TRADES.
  • AT papers use inconsistent basic settings, including weight decay, learning-rate schedules, initialization, and batch-normalization mode.
  • The study evaluates training tricks such as warmup, early stopping, weight decay, batch size, and batch-normalization mode for adversarially trained models.
  • A baseline PGD-AT recipe is provided, and the approach is extended to frameworks such as TRADES.
  • The authors caution that their empirical conclusions may not generalize to other datasets or tasks.

2 RELATED WORK

Related work spans empirical and certified adversarial defenses, acceleration methods, robustness benchmarks, and corruption-based evaluations. The paper uses PGD and AutoAttack as common evaluation methods.

  • Adversarial defenses include empirical adversarial training, provably robust networks, and randomized smoothing, though certified methods do not currently match state-of-the-art empirical robustness.
  • Proposed adversarial-training improvements draw on ensemble learning, metric learning, generative modeling, and related mechanisms.
  • Other work accelerates adversarial training through computation reuse, adaptive adversarial steps, or one-step training.
  • Benchmarks such as robustness curves, AutoAttack, and RayS rank existing defenses under adversarial attacks.
  • Corruption and perturbation datasets such as MNIST-C, ImageNet-C, and ImageNet-P evaluate robustness beyond adversarial attacks.

3 BAG OF TRICKS

The paper evaluates overlooked training choices for adversarially trained CIFAR-10 models and finds that robustness depends strongly on implementation details. It proposes consistent baseline settings and shows that retraining can substantially improve results.

  • Experimental setup: The study evaluates adversarial training on CIFAR-10 under the ℓ∞ threat model with ϵ = 8/255, using PGD-10 and AutoAttack.Training uses no additional data; PGD evaluation uses step size 2/255 and 5 restarts.
  • Training choices: Basic settings such as weight decay, learning-rate schedules, warmup, batch size, label smoothing, BN mode, and activation functions can materially affect robustness.The authors investigate these choices separately and in combination.
  • Training hyperparameters: Mild label smoothing improves robust accuracy by 0.5 ∼1% without affecting clean performance, whereas excessive smoothing can degrade robustness.The reported gains include AutoAttack and PGD-1000 evaluations, but the authors caution about potential gradient masking.
  • Weight decay: Up to ∼7% robust-accuracy difference arises between weight decay values 1 × 10−4 and 5 × 10−4, while clean accuracy is much less sensitive.Smaller weight decay accelerates early learning but causes earlier robust overfitting; 5 × 10−4 can enlarge decision-boundary margins.
  • Model architecture and activation: Smooth activations benefit ResNet-18 more than larger WRN models, suggesting greater usefulness when model capacity is insufficient for adversarial training.On CIFAR-10, zero-truncated activations perform better than activations with negative output values such as ELU, LeakyReLU, and Tanh.
  • Re-implementation: Changing TRADES weight decay from 2 × 10−4 to 5 × 10−4 improves clean accuracy by ∼1% and AutoAttack accuracy by ∼4%, surpassing previously reported state-of-the-art models.The result supports standardized training settings for fair comparisons.

4 CONCLUSION

The paper emphasizes that overlooked implementation details can substantially affect adversarially trained models, motivating careful training-setting selection. It also situates PGD and AutoAttack as robustness-evaluation tools.

  • Implementation details can strongly affect adversarially trained model performance, so training settings require careful examination.
  • PGD is a commonly studied gradient-based attack that iteratively crafts adversarial examples from a randomly perturbed input.
  • The clipping function constrains each PGD update, while adversarial loss L determines the optimization objective.
  • AutoAttack combines APGDCE, APGDDLR, FAB, and square attack into an ensemble for robustness evaluation.

A.2 REFERENCE CODES

The paper provides code links for referred defenses and derives summarized training settings from papers or their implementations.

  • The paper supplies code links for the defense methods discussed in Table 1.
  • Summarized training settings come from descriptions in the papers or manual retrieval from their code implementations.

A.3 MODEL ARCHITECTURES

The study examines diverse hand-crafted architectures and relates architecture choice to robustness, efficiency, and inference considerations. It also places the work alongside other studies of adversarial-training tricks.

  • A.3 MODEL ARCHITECTURES: The evaluated architectures include DenseNet, GoogleNet, ResNet variants, SENet, WRN, DPN, ResNeXt, and RegNetX.
  • A.3 MODEL ARCHITECTURES: The selected architectures have comparable parameter counts and are implemented using the pytorch-cifar repository.
  • A.3 MODEL ARCHITECTURES: Inference-phase defenses include input transformations, image processing, specified inference principles, and detection-based interventions.
  • A.3 MODEL ARCHITECTURES: The paper complements related training-trick analyses that study factors such as batch size, label smoothing, weight decay, activation functions, weight moving average, and data quality.

B ADDITIONAL RESULTS

This section presents additional results intended to further support the conclusions stated in the main text.

  • B ADDITIONAL RESULTS: Additional results are provided to further support the conclusions in the main text.

B.1 EARLY DECAYS LEARNING RATE

Early learning-rate decay can prevent overfitting for lower weight decay, preserving clean accuracy but not matching the robustness of higher weight decay. Clean accuracy is comparatively insensitive to weight decay in both standard and adversarial training.

  • Early decay: Lower weight decay values train faster but tend to overfit, motivating learning-rate decay at 40 or 45 rather than 100 or 105 epochs.The earlier decay is applied just before overfitting.
  • Early decay: Models with early learning-rate decay achieve the same clean accuracy as weight decay 5 × 10−4 but worse robustness.
  • Clean versus robust accuracy: Different weight-decay values produce comparable final clean accuracy under standard training on CIFAR-10.The passage states that this observation also holds for adversarially trained models.
  • Additional evaluation: Softplus and batch-normalization mode are evaluated on ResNet-18 in Table 13.
  • Additional evaluation: Table 13 compares smooth activation functions and evaluation-mode batch normalization for TRADES.

B.3 RESULTS OF EARLY STOPPING, WARMUP, AND OPTIMIZERS ON WRN-34-10

This section reports WRN-34-10 CIFAR-10 evaluations of early stopping, warmup, and optimizer choices. The supplied passages identify the evaluated settings and learning-rate initialization, but do not provide their accuracy outcomes.

  • Evaluation setup: Tables 14 and 15 report WRN-34-10 test accuracy on CIFAR-10 for early stopping, warmup, and different optimizers.
  • Early stopping and warmup: For early-stopping attack iterations, notation such as 40 / 70 denotes epochs when the tolerance step increases by one.
  • Early stopping and warmup: Warmup linearly increases the learning rate and maximal perturbation from zero to preset values during the first 10, 15, or 20 epochs.
  • Optimizers: Adam and AdamW use an initial learning rate of 0.0001, whereas the other optimizers use 0.1.

B.4 RANK IN THE AUTOATTACK BENCHMARK

The AutoAttack comparison uses retrained models and shows that changing TRADES weight decay alone places the model at the state-of-the-art position in the benchmark.

  • Benchmark comparison: Changing TRADES weight decay from 2 × 10−4 to 5 × 10−4 sends the model back to the state-of-the-art position in the AutoAttack benchmark.
  • Benchmark comparison: The evaluated methods use no additional training data on CIFAR-10, and the reported results for the authors’ model are means and standard deviations over five random seeds.
  • Benchmark comparison: The authors’ TRADES model differs from original TRADES only in weight decay while using evaluation-mode batch normalization and ReLU activation.

B.5 MORE EVALUATIONS ON LABEL SMOOTHING

The label-smoothing evaluation tests ResNet-18 trained by PGD-AT across strong gradient-based and SPSA attacks. The supplied passages specify the attack configurations but not the resulting accuracies.

  • Label smoothing: Table 17 evaluates different label-smoothing values for ResNet-18 trained by PGD-AT on CIFAR-10.
  • Attack evaluation: PGD-1000 evaluation varies the number of restarts and step sizes using cross-entropy and C&W objectives.
  • Attack evaluation: SPSA evaluation uses 10,000 iterations, batch size 128, perturbation size 0.001, and learning rate 1/255.
Loading 2010.00467v3…