PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Robust Training via Backward Smoothing

Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu, Jingjing Liu

arXiv:2010.01278v2cs.LGcs.AIstat.ML

TL;DR

Adversarial training is robust but computationally expensive, while single-step Fast AT is faster yet less stable and less robust. This paper interprets random initialization as randomized smoothing and introduces backward smoothing, achieving robustness similar to TRADES with approximately 3x less training time under the same schedule.

  • Problem

    Adversarial training requires costly iterative attacks, while single-step random-initialization training remains unstable and less robust than state-of-the-art methods.

  • Method

    The paper interprets Fast AT random initialization as randomized smoothing and proposes backward smoothing to strengthen smoothing within the perturbation ball.

  • Results

    The proposed method achieves similar model robustness to TRADES while improving training time by approximately 3x under the same schedule.

  • Takeaways & Limitations

    Randomized smoothing provides an optimization perspective for improving the stability and robustness-efficiency trade-off of single-step robust training.

  • Takeaways & Limitations

    Fast AT with random initialization can still experience catastrophic overfitting and fail to solve the inner maximization, especially with very large attack step sizes.

Abstract

from arXiv · show

Adversarial training is so far the most effective strategy in defending against adversarial examples. However, it suffers from high computational costs due to the iterative adversarial attacks in each training step. Recent studies show that it is possible to achieve fast Adversarial Training by performing a single-step attack with random initialization. However, such an approach still lags behind state-of-the-art adversarial training algorithms on both stability and model robustness. In this work, we develop a new understanding towards Fast Adversarial Training, by viewing random initialization as performing randomized smoothing for better optimization of the inner maximization problem. Following this new perspective, we also propose a new initialization strategy, backward smoothing, to further improve the stability and model robustness over single-step robust training methods. Experiments on multiple benchmarks demonstrate that our method achieves similar model robustness as the original TRADES method while using much less training time ($\sim$3x improvement with the same training schedule).

1 Introduction

Adversarial and TRADES training provide strong robustness but are costly because they solve an iterative inner maximization problem. Fast AT reduces cost with random initialization, while this work interprets that initialization as randomized smoothing and proposes backward smoothing to improve stability and robustness.

  • 1 Introduction: Adversarial training and TRADES are effective defenses, but their iterative projected-gradient attacks make training highly time-consuming.The computational burden comes from repeatedly solving the inner maximization during each outer minimization step.
  • 1 Introduction: Fast AT uses a random initialization before a single-step attack, making training faster but leaving stability and robustness below state-of-the-art robust training.The approach can still be unstable and its effectiveness was initially not explained.
  • 1 Introduction: Backward smoothing achieves similar model robustness to TRADES while improving training time by approximately 3x under the same schedule.The method is designed to improve both stability and robustness over single-step random-initialization training.
  • 1 Introduction: The paper views random initialization as randomized smoothing that makes the inner maximization easier to optimize, then strengthens this effect through backward smoothing.The proposed perspective motivates a new initialization strategy within the perturbation ball.

2 Related Work

Prior work spans adversarial attacks, robust training, and randomized smoothing. This paper builds on efficient single-step robust training while using randomized smoothing for optimization rather than certified defenses.

  • 2 Related Work: Adversarial attack research includes gradient-based, black-box, hard-label, and ensemble-based methods targeting effectiveness, efficiency, or robustness evaluation.The literature includes FGSM, PGD, finite-difference gradient estimation, hard-label attacks, and attack ensembles.
  • 2 Related Work: Adversarial training became an effective defense, followed by variants addressing robustness, training factors, extra information, and training efficiency.Efficient approaches include free adversarial training and Fast AT with single-step FGSM and random initialization.
  • 2 Related Work: Randomized smoothing was developed for nonsmooth optimization and later applied to certified defenses, whereas this paper uses it to explain Fast AT rather than provide certified guarantees.The distinction places randomized smoothing here in an optimization framework, not a certification framework.

3 Pros and Cons of Random Initialization

Random initialization stabilizes single-step adversarial training by smoothing the inner objective, but it remains vulnerable to instability and robustness gaps.

  • What Caused the Failure of One-step AT Without Random Initialization?: Fast AT without random initialization fails because its post-attack loss can decrease, indicating the one-step update does not solve the inner maximization.The attack step must be near ϵ for effective defense, but this large step can cause divergence.
  • Why Random Initialization Helps?: Random initialization acts as randomized smoothing, producing a smoother objective that permits larger step sizes and reduces divergence during single-step optimization.With u = ϵ and one stochastic gradient update, the smoothed formulation reduces to Fast AT.
  • Drawbacks of Random Initialization: Fast AT with random initialization can still suffer catastrophic overfitting and fail to solve the inner maximization, especially with very large attack step sizes.The failure may occur intermittently despite random initialization.
  • Drawbacks of Random Initialization: Fast AT reaches 46.30% robust accuracy in its best CIFAR-10 run but averages 42.11%, while remaining 6% below TRADES in robust accuracy.The variability reflects failed runs and the remaining robustness gap relative to the original TRADES method.

4 Proposed Approaches

The proposed approaches first apply randomized smoothing to TRADES for single-step training, then introduce backward smoothing to obtain stronger smoothing within the perturbation constraint.

  • A Naive Try: Randomized Smoothing for TRADES: Fast TRADES applies random initialization followed by a single-step projected gradient ascent attack to the TRADES objective.The method uses standard piecewise learning-rate decay and tests the strategy on ResNet-18 trained on CIFAR-10.
  • A Naive Try: Randomized Smoothing for TRADES: Fast TRADES slightly outperforms Fast AT but remains far behind original TRADES, motivating a stronger smoothing strategy.
  • A Naive Try: Randomized Smoothing for TRADES: Larger smoothing cannot be obtained simply by increasing the random perturbation because adversarial initialization remains constrained to the ϵ-ball.The best Fast TRADES model uses attack step size 6/255, compared with 10/255 for Fast AT.
  • Backward Smoothing: Backward smoothing samples perturbations in the output space and finds corresponding input perturbations to initialize the adversarial attack.It generates fθ(x) + γψ with ψ sampled uniformly from (−1, 1), then searches backward for the input perturbation ξ.
  • Backward Smoothing: The algorithm solves for the backward-smoothing perturbation and adversarial perturbation with single-step PGD before updating θ at x + ξ∗ + δ∗.The proposed procedure initializes ξ as zero and uses stochastic gradients for the model update.
  • Backward Smoothing: Backward smoothing produces a much smaller Hessian maximum eigenvalue than random smoothing, indicating a smoother loss during training.Figure 3 compares original, randomly perturbed, and backward-smoothed examples across training epochs; smaller eigenvalues correspond to smoother loss.

5 Experiments

Across CIFAR-10, CIFAR-100, and Tiny ImageNet, Backward Smoothing provides robustness comparable to TRADES with substantially less training time. Evaluations also indicate improved stability and robustness over fast training baselines, including under stronger attacks and additional acceleration techniques.

  • Performance Comparison with Robust Training Baselines: Backward Smoothing achieves robust accuracy nearly as good as TRADES on CIFAR-10 while using approximately 3x less training time.It also outperforms Fast AT despite roughly twice its training cost.
  • Performance Comparison with Robust Training Baselines: Backward Smoothing’s advantage is not explained by a robustness-accuracy trade-off, because it achieves better robustness at roughly the same natural accuracy as fast baselines.The comparison varies TRADES’s β parameter while fast methods provide single operating points.
  • Performance Comparison with Robust Training Baselines: Compared with Fast AT, Backward Smoothing mitigates catastrophic overfitting during later training stages.The cited comparison evaluates test accuracy across training epochs for normal and catastrophically overfitting Fast AT runs.
  • Performance Comparison with Robust Training Baselines: On CIFAR-100, Backward Smoothing achieves slightly higher robustness than TRADES with much less training time, while improving over comparable two-step fast methods.On Tiny ImageNet, it also significantly improves robustness over other single-step robust training methods.
  • Evaluation with State-of-the-art Attacks: Under AutoAttack and RayS, Backward Smoothing remains comparable to TRADES, trails it by only 2% in robust accuracy, and matches it in average decision boundary distance.It exceeds standard Adversarial Training and Fast AT on the reported robustness measures.
  • Combining with Other Acceleration Techniques: Cyclic learning-rate decay and mixed-precision training reduce training time for all methods without changing Backward Smoothing’s robustness advantage over TRADES and Fast AT+.These techniques can also be applied to Backward Smoothing for further efficiency improvements.

6 Conclusions

The paper explains single-step robust training through randomized smoothing and introduces Backward Smoothing to improve its robustness-efficiency trade-off.

  • 6 Conclusions: Backward Smoothing avoids catastrophic overfitting and improves the robustness-efficiency trade-off over previous single-step robust training methods.
  • 6 Conclusions: The paper interprets random initialization in Fast Adversarial Training as randomized smoothing for optimizing the inner maximization problem.
  • A Randomized Smoothing: Randomized smoothing transforms a loss using random perturbations to incorporate information from neighboring areas and make optimization smoother.

B Additional Experiments

The additional experiments tune several hyperparameters to maximize the robustness performance of each method.

  • The experiments tune attack step size over [1/255, 2/255, ..., 10/255] for robustness performance.
  • The learning rate is varied over [0.001, 0.01, 0.1], while γ is varied over [0.1, 0.5, 1.0, 2.0, 5.0].
  • The regularization parameter β is varied over [2, 4, ..., 10] when tuning each method.

Stability and Sensitivity

Backward Smoothing is more stable than Fast AT, is less dependent on overly large attack steps, and is useful primarily as an initialization for single-step attacks.

  • Stability and Sensitivity: Backward Smoothing has much smaller robustness variances than Fast AT and similar variance to Fast TRADES while achieving higher average robustness than Fast TRADES.
  • Stability and Sensitivity: Backward Smoothing achieves similar robustness with a slightly smaller attack step size, with best performance at 8/255, unlike Fast AT’s reliance on larger steps until training fails.
  • Does Backward Smoothing alone works?: Backward Smoothing alone provides limited robustness because its loss does not directly promote adversarial attacks.
  • Does Backward Smoothing alone works?: Backward Smoothing therefore serves as an initialization that helps single-step attacks better solve the inner maximization problem.

More Experiments for Backward Smoothing using Multiple Random Points

Using one random point for Backward Smoothing provides performance similar to multiple random points while reducing training time.

  • A single random point already achieves similar performance to multiple random points in Backward Smoothing.
  • Using a single random point saves more time, matching the method’s goal of improving adversarial-training efficiency.
  • The proposed method therefore uses one random point for randomized smoothing.

Ablation Studies on γ and ϵ

The ablations examine how γ and β affect Backward Smoothing. γ is broadly insensitive except at extreme values, while β controls the natural–robust accuracy trade-off.

  • Ablation Studies on γ and ϵ: The study includes ablations intended to provide a more detailed analysis of Backward Smoothing.
  • Ablation Studies on γ and ϵ: γ = 1 achieves the best robustness on both CIFAR-10 and CIFAR-100, while overly large or small values slightly worsen it.Overall, γ does not significantly change final robustness.
  • Ablation Studies on γ and ϵ: Larger β steadily decreases natural accuracy, while the best robustness is obtained with β = 10.0.As in TRADES, β controls the trade-off between natural and robust accuracy.

Experiments on different perturbation strength

Across perturbation strengths, Backward Smoothing outperforms the compared baselines, while its combination with PGD yields smaller gains than its combination with TRADES. Its formulation also differs from ODI in both objective and motivation.

  • Experiments on different perturbation strength: At ϵ = 4/255 and ϵ = 12/255, Backward Smoothing outperforms the other baselines on CIFAR-10.The comparisons are reported in Tables 13 and 14.
  • PGD based Backward Smoothing: Combining Backward Smoothing with PGD provides improvements, but not as much as combining it with TRADES.For adversarial training, the initialization uses the y-logit because CrossEntropy primarily depends on it.
  • Comparison of Backward Smoothing and the ODI attack: Backward Smoothing uses a KL-divergence-based formulation to smooth the objective, whereas ODI uses a simple linear formulation and targets attack effectiveness.Applying ODI to robust training produces much worse robustness than Backward Smoothing for both PGD-based and TRADES-based strategies.
Loading 2010.01278v2…