Source-linked AI summary
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, Liwei Wang
TL;DR
Adversarial training is attack-dependent and costly, motivating a scalable alternative for provable robustness. MACER directly maximizes the certified radius of randomized-smoothed classifiers without adversarial training, and experiments report better performance with lower training time across tasks. The paper also identifies practical differentiability and numerical-stability challenges in optimizing certified radii.
Problem
Adversarial training depends on attack iterations and is costly, while robust training must provide guarantees beyond a particular attack strategy.
Method
MACER trains smoothed classifiers by directly maximizing their certified radius instead of using adversarial training.
Results
MACER performs better than previous provable l2-defenses across the evaluated tasks and uses 39% less training time than adversarial training on ImageNet.
Takeaways & Limitations
The results suggest that adversarial training is not necessary for robust training and that certification-based defense is a promising research direction.
Takeaways & Limitations
Monte Carlo estimation of the indicator-based expectation is nondifferentiable from an algorithmic perspective, so the differentiability condition is not directly met.
Abstract
from arXiv · showhide
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly. In this paper, we propose the MACER algorithm, which learns robust models without using adversarial training but performs better than all existing provable l2-defenses. Recent work shows that randomized smoothing can be used to provide a certified l2 radius to smoothed classifiers, and our algorithm trains provably robust smoothed classifiers via MAximizing the CErtified Radius (MACER). The attack-free characteristic makes MACER faster to train and easier to optimize. In our experiments, we show that our method can be applied to modern deep neural networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and SVHN. For all tasks, MACER spends less training time than state-of-the-art adversarial training algorithms, and the learned models achieve larger average certified radius.
1 INTRODUCTION
MACER addresses the attack dependence and cost of adversarial training by directly maximizing the certified radius of randomized-smoothed classifiers, while targeting scalable robust training. The paper reports stronger performance and lower training time across evaluated tasks.
- Motivation: Adversarial training generates attack-specific examples during optimization, making formal robustness against other attacks uncertain and training expensive.Its multiple attack iterations make adversarial training run slowly.
- Contribution: MACER is presented as applicable to architectures of any size, unlike certification methods limited to shallow ReLU networks.This scalability is framed as a practical advantage over prior certification-based training methods.
- Method: MACER trains robust models without adversarial attacks by directly incorporating the certified radius into its objective.The approach uses randomized smoothing and is designed for scalable deep-network training.
- Experiments: 39% less training time than adversarial training is reported for MACER on ImageNet while still achieving better performance.The paper also evaluates MACER on Cifar-10, MNIST, and SVHN.
2 RELATED WORK
Prior robust-training research includes attack-based adversarial training, accelerated variants, and certification-based defenses. Randomized smoothing offers another certification route for scalable l2 robustness, while earlier certification methods face architectural or computational limits.
- Adversarial training: Adversarial training improves robustness by adding adversarial examples, and PGD-based formulations cast training as min-max optimization.Recent work also targets the speed limitations of PGD-based training.
- Adversarial training: Adversarial-training accelerators reduce cost by replaying examples, restricting PGD backpropagation, or estimating adversaries through local linearization.These methods modify attack generation or gradient computation rather than removing attack dependence.
- Provable defense: Robustness certification computes a safe radius guaranteeing unchanged predictions within a perturbation ball, but many certification-based trainers handle only shallow networks.The cited methods use linear or convex relaxations for fully connected ReLU networks.
- Randomized smoothing: Randomized smoothing certifies l2 robustness by adding Gaussian noise to inputs, extending certification to settings beyond the shallow-network methods.The passage describes Gaussian-noise certification as a newer class of certification methods.
3 PRELIMINARIES
The preliminaries define robust classification through l2 perturbation balls and certified radii, then introduce randomized smoothing as a scalable way to obtain provable guarantees for deep neural networks. Certified error measures whether the certified radius reaches a target perturbation size.
- Robust classification: An adversarial example is x′ = x + δ that changes a correctly classified input’s label, with the analysis restricting δ to ||δ||2 ≤ ϵ.Robustness requires the prediction to remain unchanged for every perturbation in the l2 ball.
- Neural-network notation: Deep neural networks map inputs to class scores, induce predictions by arg max, and use softmax-normalized outputs for cross-entropy training.The softmax output is treated as a likelihood vector over the K classes.
- Robust classification: The robust radius is the largest l2-ball radius around an input within which the classifier preserves its prediction.Training aims to obtain l2 robustness with high probability over the data distribution.
- Certification: Directly computing a deep network’s exact l2 robust radius is difficult, motivating certified lower bounds instead.The paper notes that l1 robust-radius computation is NP-hard and suggests l2 computation is likely similarly hard.
- Certification: A certified radius CR(fθ; x, y) is a lower bound on the true robust radius and therefore supports provable robustness guarantees.Certified robust error counts a sample as correct only when the certified radius reaches the target ϵ.
- Randomized smoothing: Randomized smoothing predicts with gθ by applying fθ to inputs perturbed by Gaussian noise, and it provides an analytically computable certified radius.The smoothed classifier returns the label most likely under noise distributed as N(x, σ2I).
4 ROBUST TRAINING VIA MAXIMIZING THE CERTIFIED RADIUS
MACER trains robust smoothed classifiers by maximizing a certified radius while addressing differentiability and numerical-stability challenges, without adversarial attack iterations.
- Certified-radius training: Randomized-smoothing certified-radius estimates can be computed for arbitrary deep neural networks, motivating radius-maximizing robust training.The method estimates expectations by repeatedly sampling Gaussian noise.
- Surrogate objective: The certified robust error is decomposed into classification error and robustness error, with cross-entropy and certified-radius hinge losses serving as differentiable surrogates.The surrogates are designed to upper-bound their corresponding 0/1 errors and remain numerically stable.
- Differentiable certified radius: Soft randomized smoothing replaces hard indicator predictions with a softmax-based smoothed classifier, making the certified radius sub-differentiable in practice.The soft radius converges to the hard randomized-smoothing radius as β approaches infinity and can itself provide a provable guarantee.
- Numerical stability: Directly optimizing the soft certified radius can cause exploding gradients because Φ^-1 has a large derivative near 0 and 1.The hinge robustness loss restricts nonzero-loss samples to a region where the derivative is bounded.
- Complete implementation: MACER estimates Gaussian expectations with Monte Carlo samples and minimizes its objective using a single model-update loop rather than adversarial-example generation.Avoiding additional backpropagations for attack generation makes the method faster than adversarial training.
5 EXPERIMENTS
Experiments evaluate MACER across multiple datasets and compare certified robustness, training efficiency, and hyperparameter effects against smoothed-model baselines. MACER generally achieves stronger certified-radius performance and substantially faster training than attack-based robust training.
- Experimental setup: MACER is evaluated on Cifar-10, ImageNet, MNIST, and SVHN using standard architectures for fair comparison.The experiments use LeNet for MNIST, ResNet-110 for Cifar-10 and SVHN, and ResNet-50 for ImageNet.
- Experimental setup: The baselines are Cohen et al.’s cross-entropy-trained smoothed networks and Salman et al.’s adversarially trained smoothed networks.Cohen et al.’s method is also rerun as a special case of MACER with k = 1 and λ = 0.
- Evaluation metrics: Average certified radius (ACR) complements radius-specific certified accuracy because certified accuracy varies with the chosen radius.ACR is computed by averaging estimated certified radii over the test set.
- Results: On Cifar-10, MACER consistently exceeds Cohen et al. in approximated certified test accuracy and exceeds Salman et al. in ACR across settings.For σ = 0.25 and 0.50, MACER’s ACR is about 3% larger than Salman et al.’s; at σ = 1.0, gains are smaller because both models perform poorly.
- Results: On ImageNet, MACER reaches ACR = 0.544 in 117.90 hours, compared with ACR = 0.528 in 193.10 hours for Salman et al.The comparison is performed on the same machine using the authors’ code.
- Results: With only 150 training epochs, MACER remains comparable to SmoothAdv while training four times faster.This addresses whether MACER’s performance depends on using more epochs than previous methods.
- Effect of hyperparameters: Increasing k usually improves performance, while increasing λ lowers clean accuracy but raises certified accuracy at large radii.The radius-accuracy curve for k = 16 is uniformly above that for k = 1; λ therefore controls an accuracy-robustness trade-off.
- Effect of hyperparameters: Increasing γ improves robust accuracy, whereas applying a large β only to the robustness loss preserves stability while increasing ACR.The effect of γ is less significant than λ’s, and small γ truncates large radii.
6 CONCLUSION AND FUTURE WORK
MACER is presented as an attack-free, scalable method that directly maximizes the certified radius of a smoothed classifier. Experiments report better performance than previous provable l2-defenses and faster training, while future work targets semisupervised learning.
- MACER directly maximizes the certified radius of a smoothed classifier to train robust models without adversarial training.
- Extensive experiments report that MACER performs better than previous provable l2-defenses and trains faster.
- Strong empirical results suggest that adversarial training is not required for robust training and identify certification-based defense as a promising research direction.
- The authors plan to extend MACER to the semisupervised setting, motivated by recent work on unlabeled data and robust generalization.
A.1 PROOF OF THEOREM 2
The section develops the theoretical basis and certification procedures for Soft-RS. It uses Lipschitz smoothing, confidence bounds, and sampling algorithms, then compares Soft-RS with Hard-RS empirically.
- The proof begins from a lemma stating that Φ^-1 of a Gaussian-smoothed measurable function is 1/σ-Lipschitz.
- The certification analysis reduces robustness certification to obtaining a confidence lower bound for zA and provides Hoeffding and empirical Bernstein bounds.
- The certification procedure estimates noisy class outputs, selects a predicted class, and returns a certified radius or abstains.
- Hard-RS consistently gives a larger lower bound of robust radius than Soft-RS, while the empirical Bernstein bound remains looser than the Clopper-Pearson bound.
B PROOF OF PROPOSITION 1
The proposition proof bounds the smoothed classifier’s certified-radius expression by analyzing the inverse Gaussian CDF and the class-probability constraints.
- The proof restricts attention to the case where the inverse-CDF probability difference does not exceed γ because the derivative is zero otherwise.
- Monotonicity and class-probability constraints bound p1 and p2 within intervals determined by p* and the number of classes K.
C SUPPLEMENTARY MATERIAL FOR EXPERIMENTS
The supplementary experiments identify the compared Cifar-10 and ImageNet models and describe the setup for a shortened MACER training comparison.
- C.1 COMPARED MODELS: The supplementary material lists Cifar-10 models in Table 4 and ImageNet models in Table 5.
C.2 RESULTS ON MNIST AND SVHN
The section reports MNIST and SVHN experiments, including comparisons with Cohen et al. (2019)’s method and certified-performance metrics summarized in tables.
- MNIST and SVHN experiments compare the reported results with Cohen et al. (2019)’s method.
- The reported experiments use k = 16, λ = 16.0, γ = 8.0, and β = 16.0 for all σ in one setting, and λ = 12.0 with the same remaining values in another.Training first uses λ = 0.0 before changing to λ = 12.0 after learning-rate decay.
- Table 6 reports approximated certified test accuracy and ACR on MNIST across l2 radii.
- Table 7 reports approximated certified test accuracy and ACR on SVHN across l2 radii.
C.3 MACER TRAINING FOR 150 EPOCHS
This section evaluates MACER trained for 150 epochs on Cifar-10, comparing it with SmoothAdv and a 440-epoch MACER model while examining hyperparameter effects.
- 150-epoch MACER on Cifar-10 is evaluated against SmoothAdv and MACER trained for 440 epochs.The comparison reports both performance and training time.
- Training schedule: The 150-epoch training schedule decays the learning rate by 0.1 at epochs 60 and 120, while keeping other hyperparameters as in Table 4.
- The Cifar-10 experiments use σ = 0.25 or 0.50, with detailed settings in Table 9 and results in Tables 10–13.
- Hyperparameter effects: Tables 10–13 examine the effects of k, λ, γ, and β using approximated certified test accuracy and ACR.