PaperlayerView exact revision on arXiv

Source-linked AI summary

Smooth Adversarial Training

Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, Quoc V. Le

arXiv:2006.14536v2cs.LGcs.CVcs.NE

TL;DR

Adversarial training is commonly viewed as requiring an accuracy or computational trade-off, and ReLU’s non-smoothness may weaken its gradients. The paper proposes smooth adversarial training, which replaces ReLU with smooth approximations, and reports stronger ImageNet robustness without sacrificing accuracy or adding computation. SAT also achieves strong results with EfficientNet-L1, while the paper notes exceptions on CIFAR-10 and a forward–backward discrepancy.

  • Problem

    Adversarial robustness is commonly thought to require sacrificing clean accuracy or increasing computational resources.

  • Method

    Smooth adversarial training replaces ReLU with smooth approximations to improve gradient quality during adversarial training.

  • Results

    SAT improves ResNet-50 robustness from 33.0% to 42.3% and accuracy by 0.9% on ImageNet without additional computation, while EfficientNet-L1 reaches 82.2% accuracy and 58.6% robustness.

  • Takeaways & Limitations

    Smooth activation functions can improve adversarial training robustness without sacrificing accuracy or increasing computational cost.

  • Takeaways & Limitations

    On CIFAR-10, ELU and SILU hurt adversarial training, and the paper leaves optimal settings for these activations to future work.

Abstract

from arXiv · show

It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.

1. Introduction

Adversarial training improves robustness but is commonly associated with an accuracy trade-off, while ReLU’s non-smooth gradients weaken the training process. Smooth adversarial training replaces ReLU with smooth approximations and reports stronger ImageNet results without added computation.

  • Adversarial training improves resilience by training networks with adversarial examples, but stronger robustness commonly sacrifices clean-input accuracy.
  • ReLU significantly weakens adversarial training because its gradient changes abruptly near zero.
  • Smooth adversarial training replaces ReLU with smooth approximations to improve gradient quality, generate harder training examples, and compute better updates.
  • 9.3%: SAT increases ResNet-50 robustness from 33.0% to 42.3% on ImageNet while raising standard accuracy by 0.9% without additional computation.
  • 82.2% accuracy and 58.6% robustness: EfficientNet-L1 surpasses prior art by 9.5% in accuracy and 11.6% in robustness on ImageNet.

2. Related Works

The related work frames SAT as an alternative to the prevailing accuracy–robustness trade-off and to defenses that rely on degraded gradients. It connects smooth activations to both practical adversarial training and prior theoretical robustness analysis.

  • Existing adversarial-training methods generally improve robustness by sacrificing clean-image accuracy or adding computational cost.
  • SAT is presented as improving adversarial robustness without accuracy degradation or additional computational cost.
  • Prior theory suggests smooth alternatives to ReLU can yield tractable bounds for certifying distributional robustness.
  • The paper empirically examines smooth activations in practical adversarial training on real-world data with large networks.
  • Many alternative defenses degrade gradient quality, potentially inducing gradient masking and a false sense of robustness.

3. ReLU Weakens Adversarial Training

The study decomposes adversarial training into attack generation and parameter updates, then tests how replacing ReLU gradients with Parametric Softplus gradients affects each step. Better gradients strengthen both the attacker and optimizer, producing higher robustness without the trade-offs seen from simply training longer.

  • Adversarial Training: Adversarial training combines inner maximization, which generates adversarial examples, with outer minimization, which updates network parameters.
  • Smooth Approximation: Parametric Softplus is a smooth ReLU approximation with a continuous derivative, controlled by α and empirically set to 10.
  • Gradient Quality: 1.5%: using Parametric Softplus gradients for the attacker improves robustness but reduces standard accuracy by 0.5% relative to ReLU.
  • Gradient Quality: 2.8%: using Parametric Softplus gradients for network updates improves robustness and raises accuracy by 0.6% without additional computation.
  • Gradient Quality: 3.9%: using better gradients for both attacker and optimizer improves robustness while increasing accuracy by 0.1% over ReLU.
  • Training Longer: Training for twice as long gains 2.6% accuracy but loses 1.8% robustness, unlike applying better gradients, which improves both.
  • Conclusion: Replacing ReLU with a smooth approximation in the backward pass substantially improves robustness without sacrificing accuracy or adding computation.

4. Smooth Adversarial Training

Smooth adversarial training replaces non-smooth activations with smooth alternatives in both forward and backward passes, improving gradient quality and adversarial robustness. Experiments show these changes strengthen robustness while largely preserving accuracy, including for ELU-based training and ReLU alternatives that avoid negative-input effects.

  • Method: SAT uses smooth activation functions exclusively in both forward and backward passes, while keeping other network components unchanged.This addresses the discrepancy that arises when ReLU is used forward and a smooth approximation only backward.
  • Method: Five smooth activation functions are considered as ReLU approximations, with their functions and derivatives visualized.The candidates include Softplus, SILU, GELU, ELU, and SmoothReLU.
  • Main results: 42.3% robustness and 69.7% standard accuracy are achieved by SILU-equipped ResNet-50, while all smooth activations substantially improve robustness over ReLU with nearly unchanged accuracy.Parametric Softplus provides a smaller robustness gain, from 33% to 38.7%.
  • Main results: 38.7% robustness, up from 36.9%, results when Parametric Softplus replaces ReLU in both forward and backward passes.Accuracy remains almost the same, demonstrating the importance of smoothing both passes.
  • Ruling Out the Effect From x < 0: 40.3% robustness and 69.4% accuracy are achieved by SmoothReLU, improving over ReLU by 7.3% and 0.6%, respectively.SmoothReLU modifies ReLU only for x ≥0, ruling out negative-input responses as the explanation for the gain.
  • Case Study: Stabilizing Adversarial Training with ELU using CELU: 41.4% robustness occurs for smooth ELU at α = 1.0, whereas non-smooth ELU at α = 2.0 reaches 33.2%, a 7.9% decrease.CELU reduces this sensitivity: its worst case at α = 2.0 is only 0.5% below the α = 1.0 result, compared with ELU’s 7.9% gap.

5. Exploring the Limits of Smooth Adversarial Training

SAT benefits from scaling depth, width, image resolution, and compound model factors, with larger architectures improving both accuracy and robustness. EfficientNet-L1 achieves strong performance while substantially narrowing the clean-accuracy gap associated with adversarial training.

  • Depth & width: Scaling depth and width consistently improves SAT performance over the ResNet-50 baseline.A 3× deeper ResNet-152 improves accuracy by 4.2% and robustness by 3.7%, while a 4× wider ResNeXt-50-32x8d improves accuracy by 3.9% and robustness by 2.8%.
  • Image resolution: Larger image resolutions also improve ResNet-50 performance in SAT, contrary to the expectation that they weaken adversarial robustness.The authors conjecture that stronger adversarial examples and increased representation capacity jointly produce this improvement.
  • Compound scaling: Compound scaling combines depth, width, and image resolution to produce stronger SAT models than focusing on individual scaling factors.The section presents compound scaling as the next step after confirming the value of each basic factor.
  • Comparison with standard adversarial training: In standard adversarial training, scaled models remain less robust than the ResNet-50 SAT baseline, while compound scaling narrows but does not eliminate the gap.ResNeXt-152-32x8d reaches 46.3% robustness versus 51.2% for SAT with compound scaling.
  • EfficientNet scaling: EfficientNet scaling improves SAT performance substantially, with EfficientNet-B7 reaching 57.0% robustness and 79.8% accuracy, and EfficientNet-L1 improving both further.EfficientNet-L1 adds 1.0% robustness and 0.7% accuracy over EfficientNet-B7.
  • Comparison with prior art: EfficientNet-L1 with SAT achieves 82.2% standard accuracy and 58.6% robustness, outperforming prior art by 9.5% and 11.6%, respectively.Its SAT accuracy is 1.9% below standard-training accuracy, compared with a 7.1% gap for ResNet-50.

6. Sanity Tests for SAT

The sanity tests examine whether SAT's reported robustness withstands stronger attacks and whether its loss landscape reflects the intended smoothness. Results show convergence under longer PGD attacks and a smoother landscape than the ReLU baseline.

  • Robustness vs. attack iterations: PGD robustness decreases as attack iterations increase, converging to 42.3% at PGD-200.Reported robustness falls from 48.7% at PGD-5 to 43.7% at PGD-10 and 42.7% at PGD-50.
  • Robustness vs. perturbation sizes ϵ: Increasing the perturbation budget from ϵ = 4 to 8 reduces robustness by more than 25%, while ϵ = 16 completely circumvents the model.
  • Landscape visualization: The loss-landscape visualization uses adversarial-perturbation directions on the x/y axes and cross-entropy loss on the z-axis.Compared with the ReLU baseline, SAT produces a much smoother landscape, including with other smooth activations and randomly selected images.
  • Evaluation conclusion: The authors conclude that the PGD-based robustness evaluation is properly conducted and release models for public evaluation.

7. SAT on CIFAR-10

On CIFAR-10, several smooth activations improve robustness while maintaining similar accuracy to ReLU, but ELU and SILU produce exceptions that expose sensitivity to training settings.

  • Activation comparisons: Softplus, GELU, and SmoothReLU maintain similar accuracy to ReLU while achieving stronger robustness.Robustness improvements range from 0.7% with GELU to 2.3% with Softplus.
  • Exceptions: ELU and SILU significantly hurt adversarial training on CIFAR-10 compared with ReLU.The inferior performance also appears on the training set, where SILU has 4.1% higher training error and ELU has 10.6% higher training error.
  • Open setting: The authors leave optimal ELU and SILU adversarial-training settings on CIFAR-10 for future work because the task is highly sensitive to parameter choices.

8. Conclusion

The paper proposes SAT, which replaces non-smooth activations with smooth approximations during adversarial training. Experiments report improved robustness without sacrificing accuracy or adding computation, including state-of-the-art ImageNet results with EfficientNet-L1.

  • SAT replaces non-smooth activation functions with smooth approximations during adversarial training.
  • SAT improves adversarial robustness without sacrificing accuracy or incurring additional computational cost.
  • EfficientNet-L1 with SAT achieves state-of-the-art ImageNet robustness, outperforming prior art by 9.5% for accuracy and 11.6% for robustness.
Loading 2006.14536v2…