PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks

Hanxun Huang, Yisen Wang, Sarah Monazam Erfani, Quanquan Gu, James Bailey, Xingjun Ma

arXiv:2110.03825v5cs.LGcs.CVstat.ML

TL;DR

Adversarial training can produce robust DNNs, but the architectural configurations that support robustness remain unclear. The paper investigates width and depth through controlled experiments and theoretical analysis, finding that more capacity is not always better and that its allocation matters. It concludes that a configuration may exploit robustness potential under a fixed architecture type and parameter budget.

  • Problem

    Although adversarial training has shown promising results, which architectural configurations make adversarially trained DNNs more robust remains unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper investigates width and depth using controlled experiments based on adversarially trained WRN-34-10 and theoretical analysis.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    More parameters do not necessarily improve robustness; reducing capacity in deeper layers can help, and a robust configuration may exist under a fixed architecture type and parameter budget.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Capacity allocation, especially at deeper layers, is an architectural consideration for adversarial robustness within the studied settings.

    Source: §7 · p. 10

Abstract

from arXiv · show

Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. A range of defense methods have been proposed to train adversarially robust DNNs, among which adversarial training has demonstrated promising results. However, despite preliminary understandings developed for adversarial training, it is still not clear, from the architectural perspective, what configurations can lead to more robust DNNs. In this paper, we address this gap via a comprehensive investigation on the impact of network width and depth on the robustness of adversarially trained DNNs. Specifically, we make the following key observations: 1) more parameters (higher model capacity) does not necessarily help adversarial robustness; 2) reducing capacity at the last stage (the last group of blocks) of the network can actually improve adversarial robustness; and 3) under the same parameter budget, there exists an optimal architectural configuration for adversarial robustness. We also provide a theoretical analysis explaning why such network configuration can help robustness. These architectural insights can help design adversarially robust DNNs. Code is available at \url{https://github.com/HanxunH/RobustWRN}.

1 Introduction

Adversarial training has shown promise, but which architectural configurations make adversarially trained networks more robust remains unclear. The paper investigates width and depth and finds that capacity’s effect depends on where and how it is allocated.

Motivation

  • Adversarial examples can be crafted by slightly perturbing inputs, creating deployment concerns in security-critical applications.The introduction names autonomous driving and medical diagnosis as examples.Source: §1 · p. 1
  • Prior adversarial-training research has studied learning behavior and training factors, but the architectural perspective remains unclear.Examples include robust or compact feature learning and the need for more data or higher-capacity models.Source: §1 · p. 1

Investigation

  • The study uses a finely controlled grid search over width and depth configurations based on adversarially trained WRN-34-10.WRN-34-10 is described as an extensively tested architecture in the defense literature.Source: §1 · p. 2

Architecture

  • WRN-34-10 has three sequential stages, each configured by depth and width, which control the scale of learnable parameters.Each standard stage contains five residual blocks, and each block has two convolutional layers.Source: §1 · p. 2

Key observations

  • Reducing capacity at the final WRN stage can improve robustness, and under a fixed parameter budget an optimal configuration may exist.The paper reports that width reduction at Stage-3 is slightly more effective than depth reduction, and that its configuration rule also applies to VGGs, DenseNets, and DARTS-found networks.Source: §1 · p. 2
  • Increasing width or depth does not necessarily improve adversarial robustness, despite prior beliefs that higher capacity can help.The paper attributes a trade-off to wider and deeper models increasing Lipschitzness.Source: §1 · p. 2

2 Related Work

Prior work develops adversarial training and studies factors associated with its robustness, while neural architecture search explores other routes to robust designs. This paper focuses on controlled width and depth configurations rather than blind architectural search.

2.1 Adversarial Training

  • Standard adversarial training uses an inner maximization to generate adversarial examples and an outer minimization to train the model on them.The inner process uses PGD, with projection keeping perturbations within the ε-ball around the input.Source: §2.1 · p. 2
  • Adversarial training has many proposed variants, and robust overfitting has been reported as a concern that smoothing techniques can mitigate.The variants include TRADES, DART, FAT, MART, self-training, and other training or loss modifications.Source: §2.1 · pp. 2–3

2.2 Understanding Adversarially Trained DNNs

  • Studies of adversarially trained networks report effects involving robust features, auxiliary data, weight decay, architecture, and perturbation stability.Prior work also associates wider networks with greater perturbation instability and WRNs with about 3%-5% more robustness than RNs.Source: §2.2 · p. 3
  • Neural architecture search studies robust cell connections and search strategies, whereas this work explores width and depth configurations through fine-grained controlled experiments.The passage notes that NAS can be time-consuming with adversarial training and says earlier NAS work did not investigate depth/width configurations.Source: §2.2 · p. 3

3 Wider and Deeper Models Increase Lipschitz Upper Bound

The paper analyzes how width and depth affect a neural network’s Lipschitz upper bound, which is linked to output stability under input perturbations. Its analysis predicts that wider and deeper models can have lower adversarial robustness.

Capacity trade-off

  • The theoretical analysis examines a trade-off between network capacity and the Lipschitz upper bound, against the background that adversarial training can require larger-capacity models.The paper says it will empirically examine this trade-off and its relation to robustness.Source: §3 · p. 3

Lipschitz upper bound

  • The Lipschitz constant measures the maximum rate of output change with input change and is closely related to adversarial robustness.The paper defines it through a bound on the output difference by the input difference multiplied by L.Source: §3 · p. 3
  • The Lipschitz upper bound increases with convolutional layer input and output dimensions and grows exponentially with network depth.The analysis therefore suggests wider and deeper models can exhibit larger output changes under input perturbations and lower adversarial robustness.Source: §3 · p. 4

Lipschitz regularization

  • Although some methods regularize the Lipschitz constant, a follow-up work points out experimental and theoretical limitations; the constant remains used to analyze stability and robustness.Methods named include Parseval tight frames, singular-value constraints, and Lipschitz-margin training.Source: §3 · p. 4

4 Exploring Adversarially Robust Architectures

A controlled WRN-34-10 study finds that adversarial robustness depends on where capacity is allocated: parameter count alone is insufficient, and reducing the last stage can help.

4 Exploring Adversarially Robust Architectures

  • The study varies stage-wise depth and width in WRN-34-10, training the configurations with SAT on CIFAR-10 and evaluating robustness with PGD20.The base network has three stages; each is configured by its residual-block depth and filter-width factor.Source: §4 · pp. 4–5

4.1 Exploring Different Depths

  • At ∼20M parameters, depth configurations differ by ∼6% in robustness, and deeper Stage-1/2 blocks help while deeper Stage-3 blocks hurt.The top five configurations use Stage-3 depth 1 or 3; d9-7-1 uses half the parameters of the d5-5-5 baseline.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 Exploring Different Widths

  • 17.05M parameters for w10-10-4 yields ∼54% robustness, similar to d9-7-1's 22.19M; reducing width in Stage-3 can help, unlike reducing it in shallow stages.Reducing Stage-1/2 width from 4 to 2 changes robustness little, whereas the same reduction at Stage-3 hurts robustness.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 Exploring Depth-Width Combinations

  • Removing Stage-3 entirely reduces robustness by 2% versus WRN-34-10; d5-5-1 with w10-10-2 retains ∼52% PGD20 robustness with 70% fewer parameters.None of the 25 tested depth-width combinations surpasses w10-10-4; some require more computation, including one using twice the FLOPS of WRN-34-10.Source: §4.3 · p. 6

4.4 Scaling with the Discovered Configuration

  • γ=2.0 (w20-20-8) achieves the best robustness among the tested scalings of w10-10-4; scaling down to γ=0.5 or 0.25 reduces robustness.Scaling up improves robustness, but the improvement becomes less significant above γ=1.5.Source: §4.4 · p. 7, Table 1 · p. 7

4.5 Empirical Understanding

  • Reducing depth or width lowers empirical Lipschitzness and improves perturbation stability; WRN-34-R's Stage-3 reduction mitigates higher shallow-stage Lipschitzness.Stage-1 width reduction lowers Stage-1 block Lipschitzness but not overall Lipschitzness; the theoretical analysis considers adjacent layers, not non-adjacent interactions.Source: Figure 4 · p. 7, §4.5 · p. 8

5 Adversarial Robustness Evaluation

The evaluation applies the discovered width configuration to several architectures and compares them with standard models across adversarial-training methods and white-box attacks. Reconfigured networks generally gain robustness, with some CIFAR-100 exceptions.

5 Adversarial Robustness Evaluation

  • The evaluation applies the width rule to VGG, DenseNet, NAS-discovered DARTS, and WRN-34-R on CIFAR white-box attacks.Additional CIFAR-10 black-box and ImageNet results using FastAT are in Appendix D.Source: §5 · p. 8
  • Networks are evaluated with five white-box attacks on CIFAR-10 and CIFAR-100 using a maximum perturbation of ϵ = 8/255.The comparison includes clean accuracy and adversarial robustness.Source: §5 · p. 9
  • 1–3% robustness gains are reported for reconfigured networks across architectures and adversarial-training methods, with a slight CW∞ decrease on CIFAR-100 for SAT and MART.The comparison is reported in Table 2.Source: §5 · p. 9, Table 2 · p. 9

6 Relation to Existing Understandings

The paper relates its width analysis to prior Lipschitzness findings and explains how weight decay may lower the Lipschitz upper bound. It presents architectural reconfiguration as an alternative to stronger regularization for mitigating vulnerability.

6 Relation to Existing Understandings

  • Unlike prior NTK analysis assuming equal layer widths, this work analyzes width and depth at individual layers and proposes architectural reconfiguration as an alternative to stronger regularization.The cited prior work uses stronger regularization to mitigate vulnerability associated with increased Lipschitzness.Source: §6 · p. 9
  • The paper reports that wider networks tend to increase Lipschitzness, consistent with the finding in.The cited analysis assumes all layers share the same width.Source: §6 · p. 9
  • Under normally distributed weights, weight decay reduces weight magnitudes and variance, lowering the Lipschitz upper bound and improving robustness.The paper relates this explanation to Theorems 1 and 2.Source: §6 · p. 9

7 Conclusion

The conclusion reports that more parameters do not necessarily improve adversarial robustness and that reducing capacity in deeper layers can help. It also reports that reconfiguration can improve robustness across architectures and adversarial-training methods.

7 Conclusion

  • Reducing capacity up to a limit in deeper layers can improve adversarial robustness, while more parameters do not necessarily produce more robust models.The paper reports this finding for reductions in either depth or width.Source: §7 · p. 10
  • Under the same architecture type and parameter budget, an architectural configuration may exploit more of a network’s robustness potential.The paper also reports that depth and width offer different flexibility for capacity reduction and robustness improvement.Source: §7 · p. 10
  • The width reduction and scaling rule is reported to generalize across adversarial-training methods and manually designed or NAS-discovered networks.The paper states these findings can inform the design of more robust DNNs.Source: §7 · p. 10

Border Impacts

The paper reports that adversarial robustness can improve even when capacity is reduced at deep layers. Such reductions can also help save the training cost of adversarial training.

Border Impacts

  • Reducing capacity at deep layers can make models more robust while helping save adversarial-training cost.The paper describes adversarial training as extremely time-consuming.Source: Border Impacts · p. 10
  • The paper presents architecture as a way to improve robustness, despite adversarial-training performance still needing improvement.Source: Border Impacts · p. 10
  • The authors plan to open-source the discovered architectural configurations for future research on robust architectures.Source: Border Impacts · p. 10

A Proof for Theorem 1 and 2

The proof bounds Lipschitz constants under Gaussian-weight assumptions, extending the analysis from dense networks to convolutional and residual networks. The accompanying empirical figure compares first-convolution weight distributions across three blocks.

A.1 Theoretical Proof

  • The proof assumes independent Gaussian weight entries and uses a random-matrix bound on maximum singular values to bound the network’s Lipschitz constant.The bound is stated for DNNs with 1-Lipschitz activations, including ReLU.Source: §A.1 · p. 15
  • Convolution is represented as matrix multiplication with a doubly block-circulant matrix, yielding a Lipschitz upper bound based on layer dimensions.The matrix encodes kernel weights and zeros; its dimensions depend on spatial size and kernel size.Source: §A.1 · p. 15
  • The proof extends the convolutional bound to residual blocks and then to a ResNet composed of residual blocks.The supplied passages describe these extensions but do not provide their bound values.Source: §A.1 · p. 15

A.2 Empirical Verification on Gaussian Distribution Weights.

  • Figure 6 plots kernel-density estimates of first-convolution weight matrices from blocks 1, 5, and 10 of adversarially trained WRN-34-10.The caption specifies the distributions being compared but states no empirical outcome.Source: Figure 6 · p. 16

B.1 Training method

The experiments train and configure adversarially trained networks across several methods and architectures. Reducing capacity in shallower stages decreases clean accuracy and perturbation stability, unlike the reported benefit of reducing the last stage.

B.1 Training method

  • SAT, TRADES, MART, and RST train the networks on CIFAR-10 and CIFAR-100 using PGD10 with maximum perturbation ϵ = 8/255.SAT, TRADES, and MART use 100 epochs; RST uses 400 epochs and 500K additional data.Source: §B.1 · p. 16

B.2 Network setup.

  • The network setup varies stage depth and width; the standard WRN-34-10 uses depth 5 and width 10 in each of its three stages.WRN-34-12 uses width 12 throughout, while WRN-34-R uses widths 20, 20, and 8.Source: §B.2 · p. 16, Table 3 · p. 16
  • The optimized VGG-11, DenseNet-121, and DARTS configurations reduce the width of the last stage to 40% of its original width.The passage also describes scaling WRN-34-10 to WRN-34-12 for a parameter-count comparison with WRN-34-R.Source: §B.2 · pp. 16–17

C Empirical Understanding on Reduction in Stages 1 and 2.

  • Reducing width or depth in Stages 1 or 2 decreases both clean accuracy and perturbation stability, reducing overall adversarial robustness.The comparison supports reducing capacity in the last stage, rather than shallower stages, to mitigate the capacity–Lipschitzness trade-off.Source: Figure 7 · p. 17

D.1 Black-box Robustness

The paper evaluates the reconfigured networks in black-box CIFAR-10 and ImageNet settings, and discusses a CIFAR-100 exception under CW evaluation. The reported black-box results show consistent robustness improvement, while the supplied ImageNet passage gives the evaluation setup but not its outcome.

D.1 Black-box Robustness

  • The reconfigured networks consistently improve black-box robustness on CIFAR-10 across the adversarial-training methods tested.Transferred FGSM, PGD20, and CW∞ examples are crafted with a naturally trained ResNet-50 surrogate; Table 4 reports the results.Source: §D.1 · p. 17, Table 4 · p. 18

D.2 ImageNet

  • For ImageNet, ResNet-50-R reduces the last-stage width to 40% and is scaled to match ResNet-50’s parameter count.The models use FastAT with ϵ = 4/255 and a 10-step PGD adversary; Table 5 reports robustness results.Source: §D.2 · p. 18, Table 5 · p. 18

E.1 Performance of CW attack on CIFAR-100

  • On CIFAR-100, the discovered configurations are less robust than the baseline under the CW∞ evaluation described in Table 2.The authors suggest the weaker CW∞ variant may contribute and describe AutoAttack as stronger and more reliable; they also report consistent improvements across multiple datasets, architectures, and training methods in Table 2.Source: §E.1 · p. 18

E.2 Auto-Attack leaderboards

The section situates AutoAttack improvements against the scale of gains discussed for adversarial defenses and reports an additional gain from EMA. The authors state that EMA further improves the discovered WRN configuration’s AutoAttack robustness.

E.2 Auto-Attack leaderboards

  • Table 6 reports AutoAttack leaderboard results for models using additional data as in RST.Source: Table 6 · p. 18
  • The paper notes that 1%–2% improvement on AA can be significant, and cites a similarly sized WRN-34-R achieving a 1% improvement over WRN-34-15 with stronger regularization.The passage attributes the comparison to.Source: §E.2 · p. 19
  • EMA adds 1% robustness on AA to the discovered WRN configuration, with further gains reported across adversarial-training methods.The results use the original settings and hyperparameters of the corresponding papers.Source: §E.2 · p. 19

Related papers

Loading 2110.03825v5…