PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizable Adversarial Training via Spectral Normalization

Farzan Farnia, Jesse M. Zhang, David Tse

arXiv:1811.07457v1cs.LGstat.ML

TL;DR

Adversarial training can leave DNNs with substantially worse generalization than standard training, limiting robust test performance. The paper extends margin-based analysis to adversarial settings and proposes spectral normalization, which generally improves test performance across datasets, architectures, and attack schemes.

  • Problem

    Adversarial training can generalize poorly, creating a larger training–test gap than standard empirical risk minimization and limiting adversarial test performance.

  • Method

    The paper combines adversarial margin-loss analysis and PAC-Bayes bounds with spectral normalization of DNN weight matrices as a regularizer for adversarial training.

  • Results

    SN generally improves adversarial test performance across MNIST, CIFAR-10, SVHN, multiple architectures, and FGM, PGM, and WRM training schemes.

  • Takeaways & Limitations

    Spectral normalization is supported as an effective regularization approach for improving the test and generalization performance of adversarially trained DNNs.

  • Takeaways & Limitations

    The theoretical assumptions include smooth activations with 1-Lipschitz derivatives, excluding ReLU from the stated analysis.

Abstract

from arXiv · show

Deep neural networks (DNNs) have set benchmarks on a wide array of supervised learning tasks. Trained DNNs, however, often lack robustness to minor adversarial perturbations to the input, which undermines their true practicality. Recent works have increased the robustness of DNNs by fitting networks using adversarially-perturbed training samples, but the improved performance can still be far below the performance seen in non-adversarial settings. A significant portion of this gap can be attributed to the decrease in generalization performance due to adversarial training. In this work, we extend the notion of margin loss to adversarial settings and bound the generalization error for DNNs trained under several well-known gradient-based attack schemes, motivating an effective regularization scheme based on spectral normalization of the DNN's weight matrices. We also provide a computationally-efficient method for normalizing the spectral norm of convolutional layers with arbitrary stride and padding schemes in deep convolutional networks. We evaluate the power of spectral normalization extensively on combinations of datasets, network architectures, and adversarial training schemes. The code is available at https://github.com/jessemzhang/dl_spectral_normalization.

1 Introduction

The paper addresses poor generalization in adversarially trained DNNs by proposing spectral normalization (SN) as a regularizer, supported by adversarial margin bounds and broad experiments. Across evaluated settings, SN generally improves test performance, including gains for FGM, PGM, and WRM on CIFAR-10.

  • Motivation and approach: The paper proposes SN as a computationally efficient and statistically powerful regularization scheme for adversarial training.The motivation is the poor generalization observed in standard adversarial training.
  • Theoretical analysis: Theoretical analysis extends margin-based generalization analysis to adversarial settings and derives bounds for FGM, PGM, and WRM.The analysis uses a PAC-Bayes framework and shows the adversarial generalization component vanishes when all layers have sufficiently small spectral norms.
  • Empirical evaluation: SN improved adversarial test accuracy by 9%, 11%, and 4% for FGM, PGM, and WRM, respectively, on AlexNet trained on CIFAR-10.For FGM and PGM, the perturbations had ℓ2 magnitude 2.44.
  • Empirical evaluation: Experiments across MNIST, CIFAR-10, SVHN, and multiple DNN architectures generally found better test performance after applying SN.The evaluated architectures included AlexNet, Inception, and ResNet.
  • Implementation: The work develops an efficient method for normalizing convolutional-layer spectral norms in deep convolutional networks.The method is intended for convolutional layers with arbitrary stride and padding schemes.

2 Preliminaries

This section defines supervised learning and DNN notation, then introduces adversarial attacks, adversarial training, and the margin-based evaluation framework used later.

  • 2.1 Supervised learning, Deep neural networks, Generalization error: A DNN maps inputs through layered weight matrices and activations, while standard margin analysis compares true and empirical margin risks.The network class uses spectral and Frobenius norms of its layer matrices in subsequent bounds.
  • 2.2 Adversarial attacks, Adversarial training: Adversarial training minimizes loss on perturbed samples to improve robustness on future test samples, but the underlying optimization is generally non-convex and intractable.The paper therefore analyzes gradient-based approximations including FGM, PGM, and WRM.
  • 2.2 Adversarial attacks, Adversarial training: FGM linearizes the loss to construct a perturbation, whereas PGM iteratively applies projected gradient updates within a bounded perturbation set.For the ℓ∞ norm, FGM recovers the fast gradient sign method.
  • 2.2 Adversarial attacks, Adversarial training: WRM replaces the norm constraint with a norm-squared Lagrangian penalty and is efficiently solvable under a Lipschitz-gradient assumption.The stated assumption makes the resulting optimization convex for the Euclidean norm.
  • 2.3 Adversarial generalization error: The adversarial margin loss evaluates classification margins on attack-perturbed samples, extending ordinary margin-loss analysis to adversarial settings.The empirical adversarial margin loss is defined over the training samples under FGM, PGM, or WRM attacks.

3 Margin-based adversarial Generalization bounds

The paper extends margin-based generalization analysis to adversarial settings using PAC-Bayes bounds, relating attack-specific error to network spectral norms and attack assumptions.

  • 3 Margin-based adversarial Generalization bounds: Spectral-norm-based bounds motivate regularizing trained DNN weight matrices to limit capacity and improve adversarial generalization.The framework covers adversarial margin losses for FGM, PGM, and WRM attacks.
  • 3 Margin-based adversarial Generalization bounds: The PAC-Bayes analysis extends margin-based bounds from ReLU networks to 1-Lipschitz activations and derives separate adversarial bounds for FGM, PGM, and WRM.The bounds assume norm-bounded inputs and specified smoothness or Lipschitz conditions on activations and losses.
  • 3 Margin-based adversarial Generalization bounds: FGM bounds require the input-loss gradient to remain at least κ over an ε-neighborhood of the data support.The paper notes that this change-rate assumption is checked empirically on standard image-recognition tasks.
  • 3 Margin-based adversarial Generalization bounds: PGM bounds depend on the Lipschitz constant of the input-loss gradient and can exceed the FGM bound by factor 1/(1 − (2α/κ)lip(∇ℓ○f_w)) under the stated condition.The condition is lip(∇ℓ○f_w)/κ < 1/(2α).
  • 3 Margin-based adversarial Generalization bounds: WRM requires lip(∇ℓ○f_w) < λ for convergence to the global solution, and its bound scales with 1/(λ − lip(∇ℓ○f_w)) times the FGM bound.The theorem assumes a stronger upper bound λ(1−τ) for some 0 < τ < 1.

4 Spectral normalization of convolutional layers

The paper develops an efficient way to compute and normalize convolutional-layer spectral norms while accounting for arbitrary stride and padding. It uses convolution transpose operations and modified power iteration without explicitly constructing the convolution matrix.

  • The method computes convolutional spectral norms efficiently for arbitrary stride and padding schemes.This extends approaches limited to stride 1 and zero-padding.
  • Convolutional layers are treated as linear operations whose spectral norm controls the network’s Lipschitz constant.The normalization target is to keep each linear operation’s spectral norm below a prescribed β.
  • Modified power iteration uses convolution and convolution-transpose operations to approximate the largest singular value without explicitly reconstructing the weight matrix.The procedure alternates normalized forward and transpose operations before estimating σ(W).
  • The resulting normalized convolutional weights are reported to produce faster training for supervised learning tasks.

5 Numerical Experiments

Across datasets, architectures, and adversarial-training schemes, spectral normalization generally improves test accuracy, generalization, and robustness. Experiments also show tighter margin-based bounds and reduced capacity-related normalization factors.

  • 5.1 Validation of spectral normalization implementation and bounds: Spectral normalization constrains network norm gain as β decreases, with β = 1 preventing gain above 1 in the reported experiment.The results also indicate that an alternative convolutional normalization method fails to properly control spectral norms and has worse generalization performance.
  • 5.1 Validation of spectral normalization implementation and bounds: Spectral normalization produces distributed training-sample gradient norms consistent with the minimum-gradient-norm assumption used in two theoretical bounds.
  • 5.2 Spectral normalization improves generalization and adversarial robustness: Spectral normalization yields smaller capacity norm Φ values and tighter margin-based generalization bounds, by factors of 10^2 for ERM and 10^5 for FGM and PGM.
  • 5.2 Spectral normalization improves generalization and adversarial robustness: Combining spectral normalization with adversarial training significantly improves robustness beyond spectral normalization alone.Adversarial training remains more robust than spectral normalization by itself, but the combination further improves trained-network robustness.
  • 5.3 Other datasets and architectures: Validation accuracy generally improves after spectral normalization across 42 evaluated dataset, architecture, and training-scheme combinations.Table 1 reports pre- and post-normalization test accuracies, while Figure 6 summarizes the improvements.

6 Related Works

Prior work studies adversarial robustness, generalization, and capacity from several theoretical and empirical perspectives. These passages situate the paper among robustness guarantees, adversarial-training generalization, and deep-learning generalization analyses.

  • Prior studies analyze adversarial robustness for nearest-neighbor methods, data-generating manifolds, robust learning complexity, regularized SVMs, and fixed classifiers.
  • The paper’s empirical figures compare robustness and test-accuracy changes across adversarial attacks, datasets, and architectures, with Appendix Table 1 providing further details.
  • Distributionally robust training provides optimization and generalization guarantees for Wasserstein risk minimization, but not directly for 0-1 or margin loss.
  • Existing deep-learning generalization analyses use margin bounds, VC dimension, norm-based capacity, flatness, compression, and PAC-Bayes methods.

Appendix A Further experimental results

Further experiments examine spectral normalization across architectures and report train/test accuracy, validation behavior, and runtime effects. The supplied passages identify the evaluated settings but provide limited numerical detail for these figures and tables.

  • Further experimental results: Figures 7–9 examine adversarial-training performance with and without spectral normalization across Inception, ResNet, AlexNet, CIFAR10, and ELU activations.
  • Further experimental results: Table 1 reports train and test accuracies before and after spectral normalization.
  • Further experimental results: The appendix reports that β = ∞, corresponding to no spectral normalization, achieved the highest validation accuracy in one examined setting.
  • Further experimental results: Table 2 reports runtime increases after introducing spectral normalization across datasets, architectures, and training schemes.

A.1 Comparison of proposed method to [26]’s method

The proposed convolutional spectral-normalization method is compared with [26]’s kernel-based approach on CIFAR10 AlexNet experiments. The proposed method generalizes better in the reported comparison but is less computationally efficient in principle.

  • Comparison of proposed method to [26]’s method: [26]’s approach normalizes convolution kernels rather than the overall convolution operator, failing to account for repeated amplification of a single pixel.
  • Comparison of proposed method to [26]’s method: The proposed method is less computationally efficient because each power-iteration step requires a convolution operation instead of a division operation.The TensorFlow implementation did not show a significant efficiency disadvantage in the reported comparison.
  • Comparison of proposed method to [26]’s method: 0.60 validation accuracy versus 0.55: the proposed method outperforms [26]’s spectral normalization for AlexNet trained on CIFAR10 with PGM.The corresponding train accuracies are 0.92 for the proposed method and 1.00 for [26]’s method.
  • Comparison of proposed method to [26]’s method: Figure 11 compares the proposed method with batch normalization, weight decay, and dropout under AlexNet CIFAR10 PGM training.The supplied passage specifies dropout rate 0.8 and weight decay 5e-4, and identifies the no-regularization and SN results in the leftmost plot.

A.2 Comparison of proposed method to weight decay, dropout, and batch normalization

For fully connected layers, the method estimates spectral norms with power iteration and reuses the previous singular-vector estimate across training steps. It enforces an upper-bound constraint rather than an exact spectral norm.

  • Comparison of proposed method to weight decay, dropout, and batch normalization: Power iteration approximates the spectral norm of each fully connected weight matrix using estimated left and right singular vectors.
  • Comparison of proposed method to weight decay, dropout, and batch normalization: The implementation reuses the previous singular-vector estimate and performs one iteration per training step because SGD makes only small weight updates.
  • Comparison of proposed method to weight decay, dropout, and batch normalization: Unlike [26]’s method, the approach enforces σ(W) ≤ β rather than σ(W) = β.The looser constraint is reported to result in faster training for supervised learning tasks.

C.1 Proof of Theorem 1

The proof combines PAC-Bayes bounds with perturbation control and covering arguments to derive a spectral-normalization-based generalization bound for neural networks.

  • C.1 Proof of Theorem 1: PAC-Bayes analysis bounds generalization using Gaussian weight perturbations scaled by the spectral norms of the perturbed network’s layers.The perturbation distribution is chosen with layer-specific standard deviations proportional to each perturbed layer’s spectral norm.
  • C.1 Proof of Theorem 1: A perturbation lemma controls network-output changes when each layer’s weight perturbation has bounded spectral norm.This establishes the perturbation condition needed by the PAC-Bayes lemma.
  • C.1 Proof of Theorem 1: A union bound over layers and discretization of feasible spectral norms produce a finite cover whose logarithm grows as dlog(dlog M).The cover is formed by approximating each layer’s spectral norm multiplicatively.
  • C.1 Proof of Theorem 1: The proof completes by transferring the perturbation and covering bounds to networks whose layer spectral norms are sufficiently close to those of the reference network.The argument combines the preceding perturbation, KL-divergence, and covering estimates.

C.2 Proof of Theorem 2

The proof extends perturbation bounds from FGM to PGM and WRM attacks, then applies PAC-Bayes analysis while controlling gradient smoothness through products of layer spectral norms.

  • C.2 Proof of Theorem 2: FGM perturbation analysis combines gradient stability with a lower bound on loss gradients to establish the attack-specific perturbation condition.The proof uses Lipschitz and smoothness assumptions on the network activation and loss.
  • C.2 Proof of Theorem 2: PAC-Bayes bounds follow by choosing Gaussian layer perturbations scaled by spectral norms, applying a union bound, and covering feasible norm values.The resulting cover controls networks whose layer spectral norms are multiplicatively close to the reference values.
  • C.2 Proof of Theorem 2: PGM bounds extend the FGM argument inductively across attack iterations, with the perturbation analysis depending on iteration count and step size.The base case is FGM, and the induction establishes the result for every iteration count r ≥1.
  • C.2 Proof of Theorem 2: The network’s input-gradient Lipschitz constant is bounded by the product of layer spectral norms, enabling gradient-smoothness control in the PGM and WRM analyses.This follows from repeated chain-rule arguments under the stated activation and loss assumptions.
  • C.2 Proof of Theorem 2: WRM analysis requires lip(∇ℓ○fw) < λ for global convergence, and its generalization bound is multiplied by 1/(λ−lip(∇ℓ○fw)) relative to FGM.The WRM proof uses corresponding Gaussian perturbations, union bounds, and spectral-norm coverings.
Loading 1811.07457v1…