PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing Neural Networks via Adversarial Model Perturbation

Yaowei Zheng, Richong Zhang, Yongyi Mao

arXiv:2010.04925v4cs.LGcs.AIstat.ML

TL;DR

Deep neural networks can overfit under empirical risk minimization, while effective regularization methods are sought to improve generalization. AMP minimizes empirical risk under worst-case norm-bounded parameter perturbations, with theoretical analysis indicating a preference for flatter minima. Experiments report the best performance among compared regularization schemes across benchmark datasets and modern architectures.

  • Problem

    Effective regularization is needed to alleviate overfitting and improve generalization, while many existing schemes lack strong principles or theoretical justifications.

    Source: §1 · p. 1

  • Method

    AMP replaces direct empirical-risk minimization with minimization of the highest empirical risk within a norm-bounded parameter neighborhood.

    Source: §1 · p. 1

  • Results

    AMP achieves the best performance among compared regularization schemes on benchmark datasets and modern neural network architectures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents AMP as a theoretically justified regularizer that favors flatter minima, with its theoretical analysis and image-classification performance confirmed through experiments.

    Source: §1 · p. 2, §6 · p. 8

    Each SGD iteration requires multiple gradient computations to produce adversarial perturbations, and the theoretical assumption is only an approximation that is argued to be accurate for small ϵ.

    Source: §4.1 · p. 4, §5.5 · p. 8

Abstract

from arXiv · show

Effective regularization techniques are highly desired in deep learning for alleviating overfitting and improving generalization. This work proposes a new regularization scheme, based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. This scheme is referred to as adversarial model perturbation (AMP), where instead of directly minimizing the empirical risk, an alternative "AMP loss" is minimized via SGD. Specifically, the AMP loss is obtained from the empirical risk by applying the "worst" norm-bounded perturbation on each point in the parameter space. Comparing with most existing regularization schemes, AMP has strong theoretical justifications, in that minimizing the AMP loss can be shown theoretically to favour flat local minima of the empirical risk. Extensive experiments on various modern deep architectures establish AMP as a new state of the art among regularization schemes. Our code is available at https://github.com/hiyouga/AMP-Regularizer.

1. Introduction

AMP addresses the need for effective regularization by minimizing a worst-case, norm-bounded perturbation of empirical risk, with the aim of favoring flat minima. The paper reports theoretical support for this aim and state-of-the-art performance among compared regularization schemes.

1. Introduction

  • Existing regularization techniques can lack strong principles or theoretical justifications, motivating more principled and powerful schemes.The paper notes MixUp and Flooding as examples of successful techniques that usually fall short of strong theoretical justifications.Source: §1 · p. 1
  • AMP minimizes the highest empirical risk among parameter perturbations whose norm is no greater than ϵ, rather than minimizing empirical risk directly.The paper describes this operation as max-pooling over a radius-ϵ neighborhood in parameter space.Source: §1 · pp. 1–2
  • Theoretical analysis shows AMP can favor flatter local minima, and the paper reports experimental evidence that AMP selects flatter minima than ERM.The analysis treats the narrowest width of a local minimum as its notion of flatness.Source: §1 · p. 2
  • AMP achieves the best performance among compared regularization schemes on benchmark datasets and modern neural network architectures.Experiments cover SVHN, CIFAR-10, and CIFAR-100.Source: §1 · p. 2

2. Related Work

Related work connects flat minima with generalization and surveys regularization approaches, including their proposed mechanisms and limitations.

2.1. Flat Minima and Generalization

  • Flat minima are commonly understood to tend to generalize better, though prior work also argues that sharpness can depend on parameterization and the chosen notion of flatness.The cited literature offers explanations involving model complexity, Gibbs free energy, loss-landscape visualization, and PAC-Bayesian analysis.Source: §2.1 · p. 2

2.2. Regularization

  • Prior regularizers include label smoothing, stochastic branch methods, input masking, sample mixing, flooding, adversarial training, and model-state ensembling.The passage notes that excessive adversarial-robustness training may hurt generalization and identifies a concurrent work proposing a similar scheme.Source: §2.2 · p. 2

3. AMP: Adversarial Model Perturbation

AMP replaces direct empirical-risk minimization with a loss based on parameter perturbations, aiming to steer training toward flatter minima or low-norm solutions. Its mini-batch algorithm alternates between increasing loss through bounded perturbations and updating model parameters by SGD.

3.1. From Empirical Risk to AMP Loss

  • AMP is motivated by ERM's vulnerability to overfitting and aims to improve generalization by steering training toward flatter minima or low-norm solutions.The formulation uses an L2 norm ball, while the authors expect that other norms may also be possible.Source: §3.1 · p. 3
  • AMP defines an alternative loss from empirical risk using a norm-bounded parameter neighborhood, rather than directly minimizing empirical risk.The passage identifies the perturbation radius ϵ as a small positive hyperparameter.Source: §3.1 · p. 3

3.2. Training Algorithm

  • For each mini-batch, AMP iteratively increases loss through a bounded parameter perturbation, then updates model parameters with mini-batched SGD; testing uses the learned parameters without perturbation.The batch-level objective approximates minimization of the expected AMP loss.Source: §3.2 · p. 3

4. Theoretical Justifications of AMP

Under a locally Gaussian approximation, minimizing AMP loss can favor flatter empirical-risk minima, with the narrowest-direction curvature setting a worst-case measure of flatness. AMP also admits a gradient-norm-penalty interpretation and, in input space, seeks low average loss under worst-case bounded perturbations.

4.1. AMP Finds Flatter Local Minima

  • The analysis assumes empirical-risk minima are locally approximated by inverted Gaussian surfaces, an approximation described as arguably accurate for small ε.The AMP-loss minimum is attained at the center of the surface, and its value reflects the narrowest-direction curvature.Source: §4.1 · p. 4
  • AMP loss can prefer a flatter local minimum even when that minimum has higher empirical risk, because it accounts for local curvature.The smallest eigenvalue of the curvature matrix measures width along the narrowest principal direction; a larger value indicates a flatter minimum.Source: §4.1 · pp. 4–5
  • The AMP operational region shrinks as the perturbation radius decreases, while a large radius can select high-risk minima and risk underfitting.The paper describes a sweet spot for ε that trades off the flatness and depth of selected minima.Source: §4.1 · p. 5

4.2. AMP Regularizes Gradient Norm

  • For one inner update per batch and sufficiently small inner learning rate, AMP's update is equivalent to gradient descent on a loss with a penalty on the batch empirical-risk gradient norm.This interpretation favors minima with both low loss and small nearby gradients, which the paper associates with flatter minima.Source: §4.2 · p. 5

4.3. Perspectives from the Input Space

  • In input space, AMP minimizes average training loss even under worst-case ε-bounded perturbations, which may yield smoother boundaries that keep training examples farther away.The paper illustrates this perspective with a spiral-dataset comparison of AMP and ERM boundaries.Source: §4.3 · p. 6, Figure 6 · p. 6
  • Although AMP resembles adversarial training when viewed in input space, the paper distinguishes their aims: defending against overfitting versus adversarial attacks.The paper characterizes adversarial training as defending against attacks and AMP as defending against overfitting.Source: §4.3 · p. 6

5. Experiment

The experiments evaluate AMP on benchmark image-classification datasets against regularizers and ERM, and examine augmentation, calibration, perturbation magnitude, and computational cost. Results indicate strong performance across settings, with a noted exception for CIFAR-10 using VGG16.

5.1. Comparison of the Generalization Ability

  • Across reported settings, AMP outperforms baseline methods in both classification error and test negative log-likelihood, except on CIFAR-10 when VGG16 is employed.Table 1 reports results on SVHN, CIFAR-10, and CIFAR-100 as means and standard deviations over 10 runs.Source: §5.1 · p. 7, Table 1 · p. 7
  • MixUp is competitive in classification accuracy, Flooding often has low test NLL, and RMP does not match AMP.The authors interpret the RMP comparison as evidence that adversarial rather than random parameter perturbations provide useful information to the regularizer.Source: §5.1 · p. 7

5.2. Improvement over Data Augmentation

  • AMP retains a regularization effect with vanilla augmentation, Cutout, or AutoAugment on SVHN, CIFAR-10, and CIFAR-100.The comparison uses WideResNet and PyramidNet and reports top-1 classification errors in Table 2.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Calibration Effect

  • AMP achieves strong calibration across datasets, while Flooding has lower calibration error on CIFAR-10 and label smoothing degrades calibration.Calibration is measured using expected calibration error (ECE).Source: §5.3 · p. 8, Figure 7 · p. 8

5.4. Influence of Perturbation

  • On CIFAR-10, AMP’s training empirical risk tends to rise with perturbation radius, while its test empirical risk is minimized near radius 0.06.The authors interpret the training-risk pattern as indicating a selected minimum with smaller depth and the test-risk minimum as evidence that an appropriate perturbation magnitude can yield favorable flatness and depth.Source: §5.4 · p. 8, Figure 8 · p. 8

5.5. Computational Cost

  • With one inner iteration, AMP takes around 1.8× as long as ERM training; computational cost increases significantly as the inner iteration count grows.The authors report that N = 1 is sufficient for regularization and describe the additional effort as affordable.Source: §5.5 · p. 8

6. Conclusion

The paper presents AMP as a theoretically justified regularizer motivated by flat minima and reports experimental performance, calibration, and state-of-the-art results on image-classification datasets.

6. Conclusion

  • AMP is proposed as a regularization scheme built on the idea that flat local minima lead to better generalization.The paper contrasts AMP’s theoretical justification with the heuristic nature of many data-dependent regularizers.Source: §6 · p. 8
  • Under an approximating assumption, the theoretical analysis justifies AMP and permits predictions about its behavior as hyperparameters vary.The conclusion says the assumption underlies the theoretical justification.Source: §6 · p. 8
  • Experiments on image-classification datasets support AMP’s regularization performance and improved calibration, and the authors characterize it as state of the art among regularizers.The authors also say the empirical validation appears to further confirm the connection between flat minima and generalization.Source: §6 · p. 8

A. Proof of Corollary 2

The corollary’s proof assumes equal C values, a scaled A relationship, and a positive variance ratio, then derives that the ratio exceeds one.

A. Proof of Corollary 2

  • The proof assumes C1 = C2, A2 = βA1 for β < 1, and σ2² = rσ1² for positive r; it also states γ1* < γ2*.These conditions frame the corollary’s comparison.Source: A. Proof of Corollary 2 · p. 11
  • Using log β < 0, the derivation obtains r > 1.The proof proceeds through algebraic manipulation and notes positivity in the inequality step.Source: A. Proof of Corollary 2 · p. 11

B. Proof of Theorem 2

The theorem states that, for N = 1 and a sufficiently small inner learning rate, an AMP minimization update is equivalent to gradient descent on a specified loss. The proof constructs a projected adversarial parameter perturbation and updates using the gradient at that perturbed parameter.

B. Proof of Theorem 2

  • For N = 1 and sufficiently small inner learning rate ζ, an AMP update is equivalent to a gradient-descent step on the theorem’s stated loss with learning rate η.This is the theorem’s claimed equivalence.Source: B. Proof of Theorem 2 · p. 11
  • The proof perturbs θk by a step of size ζ along the ERM-loss gradient, projecting onto the L2 ball when the perturbation exceeds ϵ.The resulting perturbed parameter is used in the next update.Source: B. Proof of Theorem 2 · p. 11
  • The parameter update uses the gradient computed at the perturbed parameter, and the proof invokes a first-order Taylor expansion for small ζ.The Taylor step is applied in the condition where the unprojected perturbation norm is at most ϵ.Source: B. Proof of Theorem 2 · p. 11

C. Why AMP is not Adversarial Training

AMP perturbs the weights to maximize overall empirical risk, whereas adversarial training perturbs each example to maximize its individual loss. A scalar example shows that the resulting solutions can differ even with one training example.

C. Why AMP is not Adversarial Training

  • The authors attribute the general difference between AMP and adversarial-training solutions to two sources, while noting peculiar cases can coincide.The supplied passage introduces those sources; the scalar example illustrates that solutions may differ even for one example.Source: C. Why AMP is not Adversarial Training · pp. 11–12
  • AMP maximizes overall empirical risk using a global weight perturbation; adversarial training maximizes each example’s loss using an individual input perturbation.For N examples, the ERM loss aggregates the individual losses ℓ(xi; θ).Source: C. Why AMP is not Adversarial Training · p. 11
  • With one scalar example, θ*ADV = θ*ERM = 0 for any perturbation radius ϵ, while θ*AMP = ϵ/3.The example sets x = 1 and ℓ(x; θ) = g(θx).Source: C. Why AMP is not Adversarial Training · p. 12

D. Definition of Expected Calibration Error

Expected Calibration Error (ECE) compares model confidence with accuracy across equal-sized bins of prediction scores.

D. Definition of Expected Calibration Error

E. Influence of Perturbation

The experiments examine AMP’s perturbation radius, robustness, training and test loss curves, selected minima, and computational cost. They report improved robustness over several schemes, lower final test loss, and flatter selected minima, while adversarial training outperforms other schemes on adversarial attacks.

E. Influence of Perturbation

F. Robustness to Adversarial Attacks

  • Adversarial training outperforms the other schemes on FGSM and PGD attacks; AMP and label smoothing also improve robustness against both attacks.The comparison uses PreActResNet18 and top-1 error on test-set adversarial examples; FGSM uses radius 4 per pixel, while PGD uses step size 1 for 10 steps at the same radius.Source: F. Robustness to Adversarial Attacks · p. 12, Table 3 · p. 12

G. Loss Curve

  • AMP converges at a similar speed to the other schemes and reaches a smaller final test loss, despite a slightly increased training loss.ERM has the smallest training loss; Flooding takes effect when training loss becomes very low, while MixUp and AMP take effect earlier.Source: G. Loss Curve · pp. 12–13, Figure 11 · p. 14

H. Flatness of Selected Minima

I. Computing Environment and Resources

  • Training PreActResNet18 on CIFAR-10 takes around 2.4 hours with ERM and 4.2 hours with AMP on a single Tesla V100 GPU.Compute time mainly depends on inner iterations, epochs, and GPU count.Source: §I · p. 13

Related papers

Loading 2010.04925v4…