PaperlayerView exact revision on arXiv

Source-linked AI summary

Three Mechanisms of Weight Decay Regularization

Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Grosse

arXiv:1810.12281v1cs.LGstat.ML

TL;DR

Weight decay is widely used, but its regularization effect is not fully explained by the traditional L2 interpretation, particularly across different optimizers and architectures. The paper empirically studies SGD, Adam, and K-FAC, finding three distinct mechanisms: increased effective learning rate, approximate Jacobian-norm regularization, and reduced effective damping.

  • Problem

    Weight decay’s regularization effect remains poorly understood because literal weight decay can outperform L2 regularization and can improve Batch-Normalized networks despite prediction-preserving weight rescaling.

  • Method

    The paper compares weight decay and L2 regularization in image classifiers trained with SGD, Adam, and K-FAC across network architectures and settings.

  • Results

    The paper identifies three mechanisms: increasing effective learning rate, approximately regularizing the input-output Jacobian norm, and reducing effective damping for second-order optimization.

  • Takeaways & Limitations

    Weight decay’s regularization effect depends on the optimization algorithm and network architecture rather than reflecting a single universal mechanism.

  • Takeaways & Limitations

    The paper does not establish how broadly the three mechanisms apply and notes that additional mechanisms may remain undiscovered.

Abstract

from arXiv · show

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

1 INTRODUCTION

Weight decay is traditionally viewed as L2 regularization, but evidence shows this interpretation is incomplete, especially for Adam and Batch-Normalized networks. The paper studies these discrepancies across optimizers and identifies distinct mechanisms underlying weight decay’s regularization effects.

  • Weight decay is traditionally interpreted as L2 regularization because it arises from the gradient of the weights’ L2 norm under gradient descent.
  • Adam can generalize substantially better with literal weight decay than with an L2 regularizer added to the objective.
  • Batch Normalization makes weight decay theoretically capacity-neutral because weight rescaling can leave network predictions unchanged.
  • The paper investigates weight decay and L2 regularization for image classifiers trained with SGD, Adam, and K-FAC.
  • The experiments identify three mechanisms: increasing effective learning rate, approximately regularizing the input-output Jacobian norm, and reducing effective damping for second-order optimization.
  • The authors caution that additional analysis is needed to determine how broadly these mechanisms apply and whether further mechanisms exist.

2 PRELIMINARIES

The preliminaries introduce gradient, natural-gradient, and Gauss-Newton optimization, along with K-FAC and Batch Normalization. These methods differ in the metrics or curvature information used to define parameter updates and normalize layer inputs.

  • Stochastic Gradient Descent: SGD minimizes empirical risk using a Euclidean steepest-descent surrogate and updates parameters by ∆θ = −η∇θL(θ).
  • Natural gradient: Natural gradient descent incorporates curvature information through the KL divergence between predictive distributions as its local distance.
  • Gauss-Newton algorithm: The Gauss-Newton algorithm uses squared output-space distance and updates parameters with the inverse Gauss-Newton matrix.
  • Gauss-Newton algorithm: The Gauss-Newton matrix is constructed using the network-output Jacobian with respect to parameters and is invariant to the network’s parameterization.
  • K-FAC: K-FAC approximates curvature with Kronecker-factored matrices, enabling efficient approximate natural-gradient updates for large neural networks.
  • Batch Normalization: Batch Normalization stabilizes layer-input distributions by subtracting the mini-batch mean and dividing by the standard deviation.
  • Batch Normalization: The setup applies Batch Normalization before activations, excludes the output layer, and ignores γ and β because they do not affect performance in practice.

3 THE EFFECTIVENESS OF WEIGHT DECAY

The paper compares weight decay with L2 regularization across optimizers, architectures, datasets, and batch sizes. Weight decay generally improves performance, while its update differs from L2 regularization under preconditioned optimization.

  • Weight decay and L2 regularization: Under gradient descent, weight decay and L2 regularization are equivalent, but they differ when the gradient update is preconditioned by Adam or K-FAC.
  • Weight decay and L2 regularization: The updates differ because preconditioned L2 regularization applies the preconditioner to the parameters, whereas literal weight decay does not.
  • Experiments: The study evaluates VGG16 and ResNet32 on CIFAR-10 and CIFAR-100 with varied batch sizes and multiple optimization algorithms.
  • Empirical effectiveness: Weight decay consistently improves generalization and outperforms L2 regularization when the two methods differ.
  • Empirical effectiveness: Weight decay closes most generalization gaps between first- and second-order optimizers and between small and large batches.
  • Empirical effectiveness: Weight decay significantly improves Batch-Normalized networks despite not meaningfully constraining their capacity, with much of the effect coming from Batch-Normalized layers.
  • Empirical effectiveness: Weight decay produces an especially strong performance boost for K-FAC when Batch Normalization is disabled.

4 THREE MECHANISMS OF WEIGHT DECAY REGULARIZATION

The paper identifies three algorithm- and architecture-dependent mechanisms behind weight decay: increasing effective learning rates, approximately regularizing input-output Jacobians, and reducing K-FAC’s effective damping.

  • Mechanism I: Higher Effective Learning Rate: In BN networks trained with SGD or Adam, weight decay reduces weight scale and increases the effective learning rate, strengthening gradient-noise regularization.This effect is concentrated in BN layers, where ordinary L2 penalties do not meaningfully constrain the represented function.
  • Mechanism I: Higher Effective Learning Rate: Without weight decay, normalization effects can reduce the effective learning rate by a factor of 10 during the first 50 epochs, whereas weight decay keeps it approximately constant within each schedule stage.The comparison uses BN networks trained with SGD and a learning-rate schedule containing three factor-of-10 reductions.
  • Mechanism I: Higher Effective Learning Rate: Transferring weight-decay-induced layer norms to a network without weight decay eliminates almost the entire generalization gap, supporting effective learning rate as the primary mechanism for BN networks.Adding weight decay to the top layer as well fully closes the remaining gap in the reported experiment.
  • Mechanism II: Approximate Jacobian Regularization: For K-FAC without BN, weight decay approximately regularizes the input-output Jacobian through the K-FAC Gauss-Newton norm, which is proportional to the squared Frobenius Jacobian norm for linear networks with whitened inputs.In practical nonlinear networks, K-FAC GN norm correlates strongly with Jacobian norm and generalization gap, and K-FAC reduces Jacobian norm more than SGD.
  • Mechanism III: Smaller Effective Damping Parameter: For K-FAC applied to BN networks, weight decay reduces effective damping and helps retain second-order behavior, with a stronger effect for K-FAC-F because its Fisher matrix shrinks during training.Weight decay on non-BN layers has the largest effect, but applying it to BN layers also produces significant gains, especially for K-FAC-F.

5 DISCUSSION

The paper identifies three mechanisms through which weight decay improves generalization, while emphasizing that neural-network training dynamics remain complex and incompletely understood.

  • Weight decay can improve generalization by increasing the effective learning rate, reducing the Jacobian norm, or reducing the effective damping parameter.Which mechanism applies depends on the architecture and optimization algorithm.
  • The authors argue that analyzing why algorithms work is important because final performance numbers can obscure interactions among training phenomena.They specifically highlight the dynamics of neural-network norms as worthy of closer study.

A EXPERIMENTS DETAILS

The experiments cover image classification with several datasets, architectures, and optimizers, using distinct training schedules for first-order methods and K-FAC; the appendix also introduces a ReLU-network gradient claim.

  • Experimental settings: Experiments use MNIST with fully connected networks, and CIFAR-10 and CIFAR-100 with VGG16 and widened ResNet32 architectures.The convolutional layers in ResNet32 are widened by a factor of 4.
  • Experimental settings: The evaluated optimization methods are SGD, Adam, and K-FAC, with K-FAC studied using Fisher-information and Gauss-Newton curvature matrices.
  • Training schedules: Default training uses batch size 128; SGD and Adam run for 200 epochs, whereas K-FAC runs for 100 epochs with separate learning-rate schedules.K-FAC curvature statistics use running averages with periodic re-estimation and inverse updates.
  • Gradient structure: The appendix states a claim for depth-L feed-forward ReLU networks without biases.
  • Gradient structure: The key lemma observation is that ReLU networks are piecewise linear up to the output, using σ(z) = σ′(z)z.
  • Gradient structure: The proof defines output logits fθ(x), layer outputs zl, activations al = σ(zl), and input activations a0 = x.
  • Gradient structure: The derivation then considers an additional relation and sums contributions over all layers to obtain the concluding equation.

C PROOF OF LEMMA 2

The proof develops a K-FAC Gauss-Newton relation for ReLU feed-forward networks and connects the resulting quantities to Jacobian-based expressions under stated structural conditions.

  • The appendix states a claim for a depth-L feed-forward network with ReLU activation.
  • The derivation invokes an equation for the K-FAC Gauss-Newton norm and then applies the definition of the Gauss-Newton norm.
  • The proof relates the parameter-space Jacobian Jθθ to the input-space Jacobian Jxx through the network output fθ(x), then combines these equalities.
  • For linear networks, the Kronecker-product approximation is exact, and GK-FAC is the diagonal-block version of the Gauss-Newton matrix G.
  • The final steps use Lemma 1 and an additional derived relation to reach the stated conclusion.

D DERIVATION OF EQUATION 14

This appendix derives the gradient of the Gauss-Newton norm and explains K-FAC’s curvature approximation, efficient computation, and separate treatment of L2 regularization and weight decay.

  • Gradient derivation: The natural-gradient update is introduced before being rewritten into an alternative form for the derivation.
  • Gradient derivation: The squared norm G = (L + 1)^2 E_x[⟨fθ(x), fθ(x)⟩] is differentiated to obtain its gradient structure.
  • Gradient derivation: Using Lemma 1, the output fθ(x) is replaced by a parameter-Jacobian expression to rewrite equation 18.
  • Gradient derivation: The resulting gradient has the same form as when the Gauss-Newton matrix is treated as constant, up to a factor of (L + 1).
  • K-FAC curvature: K-FAC approximates curvature using Kronecker factors, including Fisher and Gauss-Newton metrics, to perform efficient approximate natural-gradient updates.
  • K-FAC curvature: For a layer, the weight gradient is ∇WlL = al(∇slL)⊤, and K-FAC decouples the layer’s Fisher matrix using approximations.
  • K-FAC curvature: The approximation assumes independence between activations and output gradients, while between-layer independence yields a block-diagonal Fisher approximation.The factorization reduces memory requirements and enables efficient natural-gradient computation.
  • K-FAC curvature: K-FAC natural-gradient computation uses matrix transformations comparable in size to each layer’s weight matrix.

G.1 LARGE-BATCH TRAINING

Large-batch training creates a substantial generalization gap, but tuned weight decay nearly closes it for K-FAC on CIFAR-10 and reduces it on CIFAR-100. Weight decay also appears to improve optimization by preventing the training loss from reaching zero without it.

  • G.1 LARGE-BATCH TRAINING: Large-batch K-FAC is studied because it scales favorably with larger mini-batches, despite prior evidence that large-batch methods generalize worse.
  • G.1 LARGE-BATCH TRAINING: Weight decay almost closes the large-batch generalization gap for K-FAC on CIFAR-10 and causes much of the gap to diminish on CIFAR-100.The comparison interpolates between BS128 and BS2000.
  • G.1 LARGE-BATCH TRAINING: The generalization gap of SGD also disappears with well-tuned weight decay regularization.Weight decay factors and learning rates were tuned using a held-out validation set.
  • G.1 LARGE-BATCH TRAINING: Without weight decay, the training loss cannot decrease to zero, suggesting that weight decay may also speed up training.

G.3 OPTIMIZATION PERFORMANCE OF DIFFERENT OPTIMIZERS

The paper compares optimization speed across SGD, Adam, and K-FAC variants on CIFAR-10 using VGG16 and widened ResNet32 models. K-FAC-G achieves the strongest optimization performance per epoch and in compute time across the reported experiments.

  • G.3 OPTIMIZATION PERFORMANCE OF DIFFERENT OPTIMIZERS: Optimization is evaluated by both per-epoch performance and wall-clock time.
  • G.3 OPTIMIZATION PERFORMANCE OF DIFFERENT OPTIMIZERS: The experiments compare K-FAC-G and K-FAC-F with SGD and Adam on CIFAR-10 using VGG16 and ResNet32 architectures.
  • G.3 OPTIMIZATION PERFORMANCE OF DIFFERENT OPTIMIZERS: K-FAC-G reduces training loss to 1e-4 within 10 epochs and improves wall-clock performance over the baselines.Although K-FAC methods take more time per epoch, they still show wall-clock improvements.
  • G.3 OPTIMIZATION PERFORMANCE OF DIFFERENT OPTIMIZERS: K-FAC-G outperforms K-FAC-F and the other baselines in optimization per epoch and compute time on widened ResNet32.The network uses a widening factor of 4.
Loading 1810.12281v1…