Source-linked AI summary
L2 Regularization versus Batch and Weight Normalization
Twan van Laarhoven
TL;DR
Normalization makes neural-network functions invariant to weight scaling, raising the question of why L2 regularization is used with normalized models. The paper analyzes this interaction theoretically and experimentally, finding that L2 does not regularize the function but changes weight scale and effective learning rates. It also examines optimization methods and discusses ways to mitigate the resulting dependence.
Problem
Normalization makes learned functions invariant to weight scaling, challenging the assumed regularizing role of L2 penalties in normalized networks.
Method
The paper investigates L2 regularization with Batch, Weight, and Layer Normalization through theoretical and experimental analysis.
Results
L2 regularization has no regularizing effect with normalization but strongly influences weight scale and the effective learning rate.
Takeaways & Limitations
No first-order gradient method examined fully eliminates normalization’s learning-rate dependence, while enforcing ∥w∥ = 1 directly solves the problem and removes one hyperparameter.
Takeaways & Limitations
The experiments do not regularize Batch Normalization’s γ and β parameters, although preliminary experiments suggest this does not affect results.
Abstract
from arXiv · showhide
Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.
1 Introduction
Normalization standardizes neural-network training inputs and can accelerate training, while making the learned function invariant to weight scaling. The paper investigates why L2 regularization remains useful in this setting despite that invariance.
- 1 Introduction: Batch Normalization normalizes unit inputs using batch means and variances, with fixed statistics at test time.Parameters γ and β preserve the ability to represent different input ranges.
- 1 Introduction: In a neural network unit, X is the input and w and b are learned weights and bias, while g is a nonlinearity.Examples of g include ReLU and sigmoid; convolutional weights may be shared across units.
- 1 Introduction: L2 regularization combined with gradient descent produces weight decay by scaling weights slightly below one at each update.The paper notes that this continued use is surprising when normalization makes function values insensitive to weight scale.
- 1 Introduction: Normalization strategies make the learned function invariant to scaling of the weights.The paper expresses this invariance as y(X; αw, γ, β) = y(X; w, γ, β).
- 1 Introduction: The paper studies L2 regularization with Batch, Weight, and Layer Normalization and finds no regularizing effect, while the term strongly influences learning rate.This is the paper’s central contribution stated in the introduction.
2 L2 Regularization?
With normalization, the objective’s loss component is unchanged by rescaling weights, so L2 regularization shrinks weights without simplifying the computed function. Consequently, λ does not affect the optimum because weight scale can compensate.
- 2 L2 Regularization?: The L2-regularized objective combines an unregularized objective with a weight penalty.The loss terms are defined with respect to each unit’s output for a sample.
- 2 L2 Regularization?: Normalization makes the unit output unchanged when weights are rescaled by α.Thus the unregularized objective depends on the rescaled weights through the same output.
- 2 L2 Regularization?: The L2 penalty makes weights smaller but does not regularize by making the computed function simpler.The function remains exactly the same regardless of weight scale.
- 2 L2 Regularization?: λ has no impact on the optimum because weights can be rescaled to compensate for the penalty.This conclusion follows from scale invariance of the normalized function.
3 Effect of the Scale of Weights on Learning Rate
Normalization makes the learned function invariant to weight scaling, but first-order updates still change with weight scale. Consequently, L2 regularization affects the effective learning rate rather than simplifying the computed function.
- Effect of the Scale of Weights on Learning Rate: Weight scaling leaves the normalized function unchanged, while the gradient of the objective changes with the weight scale.This holds for Batch, Layer, and Weight Normalization, with Layer Normalization requiring uniform scaling of the whole layer matrix.
- Effect of the Scale of Weights on Learning Rate: For normalized weights w′ with norm 1, the effective learning rate is ηeff = η/∥w∥2.The weight norm represents the scale removed by reparameterizing the weights.
- Effect of the Scale of Weights on Learning Rate: Decreasing the weight scale through weight decay increases the effective learning rate, contrary to the intuition that regularization stabilizes training.The same rescaling can be represented as an SGD update with a different learning rate and regularization amount.
- Effect of the Scale of Weights on Learning Rate: The optimal learning rate is proportional to 1/λ when changing the L2 regularization coefficient λ while preserving equivalent effective updates.For a given optimal rate η∗ at λ, rescaling implies a corresponding rate at λ′ that maintains the same effective updates.
- Effect of the Scale of Weights on Learning Rate: Without regularization, weights can grow unbounded and the effective learning rate can approach zero.Thus L2 regularization can remain beneficial for training even though it does not simplify the normalized function.
4 Effect of Regularization on the Scale of Weights
L2 regularization controls weight scale during training, while stochastic gradients increase it, causing the effective learning rate to change over time. Under the paper’s analysis, choosing η proportional to 1/λ yields a constant effective learning rate.
- Effect of Regularization on the Scale of Weights: Weight norms grow from stochastic loss gradients and shrink from L2 regularization, so the effective learning rate changes during training.The paper frames weight-scale dynamics as a balance between gradient-driven growth and regularization-driven shrinkage.
- Effect of Regularization on the Scale of Weights: The convergence analysis models inputs and output gradients as draws from simple probability distributions after the expected gradient reaches zero.This is an explicit simplifying assumption for analyzing the stationary weight scale.
- Effect of Regularization on the Scale of Weights: Table 1 summarizes effective learning rates and weight scaling for various update rules.The paper uses the table to compare update rules beyond plain stochastic gradient updates.
- Effect of Regularization on the Scale of Weights: ηeff is approximately determined by η and λ through the analyzed stationary weight scale.The displayed derivation connects the effective learning rate to the regularization coefficient after assuming convergence.
- Effect of Regularization on the Scale of Weights: Choosing η ∝ 1/λ results in a constant effective learning rate.This follows the inverse relationship between the base learning rate and the regularization coefficient established earlier.
5 Other Update Rules
The paper compares update rules under weight scaling, showing that momentum, RMSProp, and ADAM do not fully remove effective-learning-rate dependence, while second-order methods and fixed-norm weights address it.
- 5.1 Momentum: Momentum retains the same effective learning rate as SGD, η/∥w∥2, when weights are scaled by α.Scaling weights by α corresponds to updates with learning rate η/α2.
- 5.2 RMSProp: RMSProp adapts to gradient norms but still leaves the learning rate dependent on weight scale.Its effective learning rate is η/∥w∥2, and the paper describes this dependence as not completely resolved.
- 5.3 ADAM: ADAM combines RMSProp with momentum, yet its effective learning rate remains η/∥w∥2 under weight scaling.The transformation uses η′ = η/α and ϵ′ = αϵ.
- 5.4 Second Order Methods: First-order methods are sensitive to parameter scaling because their gradients increase as parameter scale decreases, and RMSProp and ADAM only partially correct this effect.The paper identifies second derivatives as a true solution to this scaling problem.
- 5.4 Second Order Methods: Newton’s method removes learning-rate dependence on weight scale, giving ηeff = η, but its Hessian is impractical to compute explicitly for neural networks.Hessian Free optimization is presented as an alternative requiring only Hessian multiplication.
- 5.5 Normalizing Weights: Fixing the weight norm at 1 prevents weight scale from changing, so the effective learning rate no longer depends on the regularization parameter λ.This update differs from Weight Normalization because it fixes the norm rather than incorporating it into gradient computation.
6 Experimental Validation
Experiments on CIFAR10 with Batch Normalization show that regularization and learning rate jointly determine weight scale and test error through the effective learning rate. Explicitly normalizing weights largely removes regularization's effect, while optimizer-specific relationships remain.
- Experimental setup: CIFAR10 experiments used a four-convolutional-layer network with Batch Normalization, data augmentation, and 100 training epochs.The learning rate was reduced by 10 after epochs 50 and 80.
- Weight norms: Weight norms depend most visibly on regularization at high η and on η at low λ, where training can converge.In these regions, the observed norm resembles the theoretical prediction, although the supplied passage truncates that expression.
- Test error: Along diagonals with constant effective learning rate η^2/λ, test error is roughly constant for Nesterov momentum.The result links comparable test errors to comparable effective learning rates rather than to λ alone.
- Test error: After each update, normalizing weights to norm 1 makes the effect of λ almost disappear from test error.Large λ still worsens error because ηλw becomes a significant part of the gradient.
- Optimal learning rate: For SGD and Nesterov momentum, cross-validation gives η∗∝1/λ; RMSProp and ADAM follow a different relationship reported in Figure 4.The supplied prose states that the results closely match theory but truncates the RMSProp and ADAM expression.
7 Discussion
Normalization makes learned functions invariant to weight scaling, while regularization strongly affects that scaling. Fixing weight norms directly removes this interaction and one training hyperparameter, but uncontrolled effective-learning-rate changes can hinder reproducibility.
- Normalization and scaling: Batch, Layer, and Weight Normalization make the learned function invariant to scaling of the weights.Regularization nevertheless strongly affects the scale of those weights.
- Mitigation: Forcing ∥w∥=1 directly solves the regularization-dependent scaling problem and removes one hyperparameter from training.This is presented as a direct alternative to relying on first-order gradient methods to eliminate the effect.
- Effective learning rate: Without regularization, weight norms tend to increase over time, causing the effective learning rate to decrease.The passage notes that this may be desirable because many training methods lower learning rates explicitly.
- Effective learning rate: The effective-learning-rate decrease can be difficult to control and depend strongly on initial training steps, making results harder to reproduce.The paper treats normalization's effect on effective learning rate as not necessarily harmful, but potentially difficult to manage.
- Scope and caveats: The experiments did not regularize Batch Normalization's γ and β parameters, although preliminary experiments reportedly showed no effect on the results.The passage attributes this to scale invariance with rectified linear activations in subsequent layers.