PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter

arXiv:1711.05101v3cs.LGcs.NEmath.OC

TL;DR

L2 regularization and weight decay are equivalent for standard SGD after learning-rate rescaling but not for adaptive methods such as Adam. The paper decouples weight decay from loss-gradient updates and reports substantially better Adam generalization, including a 15% relative test-error improvement. The authors note that these image-classification results require verification across a wider range of tasks.

  • Problem

    Adaptive optimizers such as Adam may generalize worse than SGD with momentum, partly because common implementations use L2 regularization rather than original weight decay.

  • Method

    The paper decouples weight decay from Adam’s gradient-based loss update, producing the AdamW variant.

  • Results

    15% relative improvement in test error was achieved by AdamW versus Adam across CIFAR-10 and ImageNet32x32 under the reported settings.

  • Takeaways & Limitations

    Decoupled weight decay makes optimal learning-rate and weight-decay settings more independent and helps Adam compete with SGD with momentum on image-classification tasks.

  • Takeaways & Limitations

    The image-classification results must be verified on a wider range of tasks, especially tasks where regularization is expected to matter.

Abstract

from arXiv · show

L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L$_2$ regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by \emph{decoupling} the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

1 INTRODUCTION

The paper identifies a mismatch between L2 regularization and weight decay in adaptive methods such as Adam, which may help explain their weaker generalization than SGD with momentum. It proposes decoupling weight decay from gradient updates and reports improved generalization and less coupled hyperparameter choices.

  • Adaptive gradient methods are widely used, yet SGD with momentum still achieves state-of-the-art results on popular image-classification datasets.
  • L2 regularization and weight decay are equivalent for SGD after learning-rate rescaling but not for Adam.With adaptive gradients, parameters with historically large amplitudes or gradients can be regularized less under L2 regularization.
  • Adam can benefit substantially from a scheduled global learning-rate multiplier, including cosine annealing, despite adapting rates per parameter.
  • 15% relative improvement in test error was achieved by Adam with decoupled weight decay versus L2 regularization across CIFAR-10 and ImageNet32x32.The comparison held across 100–1800-epoch budgets and fixed, drop-step, and cosine-annealing schedules.
  • Decoupled weight decay makes optimal learning-rate and weight-decay settings more independent, easing hyperparameter optimization.
  • The proposed modification aims to make Adam competitive with SGD with momentum and reduce the need for dataset- or task-specific optimizer selection.

2 DECOUPLING THE WEIGHT DECAY FROM THE GRADIENT-BASED UPDATE

This section formalizes why L2 regularization and weight decay coincide for SGD but diverge under adaptive preconditioning. It then motivates AdamW by applying weight decay separately from loss-gradient updates.

  • SGD: For standard SGD, weight decay is equivalent to L2 regularization when the L2 coefficient is rescaled by the learning rate.The equivalence requires λ′ = λα.
  • Adaptive gradients: For adaptive gradients with non-scalar preconditioners, no single L2 coefficient generally reproduces decoupled weight decay.The proposition formalizes the inequivalence when the optimizer’s preconditioner is not proportional to the identity.
  • AdamW: AdamW decouples weight decay from the loss-based gradient update, recovering the original separate decay step for an adaptive optimizer.
  • Adaptive gradients: L2 regularization adapts the combined loss and regularizer gradients, whereas decoupled weight decay adapts only loss gradients.
  • Adaptive gradients: A fixed-preconditioner analysis shows that weight decay corresponds to scale-adjusted L2 regularization, with coordinate effects depending on the preconditioner.The result provides intuition rather than directly applying to practical adaptive methods.
  • Limitation: The fixed-preconditioner proposition does not directly apply to practical adaptive algorithms because their preconditioners change at every step.The section uses the result as intuition for how regularization differs across parameters.

3 JUSTIFICATION OF DECOUPLED WEIGHT DECAY VIA A VIEW OF ADAPTIVE GRADIENT METHODS AS BAYESIAN FILTERING

The paper interprets adaptive optimization as Bayesian filtering, where uncertainty preconditions gradients and decoupled weight decay acts through the state-transition prior rather than uncertainty-dependent regularization.

  • Bayesian filtering frames stochastic optimization as inferring each parameter’s optimal value from minibatch data and the current values of other parameters.
  • The posterior uncertainty Σpost preconditions gradients, producing larger updates for parameters with greater uncertainty and smaller updates for more certain parameters.
  • Adam and RMSprop emerge as special cases of this Bayesian-filtering framework, connecting adaptive gradient methods to uncertainty-based preconditioning.
  • Decoupled weight decay is represented in the state-transition distribution as a regularizer that prevents unbounded weight growth.
  • The paper’s experiments report that AdamW outperforms Adam with L2 regularization and that cosine annealing yields clearly superior results.
  • With A = λ × I, the Bayesian formulation multiplies the current mean estimate θt by (1 − λ), independently of parameter uncertainty unlike L2 regularization.

4 EXPERIMENTAL VALIDATION

Experiments across schedules, training budgets, datasets, and optimizer variants show that decoupled weight decay separates hyperparameter effects and improves Adam’s generalization and anytime performance.

  • The experimental setup evaluates decoupled weight decay across training budgets and learning-rate schedules using ResNet models, batch size 128, and standard CIFAR augmentation.
  • 4.2 DECOUPLING THE WEIGHT DECAY AND INITIAL LEARNING RATE PARAMETERS: SGDW and AdamW produce more separable learning-rate and weight-decay hyperparameter spaces than their L2-regularized counterparts.
  • 4.1 EVALUATING DECOUPLED WEIGHT DECAY WITH DIFFERENT LEARNING RATE SCHEDULES: Decoupled weight decay outperforms L2 regularization for fixed, drop-step, and cosine-annealing schedules, with larger differences under better schedules.
  • 4.1 EVALUATING DECOUPLED WEIGHT DECAY WITH DIFFERENT LEARNING RATE SCHEDULES: Cosine annealing clearly outperforms the other learning-rate schedules and is therefore used for the remaining experiments.
  • 4.2 DECOUPLING THE WEIGHT DECAY AND INITIAL LEARNING RATE PARAMETERS: AdamW’s best settings substantially outperform Adam with L2 regularization and rival the best results of SGD and SGDW.
  • 4.3 BETTER GENERALIZATION OF ADAMW: Across 1800-epoch runs, AdamW achieves better test error than Adam at similar training-loss values, including on ImageNet32x32.
  • 4.4 ADAMWR WITH WARM RESTARTS FOR BETTER ANYTIME PERFORMANCE: AdamWR speeds up AdamW by up to a factor of 10, while AdamW achieves a 15% relative test-error improvement over Adam on CIFAR-10 and ImageNet32x32.
  • 4.4 ADAMWR WITH WARM RESTARTS FOR BETTER ANYTIME PERFORMANCE: AdamWR retains the improved results with better anytime performance, closing most of the gap to SGDWR on CIFAR-10 and matching performance on ImageNet32x32.

5 CONCLUSION AND FUTURE WORK

The paper concludes that decoupling weight decay from Adam’s gradient update improves generalization and reports warm restarts as an additional anytime-performance enhancement, while identifying broader validation as future work.

  • The paper identifies the inequivalence of L2 regularization and weight decay for Adam and reports better generalization from decoupled weight decay.
  • Warm restarts are proposed to improve the anytime performance of Adam and the decoupled-weight-decay variants.
  • The image-classification results require verification across a wider range of tasks, especially tasks where regularization is important.
  • The paper suggests integrating weight decay with other Adam improvements and expects similar results may hold for AdaGrad and AMSGrad.

A FORMAL ANALYSIS OF WEIGHT DECAY VS L2 REGULARIZATION

The formal analysis distinguishes ordinary SGD, where weight decay can be matched by a rescaled L2 penalty, from adaptive methods, where changing preconditioners prevent such equivalence.

  • The appendicial proofs compare optimizer iterates with and without weight decay under regularized objectives.
  • For adaptive optimization, the update with L2 regularization scales the penalty through the preconditioner, whereas decoupled weight decay directly multiplies the parameters by (1 − λ).
  • No L2 regularizer λ′ can make the adaptive-method iterates equivalent for all θt because equivalence would require the preconditioner Mt to be a scalar multiple of the identity.

B ADDITIONAL PRACTICAL IMPROVEMENTS OF ADAM

This section introduces two additional components intended to improve Adam’s performance in practice.

  • The paper adds two practical improvements to Adam beyond decoupled weight decay.

B.1 NORMALIZED WEIGHT DECAY

Normalized weight decay adjusts the decay factor for computational budget, aiming to make settings more robust across runs. The paper presents this normalization as one experimentally informed possibility rather than a uniquely established rule.

  • B.1 NORMALIZED WEIGHT DECAY: Optimal weight decay varies with computational budget, motivating normalization by batch size, dataset size, and training epochs.The proposed replacement uses λnorm and sets λ through a budget-dependent scaling relation.
  • B.1 NORMALIZED WEIGHT DECAY: λnorm can be interpreted as the weight decay used if only one batch pass is allowed.
  • B.1 NORMALIZED WEIGHT DECAY: The normalization choice is presented as one possibility informed by limited experiments, while the broader conclusion favors using some normalization.
  • B.1 NORMALIZED WEIGHT DECAY: For AdamWR, T denotes the total number of epochs in the current restart.

B.2 ADAM WITH COSINE ANNEALING AND WARM RESTARTS

The paper extends Adam with cosine-annealed learning-rate multipliers and warm restarts, carrying over the SGDR schedule while using normalized weight decay. AdamWR combines these components into a practical variant.

  • B.2 ADAM WITH COSINE ANNEALING AND WARM RESTARTS: Cosine annealing decays the learning-rate multiplier within each run, while warm restarts periodically increase it without restarting from scratch.Restarts retain the previous parameter values and can preserve previously acquired information such as momentum.
  • B.2 ADAM WITH COSINE ANNEALING AND WARM RESTARTS: The schedule does not adjust the maximum multiplier at each restart because doing so would introduce additional hyperparameters.
  • B.2 ADAM WITH COSINE ANNEALING AND WARM RESTARTS: The schedule begins with a short run and multiplies its duration by Tmult at each restart to improve anytime performance.An example uses an initial run of 1% to 10% of the expected total budget and Tmult = 2.
  • B.2 ADAM WITH COSINE ANNEALING AND WARM RESTARTS: AdamWR combines AdamW, the scheduled learning-rate multiplier, and normalized weight decay.The same normalized-decay setting can be used across short and long AdamWR and SGDWR runs.
  • B.2 ADAM WITH COSINE ANNEALING AND WARM RESTARTS: With an initial 100-epoch run and Tmult = 2, the next run lasts 200 epochs after the multiplier resets.

D ADDITIONAL RESULTS

Additional experiments examine long standard-Adam runs, schedule construction, normalized decay, and comparisons across CIFAR-10 and ImageNet32x32. They report consistent benefits from decoupled and normalized weight decay, with faster convergence or better generalization in several settings.

  • D ADDITIONAL RESULTS: Standard Adam was evaluated for 1800 epochs over a 4 × 4 logarithmic grid of learning-rate and L2 weight-decay settings.The grid’s coarse resolution reflects the computational expense of the long runs.
  • D ADDITIONAL RESULTS: Standard Adam was at best comparable to AdamW despite using 18 times more epochs and a larger computational budget.The comparison also involved a smaller network for AdamW.
  • D ADDITIONAL RESULTS: As runtime increases, smaller raw weight-decay values become preferable; normalized decay makes optimal settings more similar across budgets and datasets.The square-root scaling was checked on ImageNet32x32, whose epochs are 24 times longer than CIFAR-10 epochs.
  • D ADDITIONAL RESULTS: On ImageNet32x32, weight decay produced better training loss than L2 regularization and an even larger improvement in test error.
  • D ADDITIONAL RESULTS: Adam and decoupled-weight-decay variants converged faster in training loss on CIFAR-10, while restart variants generalized better than their non-restart counterparts.The convergence difference on ImageNet32x32 was small; AdamW also had lower test error than Adam at matched training loss.
  • D ADDITIONAL RESULTS: SuppFigures 5 and 6 show test-error curves in the top row and training-loss curves in the bottom row for CIFAR-10 and ImageNet32x32.
Loading 1711.05101v3…