PaperlayerView exact revision on arXiv

Source-linked AI summary

Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules

Doğay Altınel

arXiv:2509.18396v1cs.LG

TL;DR

Optimizer choice affects deep-learning training and model performance, while the literature contains numerous established and recently introduced methods. This paper reviews their update rules and techniques, concluding that optimizer development must adapt to increasingly complex models and growing data volumes.

  • Problem

    Numerous established and recently introduced optimizers are available, and optimizer updates affect both deep-learning training and model performance.

    Source: §I · p. 2

  • Method

    The paper reviews deep-learning optimizers individually, detailing their update rules, variables, techniques, contributions, and default hyperparameters.

    Source: §I · p. 2

  • Results

    The review describes optimizer development from basic gradient descent toward sophisticated methods using momentum, adaptive learning rates, and other techniques for convergence, generalization, and resource use.

    Source: §IV · p. 20

  • Takeaways & Limitations

    The evolution of optimizers demonstrates the need for methods that adapt to increasingly complex deep-learning models and growing data volumes.

    Source: §IV · p. 20

Abstract

from arXiv · show

Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep learning, a wide range of optimizers with different approaches have been developed. This study aims to provide a review of various optimizers that have been proposed and received attention in the literature. From Stochastic gradient descent to the most recent ones such as Momentum, AdamW, Sophia, and Muon in chronological order, optimizers are examined individually, and their distinctive features are highlighted in the study. The update rule of each optimizer is presented in detail, with an explanation of the associated concepts and variables. The techniques applied by these optimizers, their contributions to the optimization process, and their default hyperparameter settings are also discussed. In addition, insights are offered into the open challenges encountered in the optimization of deep learning models. Thus, a comprehensive resource is provided both for understanding the current state of optimizers and for identifying potential areas of future development.

I. INTRODUCTION

The introduction frames optimizer choice as central to balancing convergence stability, generalization, and computational cost, then surveys update rules and their development from gradient descent through momentum methods. It introduces batch, stochastic, and mini-batch updates, followed by momentum and Nesterov look-ahead.

I. INTRODUCTION

  • Optimizers iteratively minimize neural-network loss by updating trainable parameters; the paper focuses chiefly on first-order methods because second-order methods are computationally expensive.The introduction frames optimization around convergence stability, generalization, and computational cost.Source: §I · pp. 1–2

II. DEEP LEARNING OPTIMIZERS

  • The review examines optimizers individually, explains their update rules and variables, and lists them chronologically with introduction years and references.It also notes that later sections avoid repeating previously explained variables and concepts.Source: §I · p. 2, Table 1 · p. 3

A. STOCHASTIC GRADIENT DESCENT

  • Batch gradient descent uses the full dataset per update, while SGD uses one sample and mini-batch GD uses a subset; stochasticity increases loss variance but can reduce training time.Mini-batches reduce variance relative to single-sample updates and are commonly treated as a technique used with SGD or other optimizers.Source: A. STOCHASTIC GRADIENT DESCENT · pp. 2–3

B. MOMENTUM

  • Momentum combines current and past gradients in a velocity variable, which smooths loss reduction and can accelerate convergence with fewer oscillations.The velocity’s influence increases as the momentum parameter approaches 1.Source: B. MOMENTUM · p. 4

C. NESTEROV ACCELERATED GRADIENT

  • NAG computes gradients at a look-ahead weight estimate, and is described as reducing oscillation and converging faster to minima than SGD.Its theoretical excess-error convergence rate is stated to improve from O(1/T) to O(1/T^2).Source: C. NESTEROV ACCELERATED GRADIENT · p. 4

E. ROOT MEAN SQUARE PROPAGATION

RMSprop and Adadelta address Adagrad’s decreasing-learning-rate problem in different ways: RMSprop uses a decaying average of squared gradients, while Adadelta replaces the global learning rate with a scale based on weight updates.

E. ROOT MEAN SQUARE PROPAGATION

  • RMSprop replaces Adagrad’s cumulative squared-gradient scaling with an exponentially decaying average, preventing the learning rate from decreasing too quickly.The update rule is described as suitable for non-stationary cases where the data distribution changes over time.Source: E. ROOT MEAN SQUARE PROPAGATION · p. 5

F. ADADELTA

  • Adadelta addresses Adagrad’s decreasing learning rate and updates weights without manually specifying a global learning rate.It uses the root mean square of weight updates in place of the learning rate.Source: F. ADADELTA · p. 6
  • Adadelta’s numerator includes a smoothing term so updates can occur at the start of optimization and when weight updates become very small.The update calculation uses the preceding mean square of weight updates and the current mean square of gradients.Source: F. ADADELTA · p. 6

G. ADAPTIVE MOMENT ESTIMATION

Adam combines first- and second-moment gradient estimates with bias correction to adapt updates for individual weights. Adamax modifies Adam’s scaling to use an exponentially weighted infinity norm, which does not require bias correction.

G. ADAPTIVE MOMENT ESTIMATION

  • Adam combines first- and second-moment gradient estimates, correcting their initial zero bias before using them to update weights.The paper gives typical decay rates of 0.9 and 0.999 for the first and second moments, respectively.Source: G. ADAPTIVE MOMENT ESTIMATION · pp. 6–7
  • Adam adaptively adjusts each weight’s learning rate using first-order gradients and is reported to converge rapidly in most cases with minimal hyperparameter tuning.The original paper’s stated defaults for η and ϵ are 0.001 and 10^-8.Source: G. ADAPTIVE MOMENT ESTIMATION · p. 7

H. ADAMAX

  • Adamax replaces Adam’s L2 gradient scaling with an exponentially weighted infinity norm; its maximum operation avoids bias toward zero, so the norm needs no bias correction.The paper presents Adamax as an Adam variant intended to achieve more stability.Source: H. ADAMAX · p. 7

I. NESTEROV ACCELERATED ADAM

Nadam extends Adam’s first-moment estimate with Nesterov’s look-ahead approach, while AdamW separates weight decay from gradient-based updates. AdamW’s warm-restart variant applies cosine annealing to its learning-rate schedule.

I. NESTEROV ACCELERATED ADAM

  • Nadam scales Adam’s first-moment estimate and extends it with the Nesterov estimate, aiming for faster convergence.The formulation is anticipated to improve learning stability and convergence speed, with a slight increase in computational load.Source: §I · pp. 7–8

J. ADAM WITH DECOUPLED WEIGHT DECAY

  • AdamW decouples weight decay from gradient-based updating, because it argues that Adam’s L2-regularized weight update is not true weight decay.The paper says this provides independent weight-decay regularization and helps improve generalization performance.Source: J. ADAM WITH DECOUPLED WEIGHT DECAY · p. 8
  • AdamWR applies cosine annealing to η_t and resets the schedule when each warm-restart cycle ends.The paper describes warm restarts as a way to enhance AdamW’s anytime performance.Source: J. ADAM WITH DECOUPLED WEIGHT DECAY · p. 8

K. ADAFACTOR

Adafactor reduces the memory cost of adaptive optimization through factored second-moment estimates and additional update controls. The section also describes AMSGrad’s stability modification and AdamNC’s time-varying decay rates.

K. ADAFACTOR

  • Adafactor approximates a matrix second-moment estimate with row and column factors, requiring n + m rather than n × m memory.The factor vectors represent the matrix estimate as their outer product.Source: K. ADAFACTOR · p. 9
  • Adafactor removes Adam’s first-moment estimate, then uses an increasing second-moment decay schedule and update clipping to address large updates and instability.Updates are scaled down when their RMS exceeds the clipping threshold.Source: K. ADAFACTOR · p. 9
  • Adafactor sets its learning rate using the relative step size and weight scale, with defaults including ε1 = 10^-30, ε2 = 10^-3, and τc = 1.The listed defaults also include qt = min(10^-2, 1/√t) and β̂2t = 1 − t^-0.8; the paper says it performs effectively, particularly on large-scale models.Source: K. ADAFACTOR · p. 9

L. AMSGRAD

  • AMSGrad addresses Adam’s potentially increasing learning rate by retaining the maximum of past second-moment estimates, which decay more slowly and improve optimization stability.The stated concern includes sparse gradients and high β2 values that can make Adam’s second-moment estimate decrease over time.Source: L. AMSGRAD · p. 10
  • AdamNC replaces Adam’s constant decay rates with time-varying rates, including a β2t schedule that increases over time.The passage gives β2t = 1 − 1/t as one possible choice.Source: L. AMSGRAD · p. 10

M. RECTIFIED ADAM

Rectified Adam addresses early adaptive-learning-rate variance, while LAMB combines per-dimension and layerwise normalization for large-batch training. The reported LAMB result emphasizes convergence at large batch sizes.

M. RECTIFIED ADAM

  • Rectified Adam adds a rectification term to control adaptive learning-rate variance and stabilize optimization early in training.The paper says this rectification justifies the learning-rate warmup heuristic.Source: M. RECTIFIED ADAM · p. 10
  • Rectified Adam uses variance rectification when ρt > 4 and otherwise updates from the first-moment estimate without adaptation.The paper states that it can use Adam’s hyperparameter settings.Source: M. RECTIFIED ADAM · p. 11

N. LAYERWISE ADAPTIVE MOMENTS OPTIMIZER FOR BATCH TRAINING

  • LAMB is a layerwise Adam variant that combines per-dimension normalization with layerwise normalization to support large-batch training.Its layerwise adaptation lets weights in different layers learn at different speeds.Source: N. LAYERWISE ADAPTIVE MOMENTS… · p. 11
  • LAMB improves convergence toward the target, especially at batch sizes such as 32K; N-LAMB and NN-LAMB perform comparably to LAMB.N-LAMB uses Nesterov momentum for the first moment, while NN-LAMB applies it to both moment estimates.Source: N. LAYERWISE ADAPTIVE MOMENTS… · p. 11

O. LOOKAHEAD

Lookahead combines an inner optimizer with periodic synchronization of fast and slow weights. AdaBelief modifies Adam’s second-moment estimate to base weight changes on how closely gradients match their prediction.

O. LOOKAHEAD

  • Lookahead runs a standard optimizer on fast weights and periodically synchronizes slow weights as an exponential moving average of the fast weights.The slow weights change with lower variance than the fast weights, and the paper associates their use with improved training stability.Source: O. LOOKAHEAD · p. 12

P. ADABELIEF

  • AdaBelief treats the gradient’s deviation from its moving-average prediction as a measure of belief: smaller deviations correspond to larger weight changes.It uses the exponential moving average of (gt − mt)^2, st, and its bias correction instead of Adam’s vt and v̂t.Source: P. ADABELIEF · p. 12
  • AdaBelief uses Adam’s hyperparameters and can retain the same default values.Source: P. ADABELIEF · p. 12

Q. SHARPNESS-AWARE MINIMIZATION

SAM and ASAM optimize loss sharpness as well as loss value, with ASAM adapting sharpness to be scale-invariant. Ranger21 combines AdamW with multiple optimization techniques and schedules.

Q. SHARPNESS-AWARE MINIMIZATION

  • SAM seeks low-sharpness minima by minimizing both loss value and loss sharpness, which the paper links to improved generalization.It computes the update gradient at perturbed weights rather than the original weights.Source: Q. SHARPNESS-AWARE MINIMIZATION · p. 13

R. ADAPTIVE SHARPNESS-AWARE MINIMIZATION

  • ASAM introduces scale-invariant adaptive sharpness to address SAM’s sensitivity of sharpness to weight rescaling.Its base optimizer can be replaced, and adaptive sharpness is also proposed as a generalization assessment metric.Source: R. ADAPTIVE SHARPNESS-AWARE MINIMIZATION · p. 13

S. RANGER21

  • Ranger21 uses AdamW as its main optimizer alongside eight components, including adaptive gradient clipping, gradient centralization, positive-negative momentum, learning-rate schedules, regularization, and Lookahead.The paper describes these techniques as orthogonal and synergistic in optimization.Source: S. RANGER21 · pp. 13–14
  • Ranger21’s linear warm-up starts at a low learning rate and increases it over 22% of total iterations by default; its Explore stage seeks a wide minimum region before Exploit reduces the rate toward zero.The default Exploit warm-down lasts 28% of the total iterations.Source: S. RANGER21 · pp. 14–15
  • Ranger21 combines stable weight decay with norm loss, which pushes weight vectors toward norm one and aims to address exploding or vanishing gradients.The paper defines a total step size from the two components.Source: S. RANGER21 · p. 15
  • Ranger21 adds Lookahead every k iterations and is described as expected to be more robust despite its more complex structure.Its listed defaults include k = 5 and βla = 0.5.Source: S. RANGER21 · pp. 15–16

T. ADAPTIVE NESTEROV MOMENTUM

Adan is designed for varied deep-learning networks and uses Nesterov momentum estimation to modify Adam’s moment estimates without extra computational cost. The section presents its update framework and default settings.

T. ADAPTIVE NESTEROV MOMENTUM

  • Nesterov momentum estimation uses current and previous gradients to estimate momentum direction without computing the gradient at an extrapolated point.Adan modifies Adam’s first and second moment estimates using this approach.Source: T. ADAPTIVE NESTEROV MOMENTUM · p. 16
  • The paper calls the original Adan formulation Vanilla Adan and describes it as defining the optimizer’s fundamental principles and theoretical framework.Source: T. ADAPTIVE NESTEROV MOMENTUM · p. 16
  • Experiments on ResNet, ViT, and GPT-2 demonstrate Adan’s consistent effectiveness; defaults are β1 = 0.02, β2 = 0.08, β3 = 0.01, and λ = 0.02.The learning rate can be set according to the network, and the defaults can be adjusted for fine-tuning.Source: T. ADAPTIVE NESTEROV MOMENTUM · p. 16

U. EVOLVED SIGN MOMENTUM

Lion was developed through genetic-algorithm program search as an alternative to manually designed and tuned optimizers. Its sign-based update avoids a second-moment estimate, while RLion addresses convergence problems reported for Lion in some models.

U. EVOLVED SIGN MOMENTUM

  • Lion’s genetic-algorithm development uses warm-start and restart, abstract-execution pruning, funnel selection, and program simplification.Source: U. EVOLVED SIGN MOMENTUM · p. 17
  • Lion uses the sign of momentum to control update size, needs no second-moment estimate, and applies weight decay similarly to AdamW.The simplified rule is reported to learn faster and more efficiently with less memory; Lion is used in Google’s ads clickthrough-rate model.Source: U. EVOLVED SIGN MOMENTUM · p. 17
  • Because Lion’s discrete sign function can cause convergence problems in some models, RLion uses an arctangent function and scaled momentum.Source: U. EVOLVED SIGN MOMENTUM · p. 17

V. SECOND-ORDER CLIPPED STOCHASTIC OPTIMIZATION

Sophia brings approximated second-order information into optimization using diagonal Hessian estimates and clipping. For language modeling with GPT models, the paper reports up to a 2x improvement over AdamW in total computational cost and training time.

V. SECOND-ORDER CLIPPED STOCHASTIC OPTIMIZATION

  • Sophia approximates Hessian information through diagonal approximation and Fisher estimation, combined with gradient clipping and a hybrid mini-batch strategy.It uses decoupled weight decay like AdamW; the paper describes the approach as aiming to accelerate convergence in transformer-based LLMs while reducing training time and energy consumption.Source: §V · p. 17
  • Sophia clips values between ±τc, with τc set to 1, and updates its exponential moving average of diagonal Hessian estimates every k steps, typically k = 10.The diagonal estimate can use Hutchinson’s unbiased estimator or the Gauss-Newton-Bartlett estimator; defaults include β1 = 0.96, β2 = 0.99, ε = 10^-12, δ = 0.01 or 0.05, and λ = 0.02.Source: §V · p. 17
  • For language modeling with GPT models, Sophia demonstrates up to a 2x improvement over AdamW in total computational cost and training time.Source: §V · p. 17

W. MOMENTUM ORTHOGONALIZED NEWTON-SCHULZ

Muon combines orthogonalized momentum for 2D hidden-layer weights with AdamW for other weights, while the review traces optimizer development and identifies continuing challenges in generalization, convergence, tuning, and resource use. It concludes that future optimizers must account for model scale and architecture alongside safe convergence and limited hardware.

W. MOMENTUM ORTHOGONALIZED NEWTON-SCHULZ

  • Muon orthogonalizes momentum for 2D hidden-layer weights but uses AdamW for scalars, vectors, and embedding-output-layer weights.Newton-Schulz iteration approximately orthogonalizes the update matrix, with the stated purpose of supporting learning in rare as well as dominant directions and making learning dynamics more stable.Source: W. MOMENTUM ORTHOGONALIZED NEWTON-SCHULZ · p. 18
  • On a 1.5-billion-parameter model, Muon training requires less time than AdamW, and the optimizer is reported to scale strongly to LLM training.Source: W. MOMENTUM ORTHOGONALIZED NEWTON-SCHULZ · p. 18

A. EVOLUTION OF OPTIMIZERS

  • Optimizer development progressed from SGD, Momentum, and NAG to adaptive rates, Adam and its variants, and newer methods targeting large-model compute and memory efficiency.The review notes Adagrad’s diminishing learning rate, Adam’s possible overfitting risk, and later approaches including decoupled weight decay, layerwise adaptation, and other techniques.Source: A. EVOLUTION OF OPTIMIZERS · pp. 18–19

B. OPEN CHALLENGES AND FUTURE DIRECTIONS

  • Open challenges include balancing generalization with stable, fast convergence, tuning hyperparameters for each problem, and reducing resource demands for large-scale models.The review points to more efficient gradient and second-order methods, meta-learning, distributed and large-batch training, automatic or hyperparameter-free tuning, and architecture-specific optimizers as future directions.Source: B. OPEN CHALLENGES AND FUTURE DIRECTIONS · pp. 19–20

IV. CONCLUSION

  • The conclusion calls for resource-efficient optimization that supports safe convergence while accounting for model architecture and scale, including training large models on limited hardware.Source: §IV · p. 20
Loading 2509.18396v1…