PaperlayerView exact revision on arXiv

Source-linked AI summary

Fantastic Pretraining Optimizers and Where to Find Them

Kaiyue Wen, David Hall, Tengyu Ma, Percy Liang

arXiv:2509.02046v2cs.LGcs.AIstat.ML

TL;DR

Optimizer comparisons have limited evidence from properly tuned baselines and broader model scales. This study benchmarks eleven optimizers in a controlled setup and finds that gains over well-tuned AdamW diminish with model size, reaching around 1.1× at 1.2B parameters.

  • Problem

    Optimizer comparisons have limited evidence from properly tuned baselines and broader, more realistic model scales, leaving performance in those settings unanswered.

    Source: §1 · p. 1

  • Method

    The authors benchmark eleven optimizers, including AdamW, in a rigorously controlled setup.

    Source: §1 · p. 2

  • Results

    Against well-tuned AdamW, alternative optimizers deliver at most 1.4× speedup, falling to around 1.1× for 1.2B models at 8× Chinchilla ratio.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Fair comparisons require optimizer-specific hyperparameter tuning and target-budget evaluations, since transferred settings and intermediate checkpoints can mislead.

    Source: §1 · p. 3

    The study does not evaluate models larger than 1.2B parameters.

    Source: §5 · p. 13

Abstract

from arXiv · show

AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have obscured fair comparisons and hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited or misleading evaluation setups. To address these two issues, we conduct a systematic study of ten deep learning optimizers across four model scales (0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum). We find that fair and informative comparisons require rigorous hyperparameter tuning and evaluations across a range of model scales and data-to-model ratios, performed at the end of training. First, optimal hyperparameters for one optimizer may be suboptimal for another, making blind hyperparameter transfer unfair. Second, the actual speedup of many proposed optimizers over well-tuned baselines is lower than claimed and decreases with model size to only 1.1x for 1.2B parameter models. Thirdly, comparing intermediate checkpoints before reaching the target training budgets can be misleading, as rankings between two optimizers can flip during training due to learning rate decay. Through our thorough investigation, we find that all the fastest optimizers such as Muon and Soap, use matrices as preconditioners -- multiplying gradients with matrices rather than entry-wise scalars. However, the speedup of matrix-based optimizers is inversely proportional to model scale, decreasing from 1.4x over AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.

1 Introduction

The study finds that optimizer comparisons depend on optimizer-specific hyperparameter tuning and evaluation at target training budgets across model scales and data-to-model ratios. Against tuned AdamW, matrix-based optimizers are fastest, but their advantage shrinks with model size and the best optimizer can change as the data-to-model ratio increases.

1 Introduction

  • Against tuned AdamW, alternatives top out at 1.4×; Muon and Soap fall from 1.3× on 0.1B models to around 1.1× on 1.2B models at 8× Chinchilla.Many reported 2× speedups reflect a weak baseline.Source: Figure 1 · p. 2, §1 · p. 3
  • For models under 520M parameters, matrix-based optimizers deliver approximately 1.3× speedup over AdamW, while tuned scalar-based optimizers average less than 1.2×.Matrix methods precondition gradients using matrix multiplication; scalar-based optimizers update each parameter individually using scalar operations.Source: Figure 1 · p. 2, §1 · p. 3
  • The benchmark tunes all eleven optimizers by coordinate descent across six settings for models up to 0.5B parameters, then tests four data-to-model ratios from 1× to 8× Chinchilla up to 1.2B parameters.It uses the Llama 2 architecture, an OLMo 2-like data mixture, final C4-EN validation loss, and token counts to reach a given loss as its primary comparison.Source: §1 · pp. 2–3
  • Lion’s optimal weight decay is approximately 0.6 versus AdamW’s approximately 0.1, showing that shared hyperparameters can make comparisons unfair.Tuning only the learning rate in the widely used GPT-3 recipe can yield a 2× speedup for pretraining a 100M model.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • During learning-rate decay, optimizer loss curves may cross multiple times, so intermediate checkpoints can rank optimizers differently from the target training budget.The study therefore emphasizes end-of-training comparisons.Source: §1 · p. 3
  • Muon is consistently best at smaller Chinchilla ratios, but Kron and Soap outperform it when the data-to-model ratio reaches 8× or larger.A winner in the 1× Chinchilla regime may be suboptimal as the data-to-model ratio increases.Source: §1 · p. 3

2 Related Works

The related work surveys optimizer-design families, pretraining-specific optimization and speedup claims, and evaluations that stress fair tuning and setting-dependent comparisons.

Optimizers for Deep Learning

  • Optimizer research spans adaptive Adam/AdamW variants, matrix-preconditioned methods such as Muon and Scion, Hessian-informed approaches, and discovered optimizers such as Lion.The survey also covers convergence, generalization, variance reduction, weight momentum, tuning, memory, and batch-size methods.Source: §2 · p. 4

Optimization for Pretraining

  • Pretraining studies use tensor programs and power-law fits to scale hyperparameters, while proposed optimizers claim 1.4×–3× speedups over AdamW.Examples include Sophia (2×), Soap (1.4×), Muon (2×), MARS (2×), and DION (3×) over AdamW.Source: §2 · p. 4

Re-evaluation Methodology

  • Prior evaluations find optimizer rankings are problem-specific and unequal tuning can explain most claimed speedups; the present study extends evaluation to LLM pretraining and modern optimizers.Other comparisons examine pretraining loss structure (Zhao et al. [2025]), optimizer performance across settings (Kasimbeg et al. [2025]), and Bert- and T5-like models, where rankings can flip during learning-rate decay and tuned AdamW remains competitive (Kaddour et al. [2023]).Source: §2 · pp. 4–5

Comparison with concurrent work Semenov et al. [2025]

  • Compared with Semenov et al. [2025], this work finds matrix-level optimizers such as Muon can outperform MARS, with the difference likely tied to batch size and tuning.Semenov et al. [2025] report AdEMaMix and MARS outperforming Muon; their tuned 130M experiments use 0.1M- or 0.02M-token batches, while this work uses batches of at least 0.4M tokens. At 520M scale, this work sweeps higher learning rates (4e-3 to 8e-3 versus 1e-3 to 2e-3) and separately tunes Muon’s embedding and matrix-weight learning rates.Source: §2 · p. 5

3 Methodology

The methodology combines multi-scale optimizer benchmarks with staged hyperparameter tuning, then extrapolates scaling-sensitive settings to larger models and higher data budgets.

3.1 General Experimental Setup

  • Benchmarks span approximately 130M–1.2B models and use English C4 loss as the primary metric, alongside downstream accuracy and bits-per-byte across ten benchmarks.Source: §3.1 · pp. 5–6
  • The study compares eleven optimizers chosen to include established baselines, recent proposals, and representative methods with similar update rules.Source: §3.1 · p. 6

3.2 Phase I: Fine-grained Hyperparameter Coordinate Descent

  • One-at-a-time grid sweeps across six size and data-budget regimes yielded coordinate-wise local optima for each optimizer.Sweeps continued until no hyperparameter update improved validation loss by more than Δ1 = 3 × 10−3.Source: §3.2 · p. 7

3.3 Phase II: Coordinate Descent on Scaling-Sensitive Hyperparameters

  • Phase II retains parameters that affect final loss and shift with scale, then tunes them in six additional settings to obtain near-optimal configurations for 12 regimes.Source: §3.3 · p. 8
  • Matrix-based optimizers consistently outperform scalar-based optimizers, but none exceeds 1.5× speedup over AdamW; Muon leads at 1–4× Chinchilla before Soap and Kron overtake it at higher ratios.Source: §3.3 · p. 9

3.4 Phase III: Hyperparameter Scaling Law for Further Extrapolation

  • At 1.2B parameters, Muon’s speedup falls below 1.2×; at 16× Chinchilla, NAdamW and Soap beat Muon at 130M, and Soap does so at 300M.A scaling law fitted to 12 observed settings predicted AdamW hyperparameters at 1.2B within 3e-3 of the full-sweep optimum.Source: §3.4 · p. 9

4 Empirical Findings

Across the tested scales and data-to-model ratios, optimizer rankings and speedups depend on model size, data budget, and careful hyperparameter tuning. The findings also show why evaluations should use final checkpoints and report shared norm and generalization patterns across optimizers.

4.1 Main Results

  • Matrix-based optimizers beat scalar-based counterparts, with speedups rising as data budgets grow; Muon leads at 1× Chinchilla, while Soap and Kron lead at 8×.At 16× Chinchilla, Soap also outperforms Muon; the authors conjecture that Soap and Kron’s second-order momentum helps at higher data-to-model ratios.Source: §4.1 · pp. 9, 11, Figure 3 · p. 10
  • Across 0.1B–0.5B models, optimizer speedups peak at 1.4× over AdamW; none reaches the previously claimed 2× step-wise acceleration.The estimated speedup uses a scaling-law comparison against AdamW.Source: §4.1 · p. 9, Figure 3 · p. 10
  • NAdamW, Mars, and Cautious generally outperform AdamW except in the smallest experiment; Lion and Adam-mini track AdamW within at most 5% slowdown.Muon combines matrix-based updates with Nesterov momentum; as model size grows, Lion’s disadvantage widens while Adam-mini’s narrows.Source: §4.1 · p. 10
  • Speedups shrink with scale: gains are greatest at 130M, about 1.3× for larger models, and ≈1.1× for NAdamW, Muon, and Soap at 1.2B, without downstream gains.The paper’s scaling-law fit predicts Muon may have slightly higher loss than AdamW at 7B and 1× Chinchilla.Source: §4.1 · pp. 10–11, Figure 4 · p. 11, Table 5 · p. 11

4.2 Necessity of Rigorous Benchmarking

  • Hyperparameter mis-tuning can reverse optimizer rankings, and optimal weight decay differs by optimizer; Kron needs approximately 0.5 to beat AdamW.Exhaustive sweeps found that deviations from an optimizer’s ideal learning rate or other critical hyperparameter can worsen validation loss enough to flip rankings.Source: §4.2 · pp. 11–12, Figure 5 · p. 12
  • Early losses can exaggerate or reverse final rankings, so evaluate final checkpoints; Sophia Liu et al. [2024a] showed no significant speedup below 0.5B in this setting.The cited evaluation used a data loader that did not fully randomize data order, changing AdamW’s optimal peak learning rate relative to a fully randomized setting.Source: §4.2 · p. 12

4.3 Common Phenomena Across Optimizers

  • With weight decay, parameter norms rise and fall alongside the learning-rate schedule, while gradient norms increase during decay without a corresponding loss increase.Absolute parameter and gradient norm values differ across optimizers.Source: §4.3 · p. 13, Figure 6 · p. 13
  • Across the tested regimes, training and evaluation losses follow roughly the same trend for all optimizers.The paper contrasts this shared optimizer behavior with reported differences in generalization behavior across architectures.Source: §4.3 · p. 13, Figure 6 · p. 13

5 Conclusion and Limitations · A Optimizer Definitions

The conclusion finds that carefully tuned optimizers offer smaller and less scale-stable gains over AdamW than previously reported, while early evaluations can mislead. The appendix documents the evaluated optimizers’ algorithms, notation, and implementation options.

5 Conclusion and Limitations

  • 1.1× at 1.2B parameters is the reported speedup of the best-performing alternatives, whose gains diminish with model scale.The study benchmarks 11 optimizers and notes that it does not scale beyond 1.2B parameters; future work would test whether the diminishing trend persists at larger scales.Source: §5 · p. 13
  • Fair hyperparameter sweeps eliminate most apparent optimizer advantages, which the authors attribute to insufficient tuning in prior comparisons.Hyperparameters that work well for one optimizer may not be suitable for another, making blind transfer unfair.Source: §5 · p. 13
  • Optimizer rankings can change over the full training trajectory, so early or inconsistent evaluations can mislead.The conclusion identifies changing rankings during training as a reason intermediate evaluations may not reflect final comparisons.Source: §5 · p. 13

A Optimizer Definitions

  • The optimizer appendix presents algorithms for the evaluated methods and defines shared notation for parameters, gradients, learning rate, decay, moments, and element-wise operations.The stated notation includes β1, β2, ϵ, η, λ, gradient norm, and first and second moments.Source: A Optimizer Definitions · p. 23
  • Algorithm 2 specifies Nesterov AdamW using moment state, learning rate, weight decay, and bias-corrected moments in its update rule.Its listed hyperparameters include β1, β2, ϵ, η, λ, and gradient norm.Source: A Optimizer Definitions · p. 23
  • One optimizer definition exposes options for gradient normalization, block partitioning, merging small dimensions, and probabilistic preconditioner updates.The setup also specifies a target merged dimension size, block size, and per-block state.Source: A Optimizer Definitions · p. 28

B Omitted Experiments · B.1 Scaling Law

The fitted scaling law predicts that at 7B parameters, Muon has higher loss than AdamW in the 1× Chinchilla regime. The section describes fitting and hyperparameter selection for the scaling-law analysis.

B.1 Scaling Law

  • For the 1.2B runs, scaling-law hyperparameters for Muon, NAdamW, and AdamW were fitted, rounded to the hyperparameter grid, and used for training.After training, the scaling law was fitted for Muon and AdamW.Source: §B.1 · p. 29
  • The fitted parameters were AdamW α = 21.4289, A = 0.1555, β = 276.4235, B = 0.2804, γ = 1.7324, RMS error 3 × 10−3; Muon α = 32.7458, A = 0.1864, β = 59.0221, B = 0.2074, γ = 1.8063, RMS error 5 × 10−3.Source: §B.1 · p. 29
  • At 7B parameters, Muon is predicted to have higher loss than AdamW in the 1× Chinchilla regime.The scaling law was fitted for Muon and AdamW after training.Source: §B.1 · p. 29

B.2 Sophia Experiments

In Phase I experiments, Sophia underperformed AdamW in smaller compute regimes but slightly outperformed it as model size or data size increased.

B.2 Sophia Experiments

  • Sophia tends to underperform AdamW in smaller compute regimes.Hyperparameter settings are reported in Appendix C.Source: §B.2 · p. 29
  • Sophia eventually slightly outperforms AdamW when either model size or data size increases.Source: §B.2 · p. 29

B.3 High Data-to-model Ratio

In the 130M, 16× Chinchilla setting, SOAP outperforms Muon under overtraining.

B.3 High Data-to-model Ratio

  • In the 130M, 16× Chinchilla setting, SOAP outperforms Muon under overtraining.Figure 8 presents this case study.Source: Figure 8 · p. 30

B.4 Evaluation Performance · C Hyperparameter Ablation in Phase I

Evaluation Performance presents optimizer-specific tables at three model sizes, while the Phase I ablation section explains how configurations and final validation loss are reported.

B.4 Evaluation Performance

C Hyperparameter Ablation in Phase I

C.1 Sweeping Results for AdamW … D Hyperparameter Ablation in Phase II

Appendix C reports hyperparameter-ablation tables for the listed optimizers, while Section D says Phase II results are formulated in the same way as Phase I.

C.1 Sweeping Results for AdamW; C.2 Sweeping Results for Cautious

  • AdamW and Cautious each have hyperparameter-ablation tables for 130m models on 1x Chinchilla data.Additional tables cover other data-to-model ratios and model sizes for these optimizers.Source: Table 36 · p. 41, Table 42 · p. 44

C.3 Sweeping Results for Lion; C.4 Sweeping Results for Mars

  • Lion and Mars each have hyperparameter-ablation tables for 130m models on 1x Chinchilla data.Additional tables cover other data-to-model ratios and model sizes for these optimizers.Source: Table 47 · p. 47, Table 53 · p. 53

C.5 Sweeping Results for NAdamW; C.6 Sweeping Results for Adam-Mini

  • NAdamW and Adam-Mini each have hyperparameter-ablation tables for 130m models on 1x Chinchilla data.Additional tables cover other data-to-model ratios and model sizes for these optimizers.Source: Table 59 · p. 59, Table 64 · p. 61

C.7 Sweeping Results for Kron; C.8 Sweeping Results for Soap

  • Kron and Soap each have hyperparameter-ablation tables for 130m models on 1x Chinchilla data.Additional tables cover other data-to-model ratios and model sizes for these optimizers.Source: Table 70 · p. 65, Table 76 · p. 71

C.9 Sweeping Results for Muon; C.10 Sweeping Results for Scion

  • Muon and Scion each have hyperparameter-ablation tables for 130m models on 1x Chinchilla data.Additional tables cover other data-to-model ratios and model sizes for these optimizers.Source: Table 82 · p. 76, Table 88 · p. 82

C.11 Sweeping Results for Sophia

D Hyperparameter Ablation in Phase II

D.1 Sweeping Results for AdamW … E Hyperparameter Ablation in Phase III 1.2B experiments

Appendix D reports optimizer-specific hyperparameter-ablation tables across model sizes and data budgets; the Phase III 1.2B results are formulated as in Phase I.

D.2 Sweeping Results for Cautious

  • The cited table captions identify Cautious hyperparameter ablations for 300m models on 2x and 4x Chinchilla data, specifying these two model-scale and data-multiplier conditions.Source: Table 106 · p. 93, Table 107 · p. 93

D.3 Sweeping Results for Lion

  • The cited table captions cover Lion hyperparameter ablations for 300m models at 2x and 4x Chinchilla Data, but do not report ablation outcomes or winning hyperparameters.Source: Table 112 · p. 94, Table 113 · p. 94

D.4 Sweeping Results for Mars

  • The cited Mars hyperparameter-ablation tables cover 300m models trained on 2x and 4x Chinchilla data, respectively, as indicated by the Table 118 and Table 119 captions.Source: Table 118 · p. 95, Table 119 · p. 95

D.5 Sweeping Results for NAdamW

  • The cited NAdamW ablations cover 300m models on 2x and 4x Chinchilla data; the passages provide table scope only, not ablation results or winning hyperparameters.Source: Table 124 · p. 97, Table 125 · p. 97

D.6 Sweeping Results for Adam-Mini

  • Adam-Mini hyperparameter ablations are reported for 300m models trained on 2x and 4x Chinchilla data, as indicated by the captions of Tables 130 and 131.Source: Table 130 · p. 98, Table 131 · p. 98

D.7 Sweeping Results for Muon; D.8 Sweeping Results for Scion; D.9 Sweeping Results for Kron; D.10 Sweeping Results for Soap

E Hyperparameter Ablation in Phase III 1.2B experiments

E.1 Sweeping Results for AdamW … F Hyperparameter Ablation in Phase III 16x Chinchilla experiments

The supplied passages identify hyperparameter-ablation tables for AdamW, NAdamW, Soap, and Muon on 1.2B models, covering 1x, 2x, 4x, and 8x Chinchilla data; they do not provide ablation outcomes.

E.1 Sweeping Results for AdamW

  • AdamW’s 1.2B-model hyperparameter ablation is tabulated at 1x Chinchilla data.The supplied caption identifies Table 160; it gives no ablation results.Source: Table 160 · p. 104
  • AdamW’s 1.2B-model ablations also cover 2x, 4x, and 8x Chinchilla data.These are identified as Tables 161–163; their captions provide no outcomes.Source: Table 161 · p. 104, Table 162 · p. 104, Table 163 · p. 104

E.2 Sweeping Results for NAdamW

  • NAdamW’s 1.2B-model hyperparameter ablation is tabulated at 1x Chinchilla data.The supplied caption identifies Table 164; it gives no ablation results.Source: Table 164 · p. 105
  • NAdamW’s 1.2B-model ablations also cover 2x, 4x, and 8x Chinchilla data.These are identified as Tables 165–167; their captions provide no outcomes.Source: Table 165 · p. 105, Table 166 · p. 105, Table 167 · p. 105

E.3 Sweeping Results for Soap

  • Soap’s 1.2B-model hyperparameter ablation is tabulated at 1x Chinchilla data.The supplied caption identifies Table 168; it gives no ablation results.Source: Table 168 · p. 105
  • Soap’s 1.2B-model ablations also cover 2x, 4x, and 8x Chinchilla data.These are identified as Tables 169–171; their captions provide no outcomes.Source: Table 169 · p. 105, Table 170 · p. 105, Table 171 · p. 106

E.4 Sweeping Results for Muon

  • Muon’s 1.2B-model hyperparameter ablation is tabulated at 1x Chinchilla data.The supplied caption identifies Table 172; it gives no ablation results.Source: Table 172 · p. 106
  • Muon’s 1.2B-model ablations also cover 2x, 4x, and 8x Chinchilla data.These are identified as Tables 173–175; their captions provide no outcomes.Source: Table 173 · p. 106, Table 174 · p. 106, Table 175 · p. 106

F.1 Sweeping Results for AdamW … G Comparison with Prior Work

The appendix provides optimizer-specific hyperparameter sweep tables, while the comparison with prior work links reported speedups to differences in baselines, training setups, and evaluation conditions.

F.1 Sweeping Results for AdamW

  • AdamW sweep tables cover 130m and 300m models trained on 16x Chinchilla data.The provided captions identify hyperparameter ablations but do not include sweep outcomes.Source: Table 176 · p. 106, Table 177 · p. 107

F.2 Sweeping Results for NAdamW

  • NAdamW sweep tables cover 130m and 300m models trained on 16x Chinchilla data.The provided captions identify hyperparameter ablations but do not include sweep outcomes.Source: Table 178 · p. 107, Table 179 · p. 107

F.3 Sweeping Results for Soap

  • Soap sweep tables cover 130m and 300m models trained on 16x Chinchilla data.The provided captions identify hyperparameter ablations but do not include sweep outcomes.Source: Table 180 · p. 107, Table 181 · p. 107

F.4 Sweeping Results for Muon

  • Muon sweep tables cover 130m and 300m models trained on 16x Chinchilla data.The provided captions identify hyperparameter ablations but do not include sweep outcomes.Source: Table 182 · p. 107, Table 183 · p. 107

G Comparison with Prior Work

  • Sophia, MARS, Cautious, Block-wise Learning Rate, and FOCUS report 2× speedups, with Sophia’s setup using a peak learning rate below 6e-4.The paper notes that higher AdamW learning rates can improve the baseline and that random data shuffling matters for stable use of a large learning rate.Source: G Comparison with Prior Work · p. 108
  • Soap achieved 1.2 to 1.3× speedup on 300M and 520M models versus its claimed 1.4×; SPlus reported 2× with a constant learning rate.The paper notes its Soap implementation uses weight blocking and bfloat16 momentum in 1.2B experiments, which may lower step-wise performance.Source: G Comparison with Prior Work · p. 108
  • Muon reports range from 1.3× in the original Nanogpt speedrun to 2× in Kimi, while independent work finds 1.1 to 1.2× token efficiency over AdamW.The paper attributes Kimi’s comparison partly to its low AdamW learning rate and notes its MoE comparison used not-fully-decayed learning rates; SWAN and DION report 2-3× below 1× Chinchilla.Source: G Comparison with Prior Work · p. 108

Related papers

Loading 2509.02046v2…