Benchmarking Optimizers for Large Language Model Pretraining

1 Introduction

• Prior optimizer benchmarks and ablations often examine isolated design aspects, limiting trustworthy guidance for practitioners and research directions.Recent alternatives claim advantages in speed, communication efficiency, or downstream performance, but comparisons vary in scope.Source: §1 · p. 1
• The study varies batch size, model size, and training iterations in standardized experiments, tuning methods and providing best practices across optimizer choices.The stated goal is to identify well-performing methods for researchers and practitioners.Source: §1 · p. 1
• At 720M parameters, AdEMAMix and MARS are the best-performing optimizers in the FineWeb setup, with a noticeable gap over other methods.The comparison uses a 1M-token batch and includes training durations below and above the 14.4B-token Chinchilla optimum.Source: Figure 1 · p. 2
• The benchmark covers 11 optimization methods across diverse LLM scenarios, with compute accounting and extensive tuning for dense and MoE architectures.The work also reports hyperparameter ablations and releases training scripts, evaluation pipelines, and configurations for reproducible research.Source: §1 · p. 2

Optimizers

• Modern alternatives build on AdamW through modified moments, sign-based updates, approximate second-order structure, parameter-free scheduling, or variance reduction.ADOPT adjusts second-moment normalization; AdEMAMix adds a slower momentum buffer; Signum and Lion use sign-based structures; Sophia, SOAP, Muon, and Scion exploit second-order or matrix structure; SF-AdamW, Prodigy, and MARS extend parameter-free approaches.Source: §2 · p. 3

Benchmarks

• Optimizer comparisons can change with training horizon: strong short speedruns may not identify the best method for longer LLM training.The cited discussion connects training-horizon and compute-budget questions to scaling laws and notes that roughly 20 tokens per parameter may be insufficient for production-ready models.Source: §2 · pp. 3–4
• Some prior pretraining and benchmark setups add z-loss regularization to cross-entropy, a loss-function choice the paper says needs further investigation.The authors state that their work also addresses the choice of loss function.Source: §2 · p. 4
• Prior optimizer comparisons may not follow recent practices for weight decay, learning-rate decay, and hyperparameter tuning, concerns revisited in this work.Source: §2 · p. 4

Optimizers

• The benchmark tunes 11 optimizer families and variants, including momentum choices, Newton–Schulz steps, learning rates, clipping, warmup, weight decay, initialization, and schedulers.The listed methods include AdamW, ADOPT, AdEMAMix, Lion, Signum, Muon, D-Muon, SOAP, Sophia, SF-AdamW, Prodigy, and MARS.Source: §3 · p. 4
• Across dense-model sizes, D-Muon with weight decay applied to all parameter groups improves over basic Muon.Source: Figure 4 · p. 4

Models & Data

• Experiments use Llama-like models from 124M to 720M parameters, plus a 520M MoE model, trained on a 100B-token FineWeb subset.The corpus is cleaned and deduplicated, and tokenized with GPT-2 before splitting into training and validation sequences.Source: §3 · p. 4

Iterations & Batch size

• Batch sizes and iteration counts vary by model, with training durations covering Chinchilla-optimal lengths and longer runs.For 124M and 210M models, durations range from 1B to 16.8B tokens; the 583M setup uses 13B tokens, while 720M runs use 1M-token batches.Source: §3 · pp. 4–5

Loss

• Models use classical cross-entropy next-token prediction without z-loss, which the authors found had little impact.An ablation comparing results with and without z-loss is reported in § 4.Source: §3 · p. 5

Hyperparameter Tuning

• For each model size, batch size, and optimizer, hyperparameters are tuned near Chinchilla-optimal token budgets, then generally held fixed for longer runs.The study compares cosine, linear, and warmup-stable-decay schedules, usually uses cosine, decays learning rate to 0.01 × γmax, and reports 2,900 trained models and roughly 30,000 GPU hours.Source: §3 · p. 5

4.1 Benchmarking & Ablations at Small Scale: Training Models of 124M Parameters

• At 124M, AdEMAMix leads as training duration increases; Signum, Lion, and MARS benefit from larger batches, while SOAP behaves similarly across batch settings.At small batches, D-Muon is strongest among methods at the shortest duration, while Prodigy, basic Muon, and SF-AdamW underperform AdamW; at larger batches, AdEMAMix leads at longer durations.Source: §4.1 · pp. 5–6, Figure 5 · p. 6
• After retuning for 16.8B tokens, optimizer rankings remain unchanged when training is doubled to 33.6B tokens, while Sophia diverges shortly after 130k steps.For longer horizons, AdamW performs better with β2 = 0.999 than β2 = 0.95, and AdEMAMix's β3 is increased from 0.999 to 0.9999; SOAP and D-Muon narrow their gap with AdEMAMix at 16.8B tokens.Source: §4.1 · pp. 6–7, Figure 6 · p. 7
• Weight decay substantially affects optimizer behavior: 0.5 benefits shorter runs, while 0.1 is robust for longer runs and zero weight decay is generally worse.Signum and Lion with weight decay of 0.5 outperform AdamW with weight decay of 0.1 in most reported durations; applying weight decay across parameters also improves D-Muon over basic Muon.Source: §4.1 · pp. 7–8, Figure 7 · p. 8, Figure 8 · p. 8
• Learning-rate, warmup, and scheduler preferences vary by optimizer, so these settings should be tuned rather than assumed to transfer uniformly.The learning rate tuned near the Chinchilla-optimal horizon transfers for most optimizers, but sign-based methods and Sophia diverge at γmax = 2e−3; MARS remains consistent across the sweep. Longer warmup benefits SF-AdamW, Sophia, and Signum, while Lion with increased warmup surpasses AdamW; cosine leads for most methods, linear matches or beats it for sign-based methods, SOAP, and MARS, and WSD is best for Muon.Source: §4.1 · pp. 9–11

4.2 Benchmarking & Ablations at Medium Scale: Training Models of 210M Parameters

• At 210M, optimizer rankings remain much like those at 124M, but sign-based methods require learning-rate retuning; cosine and WSD work best with γend = 0.01 × γmax, while linear works best with γend = 0.001 × γmax.Decaying only to 0.1 × γmax performs worse than the lower final learning rates considered.Source: §4.2 · pp. 11–12, Figure 15 · p. 12

4.3 Scaling Up: Benchmarking models of 583M and 720M Parameters

• At 583M, choices such as final learning rate, z-loss, and weight decay affect both absolute performance and optimizer rankings.In the AdamW and SOAP comparison, removing z-loss and increasing weight decay to 0.1 improves results.Source: §4.3 · p. 12, Figure 16 · p. 12
• SOAP beats AdamW at the Chinchilla-optimal duration and below, but AdamW narrows the gap and eventually surpasses it during longer training.The SOAP advantage at large batches is setup-dependent: with the 2M-token setup, proper tuning makes AdamW better even at the Chinchilla-optimal duration, and at 1M tokens SOAP's gain is less pronounced than at smaller model scales.Source: Figure 16 · p. 12, §4.3 · p. 13, Figure 17 · p. 13
• At 720M parameters with 1M-token batches, AdEMAMix and MARS dominate; MARS ranks second, while Signum and Lion scale poorly and D-Muon remains consistent.SF-AdamW also scales poorly, and Sophia degrades after outperforming AdamW on short runs. The ranking of SOAP versus AdamW changes with training duration and setup.Source: Figure 17 · p. 13, §4.3 · p. 13
• Across 30M–1B models, most optimizers took similar time per 100 iterations, sign-based methods were slightly faster, and SOAP slowed notably as model size rose.The timing ablation used one GPU, batch size 16, 100 iterations, and five seeds, without gradient accumulation or torch.compile.Source: §4.3 · pp. 13–14, Figure 18 · p. 13

4.4 Extension to MoEs

• At 336k MoE steps, SOAP outperformed AdEMAMix, though retuned beta parameters might reverse the ordering in longer training.Source: Figure 19 · p. 14
• For a 44B-token MoE training horizon, AdamW by far outperformed Sophia, SF-AdamW, MARS, and sign-based methods, while Prodigy followed AdamW as in the dense-model comparison.The figure caption says results closely resemble the dense-model results.Source: Figure 20 · p. 14
• Optimizer rankings on 520M MoEs closely mirror dense-model rankings, indicating that dense-model benchmarking results transfer to corresponding MoEs.The MoE tests used an 8-expert Switch-Transformer variant and retained untuned optimizer hyperparameters from the corresponding dense models; training used a 256 × 512 batch for 42k or 336k iterations.Source: §4.4 · p. 14, Figure 19 · p. 14

5 Discussion

• AdEMAMix is the best optimizer overall; D-Muon and MARS are reliable at scale, while ADOPT and Prodigy scale similarly to AdamW.The discussion also reports that MARS benefits from larger batches and model scales, and D-Muon addresses basic Muon’s small-batch weakness.Source: §5 · p. 15
• For shorter runs, larger weight decay can speed convergence; a 25%-of-training warmup benefits sign-based methods, Sophia, and SF-AdamW.The discussion also recommends decaying the learning rate below 0.1 × γmax and notes optimizer-specific scheduler preferences.Source: §5 · p. 15
• The benchmark covers models up to 720M parameters and almost 50B tokens, evaluates loss dynamics rather than downstream performance, and omits several optimizer families.The authors note that behavior may change at practical scales and longer durations, and call for further work connecting loss minimization with downstream performance.Source: §5 · p. 15

A Optimizers we study

• The surveyed methods are grouped as Adam-like, sign-based, approximate second-order, schedule-free, and MARS optimizers.The groups include AdamW, ADOPT, AdEMAMix, Lion, Signum, Muon, SOAP, Sophia, Schedule-Free AdamW, and Prodigy.Source: A Optimizers we study · p. 26
• The notation defines |ξ| as batch size, γ as learning rate, λ as weight decay, β as momentum, and m and v as first- and second-moment estimates.The algorithms use t and T for the iteration counter and total iterations, respectively.Source: A Optimizers we study · p. 26

AdamW

• The benchmark uses AdamW as its baseline and applies decoupled weight decay to all methods for consistent comparisons.The authors identify correct weight decay as important for optimizer comparison, tuning, and final performance.Source: §A.1 · p. 26

ADOPT

• ADOPT removes the current gradient from its second-moment estimate and updates parameters using the previous estimate; practical clipping addresses early instability from near-zero gradients.The clamp operation preserves the gradient from division by near-zero values.Source: §A.1 · pp. 26–27
• ADOPT’s practical performance varies significantly with β2, despite the claimed convergence bound being independent of β2; ε = 10−6 is more suitable than AdamW’s typical ε = 10−8.The authors report that the smaller ε value also holds in their experiments.Source: §A.1 · p. 27

AdEMAMix

• AdEMAMix combines fast and slow EMAs to retain older gradients, and its α and β3 schedulers mitigate early instability.Settings Tα and Tβ3 longer than the first WSD checkpoint do not noticeably harm performance, so the method can be combined with WSD.Source: §A.1 · p. 27

Sign-based methods

• With sufficiently large batches at moderate model scales, Signum and other sign-based methods perform on par with Adam and sometimes outperform it.Signum, MARS, Lion, and Prodigy greatly benefit from increased batch size, according to the cited discussion of larger batches.Source: §A.2 · p. 28
• Lion interpolates the prior EMA and current gradient for its sign update, whereas Signum uses a current EMA and a different interpolation term.Lion’s gradient interpolation is described as similar in idea to AdEMAMix.Source: §A.2 · p. 28

Signum implementation

• Many PyTorch Signum implementations may impair performance; decoupled weight decay can alter the update’s sign and potentially send optimization in the wrong direction.This risk occurs if weight decay is sufficiently large; the authors emphasize correct weight-decay implementation for sign-based methods.Source: §A.2 · pp. 28–29
• The authors propose a PyTorch-compatible Signum variant after finding that Nesterov momentum improves Signum.They study dampening and Nesterov momentum in the variant.Source: §A.2 · p. 29

A.3 Muon & D-Muon, SOAP, Sophia

• Muon uses AdamW for 1D parameters, embeddings, normalization parameters, and the output layer, while MuonNon1D handles parameters with two or more dimensions.MuonNon1D uses Nesterov momentum followed by Newton-Schulz orthogonalization of the gradient matrix.Source: §A.3 · p. 31
• For basic Muon, the weight-decay gap between 0.1 and 0 nearly disappears, while 0.5 gives the worst result.In the described implementation, weight decay does not apply to MuonNon1D matrix parameters; AdamW supplies the only weight decay during training.Source: §A.3 · p. 31
• D-Muon applies an RMS heuristic to share learning rate and weight decay between matrix and non-matrix parameters.The heuristic scales Muon's RMS update to the AdamW range, avoiding separate tuning for 1D and non-1D parameters.Source: §A.3 · p. 32
• D-Muon scales better than basic Muon in all settings the authors considered.The authors compare the methods in multiple figures and discuss their connection to weight decay on non-1D parameters.Source: §A.3 · p. 32
• SOAP runs AdamW on 1D layers and maintains second-moment statistics for 2D-layer gradients in a slowly changing eigenvector basis.It periodically updates that basis using QR decomposition.Source: §A.3 · p. 32
• The study uses Sophia's Gauss-Newton-Bartlett preconditioner and reports that its actual update rule differs from the original paper's description.The authors say the original rule should be corrected to match line 16 of Algorithm 11.Source: §A.3 · pp. 32–33

A.4 Schedule-Free AdamW, Prodigy

• Schedule-Free AdamW replaces learning-rate schedulers with iterate averaging and uses internal warmup to stabilize scalable training.Its performance is sensitive to β parameters, and the study finds that it performs best with more warmup iterations than other methods.Source: §A.4 · p. 34
• The authors find that Schedule-Free AdamW's β2 choice may depend on training length, so the method may not be fully schedule-free.They also report that, contrary to the cited authors' guidance, disabling gradient clipping is not crucial in their runs; they find the opposite effect.Source: §A.4 · p. 34
• Prodigy scales reliably, does not require extensive learning-rate tuning, and remains compatible with learning-rate schedules.The authors typically initialize its learning rate at 1 and report that it remains stable in their sweeps.Source: §A.4 · p. 35

A.5 MARS

• MARS combines adaptive or approximate second-order methods with a variance-reduction update, whose gradient-correction scale is controlled by η.For LLM training, the authors use MARS-like methods on 2D parameters and AdamW on 1D parameters, embeddings, scalar parameters, and the final layer.Source: §A.5 · pp. 35–36
• The study includes MARS-AdamW as its main MARS baseline and also reports MARS-Lion and MARS-Shampoo.MARS-Lion uses sign-based updates without second-moment estimates for 2D parameters; MARS-Shampoo uses five Newton-Schulz orthogonalization iterations in these experiments.Source: §A.5 · p. 36

B Implementation

• The code extends nanoGPT and uses PyTorch and FlashAttention.Source: B Implementation · p. 37
• Training uses bfloat16 except for normalization modules and softmax, which use float32; optimizer states are stored in float32.Source: B Implementation · p. 37
• Experiments used A100, H100, and GH200 GPUs in single-GPU and 2-to-8-GPU DDP runs.The project reports roughly 30,000 GPU hours and 2,900 trained models.Source: B Implementation · p. 37

C Model & Data

• The models use a decoder-only Llama-like architecture with SwiGLU, RoPE, RMSNorm, a GPT2 tokenizer, and weight tying.The vocabulary contains 50,304 tokens.Source: C Model & Data · p. 37
• The model parameter count is configurable, and the paper lists the experiment configurations in Table 1.Source: C Model & Data · p. 38, Table 1 · p. 38
• The main findings use a cleaned, deduplicated 100B-token FineWeb subset, with final validation loss and perplexity computed after training.During training, validation loss curves use 32 fixed batches; initial results were also run on OpenWebText2.Source: C Model & Data · p. 38

• The additional experiments use 124M and 210M models.Source: D Additional Results · p. 38
• Ablations cover warmup, weight decay, learning-rate schedulers and decay, gradient norms, and other optimizer-related phenomena.Source: D Additional Results · p. 38
• The section concludes with optimizer wall-clock performance, with hyperparameter-search details provided in Appendix E.Source: D Additional Results · p. 38

Scaling the number of iterations

• Retuning β2 for SOAP and β3 for AdEMAMix preserves AdEMAMix’s lead at 33.6B tokens, while without retuning SOAP outperforms it.For the longer run, the authors use β2 = 0.9999 for SOAP and β3 = 0.9999 for AdEMAMix; they recommend retuning Adam-like methods’ β2 when changing training duration.Source: §D.1 · p. 39, Figure 21 · p. 39

Warmup ablation

• For most optimizers, 2000 warmup steps are more suitable than 25% of the training duration; exceptions include Signum, Lion, Sophia, and SF-AdamW.The study contrasts this result with prior work extrapolating from 85M models.Source: §D.1 · p. 39

Weight decay ablation

• Large weight decay performs better in short runs, while λ = 0.1 begins to outperform it at 8.4B and 16.8B tokens, with Muon an exception.Muon’s λ = 0.5 runs underperform λ ∈{0, 0.1} even on short runs; D-Muon instead shows a pattern similar to other methods.Source: §D.1 · p. 40, Figure 23 · p. 41

Learning rate sensitivity and schedulers

• On OWT2, cosine leads the tested schedulers; on FineWeb, tuned cosine performs similarly to linear, with both usually slightly outperforming WSD.For the learning-rate sweep, only SOAP, SF-AdamW, and D-Muon improve with γmax = 0.002; Sophia and sign-based methods diverge at that value, while MARS and Prodigy are consistent across the sweep.Source: §D.1 · pp. 40–42, Figure 25 · p. 42, Figure 24 · p. 42

Gradient norm patterns

• WSD produces flatter gradient-norm trajectories than cosine and linear; larger weight decay often increases norms, while smaller learning rates typically yield larger norms except for Signum and Lion.Weight-decay effects on gradient norms vary by optimizer, and large decay can increase norms late in training.Source: Figure 27 · p. 44, §D.1 · p. 45, Figure 28 · p. 45

Learning rate decaying for 124M model

• For 124M models, decaying the learning rate only to 10% of its maximum gives significantly worse final performance, and the optimal γend varies with model size.For 124M models, WSD performs best at γend = 10^-6 × γmax, versus 10^-2 × γmax for 210M models; the ablation tests AdamW only.Source: §D.1 · p. 46, Figure 30 · p. 47

Fail of Sophia

• Sophia diverges beyond 2.1B tokens in the small-batch setting and after 16.8B tokens in extended 256 × 512-batch training.In the latter setting, divergence begins at t = 129720 during a 33.6B-token run; the authors caution against relying on Sophia in its current form.Source: §D.1 · pp. 47–48, Figure 31 · p. 47, Figure 32 · p. 48

Clipping & SF-AdamW

• Disabling gradient clipping destabilizes SF-AdamW, and even after lowering the learning rate to 0.0005, the best clipped run with threshold 0.5 performs better.This finding contrasts with the recommendation to disable clipping in Defazio et al..Source: §D.1 · p. 48

Betas sensitivity

• Prodigy can diverge when β2 changes from 0.999 to 0.9999 at small batch size; bias correction resolves this instability, while SF-AdamW and ADOPT also require careful beta tuning.For SF-AdamW, (β1 = 0.9, β2 = 0.9999) performs best among the reported settings; ADOPT’s practical performance varies substantially despite convergence guarantees for any β2.Source: Figure 35 · p. 49, §D.1 · p. 49, Figure 34 · p. 49

Muon’s Newton-Schulz iterations

• With 124M models trained for 16k steps, Muon’s TNS = 5, 10, and 20 perform comparably, while TNS = 1 significantly degrades performance.TNS = 5 slightly outperforms the other tested settings; increasing TNS beyond 5 does not improve performance and increases wall-clock time.Source: Figure 37 · p. 50, §D.1 · p. 50

Signum types

• For Signum, Nesterov momentum outperforms dampening and plain updates, with the reported PPL margin narrowing as training duration increases.The margins are ∼0.45 PPL after 2.1B tokens and ∼0.11 PPL after 16.8B tokens.Source: Figure 38 · p. 51, §D.1 · p. 51

MARS types

• Among the tested MARS variants, MARS-AdamW performs best; MARS-Lion favors WSD over cosine and longer warmup, while neither MARS-AdamW nor MARS-Lion beats AdamW at batch size 256 × 512.Both MARS-AdamW and MARS-Lion benefit from larger batch size, and MARS-Shampoo performs worst among the variants.Source: §D.1 · pp. 51–52, Figure 39 · p. 52

On learning rates of Prodigy

• Prodigy’s effective learning rate has a longer warmup than the explicit setting and, with cosine scheduling, reaches ∼1.08× the AdamW rate of 0.001.The authors suggest tuning Prodigy betas with γ = 1, tracking γeff max, and using that peak as a learning-rate proxy for Adam-like optimizers.Source: §D.1 · pp. 53–54, Figure 40 · p. 53

ADOPT ε sensitivity

• ADOPT’s suggested ε = 10^-6 is reported as best; ε values from 10^-6 to 10^-10 show no noticeable convergence difference, while 10^-5 and above perform worse.The sweep holds other hyperparameters at their tuned values.Source: Figure 41 · p. 54, §D.2 · p. 54

Initialization and warmup

• Smaller initialization (std = 0.006) performs worse with very short warmup, while longer warmup benefits it relative to conventional std = 0.02.The experiments use AdamW and vary warmup, batch size, and training horizon.Source: §D.2 · p. 54
• Increasing batch size reduces the performance gap between smaller and conventional initializations.The reported ablation compares batch sizes of 512 × 512 and 256 × 512 tokens.Source: §D.2 · p. 54

Initialization and warmup

• Smaller initialization (std = 0.006) benefits from longer warmup, while very short warmup favors conventional std = 0.02 initialization.Increasing batch size reduces the performance gap for longer training runs.Source: Figure 42 · p. 55

Runtime by model scale

• Figure 44 tracks wall-clock time per 100 iterations as model size increases for optimizers omitted from the main comparison.SOAP, AdEMAMix, Muon, and AdamW are omitted because their results appear in Figure 18.Source: §D.3 · p. 55

Wall-clock comparisons

• SOAP is consistently the slowest optimizer and Signum the fastest in 100-iteration, single-GPU runs, though most optimizer timing differences are small.Runs use batch size 16 × 512 tokens without gradient accumulation or torch.compile; absolute times may vary by hardware.Source: §D.3 · pp. 54–55, Figure 43 · p. 56

Tuning scales and horizons

• For 124M models, tuning uses batch sizes 32 and 256 at 512-token sequence length, with gradient accumulation when needed to preserve effective batch size.For 210M models, runs use batch size 256 × 512 tokens and training durations from 8k to 128k steps.Source: E Hyperparameter tuning · pp. 56, 64
• Large-model benchmarks use batch sizes of 3936 for 583M models and 1984 for 720M models, corresponding to approximately 2M and 1M tokens.For 1M-token runs, training horizons are T ∈ {8, 16, 48}k steps.Source: E Hyperparameter tuning · pp. 56, 69

Tuning workflow

• The tuning workflow starts with 124M models, reuses their best settings for 210M runs except learning rate, and tunes selected parameters or settings at larger scales.At 583M, the reported ablation isolates z-loss for AdamW and SOAP; 720M models are tuned at 16B tokens, and MoE tuning is covered separately.Source: E Hyperparameter tuning · p. 57

Comparison and ablation setup

• Hyperparameter ablations assume the remaining settings are already tuned, and most settings are maintained across optimizers while learning rates are adjusted.Muon and MARS also receive reductions to AdamW’s learning rate and the rate for non-1D parameters.Source: E Hyperparameter tuning · p. 58

Scheduler settings

• For WSD scheduling, the tuning rule halves the cosine-optimal learning rate and assigns 20% of iterations to cooldown, without stochastic weight averaging.The cooldown uses decay shape (1 − √x), which the authors report works better in their setting.Source: E Hyperparameter tuning · p. 58

Large-model tuning outcomes

• With large batches of 1M tokens and shorter training runs, smaller β2 gives better results for SOAP, D-Muon on 1D parameters, and Prodigy.The appendix also describes re-tuning optimizer β parameters when training iterations change.Source: E Hyperparameter tuning · p. 69

Model configuration

• The Llama-like MoE model has about 520M parameters and matches the 124M dense model’s hidden size, layer count, and attention heads.It uses softmax linear gating with top-k routing (k = 2) over eight experts.Source: §E.5 · p. 73

Training setup

• The MoE experiment transfers optimizer hyperparameters from dense-model tuning, with Sophia’s learning rate adjusted to 0.0005 from 0.001.Training uses batch size 256 × 512 tokens.Source: §E.5 · p. 73

Study purpose

• The authors frame the comparison as a test of dense-tuned hyperparameter reuse in realistic small-scale deployment settings.The passage describes the purpose of the ablation, not a comparative optimizer outcome.Source: §E.5 · p. 73