Blog: Survey of Optimizers

Ruoran Xu

Source-linked AI summary

Blog: Survey of Optimizers

Ruoran Xu

arXiv:2608.28557v1Aug 28, 2026cs.LGcs.AI

TL;DR

The survey addresses how neural-network optimization has expanded beyond named Adam variants into interacting choices about time, geometry, horizon, parameter classes, and state representation. It organizes recent methods along these axes and compares their evidence across training and systems conditions. Its central conclusion is that matrix-aware methods are a genuine advance, but no context-independent replacement for AdamW exists.

Problem
Recent optimizer comparisons lack a unified framework for separating temporal estimation, geometry, horizon management, and state representation across changing training conditions.
Method
The survey organizes recent optimizers as compositional stacks spanning temporal estimators, matrix and coordinate geometries, memory representations, and training policies.
Results
The survey concludes that matrix-aware methods represent a genuine advance, while optimizer rankings vary with protocol and resource metric rather than yielding one universal winner.
Takeaways & Limitations
Optimizer selection should match the parameter block, training horizon, precision, and system budget, with those conditions reported clearly for reproducibility.
Takeaways & Limitations
Optimizer claims are protocol-dependent, and slow temporal averages interact with batch size and training duration.

Abstract

from arXiv · show

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.

1 The optimizer question has changed

Contemporary optimizer design is best understood as a stack of temporal estimation, update geometry, horizon policy, and state representation rather than as a sequence of named algorithms. The survey emphasizes that these choices interact with parameter classes, evidence levels, and resource metrics.

Optimizer stacks: An optimizer stack separates temporal estimators, geometric transforms, stored representations, and learning-rate or regularization policies.The same optimizer name may modify one component, while one implementation can combine ideas from all four.
Evidence and evaluation: Optimizer comparisons must distinguish scaled, controlled, and emerging evidence rather than treating all results as equally general.Reported speedups are not directly comparable because faster may mean fewer tokens, FLOPs, wall-clock time, or more feasible memory.
Update geometry: Modern optimizers extend beyond coordinate-wise updates to matrix, low-rank, spectral, and Kronecker geometries.Momentum SGD, Adam, Shampoo, Muon, and low-rank methods correspond to different choices of temporal estimator and left/right geometry.
Evidence and evaluation: The survey treats optimization as a budgeted policy because statistics, representation, and implementation determine whether mathematical benefits appear in the constrained resource metric.Per-token, per-step, wall-clock, and memory outcomes can favor different methods.
State and time: Persistent optimizer state is a compressed summary of gradient history whose effective memory changes with the sampling unit and batch size.For an EMA, nominal memory is approximately (1 −β)^-1 steps, or roughly B/(1 −β) tokens when a step contains B tokens.
Parameter classes: Matrix transforms do not automatically apply to every parameter class, so practical stacks often route vectors, embeddings, and other non-hidden parameters to AdamW.Transformer parameter classes differ in shape and symmetry, making parameter-specific optimizer choices necessary.

3 Why AdamW remains the reference point

AdamW remains the reference point because its mature engineering contract spans predictable tensorwise behavior, routine sharding, low-precision implementations, and established training recipes. Recent methods target specific weaknesses in temporal estimation, normalization timing, variance, agreement, spikes, and optimizer-state memory, but their benefits remain context- and protocol-dependent.

AdamW remains difficult to displace because its engineering contract includes predictable tensorwise behavior, routine sharding, low-precision implementations, and established recipes.A new optimizer must beat this total operational risk, not merely improve a loss curve.
Recent work addresses three Adam weaknesses: forgetting useful old gradients, coupling the current gradient to its denominator, and the cost of storing a scalar second moment.These correspond to longer-timescale momentum, reordered normalization, and more selective or compressed state.
More careful use of gradient history: AdEMAMix uses fast and slow exponential averages, and its largest comparison matched AdamW trained on 197B tokens with a 1.3B model trained on 101B tokens.The survey treats this as evidence that one momentum timescale is restrictive, not as a portable twofold guarantee.
Agreement filters, resets, and robustness: Agreement filters and spike handling modify updates locally: cautious methods mask sign-disagreeing coordinates, while SPAM clips spikes, resets contaminated state, and can retain sparse momentum.These interventions add robustness or reduce memory without necessarily changing the base optimizer’s geometry.
A closer look at temporal estimators: MARS separates temporal variance reduction from spatial preconditioning, allowing AdamW-, Lion-, and Shampoo-like variants rather than one fixed optimizer rule.Its recursive estimator combines current and previous stochastic gradients before applying a chosen preconditioner.
Agreement filters, resets, and robustness: Stable-SPAM reports a protocol-specific case where 4-bit training outperformed a higher-precision Adam baseline, linking numerical precision to gradient stability.The result suggests stabilization may matter more than simply adding bits when updates are poorly scaled.
ADOPT and the timing of normalization: ADOPT improves convergence analysis by reordering normalization and moment accumulation, yet empirical performance can still depend on β2.Removing β2 from a theorem does not remove it from practical tuning.

4 The matrix-aware renaissance

Matrix-aware optimization expands update geometry beyond coordinatewise scaling, spanning historical whitening, polar normalization, and hybrids that selectively restore variance adaptation. Evidence supports meaningful gains at scale, but comparisons remain protocol-dependent rather than universally ranking one method.

Matrix optimizers exploit row and column structure in dense weights, attention projections, embeddings, experts, and convolutional kernels.
Shampoo and SOAP use historical matrix statistics, with SOAP adding Adam-style coordinatewise adaptivity in evolving eigenbases.Shampoo uses left and right statistics and inverse roots; SOAP rotates gradients into those eigenbases before updating.
SPlus reached Adam-level validation performance in an average of 44% of the steps and 62% of wall-clock time across its benchmark.
Muon approximately orthogonalizes momentum updates, replacing singular values by one while preserving singular directions rather than estimating a Hessian.Its polar-style transform corresponds to the linear minimization oracle for a spectral-norm constraint.
The scalable Muon study matched AdamW with roughly 52% of training FLOPs under its fitted protocol and demonstrated a 16B-parameter mixture-of-experts run.The reported result depends on tensor partitioning, shape scaling, and update-RMS alignment.
Adaptive and hybrid methods restore variance information at different costs: NorMuon reports 21.74% greater efficiency than Adam and 11.31% over Muon, while Mousse reports roughly 12% fewer steps than Muon with about 3% wall-clock overhead.COSMOS combines leading-subspace SOAP treatment with Muon-like residual updates; AdaMuon adds elementwise second moments.
Optimizer rankings vary with scale, data-to-parameter ratio, horizon, batch size, schedule, and implementation, so an optimizer winner is a statement about a protocol.A large-scale study found SOAP and Muon outperforming AdamW in tested trillion-token regimes, with a SOAP variant slightly ahead.

5 Dissecting matrix optimizers

Matrix optimizers can be decomposed into momentum, basis choice, singular-value transformation, variance adaptation, and output scaling. This decomposition clarifies why spectral normalization, historical whitening, hybrid adaptation, and distributed implementation make different trade-offs across horizons and parameter blocks.

A mechanism-level taxonomy separates momentum formation, basis choice, singular-value transformation, variance adaptation, and output scaling.
Muon uses finite-step polar computation, so coefficients, normalization, iteration count, and precision affect both approximation quality and kernel performance.Rectangular matrix handling and shape-dependent scale rules also matter.
Historical whitening reduces directional anisotropy and adapts to persistent variance over time, unlike a purely instantaneous spectral transform.
Controlled comparisons report that variance adaptation explains much of the gap between spectral accuracy and per-step progress.Muon can produce a tighter singular-value spread while SOAP achieves stronger per-step progress in the studied setup.
Hybrid methods allocate expensive geometry selectively: COSMOS uses a leading eigensubspace, Mousse composes whitening with a polar step, and AdaMuon or NorMuon add variance elementwise or rowwise.
Distributed matrix optimizers must report sharding, factorization exactness, refresh intervals, precision, and communication overlap because these choices can change the practical algorithm.
Pure spectral normalization is often strong early, whereas longer horizons more often reward variance adaptation.

6 The emerging layerwise optimizer stack

The emerging design is a layerwise optimizer stack that dispatches geometry according to parameter class and symmetry rather than applying one rule to every tensor. This flexibility introduces calibration, state-management, and attribution risks that must be specified as part of the algorithm.

4.1 Equivariance as a routing principle: Symmetry-compatible optimization treats update transformations as commuting with function-preserving parameter transformations.
4.1 Equivariance as a routing principle: Different parameter classes have distinct shapes and symmetries, motivating two-sided spectral, one-sided, row-aware, and hybrid updates.Embeddings, routers, normalization parameters, and output heads do not automatically suit a dense-matrix transform.
4.1 Equivariance as a routing principle: The routing principle supports dispatching geometry from architecture metadata rather than prescribing one universal matrix rule.
6.2 Calibration and failure surfaces: Layerwise stacks add failure surfaces involving relative learning rates, weight decay, state precision, sharding, residual-branch imbalance, and attribution.
6.2 Calibration and failure surfaces: A common calibration layer should match update RMS, align token-based timescales, and make non-matrix fallbacks explicit.These calibration choices are presented as part of the algorithm specification.

7 Memory is part of the algorithm

Memory-efficient optimization compresses different statistical objects, so state size alone does not determine the resulting training behavior. The survey therefore treats memory, throughput, batch size, projection freshness, and total workload as coupled parts of the optimization experiment.

At scale, the central memory question is where persistent state deserves allocation, not simply whether preconditioning is used.Full-precision Adam commonly stores two moment values per parameter in addition to weights, gradients, and often master weights.
Adam-mini reports roughly 45–50% lower optimizer-state memory, while SlimAdam reports up to 98% second-moment savings in its experiments.
Low-rank state need not imply a low-rank update; compressed statistics can choose full-space scales while retaining full-rank residual updates.
SWAN reports roughly 50% lower end-to-end memory and about twofold token efficiency over Adam, while SCALE reports Adam-like performance at 35–45% of total training memory.
End-to-end comparisons should report validation loss, wall-clock time, peak memory, robustness, workload reinvestment, and whether communication or batch-size effects drive gains.A fixed-memory benchmark can be more informative than bytes per parameter alone.
Memory-efficient methods use factorization, projection, sparsification, or elimination, each encoding different statistical biases.
Projection methods must manage changing bases because stale coordinates can corrupt state meaning; LDAdam transports states, while GaLore2 reduces projection and sharding bottlenecks.

9 Time: schedules, averaging, and unknown horizons

Training time is part of optimizer design: schedules, averaging, decay, and horizon assumptions shape trajectories and can change which checkpoint or method looks best.

Learning-rate schedules define temporal behavior because momentum, weight decay, averaging, and late decay all accumulate over training.
Schedule-free methods internalize horizon handling through fast and averaged iterates rather than removing temporal policy.The coefficients encode interpolation and averaging without specifying the final training horizon.
Warmup-stable-decay and reusable cooldowns maintain a stable trunk, then expose a decay branch when a checkpoint is needed.This allows one trajectory to support multiple endpoints as additional data or compute arrives.
Matrix-aware schedule-free methods are promising across multiple model sizes and horizons, but their composition with weight decay is nontrivial.SF-NorMuon reports competitive results for 125M and 772M models across 1×–8× horizons, with fast-iterate decay essential; ScheduleFree+ reports a 31% advantage over WSD at a high tokens-per-parameter ratio.
Batch-size correction preserves EMA half-life in processed examples, while small-batch experiments show momentum SGD can become competitive after conversion.Gradient accumulation may improve device utilization yet waste optimization opportunities.

10 Horizon management in detail

Horizon management is a policy problem spanning explicit decay, stable trunks, schedule-free averaging, checkpoint merging, and decay placement; each changes the trajectory and its evaluation.

Horizon strategies: A learning-rate curve governs exploration, progress revelation, and checkpoint availability through training time.The survey distinguishes explicit decay, stable trunks with cooldowns, and schedule-free averaging.
Explicit decay: Cosine decay embeds the planned horizon in every update, making extensions and intermediate checkpoints path-dependent.A midpoint under one planned horizon is not equivalent to the midpoint of a differently planned run.
Evaluation: Comparing methods before the intended endpoint can favor the wrong mechanism because high-rate tolerance and late-decay benefits emerge at different times.
Cooldowns: Warmup-stable-decay and reusable cooldowns separate a reusable high-rate trunk from a short decay branch, supporting multiple endpoints.The practical benefit is continuing the trunk when more data or compute arrives without changing earlier rates.
Schedule-free: Schedule-free optimization uses fast, averaged, and interpolated iterates to induce effective decay and averaging without encoding the final step.It is horizon-independent, not dynamically constant.
Composition: Decay location matters in multi-iterate methods: applying it to the fast, averaged, or evaluation iterate produces different trajectories.SF-NorMuon identifies fast-iterate decay as essential in its spectral construction.
Schedule-free: ScheduleFree+ targets long language-model runs, with its largest claimed benefit at an extreme tokens-per-parameter ratio requiring independent replication.
Averaging: Checkpoint merging acts as a temporal kernel whose window controls whether averaging is local or mixes functionally distant models.This connects stochastic weight averaging, online schedule-free averaging, and post-hoc merging.

11 Low precision: state semantics matter more than bit count

Low-precision optimization depends on the semantics and recurrence of each state, not merely its bit width, with quantization affecting memory, speed, timescales, and deployment behavior.

State semantics: Optimizer states are recursively updated, so rounding errors can alter their effective timescale rather than disappearing after one computation.
Representation: SOLO reports roughly 45GB of savings for a 7B model using 2–3 bit state and precision-specific momentum.
Muon: 8-bit Muon matches full-precision Muon at 1.6B scale while reducing optimizer state by 74% relative to full-precision Muon and 86% relative to FP32 AdamW.The study attributes this tolerance partly to momentum being used before a polar transform, unlike Adam’s denominator-sensitive second moment.
Failure modes: Quantization staleness can freeze an EMA when increments fall below a quantization cell, making its effective half-life much longer than (1 −β)^−1.Resets can restore responsiveness, but excessive resets destroy useful history.
Robustness: Singularity-aware Adam uses randomized probes to damp updates during local disagreement, complementing resets that remove contaminated history.
Representation: Quantization should match state topology and recurrence: logarithmic unsigned codes suit second moments, while signed blockwise representations suit momentum.
Dynamic allocation: Dynamic precision allocation can reduce state memory, but evaluations should report the precision path because transitions perturb the quantization lattice.STQuant reports an average width near five bits.

13 A synthesis: five claims that survive the leaderboard

The survey’s synthesis is conditional: optimizer performance depends on geometry, parameter-block structure, systems context, protocol variables, and evaluation objectives rather than a universal leaderboard.

Claims: Matrix-aware methods address distinct problems, and their gains are consistent enough to matter without making them uniformly second order.The key questions are which statistics are historical, which norm is controlled, and which information is discarded.
Claims: Pure spectral normalization is strong early and at shorter data ratios, while longer horizons often reward variance-aware methods.
Claims: Layerwise optimizer assignment is more appropriate than one universal rule because parameter blocks differ in symmetry and role.Practical stacks may combine spectral, one-sided, row-aware, and AdamW-like updates.
Claims: Communication, matrix operations, sharding, and quantization kernels are part of the practical optimizer, so convergence without systems context is incomplete.
Claims: A well-tuned matrix-aware method can reduce tokens to target loss, but AdamW remains preferable when robustness, compatibility, maturity, or switching cost dominates.The survey’s conclusion is a decision procedure rather than a universal winner.
Protocol: Optimizer rankings reverse across horizon, batch size, and study protocol, making apparent contradictions compatible when protocol is treated as part of the result.
Protocol: Matrix-optimizer gains can shrink with model scale while rankings also change with training duration.The cited study spans 0.1B–1.2B parameters and 1×–8× a standard compute-optimal data ratio.
Protocol: Batch size changes gradient variance, token-time momentum half-life, optimizer-step count, and hardware utilization, altering method rankings.

15 A practical decision guide

A practical optimizer study should proceed conservatively: establish an AdamW frontier, vary one design axis at a time, add interactions only afterward, and define the budget metric before selecting a winner.

Establish a reproducible AdamW frontier by jointly sweeping learning rate and weight decay while fixing data order and precision.Log update RMS by parameter class.
Change one optimizer axis at a time, pairing conventional schedules with Muon, AdamW geometry with schedule-free training, and full-precision algorithms with quantized state.
Add combined methods only after their components work independently, because schedule-free matrix optimizers and low-bit SOAP alter multiple axes simultaneously.
Choose the budget metric before declaring a winner: tokens measure sample efficiency, FLOPs include matrix operations, time-to-target includes systems costs, and fixed HBM captures workload capacity.

16 A minimum credible evaluation protocol

A credible optimizer comparison must control training conditions, tune methods fairly, measure statistical and systems outcomes across horizons, and disclose tuning cost and scope.

Report more than final loss, including deployment-relevant downstream adaptation, retention, or quantization quality when applicable.
Match initialization, data mixture and order, tokenizer, precision, parameterization, and optimizer treatment of embeddings, heads, normalization, biases, and non-matrix tensors.
Give each optimizer specific sweeps for learning rate, momentum or decay constants, weight decay, and schedule rather than transplanting defaults blindly.
Measure update RMS and weight RMS by parameter class, especially when comparing Adam-like and spectral methods.
Report loss at fixed tokens and FLOPs, time to a prespecified target, peak memory, optimizer-step time, and communication volume.
Evaluate multiple horizons, including the actual endpoint, because optimizer rankings can flip late; use several seeds when expected gaps are small and record instability.
State tuning compute because a win requiring a much larger sweep has an adoption cost, even for long trillion-token runs.
Open questions include hybrid-geometry theory, scaling laws incorporating state, batch, horizon, and tuning budget, optimizer-aware parameterization, state transport, and reproducible systems evidence.The survey frames optimizer progress as composed training policies spanning temporal estimation, matrix geometry, horizon, and representation.

A Compact literature map

The literature map organizes representative optimizer work by functional family, intervention, and evidence emphasis, spanning memory, horizon, low precision, and matrix-scaled methods.

Matrix and scaling: Matrix and scaling methods include low-rank eigen-Adam, row-column-scaled SGD, and column-normalized SGD with minimal momentum.
Memory: Memory methods include SWAN, SCALE, SlimAdam, APOLLO, LDAdam, GaLore2, and Memory Alice/RACS, using stateless normalization, minimal momentum, low-rank state, or projected moments.
Horizon: Horizon methods include Schedule-Free, WSD, Power Scheduler, WSM, and SF-NorMuon, covering endpoint-free iterates, decay branches, scaling laws, checkpoint merging, and schedule-free spectral updates.
Low precision: Low-precision methods span COAT, SOLO, ECO, 8-bit Muon, state-staleness analysis, STQuant, and OSP, with quantized states, error feedback, resets, adaptive precision, or outlier suppression.