PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Siyuan Li, Jiabao Pan, Yumou Liu, Zhuoli Ouyang, Xin Jin, Xinglong Xu, Jingxuan Wei, Shengye Pang, Jintao Che, Xuanhe Zhou, Conghui He, Cheng Tan

arXiv:2607.04033v1cs.LGcs.AI

TL;DR

Optimizer research is fragmented across mechanisms and protocol-sensitive evaluations, making method selection difficult. OmniOpt unifies optimizers through a staged pipeline, geometric formulation, taxonomy, and cross-domain benchmark, finding that optimizer rankings systematically cross with scale, context length, and domain.

  • Problem

    Optimizer literature is fragmented by local mechanisms and protocol-sensitive empirical claims, limiting systematic comparison across update stages and training conditions.

    Source: §1.1 · p. 4

  • Method

    OmniOpt combines a five-stage optimizer meta-pipeline, LMO-based geometric decomposition, and mechanism-aligned cross-domain benchmarking across optimizer families and effect objectives.

    Source: §1.2 · p. 5

  • Results

    Optimizer rankings exhibit systematic crossings with model scale, context length, and domain across the benchmark’s evaluated tasks and objectives.

    Source: §1.2 · pp. 5–6

  • Takeaways & Limitations

    Optimizer choice should be evaluated in relation to scale, context length, and domain rather than treated as universally ranked.

    Source: §1.2 · p. 6

    The LMO reports direction and boundary point but discards magnitude, requiring gradient scale to be reintroduced through the learning rate, radius, or preconditioner.

    Source: §3.2.1 · p. 17

Abstract

from arXiv · show

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.

1 Introduction

OmniOpt frames optimizer selection as a mechanism- and objective-aware system-level problem, unifying optimizer structure, geometry, taxonomy, and benchmarking. It connects these views to explain and evaluate trade-offs across training constraints, scales, domains, and objectives.

1.2 Contributions and Paper Organization

  • OmniOpt contributes a five-stage meta-pipeline—routing, gradient transformation, state evolution, update reconstruction, and finalization—where most optimizers modify only one or two stages.The identity-mapping principle distinguishes primary interventions from default operations.Source: §1.2 · p. 5

1.1 Background and Motivation

  • The paper addresses fragmented optimizer analysis with an LMO-driven four-axis decomposition and dual taxonomy linking mechanism families to measurable objectives such as convergence, cost, memory, stability, tuning robustness, and generalization.The four axes describe update domain, state estimator, geometry-and-precondition operator, and finalization wrapper.Source: §1.2 · p. 5, §1.1 · p. 5

1.2 Contributions and Paper Organization

  • The unified benchmark finds no optimizer dominates all objectives: state compression excels at short context but degrades with complexity, while structured-matrix methods transfer stably at substantial per-step cost.Optimizer rankings cross systematically with scale, context length, and domain.Source: §1.2 · pp. 5–6

1.3.1 Prior Literature Strands and Survey-Theory Baselines

  • Prior surveys, theory, and benchmarks separately cover optimizer history, geometric update interpretations, or realistic protocols, but do not jointly map intervention stages, compatible mechanisms, and benchmark objectives.Relevant works include Zhang et al. [134], Altinel and Ozcan [3], Ranganath et al. [91], Bernstein et al. [10], Pethick et al. [87], and Sfyraki et al. [98].Source: §1.3.1 · pp. 6–7

1.3.2 Benchmark Evidence and Taxonomy Mapping

  • Existing benchmark evidence shows optimizer rankings change with tuning budget, model and batch scale, duration, data regime, and evaluation endpoint, motivating family- and objective-specific reporting rather than a single aggregate rank.The paper’s taxonomy maps benchmarked optimizers to primary mechanism families and evaluates protocol variation, costs, tuning, downstream quality, and mechanistic behavior.Source: §1.3.2 · p. 7, §1.3.3 · p. 8

2 Preliminaries

This section establishes the optimization, parameter-topology, memory, and notation conventions for the survey, then interprets classical and modern optimizers through smoothing, curvature estimation, scaling, and regularization. It connects these preliminaries to the later unified framework by treating parameter geometry and curvature approximations as foundations for optimizer scoping and LMO-driven decomposition.

2.1 Problem Formulation

  • Large-language-model pretraining and supervised fine-tuning are formulated as empirical-risk minimization with stochastic mini-batch gradients whose noise and numerical behavior reflect the surrounding training system.Transformer parameters are distinguished by tensor geometry, while mixed-precision memory accounting shows that Adam states alone require about 56 GB for a 7B-parameter model.Source: §2.1 · p. 9

2.2 Classical Optimization Background

  • Classical optimizers are organized by gradient smoothing, curvature-proxy maintenance, direction scaling, and final-write regularization, with AdamW serving as the reference Adam-like baseline.SGD uses identity geometry, momentum smooths the gradient, adaptive methods use diagonal curvature proxies, and AdamW decouples weight decay from adaptive scaling.Source: §2.2 · pp. 9–11
  • Newton, natural-gradient, and matrix-structured methods motivate a unified preconditioning view in which tractable approximations determine the state, curvature estimator, and geometry axes of the LMO-driven decomposition.Modern examples replace exact inverse Hessians or Fisher metrics with diagonal, structured, eigenbasis, polar, Kronecker, low-rank, or clipped-Hessian approximations.Source: §2.2 · pp. 10–11

3 Unified Theoretical Framework for Optimizers

This section develops a unified theoretical framework for optimizers by organizing update mechanisms across ordered intervention stages and complementary update axes. It instantiates representative methods within this structure and relates norm-constrained linear minimization oracles to update geometry and preconditioning.

3.1.1 Operational Steps

  • The Universal Meta-Pipeline represents each optimizer update as signal acquisition followed by routing, transformation, state evolution, reconstruction, and finalization, with most methods modifying only one or two stages.The update is formalized as Δt = S5(S4(S3(S2(S1(Gt); St−1); St−1); St−1, Wt), followed by Wt+1 = Wt + Δt.Source: §3.1.1 · pp. 12, 14, §3.1.2 · p. 14

3.1.2 Identity Mapping and Representative Instantiations

  • Representative optimizers are characterized by their active pipeline stages: AdamW is primarily S3/S5, Muon S1/S2, GaLore S1–S4, Lion S2/S3, and SAM S0/S5.GaLore relocates computation and optimizer state into a lower-dimensional space through the S2/S4 dual, whereas SAM places its sharpness correction in signal acquisition and finalization.Source: §3.1.2 · pp. 15–16

3.2 A Mathematical Unification of Optimizer Updates

  • The LMO framework treats constraint geometry and optimizer behavior as two readings of one direction operator, covering sign updates, spectral orthogonalization, Kronecker preconditioning, curvature weighting, variance reduction, and state compression.The four axes separately encode update domain, state estimator, geometry or preconditioner, and finalization wrapper.Source: §3.2 · pp. 16–17

3.2.1 LMO Foundations and Norm-Induced Directions

  • Norm-constrained LMOs select steepest directions from the chosen geometry, but discard magnitude, so gradient scale must be restored through the learning rate, radius, or a preconditioner.The Euclidean ball yields normalized gradients, the max-norm ball sign directions, and the spectral-norm ball matrix polar directions; Adam is interpreted as an LMO over a state-generated adaptive box.Source: §3.2.1 · pp. 17–18

3.2.2 A Four-Axis Decomposition of Optimizer Updates

  • The four-axis decomposition moves momentum, second moments, variance reduction, and projection state into Axis II before Axis III forms the direction, while Axis I fixes the update space and Axis IV commits the result.SGD, AdamW, and Muon update in the original parameter or matrix space, whereas GaLore uses a low-rank projected space and maps the direction back through the projection operators.Source: §3.2.2 · p. 18
  • Muon demonstrates Axis III’s dual interpretation: its spectral-norm LMO and Gram-based preconditioner produce the same update.The framework identifies Muon as both a spectral-norm LMO and a Gram-based preconditioner.Source: §3.2.2 · p. 19
  • The four axes select the update domain, estimate optimizer state, generate the direction through geometry or preconditioning, and finalize the writeback with wrappers such as decay or projection.These axes align step by step with the meta-pipeline: domain choice, state estimation, direction generation, and finalization; state estimation precedes geometry.Source: §3.2.2 · p. 19

3.2.3 Instantiations

  • AdamW estimates first- and second-moment states, then applies diagonal second-moment preconditioning, interpretable as an adaptive ℓ∞ box or a diagonal metric.MARS-AdamW preserves AdamW’s Axis III form while moving variance reduction into the preceding state-estimation axis.Source: §3.2.3 · p. 20
  • GaLore-Adam makes Axis I explicit by placing the update in a low-rank projected subspace defined by projection matrices with rank much smaller than the matrix dimensions.The broader framework distinguishes full-space, matrix-space, and low-rank projected-subspace update domains.Source: §3.2.2 · p. 18, §3.2.3 · p. 20

L MtQR, (39)

The section decomposes optimizer updates into four axes and instantiates projected adaptive methods such as MtQR alongside SOAP. MtQR performs Adam-style state estimation and direction formation in a low-rank subspace, while SOAP adapts in a periodically refreshed Shampoo eigenbasis.

MtQR

  • MtQR stores projected first- and second-moment states in an r × r subspace, reducing optimizer memory from O(mn) to O(r2).Its projected direction is formed entrywise using Adam-style adaptive scaling before being lifted back to the original weight space.Source: L MtQR, (39) · p. 20
  • MtQR combines projected Adam box geometry with a projected diagonal metric, then rotates the adaptive direction back through the subspace bases.This gives the method both an LMO interpretation and a preconditioner interpretation.Source: L MtQR, (39) · p. 20

SOAP

  • SOAP uses full-rank Shampoo eigenvectors as a rotated coordinate system, refreshing its basis every f steps while updating the rotated second moment each step.It retains the first moment in the original space and applies the Adam box in the eigenbasis before rotating the direction back.Source: L MtQR, (39) · p. 22
  • Unlike Shampoo’s fixed inverse-root metric, SOAP separates a basis held for f steps from per-step adaptive variance, enabling robustness at low preconditioning frequency.The section attributes this robustness to allowing the basis to become stale while Vt continues to carry adaptivity.Source: L MtQR, (39) · p. 22

Framework summary

  • The four-axis framework separates update domain, state estimation, direction geometry, and write-back, preventing momentum or variance estimation from being confused with LMO geometry.Axis II contains moments, projection bases, and variance reduction, whereas Axis III contains the LMO constraint or metric geometry.Source: Table 5 · p. 21, L MtQR, (39) · p. 22

4 Dual-Dimension Taxonomy

OmniOpt organizes optimizers with a dual taxonomy: a single primary mechanism family and multi-label effect objectives, creating a mechanism-aware framework for survey organization and benchmark design.

4 Dual-Dimension Taxonomy

  • The taxonomy combines a five-stage meta-pipeline view with an LMO-driven four-axis decomposition to support mechanism-based literature organization, experimental grouping, and trade-off analysis.It distinguishes where an optimizer intervenes from how it changes direction, curvature, gradient estimation, or state representation.Source: §4 · p. 22

4.1 Taxonomy Design Principles

  • Dimension A assigns each optimizer one primary mechanism family, while Dimension B permits multiple objective labels for convergence, cost, memory, stability, robustness, and generalization.This separation distinguishes what an optimizer changes from the outcome it is intended to improve.Source: §4 · p. 22, §4.1 · p. 23

4.2 Methodological Taxonomy

  • Dimension A organizes 108 optimizers into five families and fifteen subclasses, classified by the dominant incremental mechanism rather than optimizer names, chronology, or inherited base methods.The families are T1 element-wise adaptive moments and scalar control, T2 matrix-level structure, T3 directional discretization, T4 state compression, and T5 geometric regularization.Source: §4.2 · pp. 23–24, §4.1 · p. 23

4.3 Objective-Oriented Taxonomy

  • Dimension B defines six non-exclusive objectives—O1 convergence, O2 cost, O3 memory, O4 stability, O5 robustness, and O6 generalization—measured through logged, derived, or cross-configuration metrics.These labels are evaluation prompts rather than performance rankings, and design intent must remain distinct from experimentally verified effects.Source: §4.3 · pp. 24–25

4.4 Cross-Dimension Analysis

  • The cross-dimension matrix supplies benchmark priors and exposes composition opportunities, while making explicit that protocol-dependent effects and meta-pipeline conflicts constrain practical combinations.Low-rank projection can combine with state quantization, but overlapping transformations, streaming updates, extra evaluations, and expensive preconditioning may require ordering or become impractical.Source: §4.4 · p. 26

5 Optimizer Method Families · 5.1 T1: Element-Wise Adaptive Moment and Scalar Control

OmniOpt organizes optimizer methods into mechanism families, with T1 covering coordinate-wise adaptive-moment and scalar-control methods whose main variations affect state evolution or outer step-size and regularization control. T1 serves as the calibration family because AdamW-like methods provide a mature, stable reference stack for comparing mechanisms and objectives.

5.1.1 Family Overview and Meta-Pipeline Position

  • T1 defines the element-wise family: methods preserve coordinate-wise updates, avoid cross-parameter routing, and mainly alter scalar state, adaptive moments, or outer control rules.T1 excludes methods whose defining contributions are matrix routing, irreversible sign discretization, state compression, or sharpness-aware gradients.Source: §5.1.1 · pp. 27–28
  • The taxonomy splits T1 mechanistically into direct Adam variants, multi-timescale or corrected estimators, and iterate-averaging or automatic-tuning methods.The subclasses identify the dominant non-identity operation added around the AdamW coordinate-wise template rather than following chronology.Source: §5.1.1 · p. 28
  • T1 is mostly identity-like in routing, gradient input, and reconstruction, while its primary variation is S3 state evolution and S5 control over decay, global learning rate, or schedule replacement.AdamW decouples weight decay, RAdam rectifies early variance, D-Adaptation and Prodigy infer global scale, and Schedule-Free methods use iterate averaging.Source: §5.1.1 · pp. 28–29

5.1.2 LMO-Driven Four-Axis Interpretation

  • In the four-axis view, T1 uses the identity basis, diagonal curvature, and full coordinate-resolution state; AdamW therefore combines element-wise direction with diagonal scaling rather than being simply sign or Newton descent.The LMO view interprets normalized coordinate directions through an ℓ∞-type constraint, while the diagonal denominator creates an anisotropic coordinate-wise trust region.Source: §5.1.2 · p. 29

5.1.3 Representative Methods

  • Representative T1 methods modify AdamW locally, add temporal or variance-reduced signals, or change outer averaging, scale inference, and optimizer switching.Examples include RAdam, AdaBelief, Adan, ADOPT, AdEMAMix, MARS, TAM, Schedule-Free, D-Adaptation, Prodigy, and SWATS, while composite bundles retain an Adam or RAdam core.Source: §5.1.3 · pp. 29–31
  • AdamW is the calibration baseline because coordinate-wise normalization, momentum smoothing, decoupled weight decay, and a mature schedule interface support stable Transformer pretraining and strong early loss reduction.AdamW stores two full-size moment buffers, making it more expensive than SGDM; optimizer rankings can also shift with model scale and protocol choices, as reported by Zhao et al., Semenov et al. [97], and Wen et al. [113].Source: §5.1.3 · p. 32
  • T1 frames optimizer comparison around which AdamW control-stack components remain necessary after fixing model scale, batch size, training length, and tuning budget.This reference framing distinguishes mechanism-informed priors from final empirical rankings and provides the coordinate system for comparing later optimizer families.Source: §5.1.3 · pp. 32–33
  • T1.1 favors fast early reduction and robustness across heterogeneous coordinate scales, T1.2 targets smoother trajectories and token efficiency, and T1.3 targets broader learning-rate usability and fewer schedule decisions.These benefits trade off against full state, extra computation, or the possibility that automatic rules trail carefully tuned schedules; MARS-AdamW is described as the most stable AdamW-style enhancement but requires more state and compute.Source: §5.1.3 · p. 33

5.2 T2: Matrix-Level Structural Methods

T2 defines optimizers by operating directly on Transformer weight matrices, using spectral, Kronecker-factored, or low-rank geometry rather than independent coordinates. Its benefits and costs are multi-objective and depend on whether exploited structure is statistically useful and computationally affordable.

5.2.1 Family Overview and Meta-Pipeline Position

  • T2 operates directly on matrix structure, routing dense matrix weights through matrix transforms while typically sending biases, normalization parameters, embeddings, and some heads to AdamW or SGD fallbacks.Its subclasses are spectral direction selection, Kronecker-factored preconditioning, and low-rank subspace projection; S1 routing determines where matrix operations can apply.Source: §5.2.1 · p. 33

5.2.2 LMO-Driven Four-Axis Interpretation

  • The four-axis interpretation separates matrix analysis bases, direction maps, structured curvature proxies, and optimizer-state geometry, with Muon, Shampoo, SOAP, and GaLore occupying distinct combinations.Muon flattens singular values through a polar direction, Shampoo uses Kronecker curvature, SOAP adapts in a learned matrix basis, and GaLore projects state into rank r ≪ min(m, n).Source: §5.2.2 · pp. 35–36

5.2.3 Representative Methods

  • Representative methods share mechanism-specific trade-offs: Muon selects spectral directions, Shampoo and SOAP scale in structured curvature coordinates, and GaLore or Fira train in low-rank subspaces.Fira addresses information loss from insufficient rank or stale bases, while T2.3 comparisons require matched memory, rank, refresh, batch-size, and wall-clock conditions.Source: §5.2.3 · pp. 36–38

5.2.4 Effect-Target Assessment

  • T2 has a multi-objective profile: spectral methods may improve loss per token but slow steps, Kronecker methods may improve conditioning but incur overhead, and low-rank methods save memory subject to rank and refresh quality.Rankings vary with scale, tuning effort, and training duration, so T2 is not a uniform upgrade over AdamW.Source: §5.2.4 · p. 38

5.3 T3: Discretization and Directional Quantization

T3 discretizes update directions through sign-like transformations, distinguishing it from moment scaling and state compression. Its main benefit is resource efficiency, while fixed-magnitude updates can weaken tuned perplexity relative to leading T1 and T2 methods.

5.3.1 Family Overview and Meta-Pipeline Position

  • T3 changes update geometry by applying hard or smooth sign-like maps that preserve coordinate orientation while discarding, clipping, or coarsening magnitude information.Unlike T1’s moment estimation or T4’s state-management mechanisms, T3 directly transforms the direction itself.Source: §5.3.1 · pp. 39–40

5.3.2 LMO-Driven Four-Axis Interpretation

  • Under the ℓ∞-ball, sign updates are linear minimization oracle steps because the ball’s extreme points are coordinate-wise sign patterns.SignSGD applies this geometry directly, while Lion applies the same map to a smoothed current-gradient and momentum signal [11].Source: §5.3.2 · p. 40

5.3.3 Representative Methods

  • T3 spans pure sign directions, signed momentum, and smooth or hybrid variants that soften sign decisions, add trajectory attraction, or restore limited magnitude channels.SignSGD removes continuous magnitudes [11], Lion is the central signed-momentum representative [19], and RLion, FOCUS, and Ano extend the sign-direction core [68] [50].Source: §5.3.3 · pp. 41–42

5.3.4 Effect-Target Assessment

  • T3 is resource-efficient and locally tolerant to learning-rate perturbations, but its tuned perplexity is generally weaker than the best T1 and T2 alternatives.Removing AdamW’s second-moment state reduces optimizer memory, but fixed-magnitude updates remain sensitive to learning rate, warmup, and weight decay.Source: §5.3.4 · pp. 42–43

5.4 T4: State Compression and Structural Aggregation

T4 comprises optimizers that reduce optimizer-state memory or gradient lifetime by changing how state is represented, reconstructed, shared, or consumed. Its taxonomy spans four subclasses and evaluates their O3 memory benefits against approximation, overhead, convergence, and protocol-dependent trade-offs.

5.4.1 Family Overview and Meta-Pipeline Position

  • The four T4 subclasses are row-column factorization, low-bit quantization, block- or layer-level state sharing, and fused backprop-update.They include AdaFactor and CAME; 8-bit Adam and Q-GaLore; Adam-mini, APOLLO, SM3, Conda, and NovoGrad; and LOMO and AdaLOMO, respectively.Source: §5.4.1 · p. 43
  • T4 begins when optimizer state undergoes nontrivial compression, sharing, quantization, or lifetime shortening, with usable state reconstructed or exposed at update time.Full-state AdamW is the special case in which compression and reconstruction maps are identities.Source: §5.4.1 · p. 44

5.4.2 LMO-Driven Four-Axis Interpretation

  • T4 preserves or approximates Adam-like update geometry while compressing statistics through factorization, quantization, sharing, or streaming, with compression quality depending on structural alignment and numerical controls.Factoring can reduce matrix state from O(mn) to O(m + n); low-bit methods depend on block size, dynamic-range tracking, stochastic rounding, and sensitive-tensor handling; shared-state methods trade granularity against memory savings; streaming methods restrict global-gradient operations.Source: §5.4.2 · p. 45, §5.4.3 · pp. 46–47

5.4.4 Effect-Target Assessment

  • T4’s primary objective is O3 memory reduction, but evaluation must compare absolute and memory-normalized performance because ranking depends on hardware feasibility, overhead, convergence, and protocol.Relevant measures include peak memory, persistent state, gradient-buffer lifetime, step time, token-normalized and wall-clock loss, and sensitivity to tuning and quantization granularity; streaming methods impose especially strong composition constraints.Source: §5.4.3 · pp. 47–48, §5.4.4 · p. 48

5.5 T5: Curvature-Aware and Geometric Regularization

T5 groups optimizers that impose curvature or geometry constraints on gradients, candidate updates, or parameter writeback, spanning perturbation, curvature estimation, filtering, and layer-wise scaling. The family is competitive in limited regimes but shows no uniform advantage under the current LLM benchmark, with comparisons sensitive to compute accounting.

5.5.1 Family Overview and Meta-Pipeline Position

  • T5 primarily acts near S5, while subclasses enter through perturbed-gradient evaluation, curvature storage and clipping, update filtering, or layer-wise trust ratios.T5 labels wrapper-level geometry: SAM can wrap SGD or AdamW, Cautious Lion can wrap Lion, and LAMB can wrap an Adam-style update.Source: §5.5.1 · p. 50

5.5.2 LMO-Driven Four-Axis Interpretation

  • In the LMO view, T5 appends an outer geometric operator to a base optimizer, usually leaving its direction coordinates intact while constraining or rescaling the written update.SAM instead changes the gradient evaluation point and requires a second forward-backward pass for the perturbed gradient.Source: §5.5.2 · p. 51

5.5.3 Representative Methods

  • Representative branches include SAM-style perturbed gradients, Sophia and AdaHessian curvature proxies, projection or selective filtering, and LAMB or AGC parameter-relative scaling.T5.3 methods operate after the base direction is formed, whereas layer-wise methods determine step scale from parameter-group geometry.Source: §5.5.3 · pp. 52–54

5.5.4 Effect-Target Assessment

  • T5 methods are competitive in limited regimes but show no uniform advantage under the current LLM benchmark, and fixed-step comparisons can change their apparent ranking relative to fixed-compute comparisons.The family incurs protocol-dependent costs, including extra forward-backward work for SAM, curvature-proxy choices for Sophia, and layer-norm conventions for LAMB.Source: §5.5.4 · pp. 54–55

6 Benchmark Study

This section benchmarks 24 optimizers across language-model and vision settings using mechanism-aware, multi-objective evaluations of quality, efficiency, stability, robustness, and transfer. It examines how optimizer behavior depends on scale, architecture, context length, and design components such as orthogonalization, including their interactions with standard Transformers and Gated DeltaNet topology.

6.1 Experimental Setup

  • The benchmark covers 24 optimizers across all five families and evaluates quality, runtime, memory, stability, learning-rate robustness, and cross-scenario generalization under controlled protocols.Stage 1 uses C4 with LLaMA at sequence length 256 across 60M–1B scales, while Stage 2 transfers stronger methods to FineWeb-Edu with 32k sequences and four architectures; only optimizer hyperparameters are tuned, with regularization disabled in Stage 1 and shared in Stage 2.Source: §6.1 · pp. 55–56

6.2.1 Stage 1: Broad Screening on C4

  • Stage 1 identifies distinct quality–runtime–memory frontiers: APOLLO achieves the best 1B PPL at 13.53, RMNP offers the clearest quality–runtime compromise, and AdaFactor minimizes memory.APOLLO combines low memory with the best 1B PPL, while heavy matrix methods incur runtime overhead and family labels do not determine metric-level trade-offs.Source: §6.2.1 · pp. 56–57, Table 13 · p. 57, §6.2.2 · p. 58

6.2.2 Stage 1 Pareto Analysis over PPL, Runtime, and Memory

  • Pareto analysis shows that heavy matrix methods improve quality at substantial runtime cost, whereas RMNP retains strong PPL near the efficient frontier and APOLLO is the strongest memory-frontier point.AdaFactor uses the least optimizer-state memory but has only moderate PPL; lightweight T2 variants are preferable when runtime dominates, while T4 is strongest when memory dominates.Source: §6.2.2 · pp. 58–59, Figure 14 · p. 58

6.2.3 Stage 2: Generalization across Data, Context Length, and Architectures

  • The benchmark’s mechanism-level findings show that SOAP transfers across attention types, Muon is architecture-sensitive, and T1/T2 generally form stronger long-context quality bands than T4.Muon moves from the middle on standard attention toward the top on GLA, where it achieves the best CS Avg. at 340M; its matrix geometry interacts with target parameter topology.Source: §6.2.3 · pp. 61–62, Figure 16 · p. 61
  • SOAP is the most stable long-context optimizer, while APOLLO reverses from Stage-1 champion to near-worst across eight FineWeb-Edu scenarios; downstream CS Avg. remains scenario-dependent.SOAP holds the top PPL position in seven of eight scenarios and never leaves the top two, whereas APOLLO is last or near-last under both PPL and CS Avg.; MARS-AdamW, RMNP, and Muon lead selected CS Avg. scenarios.Source: Table 14 · p. 60, §6.2.3 · pp. 61–62

6.2.4 Scenario Sensitivity of Memory-Efficient Optimizers

  • APOLLO’s PPL rises from 13.53 at 256 tokens to 35.40 at 32k, a +21.87 degradation versus AdamW’s +7.39, because low-rank state compression loses information as long-context gradient structure becomes more complex.Lion degrades by +6.29, slightly less than AdamW, and the comparison fixes the dataset and LLaMA architecture while varying context length.Source: §6.2.4 · pp. 62–63, Table 15 · p. 63

6.2.5 Optimizer Robustness Across Vision Backbones

  • Vision results are backbone-dependent: AdaBelief reaches 80.53% on ResNet50, Muon reaches 77.38% on DeiT-S, and Adan reaches 84.89% on CAFormer-S12.Muon improves DeiT-S over AdamW by more than five percentage points, while adaptive-moment methods remain broadly competitive and MARS-Lion falls to 33.70 on DeiT-S.Source: §6.2.5 · pp. 63–64

6.2.6 Auxiliary Stability Analysis from Gradient-Norm Dynamics

  • Muon attains the best aggregate gradient-norm stability rank, placing first or second in all eight scenarios, while GLA causes most optimizers’ GNormCV to rise to 10–160.All runs finish without NaN/Inf events, yet GNormCV ranges from ∼0.32 to >160, revealing architecture-driven soft instability that completion checks miss.Source: Figure 17 · p. 65, §6.2.6 · pp. 65–66

6.2.7 Auxiliary Learning-Rate Perturbation Robustness

  • Lion and MARS-Lion have the flattest local learning-rate responses, with sLR = 0.7% and 7.7%, whereas APOLLO has the largest sensitivity.The diagnostic is local around the tuned learning rate, and flatness can coexist with weak tuned quality; AdamW, MARS-Shampoo, and APOLLO are sensitive under perturbation.Source: §6.2.7 · pp. 66–67, Figure 18 · p. 67

6.2.8 Family-Level Objective Summary

  • Family profiles separate strengths: T2 offers matrix-structured quality and generalization, T3 offers efficiency and local LR robustness, T4 offers short-context memory efficiency, and T5 lacks consistent advantage.T1 remains a stable baseline, while APOLLO exemplifies short-context quality–memory gains that fail to transfer reliably to long-context training.Source: Figure 19 · p. 68, §6.2.8 · p. 68

6.2.9 Tiered Optimizer Summary

  • AdamW remains the default reference, RMNP the balanced matrix-structured alternative, SOAP the long-context quality ceiling, and Muon a strong interpretable optimizer requiring topology awareness.APOLLO and AdaFactor serve memory-constrained short-context training, while Lion supports low-cost exploration with an expected quality gap.Source: §6.2.9 · p. 69

6.3.1 Single-scene decomposition on C4-350M

  • On C4-350M, removing AdamW’s second moment raises PPL from 17.78 to 70.74, while adding Newton–Schulz orthogonalization recovers it to 16.86 and beats AdamW.Scaling and post-NS Nesterov refine the result to 16.60, while symmetric two-way scaling reaches the best single-scene result of 16.52.Source: Figure 20 · p. 70, §6.3.1 · pp. 70–71

6.2.9 Tiered Optimizer Summary

  • Optimizer selection is constraint-driven rather than a universal ranking: T1 is safest, T2 offers quality or stability ceilings, T3 favors runtime or local LR tolerance, and T4 targets short-context memory pressure.The tiered grouping summarizes short-context quality–efficiency, cross-scenario transfer, and mechanistic value, not a strict global leaderboard.Source: §6.2.9 · pp. 68, 70

6.3.2 Cross-scale and cross-architecture validation

  • Muon’s combined gains improve standard-Transformer PPL to 16.51 at 350M and 13.58 at 1B, but on Gated DeltaNet combination reaches 24.12, matching the best single gain.The gains transfer and partially stack across standard Transformers, whereas their additivity is architecture-dependent on Gated DeltaNet.Source: Table 18 · p. 71, §6.3.2 · pp. 71–72

6.3.1 Single-scene decomposition on C4-350M

  • Muon’s operations are order-constrained: momentum after orthogonalization worsens PPL to 23.01, and scaling before Newton–Schulz worsens it to 16.70.The ablation identifies Newton–Schulz orthogonalization as the core mechanism, with downstream scaling and Nesterov acting as geometry-aware refinements.Source: §6.3 · p. 70, §6.3.1 · p. 71

7 Discussion

The framework reframes optimizer choice as a technique-level, multi-objective design problem: mechanisms acting on different stages or axes tend to compose, while shared-slot mechanisms can conflict. Its conclusions are bounded by tested instances, controlled protocols, and qualitative mechanism attribution, motivating predictive diagnostics, architecture-aware transfer, and standardized evaluation.

7.1 Limitations

  • The study supports claims primarily on language-model pretraining and a targeted CIFAR100 cross-architecture check, while broader modality conclusions remain future work.The language experiments span 60M–1B parameters, four architectures, and 256–32k-token contexts; the vision study covers ResNet50, DeiT-S, and CAFormer-S12.Source: §7.1 · p. 72
  • Conclusions are protocol-relative because only optimizer hyperparameters were tuned, baseline retuning can erase gains, long-context comparisons lack matched token budgets, and coverage includes 24 of over 100 methods.Mechanism attributions are also qualitative, motivating metrics for effective rank, basis staleness, and intrinsic versus protocol-induced benefit.Source: §7.1 · pp. 72–73

7.2 Technique-Level Lessons

  • Mechanisms acting on different pipeline stages or axes tend to compose, whereas shared-slot mechanisms require explicit ordering and may conflict.This locality principle shifts optimizer design from choosing a universally best method to composing techniques for a specific budget, context length, and architecture.Source: §7.2 · pp. 73–74
  • Spectral orthogonalization, structured state, and variance reduction carry the strongest gains, while geometry-preserving factorization and quantization provide reliable memory savings.Muon’s orthogonalization recovers and surpasses AdamW quality, SOAP transfers strongly at high runtime and memory cost, and MARS improves state estimation with extra computation.Source: §7.2 · p. 73
  • Low-rank projection and spectral orthogonalization can conflict, fused streaming can invalidate gradient-statistics mechanisms, and APOLLO’s degradation reaches +21.87 PPL at 32k tokens.Muon’s gains are architecture-conditional and operator-order dependent; APOLLO is best at 256 tokens but worst at 32k as effective gradient rank rises.Source: §7.2 · p. 74

7.3 Open Problems and Future Directions

  • Future work should develop predictive diagnostics that adapt compression to rising gradient rank, measure preconditioner staleness, and distinguish intrinsic algorithmic gains from tuning artifacts.These diagnostics aim to prevent short-context memory gains from becoming long-context quality failures and to make the taxonomy predictive.Source: §7.3 · p. 75
  • Open directions include architecture-aware geometry transfer, cheaper curvature estimation, and standardized matched-budget protocols separating algorithmic progress from tuning effects.These directions target measurable architecture and training-dynamics properties, clarify when curvature overhead is justified, and improve long-context and learning-rate-transfer comparisons.Source: §7.3 · p. 75
  • Optimizer selection should become Pareto- and cost-aware, while compositional search automates axis-compatible combinations, same-slot ordering, and co-design of state compression, variance reduction, and geometry.The open problem arises because O1–O6 trade off and no method is uniformly best.Source: §7.3 · p. 75

8 Conclusion

OmniOpt concludes that optimizer choice is a constraint-driven composition rather than a universal ranking, enabled by a shared pipeline, LMO geometry, dual taxonomy, and cross-domain benchmark. It provides practical selection guidance and an operational coordinate system for future optimizer design.

Community contribution

  • OmniOpt’s coordinate system locates existing and future optimizers by pipeline stages, four-axis mechanisms, and objectives, enabling comparison, composition prediction, and constraint-aware design.Future work includes quantitative interpretability diagnostics and extending the benchmark beyond language modeling.Source: §8 · pp. 76–77

Main conclusion

  • There is no universal best optimizer; geometry-sensitive direction maps and structured state yield the strongest tested quality gains, while scalar AdamW refinements contribute little.Most element-wise AdamW variants fail to outperform a retuned baseline.Source: §8 · p. 76

Design constraints

  • Aggressive state compression is rank-bounded, spectral geometry is architecture-conditional, and composability depends on locality across the four axes and pipeline stages.Different axes or stages can stack, whereas methods competing for the same slot require explicit ordering.Source: §8 · p. 76

Method guidance

  • AdamW is the stable, inexpensive reference; RMNP [26] targets architecture-aware quality–efficiency balance, while SOAP offers the highest quality and transferability at substantially greater cost.AdamW is not strongest on any single metric, RMNP exploits row-wise structure, and SOAP suits quality-dominant regimes without compute or memory bottlenecks.Source: §8 · p. 76
  • Muon is a transparent matrix method requiring topology awareness, whereas AdaFactor is a safe low-memory baseline and APOLLO trades short-context gains for long-context collapse.APOLLO is therefore high-reward but high-risk, and its short-context performance does not certify it generally.Source: §8 · p. 76

Selection rule

  • The decision rule is to identify the run’s binding constraint—stability, quality, runtime, memory, or transfer—and select the method whose dominant strength addresses it.Starting from AdamW, practitioners should move to RMNP, SOAP, or a memory-efficient method only when a specific constraint demands it.Source: §8 · pp. 76–77

Appendix · A Additional Experimental Hyperparameter Configurations

This appendix documents the main optimizer hyperparameters used in the experiments to improve benchmark reproducibility, complementing the main text’s results on performance, runtime, and optimizer-state memory.

A Additional Experimental Hyperparameter Configurations

  • The appendix lists learning rates, momentum coefficients, numerical stability constants, and method-specific auxiliary parameters for the benchmark training configurations.These configurations complement the main text’s reported perplexity, downstream performance, per-step optimizer runtime, and optimizer-state memory.Source: A Additional Experimental… · p. 86

A.1 Stage-1 C4-LLaMA Short-Context Screening

Stage-1 C4-LLaMA short-context screening used tuned, scale-specific optimizer configurations, with core hyperparameters and method-specific auxiliary settings documented in Table 19.

A.1 Stage-1 C4-LLaMA Short-Context Screening

  • Table 19 reports each optimizer’s momentum coefficients, numerical stability constant, selected learning rate at each model scale, and auxiliary settings such as projection rank and interval.The table note summarizes method-specific auxiliary configurations.Source: §A.1 · p. 86

A.2 Stage-2 FineWeb-Edu 32k Long-Context Experiments

This section reports the best-run hyperparameter configurations for Stage-2 FineWeb-Edu 32k experiments at 340M and 1B model scales.

A.2 Stage-2 FineWeb-Edu 32k Long-Context Experiments

  • Tables 20 list the best-run learning rates, momentum coefficients, numerical stability constants, and optimizer-specific auxiliary parameters for the 340M and 1B experiments.The optimizer-specific parameters are encoded in the log filenames.Source: §A.2 · p. 86

B Detailed Stage-2 Commonsense Reasoning Results

This section reports detailed Stage-2 Commonsense Reasoning results across four architectures at 340M and 1B scales. It defines CS Avg. as the mean across ten commonsense benchmarks.

Detailed results

Evaluation metric

  • CS Avg. averages performance on ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA, OpenBookQA, BoolQ, COPA, LAMBADA-OpenAI [85], and SciQ [112].The metric aggregates ten named tasks into a single commonsense-reasoning summary.Source: B Detailed Stage-2 Commonsense… · p. 86

Related papers

Loading 2607.04033v1…