PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Siyuan Li, Jiabao Pan, Yumou Liu, Zhuoli Ouyang, Xin Jin, Xinglong Xu, Jingxuan Wei, Shengye Pang, Jintao Che, Xuanhe Zhou, Conghui He, Cheng Tan

arXiv:2607.04033v1cs.LGcs.AI

TL;DR

Optimizer research is fragmented across mechanisms and protocol-sensitive evaluations, complicating comparisons among methods. OmniOpt introduces shared pipeline and geometric views plus a cross-domain benchmark, finding that optimizer rankings systematically cross with scale, context length, and domain.

  • Problem

    Optimizer studies use fragmented mechanism labels and protocol-sensitive evaluations, limiting unified comparison across update stages and training conditions.

  • Method

    OmniOpt unifies optimizers with a five-stage update pipeline, norm-constrained LMOs, a multi-axis decomposition, and a cross-domain benchmark across model scales, architectures, tasks, and objectives.

  • Results

    Optimizer rankings exhibit systematic crossings with scale, context length, and domain across the benchmark’s evaluated effect objectives.

  • Takeaways & Limitations

    Optimizer selection should be assessed by mechanism and objective under the relevant scale, context length, and domain rather than by a single universal ranking.

  • Takeaways & Limitations

    The norm-constrained LMO discards gradient magnitude, requiring its scale to be reintroduced through the learning rate, radius, or a preconditioner.

Abstract

from arXiv · show

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.

1 Introduction

OmniOpt frames optimizer selection as a system-level decision shaped by training constraints and introduces a mechanism-aware framework linking a five-stage meta-pipeline, LMO geometry, dual taxonomies, and cross-domain benchmarking. Its benchmark emphasizes that optimizer rankings can change with scale, context length, and domain rather than forming a stable global ordering.

  • Motivation: Optimizer choice depends jointly on empirical-loss reduction and constraints including accelerator, memory, communication, batch-size, data-scale, tuning, and evaluation budgets.Adam and AdamW established the dominant element-wise adaptive-moment template for deep learning [54] [70].
  • Framework: The LMO-driven four-axis decomposition unifies optimizer directions through update domain, state estimator, geometry-and-precondition operator, and finalization wrapper.It uses norm-constrained linear minimization oracles to provide a common geometric language for direction selection.
  • Taxonomy: OmniOpt classifies more than one hundred optimizers by primary mechanism family and by the measurable training objectives they aim to improve.The methodological dimension is non-overlapping, while the objective-oriented dimension records intended training effects.
  • Benchmark: The benchmark spans language-model pretraining from 60M to 1B parameters, four architectures, context lengths from 256 to 32k tokens, and CIFAR100 image classification.Evaluation compares representative optimizers at family and axis levels across six effect objectives.
  • Findings: Optimizer rankings exhibit systematic crossings with scale, context length, and domain, motivating effect-specific comparisons rather than a single global leaderboard.The paper connects mechanism-aware organization, LMO decomposition, dual taxonomies, and benchmarking into an integrated whole.

2 Preliminaries

This section fixes the stochastic optimization framework, Transformer parameter geometry, and memory conventions, then establishes classical preconditioning mechanisms and AdamW as the baseline for later optimizer analyses.

  • 2.1 Training objective and stochastic gradients: Pretraining and supervised fine-tuning are formulated as empirical risk minimization with mini-batch stochastic gradients whose noise and numerical behavior reflect system details such as accumulation, clipping, and mixed precision.The stochastic-oracle model assumes unbiased gradients with bounded variance, while practical pipelines determine the update rule’s noise level, numerical range, and memory traffic.
  • 2.1 Transformer parameter topology: Transformer optimizers must account for heterogeneous tensor geometry: matrices support structured operations such as factorization and orthogonalization, unlike vector-like parameters.The distinction is defined by tensor geometry rather than layer names, motivating matrix-structured optimizer designs.
  • 2.1 Memory budget under mixed-precision training: A 7B-parameter model requires about 56 GB for Adam states alone because Adam stores two FP32 state tensors, excluding activations, temporary buffers, and distributed-training overhead.This motivates state-compressed optimizer methods under mixed-precision memory constraints.
  • 2.2 Classical optimization mechanisms: The preliminaries organize optimizer design around gradient smoothing, curvature estimation, direction scaling, and final-write regularization, with Newton’s method providing the preconditioning perspective.SGD uses identity geometry, momentum smooths the gradient signal, and modern methods replace exact Hessian inversion with tractable diagonal or structured approximations.
  • 2.2 Classical optimization mechanisms: AdamW [70] is the survey’s Adam-like baseline: it retains Adam’s diagonal data-gradient preconditioner while decoupling weight decay from adaptive scaling and applying it in the final assignment.Later optimizers are commonly compared against this AdamW-inherited state-evolution template.
  • 2.2 Classical optimization mechanisms: Natural gradient extends the Newton-style view by using Fisher geometry, while practical diagonal, block-diagonal, Kronecker, low-rank, and stochastic approximations make it usable for large models.The resulting curvature and geometry choices become Axis II and Axis III in the paper’s LMO-driven decomposition.

3 Unified Theoretical Framework for Optimizers

OmniOpt unifies optimizer analysis through a five-stage meta-pipeline that identifies where methods intervene and an LMO-based geometric view that links update directions to norm-ball constraints. Together, these views show that most optimizers modify only one or two stages while placing diverse mechanisms within a common direction-operator framework.

  • 3.1 Universal Meta-Pipeline: The Universal Meta-Pipeline represents each optimizer step as five operators transforming a training signal into a parameter increment, with optimizer state updated internally and identity stages allowed.The pipeline formalizes parameter tensors, training signals, optimizer state, and the update equation Δ_t = S5(S4(S3(S2(S1(G_t))))).
  • 3.1 Universal Meta-Pipeline: The pipeline separates signal acquisition, parameter routing, gradient transformation, state evolution, and update reconstruction, while S2 and S4 form a dual pair for dimension-reducing or basis-rotating transformations.The return map is exact for orthogonal basis rotations and approximate for projection or factorized-state methods; GaLore relocates computation and state into a lower-dimensional space through this S2/S4 dual.
  • 3.1 Universal Meta-Pipeline: Most optimizers make nontrivial choices at only one or two pipeline stages, leaving the remaining stages as identity maps or standard defaults.This identity-mapping principle supports compact cross-method characterization and allows coupled optimizer families to span multiple stages.
  • 3.2 LMO-Driven Geometric View: The LMO view interprets optimizer behavior geometrically: the same gradient produces different extremal directions under different norm balls, while magnitude must be restored through the learning rate, radius, or preconditioner.This connects norm-constrained LMOs with steepest descent and treats constraint geometry and optimizer behavior as two readings of one direction-selection mechanism.
  • 3.2 LMO-Driven Geometric View: The framework groups LMO and preconditioning interpretations into a direction operator Φ_t, while moving momentum, second moments, variance reduction, and projection state earlier as inputs to that operator.Under this organization, Muon is simultaneously a spectral-norm LMO and a Gram-based preconditioner.

L MtQR, (39)

L MtQR performs Adam-style optimization in a low-dimensional projected subspace, reducing state memory while lifting the adaptive direction back to the original parameter space. Its construction is simultaneously a projected adaptive-box LMO and a rotated diagonal preconditioner.

  • L MtQR: L MtQR stores projected first- and second-moment states in an r × r subspace, reducing memory from O(mn) to O(r^2).The projected momentum, first moment, and coordinate-wise variance estimate are maintained inside the small subspace.
  • L MtQR: The method forms an adaptive direction in projected coordinates, then lifts it to the original weight space through the subspace bases.As an LMO, it is a projected adaptive box; as a preconditioner, it is a projected diagonal metric rotated back by the bases.
  • SOAP: SOAP rotates Adam’s box into eigenbasis coordinates, refreshing full-rank bases every f steps while updating its second moment each step.Unlike Shampoo’s fixed inverse-root metric, SOAP lets the per-step second-moment state provide adaptivity, supporting robustness at low preconditioning frequency.
  • Summary: The four-axis decomposition separates update domain, state estimation, direction formation, and write-back, with momentum and variance reduction assigned to Axis II.Axis III contains the LMO geometry and direction operators, including Adam’s diagonal metric and Muon’s polar map.

4 Dual-Dimension Taxonomy

OmniOpt organizes optimizers with two complementary dimensions: a single primary mechanism family and multi-label effect objectives. This dual taxonomy aligns families with the meta-pipeline and provides a mechanism-aware plan for benchmarking and classifying new methods.

  • Taxonomy design principles: Each optimizer receives one primary mechanism label, while effect objectives may be multi-label annotations covering convergence, cost, memory, stability, robustness, and generalization.This prevents mechanisms from being confused with their intended or observed effects, which need not improve together.
  • Taxonomy design principles: The taxonomy resolves overlapping descriptions by assigning methods according to the component whose removal collapses them to a simpler baseline.For example, Lion is classified by its sign-map mechanism rather than its memory-saving effect, while Q-GaLore adds quantized state to GaLore’s subspace mechanism.
  • Dimension A: mechanism families: The family assignments distinguish mechanisms including element-wise adaptive moments, matrix-level structure, directional quantization, state compression or gradient-lifetime changes, and selective or wrapped update operations.Examples include Lion’s sign geometry, GaLore’s subspace mechanism, SAM’s sharpness-aware wrapper, LAMB’s trust-ratio writeback, and selective masks such as Magma and MGUP.
  • Dimension A: mechanism families: Dimension A organizes 108 optimizers into five mechanism families and fifteen subclasses, with boundaries aligned to the five-stage meta-pipeline.T1 centers on element-wise state evolution and scaling; T2 on matrix routing and transformation; T3 on direction discretization; T4 on state representation and gradient lifetime; and T5 on update writeback.
  • Dimension B and benchmark use: The resulting taxonomy is a mechanism-informed benchmark plan: Table 7 supplies objective priorities, while detailed records combine primary labels, secondary mechanism tags, and O1–O6 effect annotations.Cross-family combinations may require explicit ordering and descent-quality arguments when mechanisms occupy the same pipeline slot.

5 Optimizer Method Families · 5.1 T1: Element-Wise Adaptive Moment and Scalar Control

OmniOpt organizes optimizer methods into five mechanism families, with T1 covering element-wise adaptive-moment and scalar-control methods. T1 preserves coordinate-wise topology while varying local adaptive terms, temporal signals, or outer control logic.

  • 5 Optimizer Method Families: The five method families share a common template covering meta-pipeline position, LMO-driven interpretation, representative methods, and effect-target assessment.This organization preserves compact progression while retaining method-level comparison granularity for benchmark design.
  • 5.1.1 Family Overview and Meta-Pipeline Position: T1 is defined by scalar or coordinate-wise state, independent parameter normalization or control, and no information exchange across rows, columns, heads, or layers.Its canonical AdamW structure uses a diagonal second-moment preconditioner and preserves parameter topology; T1 is mostly identity-like in S1, S2, and S4, with major variation in S3 and S5.
  • 5.1.1 Family Overview and Meta-Pipeline Position: T1 divides mechanistically into direct Adam variants, multi-timescale or corrected estimators, and iterate-averaging or automatic-tuning methods.The subclasses modify AdamW’s local numerator, denominator, bias correction, or writeback; temporal or variance-reduced signals; or outer averaging, scale inference, and phase switching.
  • 5.1.2 LMO-Driven Four-Axis Interpretation: In the four-axis view, T1 uses the identity basis, diagonal curvature, and full coordinate-resolution state, making AdamW a diagonal-curvature analogue without off-diagonal coupling or basis changes.Its normalized direction can be interpreted through an ℓ∞-type LMO and an anisotropic coordinate-wise trust region, so AdamW is neither simply sign descent nor Newton descent.
  • 5.1.3 Representative Methods: Representative T1 methods modify AdamW locally, add temporal or corrected signals, or alter outer control through averaging, global-scale inference, or switching.Examples include RAdam’s early-variance rectification, AdaBelief’s prediction-error statistic, AdEMAMix’s multiple momentum timescales, MARS’s corrected estimator, TAM’s alignment gate, and Schedule-Free’s internal averaging.
  • 5.1.3 Representative Methods: T1 serves as the calibration family because AdamW combines coordinate-wise normalization, momentum smoothing, decoupled weight decay, and a mature schedule interface.The trade-off is memory: AdamW stores two full-size moment buffers, making it more expensive than SGDM while often supporting strong early loss reduction and stable Transformer pretraining.
  • 5.1.3 Representative Methods: T1.1 typically offers fast early loss reduction and stable behavior under heterogeneous coordinate scales, while T1.2 targets smoother trajectories and token efficiency under noisy or temporally biased gradients.These benefits come with full element-wise state for Adam-like methods or extra state, update logic, and per-step cost for temporal and corrected estimators.
  • 5.1.3 Representative Methods: T1.3 prioritizes wider usable learning-rate ranges, fewer schedule decisions, and smoother phase behavior rather than necessarily achieving lower best-tuned loss.Automatic rules may trail carefully tuned fixed schedules when sufficient search budget is available.

5.2 T2: Matrix-Level Structural Methods

T2 comprises optimizers that operate on Transformer weight matrices rather than independent coordinates, using spectral, Kronecker-factored, or low-rank subspace mechanisms. Its benefits and costs are multi-objective, spanning loss, conditioning, memory, step time, and information preservation.

  • Family definition: T2 methods directly couple matrix rows, columns, singular directions, Kronecker factors, or low-rank subspaces, while non-matrix parameters typically fall back to AdamW or SGD.This matrix routing is the S1 parameter-scoping stage and targets attention and MLP projection weights most naturally.
  • Subclass structure: T2 splits into spectral orthogonalization, Kronecker-factored preconditioning, and low-rank subspace projection, represented centrally by Muon, Shampoo or SOAP, and GaLore, respectively.The subclasses differ by their primary matrix operation: spectral direction selection, structured curvature or covariance scaling, and projected-state training with reconstruction.
  • Meta-pipeline: T2 makes parameter routing, matrix transformation, structured state, and reconstruction first-class design choices, while final writeback often remains SGD- or AdamW-style.Its generic matrix update rotates or projects gradients, applies a direction or preconditioning rule, and maps the transformed update back to the parameter matrix.
  • Mechanism distinctions: T2.2 uses row, column, or Fisher-coordinate magnitude information, whereas T2.3 makes low-rank projection the primary state space and treats memory savings as an effect.Low-rank methods require sufficient rank, refresh frequency, or rank-allocation strategies to avoid losing high-rank gradient information.
  • Trade-offs: T2 has no single upgrade profile: spectral methods may improve loss per token but increase step time, Kronecker methods may lose wall-clock benefits, and low-rank methods trade memory for rank or refresh requirements.Accordingly, evaluations should match token budgets, wall-clock accounting, memory budgets, rank, refresh interval, and batch size against relevant baselines.

5.3 T3: Discretization and Directional Quantization

T3 groups optimizers whose defining operation discretizes or smooths the update direction through sign-like maps, removing or coarsening continuous magnitude. Its geometry follows the ℓ∞ linear minimization oracle, while variants differ through momentum signals, smoothing, trajectory control, or bounded magnitude channels.

  • Family definition: T3 defines updates through hard or smooth sign-like maps that preserve coordinate orientation while discarding, clipping, or coarsening continuous gradient or momentum magnitude.This distinguishes T3 from adaptive-moment methods whose defining mechanism remains variance estimation and diagonal scaling.
  • Mechanism groups: The family comprises pure sign directions, signed momentum, and smooth or hybrid directions represented by SignSGD, Signum [11], Lion [19], MARS-Lion [128], RLion, FOCUS [68], and Ano [50].These routes respectively alter the input signal, sign decision, trajectory behavior, or magnitude channel while retaining a directional-quantization core.
  • Family boundary: T3 includes methods whose sign-like map directly defines the base direction, but excludes binary or stochastic masks applied after AdamW-, Lion-, or Muon-style updates, which belong to T5.3.This boundary places SignSGD, Lion, RLion, FOCUS, and Ano in T3 while assigning Magma, MGUP, cautious, and top-k filters elsewhere.
  • Geometry and LMO: Under the ℓ∞-ball LMO, d* = −sign(g), so sign updates are steepest linearized steps under fixed maximum coordinate displacement in the identity basis.SignSGD applies this geometry directly, while Lion applies it to a smoothed current-gradient and momentum signal, using less state than Adam’s two-buffer design.
  • Practical trade-offs: T3 primarily offers resource efficiency and local tolerance to learning-rate perturbations, while fixed-magnitude updates remain sensitive to learning rate, warmup, weight decay, scale, token budget, batch size, and tuning effort.Removing Adam’s second-moment state can reduce optimizer memory, but it does not guarantee a better loss trajectory or peak quality.

5.4 T4: State Compression and Structural Aggregation

T4 comprises optimizers that reduce optimizer-state memory, precision, granularity, or lifetime through factorization, low-bit representations, shared statistics, or fused gradient consumption. These methods primarily modify state maintenance and exposure, while their trade-offs depend on approximation quality, grouping granularity, and composition constraints.

  • Family definition: T4 reduces optimizer-state overhead by changing the optimizer’s state representation or gradient-consumption pipeline rather than relying on external systems techniques.Relative to AdamW’s two full-size state tensors, T4 compresses, shares, quantizes, reconstructs, or eliminates state during backpropagation.
  • T4.1 Row-column factorization: Row-column factorization methods AdaFactor and CAME replace full matrix second moments with row and column summaries, reconstructing approximate denominators at update time [99] [74].AdaFactor reduces matrix state from O(mn) to O(m + n), while CAME adds confidence-guided correction for unreliable factorized surrogates [99] [74].
  • T4.2 Low-bit quantization and error compensation: Low-bit methods 8-bit Adam and Q-GaLore preserve state structure while storing statistics or projections in quantized form, using scaling, rounding, or compensation to limit numerical error [27] [137].Q-GaLore is included for its additional quantization mechanism, while its low-rank update geometry belongs to T2.3 [138] [137].
  • T4.3 Block- and layer-level state sharing: Shared-state methods Adam-mini, APOLLO, SM3, Conda, and NovoGrad reduce adaptive-statistic dimensionality by sharing values across blocks, covers, columns, layers, or auxiliary structures [6] [111] [35].Adam-mini selects Hessian-motivated Transformer blocks, maintains one statistic per block, and broadcasts it to block coordinates. Finer sharing preserves heterogeneity but saves less memory, whereas coarser sharing risks underfitting heterogeneous curvature.
  • T4.4 Fused backprop-update: Fused backprop-update methods LOMO and AdaLOMO consume gradients as soon as they are produced, but streaming can conflict with global clipping, delayed basis refresh, SAM, and full-statistics preconditioners [76] [75].Their writeback occurs during the backward pass, giving T4.4 stronger composition constraints than ordinary state compression.

5.5 T5: Curvature-Aware and Geometric Regularization

T5 groups optimizers that impose curvature or geometric constraints through perturbed-gradient evaluation, curvature estimation, update filtering, or layer-wise trust scaling. Its methods are competitive in limited regimes but show no uniform advantage under the current LLM benchmark, with matched-compute comparisons remaining important.

  • Taxonomy: T5 covers four geometric subclasses: adversarial perturbation, diagonal Hessian estimation, update post-processing and selective filtering, and layer-wise trust-region scaling.Figure 13 presents the complete T5 membership taxonomy.
  • T5.1 Adversarial perturbation: SAM changes the gradient evaluation point through a local adversarial perturbation, while ASAM and later variants modify perturbation geometry, sharpness objectives, or probabilistic interpretations [32] [57] [142] [82].The perturbed-gradient branch can wrap SGD, AdamW, or another base optimizer, but ordinarily incurs a second forward-backward pass.
  • T5.2 Diagonal Hessian estimation: Sophia estimates diagonal curvature and clips the resulting ratio, while AdaHessian uses Hutchinson-style diagonal-Hessian estimation and Hessian-powered adaptive scaling [65] [123].Sophia-H uses stochastic Hessian-vector products, whereas Sophia-G uses a cheaper gradient-based curvature proxy.
  • T5.3–T5.4 Update geometry: Post-processing methods project, normalize, mask, stabilize, or selectively amplify updates, whereas LAMB and AGC apply layer- or parameter-group-level trust scaling [42] [114] [64] [15] [44] [125] [14].These operations generally leave the base direction geometry intact while constraining which coordinates are written back or rescaling the whole layer.
  • Benchmark caveat: T5 methods are competitive in limited regimes but do not show a uniform advantage under the current LLM benchmark, and fixed-step comparisons can change their apparent ranking.Matched-compute comparisons are essential because SAM-style methods may require two sequential forward-backward passes per step.

6 Benchmark Study

OmniOpt benchmarks optimizers across quality, runtime, memory, families, and architectures, revealing objective-dependent trade-offs rather than a universal winner. Stage 1 favors APOLLO and RMNP locally, while cross-architecture results identify SOAP as stable and APOLLO as the sharpest generalization failure.

  • Stage 1 and cross-architecture generalization: APOLLO is the best 1B short-context quality point at 13.53 and uses much less optimizer-state memory than AdamW, but it ranks last or near-last across all eight long-context scenarios.This reverses APOLLO’s Stage 1 short-context success and marks the benchmark’s sharpest generalization failure.
  • Stage 1 Pareto analysis: Stage 1 reveals distinct objective frontiers: T4 is strongest under memory constraints, lightweight T2 variants under runtime constraints, and T1 when neither cost dimension dominates.AdaFactor minimizes optimizer-state memory but has only moderate PPL, whereas APOLLO combines low memory with the best 1B PPL.
  • Stage 1 Pareto analysis: RMNP provides the clearest quality–runtime compromise among matrix-structured methods, staying much faster than heavy alternatives while retaining strong PPL.SOAP, Muon, MARS-Shampoo, and Shampoo incur substantial runtime overhead from matrix-level transformations or preconditioning.
  • Cross-architecture evaluation: Within-architecture ranking is used instead of raw perplexity comparisons, separating transferable optimizers from architecture-specific strengths and systematic failures.The benchmark analyzes optimizer and family rankings across architecture-scale pairs because absolute perplexity is not comparable across architectures.
  • Cross-architecture generalization: SOAP holds the top PPL position in seven of eight architecture-scale scenarios and never leaves the top two, making it the most stable cross-scenario optimizer.Its stability suggests that Kronecker or Fisher basis preconditioning transfers across standard-attention and linear-attention architectures.
  • Cross-architecture generalization: MARS-AdamW consistently outperforms AdamW and Adan, whereas Muon’s ranking depends on architecture, improving on GLA but weakening on Transformer++ and DeltaNet.MARS-AdamW also achieves the best CS Avg. on both Gated DeltaNet columns, while Muon achieves the best CS Avg. at 340M on GLA.

7 Discussion

OmniOpt reframes optimizer choice around mechanism locality, compatibility, and measurable objectives rather than a single best method. Its conclusions are informative but protocol-, coverage-, architecture-, and mechanism-attribution-dependent, motivating predictive diagnostics and multi-objective selection.

  • Scope and limitations: The study covers 24 representative optimizers, language-model pretraining from 60M to 1B parameters, and CIFAR100 backbones, but conclusions remain protocol- and tested-instance-relative.Only optimizer hyperparameters were tuned while architecture, data, and schedule were fixed; many apparent AdamW improvements shrink or disappear after baseline retuning.
  • Mechanism attribution: Mechanism explanations remain qualitative, motivating effective-rank, basis-staleness, and intrinsic-versus-protocol metrics to make compression, geometry, and tuning effects testable.The paper plans quantitative interpretability evaluation in a subsequent version.
  • Composition and locality: Mechanisms acting on different pipeline stages or axes tend to compose, while techniques sharing a slot require explicit ordering because they can interfere.This locality principle provides a practical compatibility test for optimizer composition.
  • Benefit carriers: PPL 17.78 →70.74 after removing Muon’s diagonal second moment, while Newton–Schulz orthogonalization recovers and surpasses AdamW at →16.86.The result associates the strongest gains with geometry-sensitive direction maps and structured state, rather than scalar control tweaks alone.
  • Limited returns: RAdam, NAdam, AdaBelief, and Prodigy occupy the weakest tuned-perplexity tier, while Sophia and LAMB are weak or only situationally competitive in this regime.These methods can add Hessian-vector products, clipping or trust-ratio parameters, or extra gradient evaluations.
  • Future selection framework: Because objectives O1–O6 trade off and no method is uniformly best, optimizer selection should become cost-aware and Pareto-aware, with transfer predicted from measurable architecture and data properties.Rankings cross dataset, context length, and architecture, especially for compression versus gradient rank and spectral geometry versus attention topology.

8 Conclusion

OmniOpt concludes that no optimizer is universally best: selection should match the training regime’s binding constraint. Its taxonomy provides an operational coordinate system for comparing existing and future optimizers under explicit mechanism and objective assumptions.

  • 8 Conclusion: OmniOpt’s operational coordinate system locates any optimizer by pipeline stages and four-axis coordinates, supporting composability predictions and explicit comparisons of mechanism and objective assumptions.The framework is intended to give practitioners a shared vocabulary for navigating a fragmented optimizer landscape.
  • 8 Conclusion: No universal best optimizer exists; geometry-sensitive direction maps and structured state drive the strongest quality gains, while scalar adaptive-moment refinements contribute little after baseline retuning.Most element-wise AdamW variants do not survive a retuned baseline under the tested protocol.
  • 8 Conclusion: AdamW is the stable, inexpensive, interpretable default reference for general-purpose pretraining, while alternatives should be selected only when a specific constraint demands them.The relevant constraint may be stability, quality, runtime, memory, or cross-scenario transfer.
  • 8 Conclusion: SOAP offers the quality ceiling for long-context training and transfers broadly across architectures, but its cost makes it appropriate only when final quality dominates compute and memory.RMNP [26] targets a quality–efficiency balance through architecture-induced row-wise structure, whereas Muon requires awareness of model topology.
  • 8 Conclusion: AdaFactor is a safe low-memory baseline at moderate quality, while APOLLO excels at short context but collapses at long context and therefore remains high-risk.Lion is a cheap exploratory option with an expected quality gap; Sophia and LAMB remain situational under the current protocol.

Appendix · A Additional Experimental Hyperparameter Configurations

This appendix documents the optimizer hyperparameters used in the benchmark to improve reproducibility, complementing the main text’s reported outcomes with training configurations.

  • A Additional Experimental Hyperparameter Configurations: The appendix reports the main optimizer hyperparameters used in the experiments to support benchmark reproducibility.It focuses on training configurations rather than the main text’s resulting metrics.
  • A Additional Experimental Hyperparameter Configurations: Configurations include learning rates, momentum coefficients, numerical stability constants, and method-specific auxiliary parameters.The main text instead reports perplexity, downstream performance, per-step optimizer runtime, and optimizer-state memory.

A.1 Stage-1 C4-LLaMA Short-Context Screening

The Stage-1 C4-LLaMA short-context screening used tuned, model-scale-specific optimizer configurations. Table 19 records core hyperparameters and method-specific auxiliary settings.

  • A.1 Stage-1 C4-LLaMA Short-Context Screening: Table 19 summarizes the tuned hyperparameter configurations used for each optimizer in the Stage-1 C4-LLaMA short-context screening.
  • A.1 Stage-1 C4-LLaMA Short-Context Screening: The configurations report momentum coefficients, the numerical stability constant, and the selected learning rate at each model scale.
  • A.1 Stage-1 C4-LLaMA Short-Context Screening: Method-specific auxiliary settings, including projection rank and projection interval, are provided in the table note.

A.2 Stage-2 FineWeb-Edu 32k Long-Context Experiments

Stage-2 FineWeb-Edu 32k experiments at 340M and 1B are documented through best-run hyperparameter configurations.

  • A.2 Stage-2 FineWeb-Edu 32k Long-Context Experiments: Tables 20 report the best-run learning rates, momentum coefficients, numerical stability constants, and optimizer-specific auxiliary parameters for the 340M and 1B experiments.The auxiliary parameters are encoded in the log filenames.

B Detailed Stage-2 Commonsense Reasoning Results

This section reports detailed Stage-2 Commonsense Reasoning results across architectures and model scales. It defines CS Avg. as the mean across ten listed evaluation tasks.

  • Evaluation metric: CS Avg. averages performance over ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA, OpenBookQA, BoolQ, COPA, LAMBADA-OpenAI [85], and SciQ [112].
Loading 2607.04033v1…