Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

• The review covers first-order, second-order, zeroth-order, and scenario-oriented paradigms, including methods that use function evaluations to avoid backpropagation's memory burden.Source: §1 · p. 3
• A controlled framework evaluates 23 optimizers across vision and language-model architectures, separating algorithmic performance from large-scale engineering optimizations.Source: §1 · pp. 3–4
• SGD collapses on highly anisotropic LLM loss landscapes, while Muon and MARS show superior hyperparameter robustness and cross-architecture generalization.Source: §1 · p. 3
• Prior surveys omit zeroth-order and scenario-oriented methods, lack a unified taxonomy, and provide no standardized large-scale benchmark for fair cross-architecture comparison.Source: §1 · p. 3
• The survey establishes a mathematical taxonomy of first-, second-, and zeroth-order methods, then analyzes how foundational algorithms are re-engineered for distributed and privacy-constrained scenarios.Source: §1 · pp. 3–4

2.1 Optimization Problem Setup

• Expected risk under unknown P is approximated by empirical risk on N i.i.d. training samples, while mini-batches provide stochastic gradient estimates for updates.Source: §2.1 · p. 5
• For convex objectives, GD has O(1/T) convergence; for strongly convex objectives, GD is linear, while SGD's noise leaves a fixed error radius unless its rate decays.Source: §2.1 · p. 5

2.2 Deep Learning Optimization

• DNN losses are non-convex, and stochastic gradient noise can help SGD escape prevalent saddle points; under stated infinite-width conditions, NTK theory approximates training by kernel regression.Source: §2.2 · pp. 5–6
• Figure 2 categorizes annual algorithms as FO, SO, and ZO and tracks total counts; its 2026 statistics are incomplete, covering data only through April.Source: Figure 2 · p. 6

2.3 Evaluation Metrics

• Evaluation considers FLOPs, activation and optimizer-state memory, and privacy; DP-SGD's clipping and Gaussian noise provide privacy guarantees but often reduce convergence speed and final accuracy.Source: §2.3 · p. 6

2.4 Optimization for Large Models

• Large-model training uses FP16 or BF16, parallelism, and gradient accumulation; FP16 needs loss scaling, while BF16 typically does not.Source: §2.4 · p. 6

2.5 History and Roadmap

• Figure 3 maps algorithmic milestones, institutions, and venues; the trajectory runs from SGD and K-FAC toward memory-efficient ZO and hardware-aware Lion, Muon, and LOZO.Source: §2.5 · pp. 6–7, Figure 3 · p. 7

3 Optimization Algorithms

• The review groups methods as first-, second-, and zeroth-order, then classifies scenario-oriented approaches for constraints such as distributed communication and differential privacy.It frames these primitives as components that can be re-engineered to address memory, communication, and privacy bottlenecks.Source: §3 · p. 7
• The section uses unified mathematical notation to distinguish methods and make their improvements over predecessors explicit.Its evolutionary figures cover first-, second-, and zeroth-order algorithms, alongside scenario-based classifications.Source: §3 · p. 7
• Base methods progress from mini-batch gradients in SGD to momentum, lookahead gradients in NAG, and adaptive learning-rate mechanisms in AdaGrad, RMSprop, and Adadelta.Momentum accumulates historical gradients; AdaGrad accumulates squared gradients, while RMSprop replaces direct accumulation with an exponential moving average and Adadelta removes the manually set global learning rate.Source: §3 · p. 7, Figure 4 · p. 8
• Existing surveys often group methods too coarsely, provide little scenario-specific guidance, and overlook optimization strategies beyond narrow gradient updates.The broadened scope includes zeroth-order estimation for memory efficiency and communication-efficient compression for distributed systems.Source: §3 · pp. 7–8

3.1 Unified Mathematical Perspective

• The formulation represents optimization methods as instances of a discrete-time dynamical system and separates the process into four dimensions.The dimensions cover gradient estimation, preconditioning, scenario transformation, and structural projection.Source: §3.1 · p. 8
• The gradient estimator may use first-order stochastic gradients or zeroth-order finite-difference forward queries that bypass backpropagation.The preconditioner warps the gradient to accelerate convergence in ill-conditioned spaces.Source: §3.1 · p. 8
• Scenario-aware transformations encode constraints through gradient clipping and Gaussian noise for privacy or lossy compression for distributed optimization.Structural projection can restrict updates to a low-dimensional subspace to reduce dense optimizer-state storage.Source: §3.1 · p. 9
• The framework is a modular decoupling perspective, not a universal convergence proof, and it identifies structural equivalences and theoretical gaps across optimization paradigms.It connects low-rank projection in memory-efficient ZO methods such as LOZO with low-rank gradient decomposition in FO preconditioning methods such as ASGO.Source: §3.1 · p. 9
• The framework identifies a design opportunity to combine memory-efficient gradient estimators with structured or localized preconditioners to pursue curvature-aware optimization under memory constraints.The passage notes that empirical-Fisher Shampoo captures gradient variance rather than true geometric curvature.Source: §3.1 · p. 9

3.2 First-Order Algorithms

• First-order methods build adaptive transformations from first-order gradient information, and the review organizes their development into eight themes spanning distinct optimization strategies.The paper traces representative algorithms and formulations in Figure 5 and compares methods in Table 2.Source: §3.2 · pp. 9–10, Figure 5 · p. 11

3.2.1 Accelerating Convergence Rate

• Momentum variants extend basic accumulation through scheduled resets, lookahead or bias correction, and multiple momentum states, improving adaptability or gradient estimates at the cost of tuning, computation, or memory.Reset triggers may interrupt useful gradient history, lookahead can be unstable under high gradient variance, and multiple states add memory overhead.Source: §3.2.1 · pp. 10–11
• Momentum research also uses gradient alignment and dynamic or frequency-domain modulation, but lacks a universal hyperparameter-free framework and faces theory-to-practice gaps under deep-learning conditions.Dynamic weights incur per-iteration computation; frequency-domain methods seek to suppress harmful high-frequency noise, while their implementation in nonlinear networks is difficult.Source: §3.2.1 · pp. 11–12

3.2.2 Adaptive Step-Size Control

• Adaptive step-size methods refine temporal estimates through moment and epsilon adjustments, prediction-deviation scaling, and momentum, while noisy batches and inconsistent historical momentum can undermine their updates.These strategies target unstable or extreme step sizes, but added auxiliary hyperparameters can make convergence more sensitive.Source: §3.2.2 · p. 14
• Adaptive methods expand from layer-wise to neuron-level scaling and combine or decouple optimizer components, addressing structural heterogeneity while introducing assumptions, memory demands, or analytical complexity.The section also identifies growing historical-state overhead as motivation for stateless adaptation.Source: §3.2.2 · pp. 14–15
• AdamS replaces temporal variance with a momentum-current gradient denominator to match SGD’s memory efficiency while retaining AdamW’s configuration; AEGDM merges momentum with transformed gradient sums for energy stability.These adaptive methods approach first-order memory limits but rely strictly on locally stationary gradient distributions; AutoDrop adjusts rates through angular velocity saturation without supplementary hyperparameters.Source: §3.2.2 · p. 15
• RLEKF reorganizes network layers for Kalman covariance tracking and approximates the dense error matrix with sparse diagonal blocks, using second-order covariance information but accumulating truncation errors across deep architectures.Source: §3.2.2 · p. 15
• Adaptive step-size control evolved from temporal moment refinement to layer- and neuron-level normalization, then toward stateless and hybrid adaptations that decouple momentum, variance, and weight decay.These later designs prioritize efficiency in large-scale pre-training.Source: §3.2.2 · pp. 15–16
• Hybrid and structurally adaptive methods’ convergence guarantees largely rely on convexity or bounded smoothness, assumptions that poorly characterize modern deep networks’ non-convex, singular landscapes.The section identifies this mismatch as a critical gap in existing theory.Source: §3.2.2 · p. 16

Preconditioning framework

• Preconditioning multiplies gradients by structure-aware approximations, increasingly viewed as variance adaptation because the empirical FIM captures gradients’ noncentral second moment.The framework aims to reduce stochastic gradient noise while preserving parameter structure.Source: §3.2.3 · p. 16

Preconditioners based on two metrics

• Dual-metric preconditioners capture row and column structure and can accelerate convergence, but storing and inverting covariance matrices imposes substantial memory overhead.Shampoo maintains separate dimension-wise preconditioners from second-order gradient statistics and achieves faster convergence with comparable per-step runtime.Source: §3.2.3 · p. 16

Preconditioners based on a single metric

• Single-metric methods reduce resource demands and can improve conditioning under convex assumptions, but represent inter-dimensional correlations less fully than dual-metric designs.The passage describes lower memory footprint and faster convergence under convex assumptions.Source: §3.2.3 · p. 16

Design trade-off

• The design trajectory shifts from dimension-wise Kronecker-factored preconditioning toward single-matrix approaches to scale to modern architectures.The former offers structural awareness at high memory cost; the latter is memory-efficient but has limited curvature representation.Source: §3.2.3 · p. 16

Open challenge

• A remaining gap is a dynamic, hardware-friendly preconditioner matching dual-metric representational capacity with single-metric efficiency, without complex tuning or quantization-related precision loss.These are identified as unresolved design requirements.Source: §3.2.3 · p. 17

Basic fixed gradient clipping

• Fixed clipping bounds gradient norms with a predefined threshold, supporting noise resilience and privacy bounds but risking gradient bias and suboptimal convergence.Clipped-SGD targets heavy-tailed noise, while DP-SGD combines clipping with Gaussian noise injection for differentially private non-convex optimization.Source: §3.2.4 · p. 17

DP-enhanced gradient clipping

• Error-feedback clipping addresses bias from rigid truncation, but persistently tracking and integrating feedback states increases computational overhead.The method is described as decoupling privacy thresholds from optimization utility.Source: §3.2.4 · p. 17

Dynamic and spike-aware clipping

• Dynamic clipping adapts thresholds from historical gradients, while spike-aware clipping targets abrupt anomalies but is mainly suited to extreme-gradient-variance regimes.Historical-statistics-based thresholds may lag during rapid loss-landscape shifts.Source: §3.2.4 · p. 17

Element-wise and layer-wise normalization

• Element-wise scaling offers flexible parameter updates but requires per-parameter statistical state, whereas layer-wise normalization reduces memory demands while assuming uniform behavior within each layer.Layer-wise normalization may suppress heterogeneous feature learning.Source: §3.2.4 · pp. 17–18

Mean-removal and noise-robust normalization

• Mean-removal methods seek smoother paths and better generalization, but their symmetry assumption can degrade performance under heavily skewed gradient noise.The section also describes noise-robust methods as mitigating batch-size constraints, with convergence analysis often requiring strict landscape-smoothness assumptions.Source: §3.2.4 · p. 18

Open challenge

• A central open gap is a proactive framework coupling stability constraints to loss-landscape geometry, rather than relying on reactive historical observations and auxiliary tuning.The passage presents this as a missing link between local gradient corrections and global topological guarantees.Source: §3.2.4 · p. 18

Learning-rate scheduling framework

• Scheduling dynamically adjusts optimizer step sizes to balance convergence speed and training stability across optimization phases.The section represents scheduling as θt+1 = θt −ηtΦ(g1:t, Ht), with Φ conditioned on gradient history and structural priors.Source: §3.2.5 · p. 18

Lightweight scaling and batch-aware scheduling

• Signal-to-noise scheduling reduces memory consumption by half compared with previous approaches while preserving competitive performance.LAMB uses layer-wise adaptive learning rates for large-batch training, preventing performance degradation on tasks requiring rapid convergence and high accuracy.Source: §3.2.5 · pp. 18–19

Element adaptation and gradient angle scheduling

• AdaBound bounds element-wise rates, while ACMo and HGM use gradient-angle information to adjust directions or rates; continuous evaluation adds computational overhead.AdaBFE also adapts rates per parameter dimension using forward-loss information.Source: §3.2.5 · p. 19

Loss-sensitive and stability-aware scheduling

• DecGD uses loss information for rate adjustment, while LyAm and Multistage SGDM use Lyapunov-based controls to improve stability or govern momentum deviation.These stability-aware methods rely on stringent theoretical assumptions that may not align with highly stochastic real-world data.Source: §3.2.5 · p. 19

Scheduler-free adaptation

• Schedule-Free, AutoDrop, AdaS, SGD-G2, AutoSGD, Step-Tuned SGD, and Amos exemplify efforts to automate or eliminate manual scheduling.Automated methods can reduce hyperparameter searches, but may sacrifice fine-grained control; a gap remains in balancing theoretical stability guarantees with low computational complexity at foundation-model scale.Source: §3.2.5 · pp. 19–20

3.2.6 Enhancing Generalization Ability

• SAM formulates optimization as minimizing the maximum loss in a local neighborhood, while later variants reduce the inner maximization’s computational burden.ESAM uses stochastic perturbation and data selection, AsyncSAM uses asynchronous perturbations, and AE-SAM reduces perturbation frequency.Source: §3.2.6 · p. 20
• Gradient-norm renormalization stabilizes SAM perturbations, multi-step ascent improves local-maximum approximation, and curvature-guided methods use higher-order geometry to navigate loss landscapes.These approaches trade stability or precision for risks including under-exploration, greater computational cost, and difficulty estimating high-dimensional geometry.Source: §3.2.6 · pp. 20–21
• Momentum approximations, gradient noise, and weight averaging offer alternatives to explicit sharpness perturbations, with trade-offs in computation, stability, or memory.Lookaround averages weights from parallel models trained with diverse data augmentations, but maintaining and synchronizing multiple states requires substantial memory.Source: §3.2.6 · p. 21
• A unified framework is still lacking for bounding generalization error and convergence when dynamic, low-rank, or asynchronous approximations are used in large-scale distributed non-convex training.The section also identifies a gap in autonomous hybrid frameworks that avoid manual tuning while preserving convergence bounds across heterogeneous loss landscapes.Source: §3.2.6 · p. 21, §3.2.7 · p. 23

3.2.7 Hybrid Methods

• Hybrid methods combine complementary operators, including global-rate adaptation, spatial or temporal smoothing, gradient filtering, and geometry-aware projections.Projection-based approaches impose geometric constraints within update dynamics, while filtering can reduce gradient noise and computational cost.Source: §3.2.7 · p. 22
• Hybrid designs can require precise transition tuning, costly projections or smoothing, complex hyperparameter interactions, and assumptions that diverge from practice.The section identifies autonomous tuning with theoretical convergence guarantees as a future challenge.Source: §3.2.7 · pp. 22–23
• Multi-objective hybrids combine adaptive rates, momentum, acceleration, efficiency strategies, or flatness-aware methods to address varied optimization goals.Examples include BAdam’s block coordinate descent with Adam for LLMs and AdaSAM’s integration of flatness-aware techniques with adaptive rates.Source: §3.2.7 · pp. 22–23

3.2.8 Towards LLMs Training

• Memory-efficient LLM optimizers use instantaneous preconditioning, low-rank projections, block-wise updates, or fused gradient computation and parameter updates.These designs reduce optimizer-state memory or active state, but may lose long-term directional stability, fine-grained gradient information, or training speed.Source: §3.2.8 · pp. 23–24

3.3.1 Hessian Approximation & Estimation

• Full Hessian computation is often impractical at scale, so second-order methods use block, diagonal, stochastic, or implicit approximations with different accuracy–overhead trade-offs.Block methods retain within-group curvature but discard cross-block dependencies; diagonal methods are efficient but ignore off-diagonal curvature, while sampling introduces variance and gradient-difference estimation can be noise-sensitive.Source: §3.3.1 · pp. 25–26

3.3.2 Fisher Information Matrix Applications

• Fisher Information Matrix methods progress from efficient diagonal approximations to block-Kronecker and trace-preserving structures, trading computational tractability against retained curvature information.Diagonal approximations ignore cross-parameter correlations; block methods capture layer-wise curvature but not inter-layer dependencies, while trace preservation improves approximation accuracy and spectral stability at added estimation cost.Source: §3.3.2 · p. 27

3.3.3 Quasi-Newton Methods

• Quasi-Newton methods avoid direct second-order derivatives through iterative curvature estimates, adapting to memory limits, stochastic noise, and distributed computation.Low-memory variants use restricted gradient and parameter histories; stochastic variants use Bayesian inference or variance reduction, while distributed methods partition curvature estimation and use momentum to mitigate stochastic noise.Source: §3.3.3 · pp. 27–28

3.3.4 Second-Order Moment Fusion

• Second-order moment fusion inserts curvature information into momentum to accelerate convergence, but its effectiveness is bounded by curvature-estimation accuracy and stochastic noise.Noise-robust approaches such as AdaHessian and Sophia use diagonal approximations with smoothing or clipping; the section identifies the lack of mechanisms that adapt curvature integration to real-time noise levels.Source: §3.3.4 · pp. 28–29

3.4 Zeroth-Order Algorithms

• ZO estimates gradient directions from forward evaluations of randomly perturbed parameters, bypassing backpropagation and its activation-memory overhead.MeZO Malladi et al. (2023) matches inference memory footprints.Source: §3.4 · p. 29

3.4.1 Adaptive Methods

• Adaptive ZO progresses from momentum-based noise smoothing to projected search and low-rank updates, trading dimensionality, omitted directions, or SVD overhead against efficiency.The section identifies strict first-order-comparable convergence in massive unstructured spaces without prohibitive sampling costs as an unresolved gap.Source: §3.4.1 · pp. 30–31

3.4.2 Perturbation Optimization

• Perturbation methods evolve from sign, sparse, or quantized constraints to paired history-aware sampling, reducing overhead or variance but risking lost directional information and added state.The literature still lacks methods that adapt perturbation geometry to the local landscape without prohibitive higher-order estimation costs.Source: §3.4.2 · pp. 31–32

3.4.3 Zeroth-First Order Hybrid

• ZO–FO hybrids blend estimates globally, divide them by layer, or correct ZO variance with FO references, trading memory savings against first-order computation and reference-point costs.The stated gap is adaptive integration based on local loss-landscape topography without static heuristics or costly periodic reference points.Source: §3.4.3 · p. 32

3.4.4 Memory-Efficient Methods

• Memory-focused ZO matches inference footprints or exploits low-rank, sparse, quantized, and scheduled updates, but unified convergence guarantees across these designs remain lacking.Inference-level methods can converge slowly in high-dimensional spaces; low-rank methods may lose expressive capacity, and quantization noise can exacerbate estimator variance.Source: §3.4.4 · pp. 32–34

3.4.5 Variance Reduction

• Variance reduction advances from moment smoothing to structured perturbations and snapshot control variates, improving stability or variance bounds but potentially requiring hardware-specific designs or checkpoint memory.The section identifies jointly bounding memory overhead and maintaining linear convergence at extreme scale as an open challenge.Source: §3.4.5 · p. 34

3.4.6 Distributed Zero-Order Optimization

• Distributed ZO aggregates local oracle estimates over a communication topology; ZoPro estimates higher-order information without exact gradients, while TOP-DP provides privacy guarantees with convergence trade-offs.Both paradigms generally achieve sublinear convergence; the cited gap is balancing query complexity, communication efficiency, and strict privacy under heterogeneous data.Source: §3.4.6 · pp. 34–35

3.5 Distributed Optimization

• Distributed optimization reduces communication bottlenecks through gradient compression and structural update strategies, but hybrid designs trade statistical efficiency against hardware utilization.Compression may use projection, quantization, or sparsification; structural strategies include local updates, decentralized communication, and federated learning.Source: §3.5 · p. 36

3.5.1 Gradient Compression & Quantization

• Quantization, sparsification, and low-rank compression reduce communication in different ways, but trade precision, index-encoding, and decomposition costs; unified scheduling under non-IID data remains an open theoretical gap.Error compensation and adaptive compression address information loss and changing network conditions, while balancing communication efficiency with convergence.Source: §3.5.1 · pp. 36–37

3.5.2 Local Update Strategies

• Local update strategies evolve from multiple local steps to momentum, variance correction, hybrid updates, and adaptive scheduling, trading fewer communications against drift, overhead, complexity, and weaker worst-case guarantees.The literature lacks a unified framework for exact consensus under simultaneous statistical heterogeneity and system-resource asymmetry without prohibitive auxiliary communication or hyperparameter optimization.Source: §3.5.2 · pp. 37–38

3.5.3 Decentralized Communication

• Decentralized methods progress toward asynchronous, robust, and privacy-aware protocols, but stragglers, gradient staleness, complexity, and privacy noise remain trade-offs without a unified guarantee of communication efficiency, strict privacy, and exact consensus.Approaches adjust communication graphs, relax synchronization, separate optimizer-state transmissions, or use privacy-preserving data and aggregation mechanisms.Source: §3.5.3 · p. 39

3.5.4 Federated Learning Optimization

• Federated optimization progresses from momentum-based first-order methods to curvature-informed methods and structural adaptations, trading simplicity against geometric awareness, matrix-inversion costs, and asynchronous instability.The cited gap is a unified theory bounding generalization error and communication complexity under dynamic, non-stationary heterogeneity; the passages call for adaptive, assumption-free algorithms.Source: §3.5.4 · pp. 40–41

3.5.5 Communication Scheduling&Threshold

• Communication scheduling ranges from update thresholds to priority ordering and delay-aware adaptation, trading bandwidth savings against precision, scheduling overhead, or system-profiling complexity; joint co-design with convergence guarantees remains unresolved.Threshold communication risks losing subtle gradient information, while priority scheduling preserves unbiased gradients and delay-aware scheduling relies on bounded delays.Source: §3.5.5 · p. 41

3.5.6 Distributed Hybrid Optimization

• Distributed hybrid optimization integrates components such as quantization, curvature estimation, and privacy protocols, spanning communication reduction, curvature adaptation, and privacy preservation.Source: §3.5.6 · p. 41
• SQuARM-SGD Singh et al. (2021) and Qsparse-local-SGD Basu et al. (2020) reduce communication bandwidth through local updates and compression, but compression variance can degrade convergence accuracy.Source: §3.5.6 · p. 42
• LQ-SGD Li et al. (2025a) combines logarithmic quantization with PowerSGD to reduce communication while maintaining model accuracy, at the cost of per-round low-rank decomposition overhead.Source: §3.5.6 · p. 42
• Fed-Sophia Elbakary et al. (2024) estimates the Hessian diagonal to use curvature without a full Hessian, accelerating convergence but remaining sensitive to estimation variance under heterogeneous data.Source: §3.5.6 · p. 42
• TOP-DP Guo et al. (2021) provides theoretical privacy guarantees for tasks such as image classification, with a trade-off between noise scale and final model utility.Source: §3.5.6 · p. 42
• Distributed hybrid optimization shifted from heuristic single-dimensional compression to joint designs, but lacks unified bounds for compression, curvature-estimation, and privacy-induced convergence errors under severe data heterogeneity.Source: §3.5.6 · p. 42

3.6 Privacy-Preserving Optimization

• Privacy-preserving optimization distinguishes privatized update mechanisms from strategies that mitigate convergence side effects, with differential privacy as its central paradigm.These methods modify vanilla optimizers to provide privacy guarantees while addressing computational inefficiencies.Source: §3.6 · p. 42

3.6.1 Differential Privacy Optimization

• DP adaptations span first-order noise and clipping methods, second-order moment corrections, and zeroth-order structured perturbations, but unified theory bounding privacy-utility trade-offs across orders remains absent.The stated gap includes avoiding extensive hyperparameter tuning or prior knowledge of the data distribution.Source: §3.6.1 · pp. 43–44

3.6.2 Gradient Noise Injection

• Noise injection spans adaptive intensity, geometric decomposition, robust second-moment or flat-region methods, and scalar perturbations; adaptive methods add computation, while zeroth-order convergence bounds assume strict convexity.The section identifies the absence of a unified framework combining tight convergence, low memory, and geometry-aware calibration across heterogeneous non-convex landscapes.Source: §3.6.2 · p. 44

3.6.3 Privacy-Utility Tradeoff

• DC-SGD adjusts clipping thresholds using private gradient-norm histograms, while DP-LSSGD denoises noisy gradients with Laplacian smoothing; both are reported to improve utility while maintaining strict DP guarantees.The methods act at different stages: clipping-threshold selection and post-processing of noisy gradients, respectively.Source: §3.6.3 · p. 44

3.6.4 Federated Privacy Enhancement

• Federated updates place randomized privacy mechanisms and client gradient estimates within the scenario transformation; transmission architecture and optimizer robustness trade communication overhead against local computation.The section also identifies the lack of a unified approach to communication, local landscape flatness, and heterogeneous data under strict privacy constraints.Source: §3.6.4 · pp. 44–45

3.6.5 Privacy-Aware Gradient Clipping

• Privacy-aware clipping advances from global thresholds to adaptive statistical calibration and curvature-aware clipping; adaptive methods add overhead, curvature approximations are costly, and unified ZO/SO clipping under strict DP remains absent.Global clipping is described as straightforward but suboptimal across heterogeneous training phases.Source: §3.6.5 · p. 45

3.7.1 Low-Memory Optimizer Design

• Low-memory designs range from stateless AlphaGrad (Sane, 2025) and factored Adafactor (Shazeer and Stern, 2018) to quantized states, blocked parameters, and layer-wise tpSGD targets.These approaches reduce storage by eliminating or compressing optimizer state, grouping parameters, or changing gradient propagation.Source: §3.7.1 · p. 46

3.7.2 Low-Rank Algorithms

• Low-rank methods project gradients or optimizer statistics into smaller subspaces, with AdaRankGrad using randomized SVD and Adapprox dynamically selecting approximation rank.Hybrid approaches such as QuZO and LQ-SGD combine low-rank structure with quantization to reduce memory and communication costs.Source: §3.7.2 · pp. 46–47

3.7.3 Optimizer State Compression

• Optimizer-state compression uses sparse momentum, low-rank second-moment approximations, and cross-parameter state sharing to reduce storage while seeking to preserve performance and stability.Examples include SPAM, MICROADAM, Adapprox, Adafactor, and AdaAct.Source: §3.7.3 · p. 47

3.7.4 Low-Rank Gradient Storage

• Low-rank gradient storage includes truncated-SVD projection in SUMO, wavelet compression in GWT, adaptive-rank approaches, and CPD-based gradient reconstruction in TeZO.Adapprox adjusts rank explicitly, while HELENE adapts gradient information through layer-wise clipping and annealing.Source: §3.7.4 · p. 47

3.7.5 Stateless Optimization

• Stateless optimization replaces historical optimizer state with instantaneous gradient statistics, parameter characteristics, or lightweight curvature estimates.Examples include SWAN, AlphaGrad, SGD-SaI, BFE/AdaBFE, and Sophia.Source: §3.7.5 · pp. 47–48

3.7.6 Memory-Efficient for Large Models

• Large-model fine-tuning methods reduce memory by improving zeroth-order estimation, fusing backpropagation with immediate parameter updates, or selectively updating parameter subsets.KerZOO uses kernel smoothing, LOMO retains at most one parameter gradient in memory, and Sparse-MeZO and LeZO select sparse parameters or layers.Source: §3.7.6 · p. 48

3.8.1 Auto-Designed Optimizers

• Auto-designed optimizers discover update rules through program search, neural controllers, symbolic derivation, evolutionary strategies, or meta-learning.Examples include Lion, KO, AGS-GD, GADAM, and WarpAdam, among others.Source: §3.8.1 · pp. 48–49

3.8.2 Robust Optimization

• Robust optimization addresses gradient noise, hardware noise, federated distribution shifts, and large-scale feasibility through clipping, correction, and structure-aware decomposition.Examples include AdaGC and AdaNorm for gradient robustness, FedCET and FAdamGC for federated settings, QuZO for hardware noise, and BC-ADMM for large-scale problems.Source: §3.8.2 · p. 49

4 Experiments

• The controlled evaluation spans ImageNet-1K with ResNet-50 and ViT-S and WikiText-103 with a 60M-parameter Llama proxy, using Top-1 accuracy for vision and perplexity for language.The study focuses on learning-rate tuning and mathematical optimizer behavior rather than peak system throughput; the Llama configuration is a computationally tractable architectural proxy, not a full-scale LLM.Source: §4 · pp. 49–50, §4.1 · p. 50, §4.2 · p. 50

4.3.1 Learning Rate Sensitivity on ViT-S

• On ViT-S, Muon maintains >75% accuracy from 0.2× to 5.0× learning rates, while Lion, MADGRAD, and MARS stay above 60% from 0.1× to 10×.Muon’s iterative momentum-matrix orthogonalization constrains updates; Lion’s sign-based updates and MADGRAD’s averaging and adaptive scaling buffer against gradient extremes.Source: Table 4 · p. 51, §4.3.1 · pp. 51–52, Figure 8 · p. 52
• At 1.0× learning rate, AdamW achieves 75.196 and outperforms Adam; NadamW reaches 75.278 versus Nadam at 72.18, while AdamP reaches 75.444 and exceeds standard Adam.SGD remains usable at 10× learning-rate scaling; unlike adaptive optimizers, it avoids division by tiny gradient values, though it oscillates violently around the optimum.Source: §4.3.1 · pp. 52–53

4.3.2 Cross-Architecture Generalization

• Muon and MARS (Yuan et al., 2025) generalize across ResNet-50 and Llama, maintaining PPL ≈12–14 on Llama under extreme learning-rate scaling.The transfer test applies ViT-S-optimal learning rates to ResNet-50 and Llama while adapting auxiliary strategies and weight decay to each architecture.Source: Table 5 · p. 53, §4.3.2 · p. 53
• On Llama, adaptive methods including Adam can fail when gradient outliers contaminate scalar second-moment estimates, while SGD-series optimizers can collapse without layer-wise or parameter-wise scaling.The reported collapse is attributed to Llama’s depth and attention mechanisms creating large gradient and parameter-scale disparities across layers.Source: §4.3.2 · pp. 53–54, Figure 9 · p. 54

4.3.3 Long-term Training Scalability

• SGD gains 9.41% on ViT-S over 300 epochs; Muon, Lion, Kron, and AdamW gain +1.896% to +4.690% across architectures, while RMSprop regresses on ResNet-50.Muon starts at 78.252 on ViT-S after 100 epochs and gains +2.622% by 300 epochs; the authors attribute modest gains for advanced optimizers to early convergence and diminishing returns.Source: §4.3.3 · pp. 54–55, Figure 10 · p. 55

4.3.4 Correlation of Optimizers

• Optimizer trajectories are highly homogeneous on ViT-S but bifurcate on ResNet-50; adaptive methods cluster together, structural methods diverge, and correlations become more homogeneous from 100 to 300 epochs.The analysis uses first-order differences in validation loss to isolate stepwise acceleration and deceleration rather than compare collinear raw loss trajectories.Source: §4.3.4 · pp. 55–56

4.3.5 Comprehensive Optimizer Evaluation.

• The comprehensive comparison evaluates more than twenty optimizers with a five-criterion star-rating paradigm in which more stars indicate stronger relative performance percentiles.Source: §4.3.5 · p. 56
• ViT-S Dosovitskiy (2020) shows larger accuracy improvements from 100 to 300 epochs than ResNet-50 He et al. (2016).Table 6 compares accuracy at both training horizons across various optimizers.Source: Table 6 · p. 56

4.3.6 Computational cost disclosure.

• Figure 11 compares Top-1 accuracy trajectories for 23 optimizers on ViT-S and ResNet-50 over 100 and 300 epochs, with heatmaps showing trajectory correlations.The inset Pearson correlation matrices use first-order differencing of validation metrics and reveal algorithmic categorization and trajectory divergence across architectures and training horizons.Source: Figure 11 · p. 57

4.3.5 Comprehensive Optimizer Evaluation.

• Muon achieves the best overall performance across convergence speed, accuracy, memory efficiency, generalization, and hyperparameter robustness.The evaluation rates accuracy across three architectures, generalization through cross-architecture experiments, robustness through learning-rate scaling, convergence by iterations to target loss, and memory by peak GPU footprint.Source: §4.3.5 · p. 56, Table 7 · p. 57

4.3.6 Computational cost disclosure.

• The strict grid-search protocol across 23 optimizers and extended 300-epoch runs required an estimated more than 1073 single NVIDIA A100 hours.The framework also included multi-scale learning-rate sweeps; the passage attributes exponential computational overhead to the rigorous methodology.Source: §4.3.6 · pp. 56–57

Challenges

• Current methods trade convergence speed, memory efficiency, and distributed scaling against generalization, computational latency, and system stability.The section also identifies hyperparameter sensitivity and distributed client drift as persistent scaling challenges.Source: §5 · pp. 57–58
• Adaptive methods can favor sharp minima, preconditioned methods have narrow stability margins, and stateless designs lose second-moment history needed to average gradient shocks.Dense optimizer states also burden memory, while structural approximations and matrix inversions can reduce global efficiency.Source: §5 · pp. 57–58
• Noise amplification destabilizes curvature estimates in anisotropic landscapes, while zeroth-order estimates incur dimensionality-sensitive variance and privacy noise degrades gradient fidelity..Source: §5 · p. 58

First-order optimization

• First-order research should pursue architecture-specific optimizer discovery, structural preconditioning or orthogonalization, and dynamic low-precision arithmetic for dense optimizer states.The proposed aims include reducing memory footprints without compromising convergence stability.Source: §5 · p. 58

Second-order optimization

• Second-order frameworks should co-design algorithms with hardware, improve robust curvature estimation, and develop sparse preconditioning to reduce per-step latency.The proposals target memory bottlenecks, gradient outliers, and computational cost while retaining convergence stability and second-order information.Source: §5 · p. 58

Zeroth-order optimization

• Zeroth-order research should cancel perturbation noise, constrain search subspaces without blocking high-quality global solutions, and adapt perturbations to local geometry.The section proposes dynamic variance and direction choices to avoid disrupting local-minimum equilibria.Source: §5 · p. 59

Scenario-oriented frameworks

• Scenario-oriented frameworks should establish communication and privacy-utility bounds, preserve useful history within memory limits, and use topology-aware compression to address client drift..Source: §5 · p. 59

Discussion

• The discussion calls for integrating first-, second-, and zeroth-order methods while optimizing global wall-clock efficiency through hardware-algorithm co-design.It also recommends combining structural updates and dynamic low precision with automated optimizer discovery, exact noise cancellation, and topology-aware consensus mechanisms.Source: §5 · p. 59

Conclusion

• The survey establishes a theoretical foundation and unified framework for understanding optimization algorithms’ evolution..Source: §6 · p. 59
• It categorizes and analyzes first-order, second-order, zeroth-order, and scenario-oriented methods by their structural principles and functional characteristics..Source: §6 · p. 59
• A controlled evaluation across task domains and architectures reports trade-offs in hyperparameter sensitivity, cross-architecture generalization, and long-term training scalability..Source: §6 · p. 59
• The survey identifies open challenges and future research directions, aiming to guide development of efficient, robust, and trustworthy optimization technologies..Source: §6 · p. 59