PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizer Memory Schedules for Outscaling the Overtraining Axis

Katie Everett, Shikai Qiu

arXiv:2609.04577v1cs.LG

TL;DR

Optimizer benchmarks commonly compare methods at a single horizon, while prior work covers a limited range of overtraining factors despite horizon-dependent rankings and hyperparameters. We train 51M–253M-parameter models with each optimizer across 1×–256× overtraining, independently sweeping base learning rate for every optimizer, model size, and overtraining factor. With log-time weight decay and momentum cooldown, ADANA has equivalent-OT scaling close to DANA theory’s 2 −κ prediction and retains an AdamW advantage after per-horizon memory tuning.

  • Problem

    Optimizer benchmarks commonly compare methods at a single horizon, while prior work covers a limited range of overtraining factors despite horizon-dependent rankings and hyperparameters.

    Source: §1 · p. 1

  • Method

    We train 51M–253M-parameter models with each optimizer across 1×–256× overtraining, independently sweeping base learning rate for every optimizer, model size, and overtraining factor.

    Source: §1 · p. 2

  • Results

    With log-time weight decay and momentum cooldown, ADANA has equivalent-OT scaling close to DANA theory’s 2 −κ prediction and retains an AdamW advantage after per-horizon memory tuning.

    Source: §1 · p. 2, §9 · p. 12

  • Takeaways & Limitations

    Optimizer evaluation should treat training horizon as an essential design axis because rankings and preferred treatments change as models train beyond compute-optimal token budgets.

    Source: §8 · p. 11

    Whether these conclusions transfer across larger models, architectures, datasets, and longer training horizons remains for future work.

    Source: §8 · p. 11

Abstract

from arXiv · show

We investigate how optimizers scale across the overtraining axis and show that relative optimizer performance and optimal hyperparameters change substantially with training horizon. In particular, we study how matrix-preconditioned methods (Muon and SOAP) and a momentum-scheduled method (ADANA) scale relative to AdamW. We compare these four optimizers across models from 51M to 253M parameters and overtraining (OT) factors from 1x to 256x, sweeping the base learning rate at every setting. The preferred learning rate schedule can reverse across the overtraining axis, the best weight decay coefficient scales approximately as sqrt(OT), and longer horizons generally favor longer fixed memory. ADANA's scaling advantage over AdamW persists after tuning AdamW's fixed memory separately at each horizon. Log-time weight decay and momentum cooldown provide substantial gains for ADANA that compound as training increases. With this treatment, ADANA outscales AdamW with an exponent advantage close to that predicted by DANA theory on power-law random features. Muon and SOAP instead provide roughly constant token-efficiency advantages over AdamW across most of the measured range, although SOAP may gain further at the highest overtraining factors. ADANA begins behind both matrix-preconditioned optimizers but closes the gaps as training increases, surpassing Muon and becoming competitive with SOAP at our highest OT factors. These results establish training horizon as an essential axis for optimizer evaluation and design.

1 INTRODUCTION

Optimizer rankings and preferred treatments change with training horizon: with log-time weight decay and momentum cooldown, ADANA outscales AdamW and closes on matrix-preconditioned methods across the overtraining axis.

1 INTRODUCTION

  • Overtraining extends training beyond the compute-optimal token ratio to reduce recurring inference costs; optimizer rankings and hyperparameters can change with horizon, a range prior benchmarks covered only narrowly.The Chinchilla scaling law corresponds to approximately 20 training tokens per parameter at fixed pretraining compute.Source: §1 · p. 1
  • The comparison tests coordinatewise AdamW against matrix-preconditioned Muon and SOAP with fixed scalar momentum, and ADANA’s scheduled memory within an Adam-style adaptive optimizer.DANA derives a momentum schedule whose memory grows throughout training.Source: §1 · pp. 1–2
  • Experiments span 51M–253M models and 1×–256× OT, independently sweeping learning rate and measuring token multipliers as baseline-to-candidate training tokens at equal loss.A token multiplier above one favors the indicated optimizer.Source: §1 · p. 2
  • The preferred learning-rate schedule can reverse across OT, best weight decay scales approximately as √f, and longer horizons generally favor longer fixed memory.Optimal learning rate decreases as memory grows; joint memory and learning-rate tuning helps particularly at the lowest and highest OT factors, while momentum cooldown and log-time weight decay provide compounding ADANA gains.Source: §1 · p. 2
  • With log-time weight decay and momentum cooldown, ADANA outscales AdamW near the DANA-predicted 2 −κ exponent; Muon and SOAP stay roughly constant, while ADANA surpasses Muon and nears SOAP.At the longest measured horizons for 51M and 124M models, SOAP overtakes Muon.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Related work links optimizer performance to memory timescales and training horizon, while prior comparisons leave behavior under substantially greater overtraining unresolved.

Long and scheduled optimizer memory

  • Prior work studies optimizer memory through fast and slow averages, iterate averaging, growing momentum timescales, and hyperparameters that vary with token horizon or batch size.AdEMAMix combines fast and slow momentum buffers (Pagliardini et al., 2024); schedule-free methods use parameter-space iterate averaging (Defazio et al., 2024), whose benefit reportedly grows with duration (Defazio, 2026). Morwani et al. (2025) connect these methods to accelerated SGD and report larger gains at small batch sizes; DANA and ADANA use growing momentum timescales (Ferbach et al., 2025; 2026). Björck et al. (2025) report a predictable decrease in optimal learning rate with token horizon, while Marek et al. (2025) find Adam’s β2 should scale with batch size to keep the second-moment half-life fixed in tokens.Source: §2 · pp. 2–3

Optimizer comparisons across training horizons

  • Figure 2 shows ADANA improves relative to AdamW as OT increases, Muon’s advantage stays roughly constant, and SOAP may improve further at the highest OT factors.The figure defines values above the equal-OT diagonal as favoring the indicated optimizer; its bottom panel reports the AdamW-relative token multiplier.Source: Figure 2 · p. 3

Long and scheduled optimizer memory

Matrix-preconditioned optimizers

  • Muon, SOAP, and Shampoo are matrix-preconditioned methods; prior experiments report approximately 1.4× compute-multiplier gains over AdamW across 190M–1.4B model widths.An idealized Muon yields a square-root spectral speedup for eigenmodes resolvable at the batch size, while unresolved modes behave like SGD (Paquette et al., 2026); the reported empirical gains require appropriate learning-rate and weight-decay scaling (Qiu et al., 2025).Source: §2 · pp. 3–4

Scaling-law functional forms

  • Scaling analyses can couple model size and training data through an interaction exponent, and sharing the model exponent across optimizers can improve identifiability.The paper adopts Videau et al.’s (2026) functional form and a shared model exponent, similar to Volkova et al. (2026).Source: §2 · p. 4

Optimizer comparisons across training horizons

  • Prior horizon comparisons find optimizer advantages can change with OT and experimental treatment, but leave behavior under substantially greater overtraining unresolved.Wen et al. (2025) report 1.2×–1.4× token multipliers for Muon and SOAP on 130M–520M models from 1× to 8× Chinchilla; Muon leads through 4×, while SOAP catches or overtakes it at 8× and 16×, and both fall to approximately 1.1× on 1.2B. Semenov et al. (2025) find AdamW narrows and eventually reverses SOAP’s short-horizon advantage, and SOAP’s long-horizon ordering against AdEMAMix depends on retuning β3. Wen et al. (2026) report standard Muon at approximately 1.1× and Muon Hyperball rising from approximately 1.2× to 1.3× across 1×–8× Chinchilla.Source: §2 · p. 4

3 SCALING OPTIMIZERS ACROSS TRAINING HORIZONS

Optimizer performance and memory choices depend on training horizon: the study independently tunes learning rates across overtraining factors and finds that rankings and token efficiency change as horizons grow.

3.1 MOMENTUM AND WEIGHT DECAY AS MEMORY

  • Momentum controls how long past gradient information remains influential, whereas decoupled weight decay controls how quickly parameters forget prior updates; time-varying decay changes that memory over training.With fixed momentum, the memory timescale remains constant across training steps; scheduled-memory optimizers vary it.Source: §3.1 · pp. 4–5

3.2 EXPERIMENTAL SETUP

  • The study compares AdamW, ADANA, Muon, and SOAP across model sizes and overtraining factors, independently sweeping base learning rate and reporting the lowest final validation loss at each setting.The models range from 51M to 253M parameters, with OT factors up to 256×, 32×, and 8×, respectively; core comparisons use sequence length 2,048 and batch size 256.Source: §3.2 · p. 5
  • Preferred learning-rate schedules reverse with horizon: linear decay performs better at 1× OT, while cosine decay performs better at 8× and 32× OT for all four optimizers.Log-time weight-decay gains depend on the learning-rate endpoint and are larger with cosine decay to 10% than with cosine decay to zero.Source: §3.2 · p. 5
  • On the 51M model, the preferred weight-decay coefficient scales approximately as √f for both uniform and log-time decay, consistent with the square-root scaling predicted for uniform AdamW by Bergsma et al. (2025a).Most experiments therefore use cuniform(f) = 8√f or clog(f) = 2√f; for optimizers that benefit from log-time decay, its advantage over uniform decay increases with OT.Source: §3.2 · p. 5

3.3 OUTSCALING AND TOKEN MULTIPLIERS

  • The token multiplier measures baseline tokens divided by indicated-optimizer tokens at equal loss, so values above one favor the indicated optimizer; outscaling means this multiplier increases with OT.The increase may reflect an improved token-decay exponent, a lower high-token validation-loss limit, or both.Source: §3.3 · p. 5
  • Figure 3 shows that longer horizons favor longer fixed effective memory, while at 8× OT the optimal base learning rate decreases as memory increases; it also compares per-horizon tuning with fixed M = 50.The figure’s right panel reports the within-optimizer validation-loss improvement from replacing fixed M = 50 with the optimal memory and learning-rate pair at each horizon.Source: Figure 3 · p. 6

3.4 OPTIMIZER RANKINGS ACROSS TRAINING HORIZONS

  • Muon and SOAP lead at most short and intermediate horizons, while ADANA’s token advantage over AdamW rises across model sizes, Muon’s stays roughly constant, and SOAP may gain at the highest OT factors.At the longest measured horizons for the 51M and 124M models, SOAP also overtakes Muon.Source: §3.4 · p. 6

4 HORIZON-TUNED FIXED MEMORY

Longer overtraining horizons generally favor longer fixed memory, but tuning memory separately does not erase ADANA’s growing advantage over AdamW; Muon and SOAP retain roughly constant advantages.

4 HORIZON-TUNED FIXED MEMORY

  • The study jointly tunes memory and learning rate at each horizon on the 51M model, with results specific to global batch size 256.The motivation is that longer averaging windows may accumulate weak, persistent gradient signals for later-learned modes, but respond more slowly to distribution changes and retain stale gradients longer.Source: §4 · p. 6

4.2 PER-HORIZON TUNING IMPROVES FIXED-MEMORY OPTIMIZERS

  • ADANA continues to gain on horizon-tuned AdamW as overtraining increases, while horizon-tuned Muon and SOAP retain roughly constant advantages over AdamW.Figure 4 compares ADANA with AdamW after tuning fixed memory separately at each horizon; its vertical axis reports the horizon-tuned AdamW OT factor needed to match validation loss.Source: Figure 4 · p. 7, §4.2 · p. 7

4.1 LONGER HORIZONS FAVOR LONGER MEMORY

  • From 1× to 128× OT, optimal memory rises from M = 20 to M = 1280 for AdamW, M = 20 to M = 640 for Muon, and M = 40 to M = 1280 for SOAP.Optimal base learning rate decreases as memory grows; very long windows may become unstable, and momentum results are specific to global batch size 256.Source: §4 · p. 6, §4.1 · p. 7

4.2 PER-HORIZON TUNING IMPROVES FIXED-MEMORY OPTIMIZERS

  • At 128× OT, per-horizon tuning improves final validation loss by 3.3 mpt for AdamW, 3.6 mpt for Muon, and 1.3 mpt for SOAP.At moderate horizons, M = 50 is already near-optimal, so β = 0.98 does not transfer uniformly across the full overtraining range.Source: §4.2 · p. 7

5 LOG-TIME WEIGHT DECAY ACROSS TRAINING HORIZONS

The preferred weight decay schedule depends on optimizer and training horizon: log-time weight decay increasingly benefits ADANA as overtraining grows, while its effects differ for SOAP, AdamW, and Muon. The tested coefficient rules scale as √f and approximate trends across optimizers rather than separately optimizing every setting.

5 LOG-TIME WEIGHT DECAY ACROSS TRAINING HORIZONS

  • Weight decay schedules determine how quickly parameters forget past updates, and log-time weight decay decreases its coefficient over training.Source: §5 · pp. 7–8
  • Log-time weight decay increasingly benefits ADANA as overtraining grows, modestly benefits SOAP, has little consistent effect on AdamW, and harms Muon at high OT.Figure 5 uses a 51M-parameter model and compares c(f) = 8√f for uniform decay with a(f) = 2√f for log-time decay, independently sweeping base learning rate for each optimizer, schedule, and OT factor. The Muon result may reflect a poorly matched log-time coefficient rather than the schedule itself.Source: Figure 5 · p. 8, §5 · p. 8
  • The tested uniform and log-time weight decay coefficients are c(f) = 8√f and a(f) = 2√f, whose √f dependence agrees with scaling predicted by Bergsma et al. (2025a).The rules were fitted from joint weight decay coefficient and learning rate sweeps; they approximate trends across optimizers rather than selecting the optimal coefficient separately for each optimizer and OT factor.Source: §5 · p. 8

6 MOMENTUM COOLDOWN FOR GROWING MEMORY

Momentum cooldown preserves ADANA’s growing memory through most of training, then shortens it during terminal learning-rate decay. Combined with log-time weight decay, it improves ADANA most at long horizons across model sizes.

6 MOMENTUM COOLDOWN FOR GROWING MEMORY

  • Momentum cooldown caps ADANA’s growing memory during terminal learning-rate decay, when responsiveness may matter more; a similar end-of-training momentum reduction was used for Muon in nanochat.The cooldown bounds memory by the learning-rate decay timescale, with a floor of one.Source: §6 · pp. 8–9
  • Across all three model sizes, combining momentum cooldown with log-time weight decay improves ADANA most at long horizons and outperforms ADANA with uniform weight decay.Improvements are measured in final validation loss; the authors defer analysis of the mechanism and alternative cooldown rules.Source: Figure 6 · p. 9, §6 · p. 9
  • The combined ADANA treatment is used in subsequent comparisons of the strongest treatments, and its AdamW-relative token multiplier appears approximately straight on log–log axes.This pattern motivates the outscaling analysis in Section 7.Source: §6 · p. 9, Figure 6 · p. 9

7 OUTSCALING ACROSS THE OVERTRAINING AXIS

Across the overtraining axis, ADANA’s relative performance improves with longer training: it outscales AdamW and closes its initial gap to matrix-preconditioned optimizers. Muon and SOAP provide roughly constant token-multiplier gains over AdamW across most measured OT factors.

7.1 ADANA OUTSCALES ADAMW

  • With log-time weight decay and momentum cooldown, ADANA’s equivalent-OT exponents are 1.15–1.20, close to the DANA prediction of 1.15.The fitted token-decay exponent advantage remains positive across the range of high-token-limit differences consistent with the data.Source: Figure 7 · p. 10, §7.1 · p. 10

7.2 MATRIX-PRECONDITIONED OPTIMIZERS PROVIDE APPROXIMATELY CONSTANT GAINS OVER ADAMW

  • Across most OT factors, Muon provides a 1.4×–1.7× token multiplier over AdamW, while SOAP provides 1.3×–1.7×.SOAP rises to approximately 1.9× at 128× OT on the 51M model, suggesting possible further gains at higher overtraining.Source: §7.2 · p. 10, Figure 8 · p. 11

7.3 ADANA OUTSCALES THE MATRIX-PRECONDITIONED OPTIMIZERS

  • At the highest tested OT factors, ADANA surpasses Muon and becomes competitive with SOAP, though whether this trend continues remains uncertain.At 2× OT, ADANA’s token multiplier is 0.59×–0.63× relative to Muon and 0.63×–0.69× relative to SOAP across the three model sizes.Source: §7.3 · p. 11, Figure 8 · p. 11

8 DISCUSSION AND LIMITATIONS

Optimizer rankings and hyperparameter choices can change with training horizon: Muon and SOAP are strong across most horizons, while ADANA’s relative efficiency increases with training. The discussion highlights scheduled memory’s added benefit, the need to test transfer beyond the studied settings, and unresolved batch-size dependence.

8 DISCUSSION AND LIMITATIONS

  • Optimizer evaluation should treat training horizon as part of experimental design because rankings and preferred learning-rate schedules, weight decay, and memory can change with overtraining.Long-horizon comparisons require hyperparameter scaling rules rather than choices based only on experiments at one horizon.Source: §8 · p. 11
  • Muon and SOAP provide large advantages over AdamW across most horizons, while ADANA’s relative efficiency increases with training horizon.Tuning fixed memory separately at each horizon improves fixed-memory optimizers but does not explain ADANA’s scaling advantage, suggesting within-run memory adaptation adds a benefit.Source: §8 · p. 11
  • Future work should test transfer across larger models, architectures, datasets, and horizons; preliminary Appendix J experiments retain most ADANA performance when preconditioning and scheduled momentum are reversed.The appendix result indicates scheduled memory can remain effective with another preconditioner ordering.Source: §8 · p. 11
  • At 8× OT, ADANA begins losing efficiency at smaller batch sizes than AdamW despite joint learning-rate and β2/g3 tuning.Main comparisons use global sequence batch 256. The fixed-token experiment cannot isolate the mechanism or establish a general batch-scaling law because batch size changes optimizer-update count and gradient noise while other hyperparameters remain fixed; controlled batch-size studies are left to future work.Source: §8 · pp. 11–12

9 CONCLUSION … B.3 COMMON TRAINING CONTROLS

Optimizer rankings and optimal treatments change beyond compute-optimal horizons: ADANA’s advantage over AdamW persists after horizon-specific memory tuning and is amplified by log-time weight decay and momentum cooldown, while matrix-preconditioned methods retain roughly constant gains.

9 CONCLUSION

  • ADANA’s scaling advantage over AdamW persists after horizon-specific memory tuning; log-time weight decay and momentum cooldown amplify it, while matrix-preconditioned methods provide roughly constant gains and ADANA closes the gap as training increases.With these treatments, ADANA’s equivalent-OT scaling approaches the 2 −κ prediction from DANA theory on power-law random features.Source: §9 · p. 12

A EXTENDED RELATED WORK

B EXPERIMENTAL DETAILS

  • Experiments use a shared model family, initialization, data, training, and evaluation protocol unless explicitly stated otherwise.Source: B EXPERIMENTAL DETAILS · p. 19

B.1 NOTATION, ARCHITECTURE, AND MODEL SIZES

  • OT denotes horizons starting at 20 tokens per nominal parameter; model labels use rounded nominal parameter counts, and compute accounting approximates 2PT forward plus 4PT backward FLOPs.The horizon convention does not assume that 1× OT is compute-optimal for every optimizer, model family, or dataset; nominal compute includes the readout but omits the input embedding.Source: §B.1 · pp. 19–20
  • The study uses decoder-only causal Transformers with SwiGLU, RMSNorm, and QK-RMSNorm, following an Enoki-ladder-style co-scaling setup with architecture changes from that baseline.The blocks are described as architecturally similar to the Llama 2 style architecture.Source: §B.1 · p. 19

B.2 INITIALIZATION

  • Residual-output projections receive a depth correction, token embeddings use width-independent standard deviation 0.02, and residual branches are added with coefficient one.The initialization uses independently drawn, untruncated, zero-mean Gaussian matrix entries, with learned normalization scales initialized to one.Source: §B.2 · p. 20

B.3 COMMON TRAINING CONTROLS

  • Candidate learning-rate schedules share the same linear warmup; the post-warmup schedule and weight-decay treatment are selected experimentally.These controls apply to the primary experiments unless stated otherwise.Source: §B.3 · p. 20

B.4 DATASET, TRAINING ORDER, SEEDS, AND EVALUATION … C.6 PARAMETER ROUTING

The paper specifies a controlled training and evaluation protocol, then defines shared update rules and optimizer-specific implementations and parameter routes. The setup uses a fixed FineWeb corpus order and a common held-out evaluation set.

B.5 HYPERPARAMETER SWEEP INTERIORITY

  • Training uses 360.5 billion FineWeb tokens in a fixed sequential order, with one shared validation set of 1,920 sequences and 3,932,160 scored targets.Configurations use fresh seed-42 initializations and one seed per sweep configuration; hyperparameter sweeps apply a one-millipoint boundary-interiority criterion to inner and outer axes.Source: §B.4 · pp. 20–21, §B.5 · p. 21

B.6 HARDWARE, NUMERICAL PRECISION, AND PARALLEL EXECUTION

  • Training uses JAX and Flax on TPU v4-8 and v6e-4, with mixed precision, global-batch gradient accumulation, and optional blockwise activation rematerialization.For sharded configurations, parameters, gradients, and optimizer states are partitioned across devices with ZeRO Stage 3.Source: §B.6 · pp. 21–22

C.1 COMMON UPDATE WRAPPER

  • The common wrapper applies scheduled optimizer updates and decoupled weight decay independently, while AdamW uses first- and second-moment estimates of clipped gradients.Weight decay is not added to the gradient or transformed by an optimizer preconditioner; it follows the normalized schedule independently of peak and layerwise learning rates.Source: §C.1 · p. 22, §C.2 · pp. 22–23

C.3 MUON

  • Muon applies Nesterov momentum and Newton–Schulz matrix orthogonalization to hidden attention and feed-forward matrices, while Adam handles other listed parameter classes.The Adam route uses native learning-rate multiplier 1.6 and the Muon route multiplier 1.Source: §C.3 · p. 23

C.4 SOAP

  • SOAP applies Adam in a Shampoo-preconditioner eigenbasis, maintaining its first moment in original coordinates and its second moment in rotated coordinates.It is applied to learned matrices, with normalization scales and biases using Adam; bases are refreshed every 10 applied updates.Source: §C.4 · p. 24

C.5 ADANA

  • ADANA uses log-time weighting for its first and second moments, with g2 = 1 and δ = 8 fixed and g3 and κ treated as momentum hyperparameters.All reported ADANA experiments disable the optional ADANA–MK4 coordinatewise cap.Source: §C.5 · pp. 24–25

C.6 PARAMETER ROUTING

  • Parameter routing uses three tunable learning-rate factors for embedding/vector-like parameters, hidden matrices, and the readout; Muon’s Adam route alone has native multiplier 1.6.The primary architecture has no bias parameters; for architectures with biases, the paper recommends the learned-normalization-scale route and embedding-group learning rate, without weight decay.Source: §C.6 · p. 25

D LEARNING RATE SCHEDULE SELECTION … E.3 COMPARING UNIFORM AND LOG-TIME WD SCHEDULES ACROSS OT FACTORS

Learning-rate and weight-decay choices change with training horizon: linear decay is favored at 1× OT, while cosine decay is favored at 8× and 32× OT. Log-time weight decay is not uniformly better than uniform weight decay, and its apparent advantage depends on the learning-rate endpoint.

D LEARNING RATE SCHEDULE SELECTION

  • Because the remaining experiments focus primarily on high OT factors, the paper uses cosine decay to zero after warmup as its primary learning-rate schedule, not as a universally optimal choice.Both schedules use linear warmup; the comparison tunes learning rate and weight decay separately for each schedule, optimizer, and OT factor.Source: D LEARNING RATE SCHEDULE SELECTION · pp. 25–26
  • At 1× OT, linear decay achieves lower loss for all four optimizers, while cosine decay achieves lower loss at 8× and 32× OT.Figure 11 summarizes the crossover after selecting the best learning-rate and weight-decay pair for each schedule; the comparison is on the 51M model with fixed optimizer-specific momentum and uniform weight decay.Source: D LEARNING RATE SCHEDULE SELECTION · pp. 25–26, Figure 11 · p. 28

E.1 PROTOCOL CHOICE FOR THE LOG-TIME WD OFFSET

  • The log-time weight-decay offset is anchored to the 1× OT horizon, using τ = 0.1S1×, to preserve the initial schedule timescale as training extends.The horizon-normalized and 1×-anchored definitions coincide at 1× OT; the anchored offset is a protocol convention, not a claim of universal optimality.Source: §E.1 · p. 28

E.2 INTERACTION WITH THE LEARNING RATE ENDPOINT

  • 32.3 and 16.9 millipoints: with a 10% learning-rate endpoint, log-time weight decay lowers final validation loss by these amounts for AdamW and ADANA, respectively; with a zero endpoint, the gains are 1.3 and 0.3.The comparison matches total parameter shrinkage from weight decay alone on the 51M model at 8× OT, rather than separately tuning each schedule for best achievable loss.Source: Figure 12 · p. 29, §E.2 · p. 29

E.3 COMPARING UNIFORM AND LOG-TIME WD SCHEDULES ACROSS OT FACTORS

  • Uniform weight decay performs better at 1× OT for AdamW, ADANA, and Muon; at higher OT, log-time weight decay is competitive or better for some optimizers, including ADANA and SOAP.Figure 13 compares schedules after separately tuning learning rate and weight-decay coefficient at each OT factor; this per-horizon tuning does not establish a coefficient scaling rule transferable across horizons.Source: §E.3 · pp. 29–30, Figure 13 · p. 30

E WEIGHT DECAY SCHEDULES AND COEFFICIENT SCALING

  • Weight-decay schedule shape and coefficient scaling across horizons are separate choices, and schedule comparisons do not by themselves yield a transferable coefficient-scaling rule.Uniform weight decay uses a constant per-update coefficient, while log-time weight decay decreases it with the shifted update index; coefficient values are not directly comparable across schedules.Source: E WEIGHT DECAY SCHEDULES AND… · p. 27, §E.3 · p. 30

E.4 WD COEFFICIENT SCALING RULES ACROSS OT FACTORS · F FIXED-MEMORY TUNING ACROSS TRAINING HORIZONS · F.1 PER-HORIZON MEMORY TUNING

Across overtraining horizons, the best weight-decay coefficient generally rises and is well approximated by square-root scaling; fixed-memory optima also shift toward longer memory as training extends. The section develops practical WD rules and examines per-horizon memory tuning, including its potential benefits and stability tradeoffs.

E.4 WD COEFFICIENT SCALING RULES ACROSS OT FACTORS

  • Jointly tuning learning rate and WD at every horizon is preferred when feasible; the proposed scaling rules substitute when that grid is impractical, not as universally optimal prescriptions.Source: §E.4 · pp. 30–31
  • The analysis extends Power Lines’ uniform-WD study of AdamW to log-time WD and the broader optimizer set considered here.Source: §E.4 · p. 31
  • The best WD coefficient generally increases with OT, and fixing the 1×-OT coefficient can incur increasing regret at longer horizons.Figure 14 shows coefficient regret after minimizing over the base learning rate; minima move across OT factors.Source: §E.4 · p. 31, Figure 14 · p. 31
  • The fitted WD exponents are 0.527 for uniform WD and 0.473 for log-time WD, motivating adopted square-root scaling for both schedules.The rules use schedule-specific coefficients anchored at 1× OT.Source: §E.4 · p. 31, Figure 15 · p. 32

F FIXED-MEMORY TUNING ACROSS TRAINING HORIZONS

F.1 PER-HORIZON MEMORY TUNING

  • On the 51M model, the study jointly sweeps learning rate and memory for AdamW, Muon, and SOAP at each OT factor.It tests memory values from 20 to 1280, with additional boundary extensions; fixed memory becomes a smaller fraction of training as OT rises.Source: §F.1 · p. 32
  • The low-regret region shifts toward longer fixed memory as the training horizon increases.At each horizon, the per-horizon treatment selects the memory and learning-rate pair with the lowest final validation loss.Source: Figure 16 · p. 33, §F.1 · p. 33

F.2 HORIZON-TUNED MEMORY RESULTS … H.2 PAIRWISE COMPARISONS UNDER UNIFORM WEIGHT DECAY

Optimizer performance changes with training horizon: ADANA and SOAP gain relative ground at long horizons, while per-horizon memory tuning improves fixed-memory methods. The paper evaluates these trends using scaling fits and a profile analysis that distinguishes possible token-decay and high-token-loss-limit advantages.

F.2 HORIZON-TUNED MEMORY RESULTS

  • At 128× OT, per-horizon memory tuning improves validation loss by 3.3 mpt for AdamW, 3.6 mpt for Muon, and 1.3 mpt for SOAP.Optimal memory generally increases with horizon, but the trajectories are not always monotonic; longer memory is generally paired with a smaller loss-minimizing learning rate. Broad low-regret regions mean the optima do not establish a transferable memory-scaling rule; the comparison is limited to the 51M model and one memory coefficient per optimizer.Source: §F.2 · pp. 33, 35, Figure 18 · p. 34

G.1 PRIMARY COMPARISON AND MEASURED SUPPORT

  • The scaling analysis fits 76 uniform-weight-decay loss values across three model sizes and OT ranges, independently sweeping the base learning rate at every optimizer, size, and horizon.The 51M model covers 1×–256× OT, the 124M model 1×–32×, and the 253M model 1×–8×.Source: §G.1 · p. 35

G.2 JOINT MODEL–TOKEN FUNCTIONAL FORM

  • The primary shared-m Skaling fit shares residual loss floor E and high-token model exponent m across optimizers while allowing optimizer-specific parameters.Because only three model sizes are available, the shared m is a modeling choice and does not imply that true optimizer-specific model exponents are equal.Source: §G.2 · p. 35

G.3 ALTERNATIVE FUNCTIONAL FORMS AND MODEL SELECTION

  • The shared-m Skaling fit is selected as the primary scaling form: among 18-parameter candidates it has the lowest observed AICc and BIC, with comparable held-out long-horizon errors.The paired-coordinate bootstrap does not consistently rank candidates, so the selection does not establish that the shared-m form is uniquely correct.Source: §G.3 · pp. 36–37

G.4 TOKEN MULTIPLIERS

  • A token multiplier above one means the baseline requires more tokens than the indicated optimizer to reach the same loss; AdamW is the usual baseline.The paper reports fit-to-fit multipliers for smooth curves and measured-to-fit multipliers at experimental settings; extrapolated points depend on baseline tokens beyond the measured budget.Source: §G.4 · pp. 37, 39

H OUTSCALING ACROSS TRAINING HORIZONS

H.1 DEFINITION AND FLOOR–DECAY TRADEOFF

  • Profile-fit analysis treats floor–decay profiles as compatible decompositions, testing whether observed outscaling requires an advantage in token-decay exponent, high-token loss limit, or both.The profile procedure is repeated across additive-OT and Skaling families with shared or optimizer-specific model exponents; its 95% display threshold is not a calibrated confidence interval.Source: §H.1 · pp. 39–40

H.2 PAIRWISE COMPARISONS UNDER UNIFORM WEIGHT DECAY

  • ADANA, Muon, and SOAP show distinct horizon-dependent comparisons with AdamW, and ADANA becomes the strongest primary treatment for the 51M model at 128× OT.Muon is strongest over most short and intermediate horizons, while SOAP’s advantage dips and then rises at the longest horizons. SOAP is strongest at 256× OT and for the 124M model at 32× OT, so ADANA does not uniformly dominate matrix-preconditioned methods.Source: §H.2 · p. 40

H.3 STRONGEST TREATMENT COMPARISONS · H.4 EQUIVALENT OT SCALING AND THE DANA PLRF PREDICTION

With each optimizer’s strongest measured treatment, ADANA’s relative token efficiency increases with overtraining, while its equivalent-OT scaling is close to the DANA effective-clock prediction across model sizes.

H.3 STRONGEST TREATMENT COMPARISONS

  • The strongest-treatment comparison uses log-time weight decay with momentum cooldown for ADANA, log-time weight decay for AdamW and SOAP, and uniform weight decay for Muon.Log-time weight decay did not improve Muon at high overtraining factors.Source: §H.3 · p. 40
  • Horizon-tuned momentum could not be included in the full three-model comparison because those experiments were available only for the 51M model.Source: §H.3 · p. 42
  • ADANA’s relative token efficiency rises with overtraining, catching or overtaking log-time-WD AdamW, uniform-WD Muon, and log-time-WD SOAP, though SOAP is especially strong at 51M and 256× OT.The comparisons favor an improved ADANA token-decay exponent, but the division between token-decay and high-token-limit advantages remains less well identified.Source: §H.3 · pp. 42–43, Figure 23 · p. 43
  • The profiles favor an improved ADANA token-decay exponent across functional forms, while the precise division between token-decay and high-token-limit advantages remains uncertain.ADANA gains on all three optimizers, but SOAP performs particularly well for the 51M model at 256× OT.Source: §H.3 · pp. 42–43, Figure 23 · p. 43

H.4 EQUIVALENT OT SCALING AND THE DANA PLRF PREDICTION

  • For fixed batch size and sequence length, optimizer updates scale with tokens and overtraining; accumulated learning rate therefore provides an effective clock beyond update count alone.Schedules reaching the same accumulated learning rate make approximately the same progress along a quadratic eigendirection.Source: §H.4 · pp. 43–44
  • ADANA’s effective optimization clock grows relative to a linear baseline with exponent 1 −κ = 0.15, yielding the equivalent-OT scaling reference 2 −κ = 1.15.DANA’s PLRF analysis predicts outscaling above the high-dimensional line before the model-capacity floor, but the loss exponent varies by phase and spectra (Ferbach et al., 2025).Source: §H.4 · p. 44
  • 1.20, 1.16, and 1.15 are the equivalent-OT slopes for combined-treatment ADANA on 51M, 124M, and 253M models, close to the 1.15 PLRF prediction; uniform-WD slopes are 1.15, 1.06, and 1.03.The authors treat 2 −κ as a theoretically motivated reference, not a direct transformer prediction, because DANA theory compares DANA with SGD on PLRF, unlike these ADANA–AdamW experiments.Source: §H.4 · p. 44, Figure 24 · p. 45

I BATCH SIZE AND OPTIMIZER UPDATE COUNT … J.2 LADANA APPLIES DANA AFTER PRECONDITIONING

At fixed tokens, batch size changes both optimizer-update count and gradient noise, complicating batch-scaling comparisons. The paper also introduces LADANA to test log-time momentum after preconditioning, while the fixed-memory comparison finds little systematic effect from changing preconditioner order.

I BATCH SIZE AND OPTIMIZER UPDATE COUNT

  • At fixed token count, increasing batch size reduces sequential optimizer updates and gradient variation, confounding batch benefits with the additional updates available at smaller batches.The passage notes that fixed-token comparisons in Ferbach et al. (2026) and Marek et al. (2025) share this identifiability limitation.Source: I BATCH SIZE AND OPTIMIZER UPDATE COUNT · p. 45
  • AdamW’s optimal second-moment memory decreases approximately as 1/B, while ADANA’s optimal g3 rises approximately linearly with B and its jointly optimized base learning rate falls.These are observed batch-scaling trends, not a general batch-scaling law for ADANA.Source: I BATCH SIZE AND OPTIMIZER UPDATE COUNT · p. 45, Figure 25 · p. 46
  • At 8× OT, ADANA’s loss begins rising at B = 64, while AdamW stays near its B = 32 performance through approximately B = 256; both lose efficiency at the largest batches.The fixed-token, fixed-sequence-length experiment varies global batch size from 32 to 1,024 on the 51M model.Source: I BATCH SIZE AND OPTIMIZER UPDATE COUNT · p. 45, Figure 25 · p. 46
  • The fixed-token, fixed-sequence-length slice does not establish a general ADANA batch-scaling law; matched-update, matched-token-per-update, and broader hyperparameter comparisons remain needed.The passages identify unresolved effects of jointly tuning g3, κ, and δ, batch-conditioned weight decay, and different batch-size/sequence-length factorizations.Source: I BATCH SIZE AND OPTIMIZER UPDATE COUNT · p. 46

J.1 ADAMW AND LAPROP ISOLATE PRECONDITIONER ORDER

  • AdamW and LaProp remain close across measured horizons; LaProp at β1 = 0.95 has a modest shorter-horizon advantage, with no separation that grows systematically with OT.This matched-preconditioner-order comparison serves as the fixed-memory control; both optimizers retain a horizon-independent first-moment window.Source: Figure 26 · p. 47, §J.1 · p. 48

J PRECONDITIONER ORDER IN FIXED- AND SCHEDULED-MEMORY OPTIMIZERS

  • LADANA reverses ADANA’s ordering by preconditioning each gradient before DANA log-time accumulation, testing whether scheduled-memory outscaling persists with normalized gradients.The paper also raises a possible stability benefit from normalizing unusual gradients before they enter growing memory, but its experiments compare final loss and do not directly establish that benefit.Source: J PRECONDITIONER ORDER IN FIXED- AND… · p. 47

J.2 LADANA APPLIES DANA AFTER PRECONDITIONING

  • LADANA stores gradients normalized by the preconditioner available when observed, unlike ADANA, which applies the current preconditioner to its accumulated raw-gradient memory.The two rules coincide when the preconditioner is constant over the effective memory window and remain close when it changes little; a fixed-v variant keeps log-time momentum but uses β2 = 0.98 for the denominator memory.Source: §J.2 · p. 49

J.3 LADANA RESULTS AND LIMITATIONS … L.6 MOMENTUM COOLDOWN SWEEPS

Momentum cooldown substantially improves ADANA, while LADANA preserves much of ADANA’s uniform-weight-decay outscaling but has unresolved long-horizon behavior under log-time weight decay. The appendix documents the learning-rate sweep evidence, while the optimal cooldown rule and its mechanism remain open.

J.3 LADANA RESULTS AND LIMITATIONS

  • LADANA with log-time denominator memory performs similarly to ADANA through 128× OT under uniform weight decay on 51M, but slightly underperforms at 256×; fixed-v LADANA falls behind at long horizons.Under log-time weight decay, LADANA remains similar through 64× OT but deteriorates sharply at 128×; because its weight-decay coefficient was not separately swept, this remains an unresolved interaction, not evidence of inherent instability. Preconditioning before accumulation therefore preserves most of ADANA’s uniform-WD outscaling, while stability and compatibility with log-time weight decay remain open.Source: §J.3 · p. 49

K.1 MOTIVATION: LONG MEMORY CAN BECOME STALE DURING TERMINAL DECAY

  • ADANA’s log-time momentum timescale grows approximately linearly with optimizer updates, but can become stale during terminal learning-rate decay as gradients from earlier, larger-step regimes persist.Cooldown is motivated as retaining growing memory while learning-rate changes are slow and shortening it when terminal decay outpaces momentum response; staleness is a motivation, not an empirically established explanation of the gains.Source: §K.1 · p. 51

K.2 DEFINITION OF MOMENTUM COOLDOWN

  • Momentum cooldown limits ADANA’s memory timescale by the learning-rate decay timescale, leaving the original rule unchanged before the timescales cross and concentrating the kernel on recent gradients during rapid decay.The construction depends on relative learning-rate changes, so it is invariant to the schedule’s overall scalar scale; the principal ADANA treatment applies cooldown to both first- and second-moment states.Source: §K.2 · p. 51

K.3 MOMENTUM COOLDOWN TREATMENT SELECTION

  • At 8× OT, first-moment cooldown improves AdamW validation loss by 2.30 millipoints; ADANA memory-window cooldown improves validation loss by 11.47 millipoints, more than the other tested interventions.For AdamW, cooling both moments gives essentially the same improvement as cooling the first moment alone, while cooling only the second moment is unstable at every tested learning rate, with losses from 15.2 to 42.9. For ADANA, combining memory-window cooldown with long-memory contribution taper improves loss by 6.13 millipoints, versus 0.92 millipoints for taper alone.Source: §K.3 · p. 52

K.4 WHERE MOMENTUM COOLDOWN HELPS AND WHAT REMAINS OPEN

  • Momentum cooldown modestly benefits AdamW but substantially improves ADANA across all three model sizes; on 51M, it complements log-time weight decay, with the combination yielding the lowest losses at the longest horizons.The final-loss comparisons are reported in Figure 6, with underlying sweeps in Figures 41 and 43.Source: §K.4 · p. 52
  • Cooldown’s optimal rule and mechanism remain unresolved: it has been tested only with cosine decay to zero, its unit-timescale crossover is not established as optimal, and final losses cannot establish causality.Analogous rules for Muon and SOAP have not been tested; the evidence supports the narrower conclusion that shortening ADANA’s memory during terminal decay produces an empirical gain.Source: §K.4 · pp. 52–53

L.1 LEARNING RATE SCHEDULE SELECTION; L.2 WEIGHT DECAY SWEEPS; L.3 UNIFORM-WD AND LOG-TIME-WD SCALING SWEEPS

  • Appendix L.1–L.3 supplies learning-rate sweep grids for schedule selection, weight-decay coefficients, and uniform- and log-time-WD scaling comparisons.The WD comparisons use nested sweeps, selecting a minimum from an independently extended learning-rate sweep at each coefficient before comparing coefficients; the scaling sweeps span model sizes and OT factors.Source: Figure 30 · p. 55, Figure 31 · p. 56, §L.2 · p. 57

L.4 FIXED-MEMORY SWEEPS; L.5 PRECONDITIONER-ORDER SWEEPS; L.6 MOMENTUM COOLDOWN SWEEPS

  • Figures 36–37 report the complete learning-rate sweeps underlying Appendix F and Figure 3 memory-tuning results, while Figures 38–40 underpin retained preconditioner-order comparisons in Appendix J and Figures 41–42 underpin Figure 6 momentum-cooldown comparisons.Figure 37 covers fixed-memory sweeps from 16× through 128× OT.Source: §L.4 · p. 62, Figure 37 · p. 63, §L.5 · p. 64

Related papers

Loading 2609.04577v1…