The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

1 Introduction

• Closer first- and second-moment decay rates in large-scale training motivate studying tied-β Adam, where the decay rates are equal.The introduction characterizes this regime as structurally simpler.Source: §1 · p. 1
• The transformed ratio yt has a stable, heavy-tailed distribution across models, tasks, and training stages, unlike raw moment magnitudes.The paper describes yt as isolating stochastic fluctuations from deterministic scaling.Source: §1 · p. 2
• A fixed 4-bit codebook can nearly match full-precision Adam without auxiliary scaling, while the ratio view also yields a learning-rate transfer rule and a Signum-type constant-state limit.The paper describes tied-β Adam as sign-based momentum attenuated by yt.Source: §1 · p. 2
• The paper does not claim that tied-β Adam universally dominates standard configurations or that the ratio law fully characterizes optimizer dynamics globally.Source: §1 · p. 2

2 Background and Related Work

• Adam’s bias-corrected update divides the first-moment EMA by the square root of the second-moment EMA, with ε for numerical stability.Source: §2 · p. 2
• Signum keeps the momentum direction and uses a fixed coordinatewise step magnitude, unlike Adam’s adaptive normalization.Source: §2 · p. 2
• Prior analyses interpret Adam through sign-based direction, adaptive magnitude, or local signal-to-noise structure; this work identifies a comparatively stable distributional quantity within its scaling term.Source: §2 · pp. 2–3
• When β1 = β2, vt ≥ mt^2 for all gradient sequences and times, supporting a cleaner online mean–variance interpretation of tied-β Adam.Source: §2 · p. 3

3 Hidden Ratio: Theoretical and Empirical Properties

• The transformed ratio yt measures the variance-like residual remaining in Adam’s denominator after removing the deterministic scale induced by tied exponential averaging.Source: §3 · p. 3

3.1 Statistical Structure of yt: Stability and Heavy Tails

• The shifted inverse-square reference retains the leading shift effect and inverse-square singularity, and its survival curve depends to second order on the pooled shifts’ second moment.Source: §3.1 · p. 4
• Across pretraining, SFT, RLHF, and the tested β values, yt has a compact single-digit bulk and persistent right tail, with only mild training-stage drift.The empirical CDFs remain close to the 2/Z^2 reference over a broad range.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Robustness of yt to coarse quantization

• In Figure 3, truncating yt’s right tail and quantizing its remaining bulk to 16 levels still yields an optimizer that closely tracks standard Adam’s behavior and final performance, with original mt and vt recursions unchanged.Source: §3.2 · p. 4, Figure 3 · p. 5

4.1 A scalar recursion for the transformed ratio

• Equation 10 gives a scalar recursion in which sign(mt) determines the update direction and yt controls attenuation magnitude.The recursive formulation permits replacing the raw state pair (mt, vt) with (mt, yt).Source: §4.1 · p. 6, §4 · p. 6

4.2 Coarse representation through the recursion

• A 16-level fixed codebook quantizes yt directly, without per-block scaling, learned quantizers, or auxiliary normalization metadata.The codebook places more resolution in the moderate range where yt concentrates while retaining larger values for its right tail.Source: §4.2 · p. 6

4.3 Experiments on low-precision quantization of the transformed ratio

• Across β = 0.90, 0.92, 0.95, and 0.97, transformed-ratio variants retain Adam-aligned favorable learning-rate regions and similar degradation at overly large rates.The sweeps compare FP32 or FP8 first-moment storage with 4-bit yt.Source: Figure 7 · p. 7, §4.3 · p. 8
• Across pre-training, SFT, and RLHF, coarse transformed-ratio representation recovers Adam-like behavior, remains competitive, and can improve late-stage performance.The low-precision variant may initially lag in SFT and RLHF, then improve steadily and often match or slightly exceed Adam later in training.Source: §4.3 · pp. 7–8

5.1 A constant-state approximation and the Signum-like limit

• Replacing stochastic yt with yconst yields a Signum-type update, with a learning-rate matching rule based on matching expected attenuation.The constant-state approximation preserves the dominant effective scale but not yt’s coordinate-wise fluctuations or temporal variation.Source: §5.1 · p. 8
• Under the shifted-reference law, the attenuation-matched constant remains slightly above 3 across β ∈[0.90, 0.97] and c ∈{0, 0.1, 0.2}.This supports using a moderate constant to capture Adam’s typical attenuation scale under that distributional model.Source: §5.1 · p. 8
• The constant-yt proxy keeps a Signum-like update in Adam’s native learning-rate parameterization, allowing Adam-style tuning rather than a separate optimizer-specific search.The matching rule is a calibration; full Adam retains yt’s fluctuations and temporal variation.Source: §5.1 · pp. 8–9

5.2 Learning-rate sweeps across β: empirical observations

• Signum’s preferred learning-rate region shifts smaller as β increases, while yconst ∈{3, 5} best matches Adam’s preferred region among tested constants.The observed shift is consistent with stronger transformed-ratio attenuation at larger β; constant proxies preserve Adam’s learning-rate scale better than pure Signum.Source: §5.2 · p. 9, Figure 8 · p. 9

6 Conclusion and Limitations

• The empirical yt distribution has a concentrated moderate-range bulk and a persistent heavy tail across training settings.The ratio representation factors out explicit β-dependent scale, exposing structure obscured in raw moment states.Source: §6 · p. 9
• A fixed 4-bit codebook for yt preserves performance close to full-precision Adam across the language-model training regimes studied.The same ratio view describes tied-β Adam as sign-momentum attenuated by yt, with a constant yt yielding a Signum-type limit and explicit learning-rate scaling.Source: §6 · p. 10
• Experiments cover Transformer language models only, up to 1B parameters; larger-scale and non-Transformer validation remains future work.Table 1 is identified as reporting y_eff const under Yc = 2/(Z + c)^2.Source: Table 1 · p. 9, §6 · p. 10

A Distributional properties of the transformed ratio state

• Under a local Gaussian window, the EMA denominator has form mt = σm(Z + a), yielding the shifted inverse-square reference 2/(Z + a)^2; numerator fluctuations enter as relative correction ηt.The calculation isolates the inverse-square denominator rather than modeling training globally.Source: A Distributional properties of the… · p. 14
• For pooled shifts A, the mixed reference uses YA = 2/(Z + A)^2, while Yc = 2/(Z + c)^2 with c^2 = E_P[A^2] matches its survival function to second order.The passage identifies A as a shift sampled from the pooled-shift distribution.Source: A Distributional properties of the… · pp. 14, 16
• Chebyshev’s inequality converts the probability bound into relative-error control for yt/Ya, for 0 < u < 2δ.The derivation also states that the fluctuation term is conditionally centered by construction.Source: A Distributional properties of the… · p. 15
• For fixed A and q = √(2/k), the mixed-reference tail probability is Pr(YA > k | A) = Φ(q − A) + Φ(q + A) − 1.A Taylor expansion in the shift yields a second-order term in A² and a remainder of order A⁴.Source: A Distributional properties of the… · pp. 15–16

• When β1 = β2, Adam’s adaptive factor is a sign-momentum direction multiplied by an attenuation factor, with a nonnegative residual preventing amplification beyond the corresponding sign-momentum magnitude.This is the regime in which the denominator decomposes into a squared mean and a variance-like residual.Source: B Additional Motivation and Related… · p. 16
• Tied β is necessary and sufficient for vt − mt^2 to remain nonnegative for every gradient sequence and time; mismatched rates can make the residual negative.The proof uses a tight bound, so when the condition fails there exists a gradient sequence violating nonnegativity.Source: B Additional Motivation and Related… · p. 17
• For classical Adam with (β1, β2) = (0.9, 0.999), the large-spike limit has 1 + rt ≈ 0.1, illustrating the mismatch’s negative-residual failure mode.The authors do not claim tied β is universally optimal; related work instead links closer decay rates to some nonstationary-training stability settings.Source: B Additional Motivation and Related… · pp. 16–18

C.1 Step-by-step derivation of the transformed-state recursion

• The appendix derives the scalar transformed-state recursion by expressing the current gradient through the first-moment recursion, substituting into the second-moment recursion, and rewriting moments through the transformed state.The section also covers initialization and numerically unstable regions.Source: C Details for the transformed-state… · p. 18, §C.1 · pp. 18–19

C Details for the transformed-state recursion and implementation

• The implementation appendix presents the recurrence derivation, initialization, numerical handling, and ablations as the key practical details for the transformed-state formulation.It identifies these as the contents of the implementation discussion.Source: C Details for the transformed-state… · p. 18

C.1 Step-by-step derivation of the transformed-state recursion

• TR-Adam pseudocode initializes the transformed ratio, applies nearest-codebook quantization, and uses a bias-corrected first moment in the parameter update.The appendix describes ablations evaluating the ratio-recursive formulation.Source: C Details for the transformed-state… · p. 18, §C.1 · p. 19

• TR-Adam remains close to full Adam in pre-training and often improves over Adam in post-training.The reported final validation metrics average the final three validation points; the distribution evidence spans five representative settings, including pre-training, SFT, and ReMax.Source: D Additional Experimental Results · p. 19

E Datasets and Implementation Details

• Optimizer comparisons keep the dataset, architecture, batch size, learning-rate schedule, and evaluation protocol fixed, changing only the optimizer update.Short-run supervised experiments use zero weight decay; the long-run 20B-token setting uses 0.1.Source: E Datasets and Implementation Details · pp. 19–20
• Pre-training learning rates are tuned separately at smaller scales, while the Pythia-1B 20B-token experiment reuses 0.001 selected from a short Adam sweep.SFT and RLHF use default VERL learning rates; the cited pre-training sweeps differ by model scale.Source: E Datasets and Implementation Details · p. 20
• The experiments include Llama-style 20M, Pythia-style 160M, and official Llama-3.2-1B and Pythia-1B models; 1B-scale runs use 8×H200 GPUs and smaller runs use 16×RTX 4090 GPUs.The 1B-scale pre-training models are trained from scratch by reinitializing parameters.Source: E Datasets and Implementation Details · pp. 20–21
• Pre-training uses FineWeb; SFT uses UltraChat with teacher forcing; and RLHF uses UltraFeedback with Llama-3.2-1B as actor and Skywork-Reward-V2-Llama-3.2-3B as reward model.SFT and RLHF use the VERL trainer and pipeline, respectively.Source: E Datasets and Implementation Details · pp. 20–21

F Broader Impact

• The work focuses on Adam’s internal adaptive dynamics and training efficiency, not downstream model behavior or human-subject research.The authors describe its mechanistic perspective as potentially guiding optimizer analysis, hyperparameter choices, and simpler Adam-like methods.Source: F Broader Impact · p. 22

NeurIPS Paper Checklist

• The checklist reports that experimental information is disclosed for reproducibility and that an anonymized code package is included in the supplemental material.A README is also provided with the code.Source: NeurIPS Paper Checklist · pp. 25, 26, 29
• Main experiments were not repeated with many independent random seeds because large-scale language-model training is computationally costly.The authors instead report complete training curves and compare trends across stages, datasets, and optimizer variants.Source: NeurIPS Paper Checklist · p. 26
• The checklist states that the research conforms to the NeurIPS Code of Ethics and that asset sources are credited, with licenses and terms reported or referenced where applicable.Source: NeurIPS Paper Checklist · pp. 27–28
• More efficient training may reduce costs and improve accessibility, while potentially amplifying both beneficial and harmful downstream uses.Source: NeurIPS Paper Checklist · p. 27
• The work does not introduce or release a new high-risk pretrained model, image generator, or scraped dataset.The checklist describes its focus as optimizer-state representations and training methodology.Source: NeurIPS Paper Checklist · p. 28