PaperlayerView exact revision on arXiv

Source-linked AI summary

When and How Does Muon Precondition? The Role of Structured Gradient Noise

Xiaoran Chen, Binghui Li, Liwei Wang, Pinyan Lu

arXiv:2610.31728v1cs.LG

TL;DR

The paper asks when Muon’s preconditioning benefits survive stochastic gradient noise, a question complicated by nonlinear normalization of mini-batch gradients. It analyzes isotropic noise, structured task sampling, label reliability, and momentum, finding that the outcome depends on noise structure and sampling while a preliminary pretraining comparison favors fast–slow momentum over Muon.

  • Problem

    The interaction of task occurrence, gradient reliability, and temporal averaging in stochastic mini-batch gradients is not captured by population-gradient analyses alone.

    Source: §1 · p. 2

  • Method

    The paper combines a low-SNR isotropic-noise diagnostic with associative-memory analyses of finite-batch sampling, label noise, sparse support, and momentum averaging and tracking.

    Source: §1 · p. 3

  • Results

    Isotropic low-SNR noise yields only a scalar first-order mean response, while structured sampling and label noise produce frequency-dependent regimes and sparse-support rotation can cause negative transfer in SignSGD.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The analysis motivates longer memory for rare, slowly varying directions and reports lower validation loss than Muon in a preliminary 0.12B pretraining experiment using fast–slow momentum.

    Source: §1 · p. 3, §7 · p. 10

    The momentum window is a local design guide, not a global convergence guarantee, and extending the analysis to evolving, non-orthogonal representations and validating across training scales remain open directions.

    Source: §5 · p. 9, §7 · p. 10

Abstract

from arXiv · show

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon's preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate. This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning. Our analysis shows that Muon can soften the dependence on task frequency from linear to square-root, helping balance the learning of frequent and rare tasks. We further identify rotation-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients. Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast-slow momentum.

1 Introduction

The paper asks when Muon’s preconditioning benefits survive stochastic gradient noise, then develops diagnostics and structured-sampling analyses. It also studies label reliability, rotation-induced negative transfer, and momentum’s averaging–tracking trade-off.

1 Introduction

  • Under isotropic low-SNR noise, Muon’s first-order conditional mean matches learning-rate-rescaled SGD, so spectral normalization alone provides no directional preconditioning.The analysis motivates examining noise structure to explain directional gains.Source: §1 · pp. 2–3
  • For heterogeneous task frequencies, the paper establishes an exact two-task balancing threshold and finds higher-rank growth depends on expected task count Bpj.Rare-task balancing requires batches on the order of the inverse rare-task frequency.Source: §1 · pp. 2–3
  • With label noise, task coverage and gradient reliability separate; a local isotropic-covariance closure predicts Muon’s frequency dependence softens from linear to square-root.The prediction assumes locally homogeneous frequencies.Source: §1 · pp. 2–3
  • In a rotated sparse-support construction, Muon preserves the missing-task margin while coordinatewise SignSGD can decrease it on frequent-task-only batches.SignSGD serves as a proxy for Adam’s coordinatewise adaptivity.Source: §1 · pp. 2–3
  • Momentum reduces sampling fluctuations but adds tracking lag, motivating longer memory for rare, slowly varying directions and shorter memory where rapid tracking is needed.The paper connects this account to curvature-conditioned and occurrence-dependent momentum designs, and reports controlled experiments plus a preliminary pretraining comparison.Source: §1 · pp. 2–3

2 Related Work

Related work studies Muon’s spectral normalization, stochastic dynamics, associative-memory learning, and heterogeneous momentum. This paper’s stated distinction is its focus on random task support and persistent label fluctuations.

Muon and adaptive optimization

  • Prior work studies Muon’s convergence and structured-matrix preconditioning, while this paper examines stochastic gradient structure and contrasts polar normalization with coordinatewise sign updates.SignSGD is treated as a coordinatewise-adaptivity comparison.Source: §2 · p. 3

Stochastic spectral dynamics

  • Recent analyses address batch-dependent spectral dynamics, gradient-noise scales, and stochasticity in Shampoo; this work instead separates task occurrence from gradient reliability.The isotropic Gaussian result is presented as a fixed-dimensional diagnostic for absent first-order directional preconditioning.Source: §2 · p. 3

Associative memory learning

  • Associative-memory research covers heterogeneous frequencies, label noise, and capacity, while this paper isolates finite-batch thresholds, frequency regimes, and rotation-induced negative transfer.Muon variants and balanced learning on imbalanced data are also discussed as prior directions.Source: §2 · p. 3

Heterogeneous momentum

  • Prior momentum work includes spectral filtering, activation-keyed memory, and curvature-conditioned designs; this paper offers a sampling-based account of variance reduction and tracking lag.Figure 1’s caption reports that isotropic noise collapses normalized directional gains near one and keeps the conditional-mean matrix within a few percent of the scalar-response prediction.Source: §2 · p. 3, Figure 1 · p. 4

3 Preliminaries

The preliminaries define the partial polar map and idealized Muon dynamics. They then show that isotropic Gaussian noise yields only a scalar first-order mean response, motivating structured-noise analysis.

3.1 Notations.

  • The partial polar map replaces each nonzero singular value with one, is positively scale invariant and orthogonally equivariant, and preserves the right nullspace.The idealized Muon update averages stochastic gradients with momentum before applying this map; β = 0 gives momentum-free Muon.Source: §3.1 · p. 4

3.2 Warmup: Isotropic Noise Limits Preconditioning

  • At fixed dimension and low signal-to-noise ratio, isotropic Gaussian noise makes Muon’s first-order conditional mean a scalar multiple of the population gradient, matching SGD with a rescaled learning rate.This equivalence concerns conditional means, not update covariances or stochastic trajectories; directional gains therefore require structure absent from the isotropic benchmark.Source: §3.2 · pp. 4–5

4 Structured Sampling in Associative Memory

The section develops stochastic associative-memory dynamics from task-coordinate gradients through finite-batch balancing, higher-rank regimes, label noise, and sparse-support comparisons. Its results distinguish task occurrence from gradient reliability and polar normalization from coordinatewise updates.

4.1 Associative memory and stochastic gradients

  • When a task is absent from a batch, momentum-free Muon leaves its task-coordinate update column zero by preserving the gradient’s nullspace.Historical momentum can still produce a nonzero update for a task absent from the current batch.Source: §4.1 · p. 5

4.2 Two tasks: an exact finite-batch balancing transition

  • For two tasks, the exact balancing threshold is Bbal = log 2/[-log(1 − pmin)]; above it, the margin gap has bounded exponential moments and vanishing asymptotic growth rate.For rare tasks, Bbal is approximately (log 2)/pmin; the theorem does not cover the critical case.Source: §4.2 · p. 5

4.3 Higher rank: frequency-dependent dynamical regimes

  • In higher-rank sampling, margins are unchanged when a task is absent, while measured late growth rates align against Bpj and approach the full-batch response as task counts increase.The local estimates do not establish uniform long-time balancing of the stochastic hierarchy.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4 Label noise: three stochastic regimes

  • Label noise adds a corruption-count scale Bpjρα: a task can be sampled often yet retain a low-SNR gradient, while a local closure predicts Muon’s contraction scales as √Bp versus p for SGD.A broad intermediate regime requires small ρα; the closure uses auxiliary covariance and locally homogeneous-frequency approximations.Source: §4.4 · pp. 6–7

4.5 Sparse support: Muon versus SignSGD

  • Under sparse support, orthogonally equivariant Muon preserves the missing-task nullspace, whereas rotated SignSGD can decrease the rare-task margin on every frequent-only batch when θ > π/4.Muon’s rare-task margin remains nonnegative and grows on its occurrence clock; the figure compares paired streams with B = 1, p1 = 0.95, and p2 = 0.05.Source: §4.5 · p. 7, Figure 4 · p. 8

5 Momentum: Averaging and Tracking

Momentum averages stochastic gradients to reduce sampling fluctuations, but longer memory increases tracking lag. The analysis gives a local window balancing these effects and cautions that variance-equivalent batch size does not capture the full batch distribution.

5 Momentum: Averaging and Tracking

  • The frozen-state EMA has variance-equivalent batch size Beff(β) = B(1 + β)/(1 −β), with relative task-frequency fluctuations vanishing as Beffpj grows.At a frozen state, EMA averages task frequencies without changing the physical mini-batch size.Source: §5 · p. 8
  • Figure 5 shows gap-bias curves nearly collapsing against Beffpmin, while late-window mean-gap error grows when η/(1 −β) is no longer small.The tracking comparison uses fixed B and β and a common horizon τmax = 50.Source: Figure 5 · p. 9
  • Replacing B by Beff matches task-frequency variances, but not task-absence probabilities or the full batch distribution.EMA updates are temporally correlated, so lower instantaneous variance need not mean lower long-time diffusion.Source: §5 · p. 9
  • Under local tracking assumptions, η ≪1 −β ≪Bpj is a sufficient window combining averaging with small frozen-state buffer error.The buffer error is at most LSηβ/(1−β) when ∥St −St−r∥F ≤LSηr; transferring this bound to polar directions needs additional local control.Source: §5 · pp. 8–9

6 Fast–Slow Momentum in Language-Model Pretraining

The paper applies curvature-conditioned fast–slow momentum in language-model pretraining, adding a slow contribution in estimated flat directions to the fast Muon update. In a single 0.12B-model run, the variant has lower validation loss than Muon at every shared checkpoint, with a final reduction of 0.0209.

6 Fast–Slow Momentum in Language-Model Pretraining

  • The variant retains the fast Muon update and adds slow momentum in estimated flat directions identified through lower singular subspaces of fast momentum.This implements a practical counterpart to the averaging–tracking trade-off without explicit task-frequency estimates.Source: §6 · p. 9
  • Across all 138 shared checkpoints, fast–slow momentum has lower validation loss than Muon, decreasing step-30,000 loss from 3.0097 to 2.9888.The comparison uses a 0.12B dense language model with the remaining training configuration fixed; results are from one run.Source: §6 · p. 9, Figure 6 · p. 10
  • The single-run comparison does not establish variability across seeds or isolate the effects of individual algorithmic components.Figure 6 presents unsmoothed validation-loss curves for steps 20k to 30k.Source: §6 · p. 9, Figure 6 · p. 10

7 Conclusion

The conclusion synthesizes results on isotropic noise, structured task sampling, label reliability, and momentum, while identifying limits to the momentum analysis and preliminary pretraining evidence. The appendix guide maps the proofs, models, and numerical protocols supporting these analyses.

7 Conclusion

  • Muon’s first-order mean update matches learning-rate-rescaled SGD under isotropic low-SNR noise, while structured sampling yields balancing thresholds, task-count effects, and square-root frequency dependence under a stated covariance closure.The conclusion also reports Muon’s protection against rotation-induced negative transfer under sparse sampling.Source: §7 · p. 10
  • The momentum analysis motivates longer memory for rare, slowly varying directions, but evolving non-orthogonal representations and validation across training scales remain open.The practical pretraining evidence is preliminary.Source: §7 · p. 10

AI use statement

  • The authors report using generative AI for some mathematical proof and derivation assistance, translation, and code and figure generation, with review of the assisted work.They state that AI did not formulate the research questions, conceptual framework, or theorem statements.Source: AI use statement · p. 10

Appendix

  • The appendices cover the isotropic response proof, associative-memory analysis, label-noise results, Muon–SignSGD contrast, EMA calculations, and numerical protocols.Appendix D supports higher-rank regimes, while Appendix H collects protocols and diagnostics.Source: Appendix · p. 15

A Isotropic Gaussian Mean Response: Proof of Proposition 3.1

The proof uses Gaussian smoothing to establish regularity of the expected partial-polar response, then uses orthogonal equivariance to constrain its derivative to scalar action. A Taylor expansion supports the local mean-response result.

A.1 Gaussian-smoothing proof

  • The proof conditions on the current iterate and uses Gaussian convolution to smooth the expected response despite the nonsmooth partial polar map.Differentiation under the expectation is justified locally by bounds on derivatives of the shifted Gaussian density.Source: A Isotropic Gaussian Mean Response:… · p. 15, §A.1 · p. 15
  • Orthogonal equivariance forces the derivative operator to act as T(X) = cX, with a common scalar across matrix units.Independent row and column permutations make the coefficients for each matrix unit equal.Source: §A.1 · p. 16

A Isotropic Gaussian Mean Response: Proof of Proposition 3.1

  • The argument uses a Taylor expansion with integral remainder and odd symmetry of the expected response to control the local approximation.The coefficient is identified by taking the trace in the orthonormal matrix-unit basis.Source: §A.1 · pp. 16–17

B Associative-Memory Setup and Gradient Derivation

The setup uses task-specific cross-entropy losses and mini-batch empirical task frequencies to form the sampled gradient and task-coordinate Muon update. The two-task finite-batch analysis reduces the dynamics to margins and contrasts full-batch behavior with occurrence-limited small batches.

B.1 Gradient of the Associative Memory Model

  • Mini-batch task frequencies define a diagonal empirical-frequency matrix, and orthogonal equivariance yields the task-coordinate Muon update.The setup begins from each task’s cross-entropy loss.Source: §B.1 · pp. 17–18

C Two-Task Finite-Batch Dynamics and Refinements

  • The exactly solvable two-task model assumes independent batches of B samples with p1 ≥ p2 > 0; its stochastic gradient is rank one, not evidence of higher-rank spectral effects.The finite-B statements are established in subsections C.1–C.4; later refinements do not assert a second phase transition.Source: C Two-Task Finite-Batch Dynamics and… · p. 18

C.1 Exact Reduction to the Margin Dynamics

  • The sampled matrix has rank one whenever at least one task is present, and taking column differences gives the margin dynamics.The passage identifies this reduction but does not provide the resulting equations in readable form.Source: §C.1 · p. 18

C.2 Full-batch Dynamics

  • In full-batch dynamics, both margins diverge, so the exponential terms dominate.The passage does not provide the ensuing rate formula.Source: §C.2 · p. 19

C.3 Small Mini-batches: Occurrence-limited Imbalance

  • In small batches, each batch containing task 2 allocates the normalized update to task 2, while a pure task-1 batch allocates it to task 1.The cited passage states this occurrence-dependent allocation without a complete readable asymptotic-rate expression.Source: §C.3 · p. 19

C.4 Large Mini-batches: Recovery of Balanced Growth

This section studies balanced growth under full-batch Muon and how mini-batch fluctuations couple to balanced contrast updates. It also characterizes how momentum averages sampling noise and task frequencies across steps while tracking evolving states, including higher-rank dynamics with sparse occurrence clocks.

C.4 Large Mini-batches: Recovery of Balanced Growth

  • In the well-sampled regime, restoring drift in both tails yields vanishing asymptotic gap growth, and the limiting drift has a unique zero at γ = 0.The late-time drift result is an approximation, not an exact invariant law of the discrete process.Source: §C.4 · pp. 20, 21, 23
  • Mini-batch fluctuations shift the effective gap positively when p1 > p2, but the shift vanishes at large batch size; the local diffusion approximation recovers the full-batch gap.The associated leading stationary variance scales as η/(2Bp1p2) in the well-sampled regime.Source: §C.4 · pp. 23–24

C.5 Momentum: Averaging Across Steps and Tracking

  • Momentum’s reduction in instantaneous variance is offset by increased cross-step correlation, leaving the leading local long-time diffusion coefficient unchanged.The result applies in the well-sampled regime around the momentum-induced equilibrium, with corrections associated with finite effective sample size and tracking error.Source: §C.5 · p. 28
  • Momentum replaces noisy within-batch task frequencies with an exponentially averaged frequency, with Beff = B(1 + β)/(1 − β); when tracking evolving margins, this drives the gap toward the full-batch value.The EMA gives both tasks positive weight almost surely, and its finite-sampling displacement of the effective gap is suppressed according to Beff.Source: §C.5 · pp. 26, 27, 29

D Higher-Rank Sampling Dynamics and Momentum Averaging

  • For higher-rank sampling, the analysis separates sparse occurrence-gated clocks, balanced full-batch contrast, well-sampled coupling, and momentum tracking; cancellation leaves contrast-to-average coupling as the leading first-order effect.In the sparse regime, the occurrence rate satisfies rj = Bpj + O((Bpj)^2), and block symmetry is preserved at every step.Source: D Higher-Rank Sampling Dynamics and… · p. 29, §D.1 · p. 30

D.1 Sparse regime: exact occurrence-gated clock and block-symmetrized reference

  • The sampled block-symmetric process is an exact finite-B benchmark; a stronger finite-horizon continuum refinement is recorded for large expected task counts and small step size.Source: §D.1 · p. 30

D.2 Block decomposition at a symmetric reference state

  • The block basis is orthogonal, and the block-average matrices L and H have rank M−1 under the stated positivity and stochasticity conditions.Source: §D.2 · p. 31

D.3 Full-batch balanced contrast dynamics

  • Full-batch Muon exactly equalizes updates across within-block contrast directions, leaving remaining nontrivial structure in the block-average subspace.Source: §D.3 · p. 32

D.4 Mini-batch perturbation decomposition around the symmetric reference

  • The mini-batch perturbation analysis is local and one-step around a fixed block-symmetric reference, measuring the polar map's instantaneous response rather than full trajectory departure.Source: §D.4 · p. 32

D.5 Well-sampled regime: first-order cancellation and residual coupling

  • First-order within-block sampling fluctuations vanish on every contrast-to-contrast block, while sufficiently large expected counts per task make contrast-to-average coupling small across blocks.The uniform condition does not require every task to appear in every mini-batch.Source: §D.5 · pp. 35–36

D.6 Momentum: averaging across steps and tracking

  • Under a frozen state, momentum replaces the single-batch frequency matrix with its exponentially weighted average across steps.Source: §D.6 · p. 38
  • Momentum's covariance formulas match those for a single batch with B replaced by Beff, but the weighted average is not distributionally equivalent to a Beff-sample empirical frequency vector.Source: §D.6 · p. 38
  • Under local O(η) state variation, momentum's frozen-state mismatch is small on the adiabatic tracking scale; this estimate is not a global stability theorem.The local regime requires both small relative sampling fluctuations and an averaging window short enough to track gradient changes.Source: §D.6 · pp. 39–40

D.7 Technical refinement: finite-horizon well-sampled continuum limit

The well-sampled hierarchy approaches the full-batch continuum trajectory on fixed time horizons, while the stochastic approximation is not uniform over the entire half-line. The stochastic-label analysis distinguishes exact categorical results from projected and covariance-closure approximations, whose local response depends on anisotropic noise.

D.7 Technical refinement: finite-horizon well-sampled continuum limit

  • On every fixed continuum-time interval, the well-sampled stochastic recursion converges to the full-batch continuum flow, but the result does not guarantee uniform approximation over the entire half-line.The nonzero singular gap may shrink during training.Source: §D.7 · pp. 40–41

E Stochastic Labels: Exact Statistics, Projected Regimes, and Local Closure

  • The label-noise analysis separates exact categorical results, an exact projected counting model, and an approximate isotropic covariance closure for an explicit local rate.The closure is not an exact formula for the full matrix process because the categorical covariance is anisotropic.Source: E Stochastic Labels: Exact Statistics,… · p. 42

E.1 Complete categorical model: gradient dynamics

  • At the noisy population optimum, task-frequency fluctuations vanish exactly, leaving conditional label-sampling fluctuations as the persistent local stochasticity.Even frequent task observations can leave low-SNR gradients near the optimum when labels continue to fluctuate.Source: §E.1 · p. 43

E.2 Complete categorical model: exact covariance and reliability

  • Expected task counts Bp_j and incorrect-label counts Bp_jρα characterize distinct reliability scales, with Bp_jα/K needed to sample each incorrect class individually.The exact covariance and reliability formulas distinguish aggregate incorrect-label fluctuations from individual-class sampling.Source: §E.2 · p. 44

E.3 Complete categorical model: local Gaussian limit and structured response

  • The fixed-dimensional local Gaussian limit retains anisotropic response, but anisotropy alone does not guarantee improved conditioning.The limit keeps K, task frequencies, and label-noise level fixed, and is not uniform as task frequencies or label noise vanish.Source: §E.3 · p. 45

E.4 Projected counting model: task and corruption counts

The projected counting analysis separates task occurrence from label corruption, while the local rate comparison uses an isotropic covariance closure and a common local frequency approximation. The appendix also derives two-task SignSGD dynamics, showing how embedding rotation couples task margins.

E.4 Projected counting model: task and corruption counts

  • In the projected counting model, a corruption-sparse window supports near-unit deterministic-label large-margin growth, while resolved corruption reverses the projected drift.In the Poisson crossover, the large-margin drift is positive for λ < log 2 and negative for λ > log 2.Source: §E.4 · pp. 47–48

E.5 Moment-matched isotropic covariance closure

  • The explicit local-rate closure replaces anisotropic categorical covariance with an isotropic covariance preserving total noise energy, and assumes a common frequency scale across the sector.The approximation is intended for the corruption-resolved local regime; the counting analysis covers occurrence-limited and corruption-sparse regimes.Source: §E.5 · pp. 48–49

E.6 Local Muon rate under the covariance closure

  • Under the covariance and locally homogeneous frequency approximations, Muon’s local contraction coefficient scales as √p, whereas SGD’s scales linearly with p_j.These rates apply to the local small-signal regime near the noisy population optimum.Source: §E.6 · pp. 50–51

F Muon–SignSGD Sparse-Support Contrast: Complete Derivation

F.1 Rank-one reduction

  • In the rank-one reduction, the answer embedding contributes a positive scalar to both margins, while the query embedding determines their coupling.This isolates which embedding controls the two-task interaction.Source: §F.1 · p. 51

F.2 Rotation-dependent margin recursion

  • The two-task SignSGD margin recursion depends on the rotated-coordinate update, with the second coordinate equal to the negative of the second margin.The supplied recursion passage states this relation but does not give a further outcome comparison.Source: §F.2 · p. 52

F.3 Full-batch reference dynamics

Full-batch dynamics balance the two tasks at leading order, while stochastic sparse support and embedding rotation can alter task margins. The appendix also checks the paper’s sampling and isotropic-noise predictions with controlled numerical experiments.

F.3 Full-batch reference dynamics

  • The full-batch gap stays in an O(η) band around Dθ, and the tasks balance at leading order, although rotation affects the switching location and common growth rate.In the small-step limit, the band collapses to Dθ.Source: §F.3 · p. 53

F.4 Sparse-support negative transfer

  • Under sparse support, SignSGD’s embedding rotation can cause persistent negative transfer from the frequent task to the rare task, unlike Muon’s task-coordinate update.The mechanism is specific to the coordinate-sensitive elementwise sign operation.Source: §F.4 · p. 54

G Supplementary EMA Bias–Variance Calculation

  • The supplementary EMA calculation is an interpretation of tracking bias–variance, not a separate Muon convergence result or method-level contribution.It considers an exogenous affine drifting signal.Source: G Supplementary EMA Bias–Variance… · p. 54

G.1 Affine-drift calculation

  • When direction-specific noise-to-drift ratios differ, a shared constant EMA coefficient is strictly suboptimal for at least one positively weighted direction.This comparison does not establish a lower bound against every adaptive scalar-momentum method or end-to-end Muon acceleration.Source: §G.1 · p. 55

Appendix protocol Role in the paper

  • The figure map distinguishes direct checks, transfer tests, and the single-run language-model pretraining comparison across the paper’s main-text figures.Table 1 likewise keeps exact and reduced-model validations separate from transfer tests.Source: Appendix protocol Role in the paper · p. 56, Table 1 · p. 57

H Numerical Experiments and Reproducibility Details

H.1 Common protocol and map from main-text figures

  • Figure 7’s trajectories show different block-margin growth rates at B = 8, while all four blocks approach the unit full-batch rate at B = 1024.The main text presents the rates against Bp_j to expose the expected-count variable.Source: Figure 7 · p. 57

H.2 Isotropic Gaussian mean response with an anisotropic signal

  • Across eight settings, the median full-matrix relative error for the isotropic-noise directional-response check is 3.05%.The signal is deliberately non-isotropic, and the coefficient estimate uses independent Monte Carlo samples.Source: §H.2 · p. 57

H.3 Two-task finite-batch transition, asymptotics, and momentum

Controlled experiments test finite-batch balancing, hierarchical sampling, label-noise predictions, and sparse-support transfer. A preliminary language-model run reports lower validation loss with fast–slow momentum than with standard Muon, while its single-run design limits the comparison.

H.3 Two-task finite-batch transition, asymptotics, and momentum

  • Across three frequency settings, the finite-batch transition collapses around B/Bbal = 1; large-batch bias and stationary-noise exponents are −1.011 and −0.531.The latter are close to the predicted −1 and −1/2; tracking error grows when η/(1 − β) is no longer small.Source: §H.3 · pp. 58–59

H.4 Hierarchical well-sampled perturbation scaling

  • For higher-rank perturbations, contrast-to-average coupling scales with fitted slopes −0.512 against expected task count and −0.509 against its momentum-adjusted equivalent.The predicted scaling in both checks is −1/2.Source: Figure 9 · p. 59, §H.4 · p. 59

H.5 Label-noise large-margin drift: projected validation and full-matrix transfer

  • The projected large-margin model has RMSE 0.00618, while the complete-matrix update shares its sign transition near λ ≃ log 2 but is not expected to match the projected formula.The complete-matrix RMSE relative to that prediction is 0.0233.Source: §H.5 · p. 60

H.6 Label-noise local response: direct closure test and categorical stress test

  • The Gaussian closure check fits a frequency exponent of 0.502, while the full-categorical stress test gives Muon exponent 0.49397 versus SGD exponent 1.The categorical result is a stress test beyond the isotropic-closure assumptions, not theorem-level validation.Source: §H.6 · pp. 60–61, Figure 11 · p. 61

H.7 Muon–SignSGD sparse-support contrast

  • With B = 1 and rotated embeddings, the final rare-task margin median is −12.35 for SignSGD versus 2.15 for Muon.The corresponding frequent-task medians are 38.40 and 40.28, respectively.Source: §H.7 · p. 61

H.8 Language-model pretraining comparison

  • At step 30,000, fast–slow momentum’s validation loss was 2.988844 versus Muon’s 3.009736, 0.020892 lower; Figure 6 shows 51 paired checkpoints from steps 20,000–30,000.Fast–slow had lower validation loss at every paired checkpoint. Checkpoints within a trajectory are not independent training replications and do not establish variability across seeds.Source: §H.8 · p. 62

Related papers

Loading 2610.31728v1…