PaperlayerView exact revision on arXiv

Source-linked AI summary

From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

Yongtao Wu, Mahdi Mansouri, Zhenyu Zhu

arXiv:2610.20112v1cs.LGcs.AI

TL;DR

The paper asks whether practical Muon retains the spectral max-margin bias established for its exact-orthogonalization counterpart. It analyzes finite Newton–Schulz updates and stabilization, finding Frobenius max-margin selection in the fixed-iteration, positive-stabilization setting while also establishing conditions for spectral recovery.

  • Problem

    Whether practical Muon’s finite Newton–Schulz iterations and positive stabilization preserve the spectral max-margin bias of idealized Muon remains unclear.

    Source: §1 · p. 2

  • Method

    The paper analyzes practical Muon by relating its update to gradient descent through a summable perturbation argument and examining iteration count, stabilization, polynomial choice, and momentum.

    Source: §1 · p. 2

  • Results

    Fixed finite Newton–Schulz iterations with positive stabilization select the Frobenius rather than spectral max-margin direction, while logarithmically growing iterations can recover spectral max-margin under stated stabilization conditions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The implicit bias of practical Muon depends on its iteration schedule, stabilization, and polynomial, with spectral recovery established for specified schedules and stabilization schemes.

    Source: §1 · p. 2

    The quintic recovery guarantee uses gradient-dependent stabilization, so whether it holds with fixed positive ε remains open.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.

1. Introduction

Prior work links optimizer geometry to different implicit biases, with exact spectral-norm steepest descent selecting spectral max-margin solutions. This paper shows that practical Muon’s finite Newton–Schulz iterations and positive stabilization instead select Frobenius max-margin, while iteration schedules and polynomial choices determine other asymptotic behaviors and can permit spectral recovery.

1. Introduction

  • Implicit bias concerns the solution direction selected by an optimizer, and prior work shows gradient descent and different norm-based steepest descent methods can select different max-margin solutions.For separable linear classification, logistic and exponential losses can approach zero only as the parameter norm diverges, making the limiting direction relevant.Source: §1 · p. 1
  • Exact spectral-norm steepest descent, including idealized Muon, has spectral max-margin behavior, while practical Muon approximates orthogonalization with finite Newton–Schulz iterations and positive stabilization.Whether these practical modifications preserve the idealized bias motivates the paper’s central question.Source: §1 · pp. 1–2
  • Practical Muon with finite Newton–Schulz iterations and positive stabilization selects the Frobenius rather than spectral max-margin direction on separable linear classification.This result covers a general class of Newton–Schulz polynomials, including the quintic used in practical Muon, and both exponential moving average and Nesterov-style momentum.Source: §1 · p. 2
  • Fixed Newton–Schulz iterations can select directions that achieve neither Frobenius nor spectral max-margin, and the cubic and quintic polynomials have distinct limiting behaviors as iteration count tends to infinity.Removing stabilization alone does not necessarily restore spectral max-margin behavior.Source: §1 · p. 2
  • Spectral max-margin behavior can be recovered with logarithmically growing Newton–Schulz iterations during training.The cubic result allows fixed positive stabilization; the quintic guarantee uses gradient-dependent stabilization.Source: §1 · p. 2

2. Preliminary

The preliminaries define the matrix and classification notation, then distinguish the max-margin behavior of gradient descent and idealized Muon. They also specify the practical Muon update, which normalizes a momentum input and applies finite Newton–Schulz iterations.

2. Preliminary

  • The preliminary section introduces the matrix notation and separable classification problem before specifying the exact and practical Muon updates used in the paper.The idealized update uses an SVD polar factor; the practical version normalizes Nesterov-style momentum and applies K Newton–Schulz iterations.Source: §2 · p. 2, Table 1 · p. 4

2.1. Notation

  • For matrices, the preliminaries define the trace inner product and Frobenius, operator, and nuclear norms, and use compact SVD notation to define polar factors and finite polynomial matrix iterations.P_K(G) applies K compositions of an odd polynomial to the singular values in a compact SVD.Source: §2.1 · pp. 2–3

2.2. Problem setting and implicit bias

  • The setup considers matrix-valued linear predictors whose scores are inner products with feature matrices, and assumes linearly separable data with logistic or exponential loss.Signed features are A_i = y_iX_i, and the classification margins are ⟨A_i, W⟩.Source: §2.2 · p. 3
  • Gradient descent with a sufficiently small constant stepsize converges in direction to the Frobenius hard-margin solution, whereas spectral-norm steepest descent, including exact-orthogonalization Muon, has spectral max-margin behavior.Under the separability assumption, normalizing a hard-margin solution by its corresponding norm gives a max-margin solution.Source: §2.2 · p. 3

2.3. Idealized and Practical Muon

  • Idealized Muon applies the polar-factor operation to an exponential moving average, while practical Muon replaces the SVD with finite polynomial iterations.The practical update normalizes a Nesterov-style momentum input by its Frobenius norm plus ε before applying K iterations; the paper’s practical-Muon definition excludes weight decay and update scaling.Source: §2.3 · p. 4, Table 1 · p. 4
  • The polynomial choice includes the practical Muon quintic and a cubic alternative; both meet the stated positive-map assumption for every finite iteration count.For the cubic, the iterated polynomial approaches one on (0, 1] as K grows, so its matrix iteration converges to the exact polar factor.Source: §2.3 · pp. 4–5

3. Frobenius Max-Margin Bias of Practical Muon

With fixed finite Newton–Schulz iterations and positive stabilization, practical Muon converges in direction to the Frobenius max-margin classifier, matching gradient descent. The paper extends this result to EMA and Nesterov-style momentum under logistic loss and proves it by showing Muon tracks gradient descent within bounded distance.

3.1. Frobenius Max-Margin of practical Muon

  • Under suitable stepsize conditions, the momentum-free result first establishes Frobenius max-margin selection for fixed finite iterations and positive stabilization.The theorem treats arbitrary initialization and logistic or exponential loss.Source: §3.1 · p. 5
  • With fixed finite Newton–Schulz iterations and positive stabilization, practical Muon converges in direction to the Frobenius max-margin separator under the stated stepsize conditions.The result covers momentum-free Muon with logistic or exponential loss, and extends to EMA and Nesterov-style momentum with logistic loss.Source: §3.1 · p. 5

3.2. Proof Outline of Theorem 1

  • As gradients vanish, the stabilized finite-iteration update becomes a gradient-descent step plus a perturbation whose norms are summable.The proof uses nonexpansiveness to keep the Muon and gradient-descent trajectories at bounded distance, transferring gradient descent’s directional limit to Muon.Source: §3.2 · pp. 6–7

4. Implicit Bias Regimes and Spectral Recovery

The paper separates how iteration count, stabilization, and polynomial choice affect practical Muon’s implicit bias. It identifies conditions for spectral max-margin recovery, while showing that finite iterations can select neither max-margin direction and that the quintic recovery guarantee requires a special stabilization scheme.

4.1. Disentangling Finite Orthogonalization and Stabilization

  • With fixed finite K and positive stabilization, Theorem 1 gives Frobenius max-margin selection; removing stabilization alone need not recover spectral max-margin.For finite iterations, the selected direction can differ from both max-margin directions.Source: §4.1 · pp. 7–8
  • For either the cubic or quintic, finite K can produce a direction differing from both Frobenius and spectral max-margin directions.In the single-example illustration, the cubic’s diagonal-entry ratio lies between 1 and 2 for every finite K, while the quintic at K = 5 has a ratio between 5/9 and 3/4.Source: §4.1 · pp. 7–8
  • As K tends to infinity with zero stabilization, the cubic recovers spectral max-margin, whereas the quintic may fail to converge in normalized direction.For the cubic, spectral max-margin also holds as K tends to infinity with any fixed stabilization level under the stated conditions.Source: §4.1 · p. 8

4.2. Recovering Spectral Max-Margin

  • Logarithmically growing Newton–Schulz iterations recover spectral max-margin for the cubic with fixed positive stabilization, while the quintic guarantee uses gradient-dependent stabilization.For the quintic scheme, the stabilization also depends on an independent random variable, limiting direct applicability to standard Muon.Source: §4.2 · pp. 8–9

5. Experiments

The experiments examine how finite Newton–Schulz iterations and stabilization affect Muon’s margin direction, and whether growing iteration counts recover the behavior of idealized Muon. Fixed iterations with positive stabilization lead toward Frobenius max-margin, while growing iterations track spectral recovery in the multi-dataset experiments.

5.1. Finite iterations and stabilization

  • The single-example experiment tracks W_t,11/W_t,22 against reference ratios of 2 for Frobenius and 1 for spectral max-margin.It initializes W_0 = 0 and runs cubic and quintic variants for 5 × 10^5 updates at step size η = 0.05.Source: §5.1 · p. 10, Figure 1 · p. 10
  • For both cubic and quintic polynomials, positive ε makes predictor ratios approach the Frobenius max-margin target, and larger ε brings an earlier transition.Column (a) fixes K = 5 while varying positive ε.Source: §5.1 · p. 10
  • With ε = 0, predictor ratios remain constant for each fixed K; increasing K brings the cubic toward the spectral target, while the quintic behaves nonmonotonically.These outcomes are reported for column (b) and are consistent with Theorem 2(i)–(ii).Source: §5.1 · p. 10
  • With ε = 10^-3, both polynomials approach Frobenius max-margin as training progresses, and K changes the transient path but not the asymptotic bias.Column (c) varies K; the text identifies its effect as transient rather than asymptotic.Source: §5.1 · pp. 10–11

5.2. Frobenius selection and spectral recovery

  • With fixed K, the cubic variant favors Frobenius max-margin, whereas growing K_t closely tracks idealized Muon in spectral margin and oracle accuracy.The figure’s results average metrics across five synthetic datasets; the fixed-K variant has lower spectral margin and larger oracle error.Source: Figure 2 · p. 11, §5.2 · p. 11

6. Related Work

Prior work connects optimization geometry to max-margin bias and studies spectral descent and Muon in several settings. This paper’s distinction from nearby Muon analyses is its focus on how practical Newton–Schulz choices shape asymptotic max-margin behavior.

6.1. Implicit bias of optimization

  • For separable linear problems, gradient descent and stochastic gradient descent select Euclidean max-margin under suitable loss and step-size conditions (Soudry et al., 2018; Nacson et al., 2019b).Related work also connects this behavior to loss tails, regularization paths, and primal-dual geometry.Source: §6.1 · p. 11
  • Steepest descent, mirror descent, and mirror flow exhibit max-margin bias governed by optimization geometry (Gunasekar et al., 2018; Sun et al., 2023; Pesme et al., 2024).Related work also studies directional convergence and margin-stationarity for homogeneous neural networks.Source: §6.1 · p. 12
  • For matrix-valued parameters, Fan et al. (2025) establish spectral max-margin behavior for spectral descent and exact-orthogonalization Muon on multiclass separable data.Other cited work examines momentum, adaptive updates, stabilization, and annealed stabilization.Source: §6.1 · p. 12

6.2. Steepest descent

  • Unlike cited convergence analyses of practical Muon, this work studies how iteration count, stabilization, and polynomial choice determine asymptotic max-margin behavior.Prior work also considers spectral descent, Muon applications and scaling, and polynomial optimization for polar approximation.Source: §6.2 · p. 12

7. Conclusion

The paper concludes that finite Newton–Schulz iterations with positive stabilization yield Frobenius max-margin selection in separable linear classification, while characterizing when spectral behavior can be recovered. Its analysis is limited to full-batch linear classification; stochastic optimization and nonlinear networks remain outside its scope.

7. Conclusion

  • Finite Newton–Schulz iterations with positive stabilization select Frobenius max-margin in separable linear classification, with conditions also given for recovering spectral max-margin.The analysis characterizes the effects of stabilization, iteration count, and polynomial choice.Source: §7 · p. 12
  • The stated scope is full-batch linear classification; stochastic optimization and nonlinear neural networks are left for future analysis.The conclusion proposes investigating whether the asymptotic connection to gradient descent appears in later stages of large-scale neural-network training.Source: §7 · p. 12

Contents of the Appendix

  • The appendix maps proofs and auxiliary lemmas to Sections 3 and 4, with supplementary experimental details and results in Section E.Section B contains proofs for Section 3, while Sections C and D cover Theorem 2 and Theorems 3–4, respectively.Source: Contents of the Appendix · p. 16, Appendix B · p. 17

Appendix A. Notation

  • Appendix A records core notation, including the momentum coefficient, input weight, hard-margin solutions, and Frobenius and spectral max-margins.It also defines the dimension bound r = min{d1, d2}.Source: Table 3 · p. 16, Appendix A · p. 16

B.1. Auxiliary lemmas; B.3. Proof of Lemma 2

  • An auxiliary exponential-loss result gives gradient descent iterates V_t = cW_F log t + O(log log t), and a later comparison states W_t = V_t + O(1).Together, these proof excerpts report the Frobenius-direction asymptotic for the compared trajectory.Source: §B.1 · p. 19, §B.3 · p. 20

B.4. Proof of Theorem 1(i)

The proof compares practical Muon iterates with gradient descent to establish Frobenius max-margin asymptotics, first for logistic loss and then for exponential loss. It extends the comparison to momentum and includes a separate appendix section for proving the three parts of Theorem 2.

B.4. Proof of Theorem 1(i)

  • For logistic loss, the proof compares Muon with a gradient-descent reference trajectory and obtains the Frobenius hard-margin asymptotic W_t = cW_F log t + O(log log t).The reference trajectory has the same expansion, and the proof plan identifies loss convergence and trajectory comparison as steps toward the limiting direction.Source: §B.4 · pp. 21–22

B.5. Proof of Theorem 1(ii)

  • For exponential loss, the proof establishes W_t = V_t + O(1) = cW_F log t + O(log log t).The comparison is made on a small-loss sublevel set.Source: §B.5 · pp. 22–23

B.6. Proof of Corollary 1

  • The momentum extension establishes vanishing gradients and reports W_t = V_t + O(1) = cW_F log t + O(log log t).Its proof first bounds the momentum error and then applies the gradient-descent comparison.Source: §B.6 · pp. 23–25

Appendix C. Proof of Section 4.1

  • Appendix C begins the proof of the three parts of Theorem 2 using r = min{d1, d2} and R* = max_i ||A_i||*.Source: Appendix C · p. 26

C.1. Proof of Theorem 2(i)

The proof constructs examples where finite Newton–Schulz iterations select directions that achieve neither Frobenius nor spectral max-margin. It establishes this behavior for cubic and quintic polynomials, with a special case when the two target directions coincide.

C.1. Proof of Theorem 2(i)

  • With at least two distinct positive singular values, the cubic polynomial produces unequal singular values not proportional to those of A, so neither margin is achieved.The argument uses the update direction obtained from the iterated polynomial applied to the normalized matrix.Source: §C.1 · p. 26
  • When all positive singular values of A are equal, the Frobenius and compact-polar directions coincide, and every admissible finite polynomial selects that common direction.For the counterexample initialization W0 = 0, the normalized iterate follows the normalized update direction at every t ≥ 1.Source: §C.1 · p. 26
  • For the quintic polynomial, a sufficiently small positive second singular value likewise yields an update direction achieving neither max-margin.The proof shows the resulting matrix is not proportional to A and has two unequal positive singular values.Source: §C.1 · p. 27

C.2. Proof of Theorem 2(ii)

The proof handles the cubic and quintic counterexamples and the associated auxiliary bounds, then uses an inexact spectral-margin argument to establish recovery guarantees. Its estimates depend on the polynomial, iteration schedule, and stabilizer.

C.2. Proof of Theorem 2(ii)

  • For the cubic with ε = 0, both logistic and exponential losses yield the asserted result for every finite initialization.The proof uses a positive alignment bound and the fact that cumulative stepsizes grow at least logarithmically.Source: §C.2 · pp. 28–29
  • For the quintic with A = diag(2, 1), W0 = 0, and ε = 0, normalized update directions alternate between disjoint ratio intervals for odd and even K ≥ 3.Thus the normalized matrices do not converge as K grows.Source: §C.2 · p. 29, §C.4 · p. 31

C.3. Proof of Theorem 2(iii)

  • For fixed ε > 0, the proof takes the iteration-count limit at each fixed horizon and then lets the horizon grow, using cumulative stepsizes of at least logarithmic order.This establishes the conclusion identified as Equation (14) in the proof.Source: §C.3 · p. 29

C.4. Auxiliary lemmas

  • For A = diag(2, 1), with DK = PK(A/∥A∥F) for the quintic in Equation (7), DK/∥DK∥F does not converge as K →∞.Source: §C.4 · p. 31

Appendix D. Proof of Section 4.2; D.1. Proof of Theorem 3

  • With fixed positive stabilization, the cubic recovery proof bounds the relative oracle error by a summable sequence under logarithmically growing cumulative stepsizes.The proof then applies an inexact spectral-margin bound to obtain its stated margin guarantee.Source: §C.4 · p. 30, §D.1 · p. 32

D.2. Proof of Theorem 4

The proof analyzes quintic updates with a randomized schedule, deriving loss and margin bounds from conditional estimates. It also describes synthetic datasets constructed around known Frobenius hard-margin solutions.

D.3. Auxiliary lemmas

  • The auxiliary argument uses conditional expectations of randomized quintic updates and bounded-variable moment estimates to control the margin analysis.The updates preserve singular vectors, and the proof bounds their operator norm by 5/4.Source: §D.2 · p. 33, §D.3 · pp. 34–36

D.2. Proof of Theorem 4

  • The randomized quintic proof establishes loss convergence and an eventually positive minimum margin, then uses its accumulated bounds to prove Equation (17).The iteration schedule and scalar mean bounds control the proof’s accumulated terms.Source: §D.2 · pp. 33–34, §D.3 · p. 35

E.1. Synthetic Dataset Construction

  • The five synthetic datasets are built around known Frobenius hard-margin solutions, which the construction makes unique; spectral solutions are computed with CVXPY and Clarabel.The samples are generated using orthogonal projections and margins specified relative to the chosen Frobenius solution.Source: §E.1 · p. 36

E.2. Additional experiment on quintic and momentum

The experiment compares cubic, quintic, and Nesterov-momentum quintic Muon across five datasets and seeds. Update directions approach negative-gradient directions faster than predictors approach the Frobenius max-margin direction; momentum changes transients, while the two quintic predictor trajectories are similar.

E.2. Additional experiment on quintic and momentum

  • Update directions approach the negative-gradient direction faster than predictors approach the Frobenius max-margin direction.Figure 3 reports the direction distances for cubic, quintic, and Nesterov-style quintic Muon; results average five random seeds.Source: §E.2 · pp. 36–37, Figure 3 · p. 37
  • Momentum affects transient behavior, while the two quintic variants exhibit similar predictor trajectories.The comparison uses five datasets and averages results over five random seeds.Source: §E.2 · pp. 36–37
Loading 2610.20112v1…