Beyond the Matrix Sign: Quadratic Spectral Descent

1 Introduction

• QSD directly optimizes a quadratic local loss model under Muon’s spectral-norm constraint, without the assumptions used by some prior curvature-aware variants.It uses curvature in the objective rather than only changing the update geometry.Source: §1 · p. 2
• QSD uses inversion-free K-FAC curvature and online Frank-Wolfe updates whose linear subproblems have matrix-sign solutions, keeping the complexity increase mild.Warm starts and closed-form Frank-Wolfe steps make three inner steps per outer iteration sufficient.Source: §1 · pp. 2–3
• Under aligned gradient and curvature singular directions, QSD’s spectral amplitudes should follow the gradient-to-curvature ratio rather than Muon’s uniform allocation.When gradient and curvature directions are not aligned, the preferred singular directions also rotate relative to Muon’s.Source: §1 · p. 2
• Even with isotropic curvature, QSD remains non-flat, whereas Newton-Muon, Mousse, and FISMO reduce to Muon.Newton-Muon relies on a weight-displacement isotropy assumption, while Mousse and FISMO use second-order information implicitly by changing update geometry.Source: §1 · p. 2
• QSD consistently improves validation loss over Muon and recent Muon variants, with up to 8.49% wall-clock training-time reduction at matched validation loss.The reported gains are on GPT pretraining.Source: §1 · p. 3

1.1 Related Work

• QSD extends spectral-norm matrix optimization from a linear objective to a quadratic one that jointly accounts for singular values and directions.Related curvature-aware methods either transform the gradient or use an anisotropic spectral geometry, whereas QSD retains the spectral-norm constraint and places curvature in the objective.Source: §1.1 · p. 3

2 Preliminaries and Problem Formulation

• A matrix update direction D produces the parameter update W + ηD, with its size measured using the spectral norm.The paper also defines the nuclear and Frobenius norms and the Frobenius inner product.Source: §2 · p. 3
• For finite updates, directions with similar first-order descent can incur substantially different second-order costs.The quadratic model uses a positive-semidefinite curvature operator in place of the costly Hessian action.Source: §2 · pp. 3–4
• QSD optimizes the quadratic local model over the same spectral-norm constraint as Muon, and reduces exactly to the Muon problem when curvature is zero.The formulation removes the constant term and divides by the learning rate.Source: §2 · p. 4

3.1 Kronecker-Factored Curvature

• K-FAC supplies a structured positive-semidefinite curvature proxy, and the method optimizes the resulting curvature-aware objective directly over the spectral-norm ball.The empirical-Fisher factors are not assumed to be an exact Hessian or generalized Gauss–Newton approximation.Source: §3.1 · p. 4

3.2 Frank–Wolfe Solver

• Frank-Wolfe approximately solves the quadratic problem by repeatedly linearizing it; each linear subproblem has a closed-form matrix-sign solution.The first atom from zero initialization is the Muon direction, and later curvature-corrected atoms refine that update.Source: §3.2 · pp. 4–5

3.3 Practical QSD

• Practical QSD periodically estimates and reuses K-FAC factors, calibrates their scale against GGN directional curvature, and applies curvature inflation with τ = 1.5.Momentum-smoothed gradients and warm starts from the preceding solution allow a small number of inner steps.Source: §3.3 · p. 5

3.4 Optimization guarantees.

• The Frank-Wolfe gap gives a computable suboptimality certificate, exact line search yields monotone surrogate descent, and the inner solver has an O(1/K) convergence rate.With zero initialization, the first step is no worse than the Muon update under the same quadratic surrogate.Source: §3.4 · p. 6

3.5 Computational Complexity

• QSD incurs 6% wall-clock overhead over Muon in the reported 124M-model experiments.Structured curvature, infrequent factor updates, and warm-started inner solves help keep the overhead modest.Source: §3.5 · p. 6
• For approximately square matrices, each of K inner steps adds optimizer cost O(Kd^3), while the main forward and backward computation scales as O(Nd^2).Warm starts allow only a few online Frank-Wolfe steps to suffice in practice.Source: §3.5 · p. 6

4 Understanding How Curvature Changes Muon Updates

• Curvature changes both the singular amplitudes and singular directions that Muon’s linear optimization keeps flat and gradient-aligned.The spectral-norm constraint controls update size but does not force the optimal update to preserve the gradient’s singular directions.Source: §4 · p. 7

4.1 Curvature Induces Nonuniform Singular Amplitudes

• When gradient and curvature share singular modes, curvature makes the optimal singular-value profile generally nonuniform rather than Muon’s uniform profile.Mode i saturates at ρ only when gi ≥ ηρκi; otherwise its optimal singular value is gi/(ηκi) < ρ.Source: §4.1 · p. 7

4.2 Curvature Can Change the Singular Directions

• Under anisotropic curvature, the optimal update can change singular directions even when the spectral-norm constraint is active.Different curvature components are rescaled unequally, while λ adjusts overall shrinkage to the constraint boundary without undoing that reweighting.Source: §4.2 · p. 8

4.3 QSD vs. Muon in Practice

• QSD differs from Muon in both singular amplitudes and directions, and both deviations remain nonzero across training layers.Its spectrum is nonuniform, and its update has substantial off-diagonal mass in the momentum matrix’s singular basis.Source: §4.3 · p. 8

5.1 Pre-training

• QSD differs from Muon in singular amplitudes and directions, with nonuniform spectra and changes in singular directions observed across layers.Figure 1 shows normalized spectra, momentum-basis update magnitudes, and layerwise deviations.Source: Figure 1 · p. 9
• With Newton-Muon [5], vocabulary size 50257, and batch size 512, QSD outperforms Muon at 6% time overhead in experiments training on RTX 5000 Ada GPUs with bfloat16 mixed precision.The AdamW learning rate was swept over [0.001, 0.0018, 0.0032, 0.0056, 0.01] and the remaining part over [0.005, 0.01, 0.02, 0.04]; the best results are reported in Table 3.Source: §5.1 · p. 9
• QSD retains performance gains when pretraining GPT-350M, using hyperparameters tuned on GPT-124M with selected adjustments for the longer schedule.The adjusted settings include learning rate and K-FAC refresh and calibration intervals.Source: §5.1 · p. 9, Table 4 · p. 9

5.2 Training Efficiency

• QSD speeds up pretraining by up to 8.49% when measured at matched validation loss in the evaluated settings.The target is QSD’s final validation loss; the comparison reports tokens and wall-clock time required to reach it.Source: §5.2 · p. 9, Table 5 · p. 10

5.3 Comparison with Muon Variants

• QSD consistently outperforms NorMuon and Newton-Muon across all three GPT-124M runs.Table 6 reports validation performance.Source: §5.3 · p. 10, Table 6 · p. 10

6 Conclusion

• QSD uses inversion-free Kronecker-factored curvature and online Frank–Wolfe updates, and consistently improves validation loss over Muon and recent Muon variants in GPT pretraining.At matched validation loss, the experiments report up to an 8.49% wall-clock speedup.Source: §6 · p. 10
• Under isotropic curvature, QSD’s spectral allocation remains nonuniform and proportional to the gradient-to-curvature ratio, while several curvature-aware Muon variants reduce to Muon.When gradient and curvature singular directions are not aligned, the optimal update’s singular directions also change.Source: §6 · p. 10

B.1 Optimality Certificate

• The Frank–Wolfe gap is a computable certificate of surrogate suboptimality, and it is zero if and only if the current iterate is optimal.A small gap indicates that no feasible direction can substantially improve the current surrogate value.Source: B Theoretical Guarantees · p. 13, §B.1 · p. 13

B Theoretical Guarantees

• With zero initialization and exact line search, the first Frank–Wolfe atom is the Muon update, and subsequent steps preserve surrogate-value dominance over Muon.This comparison concerns the quadratic surrogate, not the exact training loss.Source: §B.1 · p. 13

B.2 Convergence of the QSD Inner Solver

• With exact line search, the inner solver converges to the optimal surrogate value at rate O(1/K), from any feasible initialization.When damping d > 0, the surrogate is strongly convex and the iterates converge to the unique inner optimum.Source: §B.2 · pp. 13–14

C.1 Proof of Proposition 4.1

• For aligned gradient and curvature singular modes, the allocation problem decouples by mode; clipping the mode-wise solution to its feasible interval yields an optimal solution in the gradient’s singular subspace.If a mode’s curvature is zero and its gradient is positive, its optimal allocation is the spectral-norm radius.Source: §C.1 · pp. 15–16

C.2 Proof of Proposition 4.2

• For a rank-one gradient, the optimal update’s right singular direction is proportional to (ηA + λI)^−1v; it differs from the gradient direction when v is not an eigenvector of A.The direction change occurs with the spectral-norm constraint active.Source: §C.2 · p. 17

C.3 Proof of Proposition B.1

• The Frank–Wolfe gap is nonnegative, and a zero gap implies that the current iterate is optimal.The proof also shows the converse: an optimal iterate has zero gap.Source: §C.3 · p. 18

C.4 Proof of Theorem B.2

• The practical quadratic surrogate has a Lipschitz gradient in the Frobenius norm, and exact line search performs at least as well as any prescribed step size.These properties are used in the inner-solver convergence analysis.Source: §C.4 · pp. 18–19

D Iterate Convergence under Damping

• When damping d > 0, the surrogate is strongly convex with respect to the Frobenius norm.This supports convergence of the iterate to the unique constrained inner optimum.Source: D Iterate Convergence under Damping · p. 20

E Monotonic surrogate descent

• Exact line search cannot increase the surrogate, and it strictly decreases the surrogate whenever the Frank–Wolfe gap is positive.The zero step is feasible, while a positive gap supplies a strictly descending direction.Source: E Monotonic surrogate descent · p. 21

E.1 Relation to the Muon Update

• With zero initialization and exact line search, QSD’s first Frank–Wolfe atom is the Muon update, and subsequent refinements preserve surrogate dominance over Muon.The comparison is about the quadratic surrogate rather than exact training loss.Source: §E.1 · pp. 21–22

F Effect of Curvature Inflation

• For aligned gradient and curvature modes, the optimal amplitude depends on the gradient-to-curvature ratio rather than being uniform as in Muon.The derivation assumes shared singular directions and imposes the spectral-norm boundary on each mode’s amplitude.Source: F Effect of Curvature Inflation · p. 23
• Inflation leaves boundary-saturated modes unchanged but attenuates curvature-limited modes, with higher-curvature modes crossing into that regime earlier as τ increases.Once curvature-limited, a mode’s amplitude decreases as 1/τ.Source: F Effect of Curvature Inflation · pp. 23–24

G.1 Isolating the Effects of QSD

• Direction adaptation improves validation loss by 0.0048, and adaptive spectral shaping adds 0.0105, for a total gain of 0.0153 over Muon.The comparisons are controlled, not strictly additive, because directions and singular values are jointly optimized.Source: §G.1 · p. 25, Table 7 · p. 25

G.2 Ablation for the baseline hyperparameters

• About three warm-start Frank-Wolfe steps improve validation loss before gains become marginal, while zero start underperforms; K-FAC damping of 10^-6 performs best.Both larger and smaller damping slightly degrade validation loss.Source: §G.2 · pp. 25–26, Table 8 · p. 25, Table 9 · p. 25
• Performance is broadly stable across factor-refresh and calibration settings, but poorly smoothed, overly frequent or infrequent updates can degrade it.Using 1% of data for K-FAC factors achieves the best validation loss with little benefit from larger ratios; 1% for scale calibration matches or outperforms substantially larger ratios. Inflation helps up to τ=1.5, while excessive inflation can over-regularize.Source: §G.2 · pp. 25–26, Figure 2 · p. 25, Table 10 · p. 26

H.1 Token Sampling

• K-FAC factors use paired activation and output-gradient statistics from the same uniformly sampled token positions, with the default 1% ratio pooling estimates across devices.For each optimizer step, approximately 1% of 524288 tokens—5248 positions—is sampled.Source: §H.1 · p. 27, H K-FAC Factor Estimation Details · p. 27

H.2 Output-Gradient Rescaling

• Output gradients are rescaled to restore their per-token scale before factor estimation, and K-FAC subsampling changes the estimator’s sample count, not that underlying scale.The token count used in the rescaling is the count before K-FAC subsampling.Source: §H.2 · p. 27

H.3 Factor Refresh and EMA

• K-FAC factors refresh every 32 optimizer steps using EMA with βF=0.9, while sampled operands use bfloat16 and factor statistics and EMA states are stored in FP32.The factors are initialized from the first estimates, reused between refreshes, and do not use bias correction.Source: §H.3 · pp. 27–28, §H.4 · p. 28

I.1 Directional Curvature Matching

• A per-layer coefficient calibrates K-FAC scale by matching K-FAC and GGN directional curvatures along a recent update direction, without changing the Kronecker structure.Damping and curvature inflation are applied separately from this calibration.Source: §I.1 · p. 29, I Directional Curvature Calibration · p. 29

I.2 Finite-Difference GGN Estimate

• The GGN directional-curvature estimate approximates the needed Jacobian–vector product with a forward finite difference and uses forward evaluations plus the closed-form logit-space Hessian; it requires no explicit Jacobian or GGN matrix, or additional backward pass.Source: §I.2 · p. 30

I.3 Implementation Details

• Calibration data are sampled by sequence, nominally at 1% of the minibatch, with a minimum block size of four sequences.For a minibatch of 512, the nominal setting selects eight sequences, or 1.56% of the minibatch.Source: §I.3 · p. 30
• The finite-difference computation uses FP32 with TF32 disabled because subtracting nearby logit vectors is sensitive to reduced mantissa precision.The perturbation is normalized by the Frobenius norm of the calibration direction.Source: §I.3 · p. 30

J Experimental Details

• QSD-specific hyperparameters tuned on GPT-124M are kept unchanged for GPT-350M unless otherwise stated.The training and QSD-specific settings used in the main experiments are summarized in Table 14.Source: J Experimental Details · p. 31, Table 14 · p. 31

• The implementation’s Kronecker-factored empirical-Fisher statistics with GGN-based scale calibration may still miss aspects of the true curvature.Source: K Limitations · p. 32
• The theoretical guarantees apply to the quadratic surrogate rather than directly to the exact training loss, since higher-order effects are not modeled.Source: K Limitations · p. 32
• Experiments cover GPT pre-training at 124M and 350M scales; larger models, other architectures, and post-training settings remain future work.Source: K Limitations · p. 32