PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Muon Need Fine-Grained Spectral Shaping?

Meher Chaitanya, Tianyi Zhou, Aristides Gionis

arXiv:2610.07497v1cs.AIcs.LGstat.ML

TL;DR

Muon’s flat polar update gives equal gain to every singular direction, raising the question of whether fine-grained spectral shaping is needed. The paper introduces two-band BulkBoost and evaluates it across continued-pretraining settings. The two-band variants are competitive with fine-grained profiles, with reported average loss reductions over Muon’s flat profile of 0.073–0.147%.

  • Problem

    The paper asks whether Muon needs fine-grained, per-direction spectral shaping or whether a coarser reallocation between two groups of directions is enough.

    Source: §1 · p. 1

  • Method

    BulkBoost reweights Muon’s update with one shared bulk-to-spike gain, using either a fixed-rank split or a noise-calibrated split.

    Source: §1 · p. 2

  • Results

    Across 30 settings, BB-STATIC(5%), BB-ADAPTIVE, and BB-STATIC† reduce final loss relative to Muon’s flat profile by 0.073%, 0.077%, and 0.147% of ℓ0 on average, respectively; FREON reduces it by 0.022%.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    Across the tested continued-pretraining settings, a single shared bulk-to-spike gain captures at least as much benefit as the compared fine-grained spectral profiles.

    Source: §1 · p. 2, §5.1 · pp. 8–9

    The evaluation is restricted to Pythia continued pretraining under the reported tuning protocol.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Muon combines current and past gradients into matrix momentum. For $M=U\Sigma V^\top$, the idealized polar update $Q=UV^\top$ gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately $94$--$97\%$ of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by $0.022\%$ of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of $0.073$--$0.147\%$. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.

1 INTRODUCTION

The paper asks whether Muon needs per-direction spectral gains or whether a coarse two-band reallocation is enough. It introduces BulkBoost and evaluates its theory and performance against fine-grained profiles.

1 INTRODUCTION

  • Approximately 93.7–97.1% of pooled singular modes lie below the noise edge, and their aggregate signed alignment with a reference gradient is positive in every analyzed matrix.The modes are individually weak, but their aggregate contribution motivates the two-band split.Source: §1 · p. 2
  • BulkBoost increases the bulk’s relative weight through one shared gain, using either a fixed fraction of leading directions or a noise-calibrated edge.An exact polar-plus-low-rank identity enables an SVD-free realization.Source: §1 · p. 2
  • For a fixed partition, the theory characterizes when shifting weight toward the bulk lowers loss and how much of the best per-direction improvement two bands can capture.The contribution concerns first-order improvement.Source: §1 · p. 2
  • Giving the bulk more relative weight lowers loss after a single update in most tested states.This is reported as an empirical diagnostic result.Source: §1 · p. 2
  • Across 30 continued-pretraining settings, two-band reweighting gives slightly larger average improvement over Muon’s flat profile than the compared fine-grained profiles.The settings span Pythia-14M to 410M and six corpora.Source: §1 · p. 2

2 SETUP AND SPECTRAL REWEIGHTING

The setup defines Muon’s matrix direction and distinguishes flat, fine-grained, and two-band spectral profiles. BulkBoost’s split is based either on a fixed rank fraction or an estimated noise edge.

2 SETUP AND SPECTRAL REWEIGHTING

  • Muon forms a Nesterov spectral input from an exponential moving-average buffer and the current minibatch gradient, then uses its polar factor as the idealized direction.The polar direction gives unit gain to every singular direction; practical Muon approximates it with Newton–Schulz iterations.Source: §2 · p. 2
  • Both partition rules place leading singular directions in the spike band; BB-STATIC selects a fixed fraction, while BB-ADAPTIVE thresholds singular values.The adaptive threshold uses a noise edge calibrated for the Nesterov input; isotropic Gaussian noise is an approximate reference because real gradient noise is not isotropic.Source: §2 · p. 3
  • Fine-grained profiles assign gains from individual singular values, whereas BulkBoost uses one shared bulk-to-spike gain and norm-matches the resulting direction.At gain ratio ρ = 1, the profile is flat; ρ > 1 gives the bulk greater relative weight.Source: §2 · p. 3

3 NOISE-EDGE REWEIGHTED MUON: BULKBOOST

BulkBoost’s adaptive variant estimates a noise edge for Muon’s Nesterov input from split-minibatch gradients. Its two-band direction is realized through a polar factor and a low-rank spike correction.

3.1 NOISE-CALIBRATED PARTITIONING

  • BB-ADAPTIVE estimates sampling noise from two independent minibatch halves and propagates its variance through the momentum to set a noise-edge threshold for the Nesterov input.The half-batch difference has zero mean and the same covariance as raw full-minibatch sampling noise; the edge assumes centered, temporally independent, isotropic input noise.Source: §3.1 · pp. 3–4

3.2 STRUCTURE AND SVD-FREE REALIZATION

  • The exact two-band update consists of the flat polar direction plus a correction supported on the spike subspace, avoiding the need for bulk singular vectors.The correction has rank at most the number of selected spike directions.Source: §3.2 · p. 4
  • The numerical realization uses a polar approximation and spike-subspace tracking; finite iterations, trial-space size, or a binding rank cap can make its retained rank differ from the exact rank.Norm matching preserves the approximated direction’s Frobenius norm but does not correct polar or subspace approximation errors.Source: §3.2 · p. 4

4 THEORY OF TWO-BAND REWEIGHTING

The theory gives a first-order test for reallocating weight between two fixed bands and compares that family with all norm-matched per-direction reallocations. It also expresses the two-band update as a polar direction plus a low-rank correction.

4.1 WHAT TWO BANDS CAN CAPTURE

  • For a fixed partition, the first-order test says that increasing the bulk-to-spike gain from 1 improves loss strictly if and only if the bulk’s mean continuation utility exceeds the spike band’s.If the utility gap is zero, the first-order test is inconclusive; the condition concerns sufficiently small gain increases, not a particular finite gain.Source: §4.1 · p. 5
  • Two bands capture a fraction of the maximal first-order improvement rate determined by band sizes and the gap between their mean utilities.The uncaptured share comes from utility differences within bands, and the split rule determines only which directions enter each band.Source: §4.1 · p. 5

4.2 EXACT POLAR-PLUS-LOW-RANK STRUCTURE

  • With a common gain on the non-spike directions, the unnormalized update is the polar factor plus a correction confined to the selected spike subspace.The correction is low rank, and the normalized projector form does not require bulk singular vectors.Source: §4.2 · p. 6

5 EXPERIMENTS

Across 30 continued-pretraining settings, two-band BULKBOOST variants improve on MUON’s flat profile and are competitive with fine-grained spectral baselines. The experiments compare fixed-rank and noise-calibrated splits under a shared training and evaluation protocol, while exposing trade-offs in tuning and per-step cost.

5 EXPERIMENTS

  • Experiments continue pretraining five Pythia scales from 14M to 410M on six corpora, using about 100M tokens, three seeds, and shared held-out evaluation windows.A shared learning rate is chosen per scale and reused across methods and corpora.Source: §5 · p. 7

5.1 COMPARISON OF ALL OPTIMIZATION METHODS

  • BB-ADAPTIVE matches BB-STATIC(5%) without fraction search, with 10 wins, 6 losses, and 14 ties; per-setting tuned BB-STATIC† beats BB-ADAPTIVE in 20 settings.BB-STATIC† selects the fraction using the reported loss, while BB-ADAPTIVE chooses its split from the noise edge.Source: §5 · p. 7, §5.1 · p. 8
  • Against MUON, BB-STATIC(5%), BB-ADAPTIVE, and BB-STATIC† lower final loss by 0.073%, 0.077%, and 0.147% of ℓ0 on average, versus FREON’s 0.022%.The comparisons cover 30 model–corpus settings after about 100M tokens; BULKBOOST beats SPECTRA and DEVA in all 30.Source: §5.1 · p. 8, Table 2 · p. 9
  • Among settings where they beat MUON, BB-ADAPTIVE saves 7.99% of tokens on average but runs at 1.36–1.50× MUON’s per-step runtime; measured static configurations run at 1.19–1.27×.Per-setting tuned BB-STATIC† raises average token savings to 10%.Source: §5.1 · p. 9
  • In exact-SVD single-update tests, bulk emphasis with ρ = 1.25 or 1.5 lowers loss versus the flat update in five of six Pythia-70M settings, and a third band does not improve on two.These tests are ablations, distinct from the continued-pretraining benchmark.Source: §5.1 · p. 9
  • The evaluation is limited to Pythia continued pretraining under the reported tuning protocol, and the white-Gaussian noise edge separates signal from noise only approximately.A shared static fraction avoids per-setting search, but the best fraction varies across settings; BB-ADAPTIVE avoids that search at higher per-step cost.Source: §5.1 · p. 9

AI USE STATEMENT

The section situates BULKBOOST among matrix-aware optimization and spectral-shaping methods, explaining its two-band allocation and noise-calibrated split alongside related computational and analytical work. It also records how generative AI tools were used in manuscript development and how the authors reviewed the final work.

AI USE STATEMENT

  • Generative AI assisted with experimental-code drafting and debugging, results interpretation and presentation, and manuscript drafting and editing; authors reviewed and verified final implementations, protocols, and results.The authors state that experiments were executed by them and that they take responsibility for correctness and originality.Source: AI USE STATEMENT · p. 10

A RELATED WORK

  • BULKBOOST allocates a fixed per-matrix direction norm between two spectral bands, separating partition choice from the relative gain assigned to the bulk.Partitions are selected either by fixed rank or by a noise-calibrated edge.Source: A RELATED WORK · p. 13

A.1 MATRIX-AWARE OPTIMIZATION AND SPECTRAL GEOMETRY

  • Unlike matrix preconditioners that adapt direction-specific statistics, BULKBOOST uses a common gain within each band and preserves the Frobenius norm of the polar direction.Its noise-calibrated split selects band membership, while its fixed-rank variant requires no noise estimate.Source: §A.1 · p. 13

A.2 SPECTRAL SHAPING BEYOND MUON’S FLAT PROFILE

  • Prior spectral-shaping methods use nonconstant spectral profiles or clipping, whereas BULKBOOST tests whether a norm-matched two-band profile suffices.The compared alternatives include Freon, MUONp, DynMuon, HTMuon, Softsign, SPECTRA, and MUSEC.Source: §A.2 · p. 13

A.3 HEAD–BULK ALLOCATION AND THE CLOSEST RELATED METHODS

  • Spectra retains unequal raw tail magnitudes, while BULKBOOST’s exact target assigns equal gains within each band; the two methods also differ in partition rule and normalization.The passage identifies the Spectra optimizer of Huang et al. (2026), distinct from the SPECTRA framework of Jiang et al. (2026).Source: §A.3 · pp. 13–14

A.4 RANDOM-MATRIX NOISE EDGES AND SPECTRAL ESTIMATION

  • The noise-calibrated split estimates variance from split-minibatch differences and uses a reference-model Nesterov correction to calibrate an edge for momentum.The edge selects band membership; it is neither an exact signal–noise separator nor a rule to discard below-edge modes.Source: §A.4 · p. 14

A.5 POLAR COMPUTATION, SUBSPACE TRACKING, AND LOW-RANK UPDATES

  • BULKBOOST approximates the polar factor and selected spike subspace without computing individual bulk singular vectors, but avoiding an explicit SVD alone does not establish a runtime advantage.Tracking cost depends on trial-space width and iteration count as well as retained rank.Source: §A.5 · p. 14

A.6 CURVATURE, NOISE GEOMETRY, AND CONTINUATION UTILITY

  • The continuation utility is the negative derivative of later evaluation loss with respect to an applied singular-direction gain; its analysis is not an online procedure for selecting ρ.The theory characterizes local allocation preferences and the share of maximal first-order improvement accessible to two bands under norm matching.Source: §A.6 · pp. 14–15

A.7 CONTROLLED CONTINUED-PRETRAINING EVALUATION

The controlled evaluation compares fixed-rank and noise-calibrated reweighting from common pretrained checkpoints under matched token budgets and a shared validation protocol. Supplementary sections define the spectral notation, explain relative gains and normalization, and specify rank-degenerate cases.

A.7 CONTROLLED CONTINUED-PRETRAINING EVALUATION

  • The continued-pretraining comparison uses common pretrained checkpoints, matched training-token budgets, and a common validation protocol to assess allocation, partition selection, validation loss, token efficiency, and per-step computation.The evaluation distinguishes these measures from fraction-selection effort.Source: §A.7 · p. 15

B SUPPLEMENTARY SETUP AND NORMALIZATION; B.1 EXTENDED NOTATION AND SPECTRAL PROFILES

  • The supplementary setup establishes thin-SVD notation and the noise-scale convention; BB-ADAPTIVE estimates raw gradient variance from minibatch halves, applies the squared clipping factor, and calibrates the momentum edge.The white-noise reference is used to select band membership.Source: B SUPPLEMENTARY SETUP AND NORMALIZATION · p. 16, §B.1 · p. 16

B.2 RELATIVE AND EFFECTIVE GAINS

  • The gain ρ specifies the bulk-to-spike gain ratio, not the absolute bulk gain; with spike fraction 0.05 and ρ = 4, the gains are approximately 1.02 and 0.26.Relative bulk emphasis and relative spike suppression describe the same norm-matched reallocation.Source: §B.2 · pp. 16–17
  • When both bands are nonempty, the effective bulk-to-spike gain equals ρ; it summarizes relative allocation but not within-band variation or subspace-recovery accuracy.The common norm-matching factor cancels in this ratio.Source: §B.2 · p. 17

B.3 RANK CONVENTIONS AND DEGENERATE PARTITIONS

  • For rank-deficient inputs, the construction partitions only active singular modes, while zero modes receive no gains; empty-band cases can reduce the normalized target to the flat polar direction.The fixed-rank fraction still refers to d = min{m, n}, even when the active rank is smaller.Source: §B.3 · p. 17

B.4 DERIVATION OF NUMERICAL NORM MATCHING

The numerical construction forms a reweighted direction from a tracked spike subspace and its orthogonal complement, then rescales it to match the norm of the approximate polar direction. The exact-target identities and numerical norm guarantee rely on distinct conditions.

B.4 DERIVATION OF NUMERICAL NORM MATCHING

  • The raw numerical direction combines components from the tracked subspace and its orthogonal complement, which need not coincide with the exact spike and bulk bands.The projected norm can be evaluated on the smaller matrix side when dimensions permit.Source: §B.4 · pp. 17–18
  • With an exact polar factor and full-rank input, normalizers match the exact construction; recovering its exact target additionally requires the tracked basis to span the selected spike subspace.At ρ = 1, the numerical direction reduces to the unreweighted approximate polar direction.Source: §B.4 · p. 18
  • For a nonzero computed raw direction, direct Frobenius-norm rescaling matches its norm to the approximate polar factor, subject to finite-precision rounding.Norm matching controls magnitude, not errors in the polar or subspace approximations.Source: §B.4 · p. 18

C IMPLEMENTATION DETAILS AND SVD-FREE ALGORITHM

The implementation calibrates an approximate noise edge, tracks the spike band without a full SVD, and applies norm-matched two-band updates. Its trajectory diagnostics separately find positive aggregate alignment in the bulk.

C.1 CALIBRATION AND ESTIMATOR STATE

  • The noise-calibrated rule estimates gradient-noise variance from independent minibatch halves, then applies clipping and Nesterov adjustments to set the spectral threshold.The estimate is smoothed over time and assumes slowly changing noise; its white-noise reference cannot generally capture anisotropic or correlated noise.Source: §C.1 · p. 19

C.2 EXACT TARGET AND ORACLE COMPARISON

  • The exact-SVD oracle evaluates the chosen partition and gain at a fixed input; it does not select either one.Comparing it with the numerical update isolates approximation errors, including polar approximation, imperfect spike-subspace recovery, and adaptive rank-cap truncation.Source: §C.2 · pp. 19–20

C.3 WARM-STARTED BLOCK ITERATION

  • The adaptive tracker warm-starts from the previous spike basis, adds fresh probes, performs block-power iterations, and retains thresholded Ritz directions.A hard rank cap applies to the retained basis; the fixed-rank tracker instead iterates a shared, predetermined-width block without thresholding or fresh probes.Source: §C.3 · pp. 20–21, §C.4 · p. 22

C.4 FULL ALGORITHM

  • Each adaptive step combines noise calibration, spike tracking, and numerical norm matching, while persistent momentum, basis, and variance states carry forward.Norm matching preserves the numerical polar direction's Frobenius norm, but does not correct polar or subspace approximation errors.Source: §C.4 · pp. 21–22

D BAND CONTENT ALONG MUON TRAJECTORIES

  • Along the analyzed MUON trajectories, 93.7–97.1% of pooled singular directions lie in the bulk and its signed aggregate reference alignment is positive in every analyzed matrix.Whether increasing the bulk's relative gain improves an update is evaluated separately.Source: D BAND CONTENT ALONG MUON TRAJECTORIES · p. 23

D.1 PROTOCOL AND FRAME-CONSISTENT CALIBRATION

The diagnostics compare singular directions with frame-matched noise edges along Pythia-70M MUON trajectories. They measure each band's alignment to an independently sampled reference gradient without changing training updates.

D.1 PROTOCOL AND FRAME-CONSISTENT CALIBRATION

  • The protocol analyzes three continued-pretraining corpora at four checkpoints, covering 24 hidden weight matrices and 512 thin-SVD directions per matrix.The independent reference is an unclipped 512-sequence gradient from the same corpus and is not used for parameter updates.Source: §D.1 · p. 23
  • Noise edges are calibrated separately for raw-gradient and Nesterov frames, with the Nesterov threshold adjusted for clipping and momentum.Directions above each frame's threshold are spikes; the remaining directions are bulk, under an approximate white-noise reference.Source: §D.1 · pp. 23–24
  • Alignment is the signed projection of the reference gradient onto each unit-Frobenius direction; under the flat profile, the per-direction scores sum to polar alignment.The raw-gradient frame is diagnostic, while MUON uses the Nesterov frame.Source: §D.1 · p. 24
  • The tables pool direction counts and alignment sums before forming ratios, rather than averaging matrix-wise shares.The pooled signed bulk share measures net contribution after cancellation, while the absolute share measures projection magnitude without cancellation.Source: §D.1 · pp. 24–25

D.2 RESULTS: A BROAD BULK WITH POSITIVE AGGREGATE ALIGNMENT

The measured bulk is broad yet contributes positive signed alignment, although its relative contribution varies by frame and checkpoint. Pre-update alignment comparisons favor less bulk weight, so they do not alone determine the effect on post-update loss.

D.2 RESULTS: A BROAD BULK WITH POSITIVE AGGREGATE ALIGNMENT

  • 93.7–97.1% of pooled directions lie in the bulk, which contributes 14.6–26.0% of signed polar alignment in the raw-gradient frame and 10.0–63.5% in the Nesterov frame.Spike directions have larger mean absolute projections, but the bulk's signed alignment is positive in all 24 matrices at every measured checkpoint, in both frames and all three corpora.Source: §D.2 · pp. 25–26, Table 4 · p. 26
  • At 99.9M tokens, Nesterov bulk shares are 63.5%, 45.8%, and 25.3% on FineMath-4, CodeParrot, and Wiki-ar, versus raw-gradient shares of 18.6%, 16.6%, and 18.1%.The Nesterov bulk sums are smaller than the raw-gradient sums in Table 5, but spike sums decrease more; the share also varies non-monotonically over training.Source: §D.2 · p. 26, Table 5 · p. 26

D.3 IMPLICATIONS FOR BULKBOOST

  • The pooled 2.9–6.3% spike fraction motivates testing small fixed fractions, but is not an estimate of the best fixed rank.The diagnostic range is pooled across matrices; individual matrices can fall outside it, and fixed-rank and adaptive rules can form different partitions.Source: §D.3 · pp. 26–27

E.1 PRE-UPDATE REFERENCE ALIGNMENT

  • For fixed partitions, the pre-update score's local preference depends on the difference between mean per-direction alignment in the bulk and spikes.Increasing the bulk-to-spike gain under norm matching reallocates direction norm toward the bulk and suppresses the spikes.Source: §E.1 · pp. 27–28

E PRE-UPDATE ALIGNMENT AND POST-UPDATE LOSS

  • The pre-update aggregate derivative favors less bulk weight in all 24 corpus–checkpoint–frame comparisons, while finite bulk-emphasizing reweighting can still improve the post-update model.The pre-update score uses signed mean alignment by band; its preference is distinct from the directly measured post-update loss effect.Source: §E.1 · pp. 27–28, E PRE-UPDATE ALIGNMENT AND POST-UPDATE… · p. 27

E.2 SINGLE-UPDATE LOSS MEASUREMENTS

The single-update experiment compares shared bulk-emphasizing gains with the exact flat update while holding spectral frames and partitions fixed. Bulk emphasis lowers validation loss in most tested states, despite paired pre-update predictions favoring less bulk weight.

E.2 SINGLE-UPDATE LOSS MEASUREMENTS

  • Each comparison fixes the singular frames and noise-edge partitions, uses a common gain across 24 hidden matrices, and matches each matrix's Frobenius direction norm separately.The exact-SVD branches vary only ρ and are evaluated immediately after the update on fixed validation windows.Source: §E.2 · p. 28
  • Both ρ = 1.25 and ρ = 1.5 lower post-update validation loss relative to the exact flat update in five of six tested states.The exception is FineMath-4 at 50M tokens, where the flat update has lower loss than both bulk-emphasizing updates.Source: §E.2 · pp. 28–29, Table 7 · p. 29
  • The improvements occur even though paired pre-update linearized predictions favor less bulk weight in every tested state.The benefit is measured immediately after the current update, without additional continuation steps.Source: §E.2 · pp. 29–30
  • The experiment tests the combined effect of relative bulk emphasis and spike suppression because norm matching couples the two.Its common gain changes all matrices together, unlike the theorem's single-matrix gain variation.Source: §E.2 · pp. 29–30

F PROOFS FOR SECTIONS 3 AND 4

Appendix F supplies proofs for the paper’s main theorems and supporting results on noise identities and norm matching.

F PROOFS FOR SECTIONS 3 AND 4

F.1 PROOF OF THEOREM 1

This section establishes the two-band framework’s first-order criteria and capture limits, derives normalization and noise-calibration identities, and describes the scope of the continued-pretraining experiments. It also reports that MUON lowers validation loss across all tested corpus-and-scale settings, while initial loss does not reliably predict improvement because it reflects pretraining-distribution fit rather than adaptation gains.

F.1 PROOF OF THEOREM 1

  • A positive between-band utility gap is necessary and sufficient for a strict first-order loss improvement when increasing the bulk gain; a zero gap leaves the test inconclusive.The flat profile is first-order stationary within fixed-frame, norm-matched interventions if and only if all mode utilities are equal.Source: §F.1 · pp. 31–33
  • Two-band reweighting captures the between-band component of tangent utility, while finer shaping can add at most the within-band component to the first-order rate.If utilities are constant within each band, finer shaping cannot improve the first-order rate beyond two bands.Source: §F.1 · pp. 31, 33

F.2 PROOF OF THEOREM 2

  • The proof of Theorem 2 establishes the prescribed update identity for either rectangular orientation, without approximation or a random-matrix assumption.The correction factors through an r-dimensional space.Source: §F.2 · p. 34

F.3 EXACT FROBENIUS NORMALIZATION

  • The exact Frobenius normalizer rescales the two-level spectral profile to match the flat update’s Frobenius norm when r + ρ^2(d − r) > 0.If this quantity is zero, the raw direction is zero and cannot be rescaled to the target norm.Source: §F.3 · pp. 34–35

F.4 ONLINE CALIBRATION OF THE NOISE EDGE

  • The half-batch difference removes the unknown mean gradient and has the full-minibatch mean’s covariance; Nesterov calibration accounts for the current gradient entering directly and through the buffer.The covariance identities and noise edge do not establish that the resulting spectral split is optimal for optimization.Source: §F.4 · p. 36
  • The Marchenko–Pastur edge describes the normalized Gaussian noise matrix’s largest singular value asymptotically, not as a finite-sample bound; isotropic covariance alone does not establish it.The result assumes independent centered Gaussian entries and applies as matrix dimensions grow with a fixed aspect ratio.Source: §F.4 · pp. 37–38

G.1 CORPORA ADAPTATION.

  • The omitted-results appendix describes continued pretraining of five Pythia scales on six mathematics, code, and multilingual corpora; Table 8 reports MUON loss changes after about 100M tokens across scales.Table 8 defines the change as final minus initial held-out loss and reports mean ± standard deviation over three seeds; the supplied passage gives no cell values.Source: §G.1 · p. 39, Table 8 · p. 39
  • Initial held-out loss ℓ0 serves as a proxy for corpus shift from Pythia pretraining: CodeParrot is lowest, Wiki-de and Wiki-zh highest, and the math corpora and Wiki-ar lie between.The corpora retain this order at every model scale; the math corpora overlap with mathematical text in the Pile, while the multilingual wikis are farther from its mainly English distribution.Source: §G.1 · p. 39
  • MUON lowers validation loss in every setting, but higher initial loss does not always correspond to a larger reduction.Wiki-ar gains about twice as much as FineMath-4 at every scale despite starting below it; FineMath-4 starts above FineMath-3 and gains more at every scale.Source: §G.1 · p. 39
  • CodeParrot has the smallest reductions at 160M and 410M, with ∆= −0.479 at 14M and ∆= −0.118 at 410M.Wiki-zh and Wiki-ar retain large gains through 160M and still improve by about 0.7 at 410M.Source: §G.1 · p. 39

G.2 ABLATION STUDIES

The ablations show that noise-calibrated BB-ADAPTIVE closely matches the best fixed fraction without tuning, while a third band does not improve on two. Additional tests compare win frequency, gain sensitivity, runtime, learning-rate choices, and held-out learning curves.

G.2 ABLATION STUDIES

  • The three-band variant ends with final loss 0.002 ± 0.001 higher than BB-STATIC(5%) on average, so the extra band does not help.Bulk-emphasizing gains ρ = 1.25 and 1.5 lower immediate post-update validation loss in five of six tested corpus–checkpoint settings.Source: §G.2 · p. 40
  • The best tested gain improves on ρ = 4 by about 0.0045 on average, while ρ = 2 is best in 20 of 30 settings.The median loss range across tested gains is 0.009, and ρ = 4 remains the shared default for 100M-token runs.Source: §G.2 · p. 40, Table 12 · p. 44
  • 4.6% of dmodel directions are spikes on average under BB-ADAPTIVE, which closely matches the best fixed fraction without tuning; shared BB-STATIC(5%) has comparable aggregate loss.The adaptive rule’s advantage is avoiding fraction tuning, not improved accuracy over a tuned fixed fraction; per-setting selection lowers loss further at extra tuning cost.Source: §G.2 · pp. 39–40
  • BB-STATIC(5%), BB-ADAPTIVE, and BB-STATIC† beat MUON in 23, 24, and 25 of 30 settings, respectively, versus 19 for FREON.The counts use three-seed mean final loss and exclude ties.Source: §G.2 · p. 40
  • Shared learning-rate multipliers match the best multiplier for most methods at each scale, with exceptions at 70M for MUON and 160M for SPECTRA.Using the shared rather than each method’s best sweep rate raises loss from 2.779 to 2.783 for 70M MUON and from 2.552 to 2.553 for 160M SPECTRA.Source: §G.2 · p. 42, Table 11 · p. 43, Figure 2 · p. 43
  • Measured fixed-rank configurations take 1.19–1.59× MUON’s step time, compared with 1.36–2.13× for BB-ADAPTIVE.At 70M–410M, the respective ranges narrow to 1.19–1.27× and 1.36–1.50×; the implementations differ in calibration and subspace tracking.Source: §G.2 · pp. 42–43, Table 14 · p. 45

G.3 ADDITIONAL LEARNING CURVES

  • Final held-out loss decreases steadily with model size on every corpus, while the rate of decrease differs by corpus.Figures 4–8 show held-out loss versus tokens for FineMath-4, FineMath-3, CodeParrot, Wiki-de, and Wiki-ar; fixed-rank curves use p = 1.5%.Source: Figure 3 · p. 45, §G.3 · p. 46, Figure 4 · p. 47
Loading 2610.07497v1…