PaperlayerView exact revision on arXiv

Source-linked AI summary

From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon

Jihwan Kim, Dogyoon Song, Chulhee Yun

arXiv:2610.11309v1cs.LGcs.AImath.OCstat.ML

TL;DR

The paper asks whether differences in optimizer-induced classifier geometry lead to systematic population-accuracy differences. It analyzes optimizer-selected classifiers under isotropic and anisotropic data settings, and reports higher population accuracy for row-normalized methods than spectral and coordinate-wise alternatives within the stated scope.

  • Problem

    Whether optimizer-induced differences in classifier geometry translate into systematic population-performance differences remains unclear.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes the statistical quality of classifiers selected by different optimizer geometries in high-dimensional multiclass classification.

    Source: §1 · p. 2, §1.1 · p. 2

  • Results

    Row-normalized methods achieve strictly higher population accuracy than spectral and coordinate-wise alternatives in both full-batch and random-reshuffling settings.

    Source: §1.1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    In the analyzed settings, preserving class-mean geometry is associated with higher population accuracy for row-normalized methods than for spectral and coordinate-wise alternatives.

    Source: §6 · p. 10

    The analysis is limited to a linear last-layer readout and stylized Gaussian class means and within-class noise; the RR-Adam comparison uses AdamProxy, whose approximation is one-epoch only.

    Source: §3.2 · p. 7, §6 · p. 10

Abstract

from arXiv · show

Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam's coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage.

1 Introduction

The paper asks whether optimizer-induced classifier geometries translate into systematic population-accuracy differences, extending practical evidence for row normalization with a theoretical comparison. It reports an advantage for row-normalized methods under isotropic and specified anisotropic settings, and relates this analysis to prior optimizer-geometry work.

1 Introduction

  • Row-normalized methods are already used in LLM optimizers, but the mechanism behind their reported empirical performance remains poorly understood.Related methods use row normalization in different roles, including as a principal update, within alternating row–column normalization, or after Muon orthogonalization.Source: §1 · p. 1, §1.2 · p. 3

1.2 Related Work

  • Prior theoretical work characterizes optimizer-selected implicit biases and studies convergence or width-scaling, but does not establish that row-normalized methods select statistically better solutions.The paper adds the row norm to the implicit-bias framework and extends AdamProxy to multiclass random reshuffling.Source: §1 · p. 2, §1.2 · p. 3

1 Introduction

  • Row normalization's class-wise Euclidean geometry better preserves population decision-boundary directions, while coordinate-wise and spectral geometries introduce directional distortions.The analysis connects the optimizer geometries to the statistical quality of their limiting classifiers.Source: §1 · pp. 1–2

1.1 Our Contributions

  • Row-normalized GD and momentum variants achieve strictly higher population accuracy than spectral, coordinate-wise, and adaptive alternatives under the isotropic Gaussian-cloud model.The comparison covers full-batch methods and, under random reshuffling, stochastic Signum and exact-SVD Muon; row-normalized momentum also beats AdamProxy.Source: §1.1 · p. 2
  • With independently oriented class-mean and test-noise covariances and power-law spectra, row-normalized methods outperform Adam and Muon; a coordinate-aligned regime can reverse the advantage over Adam.Jointly rotating the covariances restores the advantage with high probability; the class-mean exponent may be any value below one, with any test-noise exponent.Source: §1.1 · pp. 2–3

2 Preliminaries

The preliminaries specify a Gaussian-cloud classification setting, define population accuracy, and formalize the optimizer updates and training conditions used in the analysis.

Data model

  • The isotropic Gaussian-cloud model uses class means, within-class noise, and an independent Gaussian test-noise vector with class-dependent label probabilities.The model defines separate training and test examples and assumes the noise vectors, labels, and class means are mutually independent.Source: §2 · pp. 3–4

High-dimensional scaling

  • The high-dimensional regime keeps class count and input dimension proportional, requires growing minimum class sample size, and excludes exponentially large datasets.The frequency condition permits class probabilities as small as subexponential in dimension.Source: §2 · p. 4

Classifier and evaluation

  • The classifier is a linear multiclass map, and population accuracy is the probability that the true class receives the strictly largest test score.The setting abstracts a language model's final linear layer.Source: §2 · p. 4

Update maps and optimizers

  • Row, entrywise-max, and spectral update maps maximize gradient alignment subject to their respective norm constraints, yielding row-normalized, sign-based, and polar-factor updates.Full-batch and random-reshuffling versions are defined, with momentum variants corresponding to RowGDM, exact-SVD Muon, and Signum.Source: §2 · pp. 4–5

Optimization conditions

  • The setup specifies Adam without bias correction or a numerical-stability constant, plus learning-rate conditions for non-momentum and momentum methods.The stated schedules include η_t = c(t + 1)^−a for c > 0 and a ∈ (0, 1].Source: §2 · p. 5

3 Main Results

The paper establishes a strict population-accuracy advantage for row-normalized methods over competing full-batch optimizers, then shows that the ordering persists under random reshuffling in the stated high-momentum regime. It attributes the advantage to how optimizer geometries shape limiting classifiers.

3.1 Comparison of Optimizers in the Deterministic Setting

  • RowGD and RowGDM achieve strictly higher population accuracy than SpecGD, Muon, SignGD, Signum, and Adam in the deterministic setting.The comparison is established through the optimizer groups’ limiting margin solutions under the paper’s assumptions.Source: §3.1 · p. 5
  • Row-normalized methods preserve class-mean directions, while coordinate-wise and spectral geometries introduce nonvanishing directional distortions.The proof compares row-, spectral-, and max-norm margin solutions and uses Gaussian pairwise-error bounds; it does not require optimizer directional convergence.Source: §3.1 · p. 5

3.2 Robustness to Stochastic Training

  • Under random reshuffling, sufficiently high momentum keeps the row-, spectral-, and max-norm methods close to their respective margin solutions, allowing the strict population-accuracy ordering to persist.The stated stochastic result applies to RR-RowSGDM, RR-Muon, and RR-Signum for every reshuffling realization under the specified conditions.Source: §3.2 · p. 6
  • The paper develops a multiclass AdamProxy for random-reshuffling Adam and proves that, as β1 and β2 approach 1, its direction approximates RR-Adam’s normalized one-epoch displacement.This is a one-epoch approximation at fixed data and dimension, not a claim of long-horizon trajectory agreement.Source: §3.2 · p. 7
  • RR-RowSGDM retains a population-accuracy advantage over AdamProxy because coordinate-wise saturation distorts class-mean geometry while high-momentum RR-RowSGDM stays close to it.The AdamProxy comparison relies on a nonvanishing distortion after scaling the classifier to unit minimum training margin.Source: §3.2 · p. 7

4 Extensions to Anisotropic Gaussian Models

The section tests whether row-normalization’s population-accuracy advantage survives anisotropy and identifies a coordinate-aligned setting where its ordering against Adam reverses. Independent orientations preserve the advantage, while a common Haar rotation restores it in the reversal setting.

4 Extensions to Anisotropic Gaussian Models

  • The anisotropic analysis uses clean training on class means and Gaussian test noise, with class-mean and test-noise covariances allowed to vary in their spectra and orientations.The model holds each covariance trace at d and takes the class-to-dimension ratio to a positive finite limit.Source: §4 · p. 8, §4.1 · p. 8

4.1 Persistence of Row-Norm Advantage under Independent Orientations

  • For power-law covariances with independent random eigenbases, row-normalized methods outperform Adam and Muon when the class-mean exponent is below one, even with arbitrary test-noise anisotropy.The theorem assumes directional convergence; its sufficient dimension threshold may depend on the covariance exponents.Source: §4.1 · p. 8
  • The condition on the class-mean exponent keeps signal spread across dimensions, helping row-norm margin solutions stay close to class-mean directions; concentration in a few directions can push them away from that geometry.The authors report fitted class-mean exponents below one in language-model representations as a possible connection to this regime.Source: §4.1 · pp. 8–9

4.2 Coordinate-aligned Reversal and Recovery under Haar Rotation

  • With diagonal, aligned class-mean and test-noise covariances, the RowGDM–Adam ordering can reverse; jointly applying a common Haar rotation restores the ordering by randomizing their orientation relative to Adam’s coordinates.The rotation preserves covariance spectra and relative alignment, so alignment with Adam’s coordinate axes is also needed for the reversal.Source: §4.2 · p. 9

5 Experiments

The experiments compare optimizer performance on synthetic Gaussian-cloud data and in last-layer-only LLaMA training. Row-normalized methods lead in both settings described.

5 Experiments

  • On isotropic Gaussian-cloud data with k = 60 and d = 20, RowGD and RowGDM outperform SpecGD, Muon, SignGD, and Adam on all 30 paired seeds (Figure 2, left).Source: §5 · p. 10

Language model experiment.

  • In last-layer-only LLaMA-60M training, RowSGDM attains the lowest final validation perplexity among RowSGDM, Muon, and Adam.The experiment freezes the trained model, zero-initializes its output head, and trains only that layer.Source: Language model experiment · p. 10

6 Conclusion

The conclusion links row-normalized methods’ population-accuracy advantage to preserving class-mean geometry and reports support across full-batch and random-reshuffling settings. It also bounds the analysis to a stylized linear last-layer model and outlines extensions and reproducibility materials.

6 Conclusion

  • For linear classification abstracting a language model’s last layer, row-normalized methods achieve strictly higher population accuracy than spectral and coordinate-wise alternatives in full-batch and random-reshuffling settings.The conclusion attributes the advantage to better preservation of class-mean geometry.Source: §6 · p. 10
  • The analysis is limited to linear last-layer readouts with Gaussian class means and within-class noise; modern language-model representations are more complex.The anisotropic independent-orientation analysis assumes directional convergence, and the Adam results use zero numerical-stability constant and AdamProxy for RR-Adam.Source: §6 · p. 10

AI use statement

  • The authors report using generative AI to assist with writing, related-work identification, appendix proof drafting, code debugging, and figure and table presentation; they reviewed the outputs.The authors state they take responsibility for the final content.Source: AI use statement · p. 10

Reproducibility statement

  • The paper states that it provides assumptions and settings, complete proofs, and experimental setup information in the appendices.Appendix H covers synthetic experiments, the language-model experiment, and representation analysis.Source: Reproducibility statement · p. 11

Overview of the Appendix

  • The appendix organizes proofs around optimizer-side implicit bias, population-accuracy comparisons, anisotropic extensions, coordinate-specific reversal and recovery, and experiments.Appendix G addresses reversal and recovery under a common Haar rotation; Appendix H supplies experimental details and evidence.Source: Overview of the Appendix · p. 15

A Additional Related Work

The related-work discussion situates the paper among implicit-bias, adaptive-optimization, matrix-geometry, and multiclass generalization studies. The appendices develop optimizer-side margin results and supporting direction-set convergence lemmas.

A Additional Related Work

  • Prior implicit-bias work characterizes optimizer-selected margin geometries, while this paper adds the row norm and compares population accuracy across complete optimizer-selected margin faces.The discussion also distinguishes the paper’s exact spectral comparator from preconditioned methods and finite-iteration polar approximations, to which its theorems do not automatically transfer.Source: A Additional Related Work · pp. 16–17
  • The discussion distinguishes Adam’s coordinate-wise normalization and sensitivity to coordinate systems from other adaptive methods, and limits its theoretical Adam result to zero-stability-constant, no-weight-decay dynamics.Its AdamProxy analysis addresses additional data dependence under incremental or mini-batch updates with random reshuffling.Source: A Additional Related Work · p. 16
  • The paper compares distinct optimizer-selected margin geometries in a proportional multiclass regime, where norm-specific margin problems may have non-unique solutions.Uniform results over complete margin faces avoid relying on a single selected solution for the optimizer-level comparison.Source: A Additional Related Work · p. 17

B Implicit Bias of Row-Normalized Methods: Full-Batch and Random Reshuffling

  • Appendix B extends full-batch normalized steepest-descent and random-reshuffling momentum analyses, then specializes them to row normalization and its margin guarantees.The random-reshuffling analysis uses a geometry-adapted primal–dual atom radius for Row, Spectral, and Max geometries.Source: B Implicit Bias of Row-Normalized… · p. 18

B.1 Common setup and auxiliary lemmas

  • Appendix B.1’s lemmas connect effective or exact margin bounds to convergence toward the associated direction sets and control cumulative remainders under general learning rates.The margin normalization lemma also relates cumulative margin growth to maximum-margin solutions.Source: §B.1 · pp. 18–19

B.2 Full-batch max–sum dominated norms

The section establishes full-batch margin results for norm-based methods and extends the analysis to random-reshuffling momentum. It also characterizes row normalization as steepest descent in the row norm and gives its margin guarantees.

B.2 Full-batch max–sum dominated norms

  • For max–sum dominated ψ, normalized steepest descent and momentum have ψ-maximum-margin accumulation points; full-batch Adam’s accumulation points are max-norm maximum-margin solutions.Source: §B.2 · pp. 20–21

B.3 Geometry-adapted random-reshuffling momentum

  • Geometry-adapted random-reshuffling momentum has a margin guarantee for arbitrary matrix norms when ρψ,β,b(X) is positive.Source: §B.3 · pp. 21, 23, 24

B.4 Specialization to row normalization

  • Row-wise normalized gradient methods are normalized steepest-descent methods for the row norm.Source: §B.4 · p. 25
  • Row-wise normalized full-batch GD has margin-convergence guarantees, with the same conclusions extending to fixed momentum under Assumption 2.6.Source: §B.4 · p. 25
  • Random-reshuffling row-wise normalized SGDM has a margin guarantee when εrow,β,b(X) is below γrow(X).Source: §B.4 · p. 26

C Proofs for Deterministic Optimizers

The appendix proves geometric properties of deterministic optimizer solutions, including feasibility, boundary separation, and controls on max-norm, spectral, and polar geometries. It transfers these results to multiclass population accuracy, establishing error bounds and the claimed high-probability optimizer ordering.

C Proofs for Deterministic Optimizers

  • Row geometry preserves population decision-boundary directions, while Max has a nonvanishing clipped-coordinate angle gap and Spectral geometry is controlled by a reduced polar factor with hard-edge truncation at α = 1.The appendix converts these geometric statements into population-error bounds and an optimizer-level accuracy ordering, and derives deterministic error exponents separating the three geometries.Source: C Proofs for Deterministic Optimizers · p. 26

C.4 Max-norm Representatives Have a Positive Angle Gap

  • Max-norm optimality imposes a coordinate cap while feasibility requires a large projection onto each empirical mean-pair direction, producing a positive angular gap.The appendix uses a clipped minimum-energy problem to show these requirements are incompatible with near-perfect alignment.Source: C Proofs for Deterministic Optimizers · p. 26, §C.4 · p. 31

C.1 Reduction from Direction Sets to Minimum-norm Representatives

  • Positive common feature rescaling leaves normalized pairwise row directions and average boundary distance unchanged, and scaling test inputs likewise preserves accuracy.The proof reduces direction-set comparisons to minimum-norm representatives using scale invariance.Source: C Proofs for Deterministic Optimizers · p. 26, §C.1 · p. 28

C.2 A Quantitative Common Gaussian-cloud Event

  • Under the stated assumptions, a quantitative common Gaussian-cloud event holds with stretched-exponential probability and controls empirical class-mean deviations from population means.The proposition states deterministic vanishing error bounds and simultaneous event conditions.Source: §C.2 · p. 28

(H5) The max-norm optimum is finite and satisfies

  • The common-event analysis also establishes uniform pairwise-direction control and constructs a finite max-norm optimum satisfying the stated condition.The supplied proof describes a feasible matrix construction for the max-norm condition and uniform concentration over ordered pairs.Source: (H5) The max-norm optimum is finite… · pp. 28–29

C.3 Row-norm Representatives Align with the Population Mean Differences

  • Uniformly over row-norm minimizers, normalized row differences align with the corresponding population mean differences as dimension grows.The deterministic rigidity result yields this alignment when empirical means and residuals satisfy small-error conditions.Source: §C.3 · pp. 30–31

C.4.1 The Clipped Minimum-energy Problem

  • The clipped minimum-energy problem has a unique minimizer with coordinates given by a thresholded form, where the threshold solves a unique scalar equation.The supplied result states uniqueness and coordinatewise clipping; the excerpt does not include the explicit coordinate formula.Source: §C.4.1 · p. 31
  • The clipped minimum-energy problem has a unique solution τ∈(0,1).The proof establishes feasibility and verifies the stated condition (60).Source: §C.4.1 · pp. 34–35

C.4.2 Application to Max-norm Minimizers

  • On a shared high-probability event, every ordered pair of every max-norm minimizer has boundary-normal distance at least δ_max−o(1).Source: §C.4.2 · p. 36

C.5 Spectral Polar Geometry

  • For Gaussian matrices with p/d→1, the hard-edge lemma gives a quantitative eigenvalue-count bound and establishes that A_d has rank min{p,d} almost surely.Source: §C.5 · p. 37
  • The all-aspect spectral input controls reduced-polar rank and norm, pairwise row separation, off-diagonal entries of S_m, and residual projections.Source: §C.5 · p. 38

(S4) The reduced polar geometry satisfies

  • The reduced polar geometry has singular values bounded away from zero when α≠1, while pairwise row-norm convergence holds in averaged L1 at α=1.Proposition C.15 also treats exact and near spectral minimizers; its exact-minimizer bound has displayed scales (η_d+ξ)^1/2 for α≠1 and (η_d+ξ)^1/4 for α=1.Source: (S4) The reduced polar geometry… · pp. 39–41
  • The reduced polar-factor analysis establishes spectral-norm boundary separation under Assumptions 2.1 and 2.2.The bound holds on the events in Proposition C.12 with the deterministic error from (87) at ξ = 0.Source: (S4) The reduced polar geometry… · p. 41

Proof of the deterministic geometry theorem

  • The deterministic geometry theorem combines the geometric bounds to establish statements (42)–(46) for sufficiently large dimension.Its proof uses a deterministic vanishing error envelope and the positive gaps δ_sp(α) and δ_max.Source: Proof of the deterministic geometry… · p. 42

C.6 From Direction-set Geometry to Population Accuracy

  • The transfer results relate pairwise Gaussian error bounds and classifier geometry to population accuracy, including for convergent direction sets.The geometry-to-accuracy theorem gives separation when its stated vanishing-error and positive-gap conditions hold; continuity lemmas transfer population error and boundary metrics along direction sets.Source: §C.6 · pp. 42–44

C.7 Proof of the Deterministic Optimizer Accuracy Ordering

  • The proof transfers the claimed population-accuracy ordering to seven full-batch optimizer trajectories with the theorem’s stated probability.Spectral-GD and exact-SVD Muon converge to spectral direction sets; RowGD and RowGDM to row direction sets; SignGD, Signum, and full-batch Adam to max direction sets.Source: §C.7 · p. 44

C.8 Proof of the Limiting Classifier Geometry

The proof establishes the population-mean spectral geometry and transfers the optimizer-selected direction and mean-angle limits to optimizer trajectories, taking the training-time limit before the dimension limit.

C.8 Proof of the Limiting Classifier Geometry

  • The proof establishes the population-mean version of the spectral geometry using an empirical-to-population polar transfer.Source: §C.8 · pp. 44, 46
  • For α > 1, Row's limiting mean angle is zero.Source: §C.8 · p. 46
  • Taking t→∞ before d→∞ transfers direction-set convergence to RowGD and RowGDM, SpecGD and Muon, and SignGD, Signum, and Adam, along with their corresponding mean-angle limits.The full-batch implicit-bias results identify each optimizer family with its corresponding maximum-margin direction set.Source: §C.8 · p. 47

C.9 Quantitative Population-error Exponents for Deterministic Geometries

The appendix quantifies how optimizer geometry separates population-error exponents and extends the geometric comparison to stochastic optimizers. Its bounds cover complete maximum-margin direction sets and establish stability under near-margin and stochastic perturbations.

Define the Row–Spectral exponent-gap constant

  • For 0 < α ≤ 1, the Row–Spectral exponent gain is exact; at α = 1, the spectral interval collapses to 32/(9π²).For α below α⋆ ≈ 1.225, exact-SVD Muon has asymptotically smaller population error than Adam; the ordering for α ≥ α⋆ remains open.Source: §C.9 · p. 49, Define the Row–Spectral exponent-gap… · p. 50

C.9 Quantitative Population-error Exponents for Deterministic Geometries

  • Row and Max directions have directed-pair exponents 1/2 + oP(1) and 1/π + oP(1), respectively, uniformly over their complete direction sets.The directed-pair exponent controls multiclass error, and for bounded exponents the normalized log error agrees with it up to o(1).Source: §C.9 · pp. 47, 49

D Proofs for Stochastic Optimizers

  • Random-reshuffling optimizers lie in near-maximum-margin direction sets, which retain enough deterministic geometric separation for a strict population-accuracy ordering at sufficiently large momentum.The comparison takes the training-time limit at each fixed dimension before d tends to infinity.Source: D Proofs for Stochastic Optimizers · p. 50

D.1 Near-margin directions and near-minimum-norm representatives

  • Near-minimum-norm representatives preserve normalized pairwise row differences while adding only bounded relative norm slack.Positive rescaling leaves those normalized pairwise differences unchanged.Source: §D.1 · p. 51

D.2 Quantitative stability of near minimizers

  • Near-minimizers are quantitatively stable: row-norm weights approach class means, while max- and spectral-norm deviations have vanishing bounds.The row-norm bound is stated as max_i ||w_i − μ_i||₂ ≤ Cq^1/4; the spectral bound depends on α.Source: §D.2 · p. 51

D.3 A dimension-independent momentum threshold

  • For fixed β ≥ β₀, uniform stochastic near-margin geometry holds for sufficiently large d, but the time threshold may depend on dimension, dataset, initialization, and learning-rate schedule.The stochastic error bound is deterministic on E_near and uniform over V in the stated family.Source: §D.3 · p. 52

D.4 Proof of the stochastic optimizer accuracy ordering

The proof establishes a population-accuracy ordering between RR-RowSGDM and AdamProxy, using bounds on effective margins and population errors. It analyzes AdamProxy as a bridge from high-momentum random-reshuffling Adam to a coordinate-induced angular gap, with Row estimates holding for every reshuffling realization and AdamProxy estimates deterministic conditional on the training data.

D.4 Proof of the stochastic optimizer accuracy ordering

  • The population-accuracy ordering in (6) holds simultaneously for every β∈[β0, 1) and every reshuffling realization.The proof transfers the margin-set ordering to optimizer trajectories and accuracy.Source: §D.4 · p. 54

E.2 RR-RowSGDM versus AdamProxy population accuracy

  • RR-RowSGDM achieves higher population accuracy than AdamProxy through a coordinate-induced angular gap relative to near-row geometry.AdamProxy’s margin-normalized iterates have minimum training margin one and satisfy a trajectory-level coordinate bound.Source: E Analysis of AdamProxy for… · p. 54, §E.2 · p. 58

E.1 One-epoch β1, β2 →1 RR Adam approximation

  • As β1, β2→1, random-reshuffling Adam is approximated over one epoch by the deterministic AdamProxy direction.The proof derives finite-population batch moments and freezes the epoch’s gradient memories to establish this approximation.Source: E Analysis of AdamProxy for… · p. 54, §E.1 · pp. 55, 57

E.2 RR-RowSGDM versus AdamProxy population accuracy

  • AdamProxy’s pairwise boundary separation is at least δAP − o(1), uniformly over pairs and classifiers in the capped set.The proof obtains the separation from uniform empirical estimates and a reverse triangle inequality.Source: §E.2 · p. 61
  • For sufficiently high momentum, RR-RowSGDM has a positive effective margin because its relative margin deficit is bounded by ϑAP < 1.The bound also covers md = 1, where the deficit is zero.Source: §E.2 · p. 62
  • Combining the Row and AdamProxy bounds yields the claimed population-accuracy ordering, with no additional exceptional event required.The Row momentum estimates hold for every reshuffling realization, and the AdamProxy estimates are deterministic conditional on the training data.Source: §E.2 · p. 63

E.3 AdamProxy margin characterization

The section characterizes AdamProxy’s limiting margin direction through a data-dependent quadratic margin problem. The selected direction corresponds to a self-consistent parameter that also determines normalized dual weights.

E.3 AdamProxy margin characterization

  • Under the stated assumptions, AdamProxy’s normalized trajectory converges to a positive-margin direction characterized by a data-dependent quadratic margin problem.The problem minimizes a quadratic form subject to unit margin constraints on the training examples.Source: §E.3 · p. 63
  • The limiting direction is characterized by the problem at a self-consistent parameter θ⋆, which coincides with the normalized dual weights.The characterization assumes linear separability, nonzero input coordinates, and convergence of the normalized proxy direction.Source: §E.3 · p. 63
  • The parameter θ⋆ is selected by the trajectory’s limiting wrong-class coefficients, and every class has positive limiting margin.The limiting coefficients are supported on margin-minimizing atoms.Source: §E.3 · pp. 64–65

F Extension to Anisotropic Gaussian with Independent Orientations

This appendix establishes a high-probability population-accuracy advantage for selected Row limits over selected Spectral and Max limits in the anisotropic Gaussian setting. It uses the independence of optimizer-selected directions from the test-noise covariance eigenbases and reduces optimizer limits to finite selectors with convergent accuracies.

Optimizer-selected directions

  • Clean training makes optimizer-selected limiting directions depend only on class means, leaving them independent of the Haar eigenbases of test-noise covariances.The proof establishes this measurability for the optimizer iterates and their normalized directional limits.Source: F Extension to Anisotropic Gaussian… · pp. 66–67

Selected-direction accuracy ordering

  • A Gaussian error-transfer lemma turns separated directed-pair signal bounds into strict multiclass population-accuracy ordering for sufficiently large dimension.It bounds Row-family errors using a union bound and competitor-family errors using a directed pair with a smaller signal bound.Source: F Extension to Anisotropic Gaussian… · p. 66

Mean geometry of the selected directions

  • The proof characterizes selected Row, Spectral, and Max directions from the class means, with the selected-direction geometry bounds holding simultaneously over each finite selector collection.For Spectral and Max, the bounds apply uniformly over their exact margin-direction faces.Source: F Extension to Anisotropic Gaussian… · pp. 67, 68, 70

Test-noise geometry

  • Conditioned on the mean geometry, selected Row test-noise energies concentrate when sΣ<1, while selected competitor directions satisfy lower bounds with polynomial failure probabilities.These bounds transfer the independently oriented Gaussian test-noise geometry to the selected directions.Source: F Extension to Anisotropic Gaussian… · pp. 70–71

Selected-direction accuracy ordering

  • When sΣ<1, selected Row classifiers strictly outperform Spectral/Max classifiers with probability ≥1−e^(−dχ); for sΣ≥1, the ordering holds with probability ≥1−Cζd^−ζ.The first regime requires sμ<1 and χ<min{1−sμ,1−sΣ}; the class/prior growth condition is log((k−1)/π_min,d)=o(d/log d).Source: F Extension to Anisotropic Gaussian… · pp. 72–73

F Extension to Anisotropic Gaussian with Independent Orientations

  • Lemma F.2 converts seven optimizer limits into finitely many F_cl-measurable Row, Spectral, and Max selectors with convergent population accuracies.Source: F Extension to Anisotropic Gaussian… · p. 73
  • With probability at least 1 − Cd^-c, every selected Row limit has strictly higher population accuracy than every selected Spectral or Max limit.The exponent c and constant C can be chosen independently of sµ and sΣ; only the sufficiently-large-d threshold depends on them.Source: F Extension to Anisotropic Gaussian… · p. 73

G Coordinate-Specific RowGDM–Adam Reversal

This section studies how coordinate alignment and test anisotropy affect the population-accuracy ordering between Row methods and Adam, including reversal and restoration of a Row advantage under a common Haar rotation; comparisons cover complete solution sets. Using limiting spectral summaries with eigenvalues in a fixed bounded range, it gives sufficient dispersion conditions for a transition and characterizes their feasibility by the spectral range and a unique upper-range threshold.

G Coordinate-Specific RowGDM–Adam Reversal

  • With aligned diagonal test anisotropy and h²γ̄₂ > 1, Row and Max have a unique transition point λ_c separating strict accuracy orderings; at λ_c, only limiting-coefficient equality is asserted.For each fixed λ ≠ λ_c, the strict ordering holds for all sufficiently large d with exponentially small failure probability.Source: G Coordinate-Specific RowGDM–Adam… · p. 77
  • The strict accuracy comparisons apply uniformly across the complete Row and Max maximum-margin solution sets.The coefficient-to-accuracy comparison gives strict uniform ordering from a fixed positive coefficient gap for sufficiently large d.Source: G Coordinate-Specific RowGDM–Adam… · pp. 74, 77, 80
  • For bounded-spectrum test covariance, a common Haar rotation yields strict Row-over-Max accuracy ordering when the mean regularity error δ_d tends to zero.The rotation-only event has failure probability at most Ck²e^−cdε⁶, and the strict ordering follows for sufficiently large d.Source: G Coordinate-Specific RowGDM–Adam… · pp. 79–80
  • For fixed λ > λ_c, the original and commonly rotated problems have opposite Row–Max orderings despite identical class Gram matrices, mean norms, pair distances, singular values, and test-noise eigenvalues.The original diagonal ordering and rotated Row-over-Max ordering each hold with exponentially small failure probability for sufficiently large d.Source: G Coordinate-Specific RowGDM–Adam… · pp. 80–81
  • The transition yields a strict comparison between RowGDM and Adam trajectory liminf and limsup accuracies without assuming directional convergence.The argument transfers the uniform complete-face ordering to positive-definite Gaussian test covariances; it also applies to RowGD.Source: G Coordinate-Specific RowGDM–Adam… · p. 81
  • Corollary G.8 states that spectral dispersion implies a transition, assuming Proposition 4.5’s conditions concerning existence of the limiting spectral summaries and an additional condition.Source: G Coordinate-Specific RowGDM–Adam… · p. 81
  • For eigenvalues in [l, u] with trace normalization, the limiting spectral variance satisfies vΓ ≤ (1 − l)(u − 1), sharply.The bound is attained asymptotically by a two-point spectral profile supported at l and u, with masses (u − 1)/(u − l) and (1 − l)/(u − l).Source: G Coordinate-Specific RowGDM–Adam… · p. 81
  • For each fixed l in (0, 1), the sufficient dispersion condition becomes feasible above a unique critical upper bound u_c(l) > 1.The condition is non-vacuous only when the spectral range permits enough variance.Source: G Coordinate-Specific RowGDM–Adam… · p. 81

H.1 Synthetic Experiments: Setup and Additional Results

The experiments test optimizer comparisons under baseline conditions, one-factor robustness changes, and anisotropic data settings. RowGD and RowGDM retain strong comparisons with the alternatives, with a coordinate-aligned setting reversing their ordering against Adam.

Additional deterministic and stochastic comparisons

  • RR-RowSGDM outperforms RR-Muon, RR-Signum, and RR-Adam on every paired seed; the deterministic panel preserves the main-text Row-over-Spectral and Row-over-coordinate-wise ordering.RR-Adam is an empirical comparator; the paper’s theoretical stochastic Adam comparison uses AdamProxy.Source: §H.1 · p. 82

Auxiliary geometries

  • Figure 4 adds FrobGD, ColumnGD, and ordinary GD as auxiliary references alongside RowGD, SpecGD, and SignGD on the same 30 paired baseline datasets.These are finite-dimensional context, not additional theoretical comparators.Source: §H.1 · pp. 82–83, Figure 4 · p. 83

Robustness of the deterministic ordering

  • Across robustness settings, RowGD and RowGDM have the two highest mean accuracies and beat Adam and SignGD on every paired seed.They beat SpecGD and Muon on all 30 seeds except with exactly one or five training examples per class, where they win 29 of 30 comparisons. The baseline uses 30 paired seeds; learning rates are selected and reported on the same evaluation sample.Source: §H.1 · pp. 82–83

Anisotropic synthetic experiments

  • Across six independent-orientation anisotropic settings, RowGD and RowGDM beat SpecGD, Muon, SignGD, and Adam on all 30 paired seeds, with mean advantages of 2.82–5.73 percentage points.Figure 8 shows a representative setting with sµ = 0.5 and sΣ = 2.Source: §H.1 · p. 83, Figure 8 · p. 86
  • In diagonal coordinate-aligned settings, RowGDM beats Adam at λ = 0.25, but Adam beats RowGDM at λ = 1; a shared Haar rotation restores RowGDM’s lead.Each reported ordering holds on all 30 paired seeds.Source: §H.1 · p. 83, Figure 9 · p. 87

H.2 Language Model Experiment

In last-layer LLaMA training, RowSGDM outperforms Adam and Muon on held-out metrics and boundary alignment, with the same qualitative ordering in a larger-model experiment. A separate shorter-horizon check finds only a 0.0152% advantage over Adam, while the main experiment examines a later training regime.

Model and training setup

  • The main experiment trains a zero-initialized output head for 22,000 updates while freezing the pretrained LLaMA-60M model body.The head has 32,000 × 512 parameters; RowSGDM, Muon, and Adam use the specified practical optimizer settings and monitoring-set learning-rate selection.Source: §H.2 · p. 88

Horizon check

  • At 11,000 updates, RowSGDM’s monitoring-set perplexity is 28.5871, versus 28.5914 for Adam and 28.6373 for Muon; its advantage over Adam is 0.0152% in this single-seed comparison.The 22,000-step experiment probes whether the qualitative ordering persists and becomes clearer at a later horizon.Source: §H.2 · p. 88

Data and evaluation

  • Training uses a shared shuffled C4 stream for each seed, while final evaluation uses a separate held-out set excluded from learning-rate, stopping, and checkpoint selection.The held-out evaluation contains 46,186,882 valid targets and has no document-hash overlap with the monitoring set.Source: §H.2 · p. 88

Boundary-alignment diagnostic

  • Boundary discrepancy and cosine measure alignment between learned classifier boundaries and empirical class-mean differences.The diagnostic uses a fixed sample of one million unordered class pairs across optimizers and seeds.Source: §H.2 · pp. 88–89

Results

  • RowSGDM has lower held-out NLL and perplexity than Muon and Adam on all five paired seeds, alongside the smallest boundary discrepancy and largest boundary cosine on every seed.The mean paired NLL differences are −0.001957 versus Muon and −0.002555 versus Adam; Table 1 summarizes held-out results and boundary alignment.Source: §H.2 · p. 89, Table 1 · p. 89

LLaMA-130M experiment

  • On LLaMA-130M, RowSGDM has the lowest held-out perplexity on every seed: 22.3797 ± 0.0014, versus 22.4096 ± 0.0030 for Muon and 22.4153 ± 0.0010 for Adam.Its mean boundary discrepancy is 1.19051, compared with 1.19664 for Adam and 1.20028 for Muon.Source: §H.2 · p. 89

H.3 Spectral Exponents for Language Model Representation

The section measures class-mean spectral exponents in language-model representations and finds estimates below one at three final checkpoints and across six sampled OLMoE snapshots. These finite-dimensional summaries are descriptive and do not directly test the theorem’s full probabilistic model.

Scope

  • These finite-dimensional spectral summaries are not intended as a direct test of the full probabilistic model underlying the theorem.The measured class-mean exponents are relevant to the theorem’s sµ < 1 regime.Source: §H.3 · p. 89

Experimental setup

  • The analysis uses final pre-head representations from OLMoE-1B-7B-0125, Qwen3-30B-A3B, and AMUSE-720M, with hidden dimension 2048 and 20 million valid C4 positions per model.The primary class-mean analysis gives equal weight to every observed next-token class.Source: §H.3 · p. 90

Class-mean spectral exponent

  • The mean exponent is estimated by an unweighted pointwise OLS fit over all 2048 eigenvalue ranks, with one regression observation per rank and no log binning or rank weighting.The class-balanced covariance uses one empirical class mean per observed class.Source: §H.3 · p. 90

Within-class covariance spectral exponents

  • Within-class covariance exponents are descriptive summaries of within-class anisotropy, using classes above model-specific reliability thresholds.Thresholds are 8192 observations for OLMoE and AMUSE and 16384 for Qwen3, leaving 218, 211, and 120 classes, respectively.Source: §H.3 · pp. 89–90

Results

  • Class-mean spectral exponents are below one at all three final checkpoints: 0.957735 for OLMoE, 0.932780 for Qwen3, and 0.985366 for AMUSE.Within-class covariance exponents are more heterogeneous and can exceed one.Source: §H.3 · p. 90, Table 2 · p. 90

OLMoE across annealing

  • Across six OLMoE snapshots spanning 0–101B added annealing tokens, every class-mean exponent estimate is below one, with a total range of 0.001830 and no monotone trend.The estimates range from 0.956530 to 0.958359.Source: §H.3 · p. 90

Related papers

Loading 2610.11309v1…