PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Representation Learning Decouple from Loss Minimization? Polar Updates Have an Answer

Akash Kumar

arXiv:2609.36240v1cs.LGmath.OCstat.ML

TL;DR

The paper asks whether representations can continue to develop while loss stalls, and studies this question for polar updates using linear toys, mean-field theory, and finite teacher–student networks. Across the reported settings, teacher-subspace AGOP alignment can improve during sustained loss oscillations, including an exact recovery result under stated mean-field conditions.

  • Problem

    Training loss can oscillate or stall without indicating whether a model's input directions are still changing, so the paper asks whether useful representation learning continues through sustained plateaus.

    Source: §1 · p. 1

  • Method

    The paper combines conditional linear-toy analysis, a population mean-field ReLU proof, and finite-network experiments using direction-only AGOP alignment measures and head-refit diagnostics.

    Source: §1 · p. 2

  • Results

    Across 33 ReLU, GELU and SiLU teacher configurations, student-AGOP alignment holds or improves during approximately period-2 oscillations, while selected clean ReLU refits have losses about 5–11 times below cycle-mean training loss.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The reported results show that loss plateaus can coexist with teacher-direction recovery, although the exact update identities alone do not establish network oscillations or AGOP alignment.

    Source: §5.2 · p. 9, §6 · p. 11

    The polar-update identities and isotropic-quadratic cycle example do not establish the observed network oscillations or AGOP alignment; the alignment-drift proposition is conditional.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Does representation learning stop when the training loss stops improving? We study this question for matrix Muon, whose polar-normalised updates have a step length set by the gradient's rank rather than its norm. Near the edge of stability, full-batch Muon on teacher-student problems enters approximately period-2 loss oscillations that persist for thousands of steps: the cycle-mean loss stays flat or rises, yet the weights keep moving and the learned features continue to align with the teacher subspace. For linear teacher-student learning toys, we derive explicit cycle and alignment formulas and conditional plateau and decay bounds. For a population mean-field ReLU model, we prove that, under stated dimension, initialisation and small-head conditions, the leading eigenspace of the average gradient outer product (AGOP) recovers the teacher subspace exactly during a loss plateau, before the loss later drops. In all 33 ReLU, GELU and SiLU teacher configurations we study, direction-only alignment metrics show the student AGOP aligned with, or still aligning to, the teacher subspace during the period-2 oscillations; projected head refitting on selected configurations shows that the learned directions are useful for prediction, and further measurements distinguish AGOP alignment from weight-mass concentration. In deep residual ReLU students, freezing the downstream layers while the first layer trains with full-batch exact polar updates recreates a nearly flat cycle-mean loss with improving input-AGOP alignment; freezing and unfreezing switch between this plateau and loss decrease, and the effect is sensitive to momentum and to the choice of orthogonaliser.

1 INTRODUCTION

The paper asks whether useful representations can keep developing while training loss stalls, and answers yes for polar updates of matrix Muon. It studies this separation through linear learning toys, a population mean-field ReLU model, and finite teacher–student networks.

Our answer

  • Polar normalisation sets nonzero gradient singular values to one, so full-rank updates have a step length determined by rank rather than gradient norm.A fixed step length alone neither creates an oscillation nor implies learning.Source: §1 · p. 2

Main finding

  • Across 33 ReLU, GELU and SiLU teacher configurations, student-AGOP alignment holds or improves during thousands of approximately period-2 loss-oscillation steps.The cycle-mean loss stays nearly constant or rises while weights keep moving.Source: §1 · p. 2

Our answer

  • In linear toys, conditional phase and margin conditions yield nearly symmetric two-cycles with loss floors scaling as η^2, while smaller components drain and teacher alignment can improve.For matrix Muon, the paper gives exact weight cycles, loss floors and teacher-alignment formulas.Source: §1 · p. 2

Measuring learning without the loss

  • The paper measures principal angles between the student AGOP's leading eigenspace and the teacher subspace, with a no-oracle head-refit probe of predictive usefulness.On six clean ReLU configurations, the cycle-mean loss is about 5–11 times the refit diagnostic over the final 30% of training; projection changes features as well as the head.Source: §1 · p. 2

Where the separation comes from

  • In deep residual ReLU students, freezing downstream layers while the first layer takes exact polar steps maintains nearly flat cycle-mean loss as input-AGOP alignment rises.Freezing and unfreezing switch between this regime and loss reduction; the effect depends on momentum and orthogonaliser choice.Source: §1 · p. 2

Our answer

  • The population mean-field ReLU result recovers the teacher subspace exactly in the leading AGOP eigenspace during a 1% loss plateau, before a later loss decrease.The stated result is conditional on dimension, initialisation and small-head conditions.Source: §1 · pp. 2–3

2 RELATED WORK

The related work connects representation learning during nonmonotone optimization to several distinct settings, while distinguishing this paper's sustained exact-polar plateaus from earlier results.

Feature learning and oscillatory dynamics

  • Silent alignment studies early NTK alignment before appreciable loss reduction, while Chen & Bruna (2023) prove teacher-direction recovery during oscillatory training for a single-neuron ReLU model under specified conditions.Catapult experiments link loss spikes to improved student–teacher AGOP alignment.Source: §2 · p. 3

AGOP and weight geometry

  • Prior work relates weight geometry to a predictor's AGOP, while this paper tracks teacher-subspace alignment and distinguishes it from weight-mass concentration.That distinction does not itself contradict an identity between weight geometry and the student's own AGOP.Source: §2 · p. 3

Muon and spectral learning

  • Practical Muon approximately orthogonalises momentum, whereas this paper isolates exact-polar, zero-momentum updates to study when AGOP alignment occurs relative to loss reduction.Fixed step length alone does not establish a cycle.Source: §2 · p. 3

3 PROBLEM SETUP

The setup spans linear learning toys and teacher–student ReLU networks, using exact polar updates and alignment measures to study how features change during loss plateaus.

3 PROBLEM SETUP

  • The study uses linear regression toys and a one-hidden-layer teacher–student network to isolate fixed-Hessian dynamics and then add an activation-dependent Hessian.The learning rate is fixed near the edge of stability; p is input dimension, and r_t and r_s denote teacher rank and student width.Source: §3 · pp. 3–4

3.1 DATA AND TEACHER–STUDENT MODEL

  • The M4 teacher uses Gaussian inputs and a teacher subspace V, while the student is a one-hidden-layer ReLU network with trainable hidden weights and a fixed head.Full-batch gradients are used, and the head is frozen so representation changes pass through the hidden weights.Source: §3.1 · p. 4

3.2 OPTIMISER: MATRIX MUON

  • Matrix Muon updates hidden weights using the gradient's polar factor, whose nonzero singular values are one and whose step length is independent of gradient norm.The analysis uses the exact polar update; practical Muon approximates it with Newton–Schulz iterations, and the one-column case reduces to normalized gradient descent.Source: §3.2 · p. 4

3.3 ALIGNMENT METRICS FOR M4

  • Alignment metrics compare the leading student-AGOP eigenspace with the teacher subspace, separately tracking direction recovery and weight Frobenius mass in that subspace.The no-oracle diagnostic projects weights onto their leading AGOP eigenspace and refits the head; the oracle diagnostic instead uses the teacher subspace.Source: §3.3 · pp. 4–5

3.4 LEARNING TOYS

  • The learning toys use linear students and fixed-Hessian regression, including normalized gradient descent and multi-output matrix Muon with teacher directions in high- or low-variance blocks.Their AGOP alignment reduces to alignment of the student's weight range with the teacher range in the multi-output case.Source: §3.4 · p. 5

3.5 POPULATION MEAN-FIELD MODEL

  • The population model uses Gaussian inputs, a ReLU teacher and student, a refitted intercept, and a fixed-head scale; its infinite-width limit represents hidden weights by a probability law.The setup also includes deep residual ReLU students whose unfrozen hidden matrices take full-batch polar updates, with freezing interventions specified separately.Source: §3.5 · pp. 5–6

4 LEARNING TOYS WITH CLOSED-FORM THEOREMS

The learning toys derive conditional plateau and decay results for scalar normalised-gradient regression and exact cycle, loss-floor, complement-drain, and alignment formulas for matrix Muon. They show how loss can remain near a step-size-set floor while teacher alignment improves, with normalised updates behaving differently from stable-step gradient descent.

Scalar regression (M2)

  • The leading residual coordinate approaches a nearly symmetric two-cycle with loss floor μ1η^2/4, while other coordinates decay geometrically and teacher alignment keeps improving.These conclusions require phase-selection and contraction-margin conditions; the symmetric-cycle rate is a reference, not a guarantee for every trajectory.Source: §4 · pp. 6–7

Matrix Muon (M3)

  • For M3, full-column-rank residuals give matrix-Muon steps of length η√r, and specified positive-definite cycle conditions yield exact weight 2-cycles.The two phase losses have mean at least η^2/4 tr(Σ̂|S); on block-isotropic Σ̂, the cycle floor is λDη^2r/4.Source: §4 · pp. 7–8
  • Near an M3 cycle, complement modes contract under the stated condition P, P′ ≻ ημr+1; on the symmetric cycle their per-step multiplier is 1−2μj/μi.This is a local linearisation, not a convergence theorem for off-cycle trajectories.Source: §4 · pp. 7–8
  • On the symmetric M3 cycle, teacher-alignment cosines are determined by residual singular values, with minimum squared cosine (1+η^2/4)^−1.The exact formula requires the stated subspace-orthogonality condition; on an exact weight cycle, loss and alignment repeat every two steps.Source: §4 · pp. 7–8

Plateau interpretation

  • The nonzero plateaus belong to the normalised updates: stable-step gradient descent reaches zero loss on the same objectives, while the observed normalised-update floors scale as η^2.M2 means are within 0–4% of μ1η^2/4 for η∈[0.01, 0.2]; M3 floors are 1.05–1.39 × λDη^2r/4.Source: §4 · p. 8

5 THE ONE-HIDDEN-LAYER RELU NN UNDER MATRIX MUON

Across one-hidden-layer and deep residual students, loss plateaus can coexist with continued AGOP alignment to teacher directions. The section combines empirical observations, exact polar-update identities, a conditional alignment-drift result, and a mean-field theorem.

5.1 EMPIRICAL PHENOMENOLOGY

  • Across 33 ReLU, GELU, and SiLU configurations, period-2 loss increments are nearly alternating while AGOP alignment rises in 29 configurations, including some with increasing training loss.The weakest teacher direction remains unrecovered in three ReLU configurations with rt ≥16.Source: §5.1 · pp. 8–9

5.2 UNIVERSAL POLAR-UPDATE IDENTITIES

  • For full-column-rank gradients with rt ≤ M, matrix Muon has exact update identities, including a fixed step length and a small-leakage update-mass split of rt : (M − rt).These identities do not imply AGOP alignment or a periodic network trajectory; the exact two-cycle result applies separately to an isotropic quadratic.Source: §5.2 · p. 9

5.3 PLATEAU HEIGHT AND ALIGNMENT DRIFT

  • The proposed leading-order plateau-loss decomposition is a conjecture, and the reported comparison does not test it directly.The comparison uses the head-refitted oracle rather than the fixed-head term in the decomposition.Source: §5.3 · p. 9
  • A conditional proposition bounds alignment drift on windows with a separated, near-teacher leading AGOP subspace and specified transverse dynamics.Under additional stationary slow-mode conditions, vk = C(θ)q^αek(1+o(1)); the assumptions are unverified for the observed trajectories.Source: §5.3 · p. 9

5.4 A MEAN-FIELD THEOREM: PLATEAU, THEN FEATURE LEARNING, THEN LOSS DROP

  • In the population mean-field ReLU model, stated dimension and small-head conditions permit exact top-r AGOP recovery of the teacher subspace during a loss plateau, before a later loss drop.The theorem bounds plateau loss within 1% of its initial value and specifies a later loss-drop time.Source: §5.4 · p. 10

5.5 DEEP RESIDUAL STUDENTS

  • Across depths 4, 8, 16, and 24, freezing layers after W1 keeps cycle-mean loss within 5% in almost all runs while input-AGOP cos2 min rises by 0.38–0.53.Depth-6–16 residual students also show oscillating loss with rising alignment; Figure 4 reports median per-run cycle-mean loss change of at most about 2.6%.Source: §5.5 · p. 10, Figure 4 · p. 10
  • At depth 8, switching downstream layers between frozen and training states switches the plateau on or off; slower downstream steps still allow alignment to rise.Training downstream layers lowers loss by 8–14% at steps η1/53 to η1/3.3, and by a median 4.7% at η1.Source: §5.5 · p. 10, Figure 5 · p. 11
  • The deep-student plateau-plus-alignment pattern is sensitive to momentum and orthogonaliser choice, and results for transformers, vision transformers, and a CNN remain limited.The one-hidden-layer small-head plateau is reported as more robust to the tested choices.Source: §5.5 · p. 11

6 DISCUSSION AND CONCLUSION

Under polar updates, a loss plateau can coexist with continued representation learning: weights move and AGOP directions align with the teacher, while the loss may remain flat or rise. The paper connects this separation to learning toys and mean-field theory, measures it in neural students, and discusses what the diagnostics and freezing interventions establish.

6 DISCUSSION AND CONCLUSION

  • LAGOP-opt and the teacher-subspace oracle lie below plateau loss, showing useful directions are present, but not that the head alone limits loss.On six cleanest ReLU configurations, LAGOP-opt is about 5–11 times below cycle-mean training loss; the remaining loss decomposition is unmeasured.Source: §6 · p. 11
  • At long-run ReLU checkpoints, 83–90% of AGOP eigenvalue mass lies in the teacher subspace, versus 29–50% of weight Frobenius mass.This challenges proportionality between the weight Gram matrix and AGOP; power-law variants of the Neural Feature Ansatz remain untested.Source: §6 · p. 11
  • Catapult-related alignment improvements are also seen during plateau intervals, including when loss rises; in deep students, gradient descent and Adam lower loss rather than holding a floor.The paper notes that the tested Newton–Schulz quintic and momentum settings disrupt the combination of flat loss and continued alignment.Source: §6 · pp. 11–12
  • Direction-only AGOP metrics and head-refit probes reveal progress hidden by the loss, but whether these probes inform practical Muon training at scale remains open.The conclusion presents direction-only measurements as a complement to training loss.Source: §6 · p. 12
  • Open questions include finite-width, finite-sample theory, interactions of approximation, momentum, and minibatch noise in practical Muon, and a quantitative account of plateau height and alignment drift.The proposed starting point for the latter is testing Conjecture 4.Source: §6 · p. 12
  • Across learning toys, a mean-field model, and neural students, representation learning can continue during a loss plateau; freezing downstream layers can switch between the plateau and loss decrease.The conclusion describes the separation as proved under stated conditions in toys and a mean-field model, measured in one-hidden-layer and deep residual students, and switched by freezing downstream layers.Source: §6 · pp. 11–12

A.2 PROOF OF THEOREM 1 (SCALAR REGRESSION)

For scalar regression with normalized-gradient updates, the analysis gives conditional results for a period-2 plateau and decay of off-leading modes. The cycle family and its phase selection also explain why invisible misalignment may drain slowly and why asymmetric plateaus are not proved for generic starts.

A.2 PROOF OF THEOREM 1 (SCALAR REGRESSION)

  • After a phase-selected handoff and under a contraction-margin condition, off-leading residual modes decay geometrically while the analysis converts that decay into improving teacher alignment.The spectral gap is strict, and the result does not assert entry into the handoff neighborhood from arbitrary initialization.Source: §A.2 · p. 16, §A.3 · p. 18
  • At η = 0.3, a unit misalignment hidden below 5% of the loss floor needs at least 443 steps per e-fold to drain on the symmetric cycle.The bound is leading-order and illustrates how low-eigenvalue misalignment can persist inside the plateau.Source: §A.2 · p. 16
  • Without the phase-selection hypothesis, the cycle may be asymmetric and its plateau height is specified, but sustained flatness for such cycles is only a numerical finding.The first-order drain rate for modes satisfying the stated smallness condition is also given.Source: §A.2 · pp. 16–17

A.3 THE NORMALISED-GRADIENT RECURSION ON A QUADRATIC

  • The scalar normalized map has a continuum of period-2 cycles {a, a − η}; only a = η/2 has constant leading loss.The theorem is stated conditionally from a phase-selected handoff because of this cycle family.Source: §A.3 · pp. 17, 19

A.4 THE SCALAR CYCLE FAMILY AND GENERIC INITIALISATION

  • Off-mode gradient content with little loss exerts little symmetrising pressure, so generic starts can freeze an asymmetric cycle; loss-visible slow modes instead symmetrise as they drain.The associated fitted slope of 1.09 is a numerical comparison, not a proof.Source: §A.4 · p. 22

A.5 PROOF OF PROPOSITION 2

The proof characterizes full-rank matrix-Muon two-cycles in an invariant teacher subspace, including their loss floor and the contraction of complementary residuals. The appendix then connects hidden-mode decay to loss envelopes and checks the theory against selected finite runs.

A.5 PROOF OF PROPOSITION 2

  • Off-cycle complementary residuals contract locally under the stated positive-definiteness condition, while exact two-cycle entry fixes the weights and prevents further alignment change along either parity.The linearisation does not establish convergence from arbitrary initialisation.Source: §A.5 · pp. 23–24
  • Every full-rank two-cycle in the r-dimensional invariant subspace has two-step mean loss at least η^2/4 times the sum of its top-r covariance eigenvalues, with equality on the symmetric cycle.The symmetric cycle is centered at the teacher and has constant loss at that floor; rank-deficient cycles may sit lower.Source: §A.5 · pp. 23–24

A.6 LOSS ENVELOPES: WHERE THE FEATURE-LEARNING SIGNAL SITS IN THE LOSS

  • For slow modes, the envelope analysis allows a small measurable loss signal while alignment improves; the reported lower-envelope-to-hidden-loss change ratio is 0.797–0.798 in the generic asymmetric run.The text cautions that the exact identity does not hold instantaneously when off-mode content changes.Source: §A.6 · p. 24

A.7 NUMERICAL CONSISTENCY CHECKS

  • In numerical checks, exact identities, constructed cycles, and alignment formulas had residuals below 10^-12, and all 50 checks met their stated tolerances.The tolerances are not interval-arithmetic certificates, and the trajectories do not establish convergence from arbitrary initialisation.Source: §A.7 · pp. 24–25

A.8 ADDITIONAL FIGURES

  • In Figure 6(b), loss reaches its floor at step 120 and stays there for 2,900 more steps as residual and teacher alignment each rise to 1.00.The slow, loss-visible off-mode symmetrises the cycle; the fast-mode case instead has a plateau 1.33 times the floor.Source: Figure 6 · p. 26
  • Figure 7 compares loss and residual energy for two matrix-Muon runs, with teacher subspace equal to D in one run and contained in D⊥ in the other.The caption reports an asymmetric-cycle loss 1.31 times the floor for run A and a nearly symmetric cycle for run B.Source: Figure 7 · p. 27
  • Figure 8 contrasts normalised-update floors with stable-step GD: NGD is within 0–4% of its reference floor, while GD reaches 4.6 × 10^-18.For M3, Muon stops at 1.05–1.39 times the population reference floor; plain GD reaches 2.5 × 10^-9.Source: Figure 8 · p. 27

B.1 PROOF OF PROPOSITION 3 (POLAR-UPDATE IDENTITIES)

The appendix proves exact polar-update identities, gives examples and constraints for period-2 orbits, and establishes a symmetry property on a block-isotropic quadratic. It also clarifies what these results do not establish about network alignment.

B.1 PROOF OF PROPOSITION 3 (POLAR-UPDATE IDENTITIES)

  • For full-column-rank gradients, the polar update has Frobenius step length η√r, and its mass splits between the teacher subspace and its complement through leakage scalars.The identities are exact without an additional smallness assumption on leakage, although the trajectories' leakage scalars were not measured.Source: §B.1 · pp. 28–29

B.2 PERIOD-2 ORBITS ON THE ISOTROPIC QUADRATIC

  • On an isotropic quadratic, matrix Muon admits a continuum of symmetric period-2 orbits; on general positive-definite quadratics, such orbits exist on invariant subspaces.The network experiments show approximately period-2 loss signatures while weights continue to move, not exact weight orbits.Source: §B.2 · p. 29

B.3 BLOCK-ORTHOGONAL EQUIVARIANCE ON A BLOCK-ISOTROPIC QUADRATIC

  • On a block-isotropic quadratic, the Muon iteration commutes with block-orthogonal transformations, preserving an invariant initialisation law over time.This symmetry does not determine relative energy in the teacher and orthogonal blocks or explain the observed network preference.Source: §B.3 · pp. 29–30

B.4 ASSUMPTIONS AND PROOF OF CONDITIONAL SECULAR DRIFT

The appendix derives conditional alignment-drift bounds under finite-order transverse-dynamics and slow-mode assumptions, while explicitly delimiting their scope. Its experiment appendix reports teacher-activation grids and describes how configurations were selected and displayed.

B.4 ASSUMPTIONS AND PROOF OF CONDITIONAL SECULAR DRIFT

  • The conditional drift analysis assumes a separated leading AGOP subspace with a graph representation and finite-order dissipative transverse dynamics; the alignment calculation does not establish plateau residence or near-period-two loss.The sharper result also requires stationary slow-mode conditions and controls against nonlinear cancellation.Source: §B.4 · pp. 30–32
  • Under the stated family and window conditions, fixed-factor error reduction takes Θθ(q^-α) optimization steps, but the exponent is not identified for the full empirical Muon family without checking its transverse dynamics.The result's relative-error claim also requires q and the error to tend to zero while kq^α grows.Source: §B.4 · pp. 30, 32

C.1 PLATEAU-SEARCH GRID: RELU, GELU, AND SILU TEACHERS

  • In the smooth-teacher sweep, SiLU with rt = 20 and η = 0.85 has loss rising from 1.92 to 2.06 while minimum squared alignment grows by 0.43 over the second half of training.The GELU rt = 16, η = 0.85 configuration shows the analogous pattern at smaller amplitude.Source: §C.1 · p. 33

C.2 PER-TEACHER TOP-6 CONFIGURATIONS AND THE FULL RELU 10-CONFIG GRID

  • Figure 2 reports rising AGOP alignment during period-2 loss plateaus in selected ReLU, SiLU, and GELU teacher configurations, using medians over 53 seeds.The figure displays training-loss summaries, test loss and head-fit oracles alongside alignment metrics.Source: §C.2 · p. 33

C.3 CONFIGURATION SELECTION

  • Table 2 summarizes 33 one-hidden-layer ReLU, GELU, and SiLU teacher configurations across five seeds, separately from the single-seed search and 53-seed figure runs.Its caption specifies the period-2 signature and cycle-mean loss ratio metrics.Source: C ADDITIONAL EXPERIMENTS AND… · p. 33, Table 2 · p. 34
  • The selection score rewards a flat second-half loss, alignment growth, and loss reduction; for SiLU and GELU it additionally favors effective rank near teacher rank.The score does not use the head-fit oracle, and the multi-seed figures retain choices made on the seed-5 sweep.Source: §C.3 · pp. 33–34

C.4 THEORY–EXPERIMENT OVERLAY AND TEST-BY-TEST DETAIL

Across ReLU, GELU, and SiLU configurations, period-2 loss oscillations coexist with teacher-subspace alignment, while several diagnostics distinguish alignment from prediction quality, weight mass, and theoretical drift scales.

Test A and Figure 3

  • Across 15 ReLU, GELU, and SiLU configurations, the period-2 loss signature is nearly perfect, while AGOP alignment is within 7% of its maximum for rt ≤12 except one ReLU case.For rt ≥16, AGOP alignment is 6–28% below maximum; the loss signature is empirical and does not establish an exact weight cycle.Source: §C.4 · pp. 35–36

Test C

  • The plateau-height comparison uses a refitted-head oracle and therefore does not test the fixed-head decomposition conjecture.The table reports the plateau loss at 10×–36× the oracle loss.Source: §C.4 · p. 36, Table 4 · p. 39

Test B

  • The polar update’s V-mass is approximately 2–5% below rt on three ReLU checkpoints, but the leakage scalars needed to attribute the gap were not measured.The exact polar-mass identity is distinct from the AGOP V-mass plotted in Figure 3(b).Source: §C.4 · pp. 35–36, Table 3 · p. 37

Test F

  • Across long-run ReLU configurations, AGOP alignment reaches cos2 ∈ [0.85, 0.96] while the weight Frobenius mass fraction Asub remains in [0.29, 0.50].This differs from the simplest Neural Feature Ansatz, which would make the two mass fractions equal; power-law variants are not tested.Source: §C.4 · p. 38, Table 6 · p. 40

Tests D and G

  • The measured alignment-drift ratio to ηrt/M ranges from 1.10 × 10^-4 to 9.16 × 10^-4 across four configurations.The comparison does not measure a one-step weight-update signal-to-noise ratio or verify the conditional drift hypotheses.Source: §C.4 · pp. 36–39

Tests C and E

  • The plateau loss is 10×–36× above the head-fit oracle, while LAGOP-opt is below training loss over most of the plateau on selected ReLU configurations.The projected directions support a better predictor, but the diagnostic does not establish that the head alone limits loss.Source: §C.4 · pp. 36–37, Table 4 · p. 39

C.5 HOW THEORY AND EXPERIMENT ALIGN: A DISCUSSION

The discussion separates exact polar-map identities from empirical learning dynamics, and distinguishes AGOP direction alignment from weight mass and loss reduction. The deep-network experiments specify a synthetic Gaussian teacher–student setup and evaluate paired loss and alignment trajectories across configured depth arms.

C.5 HOW THEORY AND EXPERIMENT ALIGN: A DISCUSSION

  • The qualitative loss/alignment separation appears across ReLU, SiLU, and GELU teachers, but the polar-mass and weight-versus-AGOP-mass tables are restricted to ReLU.Smooth teachers show the separation at tested learning rates roughly half those of the ReLU examples.Source: §C.5 · p. 40
  • The polar Frobenius identity and V/V⊥ decomposition are exact, whereas the period-2 loss signature is an empirical observation rather than an algebraic fact.The measured polar-update V-mass is about 2–5% below rt on three ReLU checkpoints, with leakage scalars unmeasured.Source: §C.5 · p. 40
  • Measured drift is about 10^-3 of ηrt/M per step and plateau height is 10–36× the head-fit oracle, while the proposed drift conditions and fixed-head decomposition remain untested.The head-refit diagnostic indicates that learned directions support a better predictor than the trained network realises.Source: §C.5 · p. 40

D.1 DATA, TEACHER AND STUDENT

  • The deep experiments use synthetic Gaussian inputs and noise-free fixed-teacher labels, with a residual ReLU student and frozen head trained by full-batch exact-polar Muon.The residual scale is 1/(L −1), and the optimiser uses no momentum, weight decay, or learning-rate schedule.Source: §D.1 · p. 41

D.2 METRICS AND SCORING PROTOCOL

  • Input-AGOP alignment compares its leading rt-dimensional eigenspace with the teacher subspace, while loss cycles are summarized by paired-step statistics and the cycle mean.The depth campaigns use disjoint seed blocks; the later freezing-study follow-ups reuse paired problems and initialisations.Source: §D.2 · p. 42, §D.3 · p. 42

D.4 DEPTH RESULTS

In deep residual ReLU students, first-layer feature learning can accompany a nearly flat cycle-mean loss when deeper layers are frozen or slow. Changing deep-layer training switches between that plateau and loss decrease, while alignment continues to rise.

D.4 DEPTH RESULTS

  • Deep runs use a first-layer step around 0.7 and deep-layer steps at most 3 × 10^-3, with synthetic Gaussian data, full-batch exact-polar updates, a frozen head, and fixed step sizes.The loss branches keep drifting during the window, so a period-2 loss does not imply a period-2 parameter orbit.Source: §D.4 · pp. 42–43, Table 9 · p. 43

E FREEZING THE DEEP LAYERS SWITCHES THE PLATEAU ON

  • With deep layers frozen or slow, cycle-mean loss stays flat while alignment rises in W1; with comparable deep-layer steps, loss decreases and some alignment moves into deeper layers.Freezing or unfreezing the deep layers mid-run switches between flat and decreasing loss while alignment rises in both cases.Source: E FREEZING THE DEEP LAYERS SWITCHES… · p. 43

E.1 SETUP

E.2 RESULTS

In deep residual students, freezing downstream layers can hold cycle-mean loss nearly flat while W1 alignment improves; training selected downstream layers converts the plateau into loss decrease. The same separation appears across several depths, with important step-size and architecture boundaries.

E.2 RESULTS

  • At depths 8 and 16, selected small deep-layer steps preserved flat loss as alignment improved, but larger steps generally lowered cycle-mean loss; plain networks were not robust across problems.At depth 8, 142 of 144 runs passed the strict test at the two smallest listed steps, while depth 16 had no runs within 5% at η1/50; depth 6 was an exception where frozen deep layers raised loss by 7%.Source: §E.2 · p. 45
  • Freezing deep layers at step 500 kept cycle-mean loss within 5% in all 48 runs while cos2 min rose by +0.12; unfreezing made loss fall 10.8% as alignment rose +0.18.At ηon = η1, freezing lowered loss about 15% before a flat plateau, while unfreezing lowered it about 20% as cos2 min rose from about 0.45 to 0.85.Source: §E.2 · pp. 45–46
  • Training only the deep layer next to the head preserved the plateau in 45 of 48 runs, whereas training the layer next to W1 lowered cycle-mean loss by 21%.With the layer next to W1 training, 90% of alignment remained when deep layers were reset; for blocks of at least four layers, at most 7 of 48 runs stayed within 5%.Source: Figure 16 · p. 46, §E.2 · p. 46

E.3 MECHANISM

  • With a fixed downstream map, the large polar step yields a flat-mean period-2 cycle while W1 directions rotate toward the teacher; training the layer immediately after W1 is enough to turn the plateau into loss decrease.The deep layers can fit targets using the features W1 currently supplies, and then carry part of the alignment.Source: §E.3 · p. 46
  • The reported scope is residual MLP students of a Gaussian teacher with full-batch exact polar updates and a frozen head; other architectures or data are not established.At depth 6, the flattest loss required slow but non-zero deep-layer steps, and the plain-network step was not tuned.Source: §E.3 · p. 46

F A FREEZING PRINCIPLE FOR THE PLATEAU WITH FEATURE LEARNING

  • The freezing principle isolates feature progress: polar W1 steps against fixed g improve AGOP alignment while loss stays on the step-set floor; training g converts that progress into loss decrease.Training W1 alone by gradient descent or Adam lowers loss, so the plateau is not required for feature learning.Source: F A FREEZING PRINCIPLE FOR THE PLATEAU… · p. 47

F.1 DEPTH AND WIDTH

  • At depths 4 and 24, frozen deep layers preserved cycle-mean loss within 5% in 24 of 24 and 23 of 24 runs while cos2 min rose; width 64 and untuned wider settings were less forgiving.The plain network failed across first-layer steps 0.05 to 1, with cycle-mean loss falling by 27–41%.Source: §F.1 · p. 47

F.2 TRAINING ONLY W1 AFTER OTHER SCHEMES

Training only W1 after other schemes separates feature alignment from loss change: polar steps can rapidly improve alignment while holding loss at a step-set floor. Small-head experiments show that the plateau's survival depends on what sets the floor, the orthogonaliser, and momentum.

F.2 TRAINING ONLY W1 AFTER OTHER SCHEMES

  • After slow GD or Adam pretraining, 15 of 16 W1-optimizer combinations reached cos2 min 0.93–1.00 within 200 steps, while continuing whole-network training reached only 0.02–0.71.The comparison changes the optimizer on W1 as well as freezing the downstream layers, so it does not isolate freezing as the cause of the speed-up.Source: §F.2 · p. 47
  • A polar W1 step of 0.66 held cycle-mean loss near its step-set floor, whereas GD and Adam on W1 continued lowering loss by 21–30% after slow pretraining.A long plateau with continuing feature improvement appeared when deep layers remained near initialization, including after Muon pretraining or freezing from initialization.Source: §F.2 · pp. 47–48

F.3 NEWTON–SCHULZ AND MOMENTUM: TWO REGIMES

  • In the small-head population ReLU setting, exact Polar and Newton–Schulz variants reached AGOP alignment Ar ≥0.9 within 120–280 steps; population momentum left alignment and plateau timing unchanged but delayed specialization slightly.With minibatch noise, momentum shortened the plateau and sped specialization; the iteration's gain on the mean direction set loss speed, while gains on remaining directions affected neuron specialization.Source: §F.3 · p. 49
  • For the deep student's large-step plateau, a convergent Newton–Schulz iteration reproduced the exact-polar cycle in 23 of 24 runs, but Muon's quintic broke the clean cycle and no run passed.Nesterov momentum often aligned features before a plateau formed, while plain EMA at 0.9 failed to align the weakest direction; the small-head regime tolerated the approximations differently because its floor was set by head scale.Source: §F.3 · p. 49

F.4 OTHER ARCHITECTURES AND SCOPE

Architecture experiments find a clean plateau with continued feature learning mainly in the residual MLP and small-head settings, with only weak evidence for the transformer. The mean-field ReLU result proves exact teacher-subspace recovery during a loss plateau under stated conditions and connects to finite-width dynamics on a restricted horizon.

F.4 OTHER ARCHITECTURES AND SCOPE

  • A clean flat-mean period-2 cycle appeared in 8–10 of 10 runs for the transformer, 4-patch vision transformer, and CNN, but only the transformer kept learning weakly; the authors claim the principle only weakly for it.Muon's quintic removed the clean cycle in all three models, and training every matrix removed the plateau; the paper does not claim the effect for vision transformers or the CNN.Source: §F.4 · pp. 50–51

G THE MEAN-FIELD THEOREM: COMPLETE PROOF

  • The proof uses computationally searched rational parameter choices, then verifies the certificate inequalities with outward-rounded interval arithmetic.The stated exact inputs and enclosure formulas specify a finite calculation that can be checked independently of the search procedure.Source: G THE MEAN-FIELD THEOREM: COMPLETE PROOF · p. 53

G.1 MAIN RESULT

  • In the mean-field ReLU model, the initially isotropic AGOP's top-r eigenspace becomes exactly the teacher subspace during a window with loss flat within 1%, before loss later drops by a fixed fraction of target variance.The theorem's setting is a balanced ReLU teacher with r ≥ 2 orthonormal directions and ordinary Gaussian initialization.Source: §G.1 · p. 53

G.1.1 SETTING

  • The finite-width dynamics converge to mean-field dynamics as width m grows, but only on horizons where Σνt remains positive definite; the theorem itself concerns mean-field dynamics.The stated convergence horizon is [0, tF].Source: §G.1.1 · p. 54

G.1.2 MAIN THEOREM

This section develops and proves Theorem 51 for population mean-field ReLU learning, establishing under stated conditions a loss plateau followed by a later loss drop and exact teacher-subspace recovery in the leading-r eigenspace of the AGOP at a finite time, with a strict eigengap. The proof combines symmetry-preserving reduced dynamics, growth and continuity bounds, and a finite-horizon certificate supported by particle-level monotonicity and probability estimates.

G.1.2 MAIN THEOREM

  • Under the theorem’s dimension and initialization conditions, sufficiently small heads yield a plateau, exact top-r AGOP recovery of U by tF, and a later loss drop of at least .03V.During the plateau, loss remains between .99V and 1.01V and varies by at most 1%; at tF, Ar = 1 and λr ≥ (1 + δ/2)λr+1. The three conclusions hold together for α ∈ (0, min{α∗(tP), a0/η̄}).Source: §G.1.2 · p. 55

G.1.3 REMARKS

  • The theorem’s feature-learning guarantee is proved first for zero head scale, then transferred to small positive heads; its certified window is long in steps but entails small particle movement.The transfer threshold a0 is not explicit, and RMS displacement through tF is at most η̄ d tF ≤ τF σ0 < .101σ0. The result concerns population Gaussian data, a balanced ReLU teacher on orthonormal directions, and a fixed shared small head; recovery is proved at tF, while the initial AGOP is isotropic.Source: §G.1.3 · pp. 55–56

G.1.4 STRUCTURE OF THE PROOF

  • The proof divides the plateau and later loss drop from the head-free feature-learning argument, then assembles the results and transfers feature learning to small heads.The proof roadmap assigns the mean-field setup and scalar reduction to G.2, plateau and drop to G.3, and head-free feature learning to G.4–G.6.Source: §G.1.4 · p. 56, Table 11 · p. 56

G.2 THE MEAN-FIELD MAP

  • The mean-field map represents width-m Polar updates as a map on particle laws, with trajectories invariant under Sr × O(D) and continuous in head scale and width.At zero head scale, particles reduce to points in R^r × R, and the AGOP has a three-block form whose top-r space is U when two scalar inequalities hold.Source: §G.2 · p. 56

G.2.1 DEFINITIONS AND THE EXACT IDENTITY

  • The exact identity makes the empirical particle measure follow the mean-field map, while scaling shows that head and step size affect the trajectory through their product.The scaling preserves whitening matrices and AGOP eigenspaces and eigenvalue ratios.Source: §G.2.1 · pp. 57–58

G.2.2 REGULARITY

  • The force is continuous and differentiable away from specified singular directions, and absolutely continuous initial laws remain absolutely continuous with finite second moments along the trajectory.In particular, the trajectory assigns zero mass to the origin; the force regularity excludes a finite union of rays.Source: §G.2.2 · pp. 58–59

G.2.3 SYMMETRY

  • Permutation symmetry on the teacher coordinates and orthogonal symmetry on their complement make the teacher, data law, force, and whitening matrix equivariant under Sr × O(D).For an invariant particle law, the whitening matrix commutes with every transformation in this symmetry group.Source: §G.2.3 · p. 60
  • At α = 0, the MF trajectory remains G-invariant with positive-definite covariance, and its reduced law evolves autonomously through ∥f∥, ā, and N.The exact reduced dynamics apply to G-invariant laws with no mass at zero.Source: §G.2.3 · pp. 61–63

G.2.5 THE MF AGOP AND THE CRITERION FOR (F)

  • For a G-invariant law, the MF AGOP’s top-r space is U with a strict gap exactly when min(λ1, λc) > λ⊥, and the gap ratio is min(λ1, λc)/λ⊥.A sufficient condition is ΓU > Γ⊥ and λ1 > λ⊥; for an O(d)-invariant law, the AGOP is a multiple of the identity.Source: §G.2.5 · p. 64

G.2.6 CONTINUITY IN THE HEAD SCALE AND IN THE WIDTH

  • The update map is continuous in both the law and head scale when the initial law has no mass at zero and the limiting covariance is positive definite.Under these conditions, covariance, particle updates, and the next-step law converge as αk → α and W2(μk, μ0) → 0.Source: §G.2.6 · p. 65
  • As α → 0, for every t ≤ T the MF law and AGOP converge to their α = 0 counterparts, and the covariance remains positive definite for sufficiently small α.There is an αT > 0 for which positive definiteness holds at every t ≤ T whenever 0 ≤ α ≤ αT.Source: §G.2.6 · p. 66
  • At fixed α, the width-m network’s law, loss, and scaled AGOP converge almost surely to their mean-field counterparts; with α > 0 and a strict eigengap, its top-r eigenspace also converges.This holds through every t ≤ T when the mean-field covariance is positive definite for t < T.Source: §G.2.6 · p. 66
  • Under the stated trajectory conditions, continuity in the head scale and width yields convergence of loss and AGOP; with a strict eigengap, the top-r spectral projector is continuous.The eigenspace claim follows from Davis–Kahan continuity; multiplying the matrix by α² > 0 does not change its eigenspace.Source: §G.2.6 · p. 66

G.3.2 THE LOSS SANDWICH

  • The loss is bounded by 1 − 2√k1 yU ≤ L/V ≤ 1 − c_x x + k1 y², relating it to output scales y, yU and mean mode x.The analysis states that these three scalar statistics of the particle law suffice for the plateau and loss-drop results.Source: §G.3 · p. 67

G.3.1 OUTPUT SCALES AND ONE MF STEP

  • A whitened mean-field step has increment covariance η̄²P, while the RMS output scales grow by at most the clock increment.P is the orthogonal projector onto the range of the gradient covariance Σν.Source: §G.3.1 · p. 67

G.3.3 THE SELF-FORCE AND THE MEAN MODE

  • The cited right-hand side is approximately 0.5946 at r = 2 and increases with r.Source: §G.3.3 · p. 71

G.3.4 THE PLATEAU

  • For sufficiently small head scales, the plateau theorem gives a nondecreasing, nonnegative mean mode; the loss-drop theorem separately bounds loss along the Gaussian-initialized trajectory.The plateau result assumes r ≥ 2, d > r, κ > 0; the loss-drop result assumes α > 0.Source: §G.3.4 · p. 72, §G.3.5 · p. 73
  • The plateau window is controlled by output-scale bounds and mean-mode monotonicity, with α* determined by the exact positive root of 1.17933 ȳ + .34085 ȳ² = .01.The root is ȳ = .0084587115…; the plateau estimate applies for α ∈ (0, α*(tP)].Source: §G.3.4 · p. 72

G.3.5 THE LOSS DROP

  • The loss-drop mechanism is a linear gain from mean-mode growth against the quadratic penalty k1y², with the initial covariance’s linear term canceling exactly.Theorem 43 states a loss-drop bound for the mean-field trajectory from a Gaussian initial law with α > 0.Source: §G.3.5 · p. 73
  • The loss-drop certificate guarantees L(0) − L(t2) ≥ .03V; at α = α∗, t2 ≤ 6.62κ + 1 when tP ≤ .1007κ.For 0 < α < α∗, the stated bound scales as t2 = ⌈(α∗/α)CR(κ + tP)⌉.Source: §G.3.5 · pp. 73–74

G.4.1 SETUP AND INITIAL LAW √

  • From ν0 = N(0, κ²Id), the α = 0 trajectory remains G-invariant, and every particle’s contrast magnitude |ξ| never decreases and is positive almost surely.Consequently, the contrast statistic Φ(t) remains positive.Source: §G.4.1 · p. 75

G.4.2 THE KICK BUDGET AND THE MEAN MODE

  • Perpendicular growth occurs only through overshoots, while the mean mode has a positive expected increment bounded below by 2√(fminfmax)/(fmin + fmax).Particles with bt ≤ 2Nt satisfy |ζ(t + 1)| ≤ |ζ(t)|.Source: §G.4.2 · p. 76

G.4.3 THE CONTRAST STEP

  • The one-step change in Φ decomposes exactly into contrast, mean-mode, and perpendicular contributions; the contrast contribution is nonnegative, and the perpendicular contribution is nonnegative when |ζ+| ≤ |ζ|.The latter condition holds for particles with b ≤ 2N.Source: §G.4.3 · p. 77

G.5.1 STATEMENT

  • At tF = ⌊τFκ⌋, the leading-r MF AGOP space equals the teacher space, with eigengap λr/λr+1 ≥ 1 + δ for r ≥ 2, d ≥ 64r, D ≥ 400, and κ ≥ κmin.The theorem gives τF ≤ 0.101 and explicit constants; in Region N, 1 + δ ≥ 1.006223, with stronger margins for r ≥ 3 and r ≥ 10.Source: §G.5.1 · p. 78

G.5.2 SETTING, INPUTS AND CONVENTIONS

  • Along the mean-field trajectory, each particle’s contrast magnitude is nondecreasing and its mean-coordinate increment is positive; the contrast contribution dU also has the stated nonnegative lower bound.The bound is dU ≥ ĥ(kp)Fρkp/√a.Source: §G.5.2 · p. 80
  • The leading AGOP eigenspace has a strict gap exactly when min(λ1, λc) > λ⊥; then its eigenvalue ratio is min(λ1, λc)/λ⊥.Source: §G.5.2 · p. 80

G.5.3 AUXILIARY LEMMAS

  • The certificate’s auxiliary bounds include chi-square lower-tail probability and inverse-moment estimates used to control the good-set conditions.The cited lemma also gives a bound uniform over dimensions D′ ≥ D.Source: §G.5.3 · p. 81

G.5.4 THE CERTIFICATE (PROPOSITION CERT)

  • For κ ≥ A/θ^3 and certificate conditions (S-Ω), (S-A), and (S-C), Proposition 57 guarantees on the good set that the scaled head stays nonincreasing and at least θ, while contrast remains at least its initial value.These guarantees hold through integer times t ≤ τ1κ.Source: §G.5.4 · p. 84
  • The certificate bounds the eigenvalue ratio by λc/λ⊥ ≥ R(τ)^2, using ΓU(t)/Γ⊥(t) ≥ R(τ) ≥ 0.Source: §G.5.4 · p. 87

G.6 EVALUATING THE CERTIFICATE

  • For every (r, D) with D ≥ max(400, 63r), Proposition 57’s hypotheses hold with margin > 1, completing the proof of Theorem 51.The certificate is reduced to finite checks: closed-form bounds for Region A and 80 interval-arithmetic certificates for Region N.Source: §G.6 · p. 88

G.6.1 CELLS, D-INTERVALS AND THE FLOOR

  • At tF = ⌊τFκ⌋, a certificate on [Dlo, Dhi] guarantees λc(tF)/λ⊥(tF) and λ1(tF)/λ⊥(tF) are each at least margin for every D in the interval and κ ≥ κ0.The interval certificate transfers the required slot conditions across D, while the floor κ0 ensures the guarantees persist for all larger κ.Source: §G.6.1 · pp. 90–91

G.6.2 REGION A: THE CLOSED-FORM CERTIFICATE

  • For every r ≥ 81, conditions (R1)–(R3) hold at D = 63r; for 2 ≤ r ≤ 80, they hold at D = DA(r), and these conditions are upward closed in D.The closed-form Region A certificate applies when d ≥ 64r and the rule conditions hold.Source: §G.6.2 · pp. 91–92
  • The Region A certificate bounds its three D-terms by 1.1(r + 1)/D and establishes a positive final bound when D > 14.25M.Its mean-eigenvalue bound exceeds 1.Source: §G.6.2 · p. 94

G.6.3 REGION N: AN EXPLICIT FINITE CERTIFICATE

  • The Region N certificate covers every required integer D interval and verifies both eigenvalue ratios are at least each row’s displayed µ for every κ ≥ κ0.Across all 80 rows, max rhsΩ/Ω is 0.999999903, max rhsA/A is 0.999997516, and max rhsC′/C′ is 0.999990967.Source: §G.6.3 · pp. 96–97

G.6.4 PROOF OF THEOREM 51

  • At tF, the top-r eigenspace of the MF AGOP is uniquely U, Ar(tF) = 1, and λr/λr+1 ≥ 1 + δ.The proof covers D ≥ DA(r) and the finite region max(400, 63r) ≤ D ≤ DA(r), with the required κ thresholds in each case.Source: §G.6.4 · p. 97

G.7 PROOF OF THEOREM 15

The proof establishes the theorem’s loss, alignment, and recovery claims under stated conditions, then supports the certificate ranges with finite endpoint checks and exact inputs.

G.7 PROOF OF THEOREM 15

  • Under the stated hypotheses, loss satisfies (P), the α=α∗(tP) dynamics has t2 ≤ CR(κ + tP) + 1 with CR ≤6.013, and finite-width trajectories converge on [0, tF + 1].The (P) bound is maxs,t≤tP L(t)/L(s) ≤1.01 and L(t) ≥.99V.Source: §G.7 · pp. 98–99
  • At tF, the top-r AGOP eigenspace is uniquely U with a positive gap, and the original trajectory preserves these eigenspaces and eigenvalue ratios.The proof also establishes the initial isotropic AGOP property.Source: §G.7 · p. 98

G.8 FINITE VERIFICATION DETAILS

  • The appendix specifies finite inequalities for verification, with table inequalities checked by integer cross-multiplication and rounded downward.The finite verification details specify the inequalities used in the preceding proof.Source: §G.8 · p. 99, §G.8.1 · p. 100

G.8.1 THE REGION A ENDPOINTS

  • Region-A endpoint claims are supported by rational enclosures and elementary bounds for π, logarithms, and square roots.The endpoint comparison uses these enclosures to verify the stated inequalities.Source: §G.8.1 · pp. 99–100

G.9 COMPLETE CERTIFICATE DATA

  • The 80 Region-N certificates prove both eigenvalue ratios meet their displayed margins at tF for every tabulated D and κ ≥κ0; exact inputs specify the calculation.The margins are conservative lower bounds, and no externally stored parameter values are needed.Source: §G.9 · p. 100, Table 12 · p. 101
Loading 2609.36240v1…