PaperlayerView exact revision on arXiv

Source-linked AI summary

The Loss Does Not See the Basis, but Adam Does

Devender Singh

arXiv:2608.05136v1cs.LGmath.OCstat.ML

TL;DR

The paper asks why gradient descent favors low-rank interpolants in factored models while Adam does not. It analyzes gauge equivariance and preconditioner geometry, finding that shared-scalar or equivariant updates restore low-rank bias whereas coordinate-wise anisotropy does not.

  • Problem

    Existing evidence shows gradient descent and adaptive optimizers can reach similar training loss yet select interpolants with different generalization and rank properties.

  • Method

    The paper characterizes gauge-equivariant update rules, proves transfer results for shared-scalar flows, and tests nine optimizers plus a preconditioning-anisotropy dial.

  • Results

    43–44% better held-out error: gradient descent outperforms Adam on two hyperspectral benchmarks at equivalent training loss and higher underdetermination.

  • Takeaways & Limitations

    Coordinate-wise anisotropy suppresses low-rank bias, while shared-scalar or equivariant preconditioning restores it across matrix-sensing and hyperspectral experiments.

  • Takeaways & Limitations

    The transfer theorem applies only to the common-scalar subclass, and gauge equivariance alone does not determine the selected interpolant.

Abstract

from arXiv · show

Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under $(U, V) \mapsto (UQ, VQ)$. Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.

1 INTRODUCTION

The paper links implicit low-rank bias in factored models to gauge-equivariance: coordinate-wise methods can break basis symmetry, while equivariant methods can preserve gradient-flow behavior without guaranteeing low-rank recovery. Experiments show that anisotropy and spectral scheduling determine recovery across synthetic, transformer, and hyperspectral settings.

  • Core criterion: Gauge-equivariance is necessary to transfer gradient flow’s low-rank mechanism, but spectral scheduling determines whether an equivariant optimizer actually reaches low-rank solutions.The paper distinguishes optimizer class membership from the singular-value dynamics induced during training.
  • Optimizer classification: Equivariant rules include GD, momentum, shared-scalar Adam, Muon, and Shampoo, whereas Adam, RMSProp, signSGD, and Lion are not equivariant under the factored-model gauge.The distinction concerns compatibility with orthogonal transformations of the latent factor bases.
  • Optimizer comparison: Equivariant optimizers achieve 0.00–0.29 recovery error versus 0.42–0.57 for coordinate-wise methods when all nine rules interpolate on matrix sensing.Matching vanishing training residuals removes residual training error as an explanation for the recovery difference.
  • Mechanism dial: Recovery and effective rank improve monotonically as preconditioning moves from Adam’s coordinate-wise denominator toward a shared scalar, isolating denominator anisotropy as the mechanism.The parameter ranges from p=1 for standard Adam to p=0 for a gauge-equivariant scalar denominator.
  • Applications: 56% separates Adam’s final per-head invariants across gauge-equivalent transformer initializations, while heavy-ball SGD and scalar-Adam remain indistinguishable at floating-point precision initially.On hyperspectral benchmarks, GD also achieves 43–44% better held-out error than Adam at higher underdetermination and lower effective rank.

2 RELATED WORK

Prior work established implicit-bias phenomena for gradient methods and basis dependence in adaptive methods, but left open how deployed optimizers affect low-rank recovery. This paper addresses that gap through gauge symmetry, transfer results, and comparisons across update rules.

  • Implicit bias of gradient methods on factored models: Gradient-flow studies linked infinitesimal initialization in factored models to minimum-nuclear-norm interpolants, extending from symmetric to deep matrix factorizations.The earliest result was proved for commuting measurement matrices, with later work emphasizing a dynamical picture.
  • Contribution relative to prior work: The paper extends prior pathwise conclusions to memoryless common-scalar flows and tests the behavior of optimizers used in practice.These experiments target a question left open by the gradient-descent and gradient-flow literature.
  • Implicit bias of Adam and adaptive methods: Adaptive-method research documented worse generalization and developed margin-based implicit-bias theories for Adam and related methods.This literature includes ℓ∞-norm-constrained characterizations and Karush–Kuhn–Tucker convergence results.
  • Basis dependence and optimizer symmetry: Prior work found rotationally invariant SGD trajectories but basis-dependent Adam trajectories, whereas this paper studies the complementary selection cost in factored low-rank recovery.The paper focuses on the internal gauge of parameter factorizations and compares nine deployed update rules.

3 SETUP: GAUGE, TASK, AND PROTOCOL

The paper studies gauge symmetry and optimizer equivariance in factored models, then compares recovery in underdetermined matrix sensing under controlled interpolation protocols. The setup restricts the symmetry to the maximal isometric subgroup and evaluates methods by recovery alongside effective rank.

  • Gauge: Gauge symmetry preserves UV^T under orthogonal right-rotations, and O(k) is the maximal subgroup preserving the Euclidean product metric.The same product symmetry extends to rectangular inputs because it acts only on the shared latent dimension k.
  • Gauge: Gauge-equivariance requires optimizer states and updates to transform consistently; coordinate-wise nonlinear operations generally fail, while scalarized second moments repair Adam’s obstruction.Adam’s entrywise second moment cannot commute with right-rotation, whereas equivariant trajectories produce gauge-invariant product matrices.
  • Task: Matrix sensing uses n=40, rank r*=3 planted targets, m = 2dof Gaussian measurements, k=n, initialization 10^-3, and no weight decay.The squared measurement loss is optimized in an underdetermined setting with infinitely many fitting matrices.
  • Task: All methods are run to interpolation below loss 10^-7, but Muon, Lion, RMSProp, and signum require dynamics-specific learning-rate decay.Recovery is the primary metric because effective rank can be low even when the recovered matrix is far from the planted target.
  • Protocol: The protocol uses symmetric scheduling and full learning-rate sweeps, while controlling weight decay, interpolation, random seeds, and deterministic full-batch updates.Constant-update-norm methods are compared with decaying schedules to avoid an advantage caused solely by inability to interpolate at a constant step size.

4 THEORY: EQUIVARIANCE AS A CLASSIFIER OF UPDATE RULES

Gauge equivariance classifies update rules that preserve the factorization’s intrinsic geometry rather than depending on an arbitrary basis. The theory establishes equivariance for gradient-based methods, scalar-preconditioned Adam, and Muon, with Muon remaining exact under real arithmetic and finite-order Newton–Schulz approximations.

  • Equivariance as a classifier: Gauge equivariance identifies update rules that maintain the intrinsic geometry of a factorization instead of depending on an arbitrary basis.Once the orthogonal gauge is fixed, covariance is straightforward to verify, but its implications determine basis-independent geometric behavior.
  • Equivariant update rules: Gradient descent and Polyak/Nesterov momentum are gauge-equivariant for linear combinations of past gradients evaluated at appropriately transformed points.The result covers both standard gradient descent and momentum variants under the stated transformation of evaluation points.
  • Equivariant update rules: Scalar-preconditioned Adam is gauge-equivariant when its per-coordinate second-moment estimate is replaced by one scalar from gauge-invariant quantities.Its denominator is a_t = √ν̂_t + ε, with update direction proportional to m̂_t/a_t.
  • Equivariant update rules: Muon is gauge-equivariant through momentum-buffer updates using the matrix sign, exactly under real arithmetic and for any finite-order Newton–Schulz approximation.The matrix sign comes from the compact SVD, with msign(0) = 0.

4. Shampoo: the step

Shampoo’s full-matrix update remains gauge-equivariant even with damping, while coordinate-wise nonlinearities such as Adam’s break the symmetry immediately. The section characterizes memoryless equivariant rules as Gram-determined left preconditioners and establishes when scalar-preconditioned flows inherit gradient-flow paths and limits.

  • Shampoo: the step: Damping λ > 0 keeps Shampoo’s matrix roots well-defined for rank-deficient early iterates while preserving exact equivariance.The experiments use λ = 1; λ = 0 is interpreted on the active subspace.
  • Gauge equivariance: Nonlinear coordinate-wise maps cannot be gauge-equivariant: under orthogonal transformations, the coordinate function must be linear, φ(x) = cx.This rules out fixed nonlinear first-step coordinate operations, including those used by Adam, RMSProp, signSGD, and Lion.
  • Gauge equivariance: Adam’s gauge defect changes the represented matrix after one step, not merely its internal state.For n = 1 and k = 2, the two products are (1−η/(1+ε))^2 and (1−η/(2^−1/2 +ε))^2, which differ for 0 < η < 2^−1/2 +ε.
  • Structure theorem: Every memoryless gauge-equivariant rule is exactly a left preconditioner determined only by the gauge-invariant Gram matrix GG^⊤.Full column rank is generic, so the theorem leaves only a measure-zero rank-deficient set unconstrained, with the update forced to vanish at G = 0.
  • Transfer theorem: Scalar-preconditioned flows follow the same parameter-space path as gradient flow after time reparameterization and reach the same limit when the transformed time diverges.Time-dependent properties such as convergence rates and hitting times do not transfer; the theorem applies to memoryless common-scalar flows, not Scalar-Adam’s EMA state.

5 THE OPTIMIZER ZOO

The optimizer zoo separates gauge-equivariant methods from coordinate-wise methods on matrix sensing: equivariant methods recover substantially more accurately, while Muon can approach exact low-rank recovery under cosine decay. These differences persist under shared scheduling and initialization controls, indicating that the observed split is not explained by those controls.

  • Interpolation: All nine optimizers reach the 10−7 interpolation threshold, although faster methods finish several orders of magnitude below it; Adam reaches 1.2×10−11 final loss.Muon, signum, Lion, and RMSProp require cosine decay to achieve interpolation under the stated conditions.
  • Recovery separation: All equivariant methods outperform every coordinate-wise method, with recovery error ≤0.286 versus ≥0.42.This leaves a 0.14 gap between the best coordinate-wise method and the worst equivariant method at the stated computational budget.
  • Convex benchmark: Relative to the min-nuclear-norm solution’s recovery error 0.0335, Muon performs better, while GD, scalar-Adam, and Shampoo remain within an order of magnitude.Coordinate-wise approaches are all worse by more than an order of magnitude than the convex benchmark.
  • Muon: Muon achieves recovery error 6.8 × 10−6 and effective rank 2.95 versus the true rank 3 when cosine decay permits interpolation.The result is attributed to Muon’s equal-rate spectral dynamics on exactly low-rank targets.
  • Controls: Under a single shared cosine decay, classification remains 9/9, and extending GD and Adam to the full 2×104-step cosine horizon does not improve their results.The split is also observed at initialization scales 10−3 and 3×10−3, becoming less pronounced at 10−2.

6 THE SAME GAUGE LIVES IN ATTENTION HEADS

Attention heads have the same orthogonal gauge symmetry as factored models, so basis-equivalent initializations should remain equivalent only under gauge-equivariant optimization. Adam violates this: its twins diverge immediately and end with a 56% disagreement in the gauge-invariant W_Q^T W_K product, which no per-head rotation can remove.

  • Gauge symmetry: Attention logits are invariant under the per-head orthogonal transformation (W_Q,W_K) → (A_hW_Q,A_hW_K), with W_Q^T W_K playing the role of UV^T.The paper restricts attention to the orthogonal subgroup of the larger invertible function-preserving symmetry.
  • Twin experiment: 3.6 × 10^-3 versus 2.9 × 10^-7: after one step, Adam’s gauge twins show a four-order-of-magnitude larger logit difference than the noise twin.Adam’s relative logit distance later saturates at 0.77, while the gauge transformation initially preserves the model function exactly.
  • Twin experiment: 56%: Adam’s per-head W_Q^T W_K invariants differ by this relative Frobenius distance, beyond any disagreement attributable to rotations.SGD and scalar-Adam remain at the numerical-noise level at step 1 and stay four to five orders of magnitude below Adam’s structural split thereafter.
  • Twin experiment: Muon’s gauge twin diverges, while its noise twin tracks the same trajectory within 0.05 decades averaged past step 100, indicating schedule-amplified chaos distinct from basis dependence.At step 100, the two Muon twins still differ by 0.24 decades, and both remain at the noise floor for about 50 steps.
  • Model merging: Adam’s 56% invariant disagreement means no per-head rotation can merge the runs, whereas shallow deterministic equivariant twins agree to floating-point error and are mergeable by one Procrustes step.At greater depth, equivariant twins can also separate, but only as far as their own noise twins do.

7 ISOLATING PRECONDITIONER ANISOTROPY WITH A DIAL

The Adam-p dial isolates preconditioner anisotropy: moving from coordinate-wise Adam (p=1) toward shared-scalar Adam (p=0) monotonically restores parameter recovery and lower effective rank. This identifies anisotropy as the cause of eroded factored-model bias, while retaining Adam’s speed advantage only at the cost of more interpolation steps.

  • Dial construction: Adam-p varies only the denominator from coordinate-wise preconditioning at p=1 to a gauge-equivariant shared scalar at p=0.Intermediate p values reduce anisotropy while preserving Adam’s adaptation, momentum, and noise properties.
  • Empirical effect: Parameter recovery and effective rank improve monotonically as p decreases, without changing the algorithm beyond the dial value.Both fixed-step and envelope experiments show the same monotone restoration, ruling out per-p learning-rate selection as the hidden cause.
  • Empirical effect: 0.576 to 0.201: parameter recovery improves as p decreases at learning rate 10−3, with every run meeting the 10−7 interpolation threshold.The corresponding fixed-step improvements are 0.570 to 0.256 at 3 × 10−3, 0.573 to 0.357 at 10−2, and 0.581 to 0.493 at 3×10−2.
  • Interpretation: The dial shows that coordinate-wise anisotropy directly explains erosion of factored-model bias, while also providing Adam’s speed advantage.The speed advantage has a cost: interpolation takes roughly eight times as many steps, increasing from 400 to 3200 on this task when comparing selected endpoints.
  • Limitation: At learning rate 10−1, recovery remains 0.55–0.62 and effective rank ≈13.8–15.3 for all p because the step is too coarse to resolve the flow limit.Thus, the restoration depth depends on rate, whereas monotonicity holds at rates that resolve the flow limit.

8 INSIDE THE EQUIVARIANT CLASS: THE SPECTRAL-TAIL PHASE DIAGRAM

Within the gauge-equivariant class, spectral schedule aggressiveness determines a trade-off between exact low-rank recovery and robustness to spectral tails. Muon’s equal-rate updates are exact for rank-3 targets but give way to GD as tail energy grows, with the boundary shaped by target, measurement, initialization, training, schedule, and algorithmic factors.

  • Spectral-tail phase diagram: Muon is exact at τ=0, but degrades fastest as tail energy grows and crosses over to GD near τ*=0.2, corresponding to approximately 4% tail energy.The tail carries τ^2 of the target’s energy.
  • Spectral-tail phase diagram: Adam has the highest recovery error at every τ, while more aggressive equivariant schedules favor accurate low-rank recovery and moderate schedules favor tail robustness.The schedule ordering is Muon, then Shampoo, then GD.
  • Schedule control: Cosine-decayed schedules preserve the performance curves and crossings of constant schedules; the crossover is τ*=0.35 on the 3-seed grid and approximately 0.2 on the 10-seed grid.This control shows that the preference boundary does not depend on the decay-schedule type.
  • Analytic mechanism: In the solvable two-timescale model, greedy updates amplify the timescale gap as initialization shrinks, whereas equal-rate updates fit all modes by the leading-mode threshold regardless of initialization scale.Greedy dynamics follow logistic growth; under equal-rate dynamics, √w_i increases linearly and saturates at s_i.
  • Interpretation: The observed transition may combine target spectral properties, measurement design, initialization scale, training length, schedule, and optimization algorithm.The analytical model connects uniform low-rank fitting under equal-rate schedules with the empirical phase behavior.

9 REAL DATA AT MATCHED TRAINING LOSS

At matched training loss on two hyperspectral image-completion benchmarks, gradient descent achieves lower held-out error and lower effective rank than Adam, especially at lower sampling density. The advantage narrows as sampling density increases, while Muon remains near full effective rank and does not outperform gradient descent.

  • Indian Pines: 44% lower held-out RMSE: GD reaches 0.0150±0.0001 at effective rank 11, versus Adam’s 0.0268±0.0005 at rank 28.The comparison uses Indian Pines at m/dof ≈1.15 and matched training loss ≤3×10^-5, consistently across all four seeds.
  • Learning trajectories: Adam’s held-out error rises from 0.0251 to 0.0268 during interpolation as effective rank increases from 22 to 28.GD’s test error instead decreases steadily, indicating that deeper fitting can degrade generalization for an optimizer that breaks gauge symmetry.
  • Sampling density: GD’s advantage narrows from +44% at m/dof = 1.15 to +28% at m/dof = 1.9, consistently across all four seeds.The gap is attributed to denser observations making the solution better constrained and reducing interpolant-selection effects.
  • Second benchmark: 43.8% improvement: Pavia University shows the same GD-over-Adam trend at m/dof ≈1.15, declining to 22.9% at m/dof ≈1.9.Results are consistent across all four seeds, and Muon does not outperform GD on either dataset or density.
  • Muon: Muon’s effective rank stays near 46–47 of capacity 48 for most of the matched-loss path, declining only near final levels.This behavior suggests the schedule dimension, rather than gauge equivariance alone, determines performance in these real-data experiments.

10 REPAIRING OUR OWN OPTIMIZER USING THE CRITERION

The section tests repairing Adam with flow-based clipping: FlowAdam narrows but does not initially close GD’s recovery gap, while whole-velocity clipping can match GD in the continuum limit. It also identifies limits of implicit bias, including weak tunability and challenges involving persistent forces or highly uneven gradient magnitudes.

  • FlowAdam: Recovery error falls from 0.573 to 0.347 with FlowAdam, versus 0.131 for GD without weight decay.FlowAdam partially closes the performance gap but remains above GD.
  • FlowAdam: Restricting the whole velocity-vector norm, rather than individual coordinates, preserves direction and lets FlowAdam match GD’s bias as c →∞.The corrected clipping recovers at 0.113 in the continuum limit.
  • FlowAdam: Effective rank reaches 4.8–5.4 for FlowAdam across seeds, closest among Adam-type algorithms to GD’s 4.51.This comparison concerns the considered Adam-type algorithms.
  • FlowAdam: +15.9% improvement occurs at learning rate 10^-3, but finer grids make the dial and FlowAdam-p empirically converge.The dial reaches 0.143 (n=40) / 0.40 (n=128), while FlowAdam-p reaches 0.148 / 0.42.
  • Limitations: Implicit bias has no separate adjustable strength parameter and clearly benefits over tuned weight decay only when optimal regularization is low and structured.The limitation is reported across varying observation density, signal conditioning, and explicit regularization.
  • Limitations: Persistent forces such as weight decay may be needed for grokking, while PINNs pose challenges when gradient magnitudes differ vastly across coordinates.These limitations follow from the procedure’s theoretical properties rather than empirical observation.

11 DISCUSSION AND LIMITATIONS

The discussion limits the claims by emphasizing modest original scale, untuned exceptions, and the distinction between interpolant selection and optimization speed. It also identifies open theory for momentum and stochasticity, while documenting reproducibility resources and conditions.

  • Scale: Original experiments used matrices from 40×40 to 2000×200 and transformers of up to six layers, prioritizing determinism and interpolation discipline over raw scale.A replication varied problem size from n=64 to n=256 across ten different random seeds on H100 GPUs.
  • Two exceptions we left untuned: Untuned Shampoo with damping (λ=1) changes relative recovery as n grows: it starts below Adam at n=40, overtakes it at n=64, then dips below again at n=96.The passage attributes this nonmonotonic behavior to retaining originally published hyperparameters.
  • Not a prescription for practice: The results do not prescribe GD over Adam: fair comparisons with equal tuning budgets have not shown any optimizer consistently outperforming well-tuned Adam.The comparison concerns which interpolant is selected, not how quickly the training objective is optimized.
  • Open theoretical directions: Equivariance enables analysis of the GD family but does not determine the interpolant, while the transfer theorem currently applies only to common-scalar methods.The discussion also notes that non-equivariant annealed sign descent shows equivariance is not strictly necessary.
  • Open theoretical directions: A momentum-invariance lemma could establish exact flow-equivalence for scalar-Adam and extend the transfer theorem to momentum-augmented methods.The proposed route uses the linear, gauge-respecting first-moment EMA.
  • Stochasticity: Mini-batch noise introduces a diffusion term whose interaction with gauge symmetry and Muon’s msign-induced chaos remains empirically probed but theoretically uncharacterized.The experiments include character-language-model twins and the noise-twin ladder; quotient geometry is suggested as a framework.

USE OF LARGE LANGUAGE MODELS

Large language models were used only for copy-editing; the authors produced and verified all substantive content.

  • USE OF LARGE LANGUAGE MODELS: LLMs supported copy-editing only, while the authors developed and checked the questions, proofs, experiments, and numbers.Verification used the derivations and experiment logs.

A INSIDE THE EQUIVARIANT CLASS … B.8 COROLLARY A.3 (DEEPER FACTORIZATIONS)

The appendix characterizes gauge-equivariant updates through Gram-determined preconditioners, spectral transfer functions, and interface-wise covariance, while identifying limitations near rank deficiency and for coordinate-wise rules. It also distinguishes equivariance from balancedness and extends the results to deeper factorizations.

  • A INSIDE THE EQUIVARIANT CLASS: Spectral transfer scales each singular direction by h(σi); GD uses h(σ)=ησ, while common-scalar preconditioning uses ht(σ)=ησ/at.The transfer function is defined through H=ψ(GG⊤) and h(σ)=ψ(σ²)σ.
  • A INSIDE THE EQUIVARIANT CLASS: The aggressiveness ratio h(σ)/h(σmax) does not fully order practical optimizers: empirical flattening follows GD < Shampoo < Muon despite identical one-step idealizations for the polar map and undamped Shampoo.Accumulation and damping make Shampoo flatten singular values less aggressively than Muon.
  • B.3 PROPOSITION 4.2 (EQUIVARIANT CLASS): GD, momentum, scalar-preconditioned Adam, Muon, and finite Newton–Schulz updates preserve gauge equivariance, with the practical finite Newton–Schulz algorithm exact in real arithmetic.The stateful construction uses covariant quantities, gauge-invariant states, and spectral functions; finite-precision sensitivity is separate.
  • A INSIDE THE EQUIVARIANT CLASS: Balancedness is conserved by gradient flow and by diagonal flows sharing one positive scalar, but not generally by mismatched preconditioners; it is nevertheless unnecessary for good low-rank recovery.Muon and Shampoo do not conserve Bt yet can achieve large final recovery performance.
  • B.4 PROPOSITION 4.3 (COORDINATE-WISE RULES BREAK THE GAUGE): Any fixed nonlinear coordinate-wise update fails gauge equivariance for internal dimension at least 2, while Adafactor also fails through its coordinate-dependent factored second moment.The failure is established using rotations for entrywise maps and a direct one-step test for Adafactor.
  • B.5 PROPOSITION 4.4 (EXACT FIRST-STEP ADAM DEFECT): Adam’s first update is exactly −ηDε(G), independent of β1 and β2, and gauge-aligned rotated and reference iterates can therefore differ immediately.The witness uses GU=GV=(1,0) versus GUQ=GVQ=(2^-1/2,2^-1/2).
  • B.6 THEOREM 4.5 (STRUCTURE): Equivariant memoryless rules have the form Φ(G)=H(GG⊤)G, so their updates are determined by the gradient Gram matrix and covariant on the right.The structure theorem derives Gram dependence using gauge orbits and the Moore–Penrose pseudoinverse.
  • B.7 PROPOSITION A.1 (SPECTRAL TRANSFER): Near rank deficiency, the Lipschitz quotient is at least |h(δ)|/δ and diverges as δ→0 when h(0+)>0, whereas GD has quotient η.This follows from comparing G+=diag(1,δ) and G−=diag(1,−δ).

B.9 THEOREM 4.6 (TRANSFER THEOREM) · B.10 PROPOSITION A.4 (BALANCEDNESS) · B.11 PROPOSITION 8.1 (GREEDY VS. EQUAL-RATE PHASE BOUNDARY)

The transfer theorem shows that common-scalar preconditioning follows the same gradient-flow path up to a gauge-independent time change, while balancedness is conserved and equal-rate dynamics create a distinct greedy phase boundary.

  • B.9 THEOREM 4.6 (TRANSFER THEOREM): A positive, gauge-invariant scalar preconditioner reparameterizes time without changing the traversed gradient-flow path, transferring every path property on the traversed portion.If the reparameterized horizon is infinite, the gradient-flow limit and its limit-point characterizations transfer; otherwise only the finite-time prefix is shared.
  • B.9 THEOREM 4.6 (TRANSFER THEOREM): The time reparameterization is gauge-independent, so the transfer theorem holds uniformly across the entire gauge orbit.This follows because the scalar is computed from gauge-invariant trajectory statistics.
  • B.10 PROPOSITION A.4 (BALANCEDNESS): Gradient flow conserves balancedness: the difference U^⊤U − V^⊤V remains constant, and common-scalar time reparameterization preserves that conservation.The two Gram-matrix derivatives are equal, so their difference has zero derivative.
  • B.10 PROPOSITION A.4 (BALANCEDNESS): As diagonal factor preconditioners converge to the same scalar, their balancedness drift vanishes when factor and gradient norms remain bounded.The common-scalar components cancel, leaving only a remainder controlled by entrywise maximum and Frobenius bounds.
  • B.11 PROPOSITION 8.1 (GREEDY VS. EQUAL-RATE PHASE BOUNDARY): In each decoupled mode, the balanced two-factor model reduces consistently to a single variable w, with the tied model differing only by a uniform time-rescaling.The balanced set u = v is invariant under both flows.
  • B.11 PROPOSITION 8.1 (GREEDY VS. EQUAL-RATE PHASE BOUNDARY): Gradient flow follows logistic dynamics, whereas equal-rate sign flow grows √w linearly and stops at the target threshold, producing stopped, greedy mode fitting.The equal-rate trajectory satisfies √w = √w0 + ηt until w reaches s.
  • B.11 PROPOSITION 8.1 (GREEDY VS. EQUAL-RATE PHASE BOUNDARY): When δ < 1 − s2/s1, the second mode reaches s2 exactly before the head reaches (1−δ)s1, independently of w0.The condition implies t2 < T1, so w2(T1) = s2 exactly.

C EXPERIMENTAL DETAILS … D.4 INITIALIZATION SCALE

The experiments define controlled sensing, transformer, hyperspectral, and FlowAdam protocols, then test whether schedules, learning rates, and initialization alter optimizer rankings and implicit-bias conclusions. Across controls, equivariant methods retain their recovery ordering, while Adam remains bias-distorted and Muon’s behavior depends on schedule and initialization.

  • C EXPERIMENTAL DETAILS: Sensing uses rank-3 Gaussian targets with 462 measurements, 40×40 factors initialized at 10^-6, 20,000 steps, interpolation threshold 10^-7, and three paired seeds.The sensing loss is the mean squared measurement error, with a 10^-3 initialization scale in factor standard deviation.
  • C EXPERIMENTAL DETAILS: Transformer experiments use modular addition, a 2-layer four-head RMSNorm model, full-batch training, and exact query–key gauge symmetry without biases or qk-norm.Drift is measured using symmetrized relative Frobenius distances for validation logits and per-head W_Q^T W_K invariants.
  • C EXPERIMENTAL DETAILS: Hyperspectral tests use 2,000 pixels from Indian Pines and Pavia University, rank-48 factorization, 10^-2 initialization, no weight decay, and training-loss-matched checkpoints.Rank 24 defines underdetermination while rank 48 over-parameterization leaves implicit bias to select the solution; four seeds are aggregated under train-only selection.
  • D THE CONTROL BATTERY: Control re-evaluations show no meaningful changes beyond noise, while additional seeds make only the phase boundary τ* more precise and strengthen conclusions about capability and RMSProp.The controls support the stability of the paper’s principal findings rather than changing their interpretation.
  • D.1 SCHEDULE SYMMETRIZATION: 9/9 methods retain their classification under a uniform cosine schedule, while Muon shifts by no more than 0.02 and Adam remains worst across all six τ values.Muon does not interpolate at constant learning rate, requiring decay; increasing seeds moves the crossover from τ*≈0.35 to τ*≈0.2.
  • D.2 THE LEARNING-RATE AXIS: 0.56 to 0.58 is Adam’s recovery range across five interpolating learning rates, whereas GD improves from 0.131 to 0.113 and scalar-Adam from 0.201 to 0.165 at smaller rates.These results motivate reporting capability through learning-rate curves rather than relying on outcome-based or fastest-interpolation selection.
  • D.3 LEARNING-RATE CURVES AND SELECTION RULES: 9/9 classification is achieved by best-recovery selection under cosine schedules, versus 8/9 under fastest-interpolation selection, with Muon the sole failure at recovery 0.91.The zoo’s selected rates are GD 0.131 at 0.01, Adam 0.573 at 0.01, and scalar-Adam 0.201 at 0.001.
  • D.4 INITIALIZATION SCALE: 10^-2 initialization removes the small-init bias for every method except Muon, while the recovery split holds at 10^-3 and 3×10^-3.Muon remains exact at the largest tested initialization scale, indicating schedule-driven rather than initialization-driven bias.

D.5 ATTENTION TWINS ACROSS SEEDS, INITIALIZATION DRAWS, AND NOISE LEVELS … F RELATION TO INVARIANT-OPTIMIZER DESIGN

Across varied seeds, scales, schedules, and optimizer designs, the experiments show that gauge-equivariance suppresses basis-dependent drift, while recovery also depends on interpolation, spectral response, and training horizon. Related work and the paper’s optimizer-design discussion frame invariant preconditioning as a practical way to control which interpolant is learned.

  • D.5 ATTENTION TWINS ACROSS SEEDS, INITIALIZATION DRAWS, AND NOISE LEVELS: Adam’s gauge-twin drift reaches 0.69–0.80, while equivariant twins remain near machine precision except at the largest tested scale.Across depths and widths, Adam’s initial deviation is about 10^-3–10^-2, whereas equivariant methods sit 12–13 orders of magnitude lower; at 6L/d=256, scalar-Adam reaches 1.8 and Muon 0.58.
  • D.6 HYPERSPECTRAL LEARNING-RATE HYGIENE: 44% lower held-out error is achieved by GD than Adam at deeper matched-loss levels, while GD’s trajectory is effectively independent of its tested learning rate.GD(lr=3) already outperforms Adam by 12% at train loss ≤3×10^-4, and the gap widens to 44% with the higher learning rate; the matched-loss path varies mainly by progress along it.
  • D.7 INTERPOLATION-BAR AUDITS: RMSProp only reaches interpolation under cosine decay, while signum’s apparent recovery advantage disappears when evaluated after successful interpolation.RMSProp reaches training loss 6.0×10^-8 and recovery score 0.5266 under cosine decay; signum changes from 0.009 at a 5×10^-2 plateau to 0.445 under decay and interpolation.
  • D.8 FLOWADAM-P IS NOT AN ARTIFACT OF EARLY STOPPING: 0.1691 mean recovery at the 10^-7 threshold matches FlowAdam-p=0’s original result and improves upon the dial-only baseline by +15.9%.Its effective rank is 4.8–5.4, closest among Adam variants to gradient descent’s 4.51, and recovery stays relatively flat across matched-loss thresholds.
  • D.9 THE H100 REPLICATION LADDER: 1.7–4.3× GD’s error is observed for Adam-type methods at every tested size, while Muon’s error remains <10^-5 for all seeds up to n=256.The ranking is consistent across problem sizes under interpolating schedules; GD and scalar-Adam improve further with smaller step sizes toward their flow-limit values.
  • E EXTENDED RELATED WORK: Related work likewise identifies rotation or group-equivariance violations in coordinate-wise optimizers and develops invariant or deliberately non-invariant alternatives across geometry, networks, and structured optimization.These studies connect shared-scalar, polar, spectral, and other structured updates to symmetry-aware optimizer design and implicit-bias effects.
  • F RELATION TO INVARIANT-OPTIMIZER DESIGN: Invariant preconditioners are presented as practical tools because basis-dependence changes what is learned, with p=0, clipping, and full matrix preconditioners spanning increasingly aggressive repairs.The design axis runs from intermediate p values that temper broken symmetry to Shampoo, Muon, and RITE-style matrix preconditioners that restore stronger invariance.
Loading 2608.05136v1…