PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

Zihan Zhu, Zhehang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, Weijie Su

arXiv:2609.12123v1cs.LGmath.OCstat.ML

TL;DR

LoRA rank expands an adapter’s representational capacity, but existing rank choices do not determine how much that capacity is used in weight-space updates. The paper introduces Iso-LoRA, which applies spectral descent to the coupled tangent update, and reports improved rank utilization and performance across language-model adaptation experiments.

  • Problem

    LoRA rank specifies adapter capacity, but the optimizer can leave some of the resulting weight-space directions underused.

    Source: §1 · p. 2

  • Method

    Iso-LoRA applies spectral descent to the induced LoRA tangent update and reconstructs compatible factor updates.

    Source: §1 · p. 3, §3 · p. 9

  • Results

    Across 0.1B–7B models, Iso-LoRA improves effective rank and validation or task performance, with strongest gains at moderate-to-large ranks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support optimizer geometry as a factor in how much LoRA rank is realized in induced tangent updates.

    Source: §7 · p. 22

    The theory uses stylized gradient models and does not fully characterize the coupled long-run dynamics of neural-network fine-tuning.

    Source: §7 · p. 22

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate ISO-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. ISO-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that ISO-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B-7B-parameter models, where ISO-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.

1 Introduction

LoRA rank sets representational capacity, but optimizer behavior determines how much of that capacity appears in induced weight-space updates. Iso-LoRA optimizes the coupled tangent update, and the paper studies its theoretical and empirical effects on rank utilization.

1 Introduction

  • LoRA rank expands representable corrections, but factor updates jointly determine the realized weight-space change and can leave available directions underused.The paper frames effective-rank utilization as an optimizer-dependent property, distinct from nominal rank.Source: §1 · p. 2
  • Iso-LoRA applies spectral descent to the induced tangent update and reconstructs compatible factor updates, retaining standard LoRA parameterization.Its objective couples the two factors rather than optimizing them as independent blocks.Source: §1 · pp. 2–3
  • Across 0.1B–7B models, Iso-LoRA improves rank utilization and validation or task performance, especially at moderate-to-large ranks.The paper also reports a stylized first-step analysis in which Iso-LoRA can attain higher stable rank than factor-wise gradient descent and Muon under stated assumptions.Source: §1 · p. 3

1.1 Related work

  • Prior LoRA work largely targets parameter, memory, or expressivity costs; Iso-LoRA instead applies spectral control to the induced tangent update at a fixed parameterization.The related-work discussion distinguishes this target from regularizing factors, full weights, or optimizer states.Source: §1.1 · pp. 4–5

1.2 Notation and Preliminaries on LoRA

  • The paper diagnoses optimizer behavior through the first-order induced weight update, the leading-order approximation to the adapted-weight change.For factors A and B, the update is formed from their one-step increments and the LoRA scale.Source: §1.2 · p. 6

2 Case Study: Optimizer-Dependent Rank Utilization in LoRA

A GPT-2 Small and E2E case study examines how optimizers use a fixed LoRA rank and how that pattern changes as rank increases. AdamW updates are spectrally concentrated, while Muon benefits more consistently from larger ranks.

2 Case Study: Optimizer-Dependent Rank Utilization in LoRA

  • The case study fine-tunes GPT-2 Small on E2E with a frozen backbone, comparing factor-wise AdamW with matrix-structured Muon on the same LoRA factors.It is designed to isolate optimizer-dependent rank utilization, not to provide a broad benchmark.Source: §2 · p. 6
  • At rank 128, AdamW has low stable and entropy rank despite high numerical rank, while Muon uses more available tangent directions.The case study treats numerical rank alone as insufficient to capture how update energy is distributed.Source: §2 · p. 6
  • AdamW is competitive at the smallest ranks, while Muon benefits more consistently as rank increases and becomes stronger at moderate and large ranks.Figure 2 reports the best validation loss from a corresponding hyperparameter sweep for each optimizer and rank.Source: §2 · p. 7, Figure 2 · p. 7
  • The observations motivate Iso-LoRA, which projects the weight-space gradient into the LoRA tangent space, applies spectral descent, and reconstructs factor updates.The design targets the update seen by the adapted network while retaining the low-rank parameterization.Source: §2 · p. 7

3 Iso-LoRA: Method and Factored Implementation

Iso-LoRA chooses the induced LoRA tangent update in weight space, rather than optimizing the two factors independently. It uses a spectrally controlled descent direction, projects it to maintain LoRA feasibility, and reconstructs compatible factor updates.

3 Iso-LoRA: Method and Factored Implementation

  • The surrogate combines a gradient-aligned linear term with a spectral-norm penalty that controls the largest singular direction of the induced update.It is motivated by an isotropic-curvature model and a worst-direction trust region.Source: §3 · p. 8
  • At fixed LoRA factors, feasible first-order weight changes form a tangent space, and only the projected gradient within that space affects linear descent.The tangent projector is orthogonal, and its output has rank at most 2r.Source: §3 · pp. 8–9
  • Iso-LoRA projects the weight-space gradient into the tangent space, takes a negative polar step, then projects again to obtain a feasible descent direction.The polar step equalizes active singular-direction magnitudes; the second projection restores tangent feasibility.Source: §3 · pp. 9–10
  • The projected-polar construction is a practical proxy, not the exact minimizer of the constrained spectral-norm problem.The exact constrained minimizer would require optimizing the spectral norm directly over the tangent subspace.Source: §3 · p. 10
  • Compatible factor increments reconstruct the selected tangent update, while the factored implementation avoids materializing the dense weight-space gradient.The implementation uses a small core of size at most 2r × 2r after QR factorization; its stated cost is O((m + d)r^2 + r^3) time and O((m + d)r) memory per adapted matrix.Source: §3 · pp. 10–11

4 Why Iso-LoRA Improves Rank Utilization: A First-Step Analysis

The first-step analysis explains how optimizers can use the same LoRA rank budget differently under a stylized spiked-gradient model. It predicts higher stable rank for Iso-LoRA than for Muon or factor-wise gradient descent, consistent with the reported rank-scaling pattern.

4 Why Iso-LoRA Improves Rank Utilization: A First-Step Analysis

  • The first-step ordering is explained by different spectral filters: gradient descent can amplify imbalance, Muon orthogonalizes the factor update, and Iso-LoRA orthogonalizes the tangent-projected gradient.The rank-ordering pattern is reported to persist qualitatively throughout training, beyond the analyzed first step.Source: §4 · pp. 11, 12, 14
  • The analysis models the initial gradient as an isotropic bulk plus a low-dimensional dominant subspace, observed by the adapter through its random subspace.Early weight-space gradient spectra are described as concentrated, motivating this spiked-gradient model.Source: §4 · p. 12
  • If the compressed spike's leading direction also controls the bulk contribution, the projected gradient can retain O(1) stable rank as r grows.The passage reports this phenomenon at the first update in GPT-2 fine-tuning.Source: §4 · p. 12
  • On GPT-2 Small + E2E NLG, stable-rank scaling exponents are β ≈1.00 for Iso-LoRA, β ≈ 0.77 for Muon, and β ≈0.04 for AdamW.The figure plots first-step update stable rank against LoRA rank; β is fitted from srank(∆W0) ∝r^β.Source: Figure 4 · p. 13
  • When the initial gradient is spectrally concentrated, factor-wise gradient descent can have stable rank O(1) in r, while Iso-LoRA reaches stable rank r and Muon scales linearly with r in the thin-adapter regime.The bound for gradient descent depends on the initial gradient stable rank; the comparison is under the theorem's assumptions.Source: §4 · p. 13

5 Stable-Rank Dynamics: Gradient Descent vs. Muon

This section analyzes how gradient descent and Muon shape the stable rank of LoRA’s per-step tangent update. Under different stylized assumptions, it bounds gradient-descent updates by ambient gradient complexity and shows Muon can preserve a fixed fraction of nominal rank.

5 Stable-Rank Dynamics: Gradient Descent vs. Muon

  • The section gives a stylized first-step explanation in which Iso-LoRA has the largest update stable rank, Muon an intermediate rank, and AdamW the most concentrated spectrum.This comparison isolates how optimizers distribute update energy under the same nominal LoRA rank, rather than modeling a full training trajectory.Source: §5 · p. 14

5.1 Gradient Descent: Projected-Gradient Control and Low Stable Rank

  • Under an exogenous dynamic-spike gradient model and a small-step condition, gradient descent’s update stable rank is bounded by a constant multiple of ambient gradient stable rank.If ambient gradient stable rank stays bounded independently of r, the update bound can also remain independent of r.Source: §5.1 · pp. 14, 17

5.2 Muon: Isotropic Orthogonalization and Rank Preservation

  • Under conditional isotropic-gradient assumptions, Muon’s tangent update has stable rank bounded below by a fixed fraction of nominal LoRA rank when factor conditioning and interference are controlled.The result is a mechanism theorem, not a claim that real training gradients satisfy the isotropic assumption.Source: §5.2 · pp. 17, 20

6 Experiments

The experiments test Iso-LoRA across rank ablations, model sizes, PEFT baselines, and fixed wall-clock budgets. Results favor Iso-LoRA particularly at moderate-to-large ranks, while gains vary by rank and model.

Rank Ablation on GPT-2 Small

  • From r = 16 onward, Iso-LoRA gives the lowest validation loss, improving over AdamW by 0.039 at r = 128 while achieving the largest stable rank at moderate-to-large ranks.At r ≤4, all methods are capacity-limited; Table 1(b) also reports Iso-LoRA has the highest BLEU and NIST among the LoRA optimizers at r = 128.Source: §6 · p. 21, Figure 6 · p. 21, Table 1 · p. 22

LLaMA-2-7B Comparison to PEFT Baselines

  • At r = 128, Iso-LoRA reaches 61.87 ± 0.64% EM versus 59.20 ± 0.39% for LoRA-Pro on LLaMA-2-7B; its advantage is smaller or absent at lower ranks.It trails LoRA-GA and StelLA at r = 8, and its 0.21-point lead over LoRA-GA at r = 32 is small relative to seed variation.Source: §6 · p. 21, Table 1 · p. 22

Model-size Scaling

  • At r = 16, Iso-LoRA has the highest mean GSM8K EM across three seeds on each evaluated Qwen2.5 and Llama-3.2 model, with a clearest separation on 3B.On 1B, Iso-LoRA is effectively tied with Muon (15.22 ± 1.82% versus 15.13 ± 1.45%); on 3B, the comparison is 42.53 ± 1.13% versus 38.49 ± 0.27%.Source: §6 · p. 21, Table 1 · p. 22

Wall-clock cost

  • After 480 s on GPT-2 and 3,600 s on LLaMA-2-7B, Iso-LoRA achieves the strongest reported downstream results despite higher per-step cost.At the LLaMA-2-7B budget, it reaches 51.71% GSM8K EM versus 49.20% for LoRA-Muon and 48.37% for LoRA-RITE.Source: §6 · p. 21

7 Discussion and Limitations

The discussion argues that optimizer geometry affects how effectively LoRA uses its rank, especially at moderate-to-large ranks. The paper also identifies limits in its theoretical models, implementation cost, and evaluation scope.

Discussion

  • Iso-LoRA targets the tangent-projected weight-space gradient and uses spectral descent to encourage energy distribution across available tangent directions.The discussion connects this design with first-step theory, dynamic mechanism results, and empirical diagnostics, which support the qualitative rank-utilization pattern.Source: §7 · p. 22

Limitations

  • The theory is stylized: it analyzes a first step under a rank-k spiked-gradient model and standard LoRA initialization, using factor-wise gradient descent as an AdamW proxy.It studies the conceptual undamped Iso-LoRA direction rather than the practical optimizer with damping, momentum, and weight decay; dynamic analysis does not characterize coupled long-run neural-network fine-tuning.Source: §7 · p. 22
  • The factored implementation approximates a tangent-gradient surrogate and adds optimizer-side cost O((m + d)r^2 + r^3), which may matter at very large ranks or in latency-sensitive settings.Evaluation remains limited to the reported architectures, datasets, target modules, and hyperparameter grids; the paper names longer horizons, adaptive optimizers, additional LoRA variants, and broader regimes as future directions.Source: §7 · pp. 22–23

A Details on Iso-LoRA

Iso-LoRA implements spectral descent on LoRA’s induced tangent update through factor updates, with an accelerated low-rank version for experiments. Its exact correspondence to the dense formulation depends on damping, rank, and momentum settings.

A Details on Iso-LoRA

  • Algorithm 4 is the practical experimental implementation; its small-core spectral step avoids materializing the rank-2r tangent signal as a dense matrix.It combines low-rank tangent operations, small-core orthogonalization, and factor-update reconstruction.Source: A Details on Iso-LoRA · p. 34
  • Optimizer-side cost is O((m+d)r2+r3) per adapted matrix, with additional memory O((m+d)r) and no dense weight-space buffers or operators.The cost includes rank-2r factor formation, small solves, QR factorizations, and a core SVD of size at most 2r × 2r.Source: A Details on Iso-LoRA · p. 34
  • With β = 0 and ε = 0, full-column-rank factors let Algorithm 4 exactly realize the current-gradient dense tangent step, up to the positive LoRA scale.For ε > 0, it exactly realizes the analogous regularized tangent-map construction and remains well defined at rank-deficient initialization such as B0 = 0.Source: A Details on Iso-LoRA · pp. 33, 35
  • With β > 0, factor-space momentum is a scalable surrogate, not an algebraically identical dense weight-space momentum method.It is closest to dense momentum when the factors change slowly over the effective momentum window.Source: A Details on Iso-LoRA · pp. 34–35

B Proofs for Section 3

  • The Section 3 proofs characterize feasible LoRA tangent updates and show that the constrained spectral objective depends on the gradient only through its tangent component.The appendix divides these results between the tangent-space characterization and the projected-gradient objective equivalence.Source: B Proofs for Section 3 · p. 35

B.1 Proof of Lemma 3.1

The proofs characterize first-order LoRA updates as a tangent space and show that the constrained spectral problem can be expressed using the tangent-projected gradient. The appendix then organizes the first-step analysis supporting Section 4.

B.1 Proof of Lemma 3.1

  • A matrix is a feasible first-order LoRA update exactly when it satisfies the tangent-space projector constraint, and such updates have rank at most 2r.The proof also gives a reconstruction of compatible factor increments.Source: §B.1 · p. 35

B.2 Proof of Proposition 3.2

  • The factor-increment objective is equivalent over induced tangent updates to the projected tangent-space problem, whose gradient dependence is only through PA,B(G).The two formulations have the same feasible set and objective values, so their minimizers correspond.Source: §B.2 · p. 36

C Proofs for Section 4

  • The Section 4 proof develops concentration, projected-gradient, GD, Muon, and Iso-LoRA components before combining them in the ordering corollary.Appendix C.1 proves Theorem 4.2; Appendix C.2 proves Corollary 4.3.Source: C Proofs for Section 4 · p. 36

C.1 Proof of Theorem 4.2

Under the Gaussian initialization and spiked-gradient assumptions, the proof compares first-step stable ranks for GD, Muon, and Iso-LoRA. Iso-LoRA’s polar map gives maximal stable rank for the projected gradient, while Muon’s update inherits the stable rank of the initialization.

C.1 Proof of Theorem 4.2

  • With rank(G0) ≥ r, the Gaussian initialization makes the projected gradient have rank r almost surely.The projected-gradient stable-rank analysis is used in the GD bound; the appendix identifies this rank fact as relevant to both Muon and Iso-LoRA.Source: §C.1 · p. 39
  • At initialization, factor-wise GD induces an update proportional to −A0A0⊤G0, further reweighting the projected gradient by A0’s squared singular values.The proof derives a stable-rank bound using random-subspace concentration under the spiked-gradient model.Source: §C.1 · pp. 40, 42
  • Muon’s first-step update has the same nonzero singular values as A0 up to scale, so its stable rank equals srank(A0).This identity holds almost surely under the stated initialization and rank conditions.Source: §C.1 · pp. 42–44
  • Iso-LoRA’s polar map gives maximal stable rank for any nonzero input; at initialization, the projected gradient has rank r almost surely when rank(G0) ≥ r.The polar map assigns equal nonzero singular values, so its stable rank equals the input rank.Source: §C.1 · pp. 39, 44

C.2 Proof of Corollary 4.3

Under the corollary’s stated separation condition, the three mechanisms from Theorem 4.2 yield a strict stable-rank ordering. The proof also notes that the condition can hold only when r ≥ 2.

C.2 Proof of Corollary 4.3

  • The corollary establishes a strict stable-rank ordering under its stated separation condition.It combines the three optimizer-specific mechanisms established in Theorem 4.2.Source: §C.2 · p. 45
  • The separation condition cannot hold for r = 1; when it is satisfiable, Gaussian A0 has srank(A0) < r almost surely..Source: §C.2 · p. 45

D.1 Proofs for Section 5.1

The proofs control gradient-descent tangent updates under a dynamic spiked-gradient model by bounding projected-gradient signal, factor growth, and subspace drift. Together, these controls support a stable-rank bound for the update.

D.1 Proofs for Section 5.1

  • Projected-gradient control and early-time factor-growth bounds transfer the initial-subspace signal to the evolving factor subspace.The transfer relies on factor drift and a bound on the change in the subspace projector.Source: §D.1 · pp. 46–49
  • A stable-rank perturbation argument completes the proof once the residual is shown to be smaller in operator norm than the dominant tangent term.The scalar rescaling in the GD update does not change stable rank.Source: §D.1 · pp. 46, 50, 51
  • The stable-rank bound follows by controlling the dominant tangent term and bounding the residual contribution from the other term.The proof uses a perturbation bound when the residual has smaller operator norm than the dominant term.Source: §D.1 · pp. 46, 51

D.2 Proofs for Section 5.2

The proofs establish a high-probability stable-rank lower bound for Muon tangent updates under factor-conditioning and interference controls. They use the conditional Haar structure of Muon directions to bound interference between the two factor contributions.

D.2 Proofs for Section 5.2

  • Conditioned on the training history, Muon’s polar update directions are Haar-distributed Stiefel frames under the stated assumptions.This conditional symmetry supports the proof’s probabilistic interference bounds.Source: §D.2 · pp. 51, 54
  • The proof controls factor conditioning and interference across steps, then combines those controls with a deterministic stable-rank bound for the sum of the two tangent terms.The resulting bound applies to the induced update because stable rank is invariant to nonzero scalar multiplication.Source: §D.2 · p. 55
  • Under the theorem’s conditions, the Muon tangent update has stable rank at least a bound proportional to the LoRA rank, reduced by factor conditioning and interference.The bound holds with conditional probability at least 1 − δ.Source: §D.2 · p. 55

E Case Study Experiment Details

This appendix documents the training, optimizer, and diagnostic protocols behind the case study and broader sweeps. Across the supplied results, Iso-LoRA performs best under fixed time budgets and shows gains at moderate-to-large ranks, with outcomes varying by model and setting.

E.1 GPT-2 Small + E2E Case-Study Setup

  • The GPT-2 case study compares AdamW and Muon on E2E NLG to examine how factor-space optimization relates to tangent-update rank and spectral concentration.The setup trains LoRA adapters on c_attn, c_proj, and c_fc; the later rank sweep adds Iso-LoRA.Source: §E.1 · p. 56, §E.4 · p. 56

E.2 Optimizer Details for AdamW and Muon

  • AdamW uses decoupled weight decay, while Muon applies polar-factor updates to each LoRA factor with Nesterov momentum and a Newton-Schulz orthogonalization backend.The optimizer grids are reported in Table 5.Source: §E.2 · p. 56

E.3 Effective-Rank Diagnostic Protocol

  • The diagnostic protocol measures the induced one-step tangent update, with numerical rank counting singular values above 10−6 times the largest singular value.For the GPT-2 case study, diagnostics are recorded every 80 optimizer steps and averaged over adapted modules unless stated otherwise.Source: §E.3 · p. 56

F.2 Hyperparameter Grids

  • The appendix selects hyperparameters by validation loss within each rank and optimizer, and the broader rank ablation extends the sweep to nine ranks while adding Iso-LoRA.The sweeps use independent optimizer grids; the GPT-2 rank-ablation learning-rate cells report best validation loss over momentum and weight decay.Source: §E.4 · p. 56, §F.2 · p. 58, §F.3 · p. 58

F.4 Rank Utilization Across LoRA Ranks

  • The rank-ablation results associate higher stable rank with lower validation loss at moderate and high ranks, while very low-rank cases remain capacity limited.Across 45 GPT-2/E2E runs, stable rank correlates −0.955 and −0.903 with validation loss by Pearson and Spearman, respectively.Source: §F.4 · p. 58

F.5 Model-Size Sweeps

  • At rank 16, Iso-LoRA reaches 40.90±0.48% GSM8K EM on Qwen2.5-0.5B-Instruct and 42.53±1.13% on Llama-3.2-3B.On Llama-3.2-1B, Iso-LoRA and Muon are effectively tied at 15.22 ± 1.82% and 15.13 ± 1.45% EM, respectively.Source: §F.5 · p. 61

F.1 Training and Evaluation Protocols

  • The fixed-time comparison measures r = 128 runs under 480-second GPT-2 and 3,600-second LLaMA-2-7B budgets; Iso-LoRA incurs model-dependent step-time and memory overhead.The appendix specifies single-seed evaluation and stopping at the first completed optimizer step after the budget.Source: §F.6 · pp. 62–63, Table 12 · p. 64

F.6 Runtime and Wall-Clock Curves

  • Iso-LoRA achieves the highest scores on all five E2E metrics within the GPT-2 time budget and reaches 51.71% GSM8K EM on LLaMA-2-7B, above the listed baselines.It completes fewer steps than AdamW but attains lower validation loss on both tasks within the fixed budgets.Source: §F.6 · p. 63, Table 13 · p. 64

F.7 Effective-Rank Diagnostics

The diagnostics distinguish numerical rank from measures of how broadly tangent-update energy is distributed. They report higher stable rank for Iso-LoRA than matched alternatives and show less spectral concentration than AdamW in the GPT-2 diagnostic.

F.7 Effective-Rank Diagnostics

  • At r = 128, Iso-LoRA’s GPT-2 tangent update has stable rank 187.76, compared with 8.26 for StelLA and 5.40 for Stiefel-LoRA.The corresponding r = 32 stable ranks are 52.12, 8.15, and 5.25.Source: §F.7 · p. 66
  • Across three seeds, Iso-LoRA’s joint spectral update yields higher stable ranks on both GPT-2 and LLaMA-2-7B, lower validation loss, and higher downstream scores than LoRA-Muon’s separate spectral subproblems.Source: §F.7 · p. 66
  • The initial dense-gradient stable rank remains small relative to ambient projection dimensions from GPT-2 Small to LLaMA-2-7B, supporting the concentrated initial-gradient condition used in Theorem 4.2.This diagnostic uses a first-backward-pass measurement with maximum sequence length 256.Source: §F.7 · p. 66

F.8 Singular-Concentration Diagnostics

  • In the GPT-2 rank-128 run, AdamW concentrates more update energy in the leading singular direction, whereas Iso-LoRA produces a flatter local spectrum.The figure’s paired views are top-1 energy fraction and final normalized leading singular values.Source: §F.8 · p. 69, Figure 19 · p. 69

F.9 Batch-Size Ablation

  • For batch sizes 32–512, validation-loss ordering is AdamW, Muon, then Iso-LoRA; Muon has the lowest loss at batch size 16.The comparison uses independently grid-searched best configurations at rank 128.Source: §F.9 · p. 69, Table 17 · p. 70
Loading 2609.12123v1…