PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking LoRA Initialization for Robust Asymmetric Learning Rates

Disen Liao, Yaoliang Yu

arXiv:2610.32142v1cs.LGcs.AI

TL;DR

LoRA learning-rate selection is brittle: asymmetric rates can improve adaptation, but standard initialization introduces noise that makes large ratios unstable. The paper analyzes this signal–noise interaction and proposes Init[A2], which scales A’s variance as Θ(n^-2) while keeping B0 = 0. Init[A2] improves stability across models and tasks, reducing best-ratio drift and improving robustness to large update multipliers.

  • Problem

    LoRA learning-rate selection remains brittle, as large asymmetric ratios can destabilize standard initialization despite their potential to improve adaptation.

    Source: §1 · pp. 1–2

  • Method

    Init[A2] scales A’s variance as Θ(n^-2) while keeping B0 = 0, suppressing initialization-dependent noise identified through a signal–noise decomposition.

    Source: §1 · p. 2

  • Results

    Init[A2] reduces best-ratio drift and improves robustness to large update multipliers across controlled transfer, RoBERTa fine-tuning, and decoder-only generation benchmarks.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Init[A2] makes high-ratio asymmetric learning rates more stable and reduces reliance on exhaustive hyperparameter sweeps, without making ρ = n a guaranteed finite-model optimum.

    Source: §5 · p. 10

    Most LR-ratio sweeps are single runs, and the evaluation focuses on supervised LLM fine-tuning without a direct ratio-setting rule for arbitrary models and tasks.

    Source: §5 · p. 10

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) is often highly sensitive to learning-rate (LR) choices. Recent empirical studies suggest that, once LRs are carefully tuned, vanilla LoRA can remain competitive with many proposed LoRA variants. This shifts the central practical question from designing ever more variants to reducing the cost and brittleness of LR selection. LoRA+ addresses this issue with asymmetric LRs, using a larger LR for the up-projection matrix $B$ than for the down-projection matrix $A$. However, the ratio $\eta_B / \eta_A$ itself becomes an additional hyperparameter: large ratios can improve adaptation, but under the standard LoRA initialization they can also cause unstable training and ratio-dependent performance collapse. We identify the source of this brittleness through a signal--noise decomposition of LoRA dynamics. Under standard \initA, the update contains an initialization-induced noise term whose magnitude can be affected by the LR for $B$, $\eta_B$, the initialization scale $\beta$, and the adapter rank $r$. In practical LLM fine-tuning, this non-vanishing noise can limit the performance gains expected from LoRA+ style asymmetric LRs. To suppress this effect, we propose \initAA, a simple initialization that scales the variance of $A$ as $\sigma_A^2 \propto n^{-2}$ while keeping $B_0=0$. Under matched rank and training time for both initializations, this reduces the initialization-noise energy by an additional factor of $1/n$, stabilizing asymmetric LoRA training while preserving zero initial adapter output. Empirically, \initAA substantially widens the stable LR region across initialization scales, adapter ranks, and LoRA update multipliers. Across tasks and models, \initAA reduces best-ratio drift, prevents the high-ratio collapse observed with standard \initA, and makes the width-based guideline $\eta_B \approx n \eta_A$ a useful starting point for finite pretrained models.

1 Introduction

The introduction frames LoRA learning-rate selection as a practical challenge: asymmetric rates can aid adaptation, but standard initialization makes large ratios brittle. It proposes Init[A2] to suppress initialization-dependent noise and reports greater stability across scales, ranks, multipliers, models, and tasks.

1 Introduction

  • LoRA rank r affects adapter capacity, while α/r can shrink activations and gradients as rank grows, motivating α/√r as an alternative [5] [6].The introduction also describes LR choice as an additional source of brittleness.Source: §1 · p. 1
  • After sufficiently broad LR sweeps, vanilla LoRA can remain competitive with proposed variants, shifting attention toward understanding hyperparameter selection [7].LoRA+ motivates ηB/ηA ≈ n under idealized scaling, but large ratios can destabilize finite pretrained models under Init[A] [8].Source: §1 · p. 2
  • Under Init[A], ηB amplifies both task-driven signal and noise from learned B interacting with random A0, making large ratios, β, and α/r brittle.In the small-LR regime common in pretrained LLM fine-tuning, the noise contribution does not vanish with width.Source: §1 · p. 2
  • Init[A2] sets σ_A^2 = Θ(n^-2) while keeping B0 = 0, reducing initialization-noise energy by 1/n at comparable rank and optimizer-update time.It preserves the LoRA architecture, objective, and zero initial adapter output.Source: §1 · p. 2
  • Across toy, RoBERTa, and decoder-only fine-tuning experiments, Init[A2] widens the stable LR region and reduces best-ratio drift.It also keeps feature norms stable as β grows.Source: §1 · p. 2

2 Background

The background defines LoRA updates and features, then reviews scaling theory’s stability and efficiency criteria and prior initialization and learning-rate analyses.

2.1 LoRA Parameterization and Features

  • LoRA parameterizes a frozen weight update as ΔW = α/r BA, with trainable B ∈ R^n×r and A ∈ R^r×n and rank r ≪ n.LoRA features are ZA = AZ and ZB = (α/r)BAZ.Source: §2.1 · p. 3

2.2 Prior Work on LoRA with Scaling Theory

  • Prior scaling analyses use feature stability and update efficiency to motivate asymmetric LRs; related work compares Init[A], Init[B], and nonzero Init[AB] under different settings [8].The scaling-theory perspective seeks feature learning rather than the lazy-kernel regime.Source: §2.2 · pp. 3–4

3 Analysis

The analysis traces asymmetric-LR brittleness to initialization-dependent noise in LoRA features and proposes Init[A2] to suppress it while preserving feature learning. Its conclusions depend on stated scaling and alignment assumptions, and it contrasts the internal-feature stability of Init[A2] with Init[AB].

3 Analysis

  • The authors report that standard Init[A] can produce initialization-induced noise that accumulates over training, while proposed Init[A2] suppresses this source of instability.Source: §3 · p. 4

3.1 Update Decomposition

  • The decomposition treats optimizer-produced matrix directions separately from LoRA-specific learning-rate scales; the algebraic identities apply to realized directions, while width-scaling estimates need optimizer-specific assumptions.The analysis assumes alignment between the A-update and input, but that condition is not asserted generically for minibatch fine-tuning.Source: §3.1 · pp. 4–5
  • The feature and its one-step update decompose into initialization noise and optimizer-driven signal; both scale with ηB, so increasing it amplifies useful adaptation and noise.The cross term between the components is signed and cannot generally be discarded because optimizer directions depend on initialization.Source: §3.1 · p. 5

3.2 Robustness Analysis

  • The robustness analysis assumes a sub-√n training horizon; at fixed rank, β, and ηB, the standard-initialization noise does not vanish with width.The noise carries a rank factor that depends on whether contraction is LLN-like or CLT-like.Source: §3.2 · pp. 5–6
  • Under Init[A2], the noise is subdominant and the total feature norm closely follows the signal norm in both Llama models shown in Figure 2.The authors state that the additional 1/n noise-energy factor follows under matched projection, contraction, and temporal-accumulation scales.Source: Figure 2 · p. 6, §3.2 · p. 6
  • Under the stated matched scaling, Init[A2] adds a factor of 1/n suppression to noise energy while preserving the signal's scaling order; it does not guarantee a 1/n reduction in total energy.The finite-width experiments test dominance and stability, not the asymptotic suppression rate; the robustness argument also requires control of signal energy and a finite-time horizon.Source: §3.2 · pp. 5–7

3.3 Init[A2] Still Enables Feature Learning

  • Under fixed-rank assumptions, Init[A2] suppresses initialization-dependent terms to o(1) while preserving Θ(1) feature updates; standard Init[A] needs a non-cancellation condition for a non-degenerate lower bound.Under the stated conditions, Init[A2] satisfies the stability and efficiency conditions; standard Init[A] guarantees bounded updates but can permit cancellation.Source: §3.3 · p. 7

3.4 Internal Stability and Bottleneck Dynamics

  • Init[AB]'s symmetric ηA = Θ(n^-1/2) scaling yields Θ(√n) internal-feature updates, whereas Init[A] and Init[A2] use ηA = Θ(n^-1) and keep them Θ(1).The paper reports that Init[AB] degrades in most tested configurations and at large β.Source: §3.4 · p. 7

4 Experiments

The experiments test hyperparameter transfer and robustness on a toy MLP and fine-tuning tasks. Init[A2] maintains broader stable LR regions than standard Init[A], and its reduced noise accompanies stable performance at larger update multipliers.

4 Experiments

  • The evaluation varies β, LR ratio, rank, and α/r across a toy MLP, RoBERTa on GLUE, and decoder-only Llama arithmetic and commonsense benchmarks.The Llama generation experiments fix ηA = 10^-5 and set ηB = ρηA.Source: §4 · p. 8, §4.2 · p. 8

4.1 Toy Example

  • At β = 5.0, Init[A]'s optimal LR drifts at width n = 4096 and convergence degrades, whereas Init[A2] maintains transfer and lower validation loss.At β = 1.0, both methods show consistent optimal LRs across widths; Init[A2] also preserves ZA stability as β grows.Source: §4.1 · p. 8, Figure 3 · p. 8

4.2 LLM Experiments

  • Init[A2] maintains broad high-accuracy regions across LR ratios, while standard Init[A] drifts toward smaller ratios and collapses at high ratios as β or rank increases.The reported qualitative pattern also holds for RoBERTa and additional Llama, Qwen, and Mistral backbones.Source: §4.2 · p. 8, Figure 4 · p. 9

4.3 Init[A2] Enables Multiplier Tuning

  • Init[A2] reaches 77.5% at α/r = 10 and 77.3% at α/r = 12, while Init[A] peaks at 76.4% at α/r = 3 and drops to 72.0% at α/r = 12.These results are from Llama 3-8B arithmetic fine-tuning under the stated β and learning rates.Source: §4.3 · p. 10

5 Conclusion and Limitations

Init[A2] remains stable at larger LoRA multipliers where Init[A] degrades, and the conclusion attributes its robustness to reduced initialization-induced noise. The authors also qualify the evidence: structured-initializer extensions are empirical, and most LR-ratio sweeps are single runs.

5 Conclusion and Limitations

  • Figure 6 reports that Init[A2] remains stable as the LoRA multiplier α/r increases to larger values, whereas Init[A] degrades beyond α/r = 3.Source: Figure 6 · p. 10
  • Init[A2] reduces the random-initial-projection noise amplified as B moves, making high-ratio asymmetric LRs less brittle while preserving zero initial adapter output.The authors report reduced best-ratio drift and robustness to large update multipliers across their controlled, encoder-only, and decoder-only evaluations.Source: §5 · p. 10, Figure 6 · p. 10
  • Variance-compressed structured initializers remain stable in selected Llama arithmetic sweeps, but the B0 = 0 theorem does not directly cover PiSSA or establish uniform gains over tuned base initializers.The extension is empirical.Source: §5 · p. 10
  • Most LR-ratio sweeps are single runs, evaluation focuses on supervised LLM fine-tuning, and no direct ηB/ηA rule is provided for arbitrary models and tasks.The authors identify broader fine-tuning settings and model-aware ratio prescriptions as future work.Source: §5 · p. 10

C. Experimental Setup

The supplied appendix material names experimental setups and initializer comparisons, then introduces LLN and CLT scaling and γ-calculus for width-dependent scaling. The latter framework tracks polynomial scaling exponents and their arithmetic.

C. Experimental Setup; E. Comparisons with Other LoRA Initializations

  • Appendix C lists initialization implementation and encoder-only and decoder-only model setups; Appendix E lists comparisons with Init[AB], an equal-LR math sweep, and variance-compressed structured initializers.The supplied passages identify these topics but do not provide their experimental details.Source: C. Experimental Setup · p. 15, E. Comparisons with Other LoRA… · p. 15

A.1 Two Scaling Laws

  • Appendix A.1 distinguishes LLN and CLT scaling regimes when analyzing the size of sums as n grows.The passage introduces the laws for i.i.d. variables with finite expectations, and the CLT additionally assumes finite variance.Source: §A.1 · p. 16

A.2 γ-Calculus

  • Appendix A.2 defines γ[v] as the exponent λ when a variable scales as Θ(n^λ), and gives rules for zero, products, and sums.For matrices or vectors, the framework describes coordinate-wise scaling.Source: §A.2 · p. 16

A.3 Update Decomposition and Noise-Dominance Theorem

This appendix develops a conditional signal–noise account of LoRA updates, specifying when Adam’s A-branch update aligns with the input and bounding initialization noise under AdamW. It examines rank and temporal-contraction dependencies and tests alignment on minibatches, reporting an additional 1/n noise-energy reduction under matched scales while noting that minibatch alignment varies across modules and training.

A.3 Update Decomposition and Noise-Dominance Theorem

  • The update and feature decompose into initialization-dependent noise and LR-dependent signal; rank coherence varies by model, initialization, and training stage, and the probe does not establish the theorem’s accumulated contractions.Llama is largely CLT-like, Qwen varies with initialization and stage, and RoBERTa is closer to coherent summation; the theorem allows distinct noise and signal rank exponents.Source: §A.3 · pp. 16, 18, 19, Figure 7 · p. 18

The total energy satisfies

  • At fixed rank and matched contraction scales, Init[A2] supplies an extra 1/n noise reduction while preserving signal scaling, without requiring equal rank exponents or LLN-like rank summation.The exact energy decomposition applies to every realized trajectory, without requiring independence between initialization and optimizer directions.Source: The total energy satisfies · pp. 19–20
  • At the analyzed finite step, Init[A2] satisfies stability and efficiency, whereas standard Init[A] can have canceling signal and initialization terms that prevent a guaranteed lower bound.For fixed rank and step, standard Init[A] has initialization terms O(1), while Init[A2] reduces them to o(1); the signal contributions are Θ(1) under the stated assumptions.Source: The total energy satisfies · pp. 20–21

A.4 Adam and AdamW as Normalized LoRA Updates

  • The Adam/AdamW analysis addresses coordinate-bounded updates and A-branch gradient–input alignment, but leaves temporal non-cancellation and rank-contraction assumptions separate.Its alignment analysis is local to a fixed input and is conditional on that input; the full-network input can evolve with depth and training time.Source: §A.4 · pp. 21–22

A.4.1 SignSGD Alignment

  • For fixed-input SignSGD, the A-gradient’s row-wise outer-product structure yields exact input alignment after the initial zero A-update.The non-degeneracy result applies after B receives a nonzero update, or when the scalar signal history is nonzero.Source: §A.4.1 · p. 22, §A.4 · p. 22

A.4.2 Adam Coordinate Boundedness

  • With width-independent Adam hyperparameters and the stated beta conditions, Adam coordinate updates are O(1) uniformly in width.The bound holds for every scalar gradient history and time step.Source: §A.4.2 · p. 23

A.4.3 Adam as Soft-Sign Gradient–Input Alignment

  • Adam’s soft-sign gradient–input alignment requires scalar EMA non-cancellation and a bound on epsilon relative to the signal.The section sets these conditions to establish the required Θ(n) A-branch alignment.Source: §A.4.3 · pp. 23–24
  • Under Lemma 5’s conditions, Adam verifies the row-wise A-branch requirement, but temporal non-cancellation and rank contraction remain assumptions.The epsilon condition matters: for fixed ε > 0, a sign-consistent example has alignment Θ(n/(1 + εn)) = Θ(1), not Θ(n).Source: §A.4.3 · p. 24

A.4.4 A Weighted-Angle View of Adam Non-Cancellation

  • For nonzero signal histories, Adam’s non-cancellation condition |κ_t,i| = Θ(1) is equivalent to a width-independent lower bound on the absolute weighted-angle cosine.Under fixed-input and small-ball assumptions, this condition and δ_t,i = O(1) ensure alignment; sign-consistent entries with width-independent magnitude bounds suffice after the initial zero gradient.Source: §A.4.4 · p. 25

A.4.5 AdamW and Effective Horizons

  • AdamW’s non-expansive weight-decay bounds do not imply that realized initialization noise decreases with decay, because damping can reduce cancellation between contributions.The effective horizons bound component scales, but matching lower bounds require assumptions on the weighted sums; scalar horizons alone do not establish temporal non-cancellation or rank contraction.Source: §A.4.5 · p. 26

A.4.6 Empirical Width Alignment and Single-Example Scope

  • Figure 8 shows predominantly sublinear q_proj alignment and generally superlinear down_proj alignment, with substantial training changes, so Θ(n) alignment is not generic for minibatch fine-tuning.The formal signal-scaling results remain conditional on the fixed single-example analysis; minibatch extensions require realized alignment and temporal accumulation, while the separate noise comparison retains its extra 1/n factor under matched assumptions.Source: Figure 8 · p. 27, §A.4.6 · p. 27

A.4.5 AdamW and Effective Horizons

  • Changing σ_A^2 from Θ(n^-1) to Θ(n^-2) gives an extra 1/n noise-energy factor under matched input, multiplier, β, decay, horizon, and rank-contraction scales.For constant learning rates and non-vanishing decay products, the effective horizons scale as H_A(t) = Θ(η_A t) and H_B(t) = Θ(η_B t); the decomposition alone does not establish matched scales across optimizer trajectories.Source: §A.4.5 · p. 27

A.4.7 Summary for the Main Theorem

The appendix derives Init[A2] from spectral scaling conditions, explains why coordinate-wise γ-calculus alone cannot diagnose LoRA noise, and reviews related work on initialization, hyperparameters, and feature learning. Its conclusions remain conditional on alignment, temporal non-cancellation, and rank-contraction assumptions.

A.4.7 Summary for the Main Theorem

  • Adam and AdamW satisfy the optimizer-facing requirements under stated conditions, while temporal non-cancellation and rank contractions remain separate assumptions.The width-alignment table reports fits at steps 5 and 620, with dot-product confidence intervals over three seeds and cosine exponents at both steps.Source: §A.4.7 · pp. 27–28, Table 2 · p. 28

A.5.3 Implementing Feature Learning via Init[A2]

  • Spectral feature-learning conditions yield Init[A2] scales of σ_A^2 = Θ(n^-2), η_A = Θ(n^-1), and η_B = Θ(1).These scalings control spectral magnitudes; non-degenerate feature learning additionally requires alignment and non-cancellation assumptions.Source: §A.5.3 · p. 29, §A.5.2 · p. 29

A.6.1 Why γ-Calculus Is Not Sufficient

  • Coordinate-wise γ-calculus can miss dependence and signed cancellation, so distinguishing initialization noise from optimizer-driven signal requires an explicit decomposition.The noise and signal may have different rank contractions, and their cross term can reinforce or cancel them.Source: §A.6.1 · pp. 29–30

A.6.2 Spectral Condition: Worst-Case Control

  • For fixed t, α/r, β, and η_B = Θ(1), worst-case noise RMS is O(√n) under Init[A] and O(1) under Init[A2].This spectral bound establishes bounded worst-case noise for Init[A2], not vanishing noise or signal dominance for every aligned input.Source: §A.6.2 · p. 30

B.1 LoRA Initialization

  • Prior initialization methods include zero-B standard LoRA, quantization- and activation-based decompositions, and orthogonal approaches.The passage describes LoftQ, PiSSA, EVA, CorDA, and OLoRA as addressing quantization, spectral, context-oriented, or orthogonal initialization objectives.Source: §B.1 · p. 30

B.2 Other LoRA Hyperparameters

  • LoRA’s rank, scaling factor, and learning rates are coupled hyperparameters, and higher rank does not automatically improve vanilla LoRA.The conventional α/r scaling can weaken the branch as rank grows; α/√r is described as preserving a more consistent update scale.Source: §B.2 · p. 31

B.3 Feature Learning and HP Transfer

  • Feature-learning parameterizations support nontrivial representation updates and can enable hyperparameter transfer across model scales.The passage describes µP as maintaining consistent dynamics across width and depth, while empirical methods normalize layer outputs and updates for width transfer.Source: §B.3 · p. 31

B.4 Training Stability

This block reviews stability-oriented approaches, specifies the initialization and fine-tuning setups, and reports a toy-example reproducibility caveat. The experiments cover RoBERTa encoder tasks and decoder-only arithmetic and commonsense benchmarks.

B.4 Training Stability

  • Training stability is treated as a primary objective complementary to feature-learning optimality, with spectral controls and normalization among related approaches.The cited examples include optimizer constraints, Layer Normalization, and transformer normalization choices.Source: §B.4 · pp. 31–32

C.1 Initialization Implementation

  • The implementation uses Kaiming-uniform A and B0 = 0, with Init[A2] variance β²/fan_in² versus β²/fan_in for Init[A].For rectangular layers, n denotes A’s fan-in.Source: §C.1 · p. 32

C.2 Toy Example

  • The MNIST toy experiment uses rank 8, widths 16–4096, 20 Adam epochs, and ηB = nηA; exact numerical reproduction has not been verified.The original toy source is not archived, and the released reconstruction’s ReLU subgradient choice affects whether the first update occurs.Source: §C.2 · p. 32

C.3 Encoder-Only Model Setup

  • RoBERTa-base is fine-tuned on four GLUE tasks with LoRA on query and value projections, using fixed-ratio and two-dimensional LR sweeps.The setup uses r = α = 8; AdamW training runs for three epochs with sequence length 128 and batch size 32.Source: §C.3 · p. 32

C.4 Decoder-Only Model Setup

  • Decoder experiments use arithmetic/math and commonsense benchmarks, with Llama 3-8B sweeps over initialization scale and Llama 3.2-1B rank sweeps.The setup also includes model-family and initialization-power sweeps; training lasts one epoch for commonsense and three for arithmetic, with separate longer-horizon diagnostics.Source: §C.4 · pp. 32–33

D.1 LR Sweep Experiment

Across LR sweeps, Init[A2] retains a broad high-accuracy region where Init[A] becomes sensitive to initialization scale. With A frozen, the best B learning rate for Init[A] shifts downward as β increases, while Init[A2] remains effective at the largest tested values.

D.1 LR Sweep Experiment

  • At β = 8, Init[A] narrows to low ηB values, while Init[A2] retains a broad high-accuracy region across the tested LRs.Both initializations have broad high-accuracy regions at β = 0.25.Source: §D.1 · p. 33, Figure 9 · p. 33

D.2 Sensitivity to the B LR

  • With A frozen, Init[A2] reaches high accuracy at the largest tested ηB values across β, while Init[A]’s best ηB shifts downward as β increases.The sweep tests ηB from 5×10^-5 to 1×10^-2 with ηA = 0.Source: §D.2 · pp. 33–34, Figure 10 · p. 34

D.3 Additional Llama Experiments

Additional arithmetic experiments test initialization robustness under quantized fine-tuning, across model families, and at more extreme LR ratios. Init[A2] preserves high accuracy at large ratios where Init[A] collapses as β grows, with a tradeoff at small ratios for more aggressive initialization scalings.

Quantized fine-tuning

  • At ratio 2000, raising β from 0.5 to 8 drops Init[A] accuracy from 71.9% to 1.8% for LoRA and from 70.7% to 4.5% for QLoRA, while Init[A2] retains roughly 77–79% at large ratios.The high-ratio sensitivity under Init[A] and broad high-accuracy region under Init[A2] both persist with quantized backbones.Source: §D.3 · p. 34

Additional model families

  • Across five model families, Init[A] collapses in high-ratio, large-β regions, while Init[A2] removes most of this failure region and remains competitive at high ratios.Init[A2] reaches roughly 58–60% on Llama 3.2-1B, 72–74% on Llama 3.2-3B, 57–66% on Qwen2.5-3B, 72–74% on Qwen3-0.6B, and 71–73% on Mistral-v0.3.Source: §D.3 · pp. 34–35, Figure 12 · p. 35

Smaller Llama backbone

  • On Llama 3.2-1B, Init[A] is near zero at ratio 20000 across tested β, whereas Init[A2] peaks around 60–62% at large ratios; Init[A3] and Init[A4] reach about 61–62% at the largest ratios.Init[A3] and Init[A4] start near 30% at ratio 1, indicating that stronger suppression requires a larger ratio to recover signal learning.Source: §D.3 · p. 35

D.4 Long-Horizon Arithmetic Fine-Tuning

Separate Llama 3-8B arithmetic runs compare Init[A] and Init[A2] across training horizons and LoRA multipliers. The reported accuracy gap persists across tested budgets, although the horizon comparisons do not isolate early stopping as its cause.

ρ = 4000 stress configuration

  • Across four horizons at ratio 4000, the reported accuracy gap is about 20–22 points, and Init[A2] remains ahead in the 10-epoch run.Init[A2] also reaches lower final training loss at longer horizons; the runs do not establish that all Init[A] configurations behave similarly.Source: §D.4 · p. 35, Table 3 · p. 35

Multiplier experiment

  • At α/r = 4, Init[A] is 0.97 points higher at one epoch, but Init[A2] leads at longer reported horizons; at α/r = 12, Init[A2] leads at every horizon.Both methods reach lower training loss with longer budgets, while an accuracy gap remains; separate schedules and absent seed-level uncertainty limit interpretation.Source: §D.4 · pp. 35–36, Table 4 · p. 36

D.5 Multi-Seed Accuracy and Tuning Robustness

Multi-seed tests distinguish robustness under fixed settings from best performance after matched retuning, while energy measurements examine signal and noise contributions. Comparisons with other initializations emphasize matched settings rather than method-specific tuned learning rates.

D.5 Multi-Seed Accuracy and Tuning Robustness

  • At fixed ratio 200 and α/r = 10 or 12, the Init[A2]-minus-Init[A] accuracy confidence intervals remain above zero across seeds; at α/r = 1, the interval includes zero.These fixed-LR results support robustness in the stress regime, not best-tuned performance.Source: §D.5 · p. 36, Table 5 · p. 36
  • On the β = 8 RoBERTa/SST-2 grid, Init[A2] increases near-best cells from 5 to 8 and reduces chance-level cells from 14 to 5, while peak scores are similar relative to uncertainty.Counts are out of 25 LR-grid cells; near-best means within one percentage point of each method’s own best.Source: §D.5 · p. 36, Table 6 · p. 36
  • After matched retuning, scores are similar at α/r = 4, while Init[A2] remains 1.33 points higher at α/r = 12 within the searched grid.Retuning recovers much of Init[A]’s fixed-ratio loss; most other LR-ratio sweeps are single-run, so small differences are not established improvements.Source: §D.5 · pp. 36–37, Table 7 · p. 37

D.6 Signed Signal–Noise Energy Measurements

  • Under Init[A2], the signal contributes approximately 86–90% of total energy, whereas Init[A] has noise dominance and a destructive cross term.The measurements cover two model widths and do not directly verify the asymptotic 1/n noise-suppression rate.Source: §D.6 · p. 37, Table 8 · p. 37

E Comparison with Other LoRA Initializations

E.1 Comparison with Init[AB]

Init[A2] meets the relevant post-update scaling while retaining zero initial adapter output, and the comparison highlights stability and implementation advantages over Init[AB]. Under symmetric LRs, the methods perform similarly in standard regimes, with Init[A2] more accurate at extreme LRs when β = 8.

E.1 Comparison with Init[AB]

  • Init[A2] attains the desired post-update scaling, γ[A0] = −1 and γ[∆B] = 0, while preserving zero initial adapter output.The comparison uses the first nonzero B update because B0 = 0 has γ[B0] = −∞ under the stated convention.Source: §E.1 · p. 37
  • Init[A2] maintains bounded internal features, avoiding the feature explosion characteristic of Init[AB].The comparison attributes this to internal stability.Source: §E.1 · p. 37
  • Init[A2] keeps B = 0, avoiding Init[AB]’s need for nonzero factor initialization and weight subtraction to preserve the pretrained mapping.This also avoids inference overhead.Source: §E.1 · p. 38
  • At extreme symmetric LRs (10^-5 and 10^-3) with β = 8, Init[A2] has higher accuracy on MNLI and SST-2; the methods perform similarly in standard regimes.Figure 13 marks cells with accuracy gaps greater than 0.5%.Source: §E.1 · p. 38, Figure 13 · p. 38

E.2 Equal-LR Sweep on Math Benchmark

The equal-LR math sweep finds no initialization that dominates across LRs, while variance-compressed structured initializers retain much higher arithmetic accuracy than their uncompressed counterparts at ρ = 4000. The appendix also records the study’s scope, experimental reporting, and reproducibility details.

E.2 Equal-LR Sweep on Math Benchmark

  • On the equal-LR math benchmark, no initialization dominates across LRs; Init[A2] remains competitive near the best results but can adapt more slowly early on.It avoids the severe large-β degradation of Init[AB].Source: §E.2 · p. 38

E.3 Variance-Compressed Structured Initializers under Asymmetric LRs

  • The structured A2 variants target RMS(A0,ℓ) = β/nℓ while retaining the base initializer’s structure; for PiSSA, the subspaces and initial product are preserved, but optimization geometry changes.The evaluated base initializers and compressed variants use arithmetic fine-tuning on Llama 3-8B with ρ ∈ {1, 200, 4000}.Source: §E.3 · pp. 38–39
  • At ρ = 4000, compressed Orthogonal, EVA, and PiSSA variants retain 77.74%, 77.19%, and 73.80% accuracy, versus 2.49%, 5.10%, and 48.10% for their uncompressed counterparts.These are single-run results; compression is not uniformly beneficial at ρ ∈ {1, 200}, and small differences remain uncertain.Source: §E.3 · p. 39, Table 10 · p. 39

F.1 Broader Impact

  • The authors state that the work has no direct negative societal impacts unique to this initialization; broader impacts depend on pretrained models and downstream uses.They report using LLMs to polish language and code, while the core research and scientific conclusions were generated by the authors.Source: §F.1 · p. 39, §F.2 · p. 39

2. Limitations

  • The authors identify limits including supervised fine-tuning focus, no direct rule for choosing asymmetric LR ratios across arbitrary models and tasks, and incomplete theory for rank and aggressive scaling.The limitation statement also notes uncertainty around more aggressive initialization scaling.Source: §2 · p. 40

4. Experimental result reproducibility

  • The paper reports its experiment setup, datasets, model choices, LoRA hyperparameters, optimizer settings, and sweep ranges in the main text and appendix.It also says code, setup instructions, and data-access recipes are publicly available.Source: §4 · p. 41, §5 · p. 42

7. Experiment statistical significance

  • Most main language-understanding and generation heatmaps are single-run, and uncertainty types and interval-construction methods are not fully documented.The appendix reports selected three- and five-seed accuracy summaries and confidence intervals.Source: §7 · p. 42

8. Experiments compute resources

  • The authors report that GPU type is documented and each fine-tuning run takes roughly three hours on one L40s GPU, while the full sweep requires substantially more compute.The larger total arises from sweeping initialization scale, LR ratio, rank, and LoRA multiplier.Source: §8 · p. 43

Related papers

Loading 2610.32142v1…