PaperlayerView exact revision on arXiv

Source-linked AI summary

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

Huiyi Wang, Daijiao Liu, Lina Yao, Dong Gong

arXiv:2609.05885v1cs.LG

TL;DR

Uniform learning rates overlook heterogeneous rank-one dynamics within LoRA adapters, which can leave nominal rank capacity underused. The paper proposes AnLR-LoRA, assigning per-component rates online from velocity and Adam SNR. Across language and multimodal settings, it reports improved adaptation and broader rank-capacity use.

  • Problem

    Uniform learning rates across LoRA rank-one components overlook within-adapter training heterogeneity associated with concentrated updates and underused nominal rank capacity.

    Source: §1 · p. 1

  • Method

    AnLR-LoRA assigns each rank-one component an online effective learning rate from function-space velocity and Adam SNR, mean-normalized per module to redistribute a fixed global rate.

    Source: §1 · p. 2, What information should the multiplier… · p. 6

  • Results

    AnLR-LoRA improves LoRA adaptation and encourages broader rank-capacity use across language and multimodal settings; on Commonsense170K, it achieves the best average accuracy on all three backbones, outperforming standard LoRA by up to 4.3%.

    Source: §5.1 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    The results support anisotropic learning-rate redistribution as a simple optimization dimension for improving LoRA adaptation and broadening rank-capacity use.

    Source: §3.3 · p. 4, §6 · p. 9

    How broadly the benefit holds across tasks, and its extension to multilingual, long-context, and larger-scale training, remain to be explored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.

1 Introduction

LoRA offers parameter-efficient adaptation, but its uniform learning-rate convention overlooks persistent differences in how rank-one components update within an adapter. AnLR-LoRA assigns components online adaptive rates using training-time signals, and experiments report broader rank-budget use and improved adaptation.

Motivation

  • LoRA is a standard parameter-efficient fine-tuning choice that balances adaptation quality with computational cost.The paper situates its study within the broader expense of full fine-tuning large language models.Source: §1 · p. 1
  • Uniform learning rates overlook persistent within-adapter velocity differences that leave nominal rank capacity underused and cannot be corrected by tuning one global rate.The paper identifies this anisotropy as arising during training, rather than from initial conditions alone.Source: §1 · pp. 1–2

Contributions

  • AnLR-LoRA redistributes a fixed global learning-rate budget across rank-one components using online velocity and Adam SNR signals.The signals are fused and mean-normalized per module; the method applies without architectural changes and with negligible computational overhead.Source: §1 · p. 2
  • Experiments report that adaptive anisotropic learning rates improve LoRA adaptation and encourage rank-budget use across settings, with robustness to global learning-rate changes and transfer to LoRA variants.The introduction states that these benefits hold across varied settings and do not require changing the global learning-rate budget.Source: §1 · p. 2

2 Related Work

The related work reviews parameter-efficient fine-tuning and LoRA design variants, then identifies rank-level adaptive learning-rate allocation as a gap in optimization methods. Existing approaches generally operate at broader granularity or intervene through mechanisms other than learning-rate scheduling.

2.1 Parameter-Efficient Fine-Tuning

  • Parameter-efficient fine-tuning reduces adaptation cost by training only a subset of model parameters, using approaches such as adapters, prompts, prefixes, and low-rank methods.The paper focuses its review on LoRA as a standard PEFT approach for large language models.Source: §2.1 · p. 2

2.2 Architectural Variants of LoRA

  • LoRA variants alter factorization, initialization, effective rank, or continual-learning use, while retaining a single global learning rate across parameters.Figure 2 summarizes observed rank-one velocity ordering, contribution magnitude differences, and spectral concentration associated with lower-velocity modules.Source: §2.2 · p. 2, Figure 2 · p. 3

2.3 Optimization and Hyperparameter Choices for LoRA

  • Prior optimization methods use module- or matrix-level rates, and none assigns each rank component an online rate derived from training signals.LoRA+ uses two matrix-level rates fixed at training start, whereas this paper derives rates from rank contribution change and Adam-gradient consistency.Source: §2.3 · p. 3

3 Analysis of Rank-One Training Dynamics in LoRA

An analysis of 224 LoRA modules in LLaMA2-7B fine-tuning on MetaMathQA finds persistent within-module differences in rank-one update velocity. Slower-moving modules learn more concentrated composite updates, leaving part of their nominal rank budget unused.

3 Analysis of Rank-One Training Dynamics in LoRA

  • The study characterizes dynamic and cumulative rank-one adaptation using each component’s Frobenius-norm rate of change and cumulative Frobenius norm.The analysis covers 224 LoRA modules from one LLaMA2-7B run on MetaMathQA.Source: §3 · p. 3

3.1 Per-Rank-One Velocity is Anisotropic within Each Module

  • Across 224 LoRA modules, the median maximum-to-minimum rank-one velocity ratio is 2×, the 90th percentile is 4×, and the most imbalanced 1% exceed 8×.The analysis uses per-rank-one velocity and cumulative contribution magnitude to characterize adaptation dynamics in LLaMA2-7B on MetaMathQA.Source: §3.1 · p. 3, §3 · p. 3

3.2 The Anisotropy is Established Early and Persists

  • Rank-one velocity ordering emerges within approximately the first 1k steps and persists to convergence, while the top-velocity quartile’s contribution-magnitude lead also persists.The ordering is stabilized early in training rather than predictable from A(0) and B(0) alone.Source: §3.2 · p. 4

3.3 Low-Velocity Modules Under-Utilize their Nominal Rank Budget

  • Lower-velocity modules develop more concentrated BA spectra, packing update energy into fewer directions and using less of the nominal rank budget.The paper describes this relationship as an association between lower mean per-rank-one velocity and sharper cumulative spectral-energy curves.Source: §3.3 · p. 4
  • For LLaMA2-7B on Commonsense170K, the best rank-32 full-fine-tuning approximation has effective rank 27.2, versus 12.7 for LoRA at rank 32.This comparison supports the paper’s claim that concentration under uniform learning rates is induced by optimization rather than inherent to the downstream update.Source: §3.3 · p. 4
  • Redistributing learning rates toward slower rank-one components is proposed as a way to encourage broader rank-capacity use and is evaluated for downstream gains later.The paper identifies spectral concentration as the target of its intervention.Source: §3.3 · p. 4

4 Methodology

The methodology replaces LoRA’s shared learning-rate convention with online, per-rank-one learning-rate multipliers, then instantiates and applies them using signals available during AdamW optimization.

4 Methodology

  • AnLR-LoRA assigns each rank-one component a time-varying multiplier to redistribute its adapter’s global learning rate.Multipliers are computed online from velocity and Adam SNR signals.Source: §4 · p. 4

4.1 The Uniform-LR Convention

  • Standard LoRA applies one global learning-rate schedule uniformly across every parameter and rank-one component.This convention uses the same schedule shape and granularity throughout each adapter.Source: §4.1 · pp. 4–5

4.2 Adaptive Anisotropic Learning-rate

  • The adaptive model controls learning rates at the rank-one level, matching the finest structured unit where LoRA anisotropy emerges.Its multipliers are computed online from training state, preserve the module-level average learning rate, and add no trainable parameters.Source: §4.2 · p. 5

4.3 AnLR-LoRA: Instantiating the Model with Two Training-Time Signals

What information should the multiplier use?

  • Velocity tracks component activity in function space, while Adam SNR estimates update reliability; log-additive fusion combines them after per-module normalization.Velocity is measured on the rank-one contribution to the update, capturing the coupling between its A and B factors.Source: What information should the multiplier… · pp. 5–6
  • The velocity signal boosts slow-moving components and dampens fast-moving ones, while SNR favors reliable directions; warmup and clamping stabilize the normalized multipliers.Per-module normalization preserves the global learning-rate average, and scheduling begins after a short warmup.Source: What information should the multiplier… · pp. 5–6

4.4 Application of Per-Rank-One LR via Post-Step Delta Scaling

  • Post-step delta scaling applies each component’s effective learning rate without changing AdamW’s optimizer state or adding forward or backward passes.The implementation snapshots parameters and rescales the resulting per-component updates after a standard AdamW step.Source: §4.4 · p. 6

5 Experiments

Across commonsense reasoning, generation, and visual instruction tuning, AnLR-LoRA improves on LoRA. The experiments also examine broader rank use, robustness across learning rates, and application to other LoRA variants.

5.1 Main Results

  • AnLR-LoRA achieves the best average commonsense accuracy on all three backbones, beating LoRA by up to 4.3% and surpassing rsLoRA and LoRA+ across model families and sizes.The comparison uses the same parameter and learning-rate budget.Source: §5.1 · p. 6, Table 1 · p. 7
  • AnLR-LoRA improves over LoRA and LoRA+ on dialogue, math reasoning, and code generation, with the largest gain on HumanEval (+3.66% over LoRA).The evaluations use MT-Bench, GSM8K, and HumanEval, respectively.Source: §5.1 · p. 7, Table 2 · p. 7
  • On seven vision-language benchmarks, AnLR-LoRA achieves the best average score, with its largest gain on VizWiz (+3.07% over LoRA).The result also holds at rank r = 128.Source: §5.1 · p. 7, Table 3 · p. 8

AnLR-LoRA encourages broader use of rank

6 Conclusion

The paper identifies within-adapter heterogeneity in LoRA rank-one training dynamics and proposes AnLR-LoRA, an adaptive per-component learning-rate method. Experiments across language and multimodal settings show improved adaptation and broader use of rank capacity.

  • The uniform-LR convention overlooks substantial heterogeneity in rank-one training dynamics within a LoRA adapter.Source: §6 · p. 9
  • AnLR-LoRA assigns rank-one components adaptive learning rates using function-space velocity and Adam SNR to encourage learning on low-velocity components with reliable update directions.Source: §6 · p. 9
  • Across language and multimodal settings, AnLR-LoRA improves LoRA adaptation and encourages broader use of rank capacity.Source: §6 · p. 9

Limitations

AnLR-LoRA is one instantiation of a broader adaptive anisotropic learning-rate framework; broader task coverage is needed to delineate where it is most beneficial.

Limitations

  • AnLR-LoRA is one instantiation, leaving alternative signals, fusion strategies, and more sophisticated update-direction uncertainty measures for future work.The analysis also leaves open how broadly its rank-utilization finding holds across tasks, as well as extensions to multilingual, long-context, and larger-scale training.Source: Limitations · p. 9

A.1 Pre- and Post-Step Scaling under AdamW

The appendix shows that pre-step gradient scaling does not install per-component effective learning rates under AdamW, whereas post-step delta scaling is equivalent to substituting the outer rate. AnLR-LoRA uses post-step scaling as a drop-in update modification.

A.1 Pre- and Post-Step Scaling under AdamW

  • Post-step delta scaling is equivalent to running AdamW with the outer learning rate replaced by the per-component scaled rate, with decoupled weight decay unchanged.The AdamW moment recurrences are unchanged because they depend on the gradient sequence and moment-decay coefficients, not the outer rate.Source: §A.1 · pp. 13–14
  • Pre-step gradient scaling by a fixed factor leaves the AdamW trajectory unchanged when ε = 0 and perturbs it only by O(ε) otherwise.Adam’s preconditioner cancels constant gradient scaling; time-varying scaling perturbs moment estimates rather than rescaling the step.Source: §A.1 · p. 13

A.2 Training Procedure of AnLR-LoRA

  • AnLR-LoRA computes per-rank-one velocity and Adam SNR, fuses their normalized signals after warmup, then applies multipliers to provisional AdamW updates.This post-step implementation leaves optimizer state unchanged and requires no extra forward or backward passes.Source: §A.2 · p. 14, §A.1 · p. 14

B.1 Datasets

The datasets span commonsense reasoning, mathematical reasoning, code generation and dialogue, alongside visual instruction-tuning and vision-language evaluation.

B.1 Datasets

  • Commonsense170K supports evaluation across commonsense and science question-answering tasks, including BoolQ, PIQA, SIQA, HellaSwag, ARC-e, and ARC-c.The dataset contains approximately 170K commonsense reasoning examples; the named benchmarks measure distinct reasoning and question-answering tasks.Source: §B.1 · p. 14
  • Math reasoning uses the 100K MetaMathQA training subset and GSM8K evaluation, while code-generation training uses a 100K Code-Feedback subset.HumanEval evaluates code generation; WizardLM and MT-Bench provide instruction-tuning data and multi-turn dialogue evaluation, respectively.Source: §B.1 · pp. 14–15
  • Visual instruction tuning uses the LLaVA-1.5 instruction dataset, with vision-language evaluation including VQAv2, GQA, VizWiz, SQA, and MMBench.The instruction dataset contains approximately 665K multimodal examples; the listed benchmarks cover visual question answering, reasoning, and multimodal evaluation.Source: §B.1 · p. 15

B.2 Implementation Details

The experiments report improved accuracy and rank utilization with AnLR-LoRA, examine its velocity and SNR signals, and test robustness across learning rates and adapter ranks. The appendix also specifies implementation settings and notes that the resources were used for academic research and evaluation.

B.2 Implementation Details

  • The implementation reports multiple-seed averages and uses LoRA modules in attention query, key, and value projections and FFN up- and down-projections.The log-additive fusion hyperparameters α, β, and κ are 2.0, 1.0, and 3.0, respectively.Source: §B.2 · pp. 15–16

C Licenses and Usage

  • The datasets and pretrained models are publicly available and were used in accordance with their licenses and terms for academic research and evaluation.The resources named include LLaMA2, LLaMA3, Qwen2.5, and LLaVA-1.5-7B.Source: C Licenses and Usage · p. 16

D.1 Analysis of Spectral Concentration

  • The rank-32 full fine-tuning approximation has effective rank 27.2, compared with 12.7 for uniform-LR LoRA; a spectrum-flattening penalty also improves math-reasoning accuracy.For math reasoning, the penalty raises accuracy from 59.59 to 60.96 at coefficient 10^-3 and to 61.11 at 10^-2.Source: §D.1 · pp. 16–17

D.2 Mechanism Analysis of the Two Signals

  • Velocity and SNR jointly identify slower components with stable directions: their combined score correlates 0.69 with subsequent component growth, and the combination reduces spectrally concentrated modules from 28.1% to 16.3%.Velocity scaling halves velocity-imbalance coefficient of variation from 0.12 to 0.06; adding SNR lowers the bottom-SNR quartile’s share of the total boost from 37.4% to 18.3%.Source: §D.2 · p. 17

D.3 Robustness to LR Choice

  • AnLR-LoRA improves over LoRA at every tested learning rate, with gains of 3.3% to 13.7%, while its total accuracy variation is 7.7% versus LoRA’s 16.5%.The sweep covers six learning rates from 5e-5 to 5e-4 on LLaMA2-7B; AnLR-LoRA also beats LoRA at LoRA’s optimal setting.Source: §D.3 · p. 17, Table 10 · p. 18

D.5 Robustness to Rank Choice

  • At ranks 4, 8, and 32, AnLR-LoRA improves over LoRA by 2.3% to 3.4% with no additional parameter cost.AnLR-LoRA at rank 4 scores 60.96%, exceeding LoRA at rank 32, which scores 58.23%.Source: Table 11 · p. 18, §D.5 · p. 18

D.4 Per-rank-one LR Scaling Signal

  • The velocity and Adam SNR fusion reaches 81.0% average accuracy, 1.7% above the parameter-sensitivity signal; random multipliers score 54.7%.The parameter-sensitivity signal scores 79.3%, above uniform-LR LoRA at 77.5%.Source: §D.4 · pp. 17–18

D.6 Robustness to Hyperparameter Choice

AnLR-LoRA's performance remains relatively stable across the evaluated hyperparameter sweeps. Three-run results also show low variance in its improvements over LoRA.

D.6 Robustness to Hyperparameter Choice

D.7 Standard Deviation Across Seeds

D.8 Experiments on Qwen2.5-14B

On Qwen2.5-14B, AnLR-LoRA achieves the best reported math-reasoning accuracy among the compared methods. It also leads additional LoRA variants across backbones, while its parameter snapshot has modest memory overhead and can be removed through optimizer-step fusion.

D.8 Experiments on Qwen2.5-14B

  • On the larger Qwen2.5-14B backbone, AnLR-LoRA attains the best accuracy, outperforming LoRA, LoRA+ and rsLoRA by 0.83, 1.32 and 4.32 points, respectively.Source: §D.8 · p. 19

D.9 Comparison with Additional LoRA Variants

  • AnLR-LoRA achieves the best performance on every backbone against AdaLoRA, DoRA and LoRA-Pro, without extra trainable parameters or expensive matrix operations.Source: §D.9 · pp. 19–20, Table 16 · p. 19

D.10 Analysis on Memory Overhead of the Parameter Snapshot

  • 56 MB (0.42% of base-weight memory) on LLaMA2-7B; snapshot overhead is estimated at 0.20% for 70B and 0.10% for 405B, or removable by optimizer-step fusion.Source: §D.10 · p. 20

E LLM Usage

  • AI assistants were used only for manuscript language refinement, not for proposing the method, designing experiments or drawing scientific conclusions.Source: E LLM Usage · p. 20
Loading 2609.05885v1…