PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning

Yu-Ang Lee, Ching-Yun Ko, Pin-Yu Chen, Mi-Yen Yeh

arXiv:2602.04998v2cs.LGcs.AIcs.CL

TL;DR

Advanced LoRA methods are often compared under fixed or narrowly tuned hyperparameters, leaving their claimed advantages insufficiently tested. The paper systematically re-evaluates nine variants against vanilla LoRA with broad searches and finds that properly tuned methods reach similar peak performance, while their preferred learning rates differ.

  • Problem

    Most prior LoRA studies lack thorough hyperparameter tuning, limiting the reliability of comparisons between advanced variants and vanilla LoRA.

  • Method

    The paper benchmarks nine representative LoRA variants against vanilla LoRA under a unified protocol with extensive searches over key training hyperparameters.

  • Results

    Across tasks, model scales, and training settings, properly tuned methods achieve similar peak performance within 1-2%, with distinct optimal learning-rate ranges and subtle rank-dependent behavior.

  • Takeaways & Limitations

    Vanilla LoRA remains a competitive baseline, and improvements observed under single training configurations may not represent consistent methodological advantages.

  • Takeaways & Limitations

    The findings cover decoder-only LLMs from 0.6B to 13B parameters and do not establish scalability to larger models or generality across untested architectures and variants.

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model (LLM) fine-tuning. Building on this paradigm, recent studies have proposed alternative initialization strategies, architectural modifications, and optimization adjustments, reporting substantial improvements over vanilla LoRA. However, these gains are often demonstrated under fixed or narrowly tuned hyperparameter settings, despite the known sensitivity of neural networks to training configurations. In this work, we systematically re-evaluate nine representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches over learning rate, batch size, rank, and training duration. Across tasks spanning mathematical reasoning, commonsense reasoning, code generation, and instruction following at diverse model scales, we find that different LoRA methods favor distinct learning rate ranges. Crucially, once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2%), with only subtle rank-dependent behaviors. These results suggest that vanilla LoRA remains a competitive baseline and that improvements reported under a single training configuration may not reflect consistent methodological advantages. Finally, a second-order analysis attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.

1 Introduction

LLM fine-tuning motivates parameter-efficient methods, with LoRA becoming standard but advanced variants often evaluated without thorough hyperparameter tuning. This study re-evaluates those comparisons and finds that properly tuned learning rates largely eliminate reported performance gaps.

  • Full-parameter fine-tuning is prohibitively expensive for billion-parameter LLMs, motivating methods that update only a small fraction of parameters.
  • LoRA has become the de facto PEFT standard, while variants report improvements through new initializations, architectures, or optimization mechanisms.
  • Only 1 of 64 surveyed studies tuned three hyperparameters simultaneously, while 46 reported results under a fixed learning rate.
  • The study re-evaluates nine advanced LoRA variants against vanilla LoRA using a unified protocol and large-scale hyperparameter searches.
  • 0.84% separates the best and least effective methods on Qwen3-0.6B after tuning, while different methods still favor distinct learning-rate ranges.
  • Hessian analysis links differing optimal learning-rate ranges to the largest Hessian eigenvalue, while rank-dependent behaviors produce only marginal performance differences.

2 Related Work and Background

Prior work shows that architectural and optimization claims can depend on incomplete or inconsistent evaluation, while LoRA research explores multiple ways to modify low-rank adaptation. This paper reviews LoRA’s formulation and organizes representative variants by their intervention point.

  • Related empirical evaluation: Earlier studies found that careful hyperparameter tuning could explain improvements attributed to complex architectures, making fair evaluation important.
  • Related empirical evaluation: Few studies directly compare vanilla LoRA with advanced variants, and fixed training hyperparameters leave practitioners without reliable method-selection guidance.
  • LoRA background: LoRA’s sensitivity to hyperparameters has motivated work on learning rate, rank, initialization, scaling, dropout, and adapter placement, but unified configuration guidelines remain unresolved.
  • LoRA background: LoRA computes h = Wresx + γrBAx using trainable down- and up-projecting matrices A and B while the pretrained weight remains represented by Wres.
  • LoRA background: Vanilla LoRA initializes B to zero and A with Kaiming noise, ensuring training begins exactly at the pretrained checkpoint.
  • Representative variants: The paper groups nine variants into initialization variants, architecture modifications, and optimization adjustments.
  • Representative variants: Initialization variants use QR, SVD, random, or one-step gradient information, while architectural methods alter LoRA’s forward design and optimization methods adjust training dynamics.

3 Learning Rate Matters, Really

The study evaluates LoRA variants under a unified framework that searches key training hyperparameters across models, tasks, and ranks. Proper learning-rate tuning produces similar peak performance across methods, while method-specific learning-rate and rank behaviors remain.

  • Experimental Setup: The experiments cover four decoder-only models spanning Qwen3-0.6B, Gemma-3-1B, Llama-2-7B, and Llama-2-13B.Tasks include commonsense reasoning, mathematical reasoning, code generation, and instruction following.
  • Experimental Setup: Learning rates are searched logarithmically from 10^-6 to 10^-3, while batch sizes and ranks are tuned over predefined grids for selected model-task combinations.Epochs, adapter placement, and the scheduler remain fixed; α=r makes γ_r=1.
  • 3.3.1 Similar Performance Levels: Across model scales and tasks, comprehensive searches yield peak-performance gaps of 0.52% for Gemma-3-1B math, 0.43% and 1.75% for Llama-2-7B math and code, and 1.81% for Llama-2-13B math.PiSSA consistently uses a lower optimal learning rate than vanilla LoRA, while other methods typically remain within LoRA’s order of magnitude.
  • 3.3.1 Similar Performance Levels: At η = 1.1×10^-3, PiSSA retains 27.83% math and 26.90% code accuracy while other methods collapse to near-zero performance on at least one task.This indicates that similar peak performance does not imply similar robustness across the full learning-rate spectrum.
  • 3.3.1 Similar Performance Levels: Learning-rate tuning matters more than batch-size tuning: PiSSA reaches 16.1% when η=2×10^-5 is fixed, versus about 20.6% when batch size is fixed and learning rate is tuned.The optimal learning rate also scales proportionally with batch size, although very large batches can reduce best achievable performance.

4 Understanding the Optimal Learning Rate via Hessian Analysis

The analysis links LoRA variants’ optimal learning rates to the curvature of their loss landscapes, measured through the largest Hessian eigenvalue. Variants initialized in higher-curvature states require lower learning rates than vanilla LoRA.

  • Sharpness and learning rates: The top Hessian eigenvalue measures maximal loss curvature and is closely linked to the optimal learning rate.The analysis estimates this curvature for trainable LoRA parameters using block-wise Hessian calculations at initialization.
  • Sharpness and learning rates: The study estimates layer-wise top Hessian eigenvalues with Lanczos algorithms and Hessian-vector products without explicitly forming the Hessian.Parameters are analyzed block-wise across matrix types and Transformer layers.
  • Relative curvature: All examined variants initialize trainable parameters in higher-curvature states than vanilla LoRA.The comparison normalizes each variant’s maximum Query-projection eigenvalue by the corresponding LoRA value across Transformer layers.
  • Relative curvature: Up to 100× higher curvature for OLoRA and LoRA-GA corresponds to a 18.2× lower learning rate requirement than LoRA.PiSSA shows ≈10× higher curvature and requires a 10× lower learning rate, while Init[AB] and MiLoRA show ≈2× higher curvature with 1.8× and 3.2× lower optimal learning rates.

5 Conclusion

The paper re-evaluates ten LoRA methods under a unified protocol and finds that improper learning rates can create a false impression of methodological progress. It concludes that vanilla LoRA often remains competitive while Hessian analysis explains variants’ differing learning-rate ranges.

  • Conclusion: The study systematically re-evaluates ten LoRA PEFT methods under a unified evaluation protocol.The comparison is motivated by the growing number of variants and insufficient tuning in prior studies.
  • Conclusion: Improper learning rates can give a false sense of LoRA advancements.The conclusion emphasizes that apparent improvements should be assessed after comprehensive hyperparameter searches.
  • Conclusion: Vanilla LoRA frequently matches or outperforms advanced variants, whose improvements often lack universality.The paper identifies only marginal, setting-dependent reasons to favor particular variants.
  • Conclusion: Hessian analysis explains disparate optimal learning-rate ranges, while five practical heuristics aim to reduce LoRA tuning costs.The authors encourage more comprehensive hyperparameter-search protocols in future PEFT research.

Limitations and Future Work

The findings are bounded by the evaluated model architectures, parameter scales, variants, and tuned hyperparameters. Future work should test broader architectures and secondary training configurations, while exploring adaptation mechanisms beyond weight-based low-rank updates.

  • Limitations: The evaluation covers decoder-only LLMs from 0.6B to 13B parameters, so scalability to larger foundation models remains unverified.The scope also may not extend to encoder-only LLMs, Vision Transformers, or Vision-Language Models.
  • Limitations: The study does not exhaustively search secondary settings such as learning-rate schedulers, warmup steps, and adapter placements.Learning rate, batch size, and LoRA rank were tuned, while these other configurations remained fixed.
  • Limitations: The conclusions may not apply to all existing advanced LoRA variants.LoRA-One is cited as a method that reported ≈2% improvement over LoRA under comprehensive learning-rate sweeps.
  • Limitations: Specific variants may have advantages beyond benchmark accuracy, including mitigating catastrophic forgetting of pretrained knowledge.The authors therefore frame accuracy as one evaluation dimension rather than the sole criterion.
  • Future work: Future work may investigate hidden-representation fine-tuning and nonlinear-function adaptation as alternatives to weight-based low-rank adaptation.The paper suggests these mechanisms could open new dimensions of efficiency.

A.1 Learning Rate Tuning on More Model–task Combinations

Across additional model–task combinations, properly tuned LoRA variants reach comparable performance while retaining distinct learning-rate preferences. The observed pattern extends to larger models, ranks, batch sizes, and other task settings, with only small reported gains over vanilla LoRA.

  • Gemma-3-1B commonsense reasoning: ≈37% peak performance is reached by all methods on Gemma-3-1B commonsense reasoning tasks after proper learning-rate tuning.LoRA-GA consistently requires learning-rate ranges more than 10× lower than LoRA, while other variants occupy distinct relative ranges.
  • Practical tuning implications: Stable relative learning-rate ranges across model–task combinations can serve as practical priors for tuning variants without exhaustive sweeps.The paper suggests this may avoid re-estimating Hessians on new downstream samples.
  • Llama-2-13B mathematical reasoning: On Llama-2-13B mathematical reasoning, the largest gains over vanilla LoRA are 0.61% for Init[AB] at r = 8 and 0.53% for DoRA at r = 128.These results are presented as validating the conclusions at a larger model scale.
  • Rank and batch-size effects: Across Llama analyses, methods show comparable performance-improvement trends as adapter rank increases.The reported best performance jointly optimizes learning rate and batch size with B ∈{16, 128}.

A.3 Varying Training Duration

With learning rates properly tuned, LoRA variants show similar performance trends as training duration, sample size, and moderate batch-size changes vary. However, performance declines for LoRA and PiSSA at very large batch sizes.

  • A.3 Varying Training Duration: Training samples from 5k to 395k and epochs from 1 to 3 produce similar performance trends across vanilla LoRA and its variants.Accuracies were averaged over three independent runs.
  • A.3 Varying Training Duration: At B ≥256, the best achievable performance of both LoRA and PiSSA begins to decay even after learning-rate tuning.The results support keeping batch size within a small-to-medium range.
  • A.3 Varying Training Duration: At batch sizes B ≤128, LoRA and PiSSA reach approximately 20% accuracy with proper learning-rate tuning.The comparison uses Gemma-3-1B on mathematical reasoning with rank 128.

D.2 Training Hyperparameter Search Ranges

The study uses broad, logarithmically spaced learning-rate searches and reports results from three independent trainings for Qwen and Gemma experiments.

  • D.2 Training Hyperparameter Search Ranges: Three independent trainings are conducted for all Qwen and Gemma experiments.Reported results include the mean and standard deviation.
  • D.2 Training Hyperparameter Search Ranges: Learning rates are tuned evenly on a logarithmic scale using four values per order of magnitude.Table 4 summarizes model-specific search ranges, while the experiments report means and standard deviations for three independent trainings on Qwen and Gemma.

D.3 Fixed Training Hyperparameters

The experiments hold most training configurations fixed across methods while standardizing LoRA scaling and documenting method-specific scheduling, warmup, dropout, implementation, and data choices.

  • D.3 Fixed Training Hyperparameters: Most training configurations remain fixed and identical across experiments, except for tunable learning rate and batch size.The fixed settings primarily follow PiSSA and therefore may differ from other PEFT methods.
  • D.3 Fixed Training Hyperparameters: MiLoRA, DoRA, and Init[AB] use linear learning-rate decay, while MiLoRA and DoRA use 100 warmup steps and 0.05 dropout.These settings differ from the shared cosine-annealing, 3% warmup, and no-dropout configuration.
  • D.3 Fixed Training Hyperparameters: All experiments set α = r, producing a LoRA scaling factor γr = 1.This configuration is adopted across the decoder-only LLM experiments.
  • D.3 Fixed Training Hyperparameters: The unified implementation integrates ten LoRA-based methods using the PiSSA framework, official PEFT interfaces, and official codebases.The methods include LoRA, OLoRA, PiSSA, LoRA-GA, DoRA, GraLoRA, RandLoRA, MiLoRA, Init[AB], and LoFT.
  • D.3 Fixed Training Hyperparameters: Experiments use datasets for commonsense reasoning, mathematical reasoning, code generation, and instruction following, with deterministic random seeds and multi-run reporting.Implementation uses PyTorch 2.7.1 and four GPUs; most fine-tuning experiments report means and standard deviations over three runs.

F Details of Hyperparameter Search Results

The appendix provides detailed hyperparameter-search results for Qwen3-0.6B, Gemma-3-1B, and Llama-2-7B across mathematical reasoning and code-generation settings.

  • F Details of Hyperparameter Search Results: Appendix Sections F.1–F.3 detail search results for Qwen3-0.6B, Gemma-3-1B, and Llama-2-7B.The referenced tables correspond to main-text figures and rank-specific appendix figures.
  • F Details of Hyperparameter Search Results: Tables 7, 11, and 14 provide detailed numerical results for Figures 1, 3a, and 3b, respectively.These tables cover the corresponding model-task evaluations described in the appendix.
  • F Details of Hyperparameter Search Results: The appendix includes Qwen3 mathematical-reasoning results at ranks 8 and 128 and Gemma-3 mathematical-reasoning results at rank 128.These are presented in Tables 6, 7, and 8.
  • F Details of Hyperparameter Search Results: Llama-2-7B results cover mathematical reasoning at ranks 8, 32, and 128 and code generation at ranks 8, 32, and 128.These configurations are presented in Tables 9–14.

G Example Model Responses

The section compares LoRA and PiSSA reasoning paths on a MATH example across learning rates. The responses illustrate that the two methods operate in distinct effective learning-rate regimes.

  • Qualitative comparison: LoRA and PiSSA produce qualitatively different reasoning paths for the same MATH testing example across learning rates.The comparison is organized in Table 15, with the example input shown separately.
  • Learning-rate examples: At 2.00 × 10−4, one response reaches 840 while another response is marked incorrect.The displayed responses contain competing reasoning traces for the same combinatorial problem.
  • Learning-rate examples: At 6.32 × 10−4, a response reaches 840 and is marked incorrect, while another reaches 17 and is marked correct.These outputs demonstrate that answer correctness varies across the displayed model responses.
  • Learning-rate examples: At 1.12 × 10−3, a displayed response computes 35 as the answer.The response applies a partial combinatorial calculation rather than the intended total count.
  • Learning-rate examples: At 2.00 × 10−3 and 6.32 × 10−3, displayed responses contain gibberish or incorrect arithmetic rather than a valid solution.The examples include repetitive text and incorrect numerical answers at these learning rates.

H Practical Heuristics for LoRA Hyperparameter Tuning

The section distills practical LoRA-tuning heuristics centered on learning rate, batch size, rank, training duration, and Hessian curvature. It also describes the Lanczos-based procedure used to estimate Hessian maximum eigenvalues.

  • I. Prioritize Learning Rate Tuning: Prioritize learning-rate tuning while fixing batch size when computational resources are limited.The authors caution that overly large batch sizes can reduce the best achievable performance even after learning-rate tuning.
  • II. Mind Batch Size Scaling: After tuning learning rate for each batch size, additional batch-size exploration is likely to yield only marginal gains.Batch-size scaling with learning rate remains useful for choosing an initial learning-rate guess.
  • III. Select Learning Rate based on Hessian: The maximum Hessian eigenvalue can indicate a variant’s relative operating learning-rate range compared with vanilla LoRA.The implementation estimates λmax with Lanczos iterations using Hessian-vector products rather than explicitly forming the Hessian.
  • IV. Increase LoRA Ranks: Increasing LoRA rank can improve downstream performance after learning-rate tuning, but the higher rank still requires renewed learning-rate tuning.The recommendation applies when performance remains unsatisfactory at a tuned lower rank.
  • IV. Increase LoRA Ranks: Optimal learning rate generally decreases as rank increases; vanilla LoRA at r = 4 requires 5.6× the learning rate used at r = 256.This rank-dependent relationship is intended to make tuning across ranks more efficient.
  • V. Prolong Training Duration: With proper learning-rate tuning, longer training can further improve LoRA methods, while the optimal learning-rate range shifts only modestly downward.The optimal rates for N = 5k and N = 395k differ by only 3.1×, so shorter-training ranges can provide a starting point.

I.2 Hessian Results on Gemma and Llama

Hessian analyses on Gemma-3-1B and Llama-2-7B show that variants’ curvature patterns explain differences in their optimal learning-rate ranges. PiSSA has substantially larger curvature than vanilla LoRA, whereas MiLoRA and Init[AB] are more similar.

  • Cross-model Hessian relationship: Hessian relationships are generally negatively correlated with each method’s optimal learning rate on Gemma-3-1B and Llama-2-7B.The distributions shown for these models extend the corresponding analysis from Qwen3-0.6B.
  • Variant comparisons: PiSSA exhibits substantially larger λmax on both models and requires 1.8–2× smaller learning rates than vanilla LoRA.The curvature difference aligns with the learning-rate results reported in Table 1 and Figure 3a.
  • Variant comparisons: MiLoRA and Init[AB] have λmax values that largely overlap vanilla LoRA’s, especially on Llama-2-7B.Their similar curvature corresponds to similar optimal learning-rate ranges.
  • Layer-wise patterns: Across Transformer layers, all methods tend to show high or low λmax values at similar layer locations.For Qwen3-0.6B, the layer-wise Query-projection heatmap provides the detailed breakdown of these patterns.
  • Layer-wise patterns: PiSSA, OLoRA, and LoRA-GA consistently exhibit larger eigenvalues than vanilla LoRA across layers and projection types.The reported distributions and heatmaps cover Query and Key projection matrices.
Loading 2602.04998v2…