PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRA Learns Less and Forgets Less

Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, John P. Cunningham

arXiv:2405.09673v2cs.LGcs.AIcs.CL

TL;DR

The paper asks whether LoRA compromises target-domain performance relative to full finetuning and how the methods differ in source-domain forgetting. It compares them across code and mathematics under continued pretraining and instruction finetuning, finding that LoRA learns less but forgets less, while full finetuning uses much higher-rank perturbations. The authors use these results to motivate practical guidance for configuring LoRA.

  • Problem

    Evidence remains mixed on whether LoRA matches full finetuning, while its effect on forgetting modern language-model capabilities has received limited study.

  • Method

    The paper compares LoRA and full finetuning for Llama-2-7B across code and mathematics, using continued pretraining and instruction finetuning and evaluating learning and forgetting.

  • Results

    LoRA underperforms full finetuning in commonly used low-rank settings, while preserving source-domain performance, reducing forgetting, and generating more diverse outputs.

  • Takeaways & Limitations

    Full finetuning finds perturbations 10-100× higher rank than typical LoRA configurations, and LoRA performance depends strongly on hyperparameter choices.

  • Takeaways & Limitations

    The spectral analysis covers continued pretraining only and does not rule out low-rank solutions or lower ranks sufficient for downstream tasks.

Abstract

from arXiv · show

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.

1 Introduction

This study compares LoRA with full finetuning for Llama-2-7B across code and mathematics, using continued pretraining and instruction finetuning. Standard low-rank LoRA learns less but forgets less, while full finetuning finds high-rank perturbations and LoRA requires careful hyperparameter tuning.

  • Motivation: LoRA reduces training memory by freezing the pretrained model and learning low-rank adapters on selected weight matrices.This parameter-efficient design trains only a small number of additional parameters.
  • Study design: The study evaluates code and mathematics with continued pretraining and instruction finetuning, measuring target-domain learning and source-domain forgetting.Target performance is evaluated on HumanEval and GSM8K, while source-domain performance covers language understanding, world knowledge, and common-sense reasoning.
  • Learning-forgetting tradeoff: LoRA forgets less of the source domain and maintains more diverse generations than full finetuning, while mitigating forgetting more than dropout and weight decay.Full finetuning tends to produce fewer unique HumanEval generations than the base model, whereas LoRA produces a wider range.
  • Perturbation structure: Full finetuning learns perturbations with ranks 10-100× higher than typical LoRA configurations, despite barely changing the base weight spectra.The perturbation rank grows as training progresses, potentially explaining performance differences.
  • Practical recommendations: LoRA is highly sensitive to learning rates, target modules, ranks, and scaling factors, making proper configuration necessary to approach full finetuning performance.The paper concludes by proposing practical best practices for these hyperparameters.
  • Learning performance: Standard low-rank LoRA substantially underperforms full finetuning, although higher ranks can close most gaps in instruction finetuning.In continued pretraining, the gap remains even with high ranks; in instruction finetuning, high ranks can match full finetuning.

2 Background

LoRA freezes pretrained weights and learns low-rank perturbations for selected transformer modules, reducing trainable parameters and gradient-computation costs. Modern practice commonly targets all transformer modules rather than only attention projections.

  • LoRA formulation: LoRA freezes W_pretrained and learns a low-rank perturbation ∆ represented by factor matrices A and B.The perturbation is scaled by γ/r and uses A ∈ R^(d×r) and B ∈ R^(r×k).
  • Efficiency: Training LoRA adapts d × r + r × k parameters per module instead of d × k, reducing memory and gradient-computation FLOPS.With r = 16 and d = k = 4096, the adapter trains less than 1% of the original parameter count.
  • Initialization and scaling: Common implementations initialize A from a normal distribution, initialize B to zero, and set γ_r = α/r using a controllable hyperparameter α.The user chooses the target modules, rank r, and α.
  • Target modules: LoRA originally targeted W_q and W_v in self-attention, whereas later best practice targets all transformer modules.This broader selection includes feedforward modules across the model’s layers.

3 Experimental Setup

The experiments compare code and mathematics adaptation using continued pretraining and instruction-finetuning datasets, then measure target-domain learning and degradation on general benchmarks. Code and math evaluations use HumanEval and GSM8K, while forgetting is assessed with HellaSwag, WinoGrande, and ARC-Challenge.

  • Overview: The study trains on code and mathematics datasets and evaluates learning and forgetting with benchmarks reported for state-of-the-art open-source LLMs.Training uses the MosaicML Composer, Streaming, and LLM Foundry repositories together with HuggingFace PEFT.
  • Training data: Coding continued pretraining uses a Python subset of permissively licensed GitHub repositories, subsampled to 20B tokens.The source includes repositories and Git commits spanning more than 80 programming languages.
  • Training data: Mathematics continued pretraining uses up to 20B tokens from OpenWebMath, repeating tokens after its initial 14.7B tokens.The dataset is derived from mathematical web pages and preserves mathematical content such as LaTeX equations.
  • Training data: Coding instruction finetuning uses Magicoder-Evol-Instruct-110k, containing 72.97M tokens of programming questions and answers.The dataset reproduces WizardCoder’s Evol-Instruct data by iteratively increasing Code Alpaca question difficulty with GPT-4.
  • Training data: Math instruction finetuning uses MetaMathQA, containing 395K question-answer pairs and roughly 103M tokens bootstrapped from GSM8K and MATH problems.GPT-3.5 rewrites the source questions with variations.
  • Evaluation: Target-domain performance is evaluated with HumanEval pass@1 and GSM8K pass@1, while forgetting is measured on HellaSwag, WinoGrande, and ARC-Challenge accuracy.HumanEval evaluates Python generation, GSM8K evaluates grade-school math problems, and the three forgetting benchmarks test commonsense, pronoun resolution, and science reasoning.

4 Results

Across code and mathematics, LoRA’s target-domain performance depends strongly on rank and training regime: standard low-rank configurations often trail full finetuning, while LoRA generally forgets less and preserves more diverse generations. Full-finetuning perturbations become high rank early in training, motivating careful choices of rank, scaling, learning rate, and target modules.

  • Target-domain performance: IFT improves evaluation scores more than CPT in both domains; code IFT reaches HumanEval 0.497 versus 0.263 for CPT.
  • Target-domain performance: At 20B tokens, Code CPT reaches HumanEval=0.224 with LoRA r = 256 versus 0.263 with full finetuning.The best LoRA model roughly matches full finetuning trained on 4B tokens, which reaches HumanEval=0.218.
  • Target-domain performance: In Code IFT, LoRA r = 256 matches full finetuning, reaching HumanEval 0.498 versus 0.497, whereas r = 16 and r = 64 score 0.358 and 0.417.
  • Target-domain performance: In Math CPT, LoRA r = 256 peaks at GSM8K=0.203, below full finetuning at 4B tokens (0.224) and its 20B-token peak (0.293).
  • Target-domain performance: In Math IFT, LoRA r = 256 peaks at GSM8K=0.634, close to full finetuning’s peak of 0.642, while r = 64 reaches 0.624 at epoch 4.
  • Learning-forgetting tradeoff: LoRA forgets less than full finetuning, with forgetting controlled by rank; in Code CPT at 20B tokens, scores are 0.617 for LoRA r = 256 versus 0.545 for full finetuning.Forgetting is also greater in IFT than CPT and in programming than mathematics, and generally worsens with training duration.
  • Learning-forgetting tradeoff: Increasing LoRA rank or training duration moves models toward learning more and forgetting more, with this pattern more consistent in IFT than CPT.
  • Learning-forgetting tradeoff: Dataset-specific tradeoff patterns make it difficult to conclude whether LoRA and full finetuning have fundamentally different learning-forgetting tradeoffs.

5 Related Work

Prior work often reported LoRA matching full finetuning on relatively easy or smaller-model benchmarks, but evidence for challenging code and mathematics domains remains limited. Related research also studies continual learning, replay, alternative parameter-efficient methods, and LoRA configuration choices.

  • LoRA extensions: LoRA extensions address initialization, scaling, sequential training, architectural changes, and alternative low-rank approximations.
  • Benchmark comparisons: The original LoRA work and many subsequent studies reported performance matching or exceeding full finetuning on GLUE, generation, SQL, summarization, and related benchmarks.
  • Benchmark comparisons: Earlier benchmarks often used models smaller than 340M parameters or tasks considered trivial for modern billion-parameter LLMs.
  • Benchmark comparisons: Only a few studies rigorously compare LoRA with full finetuning for large LLMs on challenging code and mathematics domains.
  • Configuration choices: Prior studies report mixed conclusions about LoRA target modules and rank, while this work finds that scaling α to 2r is necessary to unlock improvements by rank.
  • Continual learning: Continual-learning research specializes language models for code and mathematics through continued pretraining, instruction finetuning, synthetic data, and related pipelines.
  • Continual learning: Prompt tuning has been reported to mitigate forgetting in cross-lingual summarization, extending interest in parameter-efficient methods beyond LoRA.
  • Continual learning: Replay of source-domain data, using stored or generated examples, is a common approach for mitigating forgetting during continual learning.

6 Discussion

The study leaves model-size scaling unresolved and qualifies its spectral analysis: high-rank perturbations are typical in continued pretraining but may not be necessary for downstream tasks.

  • Model-size effects on the LoRA–full-finetuning difference remain unstudied rigorously and are left for future work.The discussion notes prior hints of scaling relationships and successful LoRA applications to 70B models, but does not provide a systematic study.
  • High-rank perturbations found by full finetuning do not rule out low-rank solutions, because downstream tasks may require less rank than full weight reconstruction.The spectral analysis characterizes solutions typically found, not the minimum rank sufficient for a task.
  • The spectral analysis covers continued pretraining only, so instruction finetuning may produce less high-rank full-finetuning solutions.

7 Conclusion

Across code and mathematics, commonly configured LoRA underperforms full finetuning while preserving more base-model behavior. The paper links this tradeoff to hyperparameter sensitivity and perturbation rank.

  • LoRA with commonly used low-rank settings underperforms full finetuning across code and mathematics domains.
  • LoRA preserves behavior closer to the base model, with diminished source-domain forgetting and more diverse generations at inference time.
  • LoRA mitigates forgetting more than classical regularization techniques such as dropout and weight decay.
  • Full finetuning finds weight perturbations that are far from low-rank, while LoRA requires careful hyperparameter choices and motivates best practices.

A Experimental Setup

The experiments compare Llama-2-7B LoRA and full finetuning across code and mathematics, using continued pretraining and instruction-finetuning settings with controlled training configurations.

  • LoRA configuration: LoRA trains ranks r = 16, 64, 256 across transformer modules with α = 2r and lora_dropout=0.05.
  • Training procedure: Continued-pretraining experiments train for 20B tokens and use intermediate-checkpoint cooldowns for selected maximum durations.
  • Model and datasets: All experiments use the Llama-2-7B base model, with separate base and chat tokenizers for continued pretraining and instruction finetuning.
  • Model and datasets: The study covers Code CPT on StarCoder-Python and Math CPT on OpenWebMath, alongside Code IFT and Math IFT datasets.
  • Hyperparameter sensitivity: LoRA learning rates are more sensitive than full-finetuning rates, with best LoRA rates of 5e−4 for code and 2e−4 for math.The corresponding best full-finetuning rates are 5e−5 and 1e−5, respectively.

B.1 Learning rate sensitivity analysis across optimizers

The sensitivity analyses examine optimizer, learning-rate, and α choices, then extend evaluation to the mixed-domain Tülu-v2-mix setting. LoRA is especially sensitive to learning rate and α scaling.

  • Optimizer comparison: Decoupled LionW performs better than AdamW on HumanEval for both LoRA and full finetuning across learning rates.
  • Alpha sensitivity: Across Magicoder-Evol-Instruct-110K and OpenWebMath, the highest LoRA accuracy occurs at α = 2r.
  • Tülu-v2-mix evaluation: The Tülu-v2-mix experiment compares full finetuning with LoRA ranks r = 16, 64, 256 over up to 6 epochs.It evaluates math, broad academic capabilities, conversation, and related averages.

C.2 Results

Across supplementary evaluations, LoRA often matches full finetuning on conversational and some task-specific benchmarks, while requiring more training and showing rank-dependent forgetting patterns.

  • MT-bench: LoRA and full finetuning both improve MT-bench over the base model, with all LoRA models within one standard error of full finetuning.The base model scores 2.74, and the benchmark averages 160 datapoints from 80 questions with two turns each.
  • GSM8K: On GSM8K, LoRA and full finetuning overlap, with LoRA r = 256 at epoch 4 achieving the best model result.Full finetuning at epoch 2 is the next-best model, and math results are ordered by LoRA rank.
  • MMLU: On MMLU, LoRA and full finetuning overlap, with LoRA r = 64 at epoch 4 outperforming the other reported models.Unlike GSM8K, MMLU shows no ordering by rank.
  • Training duration: At epoch 2, full finetuning is best across learning and forgetting evaluations, whereas LoRA needs 4 epochs and later remains competitive conversationally but weaker on math knowledge.At epoch 6, full finetuning forgets the most, while forgetting is ordered by LoRA rank.
  • Tülu-v2-mix: On Tülu-v2-mix, LoRA and full finetuning both improve upon the base model and perform comparably.The supplementary figures also report that LoRA forgets less on this more diverse instruction-finetuning dataset.

E Supplementary Figures for SVD Analysis

The supplementary analyses examine matrix spectra, solution-generation diversity, dataset construction, and practical LoRA training trade-offs.

  • SVD analysis: For the 4096 × 4096 Wq matrix, rank > 1500 is needed to explain 90% of the variance in the base weights, finetuned weights, and their difference.The figure compares singular values on the left with cumulative explained variance on the right.
  • SVD analysis: The perturbation matrices are high rank, and A, cB, and A + cB are all high rank in the supplementary spectral analysis.The analysis studies sums of random 1000 × 1000 Gaussian matrices as a comparison.
  • Solution generation diversity: In HumanEval, pass@k evaluates how performance changes as k controls the accepted sample-set size across model generations.Increasing k raises generation diversity and the chance that a passing generation appears in a random subset.
  • Solution generation diversity: For Magicoder-trained models, pass@k increases monotonically with k; full finetuning leads at k = 1, while the gap narrows as k increases.Finetuned models score substantially above the base model, and LoRA scores are ordered by rank at k = 1.
  • Training efficiency: LoRA reduces training memory, but can produce slower throughput because its computations add intermediate-activation overhead.The supplementary discussion reports memory savings alongside possible throughput slowdowns.

H.2 Training on Multiple GPUs with Fully Sharded Data Parallelism

Across multiple GPUs, FSDP can reduce per-GPU memory more than LoRA in some settings, while LoRA supports training with fewer or lower-memory GPUs.

  • Comparison with FSDP: FSDP shards parameters, gradients, and optimizer states across GPUs and can be competitive with LoRA’s memory savings in certain settings.Model parallelism has high communication overhead and poor throughput by comparison.
  • Comparison with FSDP: LoRA enables training on GPUs with far less memory and on fewer GPUs, whereas FSDP can use less memory per GPU when sharded across many devices.The relative advantage depends on the deployment configuration.
  • 7B model: For a 7B model with Adam on 8 GPUs, full finetuning with FSDP requires roughly 14 GB per GPU, compared with 1.89 GB for LoRA.The LoRA setup uses 70M additional parameters; with 32 GPUs, the figures become 3.5 GB and 0.4725 GB respectively.
  • 70B model: For a 70B model on 8 GPUs, Adam requires 140 GB per GPU, whereas Adam+LoRA requires 18.9 GB per GPU.The authors state that finetuning this model on 8 GPUs is only possible using a technique like LoRA.
  • Scaling considerations: LoRA’s efficiency gains relative to FSDP depend on model size and GPU availability or cost considerations.The supplementary analysis also extends the comparison to a 405B model.

I LoRA Throughput and Memory Measurements

Throughput and memory measurements show a trade-off: LoRA substantially lowers peak memory at small batch sizes but generally reduces throughput relative to full finetuning.

  • Measurement setup: The measured throughput comparison uses tokens per second across 8 H100-80GB GPUs, with LoRA targeting all linear-layer weights.The runs use per-GPU micro batch size 1 and include both attention and MLP modules.
  • Throughput: LoRA training runs approximately 15% lower throughput than full finetuning for a given batch size.Higher LoRA ranks are slower across batch sizes, especially rank r = 512.
  • Implementation caveat: The throughput gap is partly attributed to additional LoRA computations, though the implementation was not optimized and may permit further improvements.The measurements use the publicly available HuggingFace peft library.
  • Peak memory: At small batch sizes, LoRA reduces peak memory by approximately 40% because its optimizer state is substantially smaller.As batch size increases, intermediate activations grow and dominate memory requirements.
Loading 2405.09673v2…