PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

arXiv:2106.09685v2cs.CLcs.AIcs.LG

TL;DR

Large language models make full fine-tuning costly in hardware, storage, and task-specific deployment. LoRA freezes pretrained weights and learns low-rank update matrices, reducing resource use while preserving efficient deployment and task switching.

  • Problem

    Full fine-tuning of enormous language models is prohibitively expensive for hardware, storage, and hosting independent task-specific instances.

  • Method

    LoRA freezes pretrained weights and adapts dense layers by optimizing trainable rank-decomposition matrices representing the weight changes.

  • Results

    GPT-3 175B training VRAM fell from 1.2TB to 350GB, while a rank-4 checkpoint using query and value projections shrank roughly 10,000× from 350GB to 35MB.

  • Takeaways & Limitations

    LoRA enables lower-cost deployment and quick task switching by sharing most model parameters and swapping only the LoRA weights.

  • Takeaways & Limitations

    Absorbing task-specific LoRA matrices into the base weights makes batching different tasks in one forward pass nontrivial.

Abstract

from arXiv · show

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

1 INTRODUCTION

Adapting large pre-trained language models to many downstream tasks makes full fine-tuning costly in parameters, storage, and deployment. LoRA freezes the shared model and learns low-rank updates, reducing adaptation overhead while preserving efficiency and quality goals.

  • 175 billion trainable parameters make independent GPT-3 fine-tuned instances a critical deployment challenge.
  • Existing efficient adaptation methods can add inference latency, reduce usable sequence length, or fail to match fine-tuning quality.
  • LoRA freezes pre-trained weights and optimizes rank-decomposition matrices representing the weight changes during adaptation.The approach hypothesizes that adaptation updates have low intrinsic rank.
  • A very low rank, such as r=1 or 2, can suffice for GPT-3 175B even when the full rank is 12,288.
  • LoRA enables task switching by replacing small matrices A and B while sharing the frozen pre-trained model.
  • LoRA lowers adaptive-optimizer hardware requirements by up to 3 times and introduces no inference latency after merging its trainable matrices with frozen weights.

2 PROBLEM STATEMENT

The paper formulates downstream adaptation as maximizing conditional language-model probabilities, then replaces full parameter optimization with a much smaller task-specific representation. LoRA encodes this representation with a compute- and memory-efficient low-rank update.

  • Downstream conditional text-generation tasks are represented by context-target pairs and adapted through a conditional language-modeling objective.
  • Full fine-tuning updates all pre-trained parameters, so each task requires a separate parameter set as large as the original model.For GPT-3, the pre-trained model has approximately 175 billion parameters.
  • LoRA encodes each task-specific parameter increment with a much smaller parameter set Θ, where |Θ| ≪ |Φ0|.
  • For GPT-3 175B, the low-rank representation can reduce trainable parameters to as little as 0.01% of the pre-trained model.

3 AREN’T EXISTING SOLUTIONS GOOD ENOUGH?

Existing parameter-efficient adaptation methods reduce stored task-specific parameters but remain constrained by inference latency, sequence-length costs, or weaker performance than full fine-tuning.

  • Adapter layers: Adapter layers add sequential computation, increasing latency in online inference scenarios with small batch sizes.Their small parameter count does not eliminate the extra computation, which cannot be directly bypassed.
  • Adapter layers: Model sharding worsens adapter overhead because added depth requires more synchronous GPU operations or redundant adapter storage.
  • Direct prompt optimization: Prefix tuning is difficult to optimize, and its performance varies non-monotonically with the number of trainable parameters.
  • Direct prompt optimization: Reserving sequence positions for prefix adaptation reduces the sequence length available for downstream tasks.
  • Empirical comparison: Table 1 measures GPT-2 medium single-forward-pass latency in milliseconds across 100 trials, comparing adapter variants by trainable-parameter count.

4 OUR METHOD

LoRA freezes pretrained weights and represents adaptation updates with trainable low-rank matrices, which can be merged at deployment while reducing training and storage costs.

  • Low-rank update matrices: LoRA trains dense-layer updates indirectly through rank-decomposition matrices while keeping pretrained weights frozen.The approach is motivated by the hypothesis that adaptation updates have low intrinsic rank.
  • Low-rank update matrices: LoRA initializes A randomly and B to zero, making the initial weight update ∆W = BA zero before training.
  • Applying LoRA to Transformer: LoRA can recover full fine-tuning’s expressiveness by setting rank r to the rank of the pretrained weight matrices when adapting all weight matrices and biases.The paper’s experiments focus on selected Transformer attention weights, while MLP layers, LayerNorm layers, and biases are left for future work.
  • Applying LoRA to Transformer: LoRA merges W0 and BA into W before inference, enabling task switching by replacing low-rank matrices without additional inference latency.The merged design supports switching tasks by subtracting one update and adding another with little memory overhead.
  • Practical benefits and limitations: On GPT-3 175B, LoRA reduces training VRAM from 1.2TB to 350GB, cuts checkpoint size roughly 10,000×, and speeds training by 25% versus full fine-tuning.The reported checkpoint reduction uses r = 4 while adapting only query and value projection matrices.
  • Practical benefits and limitations: Merging task-specific matrices makes batching inputs from different tasks in one forward pass nontrivial.The paper notes that unmerged weights permit dynamic module selection when latency is not critical.

5 EMPIRICAL EXPERIMENTS

The experiments evaluate LoRA across encoder and generative models, including GPT-3 175B, against fine-tuning and parameter-efficient baselines. Across these settings, LoRA maintains competitive task performance while reducing adaptation overhead and avoiding added inference latency.

  • Experimental Setup: LoRA is evaluated on GLUE, E2E NLG, WikiSQL, MultiNLI-matched, and SAMSum across RoBERTa, DeBERTa, GPT-2, and GPT-3 175B.The evaluation spans natural language understanding and natural language generation tasks.
  • Efficiency: LoRA’s trainable matrices can be merged with frozen weights at deployment, introducing no inference latency relative to a fully fine-tuned model.This distinguishes the design from adapter layers that add computation during inference.
  • Baselines: LoRA adds trainable rank-decomposition matrix pairs to existing weight matrices, with trainable-parameter count determined by rank and adapted-weight shape.Most experiments apply LoRA to Wq and Wv for simplicity.
  • GPT-2: LoRA outperforms several GPT-2 adaptation baselines on the E2E NLG Challenge with comparable or fewer trainable parameters.The table reports results for GPT-2 medium and large, with confidence intervals for experiments run by the authors.
  • GPT-3 175B: LoRA matches or exceeds the fine-tuning baseline on all three GPT-3 175B datasets.Table 4 covers WikiSQL, MultiNLI-matched, and SAMSum, with task-specific validation or Rouge metrics.

6 RELATED WORKS

Related work frames language-model adaptation around increasingly large Transformer models, full fine-tuning, prompt-based methods, and inserted adapters. LoRA combines parameter-efficient adaptation with weight merging to avoid adapter inference latency.

  • Transformer Language Models: Transformer-based language models use self-attention and have become dominant across many NLP tasks.BERT and GPT-2 exemplify large Transformers pretrained on substantial text corpora.
  • Prompt Engineering and Fine-Tuning: Fine-tuning adapts a generally pretrained model to a specific task but becomes difficult for GPT-3 175B because each task-specific checkpoint retains the full model footprint.Prompt engineering offers another adaptation route, but its result depends heavily on prompt composition and formatting.
  • Parameter-Efficient Adaptation: Adapter methods insert bottleneck layers between existing network layers, whereas LoRA imposes a low-rank constraint on weight updates.LoRA’s learned weights can be merged with the main weights during inference.
  • Low-Rank Structures in Deep Learning: Low-rank structure is common in machine learning, and prior work reports low-rank properties in learned over-parameterized neural networks.These observations motivate studying low-rank structure in model adaptation.

7 UNDERSTANDING THE LOW-RANK UPDATES

The section empirically investigates which Transformer updates are useful, whether adaptation matrices have low intrinsic rank, and how they relate to pretrained weights. The studies find that small-rank updates capture meaningful task-relevant directions and amplify features not emphasized during pretraining.

  • Research questions: The study asks which attention weights to adapt, whether ΔW is rank-deficient, and how ΔW relates to W.The experiments use fixed parameter budgets and analyze subspace overlap, correlations, and relative magnitudes.
  • Weight selection: Adapting both Wq and Wv gives the best overall validation performance under the same parameter budget on WikiSQL and MultiNLI.The GPT-3 comparison uses a 18M-parameter budget, corresponding to r = 8 for one attention-weight type or r = 4 for two types.
  • Rank selection: Rank 1 suffices for adapting both Wq and Wv on WikiSQL and MultiNLI, whereas adapting Wq alone requires a larger rank.This result suggests that the effective update can have a very small intrinsic rank, though the authors caution that this may not generalize to every task or dataset.
  • Subspace similarity: Top singular-vector directions overlap significantly between rank-8 and rank-64 updates, with one shared dimension exceeding normalized similarity 0.5 for both ΔWv and ΔWq.The authors interpret other directions as potentially containing mostly random training noise, explaining why very low ranks can perform well.
  • Subspace similarity: Different-seed ΔWq updates share more singular directions than ΔWv updates, while random Gaussian matrices share no common singular-value directions.This pattern indicates a higher intrinsic rank for ΔWq than for ΔWv.
  • Relationship to pretrained weights: ΔW correlates more strongly with W than a random matrix, amplifies directions not emphasized in W, and reaches an amplification factor of 21.5 for r = 4.The reported factor is 21.5 ≈ 6.91/0.32, supporting the view that adaptation emphasizes task-relevant features learned but underemphasized during general pretraining.

8 CONCLUSION AND FUTURE WORK

The conclusion presents LoRA as an efficient adaptation strategy that reduces deployment costs while preserving model quality and enabling rapid task switching. It also identifies unresolved mechanisms, heuristic weight selection, and future combinations with other adaptation methods.

  • Conclusion: LoRA addresses the hardware, storage, and switching costs of fine-tuning enormous language models without adding inference latency or reducing input sequence length.The approach shares most model parameters across tasks and stores task-specific low-rank modules.
  • Conclusion: LoRA enables quick task switching by sharing the vast majority of model parameters across task-specific adaptations.The shared pretrained model remains fixed while task-specific modules are replaced.
  • Future work: The paper suggests combining LoRA with other efficient adaptation methods as a direction for potentially orthogonal improvement.The authors specifically note that LoRA is orthogonal to methods such as prefix-tuning.
  • Future work: The mechanism by which pretrained features are transformed for downstream tasks remains unclear, although LoRA may make that mechanism more tractable to study.This is presented as an open research question rather than a resolved explanation.
  • Future work: Weight-matrix selection for applying LoRA depends mostly on heuristics, motivating more principled selection methods.The authors also suggest investigating whether pretrained weights themselves may be rank-deficient.

B INFERENCE LATENCY INTRODUCED BY ADAPTER LAYERS

Adapter layers add computation sequentially to the base model, creating inference latency that LoRA avoids through parallel insertion and weight merging. The measured slowdown varies with workload and can be substantial for online short sequences.

  • Latency mechanism: Adapter layers are computed in addition to the base model, so they inevitably introduce inference latency.LoRA modules are added in parallel and can be merged with the frozen weights when deployed.
  • Experimental setup: The latency experiment averages 100 forward passes on an NVIDIA Quadro RTX8000 while varying batch size, sequence length, and adapter bottleneck dimension.It compares the AdapterH and AdapterL designs against a no-adapter baseline.
  • Findings: Larger batch sizes and sequence lengths mitigate adapter latency, but online short-sequence scenarios can incur slowdowns above 30%.Figure 5 reports percentage slow-down relative to the r = 0 no-adapter baseline for AdapterH and AdapterL.

C DATASET DETAILS

This section defines the benchmarks, datasets, evaluation settings, and training procedures used across the paper’s RoBERTa, DeBERTa, GPT-2, and GPT-3 experiments. It covers NLU, text-to-SQL, summarization, and data-to-text tasks.

  • Benchmarks: GLUE evaluates NLU across MNLI, SST-2, MRPC, CoLA, QNLI, QQP, RTE, and STS-B.Its broad task coverage makes it a standard evaluation benchmark for models such as RoBERTa and DeBERTa.
  • Datasets: WikiSQL contains 56,355 training and 8,421 validation examples for generating SQL from natural-language questions and table schemas.The input combines the table schema and query, while the target is an SQL statement.
  • Datasets: SAMSum contains 14,732 training and 819 test examples of staged two-person chats paired with abstractive summaries.Inputs concatenate utterances, and targets are summaries written by linguists.
  • Datasets: E2E, DART, and WebNLG evaluate data-to-text generation using restaurant-domain records, entity-relation triples, and subject-property-object triples.DART has 82K examples, while WebNLG has 22K examples across seen and unseen categories.
  • Training procedures: RoBERTa and DeBERTa experiments use AdamW with linear learning-rate decay, tuned task hyperparameters, frozen pretrained models, and median results over five seeds.LoRA modules are initialized from the best MNLI checkpoint for MRPC, RTE, and STS-B.
  • Training procedures: GPT-2 experiments use AdamW for five epochs and report means over three random seeds, while GPT-3 experiments use AdamW for two epochs with task-specific sequence lengths.GPT-3 uses sequence lengths of 384 for WikiSQL, 768 for MNLI, and 2048 for SAMSum.

E COMBINING LORA WITH PREFIX TUNING

The paper evaluates combining LoRA with prefix-based adaptation on WikiSQL and MultiNLI, finding that the combinations are not uniformly beneficial. Prefix-embedding tuning complements LoRA on WikiSQL, whereas prefix-layer tuning is sensitive to learning-rate choice.

  • Evaluation: The evaluation compares LoRA+PE and LoRA+PL with LoRA and prefix-based baselines on WikiSQL and MultiNLI.The experiments use the GPT-3 adaptation hyperparameters reported for the evaluated methods.
  • Combination methods: LoRA+PE combines LoRA with prefix-embedding tuning by making inserted special-token embeddings trainable.The combination inserts l_p + l_i special tokens whose embeddings are treated as trainable parameters.
  • Combination methods: LoRA+PL combines LoRA with prefix-layer tuning by replacing special-token representations after every Transformer block with trainable input-agnostic vectors.Both the token embeddings and subsequent Transformer-block activations are trainable parameters.
  • Results: LoRA+PE significantly outperforms LoRA and prefix-embedding tuning on WikiSQL, indicating that the two approaches provide complementary adaptation mechanisms.On MultiNLI, LoRA+PE does not outperform LoRA, which already reaches performance comparable to the human baseline.
  • Results: LoRA+PL performs slightly worse than LoRA despite using more trainable parameters.The authors attribute this result to prefix-layer tuning's sensitivity to the learning rate, which complicates optimization of LoRA weights.

F.1 ADDITIONAL EXPERIMENTS ON GPT-2

Additional experiments examine LoRA against prefix-based methods, the rank required for adaptation, and the structure of learned low-rank updates. Across these analyses, LoRA remains competitive while the update matrices concentrate on task-specific directions whose effective intrinsic rank is low.

  • Additional adaptation experiments: LoRA performs better than or at least on-par with prefix-based approaches on DART and WebNLG when the methods use the same number of trainable parameters.The experiments follow the setup of Li & Liang (2021).
  • Low-data evaluation: LoRA exhibits favorable sample-efficiency compared with other adaptation methods, including fine-tuning, on low-data MNLI subsets.The subsets contain 100, 1k, or 10k randomly sampled training examples, evaluated on the full validation set.
  • Measuring subspaces: The subspace similarity measure equals 1 for identical column spans, 0 for completely orthogonal spans, and lies between 0 and 1 otherwise.It compares subspaces formed from columns of left singular matrices and reverses the standard Projection Metric's distance interpretation.
  • Rank analysis: The optimal rank for GPT-2 Medium is between 4 and 16 depending on the metric, similar to GPT-3 175B.On E2E NLG, the reported performance peaks at r = 16 for validation loss and r = 4 for BLEU.
  • Update-matrix analysis: The top four directions in ∆W have normalized similarity barely above 0.2 with the top 10% of W's directions, supporting task-specific rather than dominant pretrained directions.The analysis uses Figure 8 to compare singular directions of W and ∆W across varying ranks.
  • Update-matrix analysis: For r = 4, the feature amplification factor reaches 20, whereas for r = 64 it is around 2.The authors interpret the contrast as evidence that the intrinsic rank needed to represent task-specific adaptation directions is low.
Loading 2106.09685v2…