PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning

Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li

arXiv:2308.03303v3cs.CL

TL;DR

Full-FT is costly, and LoRA can lag behind it because of low-rank gradient limitations. The paper introduces LoRA-FA, which freezes A, corrects B’s gradient in closed form, and achieves comparable performance with lower activation memory and computational cost.

  • Problem

    Full-FT is computationally expensive and memory-intensive, while LoRA may underperform Full-FT because of intrinsic low-rank gradient limitations.

  • Method

    LoRA-FA freezes the projection-down matrix A, trains B, and uses closed-form gradient corrections to reduce the discrepancy from the Full-FT gradient.

  • Results

    Across GLUE, domain-specific tasks, and other evaluated settings, LoRA-FA matches or competitively performs against PEFT methods and Full-FT while reducing activation memory and computational cost.

  • Takeaways & Limitations

    LoRA-FA provides an efficient fine-tuning approach that preserves strong performance while reducing activation storage and computational workload.

  • Takeaways & Limitations

    LoRA-FA cannot eliminate activations unrelated to A and may produce suboptimal results when base-model capacity or dataset demands exceed low-rank adaptation.

Abstract

from arXiv · show

Fine-tuning large language models (LLMs) is crucial for improving their performance on downstream tasks, but full-parameter fine-tuning (Full-FT) is computationally expensive and memory-intensive. Parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address this by optimizing only a small subset of parameters. However, LoRA may underperform Full-FT in certain scenarios due to the intrinsic limitations of its low-rank gradients. In this work, we reveal an asymmetric, collapsible structure in LoRA's update: the low-rank modification to W can be reformulated as a single-layer linear regression, implying that one of the LoRA factors can be frozen without sacrificing expressivity. Leveraging this insight, we introduce LoRA-FA, which freezes the projection-down matrix A and trains only the projection-up matrix B. We further close the gap to Full-FT by deriving closed-form gradient corrections that minimize the discrepancy between the induced low-rank gradient and the full gradient. Through extensive experiments on diverse benchmarks, including GLUE, GSM8K, MT-Bench, and HumanEval, we demonstrate that LoRA-FA consistently achieves comparable performance to existing PEFT methods and Full-FT. Experiments on system efficiency show that LoRA-FA significantly reduces activation memory consumption and computational workload in fine-tuning. Our code is available at https://github.com/huggingface/peft.

1 INTRODUCTION

Full-parameter fine-tuning is costly, while LoRA reduces trainable-state memory but can lag behind Full-FT. LoRA-FA exploits an asymmetric, collapsible update structure to freeze one factor, approximate Full-FT gradients, and improve efficiency.

  • Fine-tuning LLMs improves downstream performance, but Full-FT can require more than 1 TB of GPU memory for a Llama3-70B model.
  • LoRA freezes W and updates smaller matrices A and B, reducing trainable-parameter, gradient, and optimizer-state memory relative to Full-FT.
  • LoRA can underperform Full-FT because its low-rank gradient structure leaves asymmetric roles for A and B underexploited.
  • LoRA-FA freezes A and trains B after expressing LoRA’s update as a single-layer linear regression, while closed-form corrections approximate Full-FT gradients.
  • Across diverse experiments, LoRA-FA is reported to deliver competitive performance while reducing activation memory and computational workload compared with other LoRA variants.

2 RELATED WORK

Related work develops LoRA extensions along adaptation, asymmetry, gradient approximation, and efficiency dimensions. LoRA-FA combines asymmetric factor freezing with gradient correction to address both low-rank limitations and activation costs.

  • Low-Rank Adaptation: LoRA reduces optimizer-state and gradient memory by inserting low-rank adapters alongside frozen model layers.
  • Asymmetric LoRA: Asymmetric LoRA methods exploit different contributions from A and B through unequal optimization, shared factors, or multiple B modules.
  • Gradient approximation for LoRA: Gradient-approximation methods improve LoRA updates by matching optimization trajectories or approximating Full-FT gradients.
  • Existing variants can be limited by low-rank update subspaces and by storing forward activations for backpropagation through both A and B.
  • LoRA-FA unifies asymmetric factor freezing with closed-form gradient correction while removing the need to store large intermediate activations.

3 BACKGROUND AND MOTIVATION

LoRA uses a low-rank adapter whose gradient occupies a restricted subspace relative to Full-FT. The paper argues that the factors are collapsible, so one factor can remain fixed without sacrificing expressiveness.

  • LoRA adds a scaled low-rank update AB to the pretrained weight W0, with A and B representing projection-down and projection-up matrices.
  • The change in W depends on both factor updates through the terms dA B and A dB.
  • LoRA’s gradient is low-rank relative to Full-FT, with the comparison characterized by 2r ≪ d.
  • When ΔW = AB, updates are restricted to an r-dimensional subspace, and an equivalent factor pair can permit fixing either A or B.
  • Theorem 3.1 expresses LoRA’s low-rank update as a single-layer linear regression, establishing equivalent expressiveness with only one trainable side adapter.

4 BRIDGING THE PERFORMANCE GAP

LoRA-FA freezes A and adjusts B to reduce the discrepancy between its low-rank gradient and the Full-FT gradient. The resulting optimization has a closed-form solution that preserves non-increasing loss.

  • LoRA-FA freezes A and uses the gradient of B to update W through a low-rank gradient.
  • The gradient-adjustment objective minimizes the discrepancy between the LoRA-FA gradient and the Full-FT gradient while requiring the loss change to be non-increasing.
  • Theorem 4.1 states that this objective has an optimal closed-form solution satisfying dL ≤ 0.
  • The optimal adjustment depends solely on A and the original gradient of B, so B initialization imposes no restriction.
  • The same analysis is extended to LoRA-FB, which freezes B and fine-tunes A.

5 LORA-FA: LORA BY FIXING A

LoRA-FA freezes A and fine-tunes only B, using closed-form gradient adjustments to approach Full-FT while reducing activation memory and computation. Its fixed-A design stores only the smaller intermediate activation XA and supports kernel-level efficiency optimizations.

  • Method: The low-rank update can be represented by a single trainable linear adapter, enabling LoRA-FA to bridge the performance gap with Full-FT.Closed-form modifications to the gradient of B align LoRA-FA more closely with Full-FT than standard LoRA.
  • Method: LoRA-FA freezes both the base weight W and adapter matrix A, training only the projection-up matrix B.This requires computing only the gradient of B.
  • Memory efficiency: LoRA-FA stores only the intermediate activation XA, eliminating standard LoRA’s need to retain the full activation X for computing gradients.LoRA-FB instead freezes B and must store the full X to compute the gradient of A.
  • Computational efficiency: Because A is fixed, LoRA-FA can precompute the inverse of A^T A once and reuse it during training.For rank r = 64, storing A^T A requires approximately 8 KB.

6 EXPERIMENTS

Experiments across GLUE, domain-specific generation, larger models, and system benchmarks show that LoRA-FA is competitive with Full-FT and recent PEFT methods. It also reduces memory use, improves utilization, and benefits from distribution-aligned initialization of A.

  • 6.1 PERFORMANCE ON GLUE BENCHMARKS: LoRA-FA achieves comparable or better GLUE performance than Full-FT and consistently outperforms standard LoRA on RoBERTa-base and RoBERTa-large.Average accuracy reaches 87.5% with RoBERTa-base and 90% with RoBERTa-large, exceeding Full-FT in both cases.
  • 6.2 PERFORMANCE ON DOMAIN-SPECIFIC TASKS: On MT-Bench, GSM8K, and HumanEval, LoRA-FA improves substantially over standard LoRA and consistently matches LoRA-Pro.For Llama2-7B, gains over LoRA are 0.5 on MT-Bench, 19.1 on GSM8K, and 14.4 on HumanEval.
  • 6.2 PERFORMANCE ON DOMAIN-SPECIFIC TASKS: At the same rank, LoRA-FA uses half as many trainable parameters as LoRA-GA and LoRA-Pro while achieving comparable approximate full-gradient capability.For Llama2-7B on MT-Bench, rank 64 gives LoRA-FA 5.7 versus 5.9 for LoRA-GA and 5.8 for LoRA-Pro; at rank 128, LoRA-FA achieves superior performance after matching their parameter count.
  • 6.3 PERFORMANCE ON RECENT LARGER BASE MODELS: LoRA-FA matches LoRA-Pro on Qwen3-8B and DeepSeek-V2-Lite-Chat, supporting effective scaling to larger modern models.These evaluations use GSM8K under the non-thinking zero-shot protocol.
  • 6.4 SYSTEM EFFICIENCY: LoRA-FA achieves MFU comparable to or higher than LoRA and consistently outperforms the other evaluated PEFT methods.Removing A activations eliminates associated feed-forward and backpropagation computations, and reduced memory enables batch-size scaling from 8 to 12.
  • 6.5 DISCUSSIONS ON THE INITIALIZATION OF A: Gaussian initialization improves LoRA-FA performance over uniform initialization, especially when applied to the down-projection A adapter.Down-projection Gaussian initialization yields 75.2 versus 74.9, while global Gaussian initialization yields 75.6 versus 74.9 for global uniform initialization.

7 CONCLUSION

LoRA-FA freezes A, adjusts B using gradient corrections, and combines strong performance with lower activation memory and computational cost.

  • LoRA-FA freezes A and trains B while approximating the full gradient through a corrected B-gradient.The method minimizes discrepancy between the gradient with respect to B and the full gradient.
  • LoRA-FA matches existing PEFT methods and full fine-tuning across Dense and MoE LLMs.
  • LoRA-FA significantly reduces activation memory consumption and computational cost during fine-tuning.

A.1 PROOF OF THEOREM 2.1

The proof shows that LoRA’s low-rank update can be reformulated through a single-layer linear regression, yielding closed-form solutions for freezing one factor and adjusting the other.

  • LoRA’s update A∗B∗ can be represented as A0B′∗ when A∗ is expressed through A0 and a reparameterized matrix.The construction treats the update as a single-layer linear transformation.
  • When A∗ and A0 span the same subspace, an invertible C exists such that A∗ = A0C; otherwise, C is obtained by approximation.
  • The optimization over C has an optimal closed-form solution, making the corresponding update optimal for W.
  • The approximation residual is the projection of A∗ onto the orthogonal complement of A0’s column space.
  • The corrected LoRA-FA gradient minimizes the Frobenius discrepancy from the full gradient and yields a non-increasing loss change.

A.3 LORA-FB AND THE PROOF OF THEOREM A.1

The LoRA-FB analysis mirrors LoRA-FA: freezing B permits a closed-form correction to A’s gradient that approximates the full gradient and does not increase the loss.

  • LoRA-FB freezes B and represents its low-rank gradient as a function of the trainable A gradient.
  • The corrected A gradient minimizes the Frobenius discrepancy between the induced low-rank gradient and the full gradient.
  • The closed-form LoRA-FB solution satisfies dL ≤ 0 under the full-rank assumption on B.
  • The paper notes that the full gradient may be inaccessible directly during LoRA-FA and LoRA-FB training, motivating adjusted gradient formulations.

C PROOFS RELATED TO MEMORY COMPLEXITY

The memory analysis separates parameter, gradient, optimizer-state, and activation costs, then quantifies LoRA-FA’s activation savings over LoRA.

  • Training memory is decomposed into parameters, gradients, optimizer states, and activations, with dimensions including layers, batch size, sequence length, hidden size, and rank.
  • The analysis tabulates memory complexity for Full-FT, LoRA, LoRA-FA, QLoRA, and VeRA under mixed-precision fine-tuning.
  • LoRA-FA reduces activation memory by at least 18bsdL bytes compared to LoRA for Llama2-7B.
  • With batch size 8, sequence length 1024, d = 4096, and L = 32, theoretical LoRA-FA savings reach at least 18GB of activation memory.The reported practical reduction typically exceeds the theoretical estimate because of memory reservation and other retained activations.

D HYPERPARAMETERS AND EXPERIMENT SETTINGS

The experiments compare LoRA-FA with Full-FT, LoRA, and other PEFT baselines across memory-constrained, initialization, layer-count, and rank settings. They emphasize LoRA-FA’s memory efficiency, compatibility with other optimizations, and initialization-dependent gradient coverage.

  • Baselines: The evaluation compares LoRA-FA primarily against Full-FT, standard LoRA, and recent PEFT methods including LoRA-QV, QLoRA, VeRA, PiSSA, LoRA+, AdaLoRA, DoRA, LoRA-GA, and LoRA-Pro.The supplied settings also include configurations for different models, datasets, LoRA-FA layer counts, ranks, and large-sequence-length fine-tuning.
  • Long-sequence memory efficiency: LoRA-FA combined with FlashAttention is evaluated for long-sequence Llama2-7B fine-tuning on RTX 4090 and A100 GPUs with limited memory.The experiment uses sequence lengths of 2048 and 4096 on the two GPUs, while Figure 4 compares capacity and memory footprint under rank and layer-count changes.
  • Layer and rank effects: Increasing the number of LoRA-FA adapter layers does not substantially increase memory usage, enabling adapters on all linear layers while maintaining Full-FT-aligned accuracy.This supports broad layer coverage without the expected escalation in memory requirements.
  • Layer and rank effects: LoRA-FA’s GPU memory footprint is not significantly affected by rank changes from 1 to 128 under the tested Llama2-7B configuration.The experiment uses batch size 1, sequence length 1024, and sweeps rank while holding the other variables constant.
  • Initialization: Gaussian-random initialization avoids truncated-SVD’s possible blind spots and captures an r/m fraction of any fixed gradient in expectation.The paper therefore presents random initialization as the safer default for diverse downstream tasks, while PiSSA-style initialization underperforms Gaussian initialization on GSM8K with Llama3-8B at rank 128.
  • Memory optimizations: LoRA-FA can combine with QLoRA-style quantization, ZeRO sharding, and FlashAttention, but weight sharding introduces expensive weight-gathering communication.Because base weights remain frozen, 8-bit and 4-bit quantization can reduce model-weight memory by 2 and 4 times, respectively.

E.5 LIMITATIONS

LoRA-FA reduces activations associated with its trainable module but does not eliminate all Transformer activations. Its low-rank structure can also produce suboptimal results when model capacity or task demands are high.

  • Activation scope: LoRA-FA cannot reduce activations produced by other Transformer components, including attention-layer inputs and outputs.It eliminates only the activation associated with matrix A, despite reaching the theoretical lower bound for removable trainable-module activations.
  • Low-rank scope: LoRA-FA may yield suboptimal results when the base model has limited capacity or the dataset imposes high performance demands.This limitation follows from LoRA-FA remaining a low-rank fine-tuning method.
Loading 2308.03303v3…