A Survey on LoRA of Large Language Models

Yuren Mao; Yuhang Ge; Yijiang Fan; Wenyi Xu; Yu Mi; Zhonghao Hu; Yunjun Gao

Source-linked AI summary

A Survey on LoRA of Large Language Models

Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, Yunjun Gao

arXiv:2407.11046v4Jul 8, 2024cs.LGcs.AIcs.CL

TL;DR

Large language models require costly downstream adaptation, while prior surveys provide limited systematic coverage of LoRA. This survey organizes LoRA research across adaptation, cross-task generalization, efficiency, federated learning, applications, and future directions, concluding that LoRA is parameter-efficient, pluggable, compatible, and increasingly applicable.

Problem
Large language models need downstream adaptation, but full fine-tuning is computationally expensive and prior surveys lack a systematic, comprehensive treatment of LoRA.
Method
The survey categorizes LoRA research into downstream adaptation, cross-task generalization, efficiency improvement, federated learning, applications, and future directions.
Results
The review concludes that LoRA is parameter-efficient, pluggable, compatible, and supports cross-task generalization across broader application scenarios.
Takeaways & Limitations
Recent progress further improves LoRA’s generalization and efficiency and expands its potential use in more scenarios.
Takeaways & Limitations
LoRA in federated learning must address data, device, and model heterogeneity.

Abstract

from arXiv · show

Low-Rank Adaptation~(LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA's performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field. At last, we provide a Github page~\footnote{\href{https://github.com/ZJU-LLMs/Awesome-LoRAs.git}{https://github.com/ZJU-LLMs/Awesome-LoRAs.git}} for readers to check the updates and initiate discussions on this survey paper.

1 Introduction

LLMs require downstream fine-tuning, but full fine-tuning is costly; LoRA offers a parameter-efficient, reusable alternative whose variants support adaptation, cross-task generalization, efficiency, and privacy.

Motivation: Full fine-tuning of large language models is computationally expensive and can exceed common consumer GPU memory capacity.Fine-tuning LLaMA2-7B requires approximately 60GB of memory.
LoRA’s role: LoRA updates dense layers with pluggable low-rank matrices, enabling parameter-efficient adaptation without modifying the model architecture.The matrices can be stored and reused for related downstream tasks.
LoRA’s role: LoRA plugins can be combined to support cross-task generalization, including multi-task learning, domain adaptation, and continual learning.The supplied passages describe these uses as consequences of LoRA’s reusable plugin design.
Efficiency: Managing many accumulated LoRA modules creates additional computational costs, motivating methods that reduce per-module computation and accelerate scalable serving.The survey connects this concern to real-world Generative-as-a-Service products.
Privacy and efficiency: Federated learning with LoRA can reduce client computation and communication costs while supporting localization or encryption of personalized parameters.These properties motivate LoRA’s potential for privacy-preserving adaptation when client data cannot be centralized.
Survey scope: The survey systematically reviews LoRA methods for downstream adaptation, cross-task generalization, computational efficiency, federated learning, and applications.It also presents background knowledge, research trends, technical insights, and future directions.

2 Low-Rank Adaptation (LoRA)

LoRA adapts dense layers through low-rank updates, substantially reducing trainable parameters while retaining broad applicability. The section reviews its formulation, efficiency, theoretical analyses, and extensions to pre-training and continual training.

2.1 LoRA: LoRA replaces a full update matrix with two smaller matrices, B and A, whose product is scaled and added to the pretrained weights.The rank r is much smaller than the layer dimensions, reducing the number of trainable parameters.
2.1 LoRA: LoRA’s trainable parameter count is r × (d + k), significantly less than the d × k parameters required by full fine-tuning.The low-rank update preserves a pluggable structure and does not increase inference latency.
2.2 Theoretical Analysis: Theoretical analyses report near-equivalence to full fine-tuning in the lazy regime and characterize rank conditions for expressive adaptation.Other analyses connect rank choices and update-matrix training strategies with generalization behavior.
2.2 Theoretical Analysis: Freezing LoRA’s project-down matrix A while tuning only the project-up matrix B is reported to improve generalization and reduce parameters by 2×.The result is presented as a theoretical-analysis finding about how the two matrices function.
2.3 Efficiency in Practice: On the first FFN layer of LLaMA2-7B with r = 4, LoRA tunes 60,416 parameters versus 45,088,768 for full fine-tuning.For this layer, LoRA adjusts nearly one-thousandth of the full-fine-tuning parameter count.
2.3 Efficiency in Practice: LoRA fine-tuning of LLaMA2-7B requires about 23GB of memory, compared with approximately 60GB for full fine-tuning.The comparison uses batch size 1 on a single NVIDIA RTX4090 with 24GB of memory.
Extensions: LoRA has been extended beyond downstream fine-tuning to pre-training and continual training, including approaches targeting catastrophic forgetting.Examples include low-rank updates for high-rank networks and replay or interpolation strategies for continual training.

3 Downstream Adaptation Improving

The survey organizes downstream adaptation improvements into four directions: breaking LoRA’s low-rank bottleneck, allocating ranks adaptively, optimizing learning, and combining other paradigms.

Downstream adaptation methods target four areas: low-rank bottlenecks, dynamic rank allocation, learning-procedure optimization, and combinations with other learning paradigms.

3.1 Breaking the Low-rank Bottleneck

LoRA’s low-rank updates can limit performance on knowledge- and skill-intensive tasks. Proposed methods increase effective update rank by aggregating modules, resampling projections, or jointly updating the base model.

10-100 × higher rank is significant for full fine-tuning than LoRA, highlighting LoRA’s rank limitation in knowledge- and skill-intensive domains.
ReLoRA periodically merges LoRA modules into the LLM and reinitializes them, effectively stacking updates to increase overall rank.
MELoRA addresses possible rank overlap in merge-and-reinit methods by decomposing LoRA modules into smaller mini LoRAs.
FLoRA resamples its random projection to overcome LoRA’s fixed gradient-space compression and largely recover full-matrix SGD performance.
Delta-LoRA jointly updates the LLM and LoRA modules using differences between consecutive LoRA modules without extra memory.

3.2 Dynamic Rank Allocation

Uniform LoRA ranks are suboptimal because layer importance differs and excessive ranks can hurt performance and efficiency. Dynamic methods allocate rank through SVD-inspired pruning, incremental growth, component selection, or random sampling.

Different Transformer layers may require different ranks, while abundant ranks can degrade both performance and efficiency.
SVD-based Methods: SVD-based methods control LoRA rank by decomposing BA and filtering less important singular values.
SVD-based Methods: IncreLoRA starts from a single rank and increases it using a heuristic importance score, avoiding a predefined high-rank budget.
Component-wise Methods: DoRA decomposes BA into single-rank components and prunes components according to heuristic importance.
Sampling-based Methods: DyLoRA samples rank b from a predefined discrete distribution and trains only the corresponding row of A and column of B.

3.3 Optimizing the Learning Procedure

LoRA can converge more slowly than full fine-tuning, depend sensitively on hyperparameters, and overfit. Research therefore improves initialization, gradient updates, and overfitting mitigation.

LoRA’s slower convergence, hyperparameter sensitivity, and overfitting motivate improvements to its learning procedure.
Initialization Improvement: Init[A] is theoretically better than Init[B], but random initialization can still produce small initial gradients and slow convergence.
Initialization Improvement: PiSSA initializes LoRA with principal singular components to accelerate convergence, while MiLoRA uses minor singular components to avoid interference with important learned features.
Gradient Update Optimization: Riemannian scaled gradients and LoRA+ improve convergence and robustness through preconditioning and proportional learning rates for A and B.
Overfitting Mitigation: BiLoRA mitigates overfitting by alternately optimizing singular vectors and singular values on different training-data subsets.

3.4 Combining with other Learning Paradigms

LoRA is compatible with Bayesian, in-context, and active learning paradigms, enabling variants that address downstream adaptation challenges such as overconfidence, low-resource settings, and data efficiency.

Laplace-LoRA combines LoRA with Bayesian learning to relieve overconfidence during downstream adaptation.
PILLOW combines LoRA with in-context learning to address low-resource downstream adaptation.
STAR combines LoRA with active learning and can effectively improve data efficiency.
Table 1 compares LoRA and variants on the GLUE benchmark using task-specific correlation and accuracy metrics.CoLA uses Matthew’s correlation, STS-B uses Pearson correlation, and other datasets use accuracy.

4 Cross-task Generalization

LoRA mixture methods combine multiple LoRA modules for cross-task generalization, progressing from manually designed weights to learned task-, instance-, and token-level mixtures and trainable expert systems.

LoRA plugins can be accumulated, used independently, or mixed for multi-task learning, domain adaptation, and continual learning.The survey notes more than 20,000 compatible LoRA plugins on Hugging Face.
Mixture with Manually Designed Weights: Manually weighted mixtures are simple and computationally efficient but may produce unstable performance and limited generalization because optimal weights are difficult to find.
Mixture with Learnt Weights: Learned mixture weights can operate at task, instance, or token level to provide more adaptive mixtures for different inputs and transfer needs.LoRAHub uses CMA-ES for task-level weight optimization, while MixLoRA and X-LoRA adapt at instance or token and layer granularity.
Mixture with Learnt Weights: MixLoRA selects low-rank decomposition vectors by input instance, while X-LoRA dynamically gates LoRA modules at token level and layer granularity.These approaches report better performance in specific tasks or application scenarios.
Mixture of LoRA Experts: LoRA MoE jointly learns mixture weights and LoRA modules, using LoRA modules as experts and a router network to assign weights.It has been applied to continual learning, vision-language tasks, and multitask medical applications.
Mixture of LoRA Experts: LoRA MoE improvements address initialization, task relationship management, and efficiency, including separate LoRA initialization, balanced subspaces, and global task relationships.I-LoRA applies related expert ideas by using separate LoRAs for long-term and short-term memory in continual learning.

5 Efficiency Improving

Efficiency-improving LoRA methods reduce tunable parameters, quantize model representations, and parallelize training or inference to address memory, computation, and scalable serving costs.

Efficiency methods target parameter reduction, parameter quantization, and parallel LoRA computing frameworks.
Parameter Reduction: Parameter reduction methods use freezing, pruning, and sharing to reduce LoRA’s tunable parameters and activation-memory cost.
Parameter Freezing: Freezing methods either tune only a subset of LoRA parameters or introduce small extra parameters while freezing the original LoRA parameters.Examples include LoRA-SP, LoRA-FA, AFLoRA, DropBP, LoRA-XS, and BYOM-LoRA.
Parameter Reduction: Pruning removes unimportant LoRA parameters independently or jointly with LLM parameters, while sharing methods reuse parameters across layers or modules.VeRA, VB-LoRA, and FourierFT implement distinct sharing strategies.
Parameter Quantization: Quantization reduces parameter bit width, with PTQ methods quantizing before fine-tuning and QAT methods quantizing and fine-tuning jointly.QLoRA is introduced as a PTQ-based method, while LoftQ and ApiQ address quantization-aware training issues.
Parallel LoRA Computing: Parallel frameworks improve resource use by jointly fine-tuning or inferring multiple LoRA modules on shared hardware.ASPEN fuses input batches for parallel fine-tuning; Punica, S-LoRA, and CARASERVE target parallel inference and multi-tenant service.

6 LoRA for Federated Learning

LoRA supports federated adaptation by reducing communication and computation costs while keeping data localized, but federated settings require methods for data, device, and model heterogeneity and parameter privacy.

Federated LoRA adapts LLMs from decentralized private data while reducing the communication and computation costs of client updates.
LoRA in federated learning faces data, device, and model heterogeneity, although its pluggable parameters can support privacy protection.
Data Heterogeneity: As client data becomes more diverse, the performance gap between LoRA and full fine-tuning widens in federated learning.SLoRA addresses this issue with data-driven initialization based on sparse federated fine-tuning and SVD.
Device Heterogeneity: Device heterogeneity can create a buckets effect because using the smallest LoRA rank for all clients prevents resource-rich clients from fully using their capabilities.FedMS, FlexLoRA, and HETLORA dynamically adjust activated matrices or ranks for client resources.
Model Heterogeneity: Model heterogeneity requires knowledge transfer across clients with different architectures without sharing private data or model details.pFedLoRA uses LoRA to carry global and local knowledge through iterative training.
Parameter Privacy: LoRA can protect client-specific parameters through localized components, trusted execution, encryption, and differential privacy mechanisms.PrivateLoRA stores a trainable matrix on the client, while DP-LoRA adds Gaussian noise to LoRA updates.

7 Applications of LoRA

The survey reviews LoRA applications across language, vision, and multimodal tasks, emphasizing efficient adaptation of pretrained models to specialized objectives and domains.

Language tasks: LoRA adapts pretrained language models to traditional NLP, code, alignment, and vertical-domain tasks while reducing computational resource requirements.Examples include classification, relation extraction, code repair, Text-to-SQL, reward and policy modeling, and medical or financial applications.
Language tasks: LoRA supports code-related tasks including defect prediction, code summarization, automated program repair, Text-to-SQL, and code translation.The survey describes applications across BERT-style, encoder-decoder, decoder-only, and StarCoder models.
Vision tasks: In vision, LoRA is applied primarily to image generation and segmentation to improve training efficiency and adapt models to specialized visual tasks.Applications include diffusion-model style transfer, text-to-image generation, video and 3D generation, and SAM-based segmentation.
Vision tasks: LoRA-based methods improve image generation by learning latent-space smoothness, resolution priors, and combinations of customized concepts, while also supporting distillation.These methods target image quality, resolution adaptation, concept fidelity, and generation speed.
Multimodal tasks: LoRA is increasingly used in multimodal large language models for training efficiency and modality alignment across audio-text and image-text tasks.Examples combine LoRA with frozen language or visual components, modality adapters, abstractors, or image-token adaptation.

8 Conclusion and Future Direction

The survey identifies LoRA as a parameter-efficient, pluggable approach with applications in scalable services, continued pre-training, and autonomous agents. It highlights future directions where LoRA modules support rapid service updates, domain adaptation, and data-driven agent roles and memory.

Conclusion: The survey systematically reviews LoRA progress in downstream adaptation, cross-task generalization, efficiency, federated learning, and applications.It characterizes LoRA as parameter efficient, pluggable, compatible, and suitable for cross-task generalization.
8.1 LoRA for GaaS: In GaaS, separate LoRA modules can implement diverse functions, while combinations of previous modules can initialize rapid service updates.The survey connects this use case to LoRA’s parameter efficiency, pluggability, and cross-task generalization.
8.2 LoRA for Continued Pre-training: Continued pre-training adapts foundation models to specific domains using unlabeled user data but remains computationally expensive, motivating more efficient LoRA methods.The survey identifies this challenge as especially relevant to organizations with limited computational resources.
8.3 LoRA for Autonomous Agents: For autonomous agents, LoRA modules are proposed for role assignment, domain-specific tools, and continual-learning memory beyond prompt-based roles and RAG support.The proposal addresses complex roles and limitations associated with prompts, input-token limits, and in-context learning.