Source-linked AI summary
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, Ali Ghodsi
TL;DR
As pretrained models grow, fine-tuning becomes more expensive, while LoRA requires exhaustive rank selection and produces adapters fixed to one rank. DyLoRA trains LoRA blocks across a rank range by ordering representations, enabling dynamic search-free adaptation with comparable performance over wider ranks.
Problem
LoRA reduces fine-tuning cost but requires exhaustive rank search and produces adapters that work only with a particular trained rank.
Method
DyLoRA trains LoRA blocks for a range of ranks by sorting the representations learned at different ranks during training.
Results
DyLoRA performance is comparable with LoRA while supporting a wider range of ranks without additional time and effort.
Takeaways & Limitations
DyLoRA can select ranks without multiple retraining runs and make LoRA dynamic at inference time.
Takeaways & Limitations
The effects of different sampling distributions and rank ranges across downstream tasks require further investigation.
Abstract
from arXiv · showhide
With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.
1 Introduction
Growing pretrained models make fine-tuning increasingly expensive, motivating parameter-efficient alternatives. DyLoRA extends LoRA with dynamic, search-free rank selection while maintaining comparable performance across a wider rank range.
- Large pretrained models make downstream fine-tuning more expensive and more prone to overfitting as model parameters grow relative to labeled data.
- Parameter-efficient tuning can retain pretrained-model capacity with lighter training, and low-rank adapters avoid sequence-length increases while training faster than prompt-tuning alternatives.
- DyLoRA trains LoRA blocks across a range of ranks by sorting representations learned at different ranks during training.
- DyLoRA makes LoRA dynamic at inference time without incurring extra costs.
- DyLoRA avoids costly rank search with a negligible performance compromise.
2 Related Work
Related work covers low-rank adapters, dynamic networks, and representation ordering. Existing adapters generally require rank search and specialize to one rank, while prior dynamic approaches do not resolve this problem for low-rank adapters.
- Low-rank adapters support parameter-efficient tuning, including LoRA, Adapter, Compacter, and Parallel Adapter variants.
- DynaBERT and GradMax explore dynamic networks, but the rank-flexibility problem for factorized networks and low-rank adapters remained open.
3 Background
The background introduces nested dropout and LoRA as foundations for DyLoRA. Nested dropout orders retained representations, while LoRA freezes pretrained weights and learns a low-rank parallel adapter whose rank is otherwise fixed.
- 3.1 Nested Dropout: Nested dropout samples a truncation index and retains lower-indexed representation units while dropping higher-indexed units.
- 3.1 Nested Dropout: Nested dropout biases training toward placing more important information in lower representation indices.
- 3.1 Nested Dropout: DyLoRA adapts this ordering idea to low-rank matrix decompositions so one trained module can operate across multiple ranks.
- 3.2 LoRA: Low-rank Adapters: LoRA freezes pretrained dense-layer weights and learns parallel low-rank up- and down-projection matrices instead.
- 3.2 LoRA: Low-rank Adapters: LoRA’s rank is a task-specific hyperparameter, and the trained adapter is static rather than adaptable to other ranks.
4 Our Method: DyLoRA
DyLoRA trains a single LoRA module to operate across a range of ranks, enabling dynamic inference without exhaustive rank search. It samples and truncates rank-specific projections during training, while using an efficient loss and optional frozen updates to preserve previously learned ranks.
- DyLoRA trains and deploys low-rank adapters across a rank range under a fixed training budget, avoiding repeated training to search for the best rank.
- LoRA modules use up-projection Wup ∈ R^m×r and down-projection Wdw ∈ R^r×d, with target ranks ranging from rmin to rmax.
- The dynamic loss evaluates the network using b-truncated projections for sampled ranks and extends the static loss to the truncation process.
- DyLoRA optimizes each target rank individually instead of summing losses over all ranks, reducing the computational cost of nested-dropout-style training.
- The frozen update mode changes only the newly selected row and column, preventing earlier parameters from being forgotten and improving efficiency with a minor performance cost.
5 Experiments
The experiments evaluate DyLoRA on NLU and NLG tasks using RoBERTa and GPT backbones, comparing it with fine-tuning, LoRA, FLOP, and distribution variants. The study examines rank effects, fixed-budget dynamic inference, baselines, and the impact of rank-selection distributions.
- Experiments cover natural language understanding and natural language generation tasks using RoBERTa-base for GLUE and GPT-Medium for generation.
- The training procedure samples and truncates a rank, updates only selected-rank parameters, and can freeze lower ranks during higher-rank updates.
- The evaluation compares rank-dependent LoRA performance while holding other hyperparameters fixed and changing only the adaptation rank.
- The fixed-budget evaluation tests whether one low-rank adaptation matrix works with different ranks at inference time.
- 5.1 Baselines: Baselines include fully fine-tuned models, original LoRA, and FLOP, with FLOP representing a compression-based method lacking DyLoRA’s dynamic inference property.
- 5.1 Baselines: The ablation study examines how the rank-selection distribution affects DyLoRA performance.
5.2 LoRA rank selection problem
LoRA rank selection is difficult because performance varies across tasks and ranks, while deployment may require different model configurations. DyLoRA addresses this by supporting a wider rank range without exhaustive rank search, while retaining strong performance.
- Rank-selection challenge: LoRA has no clear rank-selection guidance because performance varies substantially across ranks and tasks.MRPC performs worst at rank 16 and best at rank 32, whereas SST-2 performs worst at rank 1 and best at rank 8.
- Rank-selection challenge: Different device configurations may require different ranks, forcing LoRA users to train multiple models or search for an optimal rank.Higher ranks introduce more parameters, which can be problematic for sensitive devices.
- Dynamic adaptation: DyLoRA preserves high performance at lower ranks while competing well with LoRA at rank 8.The comparison uses LoRA and DyLoRA models with the same model size, code, evaluation process, and training extent.
- Search cost: Exhaustive LoRA search across ranks 1 to 64 requires training and evaluating 64 models, whereas DyLoRA avoids this search.Uniform search remains 7 times more expensive than the proposed method in the reported experiment.
- Ablation study: DyLoRA’s geometric rank distribution better optimizes lower ranks, while uniform distribution provides better performance across all ranks.The authors use uniform distribution in most experiments to avoid introducing another hyperparameter.
6 Conclusion
DyLoRA makes LoRA dynamic at inference time and avoids repeated retraining for rank selection. Its performance remains comparable with LoRA across a wider range of ranks without additional training time and effort.
- Conclusion: DyLoRA addresses rank selection and dynamic inference in low-rank adapters.The method selects ranks without requiring multiple retraining runs and makes LoRA dynamic at inference time.
- Conclusion: DyLoRA avoids the process of searching for the most optimal rank without requiring multiple retraining runs.The conclusion presents this as the practical outcome of combining dynamic inference with rank selection.
- Conclusion: DyLoRA supports a wider range of ranks with performance comparable to LoRA without additional time and effort.This avoids searching for optimal ranks in many real-life scenarios.
Limitations
The paper identifies unresolved sensitivity to scalar and rank-range choices, and calls for further study across downstream tasks and rank ranges.
- Limitations: Further investigation is needed to determine how scalar choices affect LoRA results.The paper notes that a proper scalar choice can improve performance.
- Limitations: The effect of different rank distributions on downstream tasks requires further evaluation.Uniform distribution is reported as effective, but the paper calls for broader investigation against geometric distributions.
- Limitations: Further research is needed to understand the impact of selecting a particular rank range.The algorithm works over a wide range of ranks, but the consequences of choosing that range remain unresolved.
A Time complexity
DyLoRA’s training time is comparable to training LoRA once at one rank, while LoRA rank search requires repeated training. The relative advantage of DyLoRA grows as the number of candidate ranks increases.
- Time complexity: DyLoRA’s relative time complexity is inversely proportional to the number of ranks searched for LoRA.The advantage increases when LoRA would otherwise require more separate rank-specific trainings.
- Time complexity: DyLoRA training takes 408.39 seconds for all ranks in MRPC, compared with 399.95 seconds for LoRA trained once at rank 8.Both methods have comparable single-training costs in the reported experiment.
- Time complexity: Training LoRA separately for ranks 1 through 8 costs 3199.6 seconds, compared with 408.39 seconds for DyLoRA.The LoRA cost is calculated as 399.95*8, while DyLoRA trains one model covering all ranks.
- Time complexity: A more efficient implementation of DyLoRA may further improve its time complexity.This is stated as a possible implementation-level improvement.
B Hyperparameters
The experiments used original pretrained weights, unified hyperparameters, and no parameter tuning or MNLI initialization. Results were averaged across five random seeds with means and standard deviations.
- All datasets were fine-tuned from the original pretrained weights without MNLI checkpoint initialization.
- The experiments used unified hyperparameters across different tasks without parameter tuning.
- Results report the mean and standard deviation over five random seeds, unlike LoRA’s median reporting.
C GPT Experiments
Additional language-generation experiments are summarized in Table 9, while Table 8 contains the study’s hyperparameters. Table 9 marks results reported from the LoRA paper with an asterisk.
- Table 9 summarizes additional experiments conducted to demonstrate DyLoRA’s effectiveness on language-generation tasks.
- Table 8 presents all hyperparameters used throughout the study.
- Rows marked with an asterisk in Table 9 are reported from the LoRA paper.