PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao

arXiv:2303.10512v2cs.CLcs.LG

TL;DR

Large-model fine-tuning is costly, and uniform parameter budgets overlook differing importance across weight matrices. AdaLoRA uses SVD-based incremental updates and importance-guided rank allocation; experiments report consistent gains over baselines, especially at low budgets.

  • Problem

    Parameter-efficient tuning methods often distribute update budgets evenly across weight matrices despite differing module importance, limiting fine-tuning performance.

  • Method

    AdaLoRA parameterizes incremental updates as ∆ = PΛQ and dynamically allocates ranks by importance, pruning singular values of less important updates.

  • Results

    AdaLoRA consistently outperforms or matches baselines across language understanding, question answering, and generation tasks, especially under low budgets.

  • Takeaways & Limitations

    Adaptive budget allocation improves parameter efficiency and model performance relative to evenly allocated fine-tuning approaches within the evaluated tasks and models.

  • Takeaways & Limitations

    Importance estimates based on sampled mini-batches can have high variability and uncertainty because of stochastic sampling and complex training dynamics.

Abstract

from arXiv · show

Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine-tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at https://github.com/QingruZhang/AdaLoRA .

1 INTRODUCTION

Parameter-efficient tuning reduces the cost of adapting large pre-trained models, but methods that distribute trainable parameters evenly overlook differences in module importance. AdaLoRA addresses this by dynamically allocating rank-based budgets using importance scores and SVD-style parameterization.

  • Motivation: Full fine-tuning becomes impractical for large models and simultaneous downstream tasks because it updates all pre-trained parameters.BERT, T5, and GPT-3 contain up to 300 million, 11 billion, and 175 billion parameters, respectively.
  • Existing approaches: Parameter-efficient approaches add small task-specific modules or model weight updates while keeping most of the pre-trained model fixed.Adapters, prefix tuning, prompt tuning, diff pruning, and LoRA reduce the number of updated parameters relative to full fine-tuning.
  • Motivation: LoRA prespecifies identical ranks across incremental matrices, ignoring that module and layer importance can vary substantially.With equal parameter counts, feed-forward networks outperform self-attention modules, and additional parameters help critical matrices but can provide marginal gains or hurt less important ones.
  • AdaLoRA: AdaLoRA dynamically allocates parameter budgets among weight matrices by adjusting the ranks of their incremental updates according to importance.Important matrices receive higher rank, while less important updates receive lower rank to save budget and reduce overfitting.
  • AdaLoRA: AdaLoRA parameterizes each incremental update as ∆ = PΛQ, enabling singular-value pruning without repeatedly computing exact SVDs.Its importance metric scores triplets containing singular values and their corresponding singular vectors.
  • Experiments: 1.2% F1 improvement on SQuAD2.0 is achieved by AdaLoRA with less than 0.1% of full-fine-tuning parameters.The experiments cover natural language understanding, question answering, and natural language generation across DeBERTaV3-base and BART-large.

2 BACKGROUND

Transformer models combine multi-head attention and feed-forward submodules, while LoRA adapts their weights through low-rank updates represented by two smaller trainable matrices.

  • Transformer-based Models: A transformer block contains a multi-head attention module and a fully connected feed-forward network.The FFN applies two linear transformations with a ReLU activation between them.
  • Transformer-based Models: Multi-head attention uses output, query, key, and value projection matrices for its parallel attention heads.The query, key, and value projections have dimensions tied to the model width and number of heads.
  • Low Rank Adaptation: LoRA represents a weight update as the product of two smaller matrices, reducing the trainable parameter count while preserving the base weight matrix.Only the low-rank factors are updated during fine-tuning, with rank r much smaller than the original matrix dimensions.
  • Low Rank Adaptation: LoRA initializes the update at zero by using random initialization for A and zero initialization for B.This preserves the original model behavior at the beginning of training.
  • Low Rank Adaptation: LoRA was initially applied to query and value projections, while later work extended it to feed-forward matrices and unified several efficient tuning methods.The background identifies Wq, Wv, Wf1, and Wf2 as relevant target matrices.

3 ADALORA METHOD

AdaLoRA combines SVD-based adaptation with importance-aware rank allocation to control low-rank update budgets without intensive exact SVD computation. It iteratively scores singular-value triplets, prunes less important values, and schedules the global budget during training.

  • SVD-based adaptation: AdaLoRA parameterizes each incremental update as PΛQ, with Λ containing singular values and P and Q representing left and right singular vectors.The parameterization uses a low rank r much smaller than the matrix dimensions and regularizes P and Q toward orthogonality.
  • SVD-based adaptation: Iterative pruning avoids the intensive exact SVD computations required for repeatedly decomposing many high-dimensional incremental matrices.The paper gives exact SVD complexity as O(min(d1, d2)d1d2) for an incremental matrix.
  • SVD-based adaptation: Compared with structured LoRA pruning, AdaLoRA preserves pruned singular vectors for possible recovery and avoids the larger matrix changes caused by non-orthogonal LoRA doublets.The paper links doublet-wise pruning to training instability and potentially worse generalization, while AdaLoRA masks singular values instead.
  • SVD-based adaptation: The method applies SVD-based adaptation to every selected transformer weight matrix and represents each update through singular-value triplets.The adapted matrices include Wq, Wk, Wv, Wf1, and Wf2 in each transformer layer.
  • Importance-aware rank allocation: AdaLoRA assigns importance scores to triplets and zeros low-scoring singular values, leaving more budget for higher-priority incremental matrices.The proposed metric combines the singular value with the importance of its associated singular vectors and averages vector importance so scores do not scale with triplet size.
  • Importance-aware rank allocation: The importance metric can use sensitivity based on gradient-weight products, whose parameter-level removal influence is combined to estimate triplet contribution.The authors note that mini-batch stochasticity and training dynamics can make sensitivity estimates highly variable, motivating smoothing and uncertainty quantification as an alternative.
  • Global budget scheduler: A global scheduler starts above the target budget, warms up training, decreases the budget cubically, then fixes its distribution for final fine-tuning.This schedule lets AdaLoRA explore the parameter space before focusing on the most important weights.

4 EXPERIMENTS

Experiments evaluate AdaLoRA across language understanding, question answering, and generation under varied parameter budgets. AdaLoRA generally matches or improves baselines, particularly at low budgets, while ablations and rank analyses support its adaptive allocation design.

  • Experimental Setup: AdaLoRA is evaluated with DeBERTaV3-base on GLUE and SQuAD, and with BART-large on XSum and CNN/DailyMail.The implementation uses PyTorch, Hugging Face Transformers, and NVIDIA V100 GPUs.
  • Natural Language Understanding: 87.36% accuracy on RTE with a 0.3M budget is 1.8% higher than the best-performing baseline.On CoLA, AdaLoRA reaches 70.04% Mcc. with 0.3M parameters, exceeding baselines using 0.6M or 1.2M.
  • Question Answering: 88.7% F1 on SQuADv2.0 at the smallest 0.08% budget is 1.2% higher than the best-performing baseline.AdaLoRA consistently outperforms existing approaches across the tested budget levels and maintains performance as the budget changes.
  • Natural Language Generation: 21.13 R-2 at a 1.10% budget exceeds LoRA’s 19.89 on XSum, while AdaLoRA is better or comparable across tested generation budgets.The comparison covers XSum and CNN/DailyMail at 0.13%, 0.26%, 1.10%, and 2.20% parameter budgets.
  • Budget Analysis: At 0.16% budget, AdaLoRA reaches 88.78% F1 on SQuADv2.0, close to 88.89% at 4.65% and with a larger baseline gain.Across MNLI-m, SQuADv2.0, and XSum, AdaLoRA improves consistently over the baseline under all tested budgets.
  • Ablation Studies: Ablations show that SVD-based adaptation alone improves over LoRA but underperforms full AdaLoRA, while removing orthogonal regularization can degrade performance.The results support contributions from both SVD adaptation and orthogonal regularization.

5 CONCLUSION

AdaLoRA adaptively allocates parameter budgets using importance scoring and SVD-parameterized incremental updates, with experiments showing it outperforms existing approaches.

  • AdaLoRA adaptively allocates the parameter budget among incremental matrices according to importance scoring.
  • The method parameterizes incremental updates with singular value decomposition and manipulates singular values to allocate budgets.
  • AdaLoRA is evaluated on natural language processing, question answering, and natural language generation tasks.
  • AdaLoRA outperforms existing approaches in these experiments.
  • The global budget scheduler gradually decreases the budget b(t) following a cubic schedule.

B GLUE DATASET STATISTICS

This section presents GLUE dataset statistics and describes budget and rank settings used to compare AdaLoRA with other methods.

  • The GLUE benchmark dataset statistics are presented in a dedicated table.
  • For each budget level, AdaLoRA's final budget, LoRA's rank, and adapters' hidden dimensions are tuned to match budget requirements.
  • AdaLoRA can control its budget through the final average rank r̄(T) = b(T)/n.
  • Final average ranks of 2, 4, and 8 correspond to final budgets of 144, 288, and 576, respectively.

C.2 TRAINING DETAILS

Training details specify learning-rate selection, shared batch-size settings, and budget-based parameter controls for the GLUE and question-answering experiments.

  • Learning rates are selected from a fixed set, with the best learning rate chosen separately for every method.
  • Batch size is set identically for every method for each dataset.
  • The GLUE experiments include a dedicated AdaLoRA hyper-parameter setup.
  • Trainable parameters for each method are controlled according to the experimental budget.

D.2 TRAINING DETAILS

The question-answering and summarization experiments report dataset statistics, hyper-parameter configurations, and budget-based controls for trainable parameters.

  • The training configuration uses batch size 16 and selects the best-performing learning rate for every method.
  • Question-answering experiments use a dedicated AdaLoRA hyper-parameter setup.
  • The question-answering dataset statistics are summarized in Table 11, identified as SQuAD statistics.
  • Trainable parameters for each method are controlled according to the budget.
  • Summarization experiments have a dedicated detailed budget setup.

E.2 TRAINING DETAILS

The appendix specifies training hyperparameters for AdaLoRA, including batch size and learning-rate selection, and provides a summarization-task configuration table.

  • E.2 TRAINING DETAILS: Training uses a batch size of 16.
  • E.2 TRAINING DETAILS: The learning rate is selected from eight candidate values, with the best-performing value chosen separately for each method.The candidates range from 3 × 10^-5 to 1 × 10^-3.
  • E.2 TRAINING DETAILS: Table 13 lists AdaLoRA’s hyper-parameter setup for summarization tasks.

F ABLATION STUDY FOR LORA

The ablation study examines whether applying LoRA to every weight matrix improves performance over adapting only Wq and Wv, while the orthogonality analysis evaluates AdaLoRA’s regularization behavior.

  • F ABLATION STUDY FOR LORA: Applying LoRA to every weight matrix can further improve performance compared with fine-tuning Wq and Wv only.The parameter budget is fixed at 0.3M in Table 14.
  • F ABLATION STUDY FOR LORA: Table 14 reports accuracy for QQP and MRPC, accuracy(m) for MNLI, and average correlation for STS-B.
  • F ABLATION STUDY FOR LORA: With γ set to 0.1, AdaLoRA’s two orthogonality regularization terms can reach approximately 0.001 early in training.The analysis fine-tunes DeBERTaV3-base on SST-2 under the Section 4.1 training configuration.
  • F ABLATION STUDY FOR LORA: The early optimization of both regularization terms rapidly enforces orthogonality for P and Q during AdaLoRA’s initial warm-up.

H COMPARISON OF TRAINING COST

The training-cost comparison evaluates AdaLoRA against LoRA under matched settings and reports modest additional training time with similar memory usage.

  • H COMPARISON OF TRAINING COST: The comparison fine-tunes DeBERTaV3-base with both methods on a single NVIDIA V100 GPU using the same batch size and training epochs.
  • H COMPARISON OF TRAINING COST: AdaLoRA incurs 11% additional training time on MNLI and 16% on SQuADv2 under different budgets.
  • H COMPARISON OF TRAINING COST: The memory footprints of AdaLoRA and LoRA are quite close.
  • H COMPARISON OF TRAINING COST: Table 15 presents the practical training-cost comparison between AdaLoRA and LoRA.
  • H COMPARISON OF TRAINING COST: AdaLoRA evaluates importance scores only for small incremental matrices PΛQ, whose parameters are usually less than 1% of pre-trained weights.Updating these scores does not add significant computational cost relative to the full model’s forward-backward pass.
Loading 2303.10512v2…