PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Low-rank Adaptation of Pre-trained Language Models

Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, Maosong Sun

arXiv:2311.11696v1cs.CLcs.AIcs.LG

TL;DR

LoRA uses a fixed intrinsic rank, although the best rank can vary with the model and task. SoRA adds a sparse gate optimized with proximal-gradient updates to adjust rank, and reports outperforming parameter-efficient baselines with fewer parameters and shorter training time. The paper also introduces a sparsifying scheduler to examine memorization and generalization, while noting that its evaluation is limited to traditional NLP tasks.

  • Problem

    LoRA’s fixed intrinsic rank may not be ideal because the optimal rank varies with the backbone model and task.

    Source: §1 · p. 1

  • Method

    SoRA dynamically adjusts LoRA rank with a sparse gate trained by proximal-gradient updates, then removes zeroed components for inference.

    Source: §1 · p. 2

  • Results

    SoRA consistently outperforms parameter-efficient baselines with fewer parameters and 30% shorter training time across a wide range of downstream tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The sparsifying scheduler uses SoRA to examine how the number of non-zero trainable parameters relates to memorization and generalization.

    Source: §5 · p. 9

    The study evaluates SoRA on traditional NLP tasks; how its sparsity behaves in cross-modal or instruction-tuning scenarios remains unknown.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. The popular method of low-rank adaptation (LoRA) offers a notable approach, hypothesizing that the adaptation process is intrinsically low-dimensional. Although LoRA has demonstrated commendable performance, it is implemented with a fixed and unalterable intrinsic rank that might not always be the ideal choice. Recognizing the need for more flexible adaptation, we extend the methodology of LoRA to an innovative approach we call sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. We achieve this through the incorporation of a gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity of the gate. In the subsequent inference stage, we eliminate the parameter blocks corresponding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank-optimal LoRA. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters via updating in a sparse way. We further introduce a sparsifying scheduler for SoRA, aiming to examine the impact of the number of non-zero parameters on the model's memorization and generalization. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.

1 Introduction

The introduction motivates adaptive rank selection as a response to LoRA’s fixed rank, which may not suit different backbones and tasks. It presents SoRA’s sparse gate and proximal-gradient updates as an automated alternative, reporting stronger results with fewer parameters and shorter training time.

1 Introduction

  • The introduction frames parameter-efficient fine-tuning as adapting models while leaving most underlying parameters unchanged, and identifies LoRA as an efficient low-rank approach.The paper describes LoRA as avoiding the forward-propagation latency of inserted neural modules while maintaining stable performance.Source: §1 · p. 1
  • LoRA’s fixed intrinsic rank is difficult to choose because the optimal rank can vary with the backbone model and downstream task.A large rank can expand optimization capacity but may be wasteful; the paper notes that rank selection on large models is computationally costly.Source: §1 · p. 1
  • SoRA dynamically controls rank through a gate updated with a proximal-gradient method under L1 regularization, then drops zero-indexed low-rank components after training.This avoids AdaLoRA’s orthogonality regularization and its associated gradient-computation overhead.Source: §1 · p. 2
  • SoRA consistently outperforms parameter-efficient baselines with fewer parameters and 30% shorter training time across a wide range of downstream tasks.The paper also reports a sparsifying scheduler and describes considerable performance retention with very few parameters.Source: §1 · p. 2

2 A Closer Look to Adaptive Rank

This section frames adaptive rank as a way to avoid LoRA’s inflexible rank choice, then connects rank selection to a sparse representation of the weight update. It contrasts SoRA’s proximal-gradient approach with AdaLoRA’s heuristic selection and added computation.

2 A Closer Look to Adaptive Rank

  • Parameter-efficient tuning includes methods that add modules or parameters and methods that make selected parameters trainable or prunable.The section situates LoRA among approaches that use low-rank matrices to approximate weight changes.Source: §2 · p. 2
  • In LoRA, frozen pretrained weights are paired with trainable low-rank matrices whose product represents the weight change.The intrinsic rank controls the sizes of those matrices and the number of trainable parameters.Source: §2 · p. 2
  • LoRA’s discrete rank can vary in suitability across backbones and tasks; choosing a large rank may waste resources, while choosing a very small one may degrade performance or require retraining.The section presents adaptive-rank selection as a way to address this inflexibility.Source: §2 · pp. 2–3
  • The section relates the tunable weight update’s rank to its singular-value decomposition, providing a basis for selectively controlling low-rank components.The update is decomposed into orthogonal factors and a diagonal matrix of singular values.Source: §2 · p. 3
  • SoRA uses a sparse gate over the low-rank representation; the gate’s nonzero count corresponds to the update’s rank.The section contrasts this with AdaLoRA, which retains entries by a heuristic sensitivity score under a predefined budget.Source: §2 · p. 3
  • AdaLoRA’s sensitivity-based selection is described as heuristic, while its moving averages and orthogonality-regularization gradients add computation.SoRA is presented as using simplified updates backed by sparsity regularization and proximal-gradient methods.Source: §2 · p. 4

3 Our Approach

SoRA makes LoRA rank adaptive by learning sparse gates during fine-tuning, then removing zeroed ranks after training. The section also introduces a sparsifying schedule to examine performance as sparsity increases.

3.1 Sparse Low-rank Adaptation

  • SoRA starts with a high-maximum-rank LoRA factorization, learns a sparsity-promoting gate by proximal-gradient updates, then prunes zeroed ranks into a smaller LoRA.Source: §3 · p. 4, §3.1 · p. 4

3.2 Interpretation and Comparison

  • Unlike AdaLoRA [Zhang et al., 2023], SoRA uses proximal-gradient justification rather than orthogonality regularization and heuristic sensitivity scoring, and is reported to perform better with fewer parameters and less wall-clock time.Source: §3.2 · p. 5

3.3 Scheduling ξ to Explore Memorization and Generalization

  • Starting from a low threshold and training to convergence, the scheduler raises ξ incrementally to examine memorization on training data, generalization on test data, and performance loss as sparsity increases.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate SoRA’s benchmark performance and examine how sparsity, rank allocation, weight placement, and training time vary across datasets. They use GLUE tasks with DeBERTaV3-base and additional RoBERTa-large analyses.

4.2 Results

  • 0.98% and 0.52%: SoRA surpassed LoRA and AdaLoRA, respectively, on GLUE while using 31.5% and 28.3% fewer parameters, respectively.The comparison reports average results over five random seeds.Source: Table 1 · p. 6, §4.2 · p. 6

4.1 Experimental Settings.

  • The evaluation uses GLUE with DeBERTaV3-base, compares full fine-tuning and parameter-efficient baselines, and also uses RoBERTa-large for analysis.Baselines include Adapter, BitFit, LoRA, and AdaLoRA; the paper omits Adapter variants and Prompt Tuning for stated performance or convergence reasons.Source: §4.1 · pp. 5–6

4.3 Sparsifying Scheduler

  • 47,104 non-zero parameters preserve over 99% of SST-2 performance, while 4,096 retain above 90% of memorization and generalization capabilities.Figure 2 tracks performance as the sparsity indicator ξ increases every five epochs; performance eventually drops after dataset-specific inflection points.Source: §4.3 · p. 6, Figure 2 · p. 7
  • Memorization and generalization trends remain consistent during sparsification, but CoLA declines earlier and more sharply than some other datasets.The paper also reports that intermediate and deep layers tend to remain denser than shallow layers.Source: §4.3 · pp. 6–7

4.4 Rank Analysis

  • QQP’s trained parameter matrices are denser than those on the other three analyzed datasets, indicating dataset-specific rank allocation.Figure 3 shows final ranks on QQP, MNLI, QNLI, and MRPC across DeBERTaV3-base layers.Source: §4.4 · p. 7, Figure 3 · p. 8

4.5 Applying SoRA to Different Weights

  • In most cases, applying SoRA to all weight matrices improves performance over applying it to only one or several types; applying it only to WQ,K causes a considerable drop.The reported WQ,K drop is aligned with LoRA.Source: §4.5 · p. 8, Table 3 · p. 8

4.6 Efficiency Analysis

  • About 30% less training time per epoch: SoRA is faster than AdaLoRA across six datasets under matched batch sizes.The comparison uses identical compute infrastructure and batch size; the efficiency margin varies by dataset.Source: Table 4 · p. 8, §4.6 · p. 8

5 Conclusion

SoRA uses a proximal-gradient-optimized gate to regulate sparsity and dynamically adjust rank. A sparsifying scheduler examines how non-zero parameter counts relate to memorization and generalization.

5 Conclusion

  • A proximal-gradient-optimized gate regulates sparsity to provide SoRA with a dynamic alternative rank.The method is presented as expanding the optimization space while enhancing parameter efficiency.Source: §5 · pp. 8–9
  • A sparsifying scheduler examines the relationship between non-zero parameter counts and memorization and generalization.The paper reports that adaptation exhibits a strong compression capability, with even a tiny parameter portion retaining considerable performance.Source: §5 · p. 9
  • SoRA is described as theoretically supported, simple, and effective across various tasks.Source: §5 · p. 9

Limitations

This study evaluates SoRA only on traditional natural language processing tasks. Its sparsity behavior in cross-modal and instruction-tuning scenarios remains unknown, and rigorously explaining the scheduler remains challenging.

Limitations

  • The evaluation is limited to traditional natural language processing tasks, leaving SoRA’s application to cross-modal and instruction-tuning scenarios untested.How SoRA sparsity is displayed in those scenarios remains unknown.Source: Limitations · p. 9
  • Rigorously explaining the sparsifying scheduler and more efficiently assessing adaptation difficulty remain challenging.Source: Limitations · p. 9

A.1 Datasets

The evaluation uses the GLUE benchmark for natural language understanding, drawing on its eight listed datasets. Experiments use the full datasets and report averages across five random seeds for almost all experiments.

A.1 Datasets

  • GLUE evaluation covers CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE.The datasets are sourced from Huggingface Datasets, and the full dataset is used for experiments.Source: §A.1 · p. 12
  • Almost all experiments are run five times with different random seeds, with average results reported.The passage says this is done to ensure statistical significance.Source: §A.1 · p. 12
  • Table 5 reports dataset sizes and evaluation metrics, including Mcc, Acc, F1, Corr, and MNLI matched/mismatched accuracy.Mcc denotes Matthews correlation coefficient; Acc, accuracy; F1, F1 score; and Corr, Pearson correlation coefficient.Source: Table 5 · p. 12

A.2 Implementation Details

Experiments use specified learning rates, epoch counts, and regularization settings across tasks. The implementation uses Transformers and PyTorch on an RTX 3090 GPU.

A.2 Implementation Details

  • The learning rate is 8e-4, with 20 epochs for CoLA, MRPC, and STS-B and 10 for the remaining tasks; RTE uses 1.2e-3 and 50 epochs.λ is 0.1 in all experiments, while ξ is grid-searched over {1e-5, 5e-5, 1e-4}; MNLI best-checkpoint transfer is not used for MRPC, RTE, or STS-B.Source: §A.2 · p. 12
  • Experiments use Huggingface Transformers and PyTorch on an NVIDIA GeForce RTX 3090; SoRA with batch size 8 averages 6110MB GPU memory.The stated maximum GPU memory is 24268MB.Source: §A.2 · p. 12

A.3 Optimization of Hyperparameters

SoRA’s performance is stable across choices of ξ and η, with low variance for fixed settings. The authors suggest ξ at the 1e-4 level and η around 1e-1∼1e-3.

A.3 Optimization of Hyperparameters

  • SoRA’s performance is highly stable across different choices of ξ and η.The optimization experiments vary ξ and η.Source: §A.3 · p. 12, Table 6 · p. 12, Table 7 · p. 12
  • For each fixed ξ and η, the variance of performance is rather low.Source: §A.3 · p. 12
  • The authors generally suggest setting ξ to the 1e-4 level and η around 1e-1∼1e-3.Source: §A.3 · p. 12

A.4 Results with Standard Deviations

The section presents test results with standard deviations in parentheses for SoRA and other baselines on GLUE, and for SoRA at different initial rmax values compared with same-rank LoRA.

A.4 Results with Standard Deviations

  • The appendix identifies the Table 1 and Table 2 results as Tables 8 and 9, respectively.Source: §A.4 · p. 12
  • Table 8 reports GLUE test results for SoRA and other baselines, with standard deviations in parentheses.Source: §A.4 · p. 12, Table 8 · p. 13
  • Table 9 reports GLUE test results and parameter counts for SoRA initialized with different rmax values, compared with same-rank LoRA.Standard deviations are provided in parentheses.Source: §A.4 · p. 12, Table 9 · p. 13
Loading 2311.11696v1…