PaperlayerView exact revision on arXiv

Source-linked AI summary

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Peiyi Wang, Qingxiu Dong, Liang Chen, Zhifang Sui

arXiv:2402.16141v1cs.CL

TL;DR

LoRA makes fine-tuning more resource-efficient, but its low-rank updates can leave a performance gap from full fine-tuning. PeriodicLoRA accumulates low-rank updates across stages, and the paper reports stronger learning capability without increased GPU usage. Its evaluation focuses on instruction fine-tuning for difficult tasks, leaving multimodal applications for future study.

  • Problem

    LoRA’s low-rank updates can constrain adaptation on complex tasks, where the paper reports a performance gap from full-parameter fine-tuning.

    Source: §3.1 · p. 4

  • Method

    PLoRA accumulates low-rank LoRA updates across training stages by unloading them into the backbone and reinitializing LoRA states.

    Source: §1 · p. 1

  • Results

    PLoRA’s accumulated higher-rank updates show stronger learning capabilities than long-term-trained low-rank updates, without increasing GPU usage.

    Source: §6 · pp. 8–9

  • Takeaways & Limitations

    The paper presents PLoRA as a way to improve LoRA training effectiveness through higher-rank updates while retaining its resource usage.

    Source: §6 · p. 8

    The evaluation mainly covers instruction fine-tuning on difficult tasks; applying PLoRA to multimodal image-and-text tasks remains to be studied.

    Source: §7 · p. 9

Abstract

from arXiv · show

Supervised fine-tuning is the most common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning LLMs requires massive computational resources. Recently, parameter-efficient fine-tuning (PEFT) methods have been widely studied due to its cost-effectiveness. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low-dimensional. Although LoRA fine-tuning is effective, there is still a performance gap compared to full fine-tuning, since its weight update is limited to low-rank matrices. In order to break the low-rank bottleneck in LoRA Optimization, we propose PeriodicLoRA (PLoRA), which accumulates low-rank update matrices multiple times to achieve a higher update rank. PLoRA has multiple training stages. During each stage, we still update only the LoRA weights. However, at the end of each stage, we unload the LoRA weights into the backbone parameters and then reinitialize the LoRA states. Experimental results show that PLoRA has stronger learning ability, approximately 1.8 times that of LoRA's learning ability at most, but it does not increase memory usage. Further, we introduce a momentum-based unloading strategy for PLoRA to mitigate the training instability.

1 Introduction

Full fine-tuning is costly, while LoRA reduces resource demands but can lag because its updates are low-rank. PeriodicLoRA accumulates low-rank updates to raise update rank without increasing memory usage, and the paper reports stronger performance than naive LoRA.

1 Introduction

  • LoRA can lag behind full fine-tuning because its low-rank update matrix may constrain learning, while directly increasing rank raises memory overhead.The proposed method aims to increase update rank without increasing memory usage.Source: §1 · p. 1
  • PLoRA periodically transfers trained LoRA weights into the backbone and reinitializes the LoRA weights, optimizer state, and learning-rate-scheduler state.Accumulating multiple low-rank updates can produce a higher-rank update matrix.Source: §1 · p. 1
  • PLoRA is presented as a parameter-efficient method that breaks through LoRA’s low-rank bottleneck.The paper also analyzes PLoRA’s training process and reports stronger learning capability than naive LoRA.Source: §1 · p. 2
  • PLoRA consistently outperforms naive LoRA in instruction tuning, while introducing no memory overhead.The evaluation covers multi-subject multiple-choice, math reasoning, and language-understanding and reasoning tasks.Source: §1 · p. 2

2 Related Work

The related work covers instruction tuning, parameter-efficient tuning, and LoRA variants, including challenges around choosing LoRA rank and methods for flexible rank tuning.

2.1 Supervised Fine-Tuning

  • Instruction tuning uses diverse instruction-output datasets to extend language models’ ability to handle varied instructions and multi-task generalization.The section names Self-instruct, UnnaturalInstruction, Super-NaturalInstructions, and FLAN v2 among such datasets.Source: §2.1 · p. 2

2.2 Parameter-Efficient Fine-Tuning

  • PEFT methods adapt models by optimizing only a small portion of parameters while keeping the main model frozen.Examples include adapters, prefix tuning, prompt tuning, and selectively trainable parameters.Source: §2.2 · p. 2

2.3 LoRA and Its Variants

  • LoRA uses low-rank projections alongside attention matrices, and its adapter can be integrated into model weights for efficient inference.Choosing rank r remains a limitation because rank changes model structure and the ideal value may vary across tasks and backbones.Source: §2.3 · p. 2
  • Flexible-rank approaches include training multiple ranks simultaneously, regularizing projection matrices, filtering low-rank components, or constructing LoRA modules across tasks.PLoRA’s figure contrasts LoRA’s end-of-training weight update with repeated transfers and resets during training.Source: §2.3 · p. 3, Figure 1 · p. 3

3 Method: PeriodicLoRA

Naive LoRA reduces fine-tuning costs by restricting updates to low-rank matrices, but this constraint can limit performance on complex tasks. PeriodicLoRA accumulates updates across stages to increase the final update rank while retaining the LoRA training process.

3.1 Preliminary: Naive LoRA Method

  • LoRA approximates the weight update with a product of low-rank matrices, reducing trainable parameters and GPU memory requirements.After training, the matrices are merged into the backbone, leaving a final update constrained to a low-rank space.Source: §3.1 · pp. 3–4
  • On complex tasks such as GSM8K, restricting updates to a low-rank space can leave a performance gap relative to full-parameter fine-tuning.The paper notes that low-rank compression is not lossless and that updates needed for intricate tasks may not fit in the constrained space.Source: §3.1 · p. 4

3.2 PeriodicLoRA

  • PeriodicLoRA divides fine-tuning into stages, trains LoRA matrices in each stage, merges their product into the current backbone, and reinitializes the matrices for the next stage.After T stages, the final weight is W0 + B1A1 + B2A2 + ... + BTAT.Source: §3.2 · p. 4
  • The final update can have rank T × r, so increasing the number of stages can break through traditional LoRA’s low-rank constraint.Each stage remains a LoRA training process before its update is merged into the backbone.Source: §3.2 · p. 4

4 Experiments

The experiments evaluate PeriodicLoRA against LoRA and full fine-tuning using instruction-tuning data and language-model benchmarks. The reported results show improvements over same-rank LoRA on GSM8K and MMLU, with resource conservation.

4.1 Datasets

  • The 60,542-entry training set combines Orca-from-Tülu and Code-Alpaca subsets with dialogue-formatted GSM8K and ARC examples.The reported subset counts are 29,683 Orca entries, 20,016 Code-Alpaca entries, 7,473 GSM8K entries, and 3,370 ARC entries.Source: §4.1 · p. 4

4.2 Strategic Organization of Experimental Datasets

  • The dataset choices prioritize testing PLoRA on challenging benchmarks without substantially slowing iteration, rather than optimizing data quality or balancing.The stated evaluations focus on GSM8K, MMLU, and ARC.Source: §4.2 · p. 5

4.3 Experimental Setup

  • Experiments fine-tune LLaMA-7B with PLoRA, LoRA, and full fine-tuning, applying rank-1 or rank-8 matrices to attention and MLP linear layers.The setup uses AdamW with a constant learning rate of 1e-4, a global batch size of 24, and eight A100 GPUs.Source: §4.3 · p. 5

4.4 Results

  • HumanEval results are omitted because the models scored around 1% pass@20 despite Code-Alpaca making up one-third of the dataset.The reported comparisons are after one epoch, and the authors note dataset and model limitations.Source: §4.4 · p. 5
  • PLoRA improves over same-rank LoRA on GSM8K and MMLU, especially at rank 8; on GSM8K it reports a 20% improvement, and on MMLU it can match or exceed full fine-tuning while conserving resources.The results discussion also reports PLoRA outperforming full fine-tuning in some scenarios.Source: §4.4 · p. 5

5 Discussion

The discussion examines how unloading frequency and momentum affect PLoRA, then analyzes training-stage behavior and implementation choices. PLoRA can outperform LoRA early, but extended training can bring overfitting, while unloading data and learning rate require care.

5.1 Unloading Points Selection

  • Using 4.8k data as the unloading point is the authors’ empirical choice after finding that larger amounts reduce unloading frequency and training effectiveness.Too little data per unload can instead cause insufficient training, error accumulation, and eventual collapse.Source: §5.1 · p. 6

5.2 Momentum Setting

  • Momentum 0 leads to chaotic loss and collapse after the second epoch at rank 8, whereas momentum 0.1 or 0.3 keeps training stable.Lower momentum is associated with smaller losses earlier; higher momentum tends to have lower losses near the fourth epoch.Source: §5.2 · p. 6

5.3 Training Stages

  • PLoRA outperforms LoRA early, especially on GSM8K, and the authors report about half the training time saved at equal compute and speed.Performance declines as training progresses; GSM8K peaks later than MMLU, and rank 8 declines earlier than rank 1.Source: §5.3 · p. 7
  • PLoRA tends to overfit from epoch three onward, with more severe MMLU effects; extended training to six epochs markedly reduces its performance.The authors attribute this to accumulated low-rank updates producing a higher-rank update, akin to full fine-tuning.Source: §5.3 · p. 7

5.4 Learning rate and PLoRA Layers

  • For rank-8 PLoRA, a learning rate of 5e-4 performs much worse than 1e-4; applying LoRA to all linear layers also outperforms rank-8 Wq/Wv-only LoRA on GSM8K.The authors note that too high a learning rate can reduce PLoRA’s learning effectiveness; all seven linear layers are included in the all-layer setting.Source: §5.4 · p. 8, Table 3 · p. 8

6 Conclusion

PeriodicLoRA accumulates mini-batch LoRA weights to produce higher-rank updates without increasing GPU usage. The paper concludes that these accumulated updates have stronger learning capabilities than long-term-trained low-rank weights.

6 Conclusion

  • The conclusion frames PLoRA as a parameter-efficient fine-tuning method grounded in low-rank adaptation theory.Source: §6 · p. 8
  • PLoRA accumulates LoRA weights across mini-batches to obtain higher-rank update matrices without increasing GPU usage.The method is presented as theoretically capable of matching full fine-tuning’s training effectiveness.Source: §6 · p. 8
  • Accumulating low-rank weights yields stronger learning capabilities than long-term training of low-rank weights.Source: §6 · p. 9

7 Limitations

The paper evaluates PLoRA primarily for instruction fine-tuning on difficult tasks; multimodal image-and-text tasks remain an open application area, and a strict explanation of the method’s relationship to full fine-tuning remains challenging.

7 Limitations

  • The evaluation is mainly limited to instruction fine-tuning on difficult tasks.Source: §7 · p. 9
  • Applying PLoRA to multimodal tasks with more complex image-and-text datasets remains to be studied.Source: §7 · p. 9
  • The paper does not provide a strict explanation for how PLoRA fits the training process of full fine-tuning.Source: §7 · p. 9

Related papers

Loading 2402.16141v1…