PaperlayerView exact revision on arXiv

Source-linked AI summary

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen

arXiv:2406.09044v3cs.CL

TL;DR

Full finetuning of LLMs is costly, and randomly initialized LoRA updates may interfere with pretrained features. MiLoRA freezes principal singular components and adapts minor components; it consistently outperforms LoRA and PiSSA across the reported evaluation benchmarks.

  • Problem

    Full finetuning requires substantial computational resources, while LoRA-based methods optimize randomly initialized updates in an unguided subspace that may interfere with pretrained features.

    Source: §1 · p. 1

  • Method

    MiLoRA decomposes pretrained weights into principal and minor singular components, freezes the principal components, and initializes trainable low-rank updates from the minor components.

    Source: §1 · pp. 1–2

  • Results

    MiLoRA consistently outperforms LoRA and PiSSA across the reported reasoning, instruction-following, and vision-language benchmarks; on commonsense reasoning it gains 1.6 and 1.1 accuracy points over LoRA on LLaMA2-7B and LLaMA3-8B, respectively.

    Source: §4.1 · p. 5, §7 · p. 9

  • Takeaways & Limitations

    Adapting minor singular components is reported to learn finetuning tasks while better preserving pretrained knowledge, without sacrificing training or inference efficiency.

    Source: §1 · p. 2, §3 · p. 2

    Evaluation primarily covers LLaMA-family models and LLaVA-1.5, uses prior-work hyperparameter configurations rather than exhaustive task-specific searches, and leaves other tasks and models such as Mistral and Gemma for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory cost. Previous LoRA-based approaches initialize the low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might interfere with the well-learned subspace of the pretrained weight matrices. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principal singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principal matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the labeled dataset. Extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks present the superior performance of our method.

1 Introduction

Full finetuning is computationally demanding, while LoRA-based methods can optimize in an unguided subspace that may interfere with pretrained features. MiLoRA freezes principal singular components and adapts minor components, with reported gains over LoRA across several benchmark areas.

1 Introduction

  • Full finetuning requires substantial computational resources, restricting LLM use across diverse scenarios.Source: §1 · p. 1
  • LoRA reduces finetuning costs by updating low-rank matrices while freezing pretrained weights, but random initialization leaves the update subspace unguided.The authors suspect this may override important pretrained features.Source: §1 · p. 1
  • MiLoRA decomposes each weight matrix by singular-value magnitude, treating large-value components as principal and small-value components as minor.The paper characterizes principal components as capturing pretrained knowledge and minor components as containing noisy or long-tail information.Source: §1 · pp. 1–2
  • MiLoRA freezes principal components and adapts minor ones; it reports gains over LoRA of +1.6/+1.1 on commonsense reasoning, +2.0 on math reasoning, +2.9 on instruction following, and +1.4 on visual instruction following.The reported comparisons are on LLaMA2-7B/LLaMA3-8B for commonsense reasoning, LLaMA2-7B for math and instruction following, and LLaVA1.5-7B for visual instruction following.Source: §1 · p. 2

2 Preliminaries

The preliminaries introduce singular value decomposition and low-rank adaptation. LoRA keeps pretrained weights frozen and learns low-rank updates, but its random initialization does not select an optimal update subspace.

2 Preliminaries

  • SVD expresses a matrix as W = UΣV ⊤, with U and V containing left and right singular vectors and Σ holding singular values in descending order.The passage assumes m ≤ n to simplify notation.Source: §2 · p. 2
  • LoRA models weight changes with trainable low-rank matrices A and B while keeping the pretrained weight matrix frozen.Its updates can be merged into pretrained linear weights for inference without additional compute or GPU memory relative to full finetuning.Source: §2 · p. 2
  • LoRA initializes B to zero and A with a zero-mean random Gaussian, making the initial weight change zero; random initialization may fail to select an optimal update subspace.Source: §2 · p. 2

3 Methodology

MiLoRA uses singular values to separate pretrained weights into principal and minor components, initializes low-rank matrices from the minor component, and freezes the principal component during finetuning. The initialization guides updates toward a less-optimized subspace without imposing a strict orthogonality constraint.

3 Methodology

  • MiLoRA selects minor singular components for optimization, treating them as noisy or long-tail information while principal components carry important features.The method is intended to learn from finetuning data while maintaining pretrained-model knowledge.Source: §3 · p. 2
  • The weight matrix is split by singular-value magnitude, with r denoting the number of minor singular values.The minor matrix initializes low-rank matrices Am and Bm.Source: §3 · p. 3
  • During finetuning, MiLoRA keeps principal components Wp fixed and uses minor matrix Wm to initialize low-rank matrices Am and Bm.The figure depicts the principal matrix as frozen while the low-rank matrices are updated.Source: §3 · p. 3, Figure 1 · p. 3
  • Guiding optimization through initialization is intended to reduce interference with pretrained knowledge and requires only rank r, unlike vanilla LoRA’s scaling-factor and initialization hyperparameters.The authors choose this softer approach over strict orthogonality constraints, describing it as simpler, more training-efficient, and more flexible.Source: §3 · p. 3

4 Experiments

MiLoRA was evaluated against LoRA and PiSSA on NLP and multimodal tasks spanning 18 datasets. It consistently surpassed both baselines across language-model benchmarks and achieved the best results on visual instruction tuning.

4 Experiments

  • The experiments compared MiLoRA with LoRA and PiSSA across NLP and multimodal tasks covering 18 datasets.Experiments used eight NVIDIA L40 GPUs unless otherwise specified.Source: §4 · p. 3

4.1 Experiments on Large Language Model

  • Across commonsense reasoning, math reasoning, and instruction following, MiLoRA consistently surpassed LoRA and PiSSA.On LLaMA2-7B, its average gains were 1.6 and 5.4 accuracy points for commonsense reasoning, 2.0 and 3.7 EM points for math, and 1.5 LC WR and 0.5 WR on AlpacaEval 2.0, respectively; math remained 2.5 average EM points below full finetuning.Source: Table 1 · p. 4, Table 2 · p. 4, §4.1 · p. 5
  • On FollowBench, MiLoRA performed particularly well on levels 4 and 5 and improved average instruction-following results by 2.9 and 2.7 points over LoRA and PiSSA.The evaluation covered AlpacaEval 2.0, FollowBench, and IFEval.Source: §4.1 · p. 5, Table 3 · p. 5

4.2 Experiments on Vision-Language Model

  • MiLoRA achieved the best visual instruction tuning results, exceeding LoRA and PiSSA by 1.4 and 4.4 average accuracy points, respectively.The evaluation used LLaVA1.5-7B across seven vision-language tasks.Source: §4.2 · p. 6, Table 4 · p. 6

5 Understanding MiLoRA

MiLoRA is evaluated against other PEFT methods, its minor-component initialization, and analyses of its weight updates and forgetting. Across these analyses, the passages report strong benchmark performance, alignment with minor singular directions, and the lowest forgetting loss among the compared methods.

5.1 Comparison with More PEFTs

  • MiLoRA outperforms LoRA-GA by 1.1 points on GSM8K and 4.6 points on Human-eval, with training time consistent with LoRA after less than six minutes of SVD.DoRA and AdaLoRA require additional training-time overhead.Source: §5.1 · p. 6

5.2 Is Minor Singular Component Adaptation Important?

  • Minor-component initialization achieves the best performance across both math-reasoning datasets among principal, random-sampled, and minor singular-component initializations.The experiments use LLaMA2-7B on MetaMathQA 395K for one epoch.Source: §5.2 · p. 6, Table 6 · p. 6

5.3 How does the Matrix Update ∆W Compare to W?

  • LoRA and MiLoRA updates correlate with the pretrained weight slightly more strongly than a random matrix, while PiSSA’s update correlates strongly with the weight.Compared with LoRA, MiLoRA has a significantly smaller amplification factor, indicating reduced impact on features already present in the weight matrix.Source: §5.3 · p. 7, Table 7 · p. 7
  • MiLoRA’s update focuses on directions associated with bottom singular vectors, especially in MLP modules, while PiSSA aligns closely with top singular vectors.LoRA and MiLoRA do not significantly optimize in the top-singular-vector directions.Source: Figure 2 · p. 7, §5.3 · p. 8

5.4 MiLoRA Forgets Less than Baselines

  • MiLoRA has the lowest forgetting loss among the compared methods on the WikiText-103 test set after math-reasoning finetuning.The metric uses the LLaMA2-7B model and cross-entropy against the pretrained base model’s next-token distribution.Source: §5.4 · p. 8, Table 8 · p. 8

5.5 Comparison Between MiLoRA and PiSSA

  • Under PiSSA hyperparameters, PiSSA outperforms LoRA and MiLoRA; under the authors’ hyperparameter setup, all methods perform better, and MiLoRA beats LoRA and PiSSA by 1.4 and 4.3 average scores, respectively.Source: §5.5 · p. 8

6 Related Work

Related work frames parameter-efficient finetuning as a way to reduce finetuning resources, with adapter, LoRA, and prompt-based approaches as major categories. It describes LoRA variants and concurrent approaches that target rank selection, scaling, weight components, or gradient alignment.

Parameter-Efficient Finetuning of LLMs

  • PEFT seeks to reduce finetuning resources; adapter methods add feedforward submodules, LoRA updates low-rank matrices, and prompt methods update soft prompts.The passage notes that adapters add inference computation, while LoRA-based methods are widely used and add no inference computational overhead.Source: §6 · p. 9

LoRA and Its Variants

  • LoRA variants address rank selection, scaling, weight magnitude and direction, or adapting principal singular components to approximate full finetuning.The cited examples include AdaLoRA, ALoRA, rsLoRA, DoRA, and PiSSA.Source: §6 · p. 9

7 Conclusion

The conclusion presents MiLoRA as a low-rank method that adapts minor singular components to learn finetuning tasks while preserving pretrained knowledge. It reports consistent gains over LoRA and PiSSA across the evaluated language and vision-language benchmarks without sacrificing training or inference efficiency.

7 Conclusion

  • MiLoRA adapts minor singular components and is reported to outperform LoRA and PiSSA across commonsense, math, instruction-following, and vision-language benchmarks without sacrificing training or inference efficiency.The authors state that the method aims to learn finetuning tasks while better preserving pretrained knowledge.Source: §7 · p. 9

Limitations

The evaluation is concentrated on LLaMA-family models and LLaVA-1.5, and does not exhaustively search hyperparameters for each task.

Limitations

  • Limited compute confines evaluation mainly to LLaMA-family models and LLaVA-1.5, without exhaustive task-specific hyperparameter searches; other models such as Mistral and Gemma remain future work.The evaluated tasks focus on commonsense reasoning, math reasoning, and instruction following.Source: Limitations · p. 9

A Detailed Experiment Setups

The appendix defines how subspace similarity is measured and lists hyperparameter configurations for LLM and vision-language experiments.

A.1 Similarity Metric Between Subspaces

  • The similarity measure ranges from 0 for complete separation to 1 for complete overlap, comparing subspaces formed from the matrices’ left singular vectors.For ΔW and W, the method extracts r left singular vectors from each matrix before computing their subspace similarity.Source: §A.1 · p. 13

A.2 Our Hyperparameter Setup for LLM

  • The LLM-adapters hyperparameter setup follows Hu et al. (2023) and covers commonsense reasoning, math reasoning, and instruction-following tasks.Table 10 presents the configurations for these three task groups.Source: §A.2 · p. 13, Table 10 · p. 13

A.3 PiSSA Hyperparameter Setup for LLM

  • The appendix provides the detailed hyperparameter configuration used by PiSSA.Source: §A.3 · p. 13

A.4 Hyperparameter Setup for Vision-Language Model

  • Vision-language finetuning uses Liu et al. (2024a)’s LoRA configuration without tuning across PEFT methods for a fair comparison.Table 11 lists the configurations for visual instruction tuning.Source: §A.4 · p. 13, Table 11 · p. 13

B.1 Experiments on Qwen2.5-7B

On Qwen2.5-7B, MiLoRA outperforms LoRA and PiSSA on math reasoning, and results across additional architectures support its effectiveness across diverse model architectures.

  • Experiments on LLaMA2, LLaMA3, and LLaVA-1.5, alongside Qwen2.5-7B, support MiLoRA’s effectiveness across diverse model architectures.Source: §B.1 · pp. 13–14
  • MiLoRA outperforms LoRA and PiSSA by average 1.6 and 3.5 EM scores, respectively, on Qwen2.5-7B math reasoning.Table 13 reports GSM8K and MATH evaluation results.Source: §B.1 · p. 13, Table 13 · p. 14

B.2 Complementary Weighet Metrics Analysis

The appendix reports a complementary weight-matrix metric analysis and discusses MiLoRA’s training-cost implications and potential misuse of PEFT.

B.2 Complementary Weighet Metrics Analysis

  • Table 14 reports the Frobenius norm of U^⊤WV for top-r singular vectors from W, a random matrix, or PEFT ΔW, using LLaMA2-7B’s 16th-layer MLP down projection.Source: Table 14 · p. 14, §B.2 · p. 14

C Broader Impacts

  • MiLoRA’s lower training costs may help users with limited computational resources finetune large models, while PEFT’s potential misuse motivates investigating prevention strategies.Source: C Broader Impacts · p. 14

Related papers

Loading 2406.09044v3…