PaperlayerView exact revision on arXiv

Source-linked AI summary

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Fanxu Meng, Zhaohui Wang, Muhan Zhang

arXiv:2404.02948v4cs.LGcs.AI

TL;DR

Large-model fine-tuning is costly, and LoRA’s initialization can slow convergence. PiSSA initializes adapters from the pretrained matrix’s principal SVD components while freezing the residual, and experiments report consistent improvements over LoRA; QPiSSA further reduces quantization error relative to QLoRA.

  • Problem

    LoRA’s Gaussian-and-zero initialization can yield small, uninformative gradients and slow convergence during parameter-efficient fine-tuning.

  • Method

    PiSSA applies SVD to pretrained weight matrices, initializes trainable low-rank adapters from principal components, and freezes the residual components.

  • Results

    PiSSA and QPiSSA significantly outperform LoRA and QLoRA across reported NLG and NLU experiments, with QPiSSA also reducing quantization error by about 20% compared to QLoRA.

  • Takeaways & Limitations

    PiSSA provides a parameter-efficient alternative to LoRA that targets principal model components while retaining compatibility with quantization.

  • Takeaways & Limitations

    The paper leaves open whether PiSSA extends to convolutional layers and vision tasks, adaptive-rank LoRA improvements, and stronger theoretical explanations.

Abstract

from arXiv · show

To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes $ΔW \in \mathbb{R}^{m \times n}$ through the product of two matrices $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{r \times n}$, where $r \ll \min(m, n)$, $A$ is initialized with Gaussian noise, and $B$ with zeros. LoRA freezes the original model $W$ and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices $A$ and $B$ with the principal components of the original matrix $W$, and put the remaining components into a residual matrix $W^{res} \in \mathbb{R}^{m \times n}$ which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 12 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups. On the GSM8K benchmark, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, surpassing LoRA's 67.7% by 5.16%. Due to the same architecture, PiSSA is also compatible with quantization to further reduce the memory requirement of fine-tuning. Compared to QLoRA, QPiSSA exhibits smaller quantization errors in the initial stages. Fine-tuning LLaMA-3-70B on GSM8K, QPiSSA attains an accuracy of 86.05%, exceeding the performances of QLoRA at 81.73%. Leveraging a fast SVD technique, PiSSA can be initialized in only a few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA. Code is available at https://github.com/GraphPKU/PiSSA.

1 Introduction

LoRA reduces fine-tuning costs through low-rank adapters but can converge slowly because its initialization provides weak, uninformative gradients. PiSSA instead adapts principal components of the pretrained weights, while QPiSSA applies quantization to residual components to reduce error.

  • LoRA: LoRA uses low-rank matrices A and B to reduce trainable parameters and GPU memory while preserving the original model matrix W.The adapter update is ΔW = AB, with rank r much smaller than the matrix dimensions.
  • PiSSA: PiSSA applies SVD to W, trains its principal low-rank components, and freezes the residual components.The principal singular values and vectors initialize A and B, while the remaining singular components form frozen W res.
  • LoRA: LoRA’s Gaussian-and-zero initialization can produce small, uninformative gradients and slow convergence.A initially has a zero gradient while B has a random gradient, causing LoRA to spend time near its initial point.
  • PiSSA: PiSSA is designed to approximate the optimization direction of full-parameter fine-tuning by adapting the model’s principal components.The paper reports that PiSSA converges faster and outperforms LoRA.
  • Quantization: QPiSSA combines PiSSA with NF4 quantization and reduces quantization error by about 20% compared to QLoRA.QPiSSA keeps principal components in full precision and quantizes the residual matrix instead of the entire pretrained matrix.

2 Related Works

Related work develops PEFT methods that reduce trainable parameters and memory while retaining model capability, including adaptive-rank, weight-updating, pruning, and quantization-based extensions to LoRA.

  • PEFT: PEFT methods reduce fine-tuning parameters and memory requirements while targeting performance comparable to full fine-tuning.The related methods include partial fine-tuning and soft-prompt fine-tuning.
  • LoRA extensions: AdaLoRA dynamically learns layer-specific ranks, while DeltaLoRA updates original weights using adapter parameters.LoSparse combines LoRA with pruning-related procedures.
  • Quantized PEFT: QLoRA combines LoRA with 4-bit NF4 quantization, double quantization, and paged optimizers for memory-efficient large-model fine-tuning.QLoRA enabled fine-tuning a 65B-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning performance on reported tasks.

3 PiSSA: Principal Singular Values and Singular Vectors Adaptation

PiSSA applies SVD to model weights, placing principal components in trainable low-rank adapters and freezing the residual components. This preserves the pretrained model initially while directing optimization toward its most essential weight directions.

  • Method: PiSSA decomposes each self-attention and MLP weight matrix W with SVD into principal and residual singular components.The principal group contains the largest singular values; the residual group contains the remaining smaller values.
  • Method: The principal singular components initialize adapter matrices A and B, while the residual components build a frozen residual matrix.This makes the trainable adapter represent the principal low-rank matrix W pri = AB.
  • Forward pass: PiSSA’s forward pass combines the frozen residual matrix with the trainable adapter as Y = X(W res + AB), equivalent to the original Y = XW at initialization.The integration of AB with W res preserves the pretrained model’s full capability at the beginning of fine-tuning.
  • Optimization: Gradients update A and B, and because the adapter contains W’s largest-singular-value directions, optimization can more directly target essential model components.The paper contrasts this with LoRA’s Gaussian-zero initialization, which can produce small or random early gradients.
  • Practical properties: PiSSA retains LoRA’s parameter-efficient architecture, reduced-memory training, residual quantization, and straightforward deployment integration.The paper also describes storing low-rank updates instead of a dense parameter matrix.

4 QPiSSA: An Extension Method with Lower Quantization Error

QPiSSA combines PiSSA’s principal-component adapter with quantization of the residual model rather than the full base model. Removing large singular-value components narrows the residual distribution, supporting lower quantization error and better fine-tuning performance than QLoRA.

  • QLoRA: QLoRA quantizes the base model W to NF4 while keeping full-precision adapters, so its quantization error equals ||W − nf4(W)||∗.The adapter correction AB does not reduce this error under the stated formulation.
  • QPiSSA: QPiSSA quantizes only the residual matrix W res and keeps the principal components in full-precision AB, yielding error ||W res − nf4(W res)||∗.Its error is defined relative to the residual model rather than the entire base model.
  • Quantization behavior: Removing large-singular-value components gives W res a narrower value distribution than W, which benefits quantization error reduction.Figure 3 compares singular-value and data distributions for W and W res in a LLaMA 2-7B query projection layer.
  • Fine-tuning performance: QPiSSA’s gradient direction is similar to PiSSA’s, and it achieves significantly better fine-tuning performance than QLoRA.The passage attributes this comparison to both reduced quantization error and the shared PiSSA optimization direction.

5 Experiments

Experiments across NLG and NLU tasks show that PiSSA consistently outperforms LoRA, with faster convergence and stronger accuracy across model sizes, ranks, and quantization settings.

  • 5.1 Evaluating the Performance of PiSSA on both NLG and NLU Tasks: Across all evaluated NLG models and tasks, PiSSA consistently surpasses LoRA, with robustness across data amounts, epochs, precision settings, model sizes, and trainable-parameter proportions.
  • 5.1 Evaluating the Performance of PiSSA on both NLG and NLU Tasks: PiSSA outperforms LoRA on 7 of 8 NLU tasks, achieving a 1.21% overall average improvement.On MNLI, PiSSA’s final average loss is 0.17 versus LoRA’s 0.24.
  • 5.2 Experiments using Full Data and More Epochs: PiSSA’s loss drops rapidly during the first 100 steps, while its gradient norm is significantly higher than LoRA’s and follows a trend similar to full fine-tuning.
  • 5.2 Experiments using Full Data and More Epochs: PiSSA maintains lower loss than LoRA throughout training and achieves higher accuracy, surpassing full-parameter fine-tuning in most cases.The authors hypothesize that PiSSA removes noisy directions that contribute to gradient norm without reducing loss.
  • 5.3 Conducting 4-bit Quantization Experiments: PiSSA reduces quantization error by about 20% versus directly quantizing the base model, with a 49% reduction in all LLaMA-3-70B Key projection layers.
  • 5.3 Conducting 4-bit Quantization Experiments: QPiSSA reduces loss faster than PiSSA and full fine-tuning in the first 100 steps, then achieves lower loss and higher accuracy than QLoRA, LoftQ, and full-precision LoRA.
  • 5.5 Experiments on Various Ranks: Across ranks 1 to 128, PiSSA consistently outperforms LoRA on GSM8K and MATH, while increasing rank allows it to reach and surpass full-parameter fine-tuning.

6 Conclusion

PiSSA uses SVD-derived principal components to initialize a low-rank adapter while freezing residual components, and its quantized version QPiSSA outperforms LoRA and QLoRA across diverse settings.

  • PiSSA applies SVD to pretrained weights, uses principal components to initialize a low-rank adapter, and freezes the residual components.
  • Across NLG and NLU tasks, model sizes, training steps, precisions, and trainable-parameter amounts, PiSSA and QPiSSA significantly outperform LoRA and QLoRA.
  • Because PiSSA shares LoRA’s architecture, it can be used in existing LoRA pipelines as an alternative initialization method.

7 Limitation

The paper identifies unresolved questions about PiSSA’s applicability beyond language models, compatibility with adaptive-rank LoRA methods, and theoretical justification.

  • PiSSA’s adaptation to convolutional layers and potential benefits for vision tasks remain unaddressed.
  • The paper leaves open whether PiSSA can benefit from adaptive-rank methods such as AdaLoRA and DyLoRA.
  • More theoretical explanations for PiSSA’s advantages over LoRA remain an open question.

The Supplementary Material for The Paper “PiSSA: Principal Singular Values and Singular Vectors

The supplementary material combines PiSSA with LoRA improvements, evaluates faster SVD initialization, implementation alternatives, and precision choices, while reporting further performance comparisons.

  • A Enhancing PiSSA with LoRA Improvement Methods: PiSSA differs from AdaLoRA because it performs actual SVD on the original model and fine-tunes principal components while freezing residual components.
  • A Enhancing PiSSA with LoRA Improvement Methods: PiSSA’s principal singular values and orthogonal vectors make it suitable for combination with AdaLoRA and other LoRA improvements.The supplementary material reports that combined improvements further enhance PiSSA’s effectiveness.
  • B Fast Singular Value Decomposition: Fast SVD uses randomized approximate decomposition to reduce initialization time relative to conventional SVD.The comparison measures initialization time, decomposition error, and final training loss.
  • C Equivalently Converting PiSSA into LoRA: PiSSA can be converted into a directly insertable LoRA-style adapter, avoiding SVD at deployment while requiring twice the adapter storage.The alternative stores ΔA and ΔB with dimensions m×2r and 2r×n.
  • D Comparison of Fine-Tuning in BF16 and FP32 Precision: BF16 and FP32 produce materially different GSM8K results across models, and the experiments do not establish which precision is superior.For example, FP32 is 5.16% higher for LLaMA-2-7B but 7.21% and 6.52% lower for Mistral-7B and LLaMA-3-8B.

E Reducing Quantization Error through Multiple Iteration of SVD

The supplementary material studies iterative QPiSSA decomposition and singular-component initialization, finding that multiple iterations reduce quantization error and principal components improve fine-tuning outcomes.

  • E Reducing Quantization Error through Multiple Iteration of SVD: QPiSSA alternates SVD-based adapter updates with residual-weight updates after NF4 quantization.The procedure initializes from SVD(W), forms a residual, and repeatedly decomposes the quantization-adjusted matrix.
  • E Reducing Quantization Error through Multiple Iteration of SVD: QPiSSA-r64 with 5 iterations on LLaMA-3-8B reduces quantization error nearly twice as much as the 1-iteration version.
  • F Influence of Singular Values and Vectors: Principal singular values and vectors consistently reduce training loss and improve GSM8K and MATH accuracy across three evaluated models.The comparison uses LLaMA 2-7B, Mistral-7B-v0.1, and Gemma-7B fine-tuned on MetaMathQA.
  • G The Residual Matrices having a Narrower Distribution: After PiSSA initialization, residual matrices have means closer to zero and smaller standard deviations than the original matrices.The averaged difference is smaller than in the first layer, possibly because middle layers have more even eigenvalue distributions.

H Comparing the Quantization Error of QLoRA, LoftQ and QPiSSA

This section compares quantization errors in QLoRA, LoftQ, and QPiSSA, explaining why quantizing PiSSA’s residual matrix can preserve principal components more effectively.

  • QLoRA quantizes the full model matrix with NF4, leaving the adapter initialized with Gaussian noise and zeros.
  • LoftQ decomposes QLoRA’s quantization-error matrix and uses its largest singular values to reduce residual error.It removes only the largest r singular values from the QLoRA error matrix.
  • PiSSA quantizes the residual model rather than the base model, preserving the principal components in the adapter.The paper argues that preserving principal components of W is more beneficial than preserving those of the smaller error matrix.
  • PiSSA’s residual matrix has a narrower, Student’s t-distribution-compatible value distribution, enabling lower 4-bit NF4 quantization error.Removing large singular-value components makes the residual easier to fit with a Student’s t-distribution with higher degrees of freedom.

I Combining QPiSSA with Various Quantization Methods

QPiSSA extends PiSSA across NF4, GPTQ, and INT8 quantization, reducing quantization error while retaining PiSSA’s convergence and performance advantages.

  • Quantization rationale: PiSSA reduces quantization error by reshaping the weight distribution and preserving larger values in full precision.The method reduces outliers, makes values more Gaussian-like, and retains critical values outside the quantized residual portion.
  • INT8 results: 18.16% lower quantization error is achieved by QPiSSA with INT8 on LLaMA-3-8B, outperforming INT8 QLoRA.
  • GPTQ results: GPTQ-4bit perplexity on C4 decreases from 20.79 to 6.23 when PiSSA is applied to LLaMA-3-8B.For GPTQ, perplexity on WikiText-2 is used as the quantization-error proxy because the main-paper nuclear-norm measure is not applicable.
  • Overall comparison: QPiSSA maintains PiSSA’s fast convergence and performance while minimizing quantization error across quantization methods.

J Evaluating PiSSA on Mixtral and Gemma with More Training Steps

Supplementary experiments extend PiSSA comparisons to longer MetaMathQA-395K training and additional linear layers, showing favorable loss, gradient, and quantization-error behavior.

  • Extended training: Three-epoch MetaMathQA-395K experiments compare full fine-tuning, LoRA, and PiSSA on Mistral-7B and Gemma-7B using loss, gradient norm, and GSM8K accuracy.
  • Extended training: Full-parameter fine-tuning has higher first-epoch loss than LoRA and PiSSA on Mistral-7B and Gemma-7B, while its loss later decreases sharply.The passage interprets the sharp decrease as overfitting to the training data and suggests weaker first-epoch generalization than LoRA and PiSSA.
  • Quantization layers: Figure 14 evaluates quantization-error reduction for k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj layers under different ranks.
  • Quantization layers: PiSSA consistently yields lower quantization-error reduction ratios than LoftQ across the evaluated LLaMA-2-7B linear layers and ranks.

K.2 Evaluation Performance for More Model on Various Ranks

Additional evaluations examine PiSSA across models, ranks, training dynamics, and GLUE implementation settings. PiSSA generally improves early optimization and performance, but higher ranks can over-parameterize earlier on some models.

  • Performance across models: PiSSA matches or surpasses full-parameter fine-tuning on LLaMA-2-7B and Mistral-7B with fewer trainable parameters.
  • Performance across models: On Gemma-7B, PiSSA surpasses full-parameter fine-tuning at rank=1, but performance declines by rank=128 due to earlier over-parameterization.
  • Training dynamics: PiSSA reduces training loss faster than LoRA across ranks and retains a lower final loss, especially at smaller ranks.
  • Training dynamics: PiSSA maintains higher gradient norms than LoRA throughout training, indicating more efficient fitting of the training data.
  • GLUE evaluation: GLUE evaluation applies PiSSA and LoRA to DeBERTa-v3-base with 1.33M trainable parameters for each method.The comparison covers CoLA, SST, MNLI, RTE, QQP, MRPC, QNLI, and STS-B with task-specific metrics.
  • Gradient subspaces: LoRA’s matrix A gradient remains zero initially across seeds, whereas PiSSA’s gradient direction remains consistent across the tested initializations.LoRA’s zero-initialized B causes the zero gradient for A, while Gaussian initialization of B produces seed-dependent gradient directions.
  • Gradient subspaces: After five updates, PiSSA reduces loss from 0.8884 to 0.3346, compared with LoRA’s reduction to 0.5538.The experiment attributes the faster convergence to updating along principal singular value directions.
Loading 2404.02948v4…