PaperlayerView exact revision on arXiv

Source-linked AI summary

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models

Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, Tuo Zhao

arXiv:2310.08659v4cs.CLcs.AIcs.LG

TL;DR

Quantization plus LoRA fine-tuning can leave a performance gap because quantization creates a discrepancy between quantized and full-precision weights. LoftQ jointly quantizes the model and finds a low-rank LoRA initialization to reduce that discrepancy, outperforming existing methods, especially in low-bit regimes.

  • Problem

    Quantization and LoRA fine-tuning can produce a downstream-task performance gap relative to full fine-tuning, while quantization work often neglects subsequent LoRA initialization.

  • Method

    LoftQ alternates quantization and low-rank approximation of original high-precision weights to obtain a LoRA-aware initialization.

  • Results

    LoftQ outperforms existing methods across encoder-only and decoder-only models and is particularly effective in 2-bit quantization regimes.

  • Takeaways & Limitations

    LoftQ provides an effective initialization framework for quantized LLMs undergoing LoRA fine-tuning across diverse downstream tasks.

  • Takeaways & Limitations

    The paper notes that full-precision LoRA fine-tuning can overfit because of insufficient regularization.

Abstract

from arXiv · show

Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. In this work we focus on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that simultaneously quantizes an LLM and finds a proper low-rank initialization for LoRA fine-tuning. Such an initialization alleviates the discrepancy between the quantized and full-precision model and significantly improves generalization in downstream tasks. We evaluate our method on natural language understanding, question answering, summarization, and natural language generation tasks. Experiments show that our method is highly effective and outperforms existing quantization methods, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. The code is available on https://github.com/yxli2123/LoftQ.

1 Introduction

Quantization reduces model storage, but its discrepancy from full precision can undermine LoRA initialization, especially at low bit widths. LoftQ jointly performs quantization and low-rank approximation, improving downstream results across tasks and precision settings.

  • 4-bit quantization converts 16-bit floating-point parameters into integers, reducing storage overhead by 75%.
  • LoRA adapts frozen pre-trained weights through low-rank matrices, leaving the original weights unchanged.
  • Quantization introduces a discrepancy from the original weights that can impair LoRA initialization, with severe degradation below 3 bits for QLoRA.
  • LoftQ jointly combines quantization and low-rank approximation to align the initialized model more closely with the original weights.
  • LoftQ consistently outperforms QLoRA across precision levels, including Rouge-1 gains of 1.1 on XSum and 0.8 on CNN/DailyMail with 4-bit quantization.
  • At 2 bits, LoftQ gains over 8% on MNLI and more than 10% on SQuADv1.1 using both NormalFloat and uniform quantization.

2 Background

The background introduces transformer components, simulated quantization, and LoRA. It explains that quantized backbones disrupt standard LoRA initialization because the resulting weights no longer equal the original pre-trained weights.

  • A transformer layer contains multi-head self-attention and a feed-forward network, followed by a residual connection and layer normalization.
  • The feed-forward network applies two linear transformations with an activation function: FFN(X) = σ(XW_f1 + b_1)W_f2 + b_2.
  • N-bit quantization maps high-precision values to integers in {0,1,...,2^N −1} using a normalization function.
  • Simulated quantization stores encoded integers and temporarily dequantizes them to simulated high-precision matrices during multiplication.
  • LoRA attaches low-rank matrices A and B to a frozen weight matrix W, updating A and B while W remains fixed.
  • With a quantized backbone Q = q_N(W), standard zero-initialized adapters produce Q + AB^⊤ that differs from the original pre-trained weight W.

3 Method

LoftQ jointly quantizes pretrained weights and initializes low-rank LoRA adapters to reduce quantization discrepancy before fine-tuning. Alternating quantization and low-rank approximation produces an initialization closer to the pretrained model and improves downstream performance.

  • LoftQ jointly quantizes an LLM and finds a low-rank initialization for LoRA fine-tuning.
  • Standard quantization converts pretrained weights directly to quantized weights, neglecting LoRA fine-tuning and causing downstream performance degradation.
  • Alternating Optimization: The method alternates quantization of residual weight differences with low-rank approximation to approximate the original pretrained weights.
  • LoftQ is compatible with different quantization functions and can be applied once, with its initialization reused across downstream tasks.
  • Alternating Optimization: T = 1 is sufficient to mitigate quantization discrepancy, while additional alternating optimization finds a closer initialization and further improves performance.
  • LoRA Fine-Tuning: During LoRA fine-tuning, the integer backbone is frozen and only low-rank adapter parameters receive gradients, reducing training cost.

4 Experiments

Experiments evaluate LoftQ across encoder-only, encoder-decoder, and decoder-only models using low-bit quantization, multiple datasets, quantizers, and adapter ranks. Across these settings, LoftQ generally improves performance and convergence relative to QLoRA, especially at 2 bits.

  • LoftQ consistently outperforms QLoRA across GLUE, SQuADv1.1, and ANLI settings with different ranks and quantization methods.
  • Encoder-only Model: DeBERTaV3: 88.0% accuracy on MNLI-m surpasses the QLoRA baseline by 8% with uniform 2-bit quantization.
  • Encoder-only Model: DeBERTaV3: LoftQ converges in all tested CoLA cases, whereas QLoRA fails to converge; uniform quantization at rank 32 reaches 60.5.
  • Encoder-Decoder Model: BART: LoftQ outperforms QLoRA on both summarization datasets and ranks at 4 bits, surpassing full-precision LoRA on XSum.
  • Encoder-Decoder Model: BART: At 2 bits, LoftQ reaches reasonable results while QLoRA does not converge in the BART experiments.
  • Decoder-only Model: LLAMA-2: On WikiText-2, LoftQ outperforms QLoRA across tested precision settings; at challenging 2-bit precision it reaches perplexity 7.85.
  • Decoder-only Model: LLAMA-2: LoftQ achieves 20.9% GSM8K accuracy at 2 bits where QLoRA does not converge, and mixed precision yields 5.9% and 12.7% boosts.

5 Discussion

The discussion examines alternating optimization choices in LoftQ and compares an alternative update order with the primary method. Even the alternative substantially outperforms QLoRA, while the primary method performs better.

  • The alternative LoftQ update order first obtains the low-rank approximation and then obtains the quantized weight.
  • Both update orders jointly minimize the objective in (6), making the alternative a valid method.
  • The alternative method significantly outperforms QLoRA, although it performs worse than the primary LoftQ version.
  • LoftQ is not sensitive to the alternating step T and consistently enhances downstream fine-tuning performance.

6 Related Work

Prior quantization approaches trade off adaptation quality, training cost, or gradient accessibility. LoftQ uses LoRA to provide a lighter approach for downstream task adaptation.

  • Quantization-Aware Training: QAT can produce task-specific quantized models but requires full-model fine-tuning and massive training costs.Its optimization also makes gradients of quantized weights difficult to compute.
  • LoftQ: LoftQ uses LoRA to sidestep these issues and provide a light approach for downstream task adaptation.
  • Post-Training Quantization: PTQ calibrates high-precision models with a small training-data subset before quantization for task adaptation.
  • Quantization-Aware Training: QAT does not require gradient backpropagation and is therefore cost-efficient, but usually achieves lower downstream-task accuracy.

7 Conclusion

LoftQ alternates quantization and low-rank approximation to initialize LoRA fine-tuning. Across several task families and model types, it surpasses existing methods, with particular robustness in low-bit regimes.

  • Method: LoftQ alternates quantization and low-rank approximation of high-precision pretrained weights to initialize subsequent LoRA fine-tuning.
  • Evaluation: Experiments cover natural language understanding, question answering, summarization, and natural language generation.
  • Results: LoftQ surpasses existing quantization methods, including QLoRA, across encoder-only, encoder-decoder, and decoder-only models.
  • Results: No observed result showed LoftQ performing worse than QLoRA.
  • Results: LoftQ is particularly effective and robust in low-bit quantization regimes such as 2-bit quantization.

A Model Compression Ratio and Memory Footprint

The appendix reports LoftQ’s compression ratios, GPU memory costs, and execution times across model and hardware settings. It includes LLAMA-2 on GSM8K as a memory-cost example and Intel Xeon CPU timing measurements.

  • Compression and Memory: LoftQ’s compression ratio is reported in Table 7, while GPU memory footprint is reported in Table 8.
  • Memory Footprint: GPU memory cost varies with models, tasks, sequence lengths, and batch sizes; LLAMA-2 on GSM8K is reported as an example.
  • Execution Time: LoftQ execution time for individual weight matrices is reported in Table 9 and tested on an Intel Xeon E5-2650 v4 CPU.

C GLUE Dataset Statistics

The GLUE benchmark contains classification, similarity, paraphrase, and natural-language-inference tasks. Table 11 reports 4-bit DeBERTaV3-base results, with medians over four seeds and NF4 as a reference.

  • Task Composition: GLUE includes single-sentence classification tasks SST-2 and CoLA.
  • Task Composition: GLUE includes similarity and paraphrase tasks MRPC, STS-B, and QQP.
  • Task Composition: GLUE includes natural language inference tasks MNLI, QNLI, RTE, and WNLI.
  • 4-bit Results: Both 4-bit methods achieve performance close to NF4 quantization on the reported benchmark.
  • 4-bit Results: Table 11 reports 4-bit LoftQ results for DeBERTaV3-base models on the GLUE development set, using medians over four seeds.

D.2 Training Details

The DeBERTaV3-base GLUE experiments use specified learning-rate searches, batch sizes, and LoftQ iterations under uniform and nf2 quantization settings.

  • Learning rates are selected from {1 × 10−5,5 × 10−5,1 × 10−4,5 × 10−4} for both uniform and nf2 quantization experiments.
  • GLUE tasks use batch size 32, while SQuADv1.1 uses batch size 16.
  • LoftQ uses 5 iterations for all GLUE tasks, with hyperparameters summarized separately for uniform and nf2 quantization.

E.1 Training Details

The experiments specify optimizer, learning-rate, batch-size, epoch, and model-specific settings, while also extending low-rank adapters to convolutional layers and comparing LoftQ with pruning.

  • Training Details: Adam is used as the optimizer, with learning rates searched over predefined candidate sets and optimal settings reported for different experiments.
  • Training Details: LoftQ uses 1 iteration for all BART-large experiments, with detailed settings reported for CNN/DailyMail and XSum.
  • Training Details: WikiText-2 uses batch size 32 and 2 epochs, whereas GSM8K uses batch size 16 and 6 epochs.
  • Training Details: LoftQ uses learning rates of 3 × 10−4 for LLAMA-2-7b and 1 × 10−4 or 3 × 10−4 for LLAMA-2-13b across listed settings.
  • Compression Comparison: LoftQ significantly outperforms the compared pruning methods on the DeBERTaV3-base model.
  • Compression Comparison: LoftQ consistently reduces training and storage memory, whereas pruning requires training the entire full-precision matrix and provides no training-stage memory savings.
  • Extension to Convolutional Layers: For convolutional layers, input features and kernels are flattened into Z and H, after which H is approximated by a low-rank matrix using SVD.
  • Extension to Convolutional Layers: The low-rank factors are restored as convolutional operations: Z × V becomes r kernels, and M × U⊤ becomes c2 one-by-one kernels.
Loading 2310.08659v4…