PaperlayerView exact revision on arXiv

Source-linked AI summary

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models

Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, Tuo Zhao

arXiv:2310.08659v4cs.CLcs.AIcs.LG

TL;DR

Combining quantization with LoRA can leave the initialized model discrepant from pretrained weights. LoftQ jointly quantizes pretrained weights and computes a low-rank LoRA initialization; across NLU, QA, summarization, and NLG, it surpasses existing methods, especially in low-bit settings.

  • Problem

    When LoRA adapters are attached to a quantized backbone, the initialized weights can differ from the pretrained weights because of quantization.

    Source: §2.3 · p. 5

  • Method

    LoftQ jointly approximates pretrained weights through quantization and low-rank approximation to initialize subsequent LoRA fine-tuning.

    Source: §1 · p. 2

  • Results

    Across NLU, question answering, summarization, and NLG, LoftQ surpasses QLoRA, including over 8% gain on MNLI with both 2-bit NormalFloat and 2-bit uniform quantization.

    Source: §1 · p. 3, §7 · p. 14

  • Takeaways & Limitations

    LoftQ demonstrates effectiveness across encoder-only, encoder-decoder, and decoder-only models, particularly in low-bit quantization regimes.

    Source: §7 · p. 14

    The quantized update Qt is not an exact minimizer because each step fixes the preceding low-rank approximation.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. In this work we focus on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that simultaneously quantizes an LLM and finds a proper low-rank initialization for LoRA fine-tuning. Such an initialization alleviates the discrepancy between the quantized and full-precision model and significantly improves generalization in downstream tasks. We evaluate our method on natural language understanding, question answering, summarization, and natural language generation tasks. Experiments show that our method is highly effective and outperforms existing quantization methods, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. The code is available on https://github.com/yxli2123/LoftQ.

1 Introduction

LoftQ jointly quantizes pre-trained models and finds a low-rank initialization for LoRA fine-tuning to reduce quantization-induced mismatch. The paper reports improved downstream performance over QLoRA, particularly at low bit widths.

1 Introduction

  • Pre-trained language models support NLU and NLG tasks but their millions or billions of parameters create substantial compute and memory demands, especially in shared, resource-constrained deployments.Source: §1 · pp. 1–2
  • Quantization converts high-precision weights to discrete values, while LoRA adapts a model through low-rank matrices without altering its original weights (Hu et al., 2021).Converting 16-bit floats to 4-bit integers reduces storage overhead by 75%.Source: §1 · p. 2
  • QLoRA’s quantized initialization degrades severely below 3-bit, and its fine-tuning performance falls as quantization bits decrease, with failure below 3-bit.Quantization error can adversely affect LoRA initialization, particularly in the 2-bit regime.Source: §1 · pp. 2–3, Figure 1 · p. 3
  • LoftQ jointly applies quantization and low-rank approximation to pre-trained weights, reducing initialization discrepancy and providing a better starting point for LoRA fine-tuning.The paper reports that discrepancy mitigation is especially effective at 2-bit.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • LoftQ consistently outperforms QLoRA across precision levels, gaining 1.1 and 0.8 Rouge-1 on XSum and CNN/DailyMail at 4-bit, and over 8% on MNLI and more than 10% on SQuADv1.1 with both 2-bit quantizers.The 2-bit results use NormalFloat and uniform quantization; the paper reports no case where LoftQ performs worse than QLoRA.Source: §1 · pp. 2–3

2 Background

The background introduces transformer layers, quantization, and low-rank adaptation, then highlights how quantization can make a LoRA-adapted model’s starting weights differ from the pretrained weights.

2.1 Transformer Models

  • A transformer layer comprises multi-head self-attention and a feed-forward network, with residual connections followed by layer normalization.The feed-forward network uses two linear transformations and an activation function.Source: §2.1 · p. 4

2.2 Quantization

  • N-bit quantization maps high-precision values to integers; uniform quantization uses min–max normalization, while NF4 assumes normally distributed values (Dettmers et al., 2023).For simulated matrix quantization, encoded integer weights are temporarily dequantized through a lookup table for multiplication (Bai et al., 2020; Shen et al., 2020).Source: §2.2 · p. 4

2.3 Low-Rank Adaptation

  • LoRA attaches trainable low-rank matrices A and B to a frozen pretrained weight matrix W, but quantizing W means Q + AB⊤ no longer equals W at initialization.During fine-tuning, W stays fixed while A and B are updated.Source: §2.3 · p. 5

3 Method

LoftQ alternates quantization and low-rank approximation to jointly initialize a quantized LLM backbone and LoRA adapters, alleviating quantization discrepancy and improving downstream generalization.

3.1 LoRA-Aware Quantization

  • LoftQ jointly optimizes the quantized backbone Q and low-rank adapters A,B to approximate pretrained weights, unlike outright quantization that ignores subsequent LoRA fine-tuning.The joint objective minimizes the Frobenius-norm discrepancy between the original high-precision weight and the quantized weight plus its low-rank approximation.Source: §3.1 · p. 5

3.2 Alternating Optimization

  • LoftQ alternates quantizing the residual after the preceding low-rank approximation with an SVD-based rank-r approximation of the resulting quantization residual.The framework supports different quantization functions; the paper uses NF4 and uniform quantization as examples.Source: §3.2 · p. 6
  • T=1 is sufficient to mitigate quantization discrepancy, while further alternating optimization finds an initialization closer to W and further improves performance.At T=1, Q1 is the QLoRA quantized weight and A1,B1 are obtained by applying SVD to the residual W − Q1.Source: §3.2 · p. 6

3.3 Applying to LoRA Fine-tuning

  • LoRA fine-tuning initializes the backbone with LoftQ’s integer matrix M and adapters A_T,B_T, freezes M, and optimizes only the adapters.The integer weights are temporarily dequantized for forward propagation, while adapter gradients and optimizer state alone are used in backpropagation, reducing training cost.Source: §3.3 · pp. 6–7

4 Experiments

Across DeBERTaV3, BART, and LLAMA-2 experiments, LoftQ generally outperforms QLoRA and can produce usable results where QLoRA fails to converge, particularly at low bit widths. The analysis also reports that alternating optimization improves downstream performance with few steps and is not sensitive to the step count.

4.1 Encoder-only Model: DeBERTaV3

  • 88.0% accuracy on MNLI-m with uniform 2-bit quantization exceeded QLoRA by 8%, and LoftQ approached full fine-tuning on SST and SQuADv1.1.Across ranks, quantization methods, and datasets, LoftQ consistently outperformed QLoRA; 4-bit results for both methods were close to full fine-tuning.Source: §4.1 · p. 8, Table 2 · p. 9, Table 1 · p. 9
  • LoftQ converged on CoLA in all tested settings, while QLoRA failed for both quantization methods and ranks; LoftQ scored 60.5 with uniform quantization at rank 32.The passage attributes LoftQ's robust performance to preserving the pretrained weights' starting point.Source: §4.1 · p. 8

4.2 Encoder-Decoder Model: BART

  • LoftQ consistently outperformed QLoRA at both ranks on both summarization datasets in 4-bit experiments, and surpassed full-precision LoRA at both ranks on XSum.At 2 bits, LoftQ converged to reasonable results while QLoRA did not converge, which the authors attribute to narrowing the initialization gap.Source: §4.2 · pp. 9–10, Table 3 · p. 10, Table 4 · p. 10

4.3 Decoder-only Model: LLAMA-2

  • 7.85 perplexity on WikiText-2 at 2-bit precision was achieved by LoftQ where QLoRA failed to converge; on GSM8K, LoftQ achieved 20.9% accuracy under the same condition.LoftQ consistently outperformed QLoRA on WikiText-2 across precision settings for both LLAMA-2 sizes, and performed better or on par on GSM8K.Source: §4.3 · p. 11, Table 5 · p. 12
  • LLAMA-2-13b full-precision LoRA was prone to overfitting on GSM8K, while weight decay helped it but failed to help LLAMA-2-7b; the authors describe quantization as implicit regularization.LoftQ outperformed full-precision LoRA on GSM8K with LLAMA-2-13b.Source: §4.3 · p. 11, Table 5 · p. 12
  • 5.9% and 12.7% accuracy boosts on GSM8K were observed with mixed precision for LLAMA-2-7b and LLAMA-2-13b, respectively.The first four layers used 4 bits and the remaining matrices used 2 bits.Source: §4.3 · p. 11, Table 5 · p. 12

4.4 Analysis

  • Alternating optimization improved results even with few steps: LoftQ reached 88.0% accuracy on MNLI-m in 5 steps and 21.14 ROUGE-2 in 1 step.Improvements showed diminishing returns beyond a certain step count, but Figure 3 is reported to show performance was not sensitive to T.Source: §4.4 · pp. 11–12

5 Discussion

The discussion compares LoftQ’s primary alternating-optimization order with an SVD-first alternative and examines different alternating-step counts. The SVD-first alternative remains better than QLoRA but worse than the primary method.

5 Discussion

  • The discussion identifies closer approximation of pretrained weights within the low-precision regime as a potential route to further performance improvement.Both the primary and SVD-first alternatives jointly minimize objective (6).Source: §5 · pp. 12–13
  • Figure 3 compares alternating-step counts T = 0, 1, 5, and 10 for DeBERTaV3-base, LLaMA-2-13b, and BART-large settings.T = 0 denotes QLoRA; T = 1, 5, and 10 denote LoftQ runs.Source: Figure 3 · p. 13, §5 · p. 13
  • LoftQ(SVD First) significantly outperforms QLoRA but performs worse than primary LoftQ on the reported 2-bit DeBERTaV3-base GLUE tasks.The alternative switches the SVD and quantization steps, and the table reports medians over four random seeds.Source: §5 · pp. 12–13, Table 6 · p. 13

6 Related Work

The related work contrasts QAT’s costly full-model adaptation with PTQ’s cost-efficient, task-guided calibration, and presents LoftQ’s LoRA-based approach as a lighter adaptation alternative. PTQ is described as usually less accurate than QAT.

Related Work

  • QAT adapts quantized models through simultaneous quantization and full-model fine-tuning, but LoftQ uses LoRA to provide a lighter approach that avoids its high training cost and difficult quantized-weight gradients (Liu et al., 2023).The cited QAT work is used for downstream-task adaptation.Source: §6 · p. 13
  • PTQ frameworks calibrate high-precision models on a small training-data subset to produce task-specific quantized models without backpropagation, making them cost-efficient but usually less accurate than QAT (Frantar et al., 2022; Xiao et al., 2023).Source: §6 · pp. 13–14

7 Conclusion

LoftQ alternates quantization and low-rank approximation of high-precision pre-trained weights to initialize subsequent LoRA fine-tuning. Across several task types and model architectures, it surpasses existing methods such as QLoRA and is especially effective in low-bit regimes.

7 Conclusion

  • LoftQ surpasses existing methods, including QLoRA, across natural language understanding, question answering, summarization, and generation for encoder-only, encoder-decoder, and decoder-only models.The authors report no observed case of LoftQ performing worse than QLoRA.Source: §7 · p. 14
  • LoftQ alternates quantization and low-rank approximation of high-precision pre-trained weights to produce an initialization for subsequent LoRA fine-tuning.Source: §7 · p. 14
  • LoftQ is particularly effective and robust in low-bit quantization regimes, including at the 2-bit level.Source: §7 · p. 14

A Model Compression Ratio and Memory Footprint · B Quantization Time

The section reports LoftQ backbone compression ratios, training-time GPU memory footprint, and quantization execution time for individual weight matrices.

A Model Compression Ratio and Memory Footprint

B Quantization Time

C GLUE Dataset Statistics · D Natural Language Understanding · D.1 GLUE with 4-bit

GLUE spans sentence classification, similarity and paraphrase, and natural language inference tasks. In the 4-bit GLUE evaluation, both methods achieve performance close to full fine-tuning.

C GLUE Dataset Statistics

D.1 GLUE with 4-bit

  • Both methods achieve performance close to full fine-tuning in the 4-bit GLUE results.The results cover DeBERTaV3-base models on the GLUE development set.Source: Table 11 · p. 20, §D.1 · p. 20
  • The 4-bit results use NF4 quantization and report the median over four seeds.Source: §D.1 · p. 20
  • N.A. denotes a model that does not converge, and the best result on each dataset is shown in bold.Source: §D.1 · p. 20

D.2 Training Details · E Summarization

The training appendix reports shared hyperparameter choices for uniform and nf2 quantization, along with task-specific batch sizes and LoftQ iteration counts.

D.2 Training Details

  • The LoftQ implementation is based on publicly available Hugging Face code (Paszke et al., 2019).Source: §D.2 · p. 20
  • The learning-rate search used {1 × 10−5, 5 × 10−5, 1 × 10−4, 5 × 10−4} for both uniform and nf2 quantization experiments.Source: §D.2 · p. 20
  • Batch sizes were 32 for GLUE and ANLI and 16 for SQuADv1.1; GLUE tasks used five LoftQ iterations.Source: §D.2 · p. 20
  • Tables 12 and 13 summarize task hyperparameters for DeBERTaV3-base under uniform and nf2 quantization, respectively.Source: §D.2 · p. 20, Table 12 · p. 20, Table 13 · p. 21

E.1 Training Details … H Extension to Convolutional Layers

The appendices give training settings for summarization and language-generation experiments, compare LoftQ with pruning, and extend low-rank adaptation to convolutional layers.

E.1 Training Details

  • For BART-large experiments, LoftQ uses one iteration; Adam is the optimizer, and learning rates are selected from six tested values.The learning-rate and other hyperparameter setups are summarized separately for CNN/DailyMail and XSum.Source: §E.1 · p. 21, Table 14 · p. 21, Table 15 · p. 21

F Natural Language Generation

G Comparison to Pruning

H Extension to Convolutional Layers

  • A convolutional layer maps an input feature map and 2D kernels to an output feature map through 2D convolution.The input has dimensions h×w×c1, and the output has dimensions h×w×c2.Source: H Extension to Convolutional Layers · p. 22
  • The convolution is reformulated as matrix multiplication by extending each input location with its neighboring vectors in the kernel window, then flattening the extended input and concatenated kernels.The extended input is flattened into Z, and the concatenated, flattened kernels form H.Source: H Extension to Convolutional Layers · p. 23
  • The flattened kernel matrix H is approximated by a rank-r factorization obtained through SVD, with r much smaller than the matrix dimensions.The factors have shapes c2×r and c1d^2×r.Source: H Extension to Convolutional Layers · p. 23
  • The low-rank matrix products can be restored as convolutions: Z×V uses r kernels of size c1×d×d, and M×U⊤ uses c2 kernels of size r×1×1.Source: H Extension to Convolutional Layers · p. 23

Related papers

Loading 2310.08659v4…