Source-linked AI summary
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, Qi Tian
TL;DR
LLMs’ computational and memory demands hinder deployment, especially on edge devices, while combining efficient adaptation with low-bit quantization remains challenging. QA-LoRA uses group-wise operators to balance quantization and adaptation, and experiments on LLaMA families show effective accuracy and efficiency across settings.
Problem
LLMs’ heavy computational and memory requirements restrict deployment on edge devices, while post-training quantization after adaptation can reduce accuracy at low bit widths.
Method
QA-LoRA introduces group-wise operators that individually quantize weight groups while sharing adaptation parameters across each group.
Results
QA-LoRA consistently outperforms QLoRA with post-training quantization, matches QLoRA without it, and is more than 50% faster during inference.
Takeaways & Limitations
QA-LoRA provides an efficient joint quantization-and-adaptation method whose fine-tuned model remains quantized after weight integration.
Takeaways & Limitations
Low-bit quantization approximates pretrained weights and may harm language-understanding accuracy; QLoRA’s post-training quantization can cause significant accuracy loss at low bit widths.
Abstract
from arXiv · showhide
Recently years have witnessed a rapid development of large language models (LLMs). Despite the strong ability in many language-understanding tasks, the heavy computational burden largely restricts the application of LLMs especially when one needs to deploy them onto edge devices. In this paper, we propose a quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies in the imbalanced degrees of freedom of quantization and adaptation, and the solution is to use group-wise operators which increase the degree of freedom of quantization meanwhile decreasing that of adaptation. QA-LoRA is easily implemented with a few lines of code, and it equips the original LoRA with two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized (e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the LLM and auxiliary weights are naturally integrated into a quantized model without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model families and validate its effectiveness in different fine-tuning datasets and downstream scenarios. Code will be made available at https://github.com/yuhuixu1993/qa-lora.
1 INTRODUCTION
QA-LoRA addresses the difficulty of jointly fine-tuning and quantizing LLMs for efficient deployment. It uses group-wise operators to improve low-bit quantization and simplify integration of adaptation weights, while retaining performance across model scales and quantization settings.
- LLM deployment on edge devices is constrained by the computational burden of their many parameters, motivating quantized and fine-tuned models.
- Post-training quantization after parameter-efficient fine-tuning can lose accuracy at low bit widths, while existing alternatives may be costly or fail to preserve quantization after fine-tuning.
- QA-LoRA balances quantization and adaptation degrees of freedom through group-wise operators: each group is quantized individually and shares adaptation parameters.
- QA-LoRA quantizes LLM weights during fine-tuning and integrates auxiliary weights into a lightweight quantized model without requiring post-training quantization.
- Across LLaMA models fine-tuned on Alpaca, QA-LoRA outperforms QLoRA with post-training quantization and matches QLoRA without it, with larger advantages at lower bit widths.
2 RELATED WORK
Related work combines parameter-efficient adaptation and LLM quantization, but existing approaches leave a gap between efficient fine-tuning and efficient deployment. QA-LoRA is positioned to address this joint objective.
- Fine-tuning LLMs with adapters: LoRA adapts pretrained models by fine-tuning two low-rank matrices while keeping most pretrained parameters unchanged.
- Quantization of LLMs: Quantization reduces parameter or activation bit width to improve LLM efficiency and scalability, while inference methods address memory costs and quantization-induced accuracy issues.
- Joint adaptation and quantization: Joint adaptation and quantization seeks parameter-efficient adaptation together with computation-efficient tuning and deployment, but requires handling discrete values and quantization parameters.
- Joint adaptation and quantization: QLoRA quantizes pretrained weights to NF4 during fine-tuning but restores merged weights to FP16 afterward, leaving the deployed model slower.
3 THE PROPOSED APPROACH
QA-LoRA jointly quantizes pretrained weights during fine-tuning and preserves a quantized merged model for efficient deployment. It balances quantization and adaptation degrees of freedom through group-wise operations, retaining useful low-rank capacity while enabling quantized integration.
- Baseline: Low-Rank Adaptation and Low-Bit Quantization: LoRA supplements fixed pretrained weights W with adjustable low-rank matrices A and B, computing y = W^⊤x + s · (AB)^⊤x.A and B have dimensions D_in × D_int and D_int × D_out, with D_int much smaller than both input and output dimensions.
- Baseline: Low-Rank Adaptation and Low-Bit Quantization: Min-max weight quantization reduces storage and accelerates computation, but approximating W with quantized W̃ can harm language-understanding accuracy.The paper describes quantization such as FP16 to INT4 and uses scaling and zero factors with integer rounding.
- Objective: Efficient Adaptation and Deployment: QA-LoRA targets quantized weights during fine-tuning and keeps the fine-tuned merged weights W′ quantized after adaptation for computationally efficient deployment.These are the paper’s two stated objectives for efficient adaptation and deployment.
- Objective: Efficient Adaptation and Deployment: QLoRA quantizes weights during fine-tuning but restores FP16 after merging LoRA weights, while post-training quantization can significantly reduce accuracy at low bit widths.The paper also notes that QLoRA’s NF4 representation lacks operator-level optimization for accelerating fine-tuning and inference.
- Solution: Group-Wise Quantization with Low-Rank Adaptation: Column-wise quantization with unconstrained LoRA weights cannot preserve quantization after merging; forcing identical rows in A makes rank(AB) = 1 and causes a significant accuracy drop.The resulting rank limitation reduces the adaptation capacity needed for new data.
- Solution: Group-Wise Quantization with Low-Rank Adaptation: QA-LoRA partitions each weight column into L groups, using group-specific quantization factors and requiring equal A rows only within each group.A parameter-free summation reduces x from D_in dimensions to L, allowing A to have dimensions L × D_int without further constraints.
- Solution: Group-Wise Quantization with Low-Rank Adaptation: QA-LoRA increases quantization parameters from D_out to L×D_out while reducing adaptation parameters from D_in×D_int + D_int×D_out to L×D_int + D_int×D_out.The paper presents this balance as the basis for preserving quantized integration and adaptation capacity.
4 EXPERIMENTS
Experiments evaluate QA-LoRA across LLaMA and LLaMA2 models, quantization settings, datasets, and downstream tasks. QA-LoRA generally improves accuracy and efficiency, with especially strong gains at lower bit widths and over post-quantized QLoRA.
- 4.1 SETTINGS: Experiments use LLaMA models from 7B to 65B and LLaMA2 models at 7B and 13B, with MMLU and commonsense QA evaluation.The main quantization setting is group-wise asymmetric quantization with group size 32 using GPTQ.
- 4.2 MAIN RESULTS AND EFFICIENCY: QA-LoRA consistently outperforms QLoRA with GPTQ and PEQA on LLaMA MMLU in both 0-shot and 5-shot accuracy.The advantage is more significant for smaller models and lower bit widths, including INT3 and INT2.
- 4.2 MAIN RESULTS AND EFFICIENCY: More than 50% faster inference than QLoRA follows because QA-LoRA remains in INT4 after weight integration instead of converting back to FP16.INT4 operators also contribute to faster fine-tuning than QLoRA’s NF4 operators.
- 4.2 MAIN RESULTS AND EFFICIENCY: 15.0% accuracy gain is reported for 2-bit QA-LoRA over 2-bit post-quantized QLoRA on 0-shot commonsense QA.The 4-bit version is comparable with mixed-precision QLoRA and averages a 2.0% advantage over post-quantized QLoRA.
- 4.2 MAIN RESULTS AND EFFICIENCY: QA-LoRA generalizes to LLaMA2, where INT4 models fine-tuned on FLAN v2 consistently outperform the original FP16 models.LLaMA2 models fine-tuned on Alpaca report slightly lower accuracy than the original FP16 models.
- 4.3 ABLATIVE STUDIES: Larger L, corresponding to smaller group size, often yields higher accuracy, especially when the quantization bit width is small.Larger L increases quantization and adaptation freedom while adding storage and computation that remain negligible when L ≫1.
- 4.3 ABLATIVE STUDIES: On smaller fine-tuning datasets, QA-LoRA consistently outperforms mixed-precision QLoRA on LLaMA-13B and is much faster during inference.Self-instruct, Longform, and Chip2 produce somewhat weaker MMLU accuracy than Alpaca and FLAN v2.
- 4.3 ABLATIVE STUDIES: 320K examples are sufficient for both INT2 and INT4 QA-LoRA variants, although lower-bit quantization requires more data.The dataset-size study uses LLaMA-7B and subsets of FLAN v2 ranging from 160K to 480K examples.
5 CONCLUSION
QA-LoRA introduces quantization-awareness into low-rank adaptation through group-wise operations that balance quantization and adaptation degrees of freedom. Experiments on LLaMA families validate its effectiveness across language-understanding tasks.
- QA-LoRA introduces quantization-awareness into low-rank adaptation using group-wise operations.
- Balancing the degrees of freedom of quantization and adaptation is QA-LoRA’s key insight.
- Extensive experiments on the LLaMA model families validate QA-LoRA’s effectiveness.
A QUALITATIVE STUDIES
The qualitative studies compare QA-LoRA with QLoRA across quantization bit widths and conversational, categorization, HTML-generation, and dialogue prompts. QA-LoRA generally produces coherent outputs, while QLoRA with GPTQ can behave unstably, including a reported 4-bit and 2-bit crash in one case.
- QLoRA with GPTQ shows unstable behavior across qualitative cases and crashes in one case at 4-bit and 2-bit quantization.The passage attributes this mainly to uncontrollable quantization loss during post-processing.
- For expense-budgeting prompts, QA-LoRA produces five complete suggestions at 4-bit and 2-bit quantization.The corresponding QLoRA 2-bit output is corrupted, while its 4-bit output is shorter and less complete.
- For social-media responses, QA-LoRA gives a detailed de-escalatory answer, whereas QLoRA’s 4-bit answer is limited to an apology.
- For fruit categorization, QA-LoRA consistently assigns explicit colors to the listed fruits across shown bit widths.QLoRA outputs include incorrect or malformed categorizations in the displayed examples.
- For HTML generation and beach-dialogue prompts, QA-LoRA produces structured outputs at low bit widths, while QLoRA includes malformed or corrupted text in some cases.
B SIMPLE PROOF
This section presents a simple proof explaining why QA-LoRA can be merged with quantized base models. It also notes that some notation conflicts with the main script.
- The proof explains why QA-LoRA can be merged with quantized base models.
- The section uses a notation set that conflicts with some letters in the main script.
- The proof is presented as a simple explanation of the proposed approach’s mergeability.
B.1 QUANTIZATION
The quantization procedure applies group-wise quantization to columns of the pretrained weight matrix. Scaling and zero factors are defined per group, while the scaling matrix and zero matrix support de-quantization.
- Pretrained LLM weights are quantized using post-training quantization methods such as GPTQ.
- Each weight-matrix column is divided into groups with group size g for quantization.
- The scaling matrix A and zero matrix B provide the factors used in the de-quantization process.
B.2 QA-LORA TRAINING
QA-LoRA aggregates input features before they enter the LoRA adapters, using windows matched to the quantization groups. The aggregated features then pass through low-rank adapters and augment the quantized-weight output.
- QA-LoRA inserts an aggregation operation before the input reaches the LoRA adapters.
- The aggregation window size g equals the quantization group size.
- Features within each non-overlapping aggregation window are added, reducing the input feature dimension from h to h/g.
- The aggregated input passes through low-rank adapters and augments the output produced by the quantized weight.
B.3 MERGE FOR INFERENCE
QA-LoRA supports merging learned LoRA adapters into quantized weights while preserving the quantization characteristic. The merged weights can therefore be deployed directly for low-bit inference.
- QA-LoRA merges learned LoRA adapters with quantized weights while keeping their quantization characteristic.
- The resulting merged weights support direct low-bit inference deployment.
- The merge procedure derives the left term of the adapter expression from the quantized-weight representation.
- The right term is expressed using P = [p_i,j] as a substitute for L1L2, with P ∈ R^(d/l)×o.
- The adapter weights are perfectly merged by updating the zero-point matrix B to B − sL1L2 ⊘ A.