LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

1 INTRODUCTION

• LoRA reparameterizes pretrained weights with a trainable low-rank update, reducing finetuning memory by leaving the pretrained parameters fixed.This avoids storing gradients and optimizer states for those parameters.Source: §1 · p. 1
• An iterative decomposition represents each pretrained matrix as a quantized component Q plus low-rank factors L1L2, keeping Q fixed while finetuning the factors.The low-rank component captures high-variance subspaces and accounts for quantization error in its initialization.Source: §1 · p. 2
• An integer linear program assigns different quantization configurations to each matrix under an overall target bit rate.Configurations can vary in bit width and block size.Source: §1 · p. 2
• A data-aware variant weights the decomposition objective using an approximation of the Fisher information matrix from calibration samples.Source: §1 · p. 2
• LQ-LoRA meaningfully improves on QLoRA (Dettmers et al., 2023a) and GPTQ-LoRA (Frantar et al., 2022; Chai et al., 2023); its LLaMA-2-70B compression setting reaches 2.85 bits with small perplexity degradation.The comparison covers adaptation of RoBERTa and LLaMA-2 models.Source: §1 · p. 2

2.1 LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS

• LoRA reparameterizes a pretrained linear layer as X(W + L1L2), with L2 initialized to zero, and finetunes only the low-rank factors.L1 is initialized from Gaussian noise.Source: §2.1 · p. 2

2.2 WEIGHT QUANTIZATION OF LARGE LANGUAGE MODELS

• Sub-4-bit round-to-nearest quantization is difficult, and recent methods generally use calibration samples for data-aware quantization.Source: §2.2 · p. 2
• QLoRA applies NF-4 quantization to pretrained weights and learns low-rank updates, serving as the paper’s main baseline.Its double-quantization strategy quantizes the per-block absmax values.Source: §2.2 · p. 3

3.1 LOW-RANK PLUS QUANTIZED MATRIX DECOMPOSITION

• The iterative factorization alternates between optimizing the low-rank component and quantized component, but uses a heuristic stopping criterion rather than a convergence guarantee.It tracks reconstruction error and stops when that error increases.Source: §3.1 · p. 3
• LQ decomposition reduces quantization error relative to NF-3 quantization across all layers, although projection-matrix errors vary by layer.The figure also tracks decomposition error over LQ steps for query projections.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 MIXED-CONFIGURATION QUANTIZATION VIA AN INTEGER LINEAR PROGRAM

• NF-4 double quantization requires 4.127 bits per parameter on average, while the configuration search varies quantization and block parameters.The configuration tuple is (b0, b1, b2, B0, B1).Source: §3.2 · p. 4
• An integer linear program assigns each matrix a quantization configuration to minimize reconstruction error within a target memory budget.The method precomputes errors and storage costs across configurations before solving for assignments.Source: §3.2 · pp. 4–5

3.3 DATA-AWARE MATRIX DECOMPOSITION VIA FISHER-WEIGHTED SVD

• The data-aware variant weights reconstruction with a diagonal empirical Fisher approximation and uses row- and column-averaged weights in its SVD procedure.The authors report that this approach works in practice despite the homogeneous row/column assumption not holding for the Fisher matrix.Source: §3.3 · p. 6
• An alternative calibration-activation-weighted reconstruction objective underperformed the Fisher-weighted approach.Source: §3.3 · p. 6

4 EMPIRICAL STUDY

• Experiments cover continual language modeling and instruction tuning with LLaMA-2, plus GLUE finetuning with RoBERTa-Large, using QLoRA and GPTQ-LoRA as main baselines.GPTQ-LoRA was excluded from instruction-tuning experiments because training was unstable.Source: §4 · p. 7

4.1 RESULTS

• LQ-LoRA almost always outperforms QLoRA and GPTQ-LoRA at near-similar bit budgets, and it outperforms QLoRA at similar bit-widths on GLUE.At 7B, Fisher-weighted LQ-LoRA outperforms the unweighted version across target bit widths; the difference shrinks at 70B.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 LQ-LORA FOR MODEL COMPRESSION

• 2.95 bits/param for LLaMA-2 7B and 2.85 bits/param for 70B are the effective costs of 2.75-bit LQ-LoRA, which generally outperforms other sub-4-bit PTQ methods.Effective bits include LoRA components quantized to 8 bits.Source: §4.2 · p. 8, Table 3 · p. 8
• Some zero/few-shot benchmarks, including GSM8K and ARC, show nontrivial degradation, so perplexity changes do not always track downstream performance.The evaluation uses six Open LLM benchmarks.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 ANALYSIS

• At a 2.75-bit target, the ILP assigns different quantization configurations across matrices, and those allocations differ between Fisher-weighted and unweighted LQ-LoRA.Figure 4 shows bits per parameter by layer and matrix type for QLoRA and both LQ-LoRA variants.Source: §4.3 · p. 9, Figure 4 · p. 9
• With NF-3 fixed at 3.127 bits/param, QLoRA is insensitive to LoRA rank, while LQ-LoRA uses additional ranks to reduce initialization error and improve performance.The comparison uses LLaMA-2-7B and reports C4 and WikiText perplexity as a function of rank.Source: §4.3 · p. 9, Table 5 · p. 10
• At sub-3 bits, a 70B model can run on one 40GB GPU, while full forward and backward passes fit on one 80GB GPU at batch size 2 and sequence length 2048.The memory breakdown distinguishes quantized parameters, LoRA parameters, and other components.Source: §4.3 · pp. 9–10, Figure 3 · p. 9

5 DISCUSSION AND LIMITATIONS

• The iterative decomposition algorithm is empirically effective but remains heuristic, motivating more theoretically principled alternatives.The authors also note that applying LQ decomposition to quantization approaches beyond NF may yield further gains.Source: §5 · p. 10
• Periodic matrix refactorization and a hybrid initialization combining LQ-LoRA with standard LoRA did not improve results.The hybrid used half of the adaptable low-rank component from each initialization.Source: §5 · p. 10
• LQ-LoRA relies on low-rank updates and is therefore not generally applicable to other parameter-efficient finetuning methods.The authors suggest extending the ILP approach to assign different quantization precisions and ranks to different matrices.Source: §5 · p. 10

6 RELATED WORK

• Parameter-efficient finetuning methods reduce memory by keeping pretrained parameters fixed and training a smaller extension, with approaches including adapters, prompts, and partial parameter updates.Prior work also combines parameter-efficient finetuning with quantization.Source: §6 · p. 10
• Low-rank plus sparse decomposition is well studied, while prior deep-learning applications include smaller models and recent work on quantized low-rank approximations and LLM adaptation.The cited smaller-model compression work covers models with fewer than 100M parameters.Source: §6 · p. 10
• Low-rank compression for LLMs with 1B+ parameters remains underexplored, while existing LLM compression approaches have generally focused on quantization.The passage notes that slowly decaying singular values may help explain the limited exploration of low-rank approaches.Source: §6 · p. 10
• Data-aware quantization strategies use calibration data to improve weight quantization, while the ILP objective here minimizes decomposition error rather than downstream performance.The authors note that weighting LoRA parameters as less costly could address this objective mismatch.Source: §6 · pp. 10–11

7 CONCLUSION

• LQ-LoRA factorizes pretrained matrices into low-rank and quantized components, allows dynamic quantization configurations, and yields meaningful improvements over strong baselines.Source: §7 · p. 11

A IMPLEMENTATION DETAILS

• The PyTorch-only implementation supports experimentation with dynamic quantization strategies without configuration-specific custom CUDA extensions.Sub-8-bit weights are packed into supported data types and unpacked during dequantization.Source: A IMPLEMENTATION DETAILS · p. 16
• The compiled PyTorch implementation was as fast as QLoRA’s bitsandbytes implementation for LoRA finetuning.It compiles bit unpacking, dequantization, linear algebra, transpose, and casting operations.Source: A IMPLEMENTATION DETAILS · p. 16
• 14% memory savings on the 70-billion-parameter model came with a marginal (<2%) increase in training speed from CPU-based optimizer offloading.The method keeps a CPU copy of trainable parameters, moves gradients to CPU for optimizer steps, and overlaps per-matrix updates with asynchronous transfers.Source: A IMPLEMENTATION DETAILS · p. 16
• Table 6 reports rank-64 LLaMA-2 results, with C4, Wikipedia, and MMLU based on C4 finetuning and Vicuna evaluation based on OpenAssistant finetuning.The QLoRA and GPTQ-LoRA results use the authors’ reimplementations.Source: Table 6 · p. 17
• Just-in-time dequantization makes runtime no lower than dense FP32 matrix multiplication, with relatively small overhead for reasonably sized matrices.The runtimes were collected with PyTorch Profiler using TF32.Source: A IMPLEMENTATION DETAILS · p. 17

• Table 6 presents the full numeric results comparing LQ-LoRA with QLoRA and GPTQ-LoRA.Source: B FULL RESULTS · p. 17