ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

1 Introduction

• LLM serving is costly in memory and computation, with prompt processing mainly compute-bound and generation mainly memory-bound at low batch sizes with a KV cache.The paper frames PTQ as a way to reduce training prerequisites and lower weight and activation precision to INT4 or INT8.Source: §1 · p. 1
• The paper identifies a lack of comprehensive evidence on PTQ methods and model sensitivity, and asks whether methods fully exploit opportunities to reduce LLM size.It notes that existing literature does not thoroughly cover different PTQ methods or model sensitivities.Source: §1 · pp. 1–2
• The study examines weight-only, activation-only, and combined quantization using RTN, GPTQ [12], ZeroQuant [36], and variants across OPT and BLOOM models from 125M to 176B.The stated questions concern whether quantization behavior varies with model size and pretraining data, and whether existing methods fully reduce model size.Source: §1 · p. 2
• Activation quantization is generally more sensitive to weight quantization, and smaller models usually tolerate activation quantization better than larger models.Activation sensitivity also differs by family: BLOOM-176B has about 1 PPL point of drop, while OPT-30B and OPT-66B perform worse.Source: §1 · p. 2
• Existing methods barely achieve less than 0.1 PPL points degradation with INT4 weights or W4A8; FGQ recovers <0.1 PPL degradation for large models on INT4 weights, but quality drops remain.The paper introduces LoRC as a complementary technique for quality recovery with little increase in model size.Source: §1 · pp. 2–3
• LoRC combined with PTQ methods [36] [12] and fine-grained quantization sets a new quantization Pareto frontier across OPT and BLOOM models.The paper recommends different LoRC quantization settings by model size, including INT4 weights with INT8 activations for models above 10B.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

• Quantization-aware finetuning and distillation can be challenging for practitioners because training and finetuning LLMs such as GPT-3-175B and BLOOM-176B is costly.The cited methods include [29] [38] [9] [41] [1] [8] [31] [19] and quantization-aware knowledge distillation [17].Source: §2 · p. 3
• PTQ methods target quantization with no or minimal finetuning [37] [3], and prior work explored INT8 methods [36] [6], INT4 weight-only quantization [12], and activation smoothing [35].The paper notes that [35] works only for W8A8, while [7] studies weight-only RTN rather than weight-and-activation quantization or PTQ optimizations.Source: §2 · p. 3
• This study compares quantization schemes, PTQ methods, model families, quantization coverage, and granularity for a broader understanding of LLM PTQ.The comparison includes symmetric and asymmetric schemes, methods [36] [12], families [28] [40], and weight-only and weight-and-activation quantization.Source: §2 · p. 3

3 Would different model families behave similarly on quantization?

• Sensitivity is measured by zero-shot validation PPL differences before and after quantization on Wikitext-2, PTB, and C4, with higher PPL drops indicating greater sensitivity.The paper groups sensitivity into three classes, with sensitivity increasing from Class-1 to Class-3.Source: §3 · p. 4, Table 1 · p. 4
• The evaluation uses symmetric and asymmetric quantization, with per-row weight quantization and per-token activation quantization.The study is motivated by the limited prior evaluation of sensitivity across model sizes and families.Source: §3 · pp. 3–4
• INT8 weight-only quantization causes less than 0.05 accuracy loss, while INT4 weight-only quantization is better tolerated by larger models, with exceptions including OPT-66B.Asymmetric INT4 outperforms symmetric INT4 in accuracy; BLOOM-176B has about 0.3 PPL degradation.Source: §3 · p. 4
• With INT8 activation, BLOOM has no divergence issues up to 176B, whereas OPT performs poorly from 6.7B and larger models fare worse.The authors suggest OPT-family Layer Norm may contribute to the difference.Source: §3 · pp. 4–5
• INT8 activation quantization has minimal accuracy drops for small models but larger drops for large models; models above 10B have degradation exceeding 0.5 PPL points.The paper attributes smaller models’ greater tolerance to their narrower hidden dimensions and activation dynamic ranges.Source: §3 · pp. 4–5

4 Are existing quantization methods optimally harnessing the potential to minimize LLMs sizes?

• ZQ-Global∗ generally performs better on models under 1B parameters, while GPTQ excels on larger models; ZQ-Local∗ does not outperform either method.For weight-and-activation quantization, ZQ-Global∗ and ZQ-Local∗ generally outperform GPTQ, but ZQ-Global∗ trails ZQ-Local∗ on OPT-66B and BLOOM-176B.Source: §4 · p. 6
• GPTQ typically performs better for weight-only quantization, while ZeroQuant variants generally perform better for weight-and-activation quantization; tested methods rarely reach Class-1 error for INT4 weight-only or W4A8.The exception is GPTQ on OPT-30B with weight-only quantization; for W4A8, tested results fall into Class-2 or Class-3.Source: §4 · p. 6

4.1 Fine-grained Quantization and Its Evaluation

• Fine-grained quantization assigns scaling factors and/or zero points to groups of k elements, reducing quantization error; RTN still leaves an accuracy gap, so optimization-based methods are also tested.The study uses GPTQ and ZQ-Global across models and settings, and ZQ-Local for OPT-66B and BLOOM-176B.Source: §4.1 · p. 7
• Smaller blocks improve W4A16 accuracy, and models at least 13B can reach Class-1 quantization error with a smaller block size.Gains continue to block size 32 for OPT-6.7B and BLOOM-1.7B, but wane around block sizes 256/512 for most models above 10B.Source: §4.1 · p. 7, Table 4 · p. 7
• For models at least 1B, fine-grained activation quantization reduces W4A8 error versus per-row activation, and combining it with fine-grained weight quantization nearly restores W4A16 performance.The comparison uses per-row activation and block size 128, with INT4 weight quantization.Source: Table 5 · p. 7, §4.1 · p. 8
• For BLOOM-176B with INT4 weight quantized at block size 128, smaller activation blocks improve accuracy until block size 256, after which gains saturate.Uniform quantization errors persist despite INT8 representing 256 distinct values.Source: §4.1 · p. 8, Table 6 · p. 8

5 Proposed Method to Further Push the Limit of Post-training Quantization

• LoRC approximates the quantization error E = W − Ŵ with low-rank matrices and adds the approximation to Ŵ to reduce quantization error.W is the original weight and Ŵ is the quantized weight.Source: §5 · p. 8
• LoRC applies SVD to the error matrix and uses its leading components to construct the low-rank approximation.The singular values are ordered in descending order before selecting components.Source: §5 · p. 8
• For a standard transformer with hidden dimension h, layers l, and LoRC rank m, the added parameters are 18hml, or 3m/2h of the original model.The paper notes that m can be 4 or 8 while h is at least 768, making the added-parameter ratio at most 0.016.Source: §5 · pp. 8–9
• LoRC can supplement RTN, GPTQ, and ZeroQuant variants and integrate with fine-grained quantization; experiments apply GPTQ to OPT and BLOOM at 4-, 3-, and 2-bit weights.For the reported Table 7 results, LoRC uses INT8 low-rank matrices with rank m = 8; the matrices can be INT8 with hardly any performance difference from FP16.Source: §5 · p. 9, Table 8 · p. 9
• LoRC improves performance across bit and block sizes; its gains are larger at lower bit widths, and fine-grained quantization combined with LoRC gives the strongest results.With LoRC, the paper reports nearly recovering original model quality for INT4 quantization.Source: §5 · p. 9, Table 7 · p. 9
• LoRC gains begin to plateau above rank m = 4 in the tested OPT models, while OPT-6.7B performs best at m = 8.Eigenvalues of sampled error matrices flatten rapidly after index 8, which the paper offers to explain the limited gains from larger ranks.Source: §5 · pp. 9–10, Figure 2 · p. 10

6 Discussion

• LoRC works synergistically with PTQ and fine-grained quantization to enhance model quality recovery with minimal model-size increase.The conclusion also identifies PTQ methods as important to quantized-model quality and fine-grained quantization as a way to obtain an acceptable accuracy–size trade-off.Source: §6 · p. 10
• The study covers two model families and WikiText, PTB, and C4, so caution is warranted when generalizing to dissimilar tasks.The authors say computing resources constrained the choice between diversifying model sizes and varying tasks.Source: §6 · p. 10
• Fine-grained quantization lacks a real implementation in the study, and efficient odd-bit implementation remains challenging.The authors describe odd-bit packing and dequantization as an open implementation issue.Source: §6 · p. 10
• The reported observation is specific to PTQ; updating Û and V̂ during QAT leads to contrasting conclusions.The paper directs readers to Appendix D for details.Source: §6 · p. 10

A Background of Quantization

• Quantization maps floating-point values such as FP16 or FP32 to integers such as INT4 or INT8, reducing weight memory use or enabling faster integer arithmetic.The paper focuses on uniform quantization.Source: A Background of Quantization · p. 14
• Symmetric quantization uses zero point Z = 0, while asymmetric quantization uses a nonzero zero point.The quantization function Q takes floating-point input x and uses scale S and zero point Z.Source: A Background of Quantization · p. 14
• Coarse-grained per-matrix or per-tensor PTQ can cause large accuracy degradation, so the study focuses on finer-grained schemes.Per-row quantization uses one weight-matrix row or one activation token; block-k quantization assigns scaling factors and/or zero points per k elements.Source: A Background of Quantization · p. 14

B Detailed Setting Used in Section 4

• Experiments used 128 randomly selected C4 sentences of 2,048 tokens each for training.The setup follows the referenced GPTQ data selection approach.Source: B Detailed Setting Used in Section 4 · p. 14
• For ZeroQuant variants, the authors used a linear-decay learning-rate schedule and searched learning rates from 1e-3 to 5e-8; GPTQ used the author-suggested dampening factor.They used Adam and default batch size 1.Source: B Detailed Setting Used in Section 4 · p. 14
• Three-seed tests showed minimal effects on final model quality; OPT-30B had standard deviation 0.01, and subsequent experiments used seed 123 or 0.The methods ran on a V100-32GB or A100-80GB GPU; each method on BLOOM-176B cost less than one A100-80G GPU-day.Source: B Detailed Setting Used in Section 4 · p. 14, Table B.1 · p. 15

• Tables C.1 and C.2 summarize the best PTQ methods with per-row optimization.Source: C Best PTQ Methods with Per-row… · p. 15

D Quantization-aware training with LoRC

• QAT with knowledge distillation was applied to LoRC using row-wise weight and token-wise quantization.Fine-tuning used Adam with a 1e-4 learning rate and 0.05 dropout on WikiText, starting from pretrained GPT2-125M and GPT2-350M models.Source: D Quantization-aware training with LoRC · p. 15
• Larger LoRC dimensions yield better best perplexity for W4A8 models under QAT, unlike the trend observed in PTQ.The quantized models overfit swiftly, while increasing dropout to 0.1 does not significantly improve best perplexity over training.Source: D Quantization-aware training with LoRC · p. 15
• Tables E.1 and E.2 provide full Table 2 perplexity results for OPT and BLOOM on WikiText, PTB, and C4.Source: Table E.1 · p. 16, Table E.2 · p. 16
• Figure D.1 compares LoRC dimensions for GPT2-125M and GPT2-350M, with separate plots and dimension-coded curves.The figure describes dimension as pivotal to approximating the original fp16 model quality.Source: Figure D.1 · p. 16

E Tables and Figures

• Tables E.3–E.18 give OPT and BLOOM perplexity results for W4asym-A16 and W4asym-A8sym/A8asym, including ZQ-Local and ZQ-Global learning-rate results.The tables cover WikiText, OPT, and C4 evaluation datasets.Source: Table E.3 · p. 17, Table E.4 · p. 17, Table E.5 · p. 17
• The cited appendix tables report perplexity (ppl) on wikitext/opt/c4 for OPT and BLOOM using W4asym-A8sym/A8asym, with ZQ-Local and ZQ-Global results in Tables E.10–E.14.Source: Table E.9 · p. 18, Table E.10 · p. 18, Table E.11 · p. 18
• Tables E.16–E.18 report block-size results for BLOOM W4asym-A16, OPT three-bit weights, and BLOOM W3asym-A16; E.19 lists BLOOM-176B results with different quantization bits, and E.20 gives OPT full results of Table 5.The BLOOM block-size results are out of the best result from GPTQ and ZQ-Global.Source: Table E.16 · p. 20, Table E.17 · p. 21, Table E.18 · p. 22
• Table E.21 gives BLOOM full results of Table 6, while Table E.23 reports results of applying LoRC on top of ZQ-Global for INT8 activation.Source: Table E.21 · p. 23, Table E.23 · p. 24