Evaluating Quantized Large Language Models

• LLM deployment faces substantial memory consumption and computational overhead, motivating efficient quantization methods.Source: §1 · p. 1
• PTQ reduces Weight, Activation, and KV Cache memory by replacing high-precision values with lower-precision representations.Weight-only methods target memory-bound decoding, Weight-Activation methods target compute-bound prefill, and KV Cache methods reduce overhead for long texts or large batches.Source: §1 · p. 1
• Quantization design choices must account for model size, batch size, text length, hardware, and task performance because PTQ is lossy.Source: §1 · p. 1
• The evaluation extends prior work by jointly examining diverse models, task categories, tensor types, and SOTA quantization methods such as AWQ.Earlier studies omitted KV Cache evaluation or important task categories, leaving cross-task applicability insufficiently characterized.Source: §1 · pp. 1–2
• The paper evaluates quantization across 11 model families spanning 125M–180B parameters, five task types, and Weight-only, Weight-Activation, and KV Cache settings.The task categories are basic NLP, emergent abilities, trustworthiness, dialogue, and long-context processing.Source: §1 · p. 2
• The authors summarize qualitative trends and failure cases, but bit-width recommendations may not generalize to other LLMs or tasks.Source: §1 · p. 2

2.1. Quantization

• Uniform quantization maps FP16 values to low-precision integers using a scale and zero-point, with symmetric quantization setting the zero-point to zero.Source: §2.1 · pp. 2–3
• Weight-only, Weight-Activation, and KV Cache Quantization target W, X and W, or K and V tensors, respectively; W4A8 denotes 4-bit weights and 8-bit activations.Source: §2.1 · p. 3
• Weights and KV Cache use asymmetric group-wise quantization, while activations use symmetric per-token quantization with one scaling factor per token.Source: §2.1 · pp. 3–4, Figure 1 · p. 3

2.2. Benchmarks and Models

• The evaluation covers five task types and model families selected by capability, including long-context models supporting 16k or 32k context lengths.Basic NLP uses OPT, LLaMA2, Falcon, Bloomz, ChatGLM, and Mistral; other evaluations additionally include StableLM, Gemma, Mamba, LongChat, and Vicuna.Source: Table 2 · p. 3, §2.2 · p. 4

2.3. Statistical Analysis

• Tensor analysis uses AbsMax for dynamic range, standard deviation for deviation from the mean, and kurtosis for outlier conditions.Higher kurtosis indicates heavier tails and a greater likelihood of outliers, whereas lower kurtosis indicates fewer outliers.Source: §2.3 · p. 4

3.2. Effects of Quantization on Three Tensor Types

• Larger models tolerate Weight and KV Cache quantization better but tolerate Activation quantization worse as activation kurtosis and outliers increase with scale.Weight kurtosis decreases with model size, whereas Activation kurtosis exceeds 1000 and rises substantially in larger models.Source: §3.2 · p. 4
• W4, W4A8, and KV4 usually keep performance loss below 2%, while W2, KV2, and W4A4 often cause major or complete degradation.For short texts, W8KV4 is nearly lossless; for texts of at least 4K, W8KV8 is the better-supported choice.Source: §3.2 · p. 4, §3.4 · p. 5
• Weight, Activation, and KV Cache tensors have distinct layer-wise statistics, motivating tensor-specific bit-widths and scaling schemes to balance efficiency and performance.Down-projection activations and output-projection weights exhibit especially distinctive kurtosis across model families.Source: §3.2 · pp. 4–5

3.3. Effects of Quantization on Different LLMs

• Higher-than-W4, W4A8, and KV4 quantization generally preserves the relative rankings of quantized LLMs, but Mixtral-8x7B is more sensitive than similarly performing LLaMA2-70B.Its quantization sensitivity is closer to smaller LLaMA2-7B and Mistral-7B models despite comparable FP16 performance to LLaMA2-70B.Source: §3.3 · p. 5

3.4. Effects of Quantization on Different Tasks

• Quantization causes similar performance-loss trends across Chinese and English cloze tasks, with consistent findings also reported on CEval and MMLU.Source: §3.4 · p. 5
• Under a fixed memory budget, W3 can favor a larger model, but performance saturation can instead make a smaller model with higher bit-width preferable.The paper reports W3 LLaMA-70B outperforming FP16 LLaMA2-13B on RACE at 27GB, while Bloomz-3B and Bloomz-7B1 illustrate saturation.Source: §3.4 · pp. 5–6, Figure 3 · p. 5
• Extremely low-bit settings such as W2 and W4A4 remain difficult for current methods, requiring improved quantization schemes or quantization-aware training.Window-based quantization is identified as promising for KV2.Source: §3.4 · p. 6

4.2. Experimental Results

• Multi-Step Reasoning and Self-Calibration are less tolerant of quantization than Instruction-Following and In-Context Learning, especially in small LLMs.W3 or KV3 LLaMA2-7B nearly loses its Self-Calibration ability.Source: §4.2 · p. 6
• Mathematical Multi-Step Reasoning is more sensitive than Commonsense Multi-Step Reasoning, while Gemma reverses the usual ordering between In-Context Learning and mathematical reasoning.Source: §4.2 · p. 6
• Incorrect logic accounts for around 50% of quantization errors in LLaMA2-70B mathematical reasoning, versus around 20% for calculation errors.The study categorizes errors as incorrect logic, calculation error, copy mistake, and condition missing.Source: §4.2 · pp. 6–7
• For models below 13B, W8, W8A8, or KV8 is recommended to keep Multi-Step Reasoning and Self-Calibration loss within 2%; for models at least 13B, W4, W4A8, or KV4 suffices for all four abilities.Source: §4.2 · p. 7

5.1. Experimental Setups

• Trustworthiness evaluation covers Ethics, Hallucination, and Adversarial Robustness, with hallucination and adversarial results resembling basic NLP behavior.Source: §5.1 · p. 7

5.2. Experimental Results

• For models below 7B, weight-only W3 can increase ethics performance by eliciting answers to questions that FP16 models sometimes avoid.Source: §5.2 · p. 7
• KV Cache quantization makes small models more restrictive, whereas models at least 7B show lower performance as bit-width decreases.Source: §5.2 · p. 7
• Attention analysis links W3 to greater attention on original questions and KV3 to reduced attention, producing respectively more informative and less informative answers.Source: §5.2 · p. 7
• Within 2% accuracy loss, W8, W8A8, or KV8 suit smaller LLMs, while W4, W4A8, or KV4 suit larger LLMs.Source: §5.2 · p. 7

6.1. Experimental Setups

• The evaluation uses two-turn MT-bench dialogues scored from 1 to 10 by GPT-4-0613.Source: §6.1 · p. 7

6.2. Experimental Results

• Most LLM families retain dialogue performance within 2% of FP16 under W8, W8A8, or KV4, while W4 reduces GPT-4 scores by over 0.3 on LLaMA2-13B and LLaMA2-70B.Source: §6.2 · p. 8
• KV Cache quantization is generally more tolerated than weight quantization, but W3 and KV3 induce repetition, while W2, KV2, and W4A4 can produce meaningless symbols or repeated words.Source: §6.2 · p. 8
• StableLM-3B matches Falcon-40B in FP16 performance but has quantization sensitivity closer to smaller Falcon-7B.Source: §6.2 · p. 8
• KV8-to-KV3 quantization lowers LLaMA2-13B’s second-turn GPT-4 score by 1.19, compared with 0.31 for the first turn.Source: quantization · p. 8, Table 6 · p. 9
• SmoothQuant W4A4 can recover meaningful dialogue responses for some LLaMA2 models, especially in first-turn conversations, although FP16-level performance remains difficult at extreme bit-widths.Source: quantization · p. 8
• The study leaves the effect of quantization on dialogues longer than two turns for future investigation.Source: quantization · p. 9

7.1. Experimental Setups

• The evaluation covers key-value retrieval up to 16K tokens and multi-document question answering up to 6K tokens.Source: §7.1 · p. 9

7.2. Experimental Results

• Texts of at least 4K tokens are more sensitive to Weight-only and KV Cache quantization than shorter texts, while Weight-Activation quantization does not show greater degradation on long texts.Source: §7.2 · p. 9
• At equal bit-width, KV Cache quantization performs worse than Weight-only quantization on long-context tasks.Source: §7.2 · p. 9
• KV8 is nearly lossless for Vicuna and ChatGLM long texts but degrades LongChat, whereas Mistral retains lossless performance at KV4.Source: §7.2 · p. 9
• Within 2% accuracy loss, most models support W4, W4A8, or KV4 for short texts and W4, W4A8, or KV8 for long texts.Source: §7.2 · p. 9

• The evaluation excludes Quantization-Aware Training, lacks detailed studies of some hyperparameters, omits some newly available 1B–3B models, and warns that recommendations may not transfer to new tasks or models.Source: §8 · p. 9

Impact Statement

• The evaluation paper may help reduce inference-related carbon footprints, but its extensive experiments also generated considerable carbon emissions.The authors do not identify additional social impacts requiring specific discussion.Source: Impact Statement · p. 10

A.1. Large Language Model Inference

• The prefill stage is mainly compute-bound, whereas decoding is mainly memory-bound and increasingly constrained by KV Cache growth for longer contexts.The KV Cache memory overhead grows linearly with context length.Source: §A.1 · p. 13

Impact Statement

• Quantization reduces memory and computation overhead, but its impact depends on whether weights, activations, or KV Cache tensors are quantized.The study focuses on Weight-only, Weight-Activation, and KV Cache quantization rather than Activation-only quantization.Source: Impact Statement · p. 10, §A.2 · p. 13

A.2. Quantization

• Activation-only quantization is excluded because it does not notably accelerate inference, although it can reduce communication and memory costs during training.The stated scope is inference acceleration rather than training optimization.Source: §A.2 · p. 13

B.1. Introduction of Datasets

• The setup evaluates language modeling, understanding, and reasoning using CHID, Winogrande, RACE, LAMBADA, PIQA, and SIQA.The reported datasets include both Chinese and English tasks and use specified test or development splits.Source: §B.1 · p. 14

B.2. Introduction of Metrics

• Multiple-choice tasks are evaluated in PPL mode by selecting the candidate sentence with the lowest perplexity, while LAMBADA uses generation mode and checks the generated next word.OPT model-dataset combinations with near-random CHID or RACE performance are excluded from subsequent discussion.Source: §B.2 · p. 15
• Performance curves average normalized results across datasets and compare precision levels for OPT, Bloomz, Falcon, LLaMA2, and Mistral families.Figures 8–12 provide the family-level visualizations referenced by the setup.Source: §B.2 · p. 15, Figure 8 · p. 16, Figure 9 · p. 16

B.3. Additional Results on Different Tensor Types

• Most models remain close to FP16 at W4 weight quantization, suffer slight loss at W3, and collapse at W2 except Bloomz-175B and Falcon-180B.OPT is notably less tolerant, with severe degradation for most models at W3.Source: Figure 8 · p. 16, Figure 9 · p. 16, Figure 10 · p. 16
• Weight-Activation tolerance decreases with model size: most families handle W8A8 and W4A8, but larger models generally cannot use W4A8 across all tasks.W4A4 causes significant loss for nearly all models, with the stated Bloomz exceptions.Source: §B.3 · p. 17
• KV Cache quantization is usually well tolerated at KV4, while KV3 causes slight loss and often less degradation than W3, especially in larger models.The passage also reports favorable KV3 cases for Falcon-40B, Falcon-180B, and LLaMA2-70B on selected datasets.Source: §B.3 · p. 17
• < 2% performance loss is typical for W4KV4 on LAMBADA and MT-Bench, while LongEval is more sensitive and W8KV4 can lose over 10%.W8KV4 is reported as a better choice for the tested LAMBADA and MT-Bench models, whereas W4A8 or W8A8 performs better for the stated long-context cases.Source: §B.3 · p. 17

B.4. Additional Results on Different LLMs

• OPT is unusually sensitive because its higher Weight and Activation kurtosis is associated with collapse at W3, W4A8, and sometimes W8A8.OPT-6.7B has Activation kurtosis 1562.67, while OPT-66B reaches 4945.32 and cannot use even W8A8.Source: §B.4 · pp. 17, 19
• Spearman correlations between quantized and FP16 model rankings are typically above 0.9 for W4, W4A8, and KV4 or higher on the same task.The analysis uses a 6×6 block layout to compare FP16 and quantized rankings across datasets.Source: Figure 13 · p. 18, §B.4 · p. 19

B.5. Additional Results on Different Tasks

• Multiple-choice accuracy can rise after quantization because uncertain questions may be guessed correctly, so accuracy does not always reflect overall performance.For Mistral-7B on RACE, W8A8 yields a reported 3.5% accuracy increase despite higher average perplexity.Source: §B.5 · p. 20, Table 8 · p. 20, Table 9 · p. 20

B.6. Additional Results on Different Quantization methods

• AWQ improves W3 performance and approaches FP16 more closely for larger models, but cannot restore abilities lost at W2; SmoothQuant only partially recovers W4A4.The conclusions are also reported for emergent abilities, dialogue, and long-context evaluations.Source: §B.6 · pp. 20–21

B.7. Addition Results on Statistical Analysis

• Tensor distributions motivate mix-precision designs because Weight and Activation kurtosis differ across layer types, including high Activation kurtosis in down-projection layers.The paper recommends further study of mix-precision methods to balance hardware efficiency and performance.Source: §B.7 · p. 21, Table 10 · p. 21, Table 11 · p. 21
• Dynamic Activation scaling substantially reduces W4A8 performance loss relative to static scaling, especially for LLaMA2-7B and LLaMA2-70B.Applying dynamic quantization to down-projection layers and static quantization elsewhere produces perplexity close to fully dynamic quantization.Source: §B.7 · pp. 21–22, Table 12 · p. 22

C.2. Introduction of Metrics

• The emergent-ability evaluation uses MMLU, ARC-c, StrategyQA, and GSM8K with normalized accuracy relative to FP16 and theoretical minimums.Instruction-Following uses templates, Multi-Step Reasoning uses generation with chain-of-thought examples, and Self-Calibration reformulates questions as binary judgments.Source: §C.2 · pp. 22–23

C.3.1. EFFECTS OF QUANTIZATION ON DIFFERENT EMERGENT ABILITIES

• Instruction-Following and In-Context Learning are generally less sensitive to quantization than Mathematical Multi-Step Reasoning and Self-Calibration.Exceptions include LLaMA2-13B Self-Calibration improving with KV3 and Mixtral-8x7B Multi-Step Reasoning showing minimal W4A8 degradation.Source: §C.2 · p. 23, §C.3.1 · p. 24, Figure 14 · p. 24

C.3.2. MISGUIDANCE CAUSED BY EVALUATION METHOD ON STRATEGYQA

• Keyword-based StrategyQA scoring can mark incoherent or logically incorrect responses as correct after W2, W4A4, or KV2 quantization.The evaluation searches for “yes” or “no”, so guessing the keyword can conceal losses in language modeling and reasoning.Source: §C.3.2 · pp. 24–25

C.3.3. FAILURE CASES ON GSM8K

• GSM8K misjudgments are less frequent because numerical answers are harder to guess, but W3 and KV3 still produce copy, condition, calculation, and logic errors.Examples include omitting “four times”, copying a wrong formula, calculating 20+2*36 as 80, and treating 40 as the fourth-day total.Source: §C.3.3 · pp. 25, 27, 28

C.3.4. EFFECTS OF EXAMPLE NUMBER IN IN-CONTEXT LEARNING

• Increasing few-shot examples from zero to five improves FP16, W3, and KV3 LLaMA2, whereas increasing them to ten significantly harms LLaMA2-7B and LLaMA2-13B quantized models.For W2, W4A4, or KV2, accuracy is around 25% and extra examples cannot restore In-Context Learning; W2 LLaMA2-70B instead declines continuously.Source: §C.3.4 · p. 29, Table 13 · p. 30

C.3.5. EFFECTS OF QUANTIZATION METHODS

• At W3, AWQ surpasses RTN by more than 3% on GSM8K, while W2 performance collapses for both methods and SmoothQuant leaves W4A4 accuracy nearly zero.AWQ’s improvement is more significant in larger models; SmoothQuant can enhance W4A4 performance somewhat without preventing the near-zero overall accuracy.Source: §C.3.5 · p. 31, Table 15 · p. 31

D.1. Introduction of Datasets

• The evaluation covers ETHICS, TruthfulQA, and Adversarial GLUE to assess ethics, hallucination, and adversarial robustness in quantized LLMs.The supplied dataset descriptions cover moral judgments, truthful answers, and robustness under textual adversarial attacks.Source: §D.1 · pp. 31–32

D.3. Effects of Quantization on Ethics

• Weight-only quantization can improve small LLaMA2-7B and Mistral-7B ethics scores by eliciting answers, whereas KV Cache quantization makes their outputs more restricted.Large models refrain from answering fewer questions, while AWQ can slightly lower LLaMA2-7B moral accuracy despite outputs being closer to the original model.Source: §D.3 · p. 33

D.4. Effects of Quantization on Hallucination

• Quantized TruthfulQA performance consistently declines and does not follow the inverse scaling pattern observed among some full-precision models.The inverse scaling law places Mixtral-8x7B below Mistral-7B and LLaMA2-13B below LLaMA2-7B, whereas LLaMA2-70B does not follow it.Source: §D.4 · p. 34, Figure 27 · p. 36

D.5. Effects of Quantization on Adversarial Robustness

• Quantization shows no clear connection with adversarial robustness: lower bit-width sometimes reduces Acc drop, but no consistent improvement appears across models, tasks, or tensor types.The study evaluates Acc before, Acc after, and Acc drop, where Acc drop = 1 - Acc after/Acc before.Source: §D.2 · p. 32, §D.5 · p. 34, Table 16 · p. 34

E.1. Introduction of Datasets

• Dialogue quality is assessed with MT-Bench’s 80 two-turn questions and Single-answer GPT-4 grading, using reference-guided prompts for math, reasoning, and coding.The benchmark spans writing, roleplay, extraction, reasoning, math, coding, and two knowledge categories.Source: §E.1 · p. 35, §E.2 · pp. 36–37

E.3.1. CASE STUDY FOR W3 AND KV3

• Most LLMs experience notable GPT-4 score declines at W3 or KV3, with meaningless symbols and sentence-level repetition contributing to lower-quality dialogue.W3 Mixtral-8x7B can remain meaningful but occasionally emits unusual symbols, while W3 Falcon-7B-instruct repeats “Airlines: Hawaiian”.Source: §E.3.1 · pp. 37–39

E.3.2. CASE STUDY FOR W2 AND KV2

• Most LLMs completely lose dialogue ability at W2 or KV2, producing token-level randomness or repetition rather than useful responses.W2 Falcon-7B-instruct examples show both nearly random tokens and continuous repetition, while ChatGLM3-6B can remain meaningful but weakened under KV2.Source: §E.3.2 · pp. 40–41

F.1. Introduction of Datasets

• LongEval measures fine-grained key-value retrieval over 4K–16K contexts and across ten document regions to assess effective context length and positional effects.The evaluation uses 200 to 680 sentences, with 500 samples at each length and 50 region-specific questions.Source: §F.1 · p. 41, §F.2 · p. 44
• Multi-Document Question Answering tests whether models retrieve answers from 2K, 4K, or 6K contexts while varying the relevant document’s beginning, middle, or end position.The dataset combines one answer-containing Wikipedia paragraph with distractor documents and uses normalized answer matching.Source: §F.1 · p. 42, §F.2 · p. 44

F.2. Introduction of Metrics

• Long-context results are plotted as bit-width performance curves averaged across models within each family, except ChatGLM, which has one model.The supplied figures cover LongChat, Vicuna, Mistral, ChatGLM2-6B, and SOTA quantization methods.Source: §F.1 · p. 42, Figure 28 · p. 42, Figure 29 · p. 43

F.3. Additional Results on LongEval dataset

• LongChat is highly sensitive: KV8 fails beyond 6K tokens, while W8 avoids notable loss and W8A8 degrades beyond 9K tokens.Source: §F.3 · pp. 44–45
• Lower bit-widths slightly affect later context positions more for LongChat and Vicuna, although weaker early-position baselines prevent firm attribution to quantization.Source: §F.3 · p. 45
• Vicuna tolerates KV8 without significant loss, while W4 and W4A8 offer favorable efficiency–performance trade-offs; ChatGLM2-6B shows similar behavior.Source: §F.3 · p. 45
• Mistral remains more sensitive to KV Cache Quantization than Weight-only Quantization but tolerates KV4 without significant loss.Source: §F.3 · p. 45
• W3 quantization causes more than 20% accuracy loss, and AWQ improves W3 performance but does not fully recover FP16 levels.Source: §F.3 · p. 45

F.4. Additional Results on Multi-document QA dataset

• Across evaluated model families, Multi-Document QA generally shows a U-shaped performance curve over context position, with Vicuna-13B especially sensitive at the beginning under Weight-only Quantization.Source: §F.4 · p. 45, Figure 33 · p. 45