PaperlayerView exact revision on arXiv

Source-linked AI summary

Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs

Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage, Baha Rababah, Carson Leung, Murat Kantarcioglu, Cuneyt Gurcan Akcora, Rafet Sifa

arXiv:2609.07664v2cs.LGcs.AI

TL;DR

Zero-shot accuracy considers argmax decisions and can miss changes in quantized models’ full predictive distributions relative to BF16. The paper compares those distributions using TVD and JSD; across the tested setting, Q4_0 often has higher TVD than Q4_K despite similar accuracy and memory footprints.

  • Problem

    Zero-shot accuracy reduces predictions to argmax decisions and can miss changes in the full predictive distribution relative to BF16.

    Source: §I · pp. 1–2

  • Method

    The framework compares prompt-conditioned, full-vocabulary predictive distributions from uncompressed and quantized models using TVD and JSD.

    Source: B. Full-Vocabulary Probability… · p. 3, A. Design Rationale and Dual-Stream… · p. 4

  • Results

    Across tested models, benchmarks, and llama.cpp schemes, Q4_0 often yields substantially higher TVD than Q4_K despite similar task accuracy and memory footprints.

    Source: §VI · p. 7

  • Takeaways & Limitations

    TVD and JSD provide reference-fidelity signals that complement accuracy but do not establish correctness, calibration, safety, or user-perceived quality.

    Source: §VI · p. 7

    The study is limited to base models, fixed prompt templates, prompt-conditioned distributions, and llama.cpp/GGUF quantization.

    Source: §VI · p. 7

Abstract

from arXiv · show

Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, evaluating these models is largely based on zero-shot task accuracy, which depends solely on argmax predictions and is insensitive to changes in the underlying predictive distribution. Consequently, accuracy can exhibit unstable, non-monotonic behavior under progressive quantization, masking substantial fidelity loss relative to the BFloat16 (BF16) uncompressed base model and providing misleading deployment signals. We introduce a distribution-sensitive evaluation framework quantifying information loss in quantized LLMs as the divergence between full-vocabulary predictive distributions at the token decision boundary. We compute statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between outputs of full-precision and quantized models, enabling a fine-grained analysis of distributional shift. Using this framework, we quantify probability mass displacement and distributional drift relative to the BF16 reference, capturing predictive distribution changes not reflected in top-1 accuracy. We conduct a 120-run experimental matrix across five foundation architectures and four reasoning benchmarks under progressive quantization regimes, from uncompressed BF16 to Q2_K, providing a systematic fidelity analysis. Our results show divergence metrics generally increase under stronger quantization, complementing task accuracy with a fidelity signal. Across tested llama-cpp schemes, mixed-precision Q4_K generally yields lower divergence than uniform Q4_0 at similar memory footprints. These findings motivate distribution-aware evaluation as a practical diagnostic complement to task accuracy; they do not directly establish correctness, calibration, safety, or user-perceived quality.

I. INTRODUCTION

The introduction argues that zero-shot accuracy can conceal shifts in quantized models’ predictive distributions relative to BF16. It motivates full-vocabulary fidelity analysis alongside task accuracy and situates quantization within edge-device memory constraints.

I. INTRODUCTION

  • Argmax accuracy can hide substantial distributional differences, since a degraded model may guess correctly from a flattened distribution or default to a dominant prior token.The paper describes this as a misleading signal of model reliability.Source: §I · p. 2
  • The introduction calls for evaluation using continuous full-vocabulary distributions because stable aggregate accuracy can conceal internal prediction flips and uncertainty differences.It connects this concern to findings across computer vision, adversarial robustness, and medical AI.Source: §I · p. 2
  • The paper evaluates 120 configurations across five foundation models, four benchmarks, and six quantization regimes, and reports lower divergence for mixed-precision Q4_K than uniform Q4_0 at similar memory footprints.The comparison is limited to the tested llama.cpp schemes.Source: §I · p. 2

A. LLM Compression and the Precision Bottleneck

  • Post-training quantization is presented as a standard compression approach for memory-constrained edge devices, with mixed-precision K-quants among the schemes popularized by llama.cpp.The paper identifies whether quantized models preserve the BF16 reference distribution as an unresolved evaluation question.Source: A. LLM Compression and the Precision… · p. 2

B. The Illusion of Accuracy and Distributional Divergence

The related work describes limits of discrete accuracy and prior divergence analyses, while the section outlines mixed-precision quantization and the paper’s full-vocabulary representation of model outputs.

B. The Illusion of Accuracy and Distributional Divergence

  • Prior work documents erratic accuracy and prediction flips, while cited analyses based on discrete correctness do not capture the continuous predictive structure.The paper also notes that KL divergence can be sensitive to near-zero probabilities common in extreme low-bit quantization.Source: B. The Illusion of Accuracy and… · p. 2

A. Uniform vs. Mixed-Precision Quantization

  • Uniform quantization applies the same bit-width across weight tensors, whereas K-quants allocate precision dynamically, using higher precision for sensitive structures and lower precision for robust feed-forward layers.The paper describes K-quants as using double quantization to reduce metadata overhead and optimize cache-line reads.Source: A. Uniform vs. Mixed-Precision… · p. 3

B. Full-Vocabulary Probability Distribution

  • The framework represents the uncompressed and quantized models as full-vocabulary predictive distributions derived from their next-token logits for the same prompt.The two models share a tokenizer and vocabulary, differing in parameter precision.Source: B. Full-Vocabulary Probability… · p. 3

C. Statistical Divergence Metrics

The section uses bounded, symmetric divergence metrics to compare quantized-model predictions with the BF16 reference. TVD gives a probability-mass interpretation, while JSD supplies a bounded information-theoretic comparison.

C. Statistical Divergence Metrics

  • TVD and JSD are bounded and symmetric, unlike unbounded KL divergence, which can be dominated by low-probability mismatches.The paper uses them to quantify distributional drift that zero-shot accuracy may not reveal.Source: C. Statistical Divergence Metrics · p. 3
  • With base-2 logarithms, JSD lies between 0 and 1; zero indicates matching predictive distributions, while values approaching 1 indicate nearly disjoint distributions.JSD compares each distribution with their smoothed average mixture.Source: C. Statistical Divergence Metrics · pp. 3–4
  • TVD is half the L1 distance and measures the minimum fraction of probability mass that must be reassigned between distributions.A TVD of 0.25 means 25% of probability mass must be reassigned.Source: C. Statistical Divergence Metrics · p. 4

IV. METHODOLOGY AND EXPERIMENTAL SETUP

The methodology pairs standard zero-shot task scoring with prompt-conditioned, full-vocabulary comparisons to the BF16 reference. It also formalizes that mapping distributions to answer-level outputs cannot increase their discrepancy.

IV. METHODOLOGY AND EXPERIMENTAL SETUP

  • Stream 1 selects candidate answers using length-normalized sequence NLL, while Stream 2 compares prompt-level full-vocabulary distributions using TVD and JSD.For multi-token options, sequence NLL and prompt-level divergence characterize different quantities.Source: A. Design Rationale and Dual-Stream… · p. 4
  • Coarse-graining full-vocabulary distributions to answer-level distributions cannot increase TVD; the text states that the analogous JSD result is proved in the appendix.This gives a formal basis for treating answer-level aggregation as potentially losing distributional discrepancy.Source: A. Design Rationale and Dual-Stream… · p. 4
  • The evaluation setup uses the zero-shot benchmarks listed in Table II alongside the dual-stream scoring and fidelity analyses.The supplied table caption identifies Table II as the evaluation benchmarks used in the zero-shot experiments.Source: A. Design Rationale and Dual-Stream… · p. 4, Table II · p. 5

B. Quantization, Prompting, and Flip Analysis

The evaluation compares quantization states generated from BF16 baselines while controlling prompt formatting and tracking prediction flips.

B. Quantization, Prompting, and Flip Analysis

C. Metrics

The study uses TVD and JSD to compare full-vocabulary predictions with a BF16 reference, evaluates five models on four benchmarks, and shows that stable accuracy can mask changed predictions.

C. Metrics

  • TVD is the primary tabular metric because it represents the fraction of probability mass to reassign between predictive distributions.TVD and JSD are tracked alongside accuracy and prediction flips.Source: C. Metrics · p. 5

D. Evaluated Models, Datasets, and Hardware

  • Five foundation models are evaluated on MMLU, BoolQ, PIQA, and CausalBench using deterministic inference on one V100 GPU with a 2048-token context window.The stated seed is 42.Source: D. Evaluated Models, Datasets, and… · p. 5

A. The Masking Effect of Task Accuracy

  • On PIQA, Teuken-7B Q4_0 accuracy falls only 0.2 points, from 68.9% to 68.7%, despite 4.84% prediction flips and TVD 0.560.The TVD corresponds to reassigning 56% of probability mass to match the BF16 distribution.Source: A. The Masking Effect of Task Accuracy · p. 5
  • On CausalBench, Gemma-2B and LLaMA-3.2-1B remain near 50.1% accuracy through Q4_K while predicting the negative class uniformly, with zero precision and recall.On BoolQ, both models instead predict True for all examples, scoring 62.2% with zero flips; TVD complements class-level metrics by measuring divergence from BF16.Source: A. The Masking Effect of Task Accuracy · p. 5

B. Quantifying Information Loss via TVD

On PIQA, accuracy and F1 fluctuate non-monotonically across quantization states, while divergence metrics generally rise with stronger quantization. The examples show that accuracy alone does not capture distributional distance from BF16.

B. Quantifying Information Loss via TVD

  • Table III covers quantization effects on accuracy, prediction stability, and information loss across PIQA and MMLU.Source: Table III · p. 6
  • TVD and JSD generally increase under stronger quantization on PIQA, while accuracy and F1 can fluctuate non-monotonically.Figure 3 contrasts task metrics in its top panels with divergence metrics in its bottom panels.Source: B. Quantifying Information Loss via TVD · pp. 5–6, Figure 3 · p. 6
  • LLaMA-3.1-8B at Q2_K scores 74.6% accuracy versus 77.3% for BF16, with TVD 0.257.Matching the BF16 next-token distribution requires reassigning 25.7% of probability mass.Source: B. Quantifying Information Loss via TVD · p. 6

C. Mixed-Precision vs. Legacy Uniform Quantization

Across the evaluated settings, Q4_K generally has lower divergence from the BF16 reference than Q4_0 despite similar memory footprints and task accuracy. Across the tested models, benchmarks, and llama.cpp schemes, Q4_K is a favorable empirical memory–fidelity trade-off, not a universal deployment threshold.

C. Mixed-Precision vs. Legacy Uniform Quantization

  • For LLaMA-3.1-8B, Q4_0 and Q4_K have nearly identical VRAM footprints of about 5.5 GB and similar zero-shot accuracy on PIQA and MMLU.PIQA accuracy is 77.2% versus 77.1%, and MMLU accuracy is 32.1% versus 32.2%.Source: C. Mixed-Precision vs. Legacy Uniform… · p. 7
  • Across the evaluation suite, Q4_K has lower average divergence, while Q4_0 shows greater variation in prediction flips across datasets.Figure 4 compares flips and TVD averaged across the evaluation suite, with error bars showing standard deviation across datasets.Source: C. Mixed-Precision vs. Legacy Uniform… · p. 7, Figure 4 · p. 7
  • For LLaMA-3.1-8B on MMLU, Q4_K has TVD 0.055 versus Q4_0 at 0.221; for Teuken-7B, the corresponding TVDs are 0.143 and 0.541.The comparison is made despite similar task accuracy for the LLaMA-3.1-8B schemes.Source: C. Mixed-Precision vs. Legacy Uniform… · p. 7

D. The Hardware Pareto Frontier

  • Lower-memory quantization states are generally associated with higher TVD, though the magnitude varies across architectures and schemes.LLaMA-3.1-8B falls from 15.2 GB in BF16 to 5.27 GB with Q4_K while its reported TVD remains below 0.055; Q2_K shows larger divergence in some architectures.Source: D. The Hardware Pareto Frontier · p. 7
  • Q4_K is a favorable empirical memory–fidelity trade-off among the tested models, benchmarks, and llama.cpp schemes, but it is not a universal deployment threshold.The comparison does not replace latency, throughput, energy, calibration, or generation-quality evaluation on target devices.Source: D. The Hardware Pareto Frontier · p. 7

VI. CONCLUSION AND FUTURE WORK

The conclusion presents full-vocabulary JSD and TVD as measures of fidelity to an uncompressed reference, alongside accuracy. It reports a favorable Q4_K memory–fidelity trade-off in the tested setting, bounds the scope of the study, and states a coarse-graining result for JSD.

VI. CONCLUSION AND FUTURE WORK

  • Full-vocabulary JSD and TVD measure fidelity to the uncompressed reference rather than correctness, calibration, safety, or user-perceived quality.JSD gives a bounded information-theoretic comparison, while TVD quantifies probability mass that must be reassigned to transform the quantized distribution into the BF16 reference distribution.Source: §VI · p. 7
  • Across the tested models, benchmarks, and llama.cpp schemes, Q4_0 often has substantially higher TVD than Q4_K despite similar task accuracy and memory footprints.Q4_K provides a favorable empirical memory–fidelity trade-off in this setting, including up to a 65% memory reduction for the reported LLaMA-3.1-8B configuration.Source: §VI · p. 7
  • The study is limited to base models, fixed prompt templates, prompt-conditioned full-vocabulary distributions, and llama.cpp/GGUF quantization.Future work is proposed on multi-step generation, other quantization methods, prompt-template sensitivity, computational scalability, and target-hardware measures.Source: §VI · p. 7

VII. THEOREMS

  • Coarse-graining by K preserves the JSD bound: JSD(KP, KQ) is no greater than JSD(P, Q).The proof applies the log-sum inequality to each KL term and averages the resulting bounds.Source: §VII · p. 9

VIII. COMPLETE EMPIRICAL RESULTS

The appendix presents complete quantitative evaluations across 120 configurations, organized by benchmark. The supplied table passages identify MMLU and BoolQ among the benchmark tables.

VIII. COMPLETE EMPIRICAL RESULTS

  • The appendix reports zero-shot accuracy, F1, prediction flips, JSD, and TVD for five foundation models under progressive quantization regimes.The complete evaluation spans 120 experimental configurations and is organized by benchmark dataset.Source: §VIII · p. 9
  • Table IV is the complete evaluation metrics table for MMLU.Its listed metrics include VRAM, accuracy, F1, percentage of flips, JSD, and TVD.Source: Table IV · p. 10, §VIII · p. 10
  • Table VI is the complete evaluation metrics table for BoolQ.The appendix identifies accuracy, F1, flips, JSD, and TVD as reported evaluation measures.Source: §VIII · p. 9, Table VI · p. 10

LLaMA-3.1-8B

The supplied table passages identify PIQA and CausalBench as complete evaluation tables and separately show a LLaMA-3.2-1B table header; they provide no LLaMA-3.1-8B row values.

LLaMA-3.1-8B

  • Table V is labeled as the complete evaluation metrics table for PIQA.Source: Table V · p. 10
  • Table VII is labeled as the complete evaluation metrics table for CausalBench.Source: Table VII · p. 10
  • The displayed table header lists Model & Quantization, VRAM, Acc (↑), F1 (↑), % Flips (↓), JSD (↓), and TVD (↓) as its columns.Source: LLaMA-3.2-1B · p. 10
Loading 2609.07664v2…