Source-linked AI summary
The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
Jundong Hu, Shekar Ramachandran
TL;DR
Quantization damage varies across LLMs, but it is unclear where precision should be restored and how a limited budget should be allocated. The paper tests three proposed localizers with causal mixed-precision interventions across nine models, finding diffuse recovery and a consistent matched-budget advantage for globally finer granularity over local layer repair. It concludes that causal intervention is needed to identify useful precision restoration, within the tested budget setting.
Problem
Uneven quantization accuracy costs raise the problem of locating damage and allocating limited precision without relying on model-specific trial and error.
Method
The study evaluates nine open-weight models across four architecture families, testing circuit, computation-site, and weight-statistic localizers against layerwise 4-bit-to-8-bit causal intervention.
Results
Global finer granularity beats oracle-selected local layer repair by 21–52 points for all 8 group-128-compatible models, while none of the three localizers predicts marginal recovery.
Takeaways & Limitations
Within the tested matched-budget setting, global granularity is a better default than selectively protecting critical layers, and useful recovery locations require causal testing.
Takeaways & Limitations
The study does not test non-greedy layer sets, weight-level salience protection, or Hessian sensitivity, and its evaluation is capped at models of at most 8B parameters.
Abstract
from arXiv · showhide
Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground truth (raise each layer to 8-bit in turn and measure the accuracy it recovers) across 9 open-weight models in 4 architecture families, we test 3 intuitive hypotheses: that quantization damage lives in task circuits, where the model computes, or in weight statistics. None of them predicts which layers benefit from restored precision. Recovery is instead diffuse: for 8 of 9 models, recovering 75% of the gap takes roughly half the layers; the lone exception, Qwen3-8B, is sharply concentrated. At a matched precision budget, spending it globally on finer quantization granularity beats locally repairing the most recoverable layers for all 8 group-128-compatible models (all but OpenLLaMA, whose width rules out group-128), by 21-52 points, including the concentrated Qwen3-8B. We report 2 secondary findings: the residual is budget-limited (8-bit is near-lossless in our evaluation across RTN, GPTQ, and AWQ), and the location of peak recovery correlates with architecture within a family, though not across families. Within this budget setting, global granularity is a better default than selectively protecting critical layers. More broadly, cheap signals that correlate with quantization damage do not necessarily identify where restoring precision improves accuracy; this must be tested with causal intervention.
1 Introduction
The paper asks where uneven quantization damage occurs and how to allocate limited precision. Causal intervention rejects three intuitive localizers and instead supports diffuse recovery, with global finer granularity outperforming local repair at matched budget.
- Quantization damage varies across models and tasks, motivating questions about its location and precision-budget allocation.
- Three localizers—task circuits, computation sites, and weight statistics—are tested against causal mixed-precision intervention.Each candidate location is raised to 8-bit while the rest remains at 4-bit, and recovered accuracy measures marginal precision value.
- None of the three localizers reliably predicts marginal precision value; recovery is generally diffuse.
- At matched effective bits/weight, global granularity beats oracle-selected local layer repair by 21–52 points across all 8 group-128-compatible models.The comparison includes Qwen3-8B, the most concentrated model.
- Damage is diffuse, the residual is budget-limited with 8-bit near-lossless, and peak recovery locations correlate with architecture within families but not across them.
2 Related Work
The paper situates its analysis within standard 4-bit PTQ and existing sensitivity- or salience-based allocation methods. It introduces no quantization method, instead separating granularity from quantization algorithm.
- The operating range combines standard 4-bit PTQ with near-lossless mixed-precision 8-bit inference.The 4-bit setting includes RTN, group-wise scaling, GPTQ, and AWQ.
- Existing sensitivity- and salience-based allocation methods assume quantization damage is localizable.
- The paper introduces no quantization method and separates the effects of granularity and quantization algorithm.
3 Experimental Setup
The study evaluates nine open-weight models across four architecture families and diverse tasks using a fixed continuation-scoring harness. It operationalizes three damage hypotheses with calibration-set localizers and controlled quantization probes.
- Nine open-weight models spanning four architecture families and a 16× size range are evaluated on 22 tasks.Tasks cover reading comprehension, commonsense completion, factual retrieval, and formal or symbolic reasoning.
- The evaluation uses 200 samples per task with a fixed continuation-scoring harness and seed 1337.
- The three hypotheses use circuit drift, causal activation patching, and per-layer weight statistics as cheap localizers.
- Localization uses per-row RTN at 4-bit, while budget analysis adds GPTQ, AWQ, group-128, and an 8-bit tier.OpenLLaMA is excluded from the group-128 budget analysis because its intermediate dimension is 8640.
4 Three Hypotheses, None Sufficient: The Damage Is Diffuse
None of the three proposed localizers identifies where restoring precision recovers accuracy. Recovery is generally diffuse, while peak locations show limited within-family architectural regularity and do not support a universal layer-selection rule.
- Localizers: Circuit drift loses its apparent damage correlation after controlling for model and task category, while causal patching points to boundary layers that usually recover little.The adjusted drift correlation is +0.05 and nonsignificant; boundary layers recover ≤13% of the gap in 6 of 9 models.
- Localizers: Weight standard deviation and reconstruction error fail to identify recoverable layers outside LLaMA.
- Diffuse recovery: Reaching 50%, 75%, and 90% of the gap takes approximately 20%, 49%, and 73% of layers on average across all 9 models.No single layer accounts for more than approximately 44% of damage, so most recovery requires roughly half the network.
- Architecture dependence: Qwen3-8B is sharply concentrated, but peak locations predict held-out size within LLaMA-3.x and not a new architecture family.LLaMA-3.x peaks at L1 across sizes, whereas Qwen peaks move with scale.
5 Where the Next Bit Should Go: At a Matched Budget, Global Granularity Beats Local Repair
At an equal +0.146 effective bits/weight budget, globally finer granularity recovers more quantization accuracy than selectively restoring the most recoverable layers. This advantage persists across all 8 group-128-compatible models, including Qwen3-8B, and is driven mainly by granularity rather than quantization algorithm.
- Matched-budget comparison: +0.146 effective bits/weight is matched between global group-128 granularity and local 8-bit repair of top-ranked layers.Recovery is measured as a percentage of the per-row RTN4→RTN8 CORE gap.
- Matched-budget comparison: 21–52 points: global granularity yields higher recovery than oracle-selected local repair for all 8 group-128-compatible models.OpenLLaMA is excluded because its intermediate dimension is incompatible with group-128.
- Matched-budget comparison: 77 vs. 55%: global granularity beats local repair even for Qwen3-8B, the most concentrated model.The local arm is selected using each layer’s own protect-one recovery, making the comparison favorable to local repair.
- Robustness: 5.16–6.33 effective bits/weight: local repair would need substantially more budget than the tested increment to match global granularity on 7 diffuse models.This corresponds to 7–15× the disputed +0.146 effective bits/weight increment.
- Mechanism: Granularity contributes more than calibration: per-row→g128 RTN recovers +0.095 CORE on average, versus +0.020/+0.017 for GPTQ/AWQ at the same granularity.At 8-bit, per-row RTN matches fp16 within harness noise for 6/8 models.
6 Limitations
The study’s conclusions are bounded by its intervention choices, scale, evaluation design, and first-order budget accounting. Several alternatives and broader regimes remain untested.
- Intervention scope: The no-few-layer-fix conclusion applies only to greedy, recovery-ranked interventions at layer granularity.Non-greedy layer sets and weight-level salience protection are untested, as is Hessian sensitivity under H3.
- Quantization regime: The localization results use per-row RTN as a maximal-damage probe, leaving smaller GPTQ, AWQ, and g128 gaps untested.
- Scale: A single 40 GB A100 partition limited evaluation to models of at most 8B parameters, so larger scales remain untested.
- Evaluation design: Top-k curves are oracle selectors because layers are ranked and scored on the same evaluation set; small-model recoveries also approach the noise floor.Mistral and OpenLLaMA are each represented by one model.
- Budget accounting: The equal-budget comparison uses first-order, per-layer, weight-only accounting; exact-byte allocation and multi-bit rate–distortion sweeps are left for future work.
A Task Circuit Clusters
The task-circuit analysis identifies five stable groups across all nine models, with a universal formal-versus-rest split at two clusters. These groupings track computational demands more consistently than task labels.
- Cluster structure: k=2 is the strongest universal split: formal/structured tasks separate from the rest in every model across a 16× size range.The reported five-group solution is the finest split that remains stable across all nine models.
- Cluster structure: Five stable task groups recur across all 9 models, based on head-deviation profiles clustered by cosine similarity.The cluster count was selected using silhouette score without task labels.
- Clustering signal: 0.48–0.60: head-only silhouette scores at k=5 across models, and head-only clustering outperforms head+MLP in 8/9 models.Adding MLP neurons lowers quality in examples such as Llama-3-8B, from 0.54 to 0.43.
- Task-demand interpretation: 9/9 models: arc_easy and arc_challenge co-cluster, consistent with a shared multiple-choice mechanism.
- Task-demand interpretation: 7/9 models: language_identification clusters with formal tasks, while commonsense_qa forms a persistent orphan across 7 different partner sets.Similar clusters indicate relative activation patterns, not identical physical heads; each model finds its own circuits.
B Circuit Drift Is a Task-Type Confound (H1)
Circuit drift’s pooled association with quantization damage disappears after jointly controlling for model and task category, so it does not identify where precision restoration pays off.
- Circuit drift: Jointly controlling for model and task category makes drift–damage association nonsignificant and TOST-equivalent to zero within ±0.2.The pooled association remains significant under separate controls, but the joint-control result removes the apparent relationship.
- Circuit drift: The mixed-effects drift coefficient is −0.235 (p = 0.015) before category control, while within task clusters all |r| ≤0.23 and are nonsignificant.These corroborating tests locate the confound in task type rather than a stable layer-level signal.
- Alternative localizers: Protecting the causal boundary pair recovers at most 55.5% of the gap for Mistral-7B, while 6 of 9 models recover ≤13%.The boundary pair is the first and last MLP, but its causal computation location generally does not predict the repair location.
- Alternative localizers: Weight-standard-deviation fragility matches the protect-one recovery peak only in the LLaMA family.This limits weight-statistic localization as a cross-family selector.
- Alternative localizers: For Qwen3-8B, the top three recoverable layers cut reconstruction error by only ∼7% yet recover nearly all accuracy, while the remaining 285× reduction buys essentially nothing.Reconstruction error also rises toward later layers, whereas CORE recovery is concentrated earlier.
E Diffuseness and Concentration of Recovery
Layer-recovery effects are generally diffuse rather than concentrated in a few layers, although Qwen3-8B is a sharp exception.
- Recovery structure: Table 5 reports best single-layer and tested-pair recovery as percentages of the RTN4→RTN8 CORE gap, with residual defined as 100 minus the better recovery.Qwen3-8B’s tested pairs missed sweep peak L4, so its protect-one L4 recovery is reported.
- Diffuse recovery: No single layer accounts for more than ∼44% of damage in any model, consistent with damage being distributed across layers.The largest single-layer damage values range from 8.1% for Qwen3-1.7B to 43.9% for Mistral-7B.
- Recovery structure: Pair recovery closely matches the sum of its two single-layer recoveries, with ratios of 0.91–1.10 in the validation examples.This supports interpreting tested-pair effects as broadly additive in those cases.
- Diffuse recovery: 20/49/73% of layers are needed on average to recover 50/75/90% of the RTN4→RTN8 gap across all 9 models.Layers are ranked by their individual protect-one recovery, an oracle ordering.
- Concentrated exception: Qwen3-8B’s top 3 most-recoverable layers recover essentially the whole gap, with multi-seed recovery of 39.7 ± 3, 88.2 ± 2, and 102.2 ± 4 for k=1, 2, and 3.The slight result above 100% is attributed to evaluation-sampling noise around the 8-bit ceiling.
F Budget and Method Decomposition
At the tested matched budget, global granularity contributes more than quantization-method choice, while the 8-bit tier is near-lossless across the evaluated methods.
- 4-bit decomposition: +0.095 is the mean granularity gain, compared with GPTQ +0.020 and AWQ +0.017 after excluding Qwen3-8B from method means.The method gains are 0.21× and 0.18× the granularity gain, respectively.
- 4-bit decomposition: Effective bits per weight are 4.01 for per-row RTN, 4.156 for g128, and 16 for fp16.These values frame the precision-budget comparison.
- 8-bit tier: Per-row RTN at 8-bit is near-lossless, with mean gap to fp16 of −0.001 and every model within ±0.005.The Qwen3-8B GPTQ>fp16 anomaly resolves at 8-bit: GPTQ-8 is 0.418 versus fp16 0.411.
- Stability and scope: AWQ calibration is stable across seeds with 0–2% spread, whereas GPTQ varies by 4–24%, worst on Qwen3-8B.The budget analysis uses three calibration-subset seeds for GPTQ and AWQ.
- Stability and scope: Table 6 separates per-model granularity gain from GPTQ/AWQ method gain over g128-RTN, excluding OpenLLaMA from g128 because its intermediate dimension is not divisible by 128.The group-128 budget analysis therefore covers 8 models.
G Equal-Budget Allocation: Global vs. Local (Matched Effective Bits/Weight)
At a matched +0.146 effective bits/weight, globally finer group-128 quantization outperforms oracle-selected local layer repair across the compatible models, including Qwen3-8B. The comparison remains robust to task bootstrap uncertainty, while accounting limitations are bounded.
- Caveat: The local accounting is first-order, but it would need to be wrong by 7–15× to reverse the result for the 7 diffuse models.Those models require 5.16–6.33 bits/weight for local repair to match global recovery, compared with group-128’s 4.156.
- Matched-budget comparison: Qwen3-8B is the closest case, matching global recovery at 4.206 bits/weight versus group-128’s 4.156 bits/weight.Its single best layer recovers only ∼40%, and the local arm is oracle-selected, so this remains an upper-bound comparison for local repair.
- Robustness: P(global>local)≥0.95 for 8/8 models under task bootstrap resampling, with 95% margin intervals excluding the local arm for 6/8.The two exceptions are the near-noise 0.5B model and the smallest-margin singleton.
- Robustness: Qwen3-8B’s global-favoring bootstrap margin is +25, with a 95% CI of [12, 37] and P=1.00.Its per-seed single-layer recovery is tight, so the reported uncertainty is task- rather than seed-driven.