Source-linked AI summary
Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compression
Dachi Kurtskhalia
TL;DR
Capability benchmarks often treat 4-bit quantization as nearly costless, but they do not test whether models preserve diversity when multiple answers are valid. This paper compares serving precisions across Qwen3 scales and finds that int4 narrows recommendation diversity at 8B, while larger models show stylistic drift without stereotype amplification.
Problem
Existing quantization evaluations largely use tasks with nearly single correct answers, leaving distributional effects on valid recommendations and defaults insufficiently tested.
Method
The study serves Qwen3-8B, 14B, and 32B checkpoints at W4A16 AWQ, W8A16 FP8-Marlin, and bf16 under controlled hardware, software, and sampling settings, using paired prompt-level tests across pre-specified analysis waves.
Results
Int4 increases same-brand recommendation collisions by 5.1 percentage points and reduces lexical diversity at 8B, while 14B and 32B show stylistic em-dash drift; stereotype-direction tests are null across scales.
Takeaways & Limitations
Quantization amplifies determinism rather than bias in the tested settings, so deployment audits should measure concentration alongside stereotype skew.
Takeaways & Limitations
The evidence covers one model family, one quantizer per precision, an English automotive-retail battery, and scale comparisons involving 32B with a parallelism difference.
Abstract
from arXiv · showhide
Weight quantization largely determines the economics of serving open-weight LLMs. Its costs are usually assessed with capability benchmarks, on which 4-bit quantization of mid-sized models is often considered "nearly free." We examine a different question: when several answers are valid, does quantization change what a model chooses to say? We serve three checkpoints (Qwen3-8B/14B/32B) at three weight precisions (W4A16 AWQ, W8A16 FP8-Marlin, and bf16), holding the hardware, software, and sampling configuration constant, and collect approximately 71,000 completions paired by prompt and seed across two custom, leak-checked prompt batteries. We pre-specified the analyses in three waves in version control. At 8B, int4 reduces output diversity: the probability that two samples for the same scenario recommend the same brand increases by 5.1 percentage points (prompt-paired sign-flip test, Holm p = .023; reproduced at +4.4pp on a full regeneration of the arm), and lexical diversity falls substantially (TTR -0.011, standardized effect -0.51; robust to a length-controlled measure). At 14B and 32B, no content-concentration measure reaches significance; instead, stylistic drift emerges (em-dash rate +0.46/1k words at 14B and +0.61/1k at 32B, both Holm p <= .0024). Pre-specified tests of stereotype direction are null at every scale: outputs concentrate on the modal answer for each prompt rather than on stereotypical answers. Mechanistically, the token-level distribution becomes flatter (decision-token entropy +0.091 bits, p = .015) while the semantic distribution, measured directly from first-token log probabilities, becomes more concentrated (collision +2.6pp, p = .023): individual tokens become less predictable even as meanings become more repetitive. At 8B, the smallest size tested, AWQ-int4 serving measurably narrows the range of suggestions; audits should assess concentration as well as bias.
1 Introduction
Conventional evaluations suggest W4 quantization is nearly costless, but they do not test whether deployed assistants retain diverse valid answers. This paper distinguishes concentration of recommendations from stereotype bias and studies whether serving-time compression intensifies default-answer behavior.
- W4 quantization appears nearly costless on perplexity and capability benchmarks, which mainly assess tasks with one near-correct answer.Those evaluations do not measure how much of the valid answer space a deployed assistant continues to use.
- Serving-time compression may reinforce output concentration created by RLHF and instruction tuning, making deployed models more homogeneous than conventional benchmarks suggest.
- Bias favors particular brands, whereas concentration narrows the set of brands recommended; the experiments find only the latter.
- The study uses a controlled three-precision, three-size evaluation and reports 8B diversity loss, larger-model stylistic drift, no stereotype amplification, and divergent token- versus semantic-level effects.
2 Related work
Prior work establishes standard quantization methods, reduced output diversity from preference tuning, default-answer tendencies, and diversity measures from recommender systems. This paper examines how compression relates to these phenomena in LLM outputs.
- GPTQ, AWQ, and FP8 are standard serving methods, while benchmark literature generally finds W4 quantization nearly costless for mid-sized models.
- Prior studies report reduced output diversity after preference tuning and defaulting toward Japan when models are asked to name a country.
- Recommender-systems research treats aggregate diversity and catalog coverage as primary evaluation criteria, motivating collision probability as an LLM-oriented analogue.
3 Method
The study compares three Qwen3 sizes across three serving precisions using leak-checked cultural and recommendation batteries, paired prompt-level tests, and direct token-probability instruments. Its measurement and statistical design separates content, style, concentration, stereotype direction, and mechanism.
- Precision ladder: Three Qwen3 checkpoints—8B, 14B, and 32B—are served at W4A16 AWQ, W8A16 FP8-Marlin, and bf16 with matched software, flags, sampling, and within-model parallelism.
- Batteries: The batteries contain 218 country-eliciting prompts, 80 style-control prompts, and 96 brand-blind car-buying scenarios designed without explicit country or brand mentions.
- Batteries: Prompts contain no country, city, nationality, brand, or model names after adversarial review and gazetteer-based leakage checks.
- Measurement: Countries and brands are extracted with disambiguating gazetteers, while style is measured using focal-word rate, em-dash rate, constructions, and type-token ratio.
- Statistical protocol: Primary int4−bf16 contrasts use paired prompt-level two-sided sign-flip tests with 10,000 flips, bootstrap confidence intervals, rarefaction, and Holm correction.
- Mechanism instruments: A separate 8B run measures top-20 token log probabilities and maps first-token probabilities directly to brands under a fixed recommendation prefix.
4 Results
At 8B, int4 concentrates recommendations around prompt-specific modal answers and reduces lexical diversity, while larger models show stylistic rather than content drift. Token-level uncertainty rises even as semantic outputs become more repetitive.
- 4.1 At 8B, int4 collapses recommendation diversity: The collapse is modal rather than global: Toyota’s share rises from 42% to 47%, Honda’s falls from 14% to 9%, and their combined share is unchanged.The modal answer absorbs probability from alternatives within scenarios, while the modal brand varies across scenarios.
- 4.1 At 8B, int4 collapses recommendation diversity: bf16 produced four brands across 20 samples, whereas int4 produced Tesla in all 20 for the largest observed shift.This example illustrates prompt-specific collapse rather than a universal brand preference.
- 4.2 The effects dissociate by scale: At 8B, content concentration appears alone; at 14B and 32B, stylistic drift appears instead, with em-dash rates increasing by +0.46 and +0.61 per 1k words.The standardized effects are near-identical at 0.25, with relative increases of +19% and +12%.
- 4.4 Mechanism: token noise increases while semantic diversity declines: At 8B, decision-token entropy increases by 0.091 bits while exact brand collision increases by 2.6 percentage points under int4.Token distributions become flatter even as the semantic distribution becomes more concentrated.
- 4.5 Robustness: At temperature 0.4, TTR still declines by −0.0090, but the collision contrast falls below significance at +3.2pp.Colder sampling concentrates both precision tiers, reducing headroom for between-tier separation.
5 Discussion
Quantization can narrow recommendation diversity without amplifying stereotype direction, and this effect is concentrated at the smallest tested model. Deployment audits should therefore measure concentration alongside skew.
- 5 Discussion: A quantized small model can pass stereotype-direction audits while reducing the set of answers provided for a query.The paper reports this pattern at 8B, where concentration occurs without a consistent stereotypical direction.
- 5 Discussion: Deployment audits should include concentration measures such as collision probability or the number of distinct answers across repeated queries.The effect is absent at 14B and 32B in this model family.
- 5 Discussion: For small quantized models supporting recommendations, practitioners may consider a larger model at the same precision, diversity-promoting decoding, or explicit inventory conditioning.These options are presented as practical considerations within the reported automotive-recommendation setting.
6 Limitations
The study’s limitations constrain how broadly its findings can be generalized across model families, quantizers, domains, languages, scales, and sampling temperatures.
- Scope: The experiments cover only Qwen3 and one quantizer at each precision, leaving family-specific boundaries and calibration-set effects untested.A pre-specified follow-up addresses calibration-set effects.
- Scope: The recommendation battery covers automotive retail with US-market framing, and the main analysis is limited to English.
- Serving differences: 32B was served at TP=8 on a different instance type, confounding cross-scale comparisons with a parallelism difference.
- Effect heterogeneity: The 8B concentration effect is heterogeneous across prompts, with a median per-prompt delta of 0, indicating that a minority of scenarios collapse hard.
- Sampling boundary: The collision contrast falls below significance at temperature 0.4.
Reproducibility
The project’s reproducibility materials are maintained in a version-controlled repository and are designed to support local reruns against compatible inference endpoints.
- Repository contents: The repository maintains all code, prompts with construction provenance, gazetteers, analysis scripts, frozen statistics, and figures.
- Audit trail: The repository history records the declare-then-run sequence used for the analyses.
- Execution: Scoring and analysis run locally against any OpenAI-compatible vLLM endpoint.
A Complete wave-1 results
The complete wave-1 results are available in the accompanying repository, while headline rows appear in Tables 1 and 2.
- Wave-1 results: The repository provides a full 30-row wave-1 table covering style, lexical diversity, mention, entropy, country-share, and brand-share measures.
- Wave-1 results: The table includes Kobak, em-dash, not-X-but-Y, TTR, mentions/completion, and rarefied entropy measures with Miller–Madow values.
- Wave-1 results: Japan shares and Toyota+Honda shares are reported for all three models.
- Access: Headline rows appear in Tables 1 and 2, with the complete table stored at results/stats_summary.csv.
B Complete wave-2 and exploratory concentration results
Table 3 reports wave-2 confirmatory and exploratory concentration measures as int4 −bf16 contrasts across all model sizes.
- Wave-2 concentration: Table 3 presents wave-2 confirmatory concentration measures using int4 −bf16 contrasts.
- Wave-2 concentration: Table 3 also includes exploratory concentration measures alongside the confirmatory results.
- Model sizes: The table reports these concentration measures for all sizes.
C Complete wave-3 (stereotype-direction) results
Wave 3 reports stereotype-direction measures as int4 −bf16 differences across all model sizes, alongside a style-control comparison on 80 placeless prompts. The style-control drifts are not significant at any size.
- Table 4 reports wave-3 stereotype-direction measures as int4 −bf16 differences across all sizes.
- None of the em-dash or Kobak drifts reaches significance at any size in the style-control arm.The comparison supports the specificity of the cultural-prompt drift; no formal arm-by-tier interaction test was run.
- Table 5 reports Kobak and em-dash deltas for the 80 placeless style-control prompts.