Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models

Ruikang Liu; Yuxuan Sun; Manyi Zhang; Haoli Bai; Xianzhi Yu; Tiezheng Yu; Chun Yuan; Lu Hou

Source-linked AI summary

Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models

Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, Lu Hou

arXiv:2504.04823v2Apr 7, 2025cs.CLcs.AI

TL;DR

Reasoning models improve complex-task performance but incur inference overhead from extended chain-of-thought, while quantization’s effects on these models remain understudied. The paper systematically evaluates quantization across model families, precisions, components, and reasoning benchmarks, finding that suitable settings can preserve accuracy while more aggressive quantization risks degradation.

Problem
Quantization is widely used to reduce LLM inference cost, but its impact on reasoning models remains understudied despite their extended reasoning overhead.
Method
The study evaluates weight-only, weight-activation, and KV cache quantization across 1.5B–70B reasoning models, multiple algorithms and bit-widths, and mathematical, scientific, and programming benchmarks.
Results
Suitable settings preserve reasoning accuracy, whereas lower-bit quantization introduces accuracy risks that vary with task difficulty, model size, model origin, and model family.
Takeaways & Limitations
W8A8 and W4A16 are recommended lossless settings, while model-size or reasoning-step scaling can improve quantized-model performance.

Abstract

from arXiv · show

Recent advancements in reasoning language models have demonstrated remarkable performance in complex tasks, but their extended chain-of-thought reasoning process increases inference overhead. While quantization has been widely adopted to reduce the inference cost of large language models, its impact on reasoning models remains understudied. In this paper, we conduct the first systematic study on quantized reasoning models, evaluating the open-sourced DeepSeek-R1-Distilled Qwen and LLaMA families ranging from 1.5B to 70B parameters, QwQ-32B, and Qwen3-8B. Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths, with extensive evaluation across mathematical (AIME, MATH-500), scientific (GPQA), and programming (LiveCodeBench) reasoning benchmarks. Our findings reveal that while lossless quantization can be achieved with W8A8 or W4A16 quantization, lower bit-widths introduce significant accuracy risks. We further identify model size, model origin, and task difficulty as critical determinants of performance. Contrary to expectations, quantized models do not exhibit increased output lengths. In addition, strategically scaling the model sizes or reasoning steps can effectively enhance the performance. All quantized models and codes are open-sourced in https://github.com/ruikangliu/Quantized-Reasoning-Models.

1 Introduction

Reasoning models achieve strong performance but can incur substantial inference overhead from prolonged chains of thought. This study systematically evaluates quantization for reasoning models to characterize accuracy, efficiency, and practical trade-offs.

Motivation: Reasoning models perform strongly on complex tasks but may generate outputs hundreds of times longer than non-reasoning models.Their prolonged reasoning processes increase inference overhead and can cause overthinking on simple tasks.
Motivation: Quantization reduces numerical precision in weights or activations and is favored for balancing model-size reduction with performance preservation.Most existing quantization methods were developed for non-reasoning LLMs, leaving their impact on reasoning models under-explored.
Key findings: 8-bit weight-activation quantization preserves accuracy, while 4-bit weight-only and KV cache quantization achieve near-lossless results with drops of ≤1%.The study also compares quantization algorithms, task difficulty, model origins, output lengths, and scaling effects.
Significance: Quantization is presented as one component of a broader effort to make reasoning inference more efficient despite long chain-of-thought outputs.The work is among the first comprehensive evaluations of quantized reasoning models and aims to guide future quantization methods.

2 Preliminary and Related Work

The paper frames quantization as an orthogonal approach to efficient reasoning and reviews its operational forms alongside reasoning-model families and other efficiency methods. Its scope emphasizes reduced memory or computation while preserving reasoning performance.

Quantization basics: Uniform quantization maps high-precision values to lower-precision representations using b-bit integer points.The study focuses mainly on hardware-efficient uniform quantization.
Quantization configurations: Weight-only quantization stores Transformer weight matrices at low precision while leaving other parameters in BF16 to reduce model size and runtime memory access.The study focuses on GPTQ and AWQ with asymmetric quantization and group size 128.
Quantization configurations: KV cache quantization targets intermediate autoregressive-generation results to reduce memory access during long reasoning sequences.The evaluated methods include QuaRot and KVQuant*.
Quantization configurations: Weight-activation quantization applies low-precision arithmetic to Transformer matrix multiplications to further reduce computation.The evaluated methods include SmoothQuant, QuaRot, and FlatQuant.
Related work: Reasoning models include systems that acquire reasoning abilities through distinct training approaches, while efficient-reasoning methods also target their long output sequences.Related approaches include length penalties, token budgets, forced termination or extension, and direct performance optimization.
Related work: Quantization is characterized as orthogonal to other efficient-reasoning methods, motivating a comprehensive study of its accuracy-efficiency trade-offs.The paper argues that such an evaluation is needed to understand quantized reasoning models.

3 Evaluation of Quantized Reasoning Models

The evaluation examines quantized reasoning models across benchmarks, model sizes, origins, quantization methods, and bit-widths. Results show that 8-bit and selected 4-bit settings can preserve accuracy, while lower-bit and more aggressive configurations are especially risky on difficult tasks and smaller models.

3.2 Lossless Quantization Bit-width: Table 1 classifies quantization degradation as lossless at ≤1%, fair at 1%-3%, and risky at ≥3%, with severe low-bit exceptions for 1.5B and 7B models.The table summarizes DeepSeek-R1-Distill-Qwen results across reasoning benchmarks and quantization configurations.
3.2 Lossless Quantization Bit-width: 4-bit weight-only and KV-cache quantization are comparatively safe, while 3-bit settings become risky, with KV-cache drops exceeding 5% for 1.5B and 7B models.For weight-only quantization, the 1.5B and 32B models show 2.1% and 0.4% drops at 4 bits, versus over 7% and 3% at 3 bits.
3.2 Lossless Quantization Bit-width: 8-bit quantization of weights, activations, and KV cache remains below a 1-point performance drop, whereas 4-bit weight-activation quantization drops 2.9% on 32B and over 10% on 1.5B and 7B models.The authors identify improving 4-bit weight-activation quantization for reasoning models as an open challenge.
3.3 Comparisons of Quantization Algorithms: AWQ is recommended for weight-only quantization, QuaRot generally for KV-cache quantization, SmoothQuant for 8-bit weight-activation quantization, and FlatQuant for the 4-bit setting.Exceptions include Qwen 1.5B and 7B for KV-cache quantization, where bias outliers favor KVQuant*, while FlatQuant still suffers substantial losses on smaller models under W4A4KV4.
3.4 Impacts of Task Difficulty: Quantization causes larger performance drops on harder tasks: for 32B W4A4KV4, drops are 3.9% on AIME-120, 1.2% on MATH-500, and 0.0% on GSM8K.Model origin also affects robustness: Qwen-32B is more robust to KV-cache quantization, QwQ-32B to weight and weight-activation quantization, Qwen-7B to weight-only quantization, and LLaMA-8B to KV-cache and activation quantization.

4 Discussions

Quantized reasoning models generally preserve output lengths, but calibration data, model size, latency, and test-time reasoning budgets materially shape accuracy and efficiency. Larger models and 4-bit configurations can improve trade-offs, while excessive reasoning or mismatched calibration can hurt performance.

Output Length: Quantized models generate output lengths closely aligned with BF16 models across configurations, despite reduced numerical precision.This includes W4G128, W8A8KV8, and KV4 settings.
Calibration Data: Calibration-domain effects are strongly method-dependent: GPTQ gains 9.8% average accuracy when switching from WikiText2 to Numina-Math-1.5.Reasoning and pre-training data exhibit domain gaps that can create training-inference inconsistency for calibration-sensitive methods.
Model Size Scaling: At similar model sizes, 3-bit quantized QwQ-32B and Qwen-32B outperform BF16 Qwen-7B by around 17% on LiveCodeBench.Accuracy improves rapidly as model size increases, and larger low-precision models can outperform smaller full-precision models at comparable sizes.
Latency Scaling: 4-bit quantized models provide superior accuracy–latency trade-offs to 8-bit and BF16 models on LiveCodeBench while maintaining the same reasoning length.The advantage mainly comes from increased generation speed.
Test-time Scaling: Longer reasoning generally improves accuracy, but quantized models improve more slowly than BF16 and degrade when reasoning becomes excessively long.The paper attributes this pattern to accumulated quantization error and unnecessary reasoning steps, recommending an appropriate scaling range.

5 Conclusion

The paper presents a systematic empirical study of quantized reasoning models and identifies settings and factors associated with preserving or degrading performance. It concludes that accuracy–efficiency optimization remains incompletely understood.

5 Conclusion: The study systematically evaluates state-of-the-art quantization methods across reasoning models and configurations, recommending W8A8 and W4A16 as lossless settings.It also examines task difficulty, model origins, training recipes, model families, and test-time compute scaling.
5 Conclusion: More difficult tasks incur more severe quantization-related performance degradation, while quantized models improve with increased test-time compute over an appropriate range.Performance may worsen after the useful scaling range.
5 Conclusion: The mechanisms driving quantized reasoning models’ performance effects remain incompletely understood, motivating future work on causal relationships and accuracy–efficiency optimization.The paper identifies the influencing factors but does not fully explain their underlying mechanisms.

A Quantization Algorithms and Implementation Details

The paper implements weight-only, KV-cache, and weight-activation quantization methods using calibration, scaling, transformations, and mixed-precision strategies. These implementations cover methods including GPTQ, AWQ, KVQuant, QuaRot, SmoothQuant, FlatQuant, and MXFP4.

Weight-only Quantization: GPTQ minimizes squared error between full-precision and quantized outputs using iterative weight updates informed by approximate inverse-Hessian information.Cholesky decomposition is used for numerical stability and computational efficiency.
Weight-only Quantization: AWQ protects salient weights through per-channel scaling that balances activation and weight magnitudes while avoiding runtime overhead after merging scaled weights.Its calibration data consists of 128 Pile sequences of length 512.
KV-Cache Quantization: KV-cache quantization targets memory access during long autoregressive sequences, using QuaRot and a simplified KVQuant* implementation.QuaRot uses asymmetric per-token quantization with group size 128, while KVQuant exploits outlier channels and non-uniform cache distributions.
Weight-Activation Quantization: QuaRot enables end-to-end 4-bit inference by applying Hadamard transformations to mitigate activation and KV-cache outliers before quantization.It also integrates GPTQ for weight-error reduction and uses distinct per-token or per-channel quantization schemes by component.
Weight-Activation Quantization: SmoothQuant migrates quantization difficulty from activations to weights, while FlatQuant learns layer-specific transformations and MXFP4 uses grouped 4-bit floating-point scaling.The study evaluates these approaches for weight-activation quantization, with calibration settings specified for each method.

B.1 Results on More Quantized LLMs

Across additional reasoning models and benchmarks, 4-bit weight-only, 4-bit KV-cache, and 8-bit weight-activation quantization are generally near-lossless, whereas lower-bit settings become risky, especially on harder tasks. Quantization damage increases with task difficulty, but minor-degradation settings do not lengthen model responses.

Quantization results: 4-bit weight-only quantization is lossless or nearly lossless across QwQ, DeepSeek-R1-Distill-LLaMA, and Qwen3-8B, while lower-bit quantization causes substantial accuracy losses.QwQ shows over 7% LiveCodeBench degradation at 3 bits; Qwen3-8B exceeds 13% average degradation at 3-bit weight-only quantization.
Quantization results: 8-bit weight-activation quantization remains lossless across the evaluated models and tasks, while only FlatQuant preserves accuracy in the more challenging 4-bit setting on QwQ.SmoothQuant is sufficient for 8-bit weight-activation quantization, whereas FlatQuant leads in the 4-bit scenario.
Task difficulty: Quantization-induced accuracy loss becomes more pronounced as MATH-500 difficulty increases, with FlatQuant-W4A4KV4 losing over 2% at level 5.The same setting performs comparably to the full-precision 14B model overall, but complex tasks are more vulnerable to precision reduction.
Output length: Reasoning models generate more tokens on harder problems, but quantized models with minor accuracy degradation do not generate longer responses than BF16 models.Lower bitwidths on smaller models can affect this pattern, particularly for DeepSeek-R1-Distill-Qwen-1.5B.

C.1 Choice of Calibration Data

Calibration data from reasoning and pre-training domains has similar channel-wise magnitude structure, including shared outlier channels, while t-SNE reveals domain gaps. Activation distributions across reasoning datasets remain close despite dataset category differences.

Distribution analysis: t-SNE visualizations show domain gaps between reasoning and pre-training activations.The visualization samples 128 tokens from each domain.
Distribution analysis: Reasoning and pre-training calibration data share similar channel-wise magnitude distributions and the same outlier channels.The comparison uses Numina-Math-1.5, LiveCodeBench, and WikiText2 data.
Calibration data: Activation distributions across different reasoning datasets are close to each other regardless of dataset category.Calibration data includes self-generated reasoning data from Numina-Math-1.5 and LiveCodeBench alongside WikiText2 pre-training data.

C.2 Analysis on DeepSeek-R1-Distill-Qwen-1.5B and 7B Models

DeepSeek-R1-Distill-Qwen-1.5B and 7B models contain extreme K-cache outlier channels linked to unusually large key-projection biases inherited from their pretrained Qwen models. These outliers make KV-cache quantization more difficult and help explain method-specific performance differences.

Quantization impact: The large inherited biases significantly increase KV-cache quantization difficulty in the two DeepSeek-R1-Distill-Qwen models.The outliers help explain the large performance gap between KVQuant* and QuaRot reported for these models.
Visualization: Figure 10 compares K-cache visualizations before and after adding the key-projection bias term in the 1.5B and 7B models.The plotted quantity is labeled Ypre bias.
Outlier-channel origin: DeepSeek-R1-Distill-Qwen-1.5B and 7B models exhibit extreme K-cache outlier channels caused by large key-projection bias terms.The maximum absolute key-projection bias reaches 402 in Qwen-1.5B, and the distilled models inherit these biases.

C.2.2 Recipes for Extreme Outlier Channels on K Cache

The paper proposes two practical remedies for extreme K-cache outliers: quantizing keys before bias addition and extending calibration sequence length for FlatQuant. Pre-bias quantization improves low-bit KV-cache accuracy, while long-sequence calibration addresses reasoning models’ extended thinking steps.

Pre-bias quantization: Pre-bias K-cache quantization reduces the quantization range by quantizing keys before adding the bias term.The pre-bias K cache appears smoother and more compatible with quantization.
Extended calibration: Figure 11 reports channel-wise K-cache magnitude distributions before and after RoPE across sequence-length ranges.Yk denotes self-attention key-projection outputs.
Pre-bias quantization: 7.81% average accuracy improvement is obtained with pre-bias quantization for 3-bit KV-cache-only quantization.The improvement is reported consistently across different models and bit-widths.
Extended calibration: Extending calibration sequence length targets FlatQuant’s degradation over long reasoning sequences.Reasoning models generate long thinking steps, while RoPE rotates key-projection outliers into adjacent channels.

D Qualitative Examples

Qualitative examples show that quantized models generally preserve the BF16 model’s reasoning pattern, while occasionally making calculation errors or using different verification strategies. Examples include both correct and incorrect answers on MATH-500.

Reasoning patterns: Quantized models generally follow the BF16 model’s four-part reasoning pattern: understand, process, check, and final answer.The paper uses different colors to distinguish these reasoning stages.
Reasoning patterns: Quantized models can preserve this reasoning pattern even when they answer a problem incorrectly.The paper illustrates this behavior with an incorrect MATH-500 example.
Observed deviations: Quantized models may occasionally exhibit calculation errors or adopt slightly different verification strategies.These differences are presented as occasional deviations from the overall shared reasoning pattern.
Correct examples: The examples include correctly answered MATH-500 problems from BF16, GPTQ-W4A16G128, and quantized-model settings.Tables 8–10 provide correct-answer examples across these model configurations.
Incorrect example: One MATH-500 example was incorrectly answered by the MXFP4-quantized DeepSeek-R1-Distill-Qwen-32B model.The example concerns computing the angle ABC from three-dimensional point coordinates.