PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

Sanae Lotfi, Polina Kirichenko, Steven Li, Zechun Liu

arXiv:2606.00206v1cs.LG

TL;DR

The paper examines how post-training quantization affects reasoning models, finding that aggressive quantization is associated with longer chains of thought and lower accuracy, including failures to retain correct intermediate answers. It analyzes token-level distribution differences and tests a training-free penalty on overthinking markers, reducing reasoning length while preserving or improving accuracy.

  • Problem

    Although post-training quantization is widely used for efficient LLM deployment, its effects on reasoning models are not well understood.

    Source: §1 · p. 1

  • Method

    The paper compares quantized and full-precision token distributions using KL divergence, then applies a training-free logit penalty to curated overthinking markers.

    Source: §1 · p. 2

  • Results

    Across five models, three quantization methods, and five benchmarks, penalizing overthinking markers reduces CoT length by 12–23% while preserving or improving accuracy.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Penalizing overthinking markers yields a more favorable accuracy–reasoning-cost Pareto frontier than penalizing other token sets.

    Source: §1 · p. 2

    The evaluation covers math, coding, and science benchmarks, leaving open whether the same overthinking patterns arise in other reasoning and planning domains.

    Source: §7 · p. 10

Abstract

from arXiv · show

Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood. Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length. Surprisingly, we show that in up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer. To understand why quantization leads to this increase in overthinking errors, we measure the token-level KL divergence between quantized and full-precision output distributions. Positions with high KL divergence correlate strongly with high next-token entropy, and at these positions quantized models disproportionately sample overthinking markers such as "wait", "but", and "alternatively". We show that simply introducing a training-free logit penalty on a curated set of overthinking markers can reduce CoT length by 12--23% while preserving or improving accuracy across 5 models (1.5B-32B parameters), 3 quantization methods, and 5 benchmarks, yielding a favorable Pareto frontier of accuracy against reasoning cost compared to penalizing other token sets. Overthinking errors produced by quantized models are particularly reduced by up to 58%.

1 Introduction

Aggressive post-training quantization can make reasoning models abandon correct intermediate answers, contributing to longer chains and lower accuracy. The paper analyzes token-level divergence and tests a training-free penalty on overthinking markers.

1 Introduction

  • The paper compares quantized and full-precision output distributions using token-level KL divergence under identical generation prefixes to isolate quantization effects.PTQ is described as requiring no gradient updates or training-pipeline access and as usable with limited calibration data.Source: §1 · pp. 1–2
  • High-divergence tokens include hesitation and branching markers, which concentrate at high-entropy positions; mathematical tokens carrying reasoning content have low divergence.Markers appear 2–4× more frequently among the top-20 predictions at high-entropy steps than at low-entropy ones.Source: §1 · p. 2
  • A training-free logit penalty on overthinking markers cuts chain-of-thought length by 12–23% on average while preserving or improving accuracy and improving the accuracy–reasoning-cost Pareto frontier.Controlled ablations find that random-token penalties have no consistent effect, while penalizing lowest-KL tokens can increase CoT length by up to 41% and reduce accuracy by up to 9.5%.Source: §1 · p. 2
  • In up to 52% of failures under aggressive quantization, models reach the correct answer during reasoning but fail to commit to it as the final answer.Overthinking accounts for 52% of errors under AWQ 3-bit versus 26% in BF16.Source: §1 · p. 2

2 Related Work

Related work covers methods for low-bit post-training quantization, research on reasoning efficiency, and studies of quantization’s effects on reasoning. This paper focuses on token-level trace changes and a training-free mitigation.

2 Related Work

  • Low-bit PTQ work includes GPTQ, AWQ, SmoothQuant, ZeroQuant, QuIP, QuIP#, QuaRot, SpinQuant, and FlatQuant, which use different strategies to reduce quantization error or distortion.The paper evaluates AWQ and GPTQ at 3–4 bits and FlatQuant at 4-bit or 8-bit configurations.Source: §2 · p. 3
  • Prior efficient-reasoning work addresses over-allocation, repetitive looping, non-monotonic links between trace length and correctness, and mitigation through training or decoding strategies.The paper distinguishes its focus as understanding quantization-related overthinking and mitigating it without training.Source: §2 · p. 3
  • Studies report that low-bit PTQ can reduce reasoning accuracy and increase chain-of-thought length, while this paper examines token-level trace changes and overthinking mitigation.The cited work also describes quantization effects as task- and metric-dependent.Source: §2 · p. 3

3 Experimental Setup

The experiments evaluate five reasoning-specialized models with weight-only and end-to-end quantization across math, coding, and science benchmarks. The setup includes an illustrative accuracy and chain-of-thought comparison for one model.

3 Experimental Setup

  • The study uses GPTQ and AWQ for 3-bit and 4-bit weight-only quantization, and FlatQuant for W4A4KV4 and W8A8KV8 end-to-end quantization.GPTQ uses second-order information; AWQ uses calibration activation statistics to identify salient channels and rescale them.Source: §3 · p. 3
  • Five reasoning-specialized models span 1.5B–32B parameters, including four DeepSeek-R1-Distill models and QwQ-32B.QwQ-32B acquires reasoning capabilities through reinforcement learning with verifiable rewards rather than distillation.Source: §3 · p. 3
  • The five benchmarks span mathematics, coding, and science, including GSM8K, MATH-500, AIME-120, LiveCodeBench, and GPQA-Diamond.The setup uses default decoding with T=0.6 and top-p=0.95.Source: §3 · pp. 3–4
  • For DeepSeek-R1-Distill-Qwen-1.5B on MATH-500, 3-bit AWQ lowers accuracy from 85.6% to 47.0% and raises CoT length from 5.2K to 23.4K tokens, a 4.5× increase.Figure 2 compares accuracy and CoT length as quantization becomes more aggressive.Source: Figure 2 · p. 4

4 Quantization Exacerbates Overthinking in Reasoning Models

Aggressive post-training quantization reduces reasoning accuracy while lengthening chains of thought, and it increases failures in which models reach but do not commit to a correct answer. Token-level comparisons associate this overthinking with divergence at uncertain positions and a greater prevalence of hesitation and branching tokens.

4.1 Accuracy drops while CoT length increases under PTQ

  • On MATH-500, 3-bit AWQ lowers accuracy from 85.6% to 47.0% while increasing average CoT from 5.2K to 23.4K tokens; GPTQ 3-bit retains 71.6% accuracy with 6.8K tokens.Across additional models, aggressive quantization also lengthens traces, with the effect most pronounced for smaller models and harder benchmarks; Qwen-14B loses 4.0% average accuracy under 3-bit AWQ.Source: §4.1 · p. 4
  • Across 28 model–quantization pairs, accuracy degradation and CoT-length increase correlate at ρ = −0.73.Quantized models with larger accuracy losses tend to produce longer reasoning traces.Source: §4.1 · p. 4

4.2 Quantization increases overthinking errors

  • Under 3-bit AWQ, overthinking rises from 19 to 139 errors, accounting for 52% of 265 errors; longer traces include hesitation that can turn correct intermediate answers into wrong finals.Overthinking means reaching a correct solution during reasoning but failing to commit to it as the final answer.Source: §4.2 · p. 5

4.3 Quantized models diverge from non-quantized ones in branching positions

  • With identical generation prefixes, token-level KL analysis finds the greatest divergence on branching tokens such as “Wait” and “Alternatively,” while mathematical and formatting tokens diverge least.Position-level KL divergence correlates strongly with full-precision next-token entropy (Spearman ρ = 0.92).Source: §4.3 · p. 6, Figure 4 · p. 6
  • Under 3-bit AWQ, high-KL hesitation tokens outnumber low-KL mathematical tokens per unit of CoT, reversing the BF16 ratio of 0.57 to 1.15.The paper associates this token-composition shift with less efficient reasoning.Source: §4.3 · p. 6

5 Penalizing Overthinking Markers

The paper applies a training-free logit penalty to a curated set of overthinking markers. Across configurations, the penalty shortens reasoning while generally preserving accuracy, including across multiple models and in overthinking-error counts.

5.1 Overthinking penalty

  • The method penalizes a curated set of 50 tokens expressing hesitation, self-doubt, or redirection, excluding high-KL tokens with other roles such as “so.”The evaluation compares this set with equally sized high-KL, low-KL, and random token lists.Source: §5.1 · pp. 6–7, §5.2 · p. 7

5.2 Penalizing overthinking markers reduces CoT while preserving accuracy on average

  • On Qwen-1.5B averaged across six quantization configurations and five benchmarks, overthinking-marker penalties reduce CoT by 12% to 23% while maintaining or slightly improving accuracy.Low-KL penalties increase CoT by up to 41% and reduce accuracy by up to 9.5%; high-KL penalties can degrade accuracy at stronger settings.Source: §5.2 · p. 7, Figure 5 · p. 7
  • Boosting rather than suppressing overthinking markers increases CoT length by up to 445% and drops accuracy by up to 34%.Random-token boosting remains neutral, while high-KL tokens show a comparable effect to overthinking markers.Source: §5.2 · pp. 7–8

5.3 Overthinking penalty improves efficiency across models

  • Across benchmarks, the penalty reduces CoT length by 4.1% to 28.0% on average across five models; on AWQ 3-bit with Qwen-1.5B, accuracy improves by 6.2% while CoT decreases by 28.0%.In most cases accuracy is preserved or slightly improved, though some cases trade accuracy for shorter CoT.Source: Table 1 · p. 8, §5.3 · p. 9
  • On AWQ 3-bit, overthinking errors fall from 139 to 58, a 58% reduction, while total errors decrease from 265 to 194.On FlatQuant W4A4KV4, overthinking errors fall from 64 to 34, a 47% reduction; the penalty also reduces such errors across three extreme GSM8K configurations from 344 to 237, a 31% reduction.Source: §5.3 · p. 9

6 Why Does Quantization Exacerbate Overthinking?

The paper tests whether quantization noise affects uncertain decoding positions more strongly, then relates those positions to overthinking markers. Its evidence supports this as intuition for one failure mode, not a complete causal account of PTQ degradation.

6 Why Does Quantization Exacerbate Overthinking?

  • On MATH-500, position-level KL divergence between full-precision and quantized models correlates with full-precision next-token entropy at ρ = 0.92.The analysis associates higher uncertainty with greater divergence between the two models.Source: §6 · p. 9
  • Overthinking markers are 2 to 4× more likely to appear among the top-20 most probable tokens at high-entropy positions than at low-entropy positions, as shown in Figure 7.Source: §6 · p. 9

7 Conclusion, Limitations, and Future Work

Aggressive quantization amplifies overthinking, while a training-free penalty on 50 curated markers reduces chain-of-thought length and improves the accuracy–reasoning-cost tradeoff. The conclusion also identifies open questions about adapting the penalty, constructing marker lists, and extending evaluation beyond math, coding, and science.

7 Conclusion, Limitations, and Future Work

  • A 50-marker training-free logit penalty reduces CoT length by 12–23% and improves the accuracy–reasoning-cost Pareto frontier across five models, three quantization methods, and five benchmarks.Aggressive quantization exacerbates overthinking: models reach correct intermediate answers but fail to commit, producing longer CoTs and lower accuracy.Source: §7 · p. 10
  • Overthinking also occurs in BF16 models, where the penalty reduces CoT length by 5.6% to 10.0%, depending on the model.Quantization amplifies a failure mode already present in reasoning models.Source: §7 · p. 10
  • Comparing token-level divergences between a reference model and its compressed variant could diagnose which aspects of generation are most vulnerable to compression.The authors note that the analysis is not specific to quantization and may apply to perturbations such as pruning, distillation, or low-rank approximation.Source: §7 · p. 10
  • The penalty uses a fixed λ at every decoding step, and its marker list is manually curated for English reasoning models.Whether overthinking patterns arise in other reasoning and planning domains remains open; evaluation covers math, coding, and science benchmarks.Source: §7 · p. 10

A.1 Calibration Data

The supplied passages describe quantization methods and experiments examining uncertain decoding positions, KL-divergent tokens, and the effect of boosting candidate token lists. They do not provide calibration-data details for A.1.

A.2 Quantization Methods

  • GPTQ uses a blockwise sequential procedure and approximate inverse-Hessian information to minimize output reconstruction error while quantizing one layer at a time.The evaluation uses weight-only GPTQ at 3 and 4 bits with group size 128.Source: §A.2 · p. 13
  • AWQ uses calibration activations to identify salient channels and rescales weights before quantization to reduce output distortion.The rescaling is folded into stored weights, so AWQ remains weight-only at inference.Source: §A.2 · p. 13
  • FlatQuant quantizes weights, activations, and the KV cache for low-precision inference without gradient-based retraining.The evaluated settings are W4A4KV4 and W8A8KV8.Source: §A.2 · p. 13

A.3 Penalty Sweeps

  • The experiments sweep penalty strength across token lists, quantization configurations, and benchmarks, including a negative-λ sweep that boosts rather than suppresses target tokens.The token-list ablation compares overthinking markers, high-KL tokens, random tokens, and low-KL tokens; the negative sweep totals 960 runs.Source: §A.3 · p. 14

B.1 Qualitative Example of Overthinking from MATH-500

  • In a MATH-500 example, the AWQ 3-bit model reaches the correct intermediate result but then branches on repeated “wait” and “but” tokens and fails to terminate.The comparison is against the same model in BF16, which follows a short, concise solution.Source: §B.1 · p. 14, Figure 6 · p. 15

B.2 Position of Thinking Tokens in the Next-token Probability Distribution

  • Thinking tokens are ≈2–4× more likely to appear in the top 20 predictions at high-entropy than low-entropy positions across the tested models and quantized variants.High entropy is defined using a per-model threshold at the 80th percentile of its entropy distribution.Source: §B.2 · p. 14, Figure 7 · p. 15

B.3 Word Cloud Visualizations of High-KL and Low-KL Tokens

  • The 50 highest-KL tokens are dominated by hesitation and redirection, while the lowest-KL tokens are mathematical symbols, digits, and formatting tokens.Token size in the word clouds reflects average KL divergence between BF16 and AWQ 3-bit output distributions on MATH-500.Source: §B.3 · pp. 14, 16, Figure 8 · p. 16

B.4 Negative Penalty Sweep: Boosting Overthinking Markers

  • At λ = −3.0, boosting overthinking markers drops accuracy by 31% and increases CoT by 395%, while boosting high-KL tokens drops accuracy by 18% and increases CoT by 294%.Random tokens remain nearly neutral, and low-KL tokens increase CoT by only 10% even at λ = −4.0.Source: §B.4 · p. 16, Figure 9 · p. 16

B.5 Error Analysis on GSM8K

On GSM8K, the overthinking pattern generalizes beyond MATH-500: penalizing overthinking markers reduces overthinking errors, with AWQ 3-bit also gaining accuracy.

B.5 Error Analysis on GSM8K

  • 35% fewer overthinking errors and 6.1% higher accuracy on AWQ 3-bit after penalizing overthinking markers.Errors fall from 197 to 128 on GSM8K; penalizing random or low-KL tokens does not consistently reduce overthinking errors and can increase them.Source: Figure 10 · p. 17, §B.5 · p. 17

B.6 Full Quantization Evaluation

Across models and benchmarks, more aggressive quantization generally lowers accuracy while increasing CoT length. The overthinking penalty consistently shortens CoTs and usually preserves or improves accuracy, improving the efficiency–accuracy tradeoff.

B.6 Full Quantization Evaluation

  • More aggressive quantization generally decreases accuracy while increasing CoT length across models and benchmarks.The evaluation compares BF16 with weight-only GPTQ and AWQ at 3 and 4 bits, plus FlatQuant settings, using accuracy and average CoT length.Source: §B.6 · p. 17, Table 2 · p. 18

B.7 Effect of the Overthinking Penalty

  • The penalty consistently reduces CoT length across models and quantization configurations, with the largest reductions under 3-bit weight-only quantization and W4A4KV4.Accuracy is preserved or improved in most configurations; larger models show smaller absolute CoT reductions, but the intervention remains effective.Source: §B.7 · pp. 17–18

C Token Lists Used in Experiments

The appendix defines four token lists for the penalty and control experiments, and describes the GPT-5 procedure used to categorize incorrect generations.

C Token Lists Used in Experiments

  • Tokens prefixed with “_” include a leading space and therefore have distinct token IDs from their space-free variants.The overthinking-marker, high-KL, and random-token tables specify leading-space tokens.Source: C Token Lists Used in Experiments · p. 18, Table 8 · p. 21, Table 9 · p. 21
  • Overthinking markers are 50 hesitation, redirection, or backtracking tokens selected by inspecting tokens frequently sampled at high-KL positions.The list is described as consistent with a similar class of hesitation tokens in prior work (Zhao et al., 2025).Source: C Token Lists Used in Experiments · p. 18
  • The control lists contain 50 high-KL tokens, 50 low-KL tokens, and 50 random tokens; low-KL items are mainly mathematical, digit, and formatting tokens.The high-KL list partially overlaps with overthinking markers and also includes functional tokens unrelated to overthinking.Source: C Token Lists Used in Experiments · pp. 18–19
  • Tables 3–7 compare base and penalized accuracy and CoT length across five models and quantization configurations, with mean changes reported across five benchmarks.The captions identify Qwen-1.5B, Qwen-7B, Qwen-14B, Llama-8B, and QwQ-32B; they do not provide cell values here.Source: Table 3 · p. 19, Table 4 · p. 19, Table 5 · p. 20

D GPT-5 Prompt for Error Categorization

  • The GPT-5 judge assigns exactly one of four error labels and selects overthinking when the gold answer appears in reasoning but not in the final answer.It never failed to return exactly one category in the reported experiments and achieved >95% agreement with human annotations on a subset of incorrect questions.Source: D GPT-5 Prompt for Error Categorization · p. 19

Related papers

Loading 2606.00206v1…