Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories

1 INTRODUCTION

• Low-bit quantization reduces LLM memory footprints, while preserving reasoning at low precision remains challenging and QAT commonly trains on fixed response sequences.The effectiveness of fixed-response distillation depends on how well its training prefixes cover those encountered during student inference.Source: §1 · p. 1
• Fixed-response distillation supervises reference prefixes, while the student’s context increasingly drifts from them as its responses lengthen.On long-response MATH-500 questions, QAT baselines recover little full-precision accuracy and often reach the token limit when the full-precision model terminates within budget.Source: §1 · p. 1
• On student-generated prefixes, the calibrated student underpredicts some candidates and overpredicts others, with both errors more frequent at high-entropy prefixes.Forward KL more reliably corrects severe underprediction, while reverse KL more reliably corrects severe overprediction.Source: §1 · p. 2
• OnQ starts from a block-wise calibrated checkpoint, then uses student-generated prefixes for teacher supervision and candidate-wise Dynamic KL to balance correction of under- and overpredicted probabilities.On MATH-500, it recovers more accuracy than every QAT baseline at every response length and rarely reaches the token limit.Source: §1 · p. 2
• OnQ achieves the highest nine-benchmark average in all six settings, improving the Qwen3-8B average at 2 bits by 10.26 points over the strongest baseline.The study also reports shorter responses with fewer repetitive errors.Source: §1 · p. 2

2 RELATED WORK

• EfficientQAT uses block-wise optimization followed by end-to-end quantization-step refinement, while ReasoningQAT combines mixed-domain calibration with teacher-guided fine-tuning.Other QAT methods include LLM-QAT and BitDistiller.Source: §2 · p. 2
• LLM-QAT, BitDistiller, and ReasoningQAT compute teacher supervision on fixed training sequences rather than on student generations.The cited methods are LLM-QAT [Liu et al., 2024], BitDistiller [Du et al., 2024], and ReasoningQAT.Source: §2 · p. 2
• On-policy distillation trains on learner-generated prefixes; related objectives balance forward and reverse KL using teacher confidence, distribution discrepancies, entropy, or candidate-wise probability ratios.OnQ applies candidate-wise Dynamic KL on quantized student rollouts.Source: §2 · p. 3

3.1 QUANTIZATION AND INITIALIZATION

• Block-wise calibration starts from round-to-nearest quantization and trains transformer blocks sequentially with a reconstruction loss.The calibrated checkpoint provides the starting policy for on-policy distillation.Source: §3.1 · pp. 3–4

3.2 ON-POLICY DISTILLATION WITH DYNAMIC KL

• The student samples responses through its quantized forward pass, while the full-precision teacher supplies next-token distributions at those same generated prefixes.The prefixes may contain earlier student errors, so supervision follows the trajectories visited by the student.Source: §3 · p. 3, §3.2 · p. 4
• On student-generated prefixes, candidate-wise Dynamic KL weights forward and reverse KL by each candidate’s teacher–student probability ratio.Underprediction gives greater weight to forward KL, while overprediction gives greater weight to reverse KL; the objective approaches the corresponding direction for severely mispredicted candidates.Source: §3.2 · pp. 4–5
• Each training step holds sampled responses and teacher targets fixed while updating student weights and scales, then generates fresh responses from the updated policy.Responses are capped at 4,096 tokens for steps 1–50, 8,192 for steps 51–100, and 16,384 afterwards.Source: §3.2 · p. 5

4 EXPERIMENTAL SETUP

• Experiments use Qwen3-1.7B, 4B, and 8B at 2- and 3-bit weight precision, with each student supervised by its original BF16 counterpart.OnQ uses 25,600 prompts across mathematics, code, science, and general instruction following; training runs for 400 steps on 1.7B and 300 steps on 4B and 8B.Source: §4 · p. 5
• Comparisons include GPTQ and AutoRound as post-training quantization baselines and EfficientQAT, BitDistiller, and ReasoningQAT as QAT baselines.Baseline training data and configurations are detailed in Appendices A.1 and A.5.Source: §4 · p. 5
• Evaluation uses nine benchmarks in thinking mode, reporting pass@1 for code and accuracy for other tasks, with an unweighted nine-benchmark average.The response cap is 32,768 tokens, with 8,192-token exceptions for selected PTQ settings; responses reaching the output-token limit count as truncated.Source: §4 · p. 5

5 MAIN RESULTS

• OnQ achieves the highest nine-benchmark average in all six model/bit settings and the highest score on 48 of 54 benchmark–setting pairs.It outperforms both PTQ and QAT baselines on every backbone and bit width.Source: §5 · p. 6
• OnQ achieves the highest score on every benchmark at every model size at 2 bits, and the largest gains occur on long-horizon reasoning benchmarks.At every model size, its margin over the strongest baseline is larger at 2 bits than at 3 bits.Source: §5 · pp. 6–7
• At 3 bits, OnQ comes within four points of the full-precision average on Qwen3-4B and 8B.Figure 4 plots accuracy against mean generated tokens for Qwen3 models at 2 and 3 bits, averaged over three evaluation seeds.Source: §5 · p. 7, Figure 4 · p. 7

6.1 HOW DOES ONQ CHANGE LOW-BIT GENERATION?

• At 2 bits, OnQ has the highest accuracy across MATH-500, AIME, and LiveCodeBench at all three model sizes, with shorter mean responses than the most accurate baseline in eight of nine cases.At 3 bits, it has the highest accuracy among QAT methods on all three benchmarks at every model size, with shorter mean responses in six of nine cases.Source: §6.1 · p. 7
• OnQ produces repetitive errors on only 2.4% of MATH-500 questions and none on AIME, reducing repetition that accounts for most of its accuracy gain over ReasoningQAT.Figure 5 compares generation outcomes on MATH-500 for Qwen3-1.7B W2 and AIME for Qwen3-4B W3.Source: §6.1 · p. 8, Figure 5 · p. 8
• Calibration lowers average truncation at every model size and bit width, and on-policy distillation reduces it further in all six settings.On-policy distillation also closes about half or more of the remaining gap to full precision in every setting, with substantial gains on MATH-500 and LiveCodeBench.Source: §6.2 · p. 9

6.2 WHICH DESIGN CHOICES DRIVE THE GAINS OF ONQ?

• On-policy Dynamic KL exceeds fixed-response Dynamic KL on all three benchmarks, by up to 8.40 points.The comparison uses Qwen3-1.7B W2 from the same calibrated checkpoint and the same number of response tokens.Source: §6.2 · p. 9, Table 3 · p. 9
• Dynamic KL achieves the highest average across three benchmarks and the highest MATH-500 and MMLU-Redux scores.Relative to FKL (GKD) and RKL (Vanilla OPD), it improves MATH-500 by 1.0 and 4.2 points, respectively.Source: §6.2 · p. 9, Table 4 · p. 9

7 CONCLUSION

• OnQ achieves the highest nine-benchmark average for Qwen3-1.7B, 4B, and 8B at both 2- and 3-bit precision, with the largest gains at 2 bits and on long-horizon tasks.On-policy training outperforms fixed-response distillation, and Dynamic KL has the best average among the on-policy objectives compared.Source: §7 · p. 9
• The recovered models produce shorter responses with fewer repetitive errors.Source: §7 · p. 9
• The results support training on self-generated trajectories as an effective route to recovering reasoning under aggressive quantization.Source: §7 · p. 9

A.1 DATA

• Table 5 covers calibration windows, OPD prompts, and question–response pairs shared by the QAT baselines, with training data drawn from seven listed datasets.The listed sources include OpenMathReasoning, OpenCodeReasoning-2, OpenScienceReasoning-2, Dolci-Think, Tülu 3, IF-multi-constraints, and Nemotron-Instruction-Following-Chat.Source: Table 5 · p. 15, §A.1 · p. 15

A.2 TRAINING DETAILS

• Calibration uses batch size 2 and initializes ranges at 0.7 times group weight extrema; OPD samples 128 prompts per step at temperature 1.0 without top-p or top-k truncation.Learning rates begin at 2×10^-5 for 2-bit latent weights or 10^-5 for 3-bit weights, and 10^-4 for scales and zero points; AdamW uses no warmup, 0.01 weight decay, and gradient clipping at 1.0.Source: §A.2 · p. 15

A.3 DYNAMIC KL IMPLEMENTATION

• Dynamic KL is evaluated over the union of teacher and student top-20 tokens plus a residual category, with teacher categories smoothed and student categories left unsmoothed.All student categories, including the residual, retain gradients.Source: §A.3 · p. 15

A.4 MOTIVATION DIAGNOSTICS •

• Among severe underpredictions at high-entropy prefixes, FKL, RKL, and Dynamic KL correct 99.7%, 70.9%, and 95.1%; for overpredictions, they correct 75.1%, 86.3%, and 84.7%.The counts are 1,913 underpredicted and 1,828 overpredicted candidates, respectively.Source: §A.4 · p. 16

A.5 BASELINE CONFIGURATIONS

• The three QAT baselines use 32,768 question–response pairs cut at 8,192 tokens; EfficientQAT and ReasoningQAT start from OnQ’s calibrated checkpoint, while BitDistiller uses its own clipping initialization and CAKLD.Baseline configurations use group size 128 and 2- or 3-bit weights.Source: §A.5 · p. 16, Table 6 · p. 16

A.6 EVALUATION

• The evaluation uses AIME 24–26’s 90 AIME I and II problems from 2024 to 2026 and LiveCodeBench’s release-v6 set of 1,055 problems; IFEval reports prompt-level strict accuracy.Decoding uses temperature 0.6, top-p 1.0, top-k 20, and one response per question.Source: §A.6 · p. 16

• Claude Sonnet 5.5 labels incorrect responses in two independent passes with model identities withheld; disagreements and doubtful cases receive author review.Source: B RESPONSE CLASSIFICATION · pp. 16–17
• The labeling passes agree on 93.6% of MATH-500 responses (κ = 0.87) and 95.9% of AIME responses (κ = 0.88).Source: B RESPONSE CLASSIFICATION · pp. 16–17
• “Looping” in Figure 6 denotes the paper’s repetitive-error category.Source: Figure 6 · p. 17

C TRUNCATION RATES OF THE MAIN RESULTS

• OnQ has the lowest average truncation rate in all six model-and-bit settings.Table 7 reports the percentage of responses terminated by the output-token limit; Avg. is the unweighted mean across nine benchmarks.Source: C TRUNCATION RATES OF THE MAIN RESULTS · p. 17

D TRUNCATION RATES ACROSS RECOVERY STAGES

• Calibration lowers average truncation at every model size and bit width, and on-policy distillation reduces it further in all six settings.Source: D TRUNCATION RATES ACROSS RECOVERY… · p. 17

E VARIATION ACROSS EVALUATION SEEDS

• Table 7 reports truncation rates for Table 1 across evaluation seeds 42, 43, and 44, with bold marking the lowest rate per model/bit group.Source: Table 7 · p. 18
• Table 8 reports truncation rates for Table 2, with bold marking the lowest rate per model/bit group.Source: Table 8 · p. 18
• Table 9 reports sample standard deviations for QAT and OnQ scores in Table 1 across evaluation seeds 42, 43, and 44.For Avg., the nine benchmark scores are first averaged within each seed, then the SD is computed across seeds using divisor n −1 with n = 3.Source: E VARIATION ACROSS EVALUATION SEEDS · p. 17, Table 9 · p. 19