Source-linked AI summary
RECAP: Calibrating Low-Bit Reasoning Models on the History They Read
Chi Ruan, Yujie Chen, Zixiang Zhao, Jingcai Guo, Haotong Qin
TL;DR
Low-bit quantization can degrade long-chain reasoning, while standard calibration may not preserve the historical dependencies later steps use. RECAP calibrates on regenerated quantized prefixes with rotating reconstruction blocks and attention-aware cache penalties. It consistently improves on the baseline across models, benchmarks, and quantization settings.
Problem
Short calibration segments can omit earlier context needed by later reasoning steps, and cache reconstruction error alone does not capture how historical errors affect subsequent attention.
Source: §2.2 · p. 4, §3.3 · p. 6Method
RECAP regenerates quantized historical prefixes for rotating short-block reconstruction and adds a read-geometry penalty for historical key and value errors.
Source: §1 · p. 2Results
Takeaways & Limitations
The results highlight the importance of preserving historical dependencies during calibration for low-bit reasoning.
Source: §1 · p. 2Performance on standard benchmarks does not guarantee reliability across all use cases, and sensitive deployments require additional assessment and appropriate human oversight.
Source: ETHICS STATEMENT · p. 10
Abstract
from arXiv · showhide
Low-bit quantization reduces the cost of serving reasoning models but can substantially degrade long chains of thought. Each step reads cached keys and values spanning thousands of tokens, allowing quantization errors to accumulate along the trajectory. Standard calibration operates on short segments that omit the earlier context needed by later reasoning steps. Even when this context is retained, cache reconstruction error alone does not capture how quantization errors affect subsequent attention. We present RECAP, a post-training quantization method that calibrates reasoning models on the history they read. Dynamic-prefix calibration separates long historical context from short-block reconstruction. It regenerates prefix keys and values with the current quantizer and rotates the active block to supervise every position along the trajectory. Prefix read geometry further supervises historical cache errors according to their effects on attention outputs, weighting key and value errors asymmetrically. With both components applied only during calibration, RECAP outperforms the baseline across multiple reasoning models and quantization settings. Under W4A4KV4, RECAP improves accuracy over the baseline by 13.3 percentage points on MATH500 with DeepSeek-R1-Distill-Qwen-1.5B and 15.8 points on AIME-120 with Qwen3-4B. These results highlight the importance of preserving historical dependencies during calibration for low-bit reasoning.
1 INTRODUCTION
Low-bit quantization can undermine long-chain reasoning because later steps repeatedly read a growing cache of earlier states. RECAP calibrates on that history and reports gains over the baseline across reasoning models and quantization settings.
1 INTRODUCTION
- Long-chain reasoning repeatedly accesses growing histories, and quantization errors in cached keys and values can distort later attention, reduce accuracy, and increase generation length.The introduction motivates calibration that preserves historical dependencies.Source: §1 · p. 1
- Calibration must address history and position coverage as well as unequal effects of similarly sized cache errors on later attention.The introduction identifies these as challenges alongside the difficulty of capturing historical dependencies.Source: §1 · p. 2
- RECAP combines dynamic-prefix calibration, which rotates short reconstruction blocks over a regenerated quantized history, with read geometry that weights historical key and value errors by their attention effects.Both components are applied during calibration.Source: §1 · p. 2
- Up to 13.3 percentage points on MATH500 and 15.8 points on AIME-120: RECAP improves accuracy over the baseline under W4A4KV4.The method is evaluated across multiple reasoning models and quantization settings.Source: §1 · p. 2
2 MOTIVATION
The motivation section frames calibration as a trade-off among access to historical context, position coverage, and computational cost. It also explains why selecting cache states by importance or minimizing raw cache error alone is insufficient.
2 MOTIVATION
- The motivation overview identifies limited transfer of importance scores, incomplete history and position coverage, and mismatch between cache error and attention distortion as three calibration challenges.Source: §2 · p. 3
2.1 PRELIMINARIES
- Low-bit quantization uses reduced precision for weights, activations, and cached keys and values; during autoregressive reasoning, earlier cached K/V can influence later steps when read.The section introduces reconstruction-based post-training quantization and causal attention as preliminaries.Source: §2.1 · p. 3
2.2 QUANTIZATION SENSITIVITY AND CALIBRATION COVERAGE
- Importance scores fitted on three reasoning trajectories rank the most beneficial 10% of cache repairs near chance on a held-out trajectory, with AUC between 0.43 and 0.50.The analysis measures how restoring individual quantized cache units to full precision reduces teacher-forced final-answer loss across four trajectories.Source: §2.2 · pp. 3–4
- Short windows omit earlier premises, while reconstructing every position costs more per update; a long prefix with only the final segment reconstructed leaves earlier query positions unsupervised.In an 8K teacher-forced diagnostic, 4K-window calibration diverges late, while reconstructing positions 4K–8K with a 4K prefix fails in the opposite region.Source: §2.2 · p. 4
- Equal-sized cache errors can affect attention differently because the current query assigns unequal weights to cached history, making raw cache error incomplete as a measure of impact.The example relies more heavily on the reasoning step r2 = 9 than on the connecting word “so”.Source: §2.2 · p. 4
- The section presents three calibration requirements: retain historical context, supervise positions across the trajectory without full-sequence reconstruction at every update, and account for effects on later attention outputs.Source: §2.2 · p. 4
3 RECAP
RECAP retains historical cache context while reconstructing short blocks, then adds read geometry to directly weight historical K/V errors by their effects on later attention.
3.1 CALIBRATION FORMULATION
- Calibration uses full-precision generated trajectories, optimizes FlatQuant transformations, scales, and clipping with pretrained weights fixed, and minimizes teacher-forced layer-output MSE.The full-precision layer inputs and reference outputs remain fixed during optimization.Source: §3.1 · p. 5
3.2 DYNAMIC PREFIXES
- Dynamic prefixes regenerate historical K/V with current quantization parameters, while queries attend to the prefix and causal K/V in the block and reconstruct only active-block outputs.Gradients flow through prefix K/V without computing their query, attention, or feed-forward outputs.Source: §3.2 · p. 5
- Rotating the active block supervises every trajectory position; under matched transformations, quantization rules, positions, token-wise statistics, and causal support, block losses estimate the full-sequence objective.Parameter updates between blocks mean a full rotation is not identical to full-sequence optimization.Source: §3.2 · pp. 5–6
3.3 PREFIX READ GEOMETRY
- Read geometry adds a direct historical K/V penalty to active-block reconstruction, using fixed FP-target statistics and attention-output sensitivities that treat key and value errors differently.Key errors act through query alignment and value contrast; value errors enter directly, scaled by attention weight.Source: §3.3 · p. 6
- The read-geometry metric describes local head-output distortion before the output projection and omits cross-position and K–V cross terms.For grouped-query attention, it sums over query heads sharing a KV head.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments compare RECAP with quantization baselines across reasoning models, benchmarks, and bit settings, then examine prefix length, read geometry, and generation behavior. RECAP consistently improves on FlatQuant, with especially large gains on some long-horizon tasks.
4.1 SETUP
- The evaluation covers three reasoning models, five benchmarks, and W4A4KV4 and W4A8KV4, with task scores and selected generation-length measurements.Calibration uses target-model FP trajectories; the default uses a 12K prefix and rotating 4K block.Source: §4.1 · p. 7
4.2 MAIN RESULTS
- Across all six model–precision combinations, RECAP achieves the highest average score among quantized methods, tying only on R1-7B GSM8K under W4A4KV4.Under W4A4KV4, average-score gains over FlatQuant are 6.3, 4.8, and 6.1 points for R1-1.5B, R1-7B, and Qwen3-4B, respectively.Source: §4.2 · p. 7, Table 1 · p. 8
- Under W4A4KV4, AIME-120 gains are 9.7, 8.6, and 15.8 points across the three models, while LiveCodeBench gains are 4.6, 6.4, and 8.3 points.Gains on GSM8K and GPQA-Diamond never exceed 3.1 and 2.1 points, respectively, in either setting; the table does not isolate the historical-K/V mechanism.Source: §4.2 · p. 7
- RECAP’s largest average gains occur where FlatQuant degrades most: R1-1.5B gains 6.3 and 6.8 points, versus 1.1 points for Qwen3-4B under W4A8KV4.QuaRot averages at most 5.3% on both R1 models but remains competitive on Qwen3-4B.Source: §4.2 · p. 8
4.3 ABLATION STUDIES
- With a 4K active block, the 12K-prefix configuration improves MATH500 and AIME-120 accuracy by 1.2 and 1.6 points and reduces mean generation length by 14.3% and 10.8%.Differences in calibration cohorts and update budgets limit attribution to prefix length alone.Source: §4.3 · p. 8
- Across both prefix lengths, read geometry improves accuracy by 1.0–1.6 points on MATH500 and 2.3–3.3 points on AIME-120, with 4.8–11.8% fewer generated tokens.Read geometry weights historical K/V errors by their effects on subsequent attention outputs; the point estimates do not establish statistical superiority.Source: §4.3 · p. 8
- Of 101 MATH500 questions changing from correct to incorrect, 96 produce longer FlatQuant outputs than BF16, often near the generation limit.The reported association does not attribute errors to length alone.Source: §4.3 · p. 8
5 RELATED WORK
The related work reviews post-training quantization, KV-cache quantization, and calibration for reasoning models. RECAP extends reconstruction-based calibration along the sequence dimension to retain historical context while reconstructing short blocks.
Post-training quantization of LLMs
- Reconstruction-based methods such as OmniQuant and FlatQuant improve calibration through learned transformations, clipping, or layer-output reconstruction; RECAP extends this approach across the sequence dimension.Other post-training quantization approaches use sensitivity-aware quantization, activation-aware scaling, or rotations to suppress outliers.Source: §5 · p. 9
KV cache quantization
- KV-cache quantization methods address memory, attention quality, and decoding overhead through key/value asymmetry, sensitivity weighting, outlier handling, rotations, or vector codebooks.The passage also describes OSCAR’s use of attention-aware covariance statistics for rotations and clipping thresholds.Source: §5 · p. 9
Quantization and calibration for reasoning models
- Reasoning-model quantization work considers varying sensitivity across models and tasks, while calibration research examines data selection, sequence length, self-calibration, and reasoning traces.RECAP is situated alongside methods that use target-model reasoning traces and final answers during calibration.Source: §5 · p. 9
6 CONCLUSION
The conclusion presents RECAP as a post-training method that calibrates low-bit reasoning models on the history they read. It reports consistent baseline improvements, largest on long-horizon reasoning, while noting variation in the remaining gap to full precision.
6 CONCLUSION
- RECAP combines dynamic-prefix calibration, which brings long quantized history into short-block reconstruction, with prefix read geometry that supervises historical cache errors by attention effects.Both components are applied only during calibration.Source: §6 · p. 9
- Across multiple models, benchmarks, and quantization settings, RECAP consistently improves on the baseline, with the largest gains on long-horizon reasoning.The remaining gap to full precision varies across models.Source: §6 · p. 9
- Benchmark performance does not guarantee reliability across all use cases; sensitive deployments require additional assessment and appropriate human oversight.The paper also notes potential biases, inaccurate outputs, and misuse when applying language models.Source: ETHICS STATEMENT · p. 10
A IMPLEMENTATION AND CALIBRATION DETAILS
The implementation details specify calibration data and budgets, query sampling, cache-geometry estimation, and computation costs. Prefix K/V are regenerated with current quantization parameters, while reconstruction focuses on the active block.
A IMPLEMENTATION AND CALIBRATION DETAILS
- The R1-1.5B calibration pool contains 1,000 NuminaMath-1.5 FP trajectories, filtered for malformed responses and exact-match overlaps; fit cohorts retain leading segments of eligible trajectories.The 8K- and 12K-prefix budget cohorts differ, so the comparison also changes data exposure.Source: A IMPLEMENTATION AND CALIBRATION DETAILS · p. 13
- Both calibration budgets use a 4K active block, optimizer batch 4, and microbatch 1; each position is reconstructed five times at 8K prefix and eight times at 12K.These are the two configurations summarized in Table 3.Source: Table 3 · p. 13
- At each microstep, prefix K/V are regenerated from fixed FP layer inputs using current quantization parameters, with gradients retained through prefix construction.The default recipe collects geometry from the original FP model and preserves causal order and original position IDs.Source: A IMPLEMENTATION AND CALIBRATION DETAILS · p. 13
- Eligible active-block queries are divided into four position strata, with up to 16 sampled without replacement per stratum and inverse-inclusion weights used to estimate the mean.Selections remain fixed across layers and GQA heads.Source: A IMPLEMENTATION AND CALIBRATION DETAILS · p. 13
- The shared-key-direction approximation pools positive-trace blocks by position bucket and KV head, retaining the leading eight eigenvectors and an isotropic residual.The approximation preserves pooled trace; values retain scalar read strengths.Source: A IMPLEMENTATION AND CALIBRATION DETAILS · p. 13
- With T = 4B, active-block computation uses about a quarter of the full-sequence query–key pairs on average across four blocks, while feed-forward outputs are computed only for active positions.Prefix K/V are still projected each update, and read-geometry statistics are collected once per model before calibration.Source: A IMPLEMENTATION AND CALIBRATION DETAILS · p. 13
B EVIDENCE FOR THE MOTIVATION
Tested cache-importance scores transferred poorly across held-out trajectories, while calibration diagnostics show that errors vary by sequence region under short-window and fixedtail settings.
B EVIDENCE FOR THE MOTIVATION
- None of the five tested importance scores exceeds AUC 0.51 on held-out trajectories.The results come from four discovery trajectories using an earlier V-codeword configuration, not a matched W4A4KV4 comparison.Source: B EVIDENCE FOR THE MOTIVATION · p. 14
- Table 4 averages importance-score transfer over four held-out trajectories and reports AP and P@10%; future-read count is an unavailable-at-calibration reference.AP is average precision; P@10% is precision among the selected 10%.Source: Table 4 · p. 14
- Short-window and fixedtail calibration fail in opposite sequence regions, while full 8K calibration stays within 0.140–0.190 nats/token across buckets.The diagnostic averages FP-to-quantized KL within 1K position buckets over eight development trajectories.Source: B EVIDENCE FOR THE MOTIVATION · p. 14
- Rotating-prefix calibration over a 12K context stays within 0.101–0.191 nats/token across regions, but uses a different cohort and does not measure task accuracy.It is evaluated on four development trajectories, so its magnitudes cannot be directly compared with the 8K diagnostic.Source: B EVIDENCE FOR THE MOTIVATION · p. 14