PaperlayerView exact revision on arXiv

Source-linked AI summary

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

arXiv:2606.03458v1cs.LG

TL;DR

Long-horizon decoding makes KV-cache growth increasingly important, while quantization errors accumulate across timesteps because of incorrect token scales. KVarN combines Hadamard rotation with dual-scaling variance normalization and achieves state-of-the-art quality across generative benchmarks, including near loss-less 2.3bitper-element KV-Caches with 0.18% measured quantization latency overhead over baseline.

  • Problem

    Existing KV-cache quantization methods provide limited evidence under autoregressive long-horizon decoding, where quantization errors accumulate across layers and timesteps.

  • Method

    KVarN applies Hadamard rotation and variance normalization using scaling factors across both channel and token dimensions of K and V tensors.

  • Results

    KVarN achieves state-of-the-art quality on AIME24, MATH-500, HumanEval, and IF-Eval, enabling near loss-less 2.3bitper-element KV-Caches with 0.18% measured quantization latency overhead over baseline.

  • Takeaways & Limitations

    KVarN suppresses error accumulation in long decoding tasks while maintaining near-lossless compressed KV-cache quality across reasoning, coding, and instruction-following benchmarks.

  • Takeaways & Limitations

    The paper does not establish how KV-cache quantization quality is affected by models using multi-head latent attention.

Abstract

from arXiv · show

Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows. KV-cache quantization can help improve this, but current methods are evaluated under prefill-like settings and errors behave differently under autoregressive decoding. We show that in the latter regime, quantization errors accumulate across timesteps, driven primarily by incorrect token scales. We introduce KVarN, a calibration-free KV-cache quantizer that applies a Hadamard rotation followed by a dual-scaling variance normalization across both axes of the K and V matrices. We find that this combination fixes outlying token-scale errors and substantially reduces error accumulation over existing baselines. KVarN establishes a new state-of-theart for KV-cache quantization on generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision. A vLLM implementation of the KVarN method is available at https://github.com/huawei-csl/KVarN

1 Introduction

Test-time scaling increases decoding length, making KV-cache efficiency important, while autoregressive KV-cache quantization accumulates error because standard methods fail to preserve per-token scales. KVarN addresses these token-magnitude errors through variance normalization and mitigates error accumulation across timesteps, improving end-to-end generative performance.

  • Motivation: Longer test-time-scaling decodes make efficient KV-cache handling increasingly important for reasoning performance per time.Test-time scaling supports greater capability through ever-increasing decoding length.
  • Motivation: 2–4 bits per element can reduce KV-cache memory, but typical evaluations use fixed, long prefill contexts rather than on-the-fly autoregressive compression.The supplied passage identifies KIVI and TurboQuant as representative quantization methods evaluated under large prefill problems.
  • Method: Hadamard rotation alone fails to sufficiently address token scaling, motivating dual-scaling Sinkhorn-based variance normalization.The method discussion presents variance normalization as a way to further mitigate token-scaling errors.
  • Results: KVarN2 substantially improves end-to-end KV-cache quantization results over the current state of the art on AIME24, MATH500, HumanEval, and IFEval.The supplied contribution statement reports improvements on all four named generative benchmarks without providing numerical values.
  • Problem: Token magnitude errors are a key driver of outlier errors, which contribute disproportionately to end-to-end quality.The introduction identifies incorrect token scales as the cause of quantization-error accumulation across decoding timesteps.
  • Contribution: KVarN mitigates token-magnitude errors and error accumulation across decoding timesteps through a novel KV-cache compression method.The introduction names KVarN as the proposed method for addressing these errors in autoregressive decoding.

2 Preliminaries

The preliminaries define the KV-cache quantization setup, emphasizing per-head channel processing and token-wise chunking, with K quantization treated as more difficult than V. They motivate KVarN’s combination of Hadamard incoherence processing and dual-scaling variance normalization to address channel and token-wise scaling errors.

  • Quantization setup: K quantization is generally more difficult than V quantization, motivating greater emphasis on K.In Llama3.1-8B, more than 98% of the top 5% KIVI quantization errors lie in the K matrix.
  • Quantization setup: KV-cache quantization processes channels per head and tokens in chunks, forming head-dim × token-chunk tiles such as 128 × 128.Following KIVI, the setup quantizes V per token and K per channel using round-to-nearest quantization.
  • Preconditioning: Hadamard incoherence processing equalizes channel-space outliers but is insufficient alone to manage token-wise scaling errors.The Hadamard transform is suitable for online application with O(N log N) complexity and can often be absorbed into adjacent weight matrices.
  • Preconditioning: Dual-scaling variance normalization helps KV-cache quantization for reasons unrelated to approximating calibration data.The paper distinguishes this use from weight quantization, where variance-targeted normalization approximates calibration data from weight structure.
  • Central argument: The largest KV-cache errors, such as the top 5%, drive most end-to-end degradation, and KVarN addresses them through combined incoherence processing and dual-scaling.The paper identifies token-wise magnitude as the main driver of outlying errors and links their correction to reduced error accumulation over time.

3 Methods

KVarN targets autoregressive KV-cache error accumulation by separating magnitude and directional quantization errors and applying Hadamard rotation with dual-axis variance normalization. Its pseudo-decode evaluation shows that reducing token-scale errors suppresses reconstruction-error growth over longer contexts.

  • Error decomposition: Quantization error decomposes into a magnitude penalty and a directional penalty governed by norms and cosine similarity.This separates K-magnitude inflation from angular quantization noise when analyzing attention logits.
  • Error decomposition: Outlier errors are overwhelmingly caused by incorrect magnitudes.The fraction of total error due to magnitude is denoted E_M/E_T.
  • Error accumulation: During decoding, quantized-cache errors propagate across transformer blocks, making later unquantized K and V matrices already slightly wrong.Quantizing these affected matrices introduces additional error that influences subsequent representations.
  • Evaluation: KVarN achieves much lower reconstruction error than RTN (KIVI) and accumulates less error over time in the pseudo-decode setting.Incorrect token-scale multipliers can compound exponentially, making their reduction especially useful at longer contexts.
  • KVarN method: KVarN applies channel-wise Hadamard rotation and online variance scaling across both channel and token dimensions to mitigate token-scaling errors.Prior KV-cache methods scale only one dimension; the additional element-wise scaling adds one FLOP per token per channel of dequantization overhead.
  • KVarN method: Iterative row-wise and column-wise variance normalization avoids increased per-channel kurtosis while normalizing token scales and reducing tail errors from incorrect scaling.The method operates on signed matrices whose offsets make direct magnitude-based normalization unsuitable.

4 Experiments

The experiments evaluate KV-cache quantization across multiple model families and long-context tasks, emphasizing error accumulation in line retrieval, reasoning, coding, and instruction following. KVarN performs best overall on line retrieval while adding low normalization and dequantization overhead.

  • Evaluation setup: Evaluations cover Qwen3-4B, Llama-3.1-8B, and Phi-4-14B across line retrieval and instruction following, with reasoning models additionally tested on reasoning benchmarks.Qwen3-4B natively supports reasoning, Phi-4-14B has a reasoning variant, and Llama-3.1-8B does not.
  • Evaluation setup: The target tasks are MATH500, AIME24, HumanEval, and IF-Eval, covering long-horizon mathematical reasoning, coding, and instruction following.MATH500 and AIME24 require mathematical solutions and typically involve long-horizon chains of thought.
  • Line retrieval: KVarN performs best overall on line retrieval, which the authors find more informative than NiaH for assessing error accumulation over time.The paper reports comprehensive line-retrieval results across various baselines and models, while describing NiaH as comparatively easy when temporal error accumulation is ignored.
  • Efficiency: 1.9 ms is required for eight variance-normalization iterations per attention layer, compared with 1050 ms to generate 128 tokens, yielding a 0.18% measured overhead.The comparison uses Qwen3-4B through vLLM at fp16 on hardware with 500 TFLOP fp16 compute and 1.8 TB/s memory bandwidth.
  • Efficiency: About 1% dequantization slowdown is incurred by KVarN’s two-scale operation compared with RTN.The paper notes that prior methods can have larger dequantization overhead from code-book look-ups or mixed-precision handling.

5 Related Work

Prior KV-cache quantization methods exploit distinct key/value statistics, geometric rotations, or near-optimal vector quantization, while token eviction and merging reduce cache size by removing or combining entries. KVarN builds on KIVI by targeting token-scaling errors and arguing that end-to-end optimality differs from MSE optimality because outliers matter disproportionately.

  • KV-cache quantization: KV-cache quantization reduces memory by converting full-precision keys and values into low-bit representations while preserving model performance.KIVI treats keys and values differently, using per-channel key quantization and per-token value quantization.
  • KV-cache quantization: PolarQuant represents keys in polar coordinates, quantizing radius and angle separately while applying more aggressive compression to values.This approach is described alongside KIVI and KVQuant as a related KV-cache quantization method.
  • KV-cache quantization: TurboQuant applies a random rotation before scalar quantization and adds residual correction based on a 1-bit quantized Johnson–Lindenstrauss transform.The method frames KV-cache compression as a near-optimal vector quantization problem and aims to preserve inner products.
  • KVarN’s relationship to prior work: KVarN builds primarily on KIVI by addressing token scaling, finding that MSE-optimal quantization is not end-to-end optimal because outliers have disproportionate importance.The passage identifies the token scaling problem as the focus of KVarN’s contribution.
  • Token eviction and merging: Token eviction prunes KV entries, token merging combines representations, and these approaches are largely orthogonal to quantization and can be combined with it.Eviction methods estimate token importance using attention, while merging reduces tokens to a smaller informative set.

6 Conclusion · A Hadamard Transform Arrangement

The paper attributes KV-cache quantization degradation to incorrect token scaling and presents KVarN as a method that suppresses error accumulation in long decoding. Its Hadamard arrangement uses absorbed and head-wise online transforms with negligible O(N log N) complexity.

  • 6 Conclusion: KVarN addresses outlier errors caused by incorrect token scaling in current KV-cache quantization methods.The conclusion identifies incorrect token scaling as the driver of end-to-end output degradation.
  • 6 Conclusion: KVarN excels at suppressing error accumulation during long decoding tasks under KV-cache quantization.This directly summarizes the method’s stated advantage in the conclusion.
  • 6 Conclusion: KVarN achieves state-of-the-art quality on AIME24, MATH-500, HumanEval and IF-Eval.These generative benchmarks are explicitly listed in the conclusion.
  • 6 Conclusion: KVarN enables near loss-less 2.3bitper-element K.The supplied passage ends with this claim but is truncated after “K.”
  • A Hadamard Transform Arrangement: The Hadamard-transform arrangement resembles QuaRot and contains two absorbed transforms merged with WV and WO.These absorbed transforms do not need to be computed during inference.
  • A Hadamard Transform Arrangement: Two online transforms are applied after RoPE embedding and operate head-wise as block-diagonal Hadamard transforms with block size equal to the head size.The passage describes the transforms at the full-layer level.
  • A Hadamard Transform Arrangement: The online Hadamard transforms have O(N log N) complexity, described as negligible.The passage characterizes their computational cost during inference.

B Full Token-Scale Distribution Error · C Outlier Contributions to MSE

KVarN contains per-token K magnitudes near the diagonal by combining Hadamard rotation with variance normalization, whose effects are complementary across token scales. The largest 5% of errors contribute a minority of MSE but a majority of end-to-end KL-divergence, making outlier correction disproportionately important.

  • B Full Token-Scale Distribution Error: KVarN contains the per-token K-magnitude distribution close to the diagonal after quantization.This behavior is shown using the full empirical joint distribution before and after quantization.
  • B Full Token-Scale Distribution Error: Hadamard rotation and variance normalization produce different, complementary effects on token-scale distributions.The passage contrasts their behavior across the distribution rather than describing either component as sufficient alone.
  • B Full Token-Scale Distribution Error: Hadamard rotation squeezes the distribution close to the identity.Its behavior is described as strongest near the central portion of the token-scale distribution.
  • B Full Token-Scale Distribution Error: Variance normalization outperforms Hadamard rotation for very large and very small tokens.The passage specifically identifies the distribution tails as where Hadamard rotation performs much worse than variance normalization.
  • C Outlier Contributions to MSE: The largest 5% of errors cause a minority of the model’s MSE but a majority of its end-to-end KL-divergence.The comparison is made between quantile contributions to MSE and the corresponding end-to-end KL-divergence analysis.
  • C Outlier Contributions to MSE: Fixing outliers is disproportionately important because their end-to-end KL-divergence impact exceeds their share of total MSE.This conclusion follows from the quantile comparison reported for the largest errors.
  • C Outlier Contributions to MSE: The bottom 95% of entries contribute more to MSE than the top 5%.Figure 9 describes the residual MSE after replacing the largest top k% errors with high-precision values.

D Experimental details

Experiments evaluate four models across instruction-following, retrieval, long-context, mathematical reasoning, and code-generation tasks. All quantized runs use 2-bit KV-cache compression with fixed sink, quantized-body, and trailing-token regions, while baselines differ in precision and retained components.

  • Models: Four models span capability profiles and scales: Qwen3-4B, Llama-3.1-8B-Instruct, Phi-4, and Phi-4-reasoning-plus.Qwen3-4B covers all tasks, while the Phi models are assigned to specified subsets.
  • Quantization: 2-bit KV-cache compression uses FP16 sink tokens, grouped quantized body tokens, and FP16 trailing tokens.The classical configuration uses G=128, S=128, and R=128, except IFEval uses S=32.
  • Benchmarks: Evaluation covers IFEval, MATH-500, AIME 2024, HumanEval Extended, Line Retrieval, and Needle-in-a-Haystack tasks.Reasoning benchmarks use extended generation limits and report Avg@3; Line Retrieval reports exact-match accuracy over contexts of 100–600 lines.
  • Baselines: Baselines use distinct configurations: KVQuant retains 1% of K and V channels in FP16, PolarQuant requires 4-bit keys, and TurboQuant uses 3-bit K/3-bit V.TurboQuant also leaves the first and last two layers unquantized in the community vLLM implementation.
  • Effective memory overhead: 2.25 bits/element is the effective storage for KVarN, including quantized values, two FP8 scales, and one FP16 zero-point per G=128-element group.Reported bits-per-element include auxiliary storage such as scales and zero-points.

E Limitations

KVarN is limited to architectures that use KV-caches, and its quantization behavior with train-time KV-cache compression remains unclear. End-to-end serving evaluation is also hindered because public frameworks do not currently support 2-bit KV-caches.

  • Architecture compatibility: KVarN is unsuitable for architectures such as state-space models that do not require a KV-cache.The method targets KV-cache quantization and therefore does not apply to architectures without a KV-cache.
  • Compressed KV-caches: The effect of multi-head latent attention’s train-time KV-cache compression on quantization quality remains unclear.Recent models use multi-head latent attention for KV-cache compression, creating an unresolved interaction with quantization.
  • Serving evaluation: Public serving frameworks currently lack support for 2-bit KV-caches, hindering end-to-end evaluation.This limitation prevents straightforward evaluation of the method within an available serving framework.

F Comparison with Eviction Methods … I KVarN Dequantization Overhead

The appendix compares KVarN with prompt-only KV-cache eviction and evaluates retrieval under accumulated pseudo-decoding, then describes variance normalization and shows that fused dual-scale dequantization adds negligible overhead.

  • F Comparison with Eviction Methods: Eviction methods compress only the prompt cache and leave the generation cache untouched, making them orthogonal to uniform quantization across prefill and generation.The comparison includes SnapKV, PyramidKV, and KVZip on Line Retrieval with Llama-3.1-8B-Instruct.
  • F Comparison with Eviction Methods: 2.3 bits per element on average is matched by applying a 7× compression ratio to the prompt cache.This setting is used to match KVarN’s effective memory footprint.
  • G NiaH: Needle-in-a-Haystack: The accumulated pseudo-decode setting is essential for Needle-in-a-Haystack retrieval because static prefill has no error accumulation.The static version commonly used for KV-cache compression evaluation is described as much easier than accumulated pseudo-decoding.
  • H Variance Normalization Algorithm: KVarN performs variance normalization on both K and V matrices using an adaptation of Algorithm 1.The normalization procedure is adapted from prior work.
  • I KVarN Dequantization Overhead: KVarN’s dual scaling adds a per-row second scale s2, which is fused into the dequantization kernel to avoid an extra HBM round-trip.The overhead measurement uses 16 attention heads, head dimension 128, group size 128, and context lengths from 4k to 32k tokens.
  • I KVarN Dequantization Overhead: At most 1.4% separates KVarN and KIVI dequantization across all measured context lengths.The reported values are median time per call from repeated Triton GPU runs.
  • I KVarN Dequantization Overhead: KVarN’s dual-scale dequantization remains within measurement noise at 16k and 32k tokens.This supports negligible runtime overhead relative to the single-scale KIVI baseline when s2 is fused into the kernel.

J Computational Cost of Experiments

The experiments on MATH500, AIME24, HumanEval, and IFEval account for the dominant share of compute cost, totaling circa 50 GPU days under the stated hardware assumptions.

  • Compute cost: The experiments on MATH500, AIME24, HumanEval, and IFEval constitute the dominant share of compute cost.These are the experiments identified as driving most of the reported compute expenditure.
  • Compute cost: circa 50 GPU days are required to reproduce these experiments.The estimate applies to a GPU with 500 TFLOP at fp16 and 1.8 TB/s memory bandwidth.
  • Compute cost: The reproduction estimate assumes a GPU with 500 TFLOP at fp16 and 1.8 TB/s memory bandwidth.These hardware specifications define the conditions for the circa 50 GPU-day estimate.
Loading 2606.03458v1…