PaperlayerView exact revision on arXiv

Source-linked AI summary

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu

arXiv:2605.17757v1cs.LGcs.AIcs.DCcs.PF

TL;DR

Long-context LLM serving needs KV-cache compression that remains accurate at INT2, where generic rotations are not aligned with attention. OSCAR uses offline attention-aware covariance calibration to derive fixed rotations and clipping, retaining near-BF16 accuracy across models and long contexts while supporting serving-compatible deployment.

  • Problem

    INT2 KV-cache quantization offers memory savings, but existing data-oblivious rotations remain poorly aligned with the attention directions that determine ultra-low-bit accuracy.

  • Method

    OSCAR estimates attention-aware covariance offline, derives layer-wise key/value rotations and clipping thresholds, and applies fixed transforms during serving.

  • Results

    2.28 BPE keeps OSCAR near the BF16 accuracy frontier across evaluated models, with slower long-context degradation than rotation-only INT2 baselines.

  • Takeaways & Limitations

    OSCAR provides a near-2-bit KV-cache approach that remains compatible with paged-attention serving and preserves downstream attention computation better than alternative rotation targets.

  • Takeaways & Limitations

    The reported LiveCodeBench results use a 32K generation cap, truncating some long outputs and making the scores lower than under a longer 128K budget.

Abstract

from arXiv · show

INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms reduce outliers, but still degrade at INT2 because they are not aligned with downstream attention. We propose OSCAR, an Ultra-low-bit KV Cache quantization method that estimates attention-aware covariance structures offline and uses them to derive fixed rotations and clipping thresholds for quantization. In this way, it aligns KV quantization with the covariance structures that attention actually consumes. More importantly, we not only provide theoretical justification but also develop a fully deployable OSCAR system with a custom INT2 attention kernel that remains compatible with paged KV-cache serving and fused kernel pipelines, enabling seamless integration into modern LLM serving frameworks such as SGLang and vLLM. We evaluate our methods on recent reasoning models with reasoning traces of up to 32k tokens across 5 tasks. On Qwen3-4B-Thinking-2507 and Qwen3-8B, OSCAR reduces the BF16 accuracy gap to 3.78 and 1.42 points, respectively, while naive rotation INT2 collapses to nearly zero. We further scale OSCAR to Qwen3-32B and GLM-4.7 (358B params), where it remains effectively on par with BF16. On long context - RULER-NIAH up to 128K, OSCAR remains robust on both Qwen3 models, while naive rotation INT2 collapses. System-wise, OSCAR reduces KV-cache memory by approximately 8x, improves throughput by up to 7x at large batch sizes under the same memory budget, and accelerates batch-size-1 decoding by up to 3x over BF16 due to reduced memory bandwidth overhead.

1 Introduction

Long-context serving makes KV-cache memory traffic a central bottleneck, motivating hardware-friendly INT2 compression despite severe outlier-driven accuracy loss. OSCAR addresses this by deriving attention-aware covariance rotations that preserve downstream attention computation rather than merely smoothing raw activations.

  • Motivation: INT2 KV-cache quantization promises large memory reduction with a fixed-width representation, directly targeting the cache’s growth with context length, batch size, and model depth.Each decoded token must read a large fraction of the growing cache from GPU memory, so compression can increase batch size and reduce memory traffic.
  • Challenge: INT2 accuracy suffers because channel-wise outliers dominate quantization scales, forcing most normal KV values into only a few effective levels.Generic rotations can smooth activation ranges, but random rotations remain data-oblivious and do not identify directions that attention reads most strongly.
  • Design Principle: The proposed target follows attention’s correlation and score-weighted interactions, so INT2 error is pushed toward directions the model reads less strongly.This alignment is especially important because only four quantization levels are available at INT2.
  • OSCAR: OSCAR uses lightweight calibration to derive attention-aware rotations for keys and values, targeting errors in attention scores and layer outputs rather than raw cache reconstruction.The framework is theoretically analyzed through covariance-target rotations that are optimal under a natural frozen-error surrogate.

2 Preliminaries and Motivation

The section formalizes attention, KV-cache storage, and uniform quantization, then motivates OSCAR by showing that attention-aware covariance better targets downstream distortion than raw-cache reconstruction. Figure 2 shows that full OSCAR reduces errors across cache, attention-score, output, and propagated-hidden-state stages.

  • Attention and KV cache: The section defines single-head attention [19] with Q, K, and V, where row-wise scores are softmax(QKᵀ/√d) and outputs are weighted sums of value rows.During autoregressive inference, the prefixes K1:t and V1:t are stored as the KV cache.
  • Quantization notation: A symmetric uniform b-bit quantizer maps values through a scale, rounding, clipping limit, and dequantization over 2^b representable codes.The theory applies quantization element-wise to matrix rows, while experiments quantize the head dimension with block sizes 128, 64, or 32.
  • Why raw-cache reconstruction is not enough: Attention consumes keys through logits and values through weighted aggregation, so minimizing raw-cache Euclidean reconstruction error does not directly minimize downstream attention distortion.Key distortion is governed by query covariance, while value distortion depends on attention weights over value rows.
  • Why raw-cache reconstruction is not enough: Full OSCAR reduces quantization error across cache, attention-score, attention-output, and propagated-hidden-state stages, outperforming naive INT2, Hadamard-only, and clip-only variants in Figure 2.The comparison uses Qwen3-4B-Thinking-2507 on AIME and reports four error views, including relative MSE and KL divergence.

3 Algorithm Design: Offline Calibration and Justification

OSCAR calibrates fixed, attention-aware rotations and clipping thresholds offline, then applies them online through a mixed-precision cache layout. Its spectral rotations are theoretically optimal for the frozen-error surrogate under diagonal ambient-basis residual covariances.

  • Offline Covariance Calibration: OSCAR estimates attention-aware covariance matrices offline from calibration activations, eigendecomposes them, and uses their eigenvectors as layer- and head-specific base rotations.For keys, the query-aware target covariance yields Rk := UQ; for values, the heuristic target covariance yields Rv := US.
  • Scale Determination and Clipping: OSCAR fits per-token clipping thresholds offline and uses affine asymmetric INT2 quantization with scales and zero points for keys and values.Percentile-based clipping controls outliers in the quantization backend.
  • Online Deployment: During serving, the online phase applies the fixed offline transforms through a mixed-precision cache layout.All covariance and rotation estimates come from a small calibration dataset.
  • OSCAR Rotations: The final key and value rotations compose covariance-based rotations with a Hadamard transform and bit-reversal permutation to redistribute energy and balance adjacent-channel dynamic ranges.The Hadamard transform improves quantization geometry, while bit reversal interleaves large- and small-variance channels.
  • Optimality Justification: Under diagonal ambient-basis residual covariances, Rk = UQ and Rv = US minimize the corresponding frozen-error surrogate objectives on the calibration dataset.The theorem establishes optimality for the simple spectral variants under the stated residual assumption.

4 System Design: Online Serving with 2-bit KV Cache

OSCAR integrates an INT2 KV-cache mode into SGLang while preserving paged attention and high-precision sink and recent-token windows. Fused kernels update, demote, unpack, and attend over mixed BF16/INT2 cache segments, with value-rotation absorption reducing computation and latency.

  • KV Cache Layout: OSCAR integrates INT2 KV caching into SGLang [18] with full paged-attention compatibility [13], retaining BF16 sink and recent-token windows while storing middle context after fixed rotation.The cache preserves the first S0 sink tokens and most recent W tokens; the remaining middle context uses INT2 storage.
  • KV Cache Update: During prefill, a fully fused Triton kernel applies clipping and quantization, storing four 2-bit values per byte.The runtime writes BF16 key and value rows using clip values before packing the resulting INT2 representations.
  • KV Cache Update: As decoding advances, new tokens enter the BF16 recent window before a fused Triton kernel demotes the oldest token into the INT2 middle region.The same clip–quantize operation is reused during demotion, while absorbing the value rotation into projection weights saves computation and reduces latency.
  • Decoding Attention Kernel: During decoding, the attention kernel partitions cache indices into BF16 and INT2 segments, then unpacks INT2 bytes and accumulates the restored values in floating point.This supports mixed-precision attention over sink, recent, and middle-cache regions; existing decoding attention kernels use two launches to process and merge sequence segments.

5 Experiments

OSCAR is evaluated across four model configurations on five reasoning and coding benchmarks, plus RULER-NIAH long-context retrieval. It remains close to the BF16 accuracy frontier at near-2-bit precision while supporting serving-oriented evaluation and deployment.

  • Main Accuracy Comparison: 3.78 and 1.42 points are OSCAR’s BF16 accuracy gaps on Qwen3-4B-Thinking-2507 and Qwen3-8B, respectively, making it the only near-2-bit method close to the BF16 frontier under 32K generation.The comparison covers four model configurations and five benchmarks, with Table 2 reporting the main accuracy results.
  • Generation Protocol & Calibration: OSCAR uses one small calibration pass per model to estimate reusable per-layer rotations and clipping thresholds, with no task-specific calibration.Each calibration pass dumps per-layer Q, K, and V activations from 8878 tokens, then reuses the resulting parameters across benchmarks.
  • Long-Context Robustness: OSCAR degrades more slowly than rotation-only INT2 baselines on RULER-NIAH as context length increases from 4k to 128k tokens.The evaluation uses serving-compatible baselines and tests whether long-context attention remains reliable as accumulated attention-logit error grows.
  • Rotation Analysis: Attention-aware eigenbasis and Hadamard components both contribute substantially, while bit reversal improves quantization geometry and alternative rotation targets fail to match attention-aware targets.The composed rotation is R = U · HHad · Pbr; bit reversal does not change floating-point accuracy but interleaves eigenvalues for more uniform per-group ranges.
  • Ablations and System Performance: A protection window of (S, R)=(64, 256) is the accuracy-memory sweet spot, while OSCAR consistently outperforms BF16 across tested batch sizes and reaches 2.83× speedup on GLM-4.7-FP8.Larger protection windows add negligible accuracy but substantially increase BF16 KV memory; throughput is measured under long-context decoding with prefix warm-up.

B BF16 OSCAR GEMM Attn Quant Other Total GEMM Attn Quant Other Total

OSCAR delivers strong serving efficiency under concurrent long-context workloads by combining an approximately 8× smaller KV cache with accuracy-preserving INT2 execution. It scales to larger batches and remains effective across prefix-cache regimes while integrating with paged and fused serving pipelines.

  • Serving throughput: 7.83× throughput over BF16 at BS=32, with OSCAR outperforming Saw-INT4 across models and batch sizes.The advantage is already observed at BS=1 and grows with concurrency, reflecting the benefit of OSCAR’s reduced KV-cache footprint.
  • Long-context stress test: OSCAR’s uniform INT2 KV-cache scales to 28 requests with continued throughput gains in the 100k-token stress test.The stress test evaluates long-context serving on a single H100 while scaling concurrent requests.
  • Prefix-cache serving: OSCAR stays on or near the throughput-efficiency frontier as prefix-cache hit ratios increase, closely matching aggressive INT2 baselines without extra decode-time memory traffic or indirection.Higher prefix-cache hit ratios expand the throughput frontier by reducing prefill recomputation, especially at larger batch sizes.
  • Deployment and accuracy: OSCAR achieves the best accuracy among INT2 methods while preserving paged KV-cache compatibility and integrating cleanly into fused SGLang decode kernels.QuaRot-INT2 can approach the throughput ceiling but suffers significant accuracy degradation.

6 Conclusion

OSCAR uses attention-aware covariance to derive fixed INT2 KV-cache rotations and clipping thresholds through offline calibration, while its fused serving path preserves key KV tokens and retains 2-bit memory and serving benefits.

  • 6 Conclusion: OSCAR derives fixed key/value rotations and clipping thresholds from attention-aware target covariance in one offline calibration pass, rather than raw cache reconstruction.Its fused SGLang path quantizes historical KV tokens while preserving sink and recent tokens, recovering much of naive INT2’s BF16 accuracy loss while retaining 2-bit memory and serving advantages.

A Additional Details and Theoretical Analysis · A.1 Hadamard Transform · A.2 Principal Component Analysis

This section formalizes the fixed Hadamard post-rotation used in OSCAR and explains how PCA supplies an orthogonal basis aligned with attention-induced covariance directions for KV compression.

  • A.1 Hadamard Transform: The normalized Walsh-Hadamard matrix is orthogonal, with equal-magnitude entries and an identity product with its transpose.Its dimension is inferred from the transformed key or value vector, and power-of-two dimensions admit a recursive definition.
  • A.1 Hadamard Transform: OSCAR composes a fixed Walsh-Hadamard transform after the attention-aware covariance rotation to smooth rotated coordinates while preserving efficient deployment.Hadamard transforms spread outlier energy across channels before rounding in low-bit quantization [9].
  • A.2 Principal Component Analysis: PCA diagonalizes a symmetric positive semidefinite covariance matrix as A = VΛV⊤, with eigenvalues ordered nonincreasingly and nonnegative.The decomposition provides the eigenvector directions and their associated variances.
  • A.2 Principal Component Analysis: The PCA top-r proposition identifies the subspace spanned by the r largest eigenvectors as the maximizing rank-r projection.The proof uses orthonormal projected columns and assigns full weight to the largest r eigenvalues.
  • A.2 Principal Component Analysis: When the matrix is an attention-induced target covariance, PCA yields an orthogonal basis aligned with directions most important to preserve during KV compression.This alignment motivates using PCA-derived directions in OSCAR’s covariance-aware rotations.

A.3 Target Covariance · A.5 Scale Determination and Per-token Clipping Details

OSCAR derives rotations from attention-aware covariance targets—Q^TQ for keys and V^TS^TSV for values—rather than raw-cache covariance, then combines eigenspace alignment, Hadamard mixing, and bit-reversal grouping to improve low-bit quantization. These factors equalize attention importance, suppress outliers, and balance per-group quantization structure, with empirical gains appearing most clearly in attention-consumed errors.

  • A.3 Target Covariance: OSCAR uses Q^TQ for key rotations and V^TS^TSV for value rotations because these attention-induced covariances target downstream logit and output distortion rather than raw-cache reconstruction.Q^TQ weights key errors by query energy, while V^TS^TSV captures value directions remaining large after attention aggregation.
  • A.3 Target Covariance: OSCAR’s UQ, HHad, and PK factors separately align importance directions, equalize channel importance and suppress outliers, and balance importance across contiguous quantization groups.The three factors address distinct failure modes of per-group low-bit quantization and are composed in that order.
  • A.3 Target Covariance: The key and cache covariance eigenbases are nearly uncorrelated on Qwen3-8B, with top-8 self-alignment 0.05–0.15 across 36 layers versus random alignment approximately 0.09.For example, diagonal energy changes from 0.90 to 0.09 in layer 1 and from 0.16 to 0.07 in layer 16 after UQ rotation, supporting separate Q-side and K-side objectives.
  • A.3 Target Covariance: OSCAR’s advantage appears after attention: QK^T and SV errors are substantially lower even though raw K- and V-reconstruction MSE are not always dramatically smaller than rotation-only baselines.The method targets quantities consumed by attention rather than plain Euclidean reconstruction of cached tensors.
  • A.5 Scale Determination and Per-token Clipping Details: The deployment quantizer uses per-token, per-group INT2 min–max scales, making channel distributions and contiguous group composition central to rotation design.The worked example uses GK = 64, while GK = d removes PBR’s effect because the permutation stays within one group.
  • A.3 Target Covariance: 46.9× peak key-importance concentration becomes exactly 1.00× after UQ followed by Hadamard equalization, while the transformed entries occupy a tight [−6.79, 7.03] band.Hadamard mixing reduces the absolute range from 37.85 to 13.58, a 2.8× reduction, while equalizing the importance metric.
  • A.3 Target Covariance: UQ alone can worsen per-group INT2 quantization: group ranges change from 44.81 and 7.19 to 7.16 and 37.85, respectively.This motivates Hadamard mixing and group-balancing permutation after importance alignment rather than using the eigenbasis alone.
  • A.3 Target Covariance: 1.23× lower tr(EK) results from PBR reducing mean per-group max–min from 13.40 to 11.58 over 8000 tokens at GK = 64.At this setting, PBR carries most of OSCAR’s residual improvement over UQHHad; when GK = d, PK is a no-op.

A.6 Proof of Theorem. 1 … D Discussion

OSCAR’s theory derives covariance-target rotations by minimizing frozen-error surrogates, while its calibration and serving procedure operationalize fixed rotations and clipping for INT2 KV-cache quantization. The discussion positions this attention-aware approach against prior methods, notes theoretical and scope limitations, and highlights accessibility benefits alongside misuse risks.

  • A.6 Proof of Theorem. 1: Theorem 1 proves that pairing descending target-covariance eigenvalues with ascending frozen residual-covariance eigenvalues minimizes the key surrogate, with an analogous result for values.The proof uses doubly stochastic matrices, the Birkhoff polytope, and the rearrangement inequality; the frozen residual covariances are treated as fixed.
  • A.7 Justification of Surrogate Objectives in Theorem. 1: The surrogate objectives approximate position-dependent attention-error matrices with global query or value target covariances, yielding tractable frozen-error objectives for keys and values.The key target is query-covariance weighted, whereas the value target is attention-covariance aware, motivating the shared surrogate form.
  • B Algorithm Flow and Serving Procedure: Algorithm 1 implements OSCAR by calibrating per-layer, per-head rotations and clipping thresholds, rotating before cache writes, and maintaining BF16 sink/recent, rotated staging, and packed INT2 history caches.The procedure is identified as the concrete method used in experiments and the SGLang implementation.
  • C Related Work: Existing rotation-based KV methods can fail at extreme INT2 levels because their rotations are not aligned with downstream attention covariance and may lack deployable kernels or serving integration.The related-work discussion contrasts OSCAR’s attention alignment and deployment focus with these limitations.
  • C Related Work: Unlike prior covariance- or calibration-based compression methods, OSCAR estimates covariance induced by downstream attention rather than raw-cache statistics to derive fixed key/value rotations and clipping thresholds.This positions OSCAR within covariance-aware quantization while specializing the paradigm to attention and serving-time INT2 KV quantization.
  • D Discussion: The theoretical analysis establishes optimality only for the frozen-error surrogate under explicit assumptions, not for the complete autoregressive decoding process or necessarily the true attention objective.Future work includes proving when the calibration estimator is unbiased or optimal for the true attention objective; the method also remains focused on INT2 KV-cache quantization.
  • D Discussion: OSCAR can make long-context inference more accessible and energy-efficient by reducing KV-cache memory and serving cost, while retaining the underlying LLM’s existing misuse risks and safety requirements.The method does not introduce new model capabilities, but cheaper deployment of powerful generative models can lower the cost of misuse.

E Additional Experimental Results · E.1 Additional Motivation Figure

The additional motivation figures show that attention-aware rotations improve history-token quantization by combining covariance-based direction separation with Hadamard mixing. Figures 9 and 10 illustrate how OSCAR produces more uniform activation ranges than raw activations or Hadamard mixing alone.

  • E Additional Experimental Results: Figures 9 and 10 provide additional motivation for attention-aware rotations in history-token activation quantization.
  • E.1 Additional Motivation Figure: Hadamard mixing flattens raw activation peaks by spreading energy across channels.
  • E.1 Additional Motivation Figure: OSCAR further separates directions according to how much they matter to attention using the target covariance.
  • E.1 Additional Motivation Figure: Hadamard transformation then mixes each covariance-separated part into a more uniform activation range.
  • E.1 Additional Motivation Figure: The figures characterize attention-aware rotations as making history-token activations easier to quantize.

E.2 Full Table For Main Accuracy Run

This section reports full independent-run accuracy tables for OSCAR across Qwen3-4B-Thinking, Qwen3-8B, Qwen3-32B, and GLM-4.7-FP8. The tables cover the main Qwen3-4B-Thinking configuration, three Qwen3-8B INT2 groupings, Qwen3-32B runs, and detailed GLM-4.7-FP8 comparisons.

  • Qwen3-4B-Thinking: Qwen3-4B-Thinking uses the main OSCAR configuration with INT2 group size=128, sink=64, recent=256, cK = 0.96, and cV = 0.92.Table 11 reports the full results for this configuration.
  • Qwen3-8B: Qwen3-8B accuracy is reported under three INT2 grouping configurations, including the full INT2 group=128 results.Tables 12–14 summarize five independent runs for these configurations.

E.3 Ablation Detail Runs

This section provides the individual experimental runs underlying the rotation-decomposition, sink/recent-window, clipping-threshold, and calibration-data ablation studies. It also lists full INT2 results for Qwen3-8B, Qwen3-32B, and GLM-4.7-FP8 configurations.

  • Rotation decomposition: Table 21 details the individual Qwen3-8B runs behind the rotation-decomposition study reported in Table 4.These runs support the rotation-decomposition ablation analysis.
  • Sink and recent window: Table 22 details the individual Qwen3-4B-Thinking-2507 runs behind the sink/recent-window study reported in Table 5.These runs support the sink and recent-window ablation analysis.
  • Clip thresholds: Table 23 reports per-task Qwen3-4B-Thinking-2507 results for each clip-threshold setting in the sweep underlying Table 6.Each row corresponds to one (cK, cV) setting with one run per task.
Loading 2605.17757v1…