PaperlayerView exact revision on arXiv

Source-linked AI summary

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong

arXiv:2605.19660v1cs.LGcs.CL

TL;DR

KV-cache memory becomes a bottleneck under long-context and multimodal inference, while per-channel quantization loses fidelity at extreme compression. OScaR addresses Token Norm Imbalance with Canalized Rotation and Omni-Token Scaling, achieving up to 3.0× decoding speedup, 5.3× lower memory footprint, and 4.1× higher throughput versus BF16 FlashDecoding-v2.

  • Problem

    Per-channel KV-cache quantization loses effectiveness under extreme compression because Token Norm Imbalance makes shared parameters span tokens with divergent norms, amplifying quantization error.

  • Method

    OScaR combines Canalized Rotation with Omni-Token Scaling in a lightweight, training-free KV-cache compression framework for text-only, multimodal, and omni-modal LLMs.

  • Results

    Up to 3.0× decoding speedup, 5.3× lower memory footprint, and 4.1× higher inference throughput versus the BF16 FlashDecoding-v2 baseline, while achieving near-lossless INT2 performance.

  • Takeaways & Limitations

    OScaR establishes a low-complexity accuracy-efficiency Pareto front for extreme KV-cache quantization across X-LLMs.

  • Takeaways & Limitations

    Direct token-wise scaling can create artificial channel outliers and degrade quantization precision, so mitigating Token Norm Imbalance requires the proposed rotation step.

Abstract

from arXiv · show

The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. While the established per-channel quantization effectively accommodates intrinsic channel-wise outliers in Key tensors, its efficacy diminishes under extreme compression. In this work, we revisit the inherent limitations of the per-channel quantization paradigm from both empirical and theoretical perspectives. Our analysis identifies Token Norm Imbalance (TNI) as the primary bottleneck to quantization fidelity. We demonstrate that TNI systematically amplifies errors when shared quantization parameters are required to span token groups exhibiting substantial norm disparities. Instead of relying on intricate quantization pipelines (e.g., TurboQuant), we propose OScaR (Omni-Scaled Canalized Rotation), an accurate and lightweight KV cache compression framework for X-LLMs (i.e., text-only, multi-modal, and omni-modal LLMs). Advancing the per-channel paradigm, OScaR employs Canalized Rotation followed by Omni-Token Scaling to mitigate TNI-induced sequence-dimensional variance both effectively and efficiently, further supported by our optimized system design and CUDA kernels. Extensive evaluations across X-LLMs show that OScaR consistently outperforms existing methods and achieves near-lossless performance under INT2 quantization, establishing it as a robust, low-complexity, and universal framework that defines a new Pareto front. Compared with the BF16 FlashDecoding-v2 baseline, our OScaR implementation achieves a notable up to 3.0x speedup in decoding, reduces memory footprint by 5.3x, and increases throughput by 4.1x. The code for OScaR is publicly available at https://github.com/ZunhaiSu/OScaR-KV-Quant.

1 Introduction

Long-context and multimodal advances make KV-cache memory a major deployment constraint, while extreme compression exposes Token Norm Imbalance as per-channel quantization’s structural bottleneck. OScaR addresses this with a lightweight Canalized Rotation and Omni-Token Scaling pipeline, achieving near-lossless INT2 performance and strong efficiency gains.

  • Motivation: Long-context processing, streaming, and long-range audio-video understanding increase the need for efficient KV-cache memory management.These trends accompany recent advances in LLM reasoning and multimodal perception.
  • Problem: Token Norm Imbalance is identified through empirical and theoretical analysis as the fundamental bottleneck limiting per-channel quantization in X-LLMs.Per-channel quantization becomes progressively less effective under extreme compression.
  • Method: OScaR combines Canalized Rotation, which prevents Scaling-Induced Outlier Artifacts, with Omni-Token Scaling to mitigate TNI across X-LLMs.The framework is designed for text-only, multimodal, and omni-modal LLMs and avoids intricate auxiliary pipelines.
  • Results: OScaR outperforms existing methods, achieves near-lossless performance under INT2 quantization, and establishes an advantageous accuracy-efficiency Pareto front.The framework preserves high quantization fidelity while maintaining low overall complexity.
  • Results: 3.0× decoding speedup, 5.3× lower memory footprint, and 4.1× higher inference throughput are achieved against the BF16 FlashDecoding-v2 baseline.These gains come from optimized system design and dedicated CUDA kernels.

2 Related Work

Related work establishes quantization as essential for efficient LLM deployment and identifies KV-cache compression as increasingly important as context lengths grow. Existing methods differ by quantization granularity, while tensor outlier patterns motivate distinct treatment of Key and Value states.

  • KV-cache quantization: Quantization methods for weights and activations include GPTQ, AWQ, and SmoothQuant, while growing context lengths make the KV cache a dominant decoding-memory bottleneck.Existing KV-cache methods are categorized by quantization granularity, including per-token, per-channel, and per-element approaches.
  • LLM outliers: Outliers challenge high-fidelity quantization and comprise channel-wise and token-wise types based on their distributional characteristics.Channel-wise outliers concentrate in specific feature dimensions and predominantly affect Key and Query tensors more than Value tensors.
  • KIVI quantization: Key states exhibit channel-wise outliers and therefore use per-channel quantization, whereas relatively uniform Value states use per-token quantization.This Key–Value distinction is illustrated in the KIVI quantization scheme.

3 Preliminaries

Decoder-only LLMs use KV caching to avoid redundant autoregressive computations, but the cache grows linearly with sequence length and becomes a memory bottleneck. Existing hybrid quantization exploits channel-wise Key outliers and relatively uniform Value magnitudes, using per-channel Keys and per-token Values with a high-precision residual window.

  • KV caching: Decoder-only Transformers cache Key and Value states to eliminate redundant computations during autoregressive decoding.In multimodal LLMs, textual, visual, and audio representations are concatenated along the sequence dimension after projection into a shared latent space.
  • KV caching: The KV cache is updated by appending newly generated vectors, so its memory footprint grows linearly with sequence length.This growth creates a memory-bound bottleneck that motivates cache compression.
  • Hybrid quantization: Hybrid quantization applies per-channel quantization to Keys, which exhibit channel-wise outliers, while preserving per-token granularity for relatively uniform-magnitude Values.This design is motivated by the distinct numerical distributions of Key and Value states.
  • Hybrid quantization: Continuous per-channel quantization during autoregressive generation requires a high-precision residual window before accumulated tokens are block-wise quantized.Newly generated tokens remain in full precision until the buffer reaches the predefined residual number R.
  • State distributions: Query, Key, and Value states each contain a sparse yet consistent subset of exceptionally low-norm tokens.Figure 3 presents their L2-norm distributions and corresponding heatmaps.

4 Methodology

Section 4 identifies Token Norm Imbalance (TNI) as the fundamental weakness of per-channel KV quantization and introduces OScaR, a training-free framework that combines Canalized Rotation with Omni-Token Scaling to mitigate it. The methodology further specifies an efficient query/key/value pipeline built around online transforms, token-wise scaling, and per-channel quantization.

  • 4.1 Limitations of Per-Channel Quantization: Per-channel quantization assumes similar token magnitudes within each channel, so divergent tokens compromise shared parameters and degrade fidelity.The section examines this limitation through empirical observations, theoretical derivations, and quantitative error analysis.
  • 4.1 Limitations of Per-Channel Quantization: Token Norm Imbalance (TNI) appears across text-only and multi-modal LLMs, with sparse low-norm tokens expanding block dynamic ranges and weakening per-channel quantization.Multi-modal TNI also includes broader variation, inter-modality disparities, and exceptionally large-norm outliers.
  • 4.1 Limitations of Per-Channel Quantization: TNI-induced reconstruction error is governed by within-block token-norm range, while per-token Value quantization avoids cross-token interference.The theoretical and empirical analyses identify TNI as a fundamental vulnerability of per-channel Key quantization under extreme compression.
  • 4.2 OScaR Algorithmic Design: OScaR is a lightweight, training-free KV cache compression framework for text-only, multi-modal, and omni-modal LLMs.Its algorithmic design consists of Canalized Rotation followed by Omni-Token Scaling, with system optimizations and CUDA kernels discussed separately.
  • 4.2 OScaR Algorithmic Design: Canalized Rotation redistributes outlier-channel energy before Omni-Token Scaling normalizes token norms, jointly mitigating TNI without scaling-induced artifacts.Rotation alone cannot balance token norms, whereas direct scaling balances norms but introduces the Scaling-Induced Outlier Artifact.
  • 4.2 OScaR Pipeline: OScaR applies online Fast Hadamard Transforms and token-wise norm scaling to Query and Key, followed by per-channel Key quantization and inverse scaling during dequantization.The Value path uses an offline Hadamard transform, as specified in the pipeline overview.

5 Experiments

Experiments evaluate OScaR across text-only, multi-modal, and omni-modal LLMs against diverse quantization baselines, emphasizing extreme long-context tasks. OScaR delivers strong 2-bit accuracy and substantial decoding efficiency gains over BF16 FlashDecoding-v2.

  • Models and Tasks: Experiments span text-only, multi-modal, and omni-modal LLMs, with most evaluations targeting extreme long-context processing.Models include Llama-3.1-8B, Qwen3-8B, LLaVA-v1.6-vicuna-7B, Qwen3-VL-4B/8B-Instruct, and Qwen3-Omni-30B-A3B.
  • Baselines: OScaR is compared with per-channel, rotation-based per-token, and LUT-based baselines, including KIVI, OTT, QuaRot, RotateKV, and TurboQuant.OTT and RotateKV use high-precision protection for outlier tokens.
  • Results on Text-Only LLMs: OScaR achieves 41.75% average LongBench-E accuracy, surpassing the second-best method by 1.01 percentage points and incurring only a 1.7% drop versus the 16-bit baseline on Qwen3-8B.It also reaches 96.5% retrieval accuracy on NIAH.
  • Results on Multi-Modal and Omni-Modal LLMs: Under 2-bit quantization, OScaR frequently approaches the 16-bit baseline, improves OCRBench by 2.5 percentage points over the second-best method on Qwen3-VL-4B, and leads quantized methods on MMAU-Pro.These evaluations cover OCRBench, DocVQA, and MMAU-Pro across multi-modal and omni-modal models.
  • Efficiency: 3.0× decoding speedup, 5.3× lower decoding memory footprint, and 4.1× higher throughput are achieved versus BF16 FlashDecoding-v2 under the reported 128K-context and batch-size-48 settings.Efficiency experiments use Qwen3-8B on a single 141GB H20 GPU.

6 Conclusion … G Theoretical Derivation of TNI-Induced Quantization Errors

The paper identifies Token Norm Imbalance as an intrinsic source of amplified per-channel KV quantization error and proposes OScaR to mitigate it through Canalized Rotation and Omni-Token Scaling. Across text-only and multi-modal settings, the analysis connects norm disparities, attention-sink tokens, and low-bit quantization fidelity.

  • 6 Conclusion: OScaR is a lightweight, training-free KV cache compression framework that combines Canalized Rotation with Omni-Token Scaling to mitigate TNI-induced sequence-dimensional variance.The conclusion frames TNI as a primary structural bottleneck under extreme compression.
  • A Limitations and Future Directions: OScaR’s online rotation and token-wise scaling reduce quantization overhead relative to existing frameworks but remain more costly than plain per-channel quantization.Canalized Rotation requires online computation with RoPE, preventing offline fusion of Query and Key Hadamard transforms with weight matrices.
  • A Limitations and Future Directions: OScaR is designed for LLMs, multi-modal LLMs, and omni-modal LLMs, although current experiments primarily use models with LLM backbones.The paper proposes extending the framework to autoregressive models beyond LLMs that require KV caching.
  • B Algorithm of OScaR: The algorithm preprocesses WV and WO with Hadamard transforms, then applies OScaR during attention computation in both prefill and decoding.This describes the stated algorithmic workflow.
  • C Preliminaries on Low-Bit Quantization: At extreme low-bit settings such as b ≤4, reduced numerical resolution can cause severe accuracy degradation, especially in the presence of outliers.The paper motivates rotation-based redistribution, residual correction, and mixed-precision preservation as outlier-aware responses.
  • D Token Norm Imbalance in Text-Only LLMs: In text-only LLMs, sparse and consistent low-norm outlier tokens in Q, K, and V inflate per-channel scaling factors and substantially increase quantization error.These tokens are observed across the attention states and cause fidelity degradation under per-channel quantization.
  • E Outlier Tokens and Attention Sinks: Attention Sink tokens are low-norm tokens that absorb attention mass and produce minimal residual updates, while their consistency across Q, K, and V contributes to low-precision deployment challenges.The paper relates this behavior to the softmax sum-to-one constraint and no-op attention behavior.
  • F Token Norm Imbalance in Multi-modal LLMs: In multi-modal LLMs, TNI includes broader norm variation, inter-modality norm disparities, and exceptionally large-norm outlier tokens beyond attention-sink-related low-norm tokens.The results show that norm behavior can remain smooth within modalities while diverging substantially across modalities.

H Quantitative Analysis of TNI-Induced Quantization Errors

Under extreme KV cache compression, Token Norm Imbalance substantially worsens per-channel Key quantization errors, whereas per-token Value quantization confines norm variation within tokens and avoids cross-token interference.

  • Impact on Per-Channel Key Quantization: Under the same 2-bit configuration, mixed-modality groups increase visual-component per-channel Key quantization error by 140% versus single-modality groups.Mixed-modality groups contain tokens from multiple modalities, whereas single-modality groups contain tokens from one modality.
  • Impact on Per-Token Value Quantization: Per-token Value quantization prevents TNI-induced error amplification because norm variations remain confined to individual tokens.Value states lack channel-wise outliers, making per-token quantization the standard choice.
  • Conclusion: The analysis empirically confirms that TNI fundamentally undermines per-channel quantization but has negligible impact on per-token quantization.Experiments use RTN quantization with group size 32 and measure errors using MSE on LLaVA-v1.5-7B.

I Detailed Analysis of Scaling-Induced Outlier Artifact … O Experimental Results and Analysis on Needle-in-a-Haystack

The paper explains how direct token-wise scaling creates artificial channel outliers and quantization errors, then presents OScaR’s design, complexity, implementation, evaluation setup, and Needle-in-a-Haystack performance. OScaR combines Hadamard-based key/query processing with token-wise normalization and metadata, achieving 96.5% retrieval accuracy among quantized methods.

  • I Detailed Analysis of Scaling-Induced Outlier Artifact: Direct token-wise scaling can create artificial outliers when low-norm tokens are amplified relative to outlier-dominated tokens.The artifact arises because the scaling-factor ratio equals the tokens’ norm ratio, which can be much greater than one.
  • I Detailed Analysis of Scaling-Induced Outlier Artifact: 50x larger quantization steps result when channels expand from approximately 0.01 to 0.5, severely degrading precision for normal tokens.The numerical example uses N = 1, a = [1, 1, 1, 100], and b = [0.1, 0.1, 0.1, 0.1].
  • J Additional Visualizations of OScaR Processing Stages: Only complete OScaR balances token norms without incurring the Scaling-Induced Outlier Artifact; direct scaling and Canalized Rotation alone each fail one requirement.This conclusion is supported by visualizations of the OScaR processing stages.
  • K.1 Symbolic Operation Counts: OScaR combines online WHT-based Canalized Rotation with token-wise L2 normalization, adding normalization stages to per-channel key quantization.The method’s symbolic costs are derived in terms of hidden dimension d, head dimension h, and sequence length L, alongside comparisons with KIVI, QuaRot, and TurboQuant variants.
  • K Theoretical Complexity Analysis; K.2 Effective Cost Conversion; K.3 Numerical Results; K.4 Discussion: 123.0 million units is OScaR’s theoretical decode cost under d = 4096 and L = 10,000, while avoiding table lookups through Hadamard transforms and hardware-accelerated rsqrt.Theoretical costs use one arithmetic operation as 1 unit and one random table lookup as 5 units, but hardware effects can make estimates diverge from measured efficiency.
  • L Implementation Details; L.2 Fused Hadamard-Norm Kernel; L.4 Prefill and Decode Workflows: Fused CUDA preprocessing, packed/residual caches, and periodic residual flushing implement OScaR’s inference workflow while preserving unchanged FlashAttention-2 attention computation.The implementation extends BitDecoding and HadaCore, uses Tensor-Core optimization for head dimension 128, and flushes residual blocks at R = 128.
  • L.1 Overall Design; L.3 Quantization Format and Cache Organization: 2-bit representation is used for OScaR’s KV cache, with Qwen3-8B configured at head dimension 128, group size 32, and residual block size 128.Keys receive Hadamard rotation followed by token-wise normalization; token norms are stored as auxiliary metadata, and orthogonality preserves inner products.

P Experimental Results and Analysis on OCRBench

On OCRBench, OScaR achieves the highest accuracy among all evaluated 2-bit quantization methods across three models. It reaches 51.9% on LLaVA-v1.6-vicuna-7B and 85.6% on Qwen3-VL-8B.

  • Overall OCRBench results: OScaR consistently achieves the highest accuracy among all 2-bit quantization methods across all three evaluated models.OCRBench’s final score is the number of correctly answered samples.
  • Overall OCRBench results: 51.9% accuracy on LLaVA-v1.6-vicuna-7B gives OScaR a 0.6 percentage-point lead over the second-best method.
  • Overall OCRBench results: 85.6% accuracy on Qwen3-VL-8B places OScaR within 0.2 percentage points of the reported comparison point.

Q Experimental Results and Analysis on DocVQA

On DocVQA, OScaR achieves the highest accuracy among all evaluated 2-bit methods across three models. It delivers near-lossless INT2 performance, closely matching or surpassing 16-bit and competing quantized baselines.

  • DocVQA Results: OScaR achieves the highest accuracy among all 2-bit quantization methods across the three evaluated models.Table 7 reports results on DocVQA, a document-understanding visual question answering benchmark.
  • DocVQA Results: On Qwen3-VL-8B, OScaR slightly surpasses the 16-bit baseline, demonstrating near-lossless performance under INT2 quantization.
  • DocVQA Results: On Qwen3-VL-4B, OScaR trails the 16-bit baseline by 0.4 percentage points while outperforming the strongest competing 2-bit method by 2.5 percentage points.

R Experimental Results and Analysis on MMAU-Pro · S TNI Analysis Before and After OScaR

On MMAU-Pro, OScaR is evaluated on open-ended QA and audio instruction following, achieving the strongest reported results under extreme KV-cache compression. Token-norm visualizations further show that OScaR alleviates TNI across models and modalities by producing more compact, balanced distributions.

  • R Experimental Results and Analysis on MMAU-Pro: MMAU-Pro evaluation targets open-ended QA and audio instruction following, both requiring reasoning beyond standard multiple-choice tasks.The open-ended subset requires free-form response generation, while audio instruction following measures AIF accuracy.
  • R Experimental Results and Analysis on MMAU-Pro: Open-ended responses are judged by Qwen2.5-7B-Instruct on correctness, relevance, completeness, and clarity, with scores converted to percentages.Each criterion is scored from 1 to 5 for consistent comparison with multiple-choice results.
  • R Experimental Results and Analysis on MMAU-Pro: OScaR achieves the highest scores across all three MMAU-Pro evaluation metrics.All competing methods except TurboQuant+ use INT2 quantization with group size 128, while TurboQuant+ uses 2.5-bit quantization.
  • R Experimental Results and Analysis on MMAU-Pro: 1.2 percentage points: OScaR surpasses the 16-bit baseline on the open-ended subset.It also outperforms every other quantized method on that subset.
  • R Experimental Results and Analysis on MMAU-Pro: 2.0 percentage points: OScaR’s Good Rate exceeds the 16-bit baseline, while its advantage over the best quantized competitor reaches 2.8 percentage points.Good Rate is the proportion of open-ended responses receiving an overall score of 4 or higher out of 5.
  • S TNI Analysis Before and After OScaR: Across models and modalities, OScaR transforms scattered token-norm distributions into more compact and balanced patterns, empirically alleviating TNI.Figures 25–28 provide visual evidence supporting OScaR’s performance under extreme KV-cache quantization.

T Ablation Study

The ablation study shows that Canalized Rotation and Omni-Token Scaling are complementary: scaling after rotation recovers accuracy, whereas direct token-wise scaling alone worsens performance. Among scaling coefficients, ℓ2 norm and rsqrt perform best, motivating the final rsqrt implementation.

  • T Ablation Study: Omni-Token Scaling after Canalized Rotation substantially recovers accuracy from the severely degraded INT2 baseline.The complete OScaR configuration achieves the best trade-off on WorldSense with group size 128.
  • T Ablation Study: Direct token-wise scaling without Canalized Rotation further harms performance.This result aligns with the paper’s analysis of token norm imbalance in Section 4.2.
  • T Ablation Study: 14.47 is the accuracy for Max on Qwen2.5-7B, versus 42.57 with the ℓ2 norm.Max uses the maximum absolute channel value, while ℓ2 norm uses each token’s Euclidean norm.
  • T Ablation Study: ℓ2 norm and rsqrt achieve the best mutually comparable performance across evaluated model families.Because their difference is negligible, OScaR adopts the hardware-efficient rsqrt-based implementation; mean absolute value remains competitive without catastrophic drops.

U Accuracy-Efficiency Pareto Front Analysis · V Additional Decoding Efficiency Comparison

OScaR combines the highest LongBench-E accuracy with competitive decode cost, occupying an advantageous Pareto-front position. It also delivers stable, substantial decoding speedups over BF16 FlashDecoding-v2 across long contexts.

  • U Accuracy-Efficiency Pareto Front Analysis: OScaR achieves the highest LongBench-E accuracy with competitive efficiency, occupying a distinct and advantageous position on Qwen3-8B’s Pareto front.Figure 9 evaluates average LongBench-E accuracy against decode cost in million units.
  • U Accuracy-Efficiency Pareto Front Analysis: KIVI establishes the efficiency baseline with the lowest decode cost and an accuracy of 47.95, while avoiding rotation and normalization overhead.Its result illustrates the trade-off associated with minimal preprocessing.
  • U Accuracy-Efficiency Pareto Front Analysis: TurboQuant+ requires approximately three times OScaR’s and KIVI’s decode cost, yet provides only marginal accuracy gains over KIVI and falls off the Pareto front.The comparison uses theoretical decode cost and Qwen3-8B LongBench-E scores.
  • U Accuracy-Efficiency Pareto Front Analysis: OScaR offers a favorable combination of competitive computational cost and strong accuracy, giving it a distinct and advantageous Pareto-front position.The overall analysis compares OScaR with KIVI and TurboQuant+.
  • V Additional Decoding Efficiency Comparison: OScaR latency increases modestly from 24.1 ms/tok at 16K to 30.9 ms/tok at 128K, remaining stable across the evaluated context range.The comparison uses a single H20 GPU with OScaR on PyTorch and TurboQuant+ on llama.cpp.
  • V Additional Decoding Efficiency Comparison: 3.0× speedup at 128K tokens is achieved by OScaR over BF16 FlashDecoding-v2, compared with TurboQuant+’s 1.3× speedup.At 128K, the reported latencies are 92.9 ms/tok for BF16 and 30.9 ms/tok for OScaR.
Loading 2605.19660v1…