Source-linked AI summary
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Suho Yoo, Hyunjong Ok, Jongmin Choi, Jihoo Jung, Joon Son Chung
TL;DR
Omni-LLM KV caches grow substantially with multimodal audio, video, and text inputs, but existing KV-cache quantization does not address their heterogeneous structure. OmniKVQuant uses local key ranges and modality-specific value rotations, with fused decoding that avoids dense FP16 reconstruction. Across seven audio-visual benchmarks, it preserves high 2-bit performance on Qwen2.5-Omni and Qwen3-Omni while outperforming TurboQuant.
Problem
Omni-LLM KV-cache quantization is unexplored despite multimodal inputs causing substantial cache-memory growth and introducing temporal key drift and heterogeneous value geometry.
Method
OmniKVQuant is a training-free framework using local key ranges, modality-specific value rotations, and fused decoding that avoids reconstructing a dense FP16 cache.
Results
Across seven audio-visual benchmarks, OmniKVQuant retains 98.1% of FP16 performance at 2-bit precision on Qwen2.5-Omni and consistently outperforms TurboQuant on Qwen2.5-Omni and Qwen3-Omni.
Takeaways & Limitations
Accounting for the distinct key and value structures of Omni-LLMs supports low-bit KV-cache quantization across multiple models and task types.
Takeaways & Limitations
The reported analyses use Qwen2.5-Omni-3B unless otherwise noted.
Abstract
from arXiv · showhide
As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains unexplored. In this paper, we analyze how TurboQuant, a representative rotation-based KV cache quantization method, behaves on multimodal caches and identify two critical issues: temporal key drift and heterogeneous value geometry. To address these, we propose OmniKVQuant, a training-free framework that (i) sets the key quantization range over each short window of the input stream; and (ii) rotates values separately per modality. On Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks. We further provide a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built. Code: https://github.com/kaistmm/OmniKVQuant
1. INTRODUCTION
Omni-LLMs make KV-cache memory a deployment bottleneck, while existing quantization methods do not account for multimodal cache structure. OmniKVQuant addresses temporal key drift and heterogeneous value geometry with local key ranges and modality-specific value rotations, retaining performance at 2-bit precision.
- Motivation: Omni-modal inputs can exceed 10,000 tokens per minute, making their key-value cache larger than the model weights.Each token contributes a key and a value, so multimodal context length directly increases cache memory.
- Prior Work: The method extends a rotation-based quantization strategy beyond prior work that handled individual modalities or weights and activations rather than joint Omni-LLM KV caches.The paper identifies no existing method that quantizes Omni-LLM KV caches.
- Problem: TurboQuant’s fixed scale cannot track temporal key drift, while a shared value rotation poorly captures the distinct geometries of audio, video, and text.These failures arise because key distributions vary across short windows and modalities occupy different regions of value space.
- Method: OmniKVQuant adapts key codebook scales over short sequence windows and calibrates separate value rotations for each modality.The method is training-free and targets the multimodal structures that a shared treatment misses.
- Results: 98.1% of FP16 performance is retained across seven audio-visual benchmarks on Qwen2.5-Omni at 2-bit precision, consistently outperforming TurboQuant.The result motivates cache-structure-aware quantization for Omni-LLMs.
- Implementation: A fused Triton decode kernel unpacks quantized caches during attention without reconstructing a dense FP16 cache.It rotates one query per head and batches inverse value rotations after attention aggregation.
2. METHOD
OmniKVQuant quantizes multimodal KV caches by adapting key ranges locally over time and rotating values separately for each modality. It builds on rotation-based quantization while accounting for temporal key drift and heterogeneous value geometry.
- Preliminaries: Omni-LLMs organize visual, audio, and text tokens into one sequence, project hidden states into queries, keys, and values, and reuse keys and values through autoregressive decoding.Attention uses the projected queries and keys to form weights that aggregate values; the resulting representations are stored in the KV cache.
- TurboQuant: TurboQuant rotates each cached key or value toward a Gaussian-like coordinate distribution before applying a fixed codebook and inverse rotation.The method separates magnitude from direction, applies a randomized Hadamard transform, quantizes rotated coordinates, and reconstructs the vector.
- Observations: 30.40% retained Gaussian variance, with W1 and KS distances of 0.778 and 0.449, shows that cached keys substantially deviate from TurboQuant’s Gaussian target.The mismatch persists across modalities and is more pronounced for audio in the supplied analysis.
- OmniKVQuant: OmniKVQuant min-max scales rotated keys within short consecutive-token groups, stores group statistics, and then applies the shared fixed codebook to the locally scaled values.Local statistics restore the original range during reconstruction, allowing the codebook to follow temporal variation without changing the Hadamard basis.
- OmniKVQuant: For values, OmniKVQuant weights directions by their contribution to attention outputs, computes a covariance matrix separately per modality, and derives a modality-specific eigenbasis.The bases are calibrated using VGGSound data, and the reconstructed values use the corresponding modality rotation.
- Evaluation: Table 3 reports that OmniKVQuant consistently reduces quantization error compared with TurboQuant at comparable bit budgets.The supplied table caption identifies the comparison but does not provide individual error values.
3. EXPERIMENTS
Across seven audio-visual benchmarks, OmniKVQuant outperforms TurboQuant at matched 2-bit KV-cache budgets on Qwen2.5-Omni and Qwen3-Omni. Analyses attribute the gains to local key ranges and modality-specific value rotations, with performance advantages persisting across bit widths.
- Ablation Studies: Local key ranges provide the largest component-wise improvement, while modality-specific value rotation adds gains across both model scales.The ablation compares TurboQuant with successive codec components.
- Ablation Studies: OmniKVQuant consistently outperforms TurboQuant from 2-bit to 4-bit multiple-choice quantization.The comparison indicates that the advantage persists across the tested compression rates.
- Experimental Results: 98.8% and 97.3% of FP16 performance are retained by OmniKVQuant on Qwen2.5-Omni multiple-choice and captioning tasks, respectively, while Qwen3-Omni retains 90.6% and 83.4%.TurboQuant retains only 58.5% and 43.6% on the corresponding Qwen3-Omni tasks.
- Experimental Results: OmniKVQuant substantially outperforms TurboQuant at the same K2/V2 budget across four multiple-choice and three captioning benchmarks.The evaluation uses accuracy for multiple-choice tasks and an LLM judge for captioning.
- Value Rotation: Matching each modality with its own value rotation produces the lowest quantization error for text, audio, and video tokens.Pooled and shared rotations are consistently worse across all three modality groups.
- Key Adaptation: Preserving temporal contiguity reduces key range and downstream quantization error beyond modality-only grouping.Global shuffling produces the largest key error, while within-modality shuffling recovers only part of the loss.
4. CONCLUSION
OmniKVQuant is a training-free framework that quantizes Omni-LLM KV caches using local key ranges and modality-specific value rotations. Across seven audio-visual benchmarks, it consistently outperforms TurboQuant under 2-bit quantization, including multiple-choice and captioning tasks.
- 4. CONCLUSION: OmniKVQuant is a training-free framework using local key ranges for keys and modality-specific rotations for values.The design reflects the different structures of keys and values in Omni-LLM caches.
- 4. CONCLUSION: Across seven audio-visual benchmarks on Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant consistently outperforms TurboQuant under 2-bit KV-cache quantization.The gains hold across multiple-choice and captioning tasks.
- 4. CONCLUSION: The results indicate that key and value structure matters when compressing Omni-LLM KV caches.The same design carries across model scales and task types.