Source-linked AI summary
KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
Jian Chen, Zhuoran Wang, Jiayu Qin, Ming Li, Meng Wang, Changyou Chen, Yin Chen, Qizhen Weng, Yirui Liu
TL;DR
KV-cache compression research often overlooks data-dependent activation rank and layer-specific compressibility, despite growing memory-bandwidth bottlenecks. KV-CoRE addresses this with incremental, SVD-based analysis of key/value activations, finding systematic compressibility patterns and strong links between NER and compression sensitivity.
Problem
Existing KV-cache compression methods often approximate projection weights or use uniform layer ratios, overlooking data-dependent activation rank and distinct layer-wise compressibility profiles.
Method
KV-CoRE incrementally decomposes key and value activations with independent per-layer SVD, providing gradient-free, low-memory, globally optimal low-rank approximations under the Frobenius norm.
Results
Across models, domains, and languages, NER reliably tracks compression sensitivity, with Pearson r = 0.88 for values and r = 0.64 for keys against ND-PPL.
Takeaways & Limitations
KV-CoRE provides a principled framework for dataset-level, layer-wise KV-cache evaluation and supports analysis of dynamic, data-aware compression strategies.
Abstract
from arXiv · showhide
Large language models rely on kv-caches to avoid redundant computation during autoregressive decoding, but as context length grows, reading and writing the cache can quickly saturate GPU memory bandwidth. Recent work has explored KV-cache compression, yet most approaches neglect the data-dependent nature of kv-caches and their variation across layers. We introduce KV-CoRE KV-cache Compressibility by Rank Evaluation), an SVD-based method for quantifying the data-dependent low-rank compressibility of kv-caches. KV-CoRE computes the optimal low-rank approximation under the Frobenius norm and, being gradient-free and incremental, enables efficient dataset-level, layer-wise evaluation. Using this method, we analyze multiple models and datasets spanning five English domains and sixteen languages, uncovering systematic patterns that link compressibility to model architecture, training data, and language coverage. As part of this analysis, we employ the Normalized Effective Rank as a metric of compressibility and show that it correlates strongly with performance degradation under compression. Our study establishes a principled evaluation framework and the first large-scale benchmark of kv-cache compressibility in LLMs, offering insights for dynamic, data-aware compression and data-centric model development.
1 Introduction
KV-cache compression must account for input-dependent activation rank and layer-specific compressibility. KV-CoRE provides an efficient SVD-based framework for evaluating these properties and shows that NER tracks compression sensitivity across diverse models and data.
- Motivation: Growing contexts make repeated KV-cache transfers a GPU memory-bandwidth bottleneck, motivating algorithmic compression methods.KV-caches reduce redundant computation but are repeatedly fetched from high-bandwidth memory during decoding.
- Motivation: Existing approaches often approximate projection weights or use one compression ratio across layers, overlooking data-dependent activations and distinct layer-wise profiles.Domain-specific tasks may exercise only part of a model’s capacity, yielding lower intrinsic activation rank.
- KV-CoRE: KV-CoRE incrementally decomposes key and value features from large datasets with independent per-layer SVD, gradient-free computation, and low memory overhead.The method operates directly on KV-cache features rather than projection weights.
- KV-CoRE: KV-CoRE yields globally optimal low-rank approximations under the Frobenius norm and recovers layer-specific singular-value distributions for systematic compressibility analysis.These distributions support diagnosis of representational capacity usage in LLMs.
- Findings: Across models, domains, and languages, normalized effective rank correlates strongly with perplexity under compression, revealing consistent layer-wise and data-dependent patterns.The evaluation spans instruction following, code generation, medical QA, and multilingual datasets.
2 KV-CoRE Method
KV-CoRE incrementally applies SVD to dataset-derived key and value activations, producing layer-wise singular spectra and optimal data-dependent low-rank approximations. It avoids direct SVD memory costs through covariance accumulation and uses NER to evaluate compressibility.
- KV-CoRE Method: KV-CoRE incrementally computes SVD-based compressibility from key and value features over large datasets, independently for each layer.The method is gradient-free, data-dependent, and supports batch-wise computation with low memory overhead.
- SVD-based KV-cache Analysis: Truncating the largest singular values yields the best k-rank approximation under the Frobenius norm.The approximation loss is determined by the singular values, making them sufficient for evaluating rank-dependent compressibility.
- Optimal Data-dependent Compression Matrix: The optimal data-dependent compression matrix is fW^K = W^K V_k V_k^T, implementable as down- and up-projections.Replacing W^K with these factors allows caching low-dimensional key vectors instead of full-dimensional vectors.
- Efficient SVD Computation: Incremental covariance accumulation avoids directly storing and decomposing the token-by-feature matrix while preserving mathematically equivalent singular values and right singular vectors.The algorithm updates an m_hd_h-by-m_hd_h covariance matrix across tokens, then performs eigen-decomposition.
- Normalized Effective Rank: NER normalizes effective rank by actual rank, using the full singular-value spectrum rather than a fixed approximation rank.Its score lies in [1/r, 1], with lower values indicating spectra dominated by a few large singular values and higher compressibility.
3 Related Work
Prior work studies low-rank structure in Transformer weights and representations, while KV-cache compression includes architectural approaches that merge heads or jointly compress key-value states. KV-CoRE is positioned within this broader low-rank compression literature.
- Rank Analysis in Language Models: Earlier studies examine rank in Transformer weights and representations for efficiency, expressivity, and modeling capacity.This literature includes efforts to exploit low-rank structure, prevent rank collapse, and maximize rank utilization.
- Low-Rank KV-cache Compression: Multi-head Latent Attention applies joint low-rank KV-cache compression, whereas MQA and GQA reduce cache size by merging attention heads.Other related methods address low-rank compression beyond attention mechanisms, including model-wide decomposition frameworks.
4 Experiment
The experiments evaluate KV-cache compressibility across models, English and multilingual datasets, and compression-induced performance changes. They find consistent differences between keys and values, languages and domains, model KV capacity, layers, and NER-linked robustness.
- Experimental setup: The evaluation spans five open-source LLM series, five English datasets, and multilingual QA across 15 languages.Experiments use Qwen3, Mistral, Gemma, and Phi model families, with inference-only evaluation on a single GPU.
- Average NER across models and datasets: Keys are consistently more compressible than values across models and datasets, with lower NER-K than NER-V.The asymmetry indicates greater low-rank structure in the key cache, while values retain higher-rank structure.
- Average NER across models and datasets: Cross-lingual variation outweighs cross-domain variation, with Czech and German showing higher NER than Arabic and Finnish.English-domain NER remains relatively stable, whereas multilingual differences are attributed to linguistic diversity, tokenization, and training-data availability.
- Average NER across models and datasets: KV capacity governs compressibility: Gemma-7B has lower NER than Gemma-2B, while Qwen3-4B and Qwen3-8B differ only slightly.The Gemma comparison is associated with a 16× larger KV dimension in Gemma-7B; both Qwen3 models use 8 heads × 128 per head.
- Average NER across models and datasets: Low-resource languages such as Arabic, Slovenian, and Finnish exhibit unusually low NER, especially in the value cache.The paper suggests this may reflect under-trained token embeddings collapsing into low-dimensional subspaces.
- Layer-wise compressibility patterns: NER varies across layers: middle layers often have higher NER, whereas early and late layers are typically more compressible.High- and low-rank layer positions are relatively consistent across datasets, while task characteristics can still modulate layer usage.
- Performance impact of KV-cache compression: LLaMA-2-7B remains relatively stable under aggressive compression, whereas Qwen3-4B shows substantial PPL degradation.The results suggest that models with lower NER are generally more compressible; GPT-score shows a smoother degradation trajectory than PPL in some regions.
- Dataset compressibility comparison: NER correlates positively with ND-PPL, with Pearson r = 0.88 for values and r = 0.64 for keys.Multilingual datasets cluster toward low NER and low ND-PPL, while English-domain datasets show higher NER and greater compression sensitivity.
5 Conclusion
KV-CoRE provides dataset-level analysis of KV-cache compressibility through optimal low-rank approximations. The study finds that NER tracks compression sensitivity and that compressibility varies with architecture, training data, and language coverage.
- KV-CoRE directly decomposes cached key and value activations with low memory overhead to produce globally optimal low-rank approximations.
- NER correlates closely with perplexity- and GPT-based performance under KV-cache compression.
- The analysis identifies compressibility patterns associated with architectural design, training data, and language coverage.
- KV-CoRE is positioned as a diagnostic tool and benchmark for dynamic, data-aware compression and data-centric model improvements.
A.1 MHA, MQA and GQA
MHA assigns separate key and value vectors to every attention head, whereas GQA and MQA share them within groups to reduce cache size. MLA instead caches lower-dimensional latent representations and a decoupled key component.
- MHA, MQA and GQA: MHA projects each input into key and value vectors for every attention head, causing KV-cache size to scale with the number of heads.
- MHA, MQA and GQA: GQA divides mh attention heads into mg groups and maps head indices to group indices through a helper function.
- MHA, MQA and GQA: When mg = mh, GQA becomes standard MHA; when mg = 1, it becomes MQA.
- MLA: MLA projects inputs into a joint latent KV space and a decoupled key space, caching only lower-dimensional representations.
B.1 Layer-wise NER Results
The appendix presents layer-wise NER measurements for key and value representations across several models, datasets, and languages in VisR-Bench.
- Figure B.1 reports layer-wise NER for key and value representations in Qwen3-4B across five datasets and three languages.
- Figure B.2 reports layer-wise NER for Qwen3-8B, Phi-3-mini, and mistral across five datasets and three languages.
- Figure B.3 reports layer-wise NER for gemma1.1 and gemma-1.1-7b-it across five datasets and three languages.
B.2 PPL Heatmap
The appendix provides PPL heatmaps for Qwen3-4B and LLaMA-2-7B across six datasets.
- Figure B.4 presents a PPL heatmap for LLaMA-2-7B across six datasets.
- Figure B.5 presents a PPL heatmap for Qwen3-4B across six datasets.
C GPT evaluation system prompt
The evaluator assigns a binary score by comparing an original and compressed model answer for comparable usefulness, while requiring the compressed answer to be reasonable and relevant. It checks correctness, coherence, grounding, completeness, safety, and degenerate behavior before returning only the score.
- The compressed model must provide a reasonable response; empty, nonsensical, off-topic, or nonresponsive answers receive score 0.
- The evaluator checks task fulfillment, coherence, grounding, completeness, safety, and degenerate behaviors such as repetition or prompt copying.
- Score 1 requires both answers to be reasonable and in the same quality band, with comparable usefulness despite differences in wording or detail.
- Both equally poor or failed answers can receive score 1, whereas a material quality gap or serious defect in only one answer receives score 0.
- The evaluator must return only a JSON object containing a binary score of 0 or 1, illustrated by an empty compressed answer receiving 0.