PaperlayerView exact revision on arXiv

Source-linked AI summary

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, Xiaowen Chu

arXiv:2502.00299v5cs.CL

TL;DR

Long-context inference makes KV-cache memory a bottleneck, and existing compression methods focus on isolated token importance rather than semantic information. ChunkKV compresses the cache by retaining informative semantic chunks and reusing their indices across layers. Across diverse models and benchmarks, it reports higher efficiency and accuracy than existing methods while using only a fraction of the memory.

  • Problem

    KV-cache memory becomes a bottleneck for long-context LLM inference, while existing methods focus on evicting individual tokens rather than preserving semantic information.

  • Method

    ChunkKV groups related KV-cache tokens into semantic chunks and uses layer-wise index reuse to share selected indices across subsequent layers.

  • Results

    ChunkKV consistently outperforms existing methods in efficiency and accuracy across diverse LLMs and benchmarks while using only a fraction of the memory.

  • Takeaways & Limitations

    Chunk-based compression preserves crucial contextual information for long-context understanding, complex reasoning, and safety evaluations.

  • Takeaways & Limitations

    ChunkKV may be less suitable when every token must be retained, such as legal or biomedical analysis requiring absolute semantic fidelity.

Abstract

from arXiv · show

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.

1 Introduction

Long-context LLM inference makes KV-cache memory a bottleneck, while token-level compression can discard dependencies needed for semantic completeness. ChunkKV addresses this by preserving semantic chunks and reusing indices across layers, with evaluations reporting improved efficiency and accuracy.

  • A single token in a 7B-parameter model requires approximately 0.5 MB of GPU memory, so a 10,000-token prompt consumes around 5 GB.
  • Token-level KV-cache compression can preserve question-related words while omitting subjects or objects, thereby losing essential semantic information.
  • ChunkKV groups tokens into semantic chunks that are preserved or discarded as complete units, retaining structures such as subject, predicate, and object.
  • Layer-wise index reuse exploits similar preserved indices across layers by reusing indices computed for one layer group across subsequent layers.
  • ChunkKV surpasses existing KV-cache compression methods in efficiency and accuracy across LongBench, NIAH, GSM8K, and JailbreakV evaluations.

2 Related Work

Prior KV-cache compression methods primarily evict individually unimportant tokens to manage memory and preserve performance. ChunkKV instead emphasizes the semantic importance of contiguous chunks in the original cache.

  • Existing KV-cache compression methods mainly evict tokens judged unimportant, with some retaining initial and recent tokens because they receive consistently high attention.
  • These methods address GPU-memory management while aiming to maintain original performance, but they focus less on semantic information in the prompt.
  • ChunkKV finds that chunks of the original KV cache are more important than discrete tokens for preserving prompt semantics.

3 ChunkKV

ChunkKV compresses KV caches by retaining informative semantic chunks rather than isolated tokens, then reuses selected indices across layers to reduce compression overhead. The method is motivated by higher cross-layer similarity of ChunkKV’s preserved indices and is evaluated for long-context and reasoning performance.

  • KV Cache Compression: ChunkKV models KV-cache memory cost as MKV = 2 × B × S × L × N × D × 2, where batch size, sequence length, layers, heads, head dimension, KV matrices, and float16 precision determine usage.With B = 1 and S = 2048, LLaMA-3-8B-Instruct’s KV cache costs nearly 1 GB; beyond batch size 24, it exceeds an RTX 4090’s capacity.
  • Chunk Based KV Compression: ChunkKV groups related tokens into semantic chunks and retains the most informative chunks as complete units, preserving structures such as subject–verb–object relationships.The selected chunks reduce KV-cache memory usage while retaining essential information.
  • Chunk Based KV Compression: ChunkKV computes attention scores over chunks, selects the top-k chunks, retains their key and value entries in sequence order, and concatenates the observe window.The observe window is computed from recent-query attention scores; selected chunks are indexed from the original cache.
  • Layer-Wise Index Reuse: ChunkKV preserves more similar indices across neighboring layers than SnapKV, as shown by layer-wise heatmaps and higher adjacent-layer average Jaccard similarity.Heatmap cells represent similarity between preserved indices from two layers, with deeper colors indicating higher similarity.
  • Layer-Wise Index Reuse: Layer-wise index reuse shares one layer’s selected token indices across a group of subsequent layers, reducing KV-cache compression time by 20% versus FullKV with a 0.5% performance drop.ChunkKV computes indices on the first layer in each group and reuses them for the remaining layers.

4 Experiment Results

Across in-context learning and long-context benchmarks, ChunkKV generally outperforms discrete-token compression methods while retaining semantic chunks. Layer-wise index reuse improves efficiency with minimal reported performance loss, and moderate chunk sizes provide the strongest results.

  • In-Context Learning: ChunkKV outperforms other KV cache compression methods on GSM8K and many-shot GSM8K across different models and compression ratios.The reported advantage is attributed to maintaining crucial contextual information through chunk-level rather than discrete-token compression.
  • In-Context Learning: ChunkKV outperforms other KV cache compression methods on JailbreakV across different compression ratios.The paper reports chunk-level caching as more effective than discrete-token methods for this safety benchmark.
  • Long-Context Benchmark: ChunkKV outperforms other compression methods overall on LongBench English and Chinese subtasks at a 10% KV cache compression ratio.The comparison reports performance gaps relative to FullKV, with negative values indicating degradation.
  • Long-Context Benchmark: ChunkKV outperforms other KV cache compression methods on NIAH across varying token lengths and depth percentages.The figure encodes token length horizontally, depth percentage vertically, and successful needle retrieval with green cells.
  • Index Reuse: 26.5% throughput improvement over FullKV is achieved by ChunkKV_reuse, alongside up to 20.7% latency reduction.The throughput gains are particularly notable for longer input sequences.
  • Index Reuse: Layer-wise index reuse causes less than 0.6% LongBench degradation for both models while GSM8K effects are neutral or slightly positive.The results support reuse of semantic chunks across adjacent transformer layers without sacrificing reported accuracy.
  • Chunk Size: Chunk size 10 consistently achieves the best results, while sizes from 5 to 20 remain relatively stable across LongBench and NIAH.Chunk sizes of 3 fragment context, whereas size 30 makes semantic granularity too coarse and can lose fine-grained information.
  • Comparing with KV Quantization: ChunkKV achieves a 27.3% improvement in overall inference speed over KIVI at 2-bit quantization while using 10% cache size.The reported total generation times are 164.66s for ChunkKV and 226.52s for KIVI.

5 Conclusion

ChunkKV addresses semantic information loss in KV cache compression by retaining informative chunks, and its layer-wise index reuse improves computational efficiency across diverse models and benchmarks.

  • ChunkKV preserves semantic information by retaining informative chunks rather than isolated tokens.The method is evaluated across DeepSeek-R1, LLaMA-3, Qwen2, and Mistral on GSM8K, LongBench, NIAH, and JailbreakV.
  • 20.7% latency reduction and 26.5% throughput improvement are achieved through layer-wise index reuse with minimal performance impact.
  • ChunkKV maintains contextual information for complex reasoning, long-context understanding, and safety evaluations while using a fraction of the memory.

A.1 Quantitative Analysis

The quantitative analysis compares ChunkKV with discrete token-based methods using KV-cache reconstruction and attention-preservation metrics across model layers. ChunkKV generally preserves information and attention relationships better, especially in semantic-processing layers.

  • ChunkKV is evaluated against SnapKV and H2O using KV cache L1 loss and attention cosine similarity across model layers.The study uses 100 sequences from each LongBench sub-category with uncompressed KV cache and attention patterns as ground truth.
  • ChunkKV achieves lower L1 loss than SnapKV and H2O, particularly across layers 5-25.Lower L1 loss indicates better preservation of the original KV cache information.
  • ChunkKV achieves higher attention cosine similarity across most layers, especially layers 0-5 and 20-30.The result indicates stronger preservation of attention relationships between tokens.
  • ChunkKV records the lowest L1 loss and highest attention cosine similarity averaged across all layers.These averages outperform both baseline methods.
  • Approximately 2% lower L1 loss and 1.5% higher cosine similarity translate into better information retention for semantic processing.The improvements are consistent across sequences and are particularly evident in middle layers.

A.2.2 ChunkKV Method

ChunkKV retains semantically meaningful token chunks instead of isolated high-scoring tokens, preserving contextual relationships and complete information within the compressed cache.

  • ChunkKV identifies and retains semantically meaningful chunks such as complete statements about pandas’ diet.Examples include chunks describing bamboo consumption, occasional vegetation, and possible animal prey.
  • Preserving chunks maintains relevant keywords together with their contextual relationships and additional pertinent information.

A.3 Comparative Analysis

ChunkKV preserves coherent semantic context while compressing the KV cache, and its implementation and index-reuse optimizations improve efficiency. Across LongBench, NIAH, Chinese tasks, and quantization comparisons, it maintains strong performance under compression.

  • Chunk-level semantic preservation: ChunkKV preserves complete phrases or sentences, reducing ambiguity and retaining semantic relationships within the compressed cache.Chunk-level retention can preserve multiple relevant tokens in their proper context.
  • Chunk-level semantic preservation: ChunkKV’s semantic preservation supports improved accuracy, long-context processing, and complex reasoning by retaining contextually rich information.
  • Implementation: The implementation uses vectorized attention operations, binary masks, boundary checks, and single-pass selection to reduce computational and memory overhead.The algorithm selects top-scoring chunks, retains recent tokens, and returns compressed K′ and V′ caches.
  • Index reuse: Layer-wise index reuse reduces repeated compression work because index selection is typically much cheaper than full compression.The resulting speedup is approximately Nreuse when Tselect is negligible compared with Tcompress, although hardware and implementation affect the realized gain.
  • Comparative results: ChunkKV achieves the best overall LongBench average and performs consistently better as NIAH context length increases.
  • Comparative results: Less than 1% performance variation is observed across chunk sizes, with chunk sizes from 10 to 20 performing better on the evaluated benchmarks.
  • Comparative results: 41.59 at 30% compression is comparable to KIVI’s 41.43 with 8-bit quantization across LongBench subtasks.ChunkKV is particularly strong on code-related tasks, while KIVI performs better in single-document QA.

B.7 Comparison with Orthogonal and Training-Based Methods

ChunkKV is compared with training-based and other KV cache strategies on LongBench and efficiency measurements. The comparisons emphasize robust performance and improved latency or throughput, especially with layer-wise index reuse.

  • Training-Based Methods: ChunkKV is evaluated against Palu, a training-based compression method, on LongBench using LLaMA-3-8B-Instruct.The comparison positions ChunkKV as a training-free eviction method against a recent training-based approach.
  • Training-Based Methods: ChunkKV demonstrates robust performance on the diverse, out-of-distribution LongBench benchmark, whereas Palu experiences significant degradation.The reported contrast concerns performance under a benchmark distribution differing from Palu’s training data.
  • Efficiency: Table 25 compares KV cache strategies using Time to First Token (TTFT) and Token Processing Time (TPOT) across varying output lengths.These metrics characterize efficiency under different generation conditions.
  • Efficiency: ChunkKV with layer-wise reuse consistently delivers the best efficiency gains, particularly for long-context scenarios up to 16k tokens.The table caption reports latency and throughput improvements relative to the FullKV baseline.

C Theoretical Understanding

The theoretical analysis interprets KV cache compression through in-context learning, modeling sparsification as noise that affects distinguishability and prediction risk. It argues that preserving continuous chunks can reduce this noise for complete examples, while explicitly leaving a rigorous analysis of some effects for future work.

  • Theoretical Motivation: The analysis uses in-context learning to explain why continuous-sequence KV preservation can outperform sparse-token preservation.The stated goal is to interpret ChunkKV’s advantage through semantic information preserved in complete examples.
  • In-Context Learning Setup: The framework defines prompts as sequences of training examples followed by a test example, with greedy decoding producing the most likely prediction.The predictor and its expected 0-1 error are defined over the prompt distribution.
  • Sparsification as Noise: KV cache sparsification is modeled as noise that enlarges the distribution mismatch and the distinguishability requirement as the sparsity ratio increases.The analysis introduces an error term ξθ(r) associated with sparsification and states that more discarded cache increases this noise.
  • Token-Level Compression: Token-level sparsification can disrupt dependencies within training examples, causing subsequent tokens to require larger KL divergence for distinguishability.The analysis notes that noise on earlier tokens influences distinguishability for following tokens and uniformly weakens the risk bound.
  • Chunk-Level Compression: ChunkKV preserves continuous windows as complete examples, reducing ξθ(r) for those examples and lowering the corresponding 0-1 risk bound.Less important examples may be discarded, while clean and more related examples remain available for identifying the test input.
  • Scope of Analysis: The paper does not provide a rigorous proof for how KV sparsity changes distinguishability requirements or how token-level KL differences affect the risk bound.These effects are identified as future work.

E Statistics of Models

This section provides configuration information for the language models evaluated in the experiments.

  • Model Configurations: Table 27 lists the configuration parameters for the LLMs used in the experiments.The supplied passage identifies the table’s purpose but does not enumerate individual configurations.

F Statistics of Datasets

This section documents the evaluation datasets, prompts, and study-reporting context. LongBench spans multiple long-context task types, while accompanying tables provide dataset statistics and prompt materials.

  • Dataset Statistics: LongBench evaluates extended-document and complex-sequence capabilities across 17 datasets covering question answering, summarization, few-shot learning, synthetic tasks, and code generation.The benchmark is designed for multi-task evaluation of long-context inputs.
  • Dataset Statistics: Table 28 reports dataset statistics, including training and test sample counts, while NIAH test-set size varies with context length and step size.The caption identifies the meanings of # TRAIN and # TEST and notes that NIAH evaluations typically contain around 800 samples.
  • Prompt Materials: Tables 29 and 30 provide demonstration prompts and GSM8K chain-of-thought prompt exemplars used in the experiments.These materials document the prompting setup for Figure 1 and GSM8K evaluation.
  • Study Context: The study uses publicly available models and datasets and reports no human-subject, individual-data, or protected-group experiments.The authors also state that they sought fair and non-misleading experimental reporting.
  • Broader Impacts: The authors acknowledge that more efficient language models may increase deployment and usage, creating both positive and negative societal implications.They call for continued research and discussion on responsible development and application.

I Limitations

ChunkKV’s chunk-level compression has limitations when every token must be retained, when task needs differ, and when fixed chunk boundaries constrain semantic integrity. These boundaries motivate task-adaptive granularity and potentially linguistic chunking, with added latency costs.

  • Scope boundary: ChunkKV may lose crucial information in legal or biomedical analysis when every token requires absolute semantic fidelity.Its indivisible chunks prioritize semantic coherence and core gist over verbatim retention of every detail.
  • Task dependence: Hybrid compression reveals a task-dependent trade-off: pure ChunkKV excels at local retrieval, while token-level combinations can better support global understanding.The reported contrast appears between Single- and Multi-Document QA and summarization.
  • Chunk boundaries: Fixed-size chunks improve computational efficiency but may provide less semantic integrity than adaptive boundaries based on linguistic cues.Adaptive boundaries could add inference latency that must be managed carefully.
Loading 2502.00299v5…