R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

Motivation

• Reasoning models impose unsustainable memory demands as autoregressive generation rapidly grows the KV cache.A DeepSeek-R1-Distill-Llama-8B example uses 4.1GB for KV cache after generating 32K tokens.Source: §1 · p. 1
• Reasoning traces contain superfluous reflections, re-evaluations, and self-dialogue, with over half of R1-chain tokens contributing minimally to task performance.These patterns motivate compressing repetitive self-verification and intermediate calculations without compromising reasoning accuracy.Source: §1 · pp. 1–2
• Attention-only compression can retain duplicative self-reflections while pruning scattered reasoning details because repetitive sections generate high attention signals.R-KV therefore targets important and non-repetitive context rather than low-attention tokens alone.Source: §1 · p. 2

Method

• R-KV combines attention-based importance scoring, dynamic redundancy estimation from key vectors, and joint eviction.The joint mechanism balances token importance against repetition during decoding.Source: §1 · p. 2

Results

• 10–34% KV-cache retention achieves performance parity with the uncompressed model, while existing compression baselines reach only 60% of performance.On AIME-24, R-KV reaches 105% of the full-KV baseline using around 16% of the cache.Source: §1 · p. 2

Implications

• R-KV is training-free and model-agnostic, supporting deployment in reinforcement-learning rollouts and LLM serving.The method addresses the deployment trade-off between reasoning capability and practical memory constraints.Source: §1 · p. 2

2.1 Redundancy in Reasoning Models

• Reasoning models generate more than 8× longer outputs than ground truth on two mathematical reasoning datasets.Figure 2 compares DeepSeek-R1-Distill-Llama-8B, Qwen-7B, and Qwen-14B with MATH-500 and AIME 2024 ground truth.Source: §2.1 · p. 2, Figure 2 · p. 3
• Generated reasoning has 5–7× higher average 1- to 2-gram frequency than ground truth, indicating greater repetition.The repetition pattern appears consistently across the compared mathematical reasoning datasets.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Failure of Existing KV Compression Methods to Handle Redundancy

• SnapKV selects many repeated self-reflection and conclusion tokens because repetitive content receives disproportionately high attention scores.Figure 3 visualizes selected cached tokens inside red boxes; brighter colors indicate higher attention scores.Source: §2.2 · p. 3, Figure 3 · p. 3
• Attention-based methods can over-retain duplicative content while missing scattered reasoning details when selecting tokens primarily by contextual importance.This failure motivates explicitly accounting for redundancy during KV-cache selection.Source: §2.2 · p. 3

3 Redundancy-aware KV Cache Compression (R-KV)

• R-KV targets decoding-time compression for reasoning models, where generated outputs are significantly longer than prompts.Unlike existing methods focused on prefilling long-context inputs, R-KV addresses long generation outputs.Source: §3 · p. 3, §3.1 · p. 4

3.2 Importance Scoring via Attention Weights

• Attention-based importance scoring estimates which key tokens are critical from attention received by the latest α observation tokens.For GQA, query-head scores are max-pooled across each shared key/value-head group and renormalized; sliding-window max pooling stabilizes outlier importance values.Source: §3.2 · p. 4

3.3 Redundancy Estimation via Semantic Similarity

• Cosine similarity between key vectors identifies tokens whose semantic content overlaps with other cached tokens, assigning higher redundancy scores to more similar tokens.The method normalizes key vectors, computes pairwise similarity, zeros self-similarity, and retains up to β recent tokens among highly similar ones.Source: §3.3 · p. 5

3.4 Joint Selection Strategy for KV Cache Retention

• The joint selection score combines importance and redundancy under a per-head token budget, favoring tokens that are important and semantically diverse.The trade-off is controlled by λ, with higher importance increasing retention priority and higher redundancy decreasing it.Source: §3.4 · p. 5
• R-KV evaluates against SnapKV and FullKV on MATH-500 and AIME24 for R1-Llama-8B and R1-Qwen-14B using pass@1 from 64 responses per question.The supplied figure caption specifies the compared methods, datasets, models, and evaluation protocol but does not provide numeric outcomes.Source: Figure 4 · p. 6

Evaluation Setup

• Evaluation uses MATH-500 and AIME 2024 with sampled pass@1 from 64 responses per question.Source: §4.1 · pp. 6–7

• R-KV significantly outperforms SnapKV, with up to 40% accuracy improvement.Source: §4.2 · p. 7

Ratio Budget

• R-KV achieves lossless compression at 34% and 10% KV-cache budgets for R1-Llama-8B on MATH-500 and AIME-2024, respectively.Source: §4.2 · p. 7
• 105% of FullKV accuracy is achieved at 16% KV-cache budget for R1-Llama-8B and 33% for R1-Qwen-14B.Source: §4.2 · p. 7

5.2 Failure of Attention-Based Methods to Capture Redundancy

• When λ = 0, redundancy estimation controls retention and may omit the initial tokens whose eviction can impair generation.Source: §5.1 · p. 7, Figure 5 · p. 8

5.1 Hyperparameter Analysis

• λ = 0.1 yields the highest accuracy, while redundancy-only and attention-only strategies perform worst, showing the complementarity of both metrics.Source: §5.1 · p. 7, Figure 6 · p. 8

5.2 Failure of Attention-Based Methods to Capture Redundancy

• R-KV selects more diverse, broadly distributed tokens than SnapKV, capturing richer context rather than repeatedly selecting nearby tokens.Source: §5.2 · p. 8, Figure 7 · p. 8

5.3 Efficiency Analysis

• R-KV memory remains constant with sequence length through fixed buffers, unlike FullKV’s linearly growing memory footprint.Source: §5.3 · p. 8
• At sequence length 16K, R-KV provides up to 9× larger batches and over 6.6× throughput at a 10% compression ratio.Source: §5.3 · p. 9
• At fixed budget 1024, R-KV reaches 13.4× larger batches and 9.2× throughput than FullKV.Source: §5.3 · p. 9

• Existing KV-cache work includes dynamic eviction, quantization, merging, and low-rank decomposition, with eviction methods primarily using attention-based token pruning.Source: §6 · p. 9
• Efficient-reasoning approaches use RL, supervised fine-tuning, or test-time prompting to shorten chain-of-thought, but training adds cost and prompting may reduce performance.Source: §6 · p. 9

• R-KV preserves nearly full model performance using 10–34% of the original KV cache while outperforming prior compression methods.Source: §7 · p. 10
• R-KV enables up to 13× larger batches and over 9× speedup in long-sequence generation, especially under constrained memory budgets.Source: §7 · p. 10

• R-KV is specified through a pseudo-code algorithm for its cache-compression procedure.Source: §A.1 · p. 15
• Grouped-query attention requires downscaling attention scores from query-head dimensions to shared key-value-head dimensions.The described operation uses pooling to reduce the attention-score representation.Source: §A.2 · p. 15
• All experiments were completed on NVIDIA A100 80G GPUs.Source: §B.1 · p. 15

Cache compression procedure

• R-KV retains recent observation tokens, scores earlier candidates, and selects the top-Bbudget tokens before concatenating them with the observations.Compression is triggered only when the cache exceeds the budget and buffer condition; insufficient candidates are left unchanged.Source: §B.2 · p. 16

Importance and redundancy scoring

• Importance scores come from observation-token attention, while redundancy is estimated from cosine similarity among candidate key vectors.Self-similarity is suppressed, highly similar pairs are identified, and recent-token retention is enforced before joint scoring.Source: §B.2 · p. 16

Joint selection

• The joint score combines importance and redundancy terms, then averages scores across heads to rank candidate tokens.The selected candidates are concatenated with the observation cache for the next cycle.Source: §B.2 · p. 16

Memory analysis

• The memory-saving analysis allocates separate fixed-size budget and buffer storage, with additional memory for model weights and the recent query cache.The stated saving is the FullKV memory minus budget, buffer, and query-cache allocations.Source: §C.1 · p. 17

Computation analysis

• The per-segment scoring overhead is O(αBbudget + Bbudget^2), while compressed generation costs O((Bbudget + Bbuffer)Bbuffer) instead of O(BfullBbuffer).The reduction is most useful when reasoning outputs make Bfull large and Bbudget relatively small.Source: §C.1 · p. 17

Ratio Budget

• At 10% compression, R-KV supports a 9× larger batch size and delivers 6.6× the throughput of FullKV.At the other reported ratio, throughput is 1.7× higher than FullKV; larger batches eventually become compute-bound on NVIDIA A100.Source: §C.2 · p. 18

Fixed Budget

• With an 8K output and Bbudget=1024, R-KV increases batch size 6.48× and throughput 3.8× over FullKV.For Bbudget=1536, the reported gains are 4.6× in batch size and 3× in throughput.Source: §C.2 · p. 18
• With a 16K output and Bbudget=1024, R-KV achieves a 13.4× batch-size increase and a 9.19× throughput increase over FullKV.Source: §C.2 · p. 18

Comparison with SnapKV

• Despite minor redundancy-estimation overhead relative to SnapKV, R-KV’s throughput is less than 1% lower.The comparison is reported alongside Table 3, which covers memory saving, throughput, and decoding time across generation lengths and budgets.Source: §C.3 · p. 18, Table 3 · p. 18

• R-KV’s compatibility with advanced attention mechanisms such as paged attention is currently limited.Integrating the method seamlessly with these mechanisms requires further investigation.Source: D Limitations · p. 18
• Frameworks lacking dedicated KV-compression interfaces may need to reallocate memory, potentially offsetting some acceleration gains.Frameworks with specialized interfaces can manage compressed-cache operations more efficiently.Source: D Limitations · p. 19