Source-linked AI summary
TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Hong Chen, Yudong Zeng, Yongwei Huang, Zuhao Ouyang, Junyan Zhang, Xuming Hu
TL;DR
Long-context KV-cache memory limits inference, but attention-based eviction does not reliably track which context tokens causally support answers. TwinKV uses near-duplicate key vectors as a training-free, attention-free repair pass over existing eviction policies. Across benchmarks, it improves many configurations while showing clear boundaries on some baselines and task structures.
Problem
Long-context KV-cache memory is a bottleneck, while existing attention-based eviction methods rely on a token-importance premise that the paper tests as limited.
Method
TwinKV detects pairwise key redundancy and swaps evicted orphans with retained redundant donors while preserving the existing policy’s budget and scoring rule.
Results
Across four benchmark suites and three compression ratios, TwinKV improves most configurations for StreamingLLM and PyramidKV on Qwen3-4B, is near-even for SnapKV, and helps a minority for ExpectedAttention.
Takeaways & Limitations
TwinKV is most useful as a composable correction to structural gaps in existing eviction decisions rather than as a replacement policy.
Takeaways & Limitations
TwinKV adds a genuine O(n^2d) per-head, per-layer prefill computation and does not help few-shot classification exemplars on either evaluated model.
Abstract
from arXiv · showhide
Long-context inference is bottlenecked by the memory footprint of the key-value (KV) cache, especially for small models under tight resource budgets. Existing KV cache eviction methods score tokens using the model's attention distribution or, in attention-free variants, each key's distance from a global reference point. Using a controlled leave-one-out probe, we find that attention magnitude is unrelated to a token's causal contribution to the answer (Spearman $ρ=-0.004$), challenging the premise behind dominant eviction methods. We introduce TwinKV, a training-free, attention-free redundancy signal that detects whether a token's key has a near-duplicate elsewhere in context. Rather than replacing existing policies, TwinKV acts as a composable repair pass: given a policy's fixed retained set, it identifies evicted tokens with no surviving duplicate (\emph{orphans}) and retained tokens whose information is duplicated elsewhere (\emph{redundant donors}), then swaps them while preserving the original budget and scoring rule. We compose TwinKV with four recent eviction policies across LongBench, LooGLE, RULER, and a short-context MMLU-Pro no-harm control at compression ratios ${0.3,0.5,0.7}$. On Qwen3-4B, TwinKV improves a majority of configurations for two policies, is near-even for a third, and helps only a minority for a fourth adaptive baseline already near a performance ceiling; gains across the three non-ceiling policies are smallest at the loosest ratio. On RULER with Llama-3.2-1B, however, that fourth policy improves in every evaluated cell because its Alone score leaves substantial room to improve. More broadly, Llama-3.2-1B shows a smaller average LongBench gain but a higher fraction of improved cells on LongBench and LooGLE than Qwen3-4B, plus a clean RULER win. We also identify few-shot classification exemplars as a task structure where TwinKV does not help on either model.
1 INTRODUCTION
Long-context KV caches create a severe memory bottleneck, while attention magnitude does not reliably identify tokens’ causal value. TwinKV addresses this gap with an attention-free redundancy signal used as a composable, budget-preserving repair pass, improving many but not all evaluated configurations.
- Motivation: KV-cache memory grows with sequence length and is especially restrictive for small, resource-constrained models.Each new token adds key and value vectors at every attention layer, and cache memory can exceed parameter memory before the context window is exhausted.
- Motivation: Attention magnitude is statistically unrelated to causal contribution in the controlled leave-one-out probe (Spearman ρ = −0.004, p = 0.96).The probe compares received attention with the drop in answer log-probability after removing a context chunk.
- TwinKV: TwinKV identifies redundancy through near-duplicate key vectors, providing a training-free and attention-free importance signal.Tokens with equivalent copies elsewhere can be pruned, while unique tokens are treated as less safely removable.
- TwinKV: TwinKV operates as a repair pass that swaps unrecoverable evictions for redundant retentions without changing the wrapped policy’s budget or scoring rule.It uses only the retained set and budget, rather than requiring access to another compressor’s internal scoring mechanism.
- Evaluation: On Qwen3-4B, TwinKV improves a majority of configurations for StreamingLLM and PyramidKV, is near-even for SnapKV, and helps a minority for ExpectedAttention.The reported improved-cell shares are up to 64%, 57%, 41%, and 16%, respectively; gains across the three non-ceiling policies are smallest at the loosest ratio tested.
2 MOTIVATION
The paper tests whether attention reflects causal utility and finds no meaningful correlation in long-context QA. It therefore reframes eviction around recoverability from duplicate context content, while showing that redundancy varies across content types and can require architectural adaptation.
- 2.1 ATTENTION DOES NOT TRACK CAUSAL UTILITY: Attention-based eviction assumes that strongly attended tokens are tokens the model needs, but this premise is tested against output-level causal utility.The leave-one-out measure asks how much removing a context chunk reduces the model’s log-probability for the gold answer.
- 2.1 ATTENTION DOES NOT TRACK CAUSAL UTILITY: Attention and marginal utility are uncorrelated across 136 chunks (Spearman ρ = −0.004, p = 0.96).Low-attention chunks span the full range of true utility, including highly useful chunks.
- 2.2 FROM ATTENTION TO REDUNDANCY: The alternative criterion is recoverability: duplicated facts can survive eviction through another copy, whereas uniquely stated facts have no fallback.This shifts the decision from estimating importance to asking whether the remaining context can still provide the information.
- 2.2 FROM ATTENTION TO REDUNDANCY: TwinKV operationalizes recoverability with non-adjacent near-duplicate keys, treating tokens with many twins as redundant and tokens with none as irreplaceable.The local exclusion avoids mistaking smooth nearby structure for meaningful cross-context redundancy.
- 2.2 FROM ATTENTION TO REDUNDANCY: Across six LongBench content types, 37–47% of tokens have at least one twin, with the fraction highest for code and lowest for multi-hop QA.TREC has 42.7% twin-bearing tokens but unusually low across-document variance (±0.5%), reflecting repeated exemplar templates.
- 2.2 FROM ATTENTION TO REDUNDANCY: The redundancy principle must be adapted to positional-encoding choices to remain valid across model families.The paper specifically notes that architectural positional encoding affects how the underlying signal should be applied.
3 METHOD
TwinKV scores redundancy through pairwise key similarity and uses that signal to repair an existing eviction policy without changing its retained budget. The method contrasts local pairwise redundancy with global distinctiveness while accounting for protected regions, architecture-dependent thresholds, and added prefill cost.
- KV cache setup: KV eviction retains |S| = ⌈(1 −ρ)n⌉ key-value positions per head and layer under compression ratio ρ.The policy selects a budget-constrained subset during prefill and discards the remaining positions before generation.
- Redundancy scoring: TwinKV computes pairwise cosine similarity between ℓ2-normalized key vectors to identify non-adjacent near-duplicate keys.The local exclusion window prevents adjacent semantic drift from being treated as cross-context redundancy.
- Redundancy scoring: Tokens with lower redundancy receive higher scores and are retained, while more redundant tokens are evicted under the same budget.The scorer uses a negative, normalized redundancy count and protects sink tokens plus the trailing recent window regardless of score.
- Design rationale: Unlike global-centroid scoring, pairwise redundancy asks whether an equivalent copy exists anywhere else, preserving unique facts that may not be global outliers.This distinction matters when a question-answering fact is stylistically ordinary but unrepeated in the passage.
- Composable repair: TwinKV audits an arbitrary policy’s retained set by identifying orphaned evictions and redundant donors, then swaps them without changing K.The repair pass preserves the wrapped policy’s compression ratio, unlike chaining a second compressor, which multiplies retention fractions.
- Complexity: TwinKV adds a genuine O(n^2d) per-head-per-layer similarity computation during prefill, although restricting comparisons to retained positions could reduce repair cost to O(nKd).The reported implementation computes the full similarity matrix before pruning, adding this expense to the wrapped policy’s cost.
4 EXPERIMENTS
Experiments evaluate TwinKV as a fixed-budget repair pass across four eviction policies, two models, four benchmark suites, and compression ratios {0.3, 0.5, 0.7}. Results are policy-, model-, and task-dependent: TwinKV often helps non-ceiling settings, can hurt strong adaptive baselines, and is unreliable for short-context few-shot classification.
- Setup: Experiments pair TwinKV with StreamingLLM, PyramidKV, SnapKV, and ExpectedAttention across LongBench, LooGLE, RULER, and MMLU-Pro.Both wrapped-policy-alone and repaired conditions are evaluated at compression ratios {0.3, 0.5, 0.7}.
- LongBench and LooGLE: 26.03 →31.30 at ratio 0.3 is StreamingLLM’s largest Qwen3-4B LongBench gain, shrinking to 23.57 →26.07 at ratio 0.7.PyramidKV and SnapKV shift from small losses at 0.3 to small gains by 0.7, while ExpectedAttention loses at every ratio.
- LongBench and LooGLE: 24.75 →31.45 is PyramidKV’s largest Qwen3-4B LooGLE gain, while ExpectedAttention loses at every ratio.On Llama-3.2-1B, ExpectedAttention instead wins at every LooGLE ratio, reaching 15.78 →19.70 at ratio 0.3.
- Limitations and controls: 64.50 →23.00 is ExpectedAttention’s Qwen3-4B TREC regression at ratio 0.7, showing that few-shot classification exemplars can trigger a false-twin failure mode.The default setting causes sharp regressions on every wrapped policy, although a wider local window and higher threshold can make the interaction flat or positive.
- RULER: 9/9 RULER cells improve for PyramidKV and SnapKV on Qwen3-4B, but ExpectedAttention loses all 9 cells with mean −6.72.On Llama-3.2-1B, all four policies improve in every cell, including ExpectedAttention with mean +4.66.
- Cross-architecture generalization: +0.17 versus +0.46 is Llama-3.2-1B’s versus Qwen3-4B’s mean LongBench improvement, despite a higher positive-cell fraction of 54.4% versus 41.7%.On LooGLE, Llama-3.2-1B leads on both mean improvement (+1.21 versus +0.80) and positive-cell fraction (52.1% versus 37.5%).