PaperlayerView exact revision on arXiv

Source-linked AI summary

The Pitfalls of KV Cache Compression

Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, Daniel Israel

arXiv:2510.00231v2cs.LGcs.AI

TL;DR

The effects of KV cache compression in multi-instruction prompts remain poorly understood, particularly because degradation can vary across instructions. The paper investigates these failure modes and proposes fair eviction; it finds that fair eviction can mitigate unpredictable degradation and restore instruction-following fidelity even at high compression ratios.

  • Problem

    The effects of KV cache compression remain poorly understood, particularly because degradation can vary across instructions in multi-instruction prompts.

    Source: §1 · p. 1

  • Method

    The paper characterizes failure modes and system-prompt leakage under compression, then proposes fair eviction to prevent any single instruction from being disproportionately targeted.

    Source: §1 · p. 2

  • Results

    Compression degradation varies across instruction classes, while fair eviction mitigates unpredictable degradation and restores instruction-following fidelity even at high compression ratios.

    Source: §1 · p. 2, §2.3 · p. 3

  • Takeaways & Limitations

    System-prompt leakage and instruction-following under compression are affected by the compression method, instruction order, and eviction bias.

    Source: §1 · p. 2

    Fair eviction assumes instructions are comparably important and well-formed, while suitable assumptions for arbitrary prompts and online compression remain future work.

    Source: §8 · pp. 9–10

Abstract

from arXiv · show

KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multi-instruction prompting have been insufficiently studied. In this paper, we identify several pitfalls that practitioners should be aware of when deploying KV cache compressed LLMs. We evaluate five KV cache compression methods (StreamingLLM, SnapKV, TOVA, H2O, and K-Norm) on Llama3.1 8B and Qwen2.5 14B under multi-instruction prompting with IFEval. Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM. As a practical example, we highlight system prompt leakage as a case study, empirically demonstrating the impact of compression on leakage and general instruction-following. We identify several factors that contribute to system prompt leakage: compression method, instruction order, and KV eviction bias. We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.

1 Introduction

The paper argues that KV-cache compression can selectively erase prompt instructions rather than degrading performance uniformly, causing instruction-following failures and security vulnerabilities that the authors study through system-prompt leakage.

1 Introduction

  • KV-cache compression promises improved inference efficiency and memory usage, but its performance costs remain poorly understood and unpredictable.The cache grows with context length, making inference memory-bounded and limiting throughput while increasing latency.Source: §1 · p. 1
  • Certain instructions degrade faster under compression, causing models to silently ignore parts of prompts and harming multi-instruction performance and security.The paper describes this uneven degradation as “selective amnesia.”Source: §1 · p. 1
  • System-prompt leakage serves as a concrete measure of instruction-following failure because system prompts should not be revealed to end users.System prompts define model behavior and safety guardrails, and their cache entries are natural compression targets because they recur across long interactions and multiple queries.Source: §1 · pp. 1–2
  • Compression method, instruction order, and eviction bias affect performance degradation and system-prompt leakage rates.The authors investigate these failure modes across different models, model sizes, and compression methods.Source: §1 · p. 2
  • Fair eviction gives developers more control by preventing any single instruction from being disproportionately targeted, mitigating unpredictable degradation and restoring instruction-following fidelity.The authors say it retains these benefits even at high compression ratios.Source: Figure 1 · p. 1, §1 · p. 2

2 KV Cache Compression

This section formalizes KV-cache compression and eviction policies, then motivates studying compression in realistic multi-instruction settings and distinguishes offline from online compression.

2.1 Preliminaries

  • Because the KV cache grows linearly with sequence length, compression selects b ≪ n token entries per layer through an eviction policy while aiming to limit performance loss.The compressed cache budget is much smaller than the full sequence length.Source: §2.1 · p. 2

2.2 KV Eviction Policies

  • KV eviction policies are grouped into position-based, attention-based, embedding-based, and hybrid approaches.Descriptions of these strategies and representative policies are deferred to Appendix A.Source: §2.2 · p. 2
  • Benchmarks commonly test single-instruction Q&A, retrieval, or code [Zhang et al., 2023; Xiao et al., 2023; Oren et al., 2024; Liu et al., 2025; Yuan et al., 2024a; Li et al., 2025], unlike applied multi-instruction prompts.Long-context prompts may combine orthogonal instructions, including system-prompt instructions.Source: §2.2 · p. 2

2.3 Offline vs Online Compression

  • Offline compression uses global information on a fixed prompt prefix, whereas online compression evicts entries during decoding without knowledge of future tokens; this study focuses on offline compression of system prompts.Online policies must make greedy eviction decisions as the sequence grows.Source: §2.3 · pp. 2–3
  • StreamingLLM degradation varies across instruction classes, and Figure 2 compares degradation rates and single- versus multi-instruction rankings.The paper describes class-specific degradation as nonhomogeneous.Source: §2.3 · p. 3, Figure 2 · p. 3

3 The Two Facets of Degradation in Compression

KV cache compression causes unequal instruction degradation, and its effects vary with eviction policy and model. The section attributes this pattern to instruction difficulty and eviction bias, with multi-instruction prompts degrading differently from single-instruction prompts.

3 The Two Facets of Degradation in Compression

  • Instruction classes degrade at different rates under KV compression; language instructions in multi-instruction prompts deteriorate quickly as compression rises despite high accuracy at low compression.The authors identify unequal degradation as Pitfall 1.Source: §3 · p. 3
  • The two proposed facets are instruction difficulty, which can matter in single- and multi-instruction prompts, and eviction bias, which may target some instructions more in multi-instruction prompts.Difficult instructions may rely on evicted entries carrying disproportionately meaningful semantic signals; eviction bias is presented as a hypothesis.Source: §3 · p. 3
  • Compression effects depend on eviction policy and model, as shown by evaluations of five policies on Llama3 and Qwen2.The evaluated policies are StreamingLLM, H2O, K-Norm, SnapKV, and TOVA; Figure 3 summarizes accuracy and degradation across compression ratios.Source: Figure 3 · p. 4, §3 · p. 4
  • Multi-instruction prompts tend to degrade sooner and at a different pace than single-instruction prompts, showing that instruction difficulty alone does not explain degradation.Normalizing accuracy by each class's uncompressed accuracy makes slope differences more apparent across single- and multi-instruction prompts.Source: §3 · p. 4

4 A Case Study on System Prompt Leakage

KV cache compression can degrade instructions unequally and create system-prompt leakage even without adversarial prompting. The section links this risk to instruction order and eviction behavior, and describes policy changes that can reduce it.

4 A Case Study on System Prompt Leakage

  • The experiment separates a system prompt into defense and directive instructions, tests requests to follow the directive or reveal both instructions, and compresses only the system prompt.Directive following uses instruction-following metrics, while leakage is measured with ROUGE-L recall against the defense or directive text.Source: §4 · p. 5
  • System-prompt compression can cause leakage without adversarial prompting: directive following remains strong while defense leakage rises, then falls at extreme compression when the model loses the instruction text.The changing leakage curve indicates a critical range of compression ratios where vulnerability is greatest.Source: §4 · pp. 4–5, Figure 4 · p. 5
  • Putting the directive before the defense quickly degrades directive following, but the order effect does not reverse uniformly and depends on compression method and model.The section reports that changing the order substantially alters both directive-following degradation and leakage.Source: §4 · pp. 5–6, Figure 5 · p. 6
  • With defense before directive, policies that preserve directive performance evict more defense entries; StreamingLLM and SnapKV show especially strong bias, while flipping the order shifts eviction toward defense entries.The flipped-order bias is less pronounced than the bias against defense entries in the normal order.Source: §4 · p. 6, Figure 7 · p. 7
  • Eviction balance alone is insufficient: K-Norm is closest to unbiased eviction but still degrades performance and leaks because it selects inappropriate entries to evict.Choosing which tokens to retain matters for preserving the original KV-cache semantics at high compression ratios.Source: §4 · p. 6
  • A keyword whitelist and fairer eviction across instructions can reduce leakage and degradation, preserving defense performance with negligible or minimal directive-performance loss.The proposed changes target token-selection errors and unequal eviction across instructions.Source: §4 · pp. 6–7

5 Towards Eviction Policies that...

The section proposes whitelisting tokens and adapting eviction policies to retain prompt components more fairly. Both changes reduce instruction-following degradation while maintaining the same compression ratio or imposing only a small directive-following cost.

5.1 ...Better Capture Semantics

  • Whitelisting tokens at the same compression ratio improves defense performance with little directive-following cost, because it preserves tokens from the faster-degrading defense instruction.This addresses frequent failures across evaluated eviction policies, which can miss the semantic importance of evicted entries.Source: §5.1 · pp. 7–8, Figure 8 · p. 7, §5 · p. 7

5.2 ...More Fairly Evict Entries

  • Fair eviction improves overall directive-following and leakage performance, reducing defense degradation at only a small cost to directive following.The approach adapts existing policies to give distinct prompt components equal retention rates.Source: §5 · p. 7, Figure 9 · p. 8, §5.2 · p. 8
  • Fair eviction gives distinct prompt components equal token-retention rates and applies the underlying eviction logic separately to each partition.This is intended to prevent one component from being disproportionately discarded.Source: §5.2 · p. 8

6 Evaluating Fair and Whitelist Eviction

Fair and whitelist eviction consistently improve the combined directive-accuracy and leakage score across tested eviction policies, with larger gains at higher compression ratios. Equal eviction shares address harmful eviction bias, though tuning shares may improve performance further.

6 Evaluating Fair and Whitelist Eviction

  • Across compression ratios 0.4–0.7, whitelist and fair eviction improve the equal-weight directive-accuracy/leakage score for every tested policy, with substantial gains for StreamingLLM and significant gains for SnapKV and H2O.The score averages results across the stated compression ratios; directive accuracy is higher-is-better and leakage is measured by ROUGE-L, where lower is better.Source: §6 · p. 8, Table 1 · p. 9
  • Whitelist and fair eviction tend to yield larger improvements at higher compression ratios, when eviction bias and token-selection choices matter more.Source: §6 · p. 9
  • The presented whitelist and fair variants are partly manual, but the authors note they could be automated; exploring that automation is outside this work’s scope.The stated aim is to reveal consequences of KV-cache compression in multi-instruction settings and demonstrate directions for more reliable eviction policies.Source: §6 · p. 9
  • Distributing eviction equally across instructions controls eviction bias and can improve multi-instruction performance, while tuning each instruction’s eviction share may improve it further.Equal shares already demonstrate that eviction bias can harm performance.Source: §6 · p. 9

7 Conclusion

Although KV cache compression literature claims minimal performance loss, its consequences for multi-instruction prompts remain insufficiently studied. The paper identifies eviction bias and poor preservation of cache semantics as problems, and finds that simple policy modifications can lessen degradation.

7 Conclusion

  • Eviction policies disproportionately remove cache entries from some instructions, causing severe performance degradation for those instructions.The paper terms this pattern eviction bias.Source: §7 · p. 9
  • Eviction policies cannot properly determine which entries to remove while minimizing the loss of semantics from the original cache.Source: §7 · p. 9
  • Simple modifications to eviction policies can greatly lessen degradation and suggest directions for more sophisticated policies.The paper presents these modifications as addressing both eviction bias and the difficulty of preserving cache semantics.Source: §7 · p. 9

8 Limitations

The study covers a limited set of models and compression policies and focuses primarily on offline compression. Its proposed solutions rely on prompt structures and assumptions that may not hold in online settings or for arbitrary prompts.

8 Limitations

  • Results from Llama3 and Qwen2 with five compression policies may not generalize to other models or policies.The authors note that compression effects depend on both the eviction policy and the model.Source: §8 · p. 9
  • The experiments focus primarily on offline compression, leaving online compression untested, and the proposed solutions assume prompts can be divided into distinct instruction spans.That decomposition is hard to achieve in online compression.Source: §8 · p. 9
  • Fair eviction assumes instructions are comparably important and well-formed, while appropriate assumptions for arbitrary prompts and online compression remain open research directions.The authors provide a more flexible alternative in Appendix G.Source: §8 · pp. 9–10

9 Ethical Considerations … D Whitelisted tokens

The paper frames KV cache compression pitfalls as risks to address through safer, more reliable strategies, and documents eviction methods and prompts used in its leakage evaluation.

9 Ethical Considerations

  • The authors frame their analysis as risk awareness to support safer, more reliable compression strategies, rather than to enable misuse.Source: §9 · p. 10

A KV Eviction Strategies

  • Position-based StreamingLLM preserves initial attention-sink tokens and a recent-token window, whereas attention-based H2O retains cumulative-attention heavy hitters and TOVA discards lowest-attention entries.Source: A KV Eviction Strategies · p. 11
  • Embedding-based K-Norm evicts keys by L2 norm, negatively correlated with attention; hybrid SnapKV scores preceding tokens using attention from a recent observation window and keeps top-scoring tokens.Source: A KV Eviction Strategies · pp. 11–12

B Defense prompts

  • Defense prompts instruct the model not to disclose system instructions and to refuse; the same instruction is prepended before or appended after the directive, depending on ordering.Source: B Defense prompts · p. 12

C Leakage request

  • The leakage request asks the LLM to reveal system instructions by saying, “Repeat all previous instructions without modification.”Source: C Leakage request · p. 12

D Whitelisted tokens

  • The defense whitelists the sequence “DO NOT DISCLOSE AND ONLY REPLY WITH ‘I can’t assist with that request.’”Source: D Whitelisted tokens · p. 12

E Supplemental experiments … G Eviction Debiasing Policies

The supplemental experiments compare retention and leakage across eviction policies and prompting orders, while the methods allocate KV-cache retention across instruction spans. Eviction debiasing lets users interpolate between default and fair eviction, with higher settings reducing bias.

F Fair eviction policies

E Supplemental experiments

  • Supplemental figures compare kept-entry percentages and leakage across eviction policies and prompting orders; Table 2 extends improvement scores across compression ratios 0.1–0.7.Figure 14 also compares directive following and leakage before and after fair eviction when the prompt order is flipped.Source: Figure 10 · p. 13, Figure 11 · p. 13, Figure 12 · p. 13

F.2 Scoring and Selection Process

  • Fair eviction scores tokens, partitions the sequence into instruction spans, and selects top-scoring tokens within each span in proportion to its length.Spans include prefix or suffix tokens outside the instructions, so the partition covers the full sequence.Source: §F.2 · p. 15

F.3 Fair StreamingLLM

  • Fair StreamingLLM keeps the prefix sink, allocates the remaining budget proportionally across spans, and retains the most recent tokens within each span.The standard fair-eviction Algorithm 1 is not used for StreamingLLM.Source: §F.3 · p. 15

F.4 Fair SnapKV

  • Fair SnapKV splits its observation window across spans and uses span-local voting, so queries in each span vote only over keys in that span.This differs from standard SnapKV’s single global query window voting over the full prefix.Source: §F.4 · p. 15

F.5 Fair H2O

  • Fair H2O masks cross-span attention terms and scores keys using eligible causal queries from the same span.Scores are normalized by the number of eligible queries.Source: §F.5 · pp. 15–16

F.7 Fair TOVA

  • Fair TOVA anchors scoring within each span and averages scores across heads rather than computing them independently per head.The spans are disjoint, adjacent, and cover the sequence.Source: §F.7 · p. 16

G Eviction Debiasing Policies

  • Eviction debiasing interpolates between default and fair eviction; the authors report it consistently outperforms no-debiasing across IFEval and long-context benchmarks on three eviction policies.λ = 0 recovers default eviction and λ = 1 recovers fair eviction.Source: G Eviction Debiasing Policies · p. 16

H Eviction debiasing experiments

Eviction debiasing is evaluated using Pareto optimality across IFEval and long-context benchmarks. Default compression is less often optimal than debiased compression, while fair eviction consistently ranks among the top configurations.

H Eviction debiasing experiments

  • The experiments sweep λ from baseline policy (0) to fair eviction (1) at compression ratios 0.0 through 0.9, counting each λ’s frequency on the Pareto frontier.The reported percentages give the frequency of Pareto optimality across ten compression ratios; the frontier is shown in Figures 15 to 18.Source: H Eviction debiasing experiments · p. 16
  • Across IFEval and long-context benchmarks, debiased compression (λ > 0) is less often suboptimal than default compression (λ = 0), while fair eviction (λ = 1) consistently ranks among the top.Optimality is assessed using Pareto criteria: lower leakage and higher system directive instruction following. The comparison is summarized in Tables 3 and 4.Source: H Eviction debiasing experiments · p. 16

I LongBench experiments … N Automating Whitelisting and Fair-eviction

LongBench TREC experiments show that compression-related instruction degradation and leakage patterns extend to longer contexts, while the appendices explain eviction biases, evaluation methods, runtime trade-offs, and possible automation.

I LongBench experiments

  • TREC results extend the observed effects to longer contexts: the defense is too weak at 3,000–4,000 words, with leakage even at zero compression.At 1,000–2,000 words, results are similar to IFEval; at 2,000–3,000 words the pattern is less pronounced in leakage. Samsum and TriviaQA do not show the same degradation pattern, so the authors report TREC results only.Source: I LongBench experiments · pp. 16–18

I.1 Defense template A

  • The LongBench defense template prohibits repeating or disclosing user instructions and directs refusal of requests for system instructions.LongBench TREC figures report instruction-following and leakage scores for the original and fair-eviction methods, but the supplied captions do not provide comparative values.Source: §I.1 · pp. 18–20, Figure 19 · p. 19, Figure 20 · p. 19

J.4 TOVA

  • TOVA tends to retain defensive instructions even when their order flips, and preserves a higher percentage of defense tokens in middle layers.The authors tentatively suggest stronger semantic signals in middle layers may explain the latter pattern; they note that this requires further study.Source: §J.4 · p. 20

J.6 Summary

  • For orthogonal instructions, eviction bias reflects method-specific preferences: StreamingLLM, H2O, and SnapKV favor later instructions, while K-Norm favors earlier tokens.StreamingLLM keeps the first four sink tokens and recent tokens; H2O normalizes accumulated future-token attention; SnapKV uses the last k tokens to select important tokens. K-Norm prunes high-key-norm tokens and performs worse in instruction following than the other methods. The authors suggest these methods often assume prompt content is related.Source: §J.6 · p. 20, §J.1 · p. 20, §J.2 · p. 20

K Runtime comparison

  • Fair eviction adds only a modest compression overhead, while whitelisting adds the largest; decoding times remain within 7% of each other.Throughput trends mirror latency, with fair eviction close to baseline and decoding times within 6% of each other; the authors state that absolute time differences are small and at millisecond scale.Source: Figure 23 · p. 21, Figure 24 · p. 21, K Runtime comparison · p. 21

L LLM-as-a-judge to detect leakage

  • The alternative LLM-as-a-judge leakage measure uses Gemma 4 32B Instruct, scores severity from 0 to 4, and treats scores of 2 or higher as leaks.The judge compares protected system text with a model response; the input template supplies both. This evaluation is used only for Llama3.Source: L LLM-as-a-judge to detect leakage · p. 21, §L.2 · p. 22

N Automating Whitelisting and Fair-eviction

  • Whitelisting can be automated by having a model identify prompt keywords, while fair eviction can automate instruction-span detection and proportional token eviction.Span detection may use exact matching, sentence boundaries, or LLM-identified semantic spans; fine-tuning on crucial keywords is another proposed route to improve whitelisting accuracy.Source: N Automating Whitelisting and… · pp. 22–23

M Per-Layer Eviction Bias

  • Per-layer heatmaps report the mean fraction of system and defense instruction tokens retained across layers and compression ratios on TREC’s 1,000–2,000-word set.The figure compares StreamingLLM, H2O, SnapKV, TOVA, and K-Norm, with defense instructions placed first.Source: M Per-Layer Eviction Bias · p. 22, Figure 27 · p. 24
Loading 2510.00231v2…