PatchKV: Weight-Space Compensation of KV Cache

1 Introduction

• KV-cache size grows linearly with context length and can dominate serving memory and bandwidth, while discarding distant tokens can remove information needed by later queries.The cache is especially important in single-context, multi-query workloads, where query latency and memory costs recur across requests.Source: §1 · p. 1
• Aggressive cache compression can lose facts, dependencies, or query-relevant details, motivating a complementary way to recover context information beyond the cache.Existing approaches selectively evict entries or approximate the cache, but accuracy can degrade as budgets shrink.Source: §1 · pp. 1–2
• PatchKV matches compressed-cache student activations to full-cache teacher activations with block-wise ridge regression, then applies the resulting patch to MLP output projections without downstream queries.The patch is merged for the active context, leaving the generation-time forward graph unchanged and amortizing added computation at context loading.Source: §1 · p. 2
• PatchKV consistently improves token-eviction and cache-approximation methods at aggressive budgets, including below 20%, across three backbones and long-context QA and math benchmarks.It recovers a substantial fraction of the accuracy gap between compressed-cache and full-cache inference.Source: §1 · p. 2
• PatchKV treats context-specific weight patches as a complementary mechanism that can partially recover information lost through aggressive cache compression.The paper evaluates this compensation across long-context QA and math benchmarks.Source: §1 · p. 2

2.1 Problem Formulation

• Because downstream query distributions are unavailable during compression, the method approximates them with a finite reference set derived from the context.The compressed cache is reused across queries, reducing per-query serving cost by a factor of b/n.Source: §2.1 · p. 3, §2.4 · p. 5
• PatchKV adds a context-specific weight patch to the compressed cache, shifting recovery to context prefill while preserving the base forward graph and per-token generation cost.The formulation can use any off-the-shelf prefill-stage cache compressor, making weight patching complementary to token-space compression.Source: §2.1 · pp. 3–4

2.2 Closed-Form Weight Patch Derivation

• For each block, ridge regression matches compressed-cache student activations to full-cache teacher activations across reference tokens, producing a shared patch applied sequentially through the blocks.The adaptive ridge scaling uses one λ0 shared across layers, requiring no per-layer tuning.Source: §2.2 · p. 4

2.3 Efficient Computation for Long Reference Sequences

• Long reference sequences are processed in chunks, and running sufficient statistics allow the final ridge solution without retaining activations for all Nref tokens.Chunking drops cross-chunk attention; the authors expect this approximation to be mild because reference dependencies are designed to be local.Source: §2.3 · p. 5

2.4 Reference Query Construction

• Three context-derived reference strategies are used: repeat-context reconstructs context tokens, self-study uses model-generated answers, and joint pools both sets.Repeat-context covers fine-grained context facts without sampling overhead; self-study may cover higher-level summaries and reasoning patterns but adds generation cost and may diverge more from downstream queries.Source: §2.4 · p. 5

3.2 Main Results

• At cache ratios of 0.02–0.1, PatchKV substantially narrows the gap to full-cache inference and often matches or exceeds cache-only baselines with larger budgets.On SQuAD, PatchKV recovers up to 73% of full-KV accuracy with no cache, where the cache-only baseline is near zero; Figure 3 illustrates a recovered answer.Source: §3.2 · p. 7, Figure 3 · p. 8
• Across KVzip and Attention Matching, PatchKV gains hold across context lengths and task types, while reference-query strategies show different strengths by task.Self-study is strongest on contextual QA and redundancy-heavy tasks; repeat-context and the joint variant are stronger on retrieval-heavy tasks.Source: §3.2 · p. 7, §3.1 · p. 7

3.3 Analysis

• On SQuAD, patches narrow the full-versus-compressed-cache perplexity gap on downstream queries across budgets, although they do not fully close it, especially at low cache.Patches nearly match full-cache perplexity on reference queries; self-study reference queries generalize better than repeat-context in most cases.Source: §3.3 · pp. 7–8, Table 1 · p. 7
• With 80% of KV pairs evicted, each patch performs best with its source cache but still improves a disjoint cache, separating shared from subset-specific correction.Table 2 cross-evaluates patches as rows against inference caches as columns; swapped patches transfer shared correction for evicted context, while source pairing recovers subset-specific information.Source: §3.3 · p. 8, Table 2 · p. 8
• A single patch built over mixed contexts improves both constituent tasks across compression ratios, without one task dominating the other.Figure 4 evaluates RepoQA+KV and Summary+NIAH; self-study performs better on summary mixtures, while repeat-context gains an edge when retrieval is present.Source: §3.3 · pp. 8–9, Figure 4 · p. 8

3.4 Ablation Study

• At a 10% cache budget, accuracy is flat for chunk sizes from 1K to 5K tokens, then drops slightly at 8K and above as construction time falls and peak memory rises.The authors select 5K as the best balance; they attribute larger-chunk accuracy loss to noisier teacher activations from long-span reference reconstruction.Source: §3.4 · p. 9, Table 3 · p. 9
• Across NIAH, En.QA, and Math.Find, the default ridge scaling, ||SH||2_F/tr(SH), outperforms the tested trace-mean and spectral-norm alternatives.The authors report that this normalization transfers across contexts, models, and cache ratios without retuning.Source: §3.4 · p. 9, Table 4 · p. 9
• Across Qwen2.5-7B-1M, Qwen3-4B, and LLaMA3.1-8B, PatchKV improves KVzip, with the largest gains at cache ratios of 0.02–0.1.Figure 5 averages dataset scores normalized by full-KV performance; joint and self-study are strongest at aggressive budgets, while repeat-context becomes competitive as more KV pairs remain.Source: §3.4 · pp. 9–10, Figure 5 · p. 9

KV cache compression

• Token-eviction methods select cache entries, KVzip [9] makes this query-agnostic, and Attention Matching [12] approximates full-cache attention outputs.PatchKV is compatible with different compression methods because its context-specific weight patch compensates for compression-induced error.Source: §4 · p. 10

Compensating for compression error

• GRKV [27] redistributes evicted information into retained KV states, and MomentKV [28] stores compact statistics; PatchKV instead uses MLP weight updates.Source: §4 · p. 10

Converting context into weights

• Prior context-to-weight methods study weight updates as replacements for context, while PatchKV uses a closed-form patch to complement a compressed cache.The prior work includes exact low-rank updates [15], extensions to broader architectures [16], sequential optimization [17], and document distillation with Doc-to-LoRA [29].Source: §4 · p. 10

• PatchKV constructs context-derived block-wise weight patches by aligning reference-query activations under full and compressed caches with closed-form ridge regression.Source: §5 · p. 10
• Experiments show consistent improvements over compression methods in the low-cache regime across diverse benchmarks and models.Source: §5 · p. 10
• The authors conclude PatchKV complements cache-only compression and can enable more aggressive compression without increasing per-query inference cost.Source: §5 · p. 10

Cache preparation

• The algorithm takes a context, base model, cache compressor, cache budget, and ridge parameter, then prefills and compresses the full cache.Source: A Algorithms · p. 14

Block-wise patch construction

• It builds and chunks a reference set, initializes teacher and student states, and accumulates block-wise regression quantities from their activations.Source: A Algorithms · p. 14

Patch computation and return

• The algorithm computes and merges each block's patch into its weights, propagates the patched student state, and returns patched weights with the compressed cache.Source: A Algorithms · p. 14

Cache Initialization

• KVzip uses its default prefill chunk size of 2000 and global head scoring.Source: §B.1 · p. 14
• For extremely long contexts, Attention Matching uses HIGHESTATTENTIONKEYS; other hyperparameters follow its best-reported setting.Source: §B.1 · p. 14

Numerical Precision

• TF32 patch construction reduces construction time from 147.91 s to 85.35 s while achieving downstream performance comparable to FP32 across KV cache budgets.Patch construction remains in FP32, with TF32 enabled for matrix multiplications used to compute sufficient statistics.Source: Figure 6 · p. 15, §B.1 · p. 15

Dataset Selection

• Retrieve.MultiHop and ICL.ManyShot are reported separately because compression can improve their scores, complicating attribution of PatchKV’s effect.Removing distracting context can help Retrieve.MultiHop; ICL.ManyShot’s baseline scores highest with no context KV pairs, even above full-KV.Source: §B.2 · p. 15
• PatchKV improves over KVzip across evaluated nonzero cache budgets on HotpotQA, complementing gains on Retrieve.MultiHop.Repeat-context or joint reference queries further improve over KVzip on Retrieve.MultiHop across several cache budgets.Source: §B.2 · pp. 15–16

Task Variants

• Attention Matching and PatchKV both score zero on original Retr.KV and Retr.Prefix-Suffix under evaluated budgets, so shortened variants provide resolution.Source: Table 5 · p. 16, §B.2 · p. 16

Runtime Analysis

• Patch construction comprises one cache-compression stage and four stages repeated for each transformer block: teacher and student forwards, patch computation, and student-state propagation.Source: C Additional Experiment Results · p. 16

Reference Context Budget

• Using only 5K reference-context tokens recovers 15% of the compressed-to-full-KV accuracy gap on SCBench RepoQA; gains continue with diminishing returns.The context reaches 68K tokens, and the 5K budget is a single chunk.Source: C Additional Experiment Results · p. 16, Figure 8 · p. 17

Low-Rank Weight Patches

• A rank-32 patch matches full-rank performance within a small margin while reducing per-context patch storage by 99%.Source: Figure 9 · p. 17, C Additional Experiment Results · p. 17

Online Extension

• In online AIME 2025 compaction, PatchKV improves over Attention Matching at more aggressive cache budgets but provides little or no gain at larger post-compaction sizes.Compaction occurs when the cache reaches 2048 entries, with the most recent 20 protected.Source: C Additional Experiment Results · p. 17, Table 7 · p. 18

Quantized KV Caches

• PatchKV improves accuracy over the baseline with both INT8 and INT4 quantized KV caches, with gains comparable to the unquantized setting.Source: C Additional Experiment Results · p. 17, Figure 10 · p. 18

Full Experiment Results

• Qwen3-4B and LLaMA3.1-8B show consistent improvements across all reported benchmarks and variants with PatchKV over KVzip.Source: C Additional Experiment Results · p. 18

Limitations

• Reference queries proxy unknown future queries, so mismatch may limit recovery under larger distribution shifts.Source: D Limitations · p. 18
• One-time patch construction requires preprocessing and access to the full KV cache as teacher, making the method best suited to settings where cost can be amortized.Source: D Limitations · p. 18
• The primary evaluation targets single-context, multi-query use; concurrent serving of many context-specific patches needs additional mechanisms, and broader online evaluation remains future work.Source: D Limitations · p. 18