Evaluating the accuracy of KV cache reuse techniques

1 INTRODUCTION

• RAG’s retrieved chunks inflate prompts, making quadratic-cost prefill a dominant latency component that position-independent cache reuse aims to reduce.Unlike prefix caching, reuse can place a chunk cache at any position in future prompts containing that chunk.Source: §1 · p. 1
• Aggregate accuracy mixes preservation of full-prefill answers with baseline failures, parametric-memory answers, and guess-prone questions, inflating reported reuse accuracy.The proposed meaningful subset retains queries the baseline answers correctly, the model cannot answer without context, and the answer space is non-trivial.Source: §1 · pp. 1–2
• Common datasets cluster semantically similar chunks, making offline cross-attention targets easy to infer, and often offer little cross-query chunk reuse for warm-method evaluation.Boxoffice adds controlled reuse patterns, including stale cache, where a chunk recurs in conflicting contexts.Source: §1 · p. 2
• On Boxoffice, the same reuse method scores F1 0.98 or 0.00 depending only on the context in which a chunk was cached.The patterns persist across three query templates and 8B and 32B models.Source: §1 · p. 2
• Up to 42% of a reuse method’s F1 on LongBench-QA comes from queries the full-prefill baseline cannot answer.Boxoffice uses fictional-corpus chunks, making queries meaningful by construction and guaranteeing cross-query chunk reuse.Source: §1 · p. 2

2 BACKGROUND ON POSITION-INDEPENDENT KV CACHE REUSE

• A Transformer computes keys and values during prefill, then reuses them from the KV cache during token-by-token decoding.Prefill cost is quadratic in prompt length.Source: §2 · p. 2
• Isolated chunk caches omit cross-chunk attention; selective recomputation of chosen tokens against reused keys and values recovers part of the missing attention.For RoPE-based models, cached keys are also rotated to their new prompt offset.Source: §2 · p. 2
• Cold reuse starts with isolated caches and reconstructs cross-chunk attention at query time, while full prefill stores attention to preceding prompt chunks.Across cold variants, approximating full-prefill accuracy requires recomputing a substantial fraction of tokens.Source: §2 · p. 2, Figure 1 · p. 3
• Warm reuse stores cross-chunk attention ahead of query time; FusionRAG uses nearest chunks, LMCache saves first-use context, and CacheCraft keeps prefix-specific versions.Warm caches may contain attention to chunks outside the current prompt, with further recomputation used to recover attention to prompt chunks.Source: §2 · p. 3, Figure 1 · p. 3

3 TESTING ENVIRONMENT

• The evaluation covers 200-query MuSiQue, HotpotQA, and 2WikiMQA subsets, using F1 and normalized F1 relative to full prefill.A normalized F1 below 1 indicates accuracy loss.Source: §3 · p. 3
• Experiments use Llama3.1-8B, Qwen3-8B, and Mistral-7B, with selected experiments repeated on Qwen3-32B.Chunk embeddings use e5-base-v2, and similarity is measured by cosine similarity.Source: §3 · p. 3
• The comparison includes CacheBlend, FusionRAG, LMCache, and CacheCraft, with 15% of tokens selectively recomputed by default.A query-driven selection variant and CacheCraft−M variants are also implemented.Source: §3 · pp. 3–4

4.1 EVALUATION METHODOLOGIES: PITFALLS AND OUR SOLUTION.

• The meaningful subset excludes queries the full-prefill baseline cannot answer, queries answerable without context, and low-information yes/no or disjunctive questions.These filters target queries where loss attributable to KV cache reuse is observable.Source: §4.1 · p. 4
• 42% on MuSiQue and 24.9% on 2WikiMQA of aggregate F1 comes from queries the baseline scored zero but reuse partially answered.These baseline-zero flips inflate aggregate scores without measuring preservation of full-prefill accuracy.Source: §4.1 · p. 5

4.2 DATASET LIMITATIONS FOR WARM REUSE - THE OFFLINE CASE

• Cosine neighborhoods cover at least 54% of context and 69% of golden chunks on LongBench-QA, but industry co-used partners rank among the 10 nearest neighbors in only 8% of cases.LongBench-QA’s clustered chunks make offline cross-attention targets easy to identify; industry traces instead show a median shared-chunk query-set Jaccard similarity of about 0.1.Source: §4.2 · p. 6

4.3 DATASET LIMITATIONS FOR WARM REUSE - THE ONLINE CASE

• LongBench-QA has 0.7%–12.6% chunk reuse, 0.7%–26.6% golden reuse, and no query that can be served entirely from reused KV caches.By comparison, 73% of first-week RAGPulse requests consist entirely of previously seen chunks, rising to 93% after one week of warmup.Source: §4.3 · pp. 6–7

5 BOXOFFICE: A STRESS-TEST BENCHMARK FOR KV CACHE REUSE

• Boxoffice uses fictional records and context-dependent queries with reused chunks, avoiding parametric-memory answers and within-query semantic clustering.Each query has 10 candidates by default, and every candidate dossier contributes to the answer.Source: §5 · p. 7

5.1 CACHE STALENESS, A NEW KV CACHE REUSE DYNAMIC

• Boxoffice reuses every evaluation chunk after warmup and varies whether chunks were cached as winners or nonwinners, creating aligned, all-W, all-N, and flipped configurations.With multiple versions, warmup balances each chunk's winner and nonwinner roles so methods can be tested on selecting a role-matching cache.Source: §5.1 · pp. 7–8, §5 · p. 7

5.2 EVALUATING KV CACHE REUSE WITH BO X O F F I C E

• LM reaches norm-F1 ≥0.97 on aligned queries but 0.00 on flipped queries across all three models, showing strong configuration-dependent staleness effects.Intermediate configurations fall between those extremes; the dynamics also appear with other templates and Qwen3-32B.Source: §5.2 · p. 8
• +Q improves every warm method on every model, but does not close the gap to full prefill; on cold CB it helps Llama and regresses on Qwen and Mistral.The authors identify better token selection as a remaining need.Source: §5.2 · p. 8
• Multi-version methods achieve the highest overall norm-F1 on Llama, but can lose on aligned queries when selection picks the wrong cached version.With 10 chunks per query, even a small per-chunk mis-selection probability compounds.Source: §5.2 · pp. 8–9
• CC-M4+Q wins on Llama, CB on Qwen, and FR+Q on Mistral, so no reuse strategy is robust across models.The winners span different hot/cold and online/offline families.Source: §5.2 · p. 9

• Existing KV-cache-reuse evaluations report aggregate accuracy on standard QA benchmarks, while common workarounds yield clustered contexts, lack enforced golden-chunk reuse, and omit queries composed entirely of reused chunks.CacheCraft's evaluation uses a proprietary workload that is not released.Source: §6 · p. 9
• Boxoffice supports reproducible evaluation of multi-versioned strategies by enforcing parametric chunk reuse so enough cache versions accumulate.Source: §6 · p. 9
• The methodology aligns with prior work on per-capability evaluation, context-versus-memory contributions, and aggregate-score inflation from artifacts or memorized test items.Source: §6 · p. 9

• The paper releases Boxoffice to evaluate KV-cache reuse under challenging, controllable reuse patterns and support more thorough, reproducible evaluations.Source: §7 · p. 9
• Boxoffice uses three query templates over one synthetic corpus, and the main evaluation covers three open-source approximately 8B models.Qwen3-32B results verify persistence of accuracy inflation and staleness, but other architectures may differ quantitatively.Source: §7 · p. 9
• The authors used generative AI for editing, related-work discovery, and code assistance, but not to generate datasets, design experiments, or interpret results.They state that all AI-assisted code was reviewed and tested and that results came from released code.Source: AI USE STATEMENT · p. 10

A APPENDIX - PREVALENCE OF LONGBENCH DATASETS

• Each of the LongBench-QA datasets examined appears in more than 80% of reviewed KV-cache-reuse papers, while other datasets have limitations such as clustered contexts or few long chunks.Summarization datasets are excluded because they lack reuse; agentic workload datasets target different reuse dynamics.Source: A APPENDIX - PREVALENCE OF LONGBENCH… · p. 16

B EXTENDED RELATED WORK

• Prior work documents aggregate-score inflation from artifacts, memorized items, and capability aggregation, while other studies separate context contributions from parametric memory.Source: B EXTENDED RELATED WORK · pp. 16–17

C.1 EXAMPLE PROMPT COMPONENTS

• The example prompt combines an instruction, movie dossiers, and a question asking which candidate has the highest box-office field.The instruction requires ranking candidates by the BOX_OFFICE_MUSD integer field and returning one FILM-ID.Source: §C.1 · pp. 17–18

C.2 WARMUP CONFIGURATIONS

• Table 6 organizes Boxoffice configurations by whether the evaluation winner and nonwinners were cached as winners or nonwinners.Source: Table 6 · p. 18, §C.2 · p. 18

C.3 DATASET GENERATION

• The generation workflow filters candidate queries, retains rows all filter models solve with full context but not question-only, then balances warmup predecessor roles before writing dataset files.Source: §C.3 · p. 18

C.4 CHUNK DISTANCE CDFS

• LongBench-QA golden chunks are closer to other chunks in the same query, whereas Boxoffice chunk similarity is similar within and outside queries.The figure compares golden and non-golden targets in LongBench-QA and query chunks in Boxoffice.Source: §C.4 · p. 19, Figure 2 · p. 19

C.5 BOXOFFICE ADDITIONAL EVALUATION

• On Boxoffice at R=0.05, LM+Q is the average winner for each model in norm-F1 results, reported as mean±sample standard deviation across 10 generated datasets.Source: §C.5 · p. 19, Table 8 · p. 20
• With M=2, more than 75% of ten-chunk queries lose perfect alignment when one selected cache has the wrong warmup conditioning.Additional versions can also turn some nonaligned queries into mixed ones, partially recovering accuracy.Source: §C.5 · p. 20, Figure 3 · p. 21

D REUSE CHARACTERIZATION

• Evaluation chunks are reused across many queries, while winner chunks appear at arbitrary prompt positions to avoid positional bias.The plotted frequency distributions exclude warmup queries, which guarantee each evaluation chunk appears at least M=4 times in these experiments.Source: Figure 4 · p. 21, D REUSE CHARACTERIZATION · p. 22

E OFFLINE WARMUP DETAILS

• FR conditions winner and nonwinner caches on mixed neighbor-size compositions, so it has no clear advantage across query configurations.Winner chunks retrieve mostly smaller neighbors, while nonwinner chunks have a more balanced neighborhood.Source: E OFFLINE WARMUP DETAILS · p. 22

F SENSITIVITY TO THE NO-CONTEXT THRESHOLD AND SIZE OF THE MEANINGFUL SUBSET

• Across tested thresholds, ρF1 remains well below 1 for every method, supporting the conclusion that unfiltered F1 inflates reported accuracy preservation.ρF1 grows slowly as the threshold rises because fewer queries are filtered out and the filtered and unfiltered datasets overlap more.Source: F SENSITIVITY TO THE NO-CONTEXT… · p. 22, Table 9 · p. 23
• Meaningful subsets are model-dependent because accuracy preservation is measured against each model’s full-prefill ceiling.Cross-method comparisons within a model are unaffected; a shared cross-model set can use the intersection of model-specific subsets.Source: F SENSITIVITY TO THE NO-CONTEXT… · p. 22, Table 10 · p. 23
• Boxoffice retains queries answered by every filter model with full context and by none without context, yielding one shared evaluation set.Generating a set meaningful for a new group of models is automated.Source: F SENSITIVITY TO THE NO-CONTEXT… · p. 23

G LARGER MODELS: QWEN3-32B

• On Qwen3-32B MuSiQue, baseline-zero queries contribute 9%–24% of aggregate F1, and inflation persists at four times the scale of the 8B models.The meaningful subset contains 68 of 200 queries; ρF1 stays below 1 at τ=0.2 and τ=0.5.Source: G LARGER MODELS: QWEN3-32B · p. 23, Table 11 · p. 24
• On Boxoffice, Qwen3-32B LM matches full prefill on aligned queries, scores 0.00 on flipped queries, and mixed configurations remain close to flipped.Cold method CB is flat across configurations.Source: G LARGER MODELS: QWEN3-32B · p. 23, Table 12 · p. 24
• With M=4, CC-M4 and CC-M4+Q score 0.33–0.42 on non-aligned configurations versus 0.00–0.11 for LM, at the cost of aligned performance.Query-driven token selection helps LM marginally and hurts the cold method on this model.Source: G LARGER MODELS: QWEN3-32B · p. 23, Table 12 · p. 24

H ADDITIONAL QUERY TEMPLATES

• Genre uniqueness requires finding the sole candidate whose genre is not shared; warmup roles encode whether predecessors share that genre.The evaluation uses two further templates over the same fictional corpus and generation pipeline.Source: H ADDITIONAL QUERY TEMPLATES · p. 24
• Pivot join asks for the other candidate sharing a named pivot movie’s director, making the task a two-hop match.A chunk cached after a same-director predecessor carries the signal that a match exists.Source: H ADDITIONAL QUERY TEMPLATES · p. 24
• Across genre uniqueness and pivot join, LM peaks on aligned queries and collapses on flipped, while multi-version reuse progressively recovers performance.LM reaches 0.51–0.66 on aligned and 0.00–0.05 on flipped genre-uniqueness queries; for pivot join, the corresponding ranges are 0.24–0.42 and 0.01–0.06.Source: H ADDITIONAL QUERY TEMPLATES · p. 24, Table 13 · p. 25, Table 14 · p. 25

I EVIDENCE FROM PRODUCTION RAG WORKLOADS

• 97% of RAGPulse retrieved-passage slots use passages seen in at least two requests; Cache-Craft's top 5% of chunks serve 60% of requests.The evidence spans a production trace, enterprise support queries, an enterprise-workflow benchmark, and a production-workload analysis whose underlying data is not released.Source: I EVIDENCE FROM PRODUCTION RAG WORKLOADS · p. 25
• Shared chunks recur across varied contexts: median chunk-set Jaccard similarity is 0.09 in RAGPulse and about 0.1 in HERB.In HERB, an actually co-used partner appears among a chunk's top-10 offline neighbors in only 7.9% of 54,974 co-used pairs.Source: I EVIDENCE FROM PRODUCTION RAG WORKLOADS · p. 25
• Boxoffice matches this reuse pattern: 100% of chunk slots use chunks seen by at least two queries, and shared-chunk queries have median Jaccard similarity 0.11 in every seed.These values are close to RAGPulse's 97% and 0.09, respectively.Source: I EVIDENCE FROM PRODUCTION RAG WORKLOADS · p. 25
• Boxoffice lets users set cached-version count, warmup-configuration mix, candidate-set size, and recomputation budget to match measured statistics.Source: I EVIDENCE FROM PRODUCTION RAG WORKLOADS · p. 26