Source-linked AI summary
KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
Xi Shi, Qian Lou
TL;DR
KV-cache reuse is poorly evaluated for changing contexts and producer checkpoints, even though RAG and multi-agent serving depend on reusable source material. KVShareArena crosses these factors on retrieved chunks and agent reports, scoring recovered context value and resource frontiers. It finds that position alignment suffices until sources must interact, where paid repair recovers substantial value, while some unrepaired caches perform below no context.
Problem
Existing KV-cache benchmarks largely test exact-prefix reuse, leaving changing prompt contexts and producer-checkpoint differences under-evaluated for RAG and multi-agent workloads.
Method
KVShareArena crosses context and producer-weight changes across Retrieved Evidence and Agent Reports, using PGR anchors and separate compute, memory, and latency accounting.
Results
Position alignment suffices until sources must interact; then paid repair recovers substantial value, while naive assembled caches can score below the no-payload floor.
Takeaways & Limitations
Cache reuse quality depends on whether the task requires cross-source attention, so deployment comparisons should distinguish free geometric repair from repairs that rebuild information.
Takeaways & Limitations
The main rankings use one 8B model, confirmation boards replicate patterns rather than rankings, and same-backend latency covers only rows sharing an engine.
Abstract
from arXiv · showhide
LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condition: a retrieval-augmented generation server assembles a different set of retrieved chunks for every query, and a multi-agent coordinator reads reports written by other agents. Reused inside a new prompt, a cache carries the wrong positions and never attended to the other sources. The cache may also have been written by a different checkpoint of the same model family, which changes the stored values. Repair methods for such caches have appeared in three separate communities, each measured on its own terms, and existing benchmarks test only exact-prefix reuse, where nothing is lost. KVShareArena benchmarks KV-cache reuse across prompt contexts and model checkpoints on retrieved chunks and agent reports. It scores every method by the fraction of the gap it recovers between no cache and full recomputation, and charges compute, memory, and per-request latency with the cache in hand, reporting the one-time cost of building a cache separately. We find that correcting positions, which needs no recomputation, is enough until a question needs several sources at once. There, only methods that pay, by re-encoding part of the cache or by training, recover half to two thirds of the gap; unrepaired caches can be worse than no cache. Cache-compression methods that are harmless on a single prompt fall significantly behind position correction on freshly written agent reports. These patterns hold across three model boards. When a different checkpoint wrote the cache, training-free methods are barely affected, while an adapter trained on one checkpoint's caches loses quality. Harness, frozen querysets, and cost accounting ship as a pip package with an automated submission workflow and a public leaderboard.
1 INTRODUCTION
KVShareArena addresses KV-cache reuse when retrieved sources or agent reports move into new prompts or arrive from different checkpoints. It unifies these settings under a common benchmark and finds that position alignment suffices only when sources need not interact, while jointly dependent sources require paid repair.
- Motivation: RAG queries and multi-agent coordination reuse source blocks in changing prompt contexts, making exact-prefix caching ineffective despite dominant prefill costs.Each query selects different chunks or receives freshly generated reports, so cached material must move out of its birth context.
- Motivation: Cross-checkpoint reuse can preserve tensor shapes while changing cached values, because producer and receiver checkpoints may have different weights.The benchmark treats context change and producer-weight change as separate factors, including their combined hardest case.
- Problem: Reused caches suffer both misplaced positions and missing cross-source attention, although moving keys to new positions is nearly free.Position correction repairs geometry but cannot reconstruct interactions that were absent when blocks were encoded independently.
- Research gap: Prior evaluations used separate tasks, ceilings, and cost accounting, while the closest benchmark tested only exact-prefix reuse or varied one factor at a time.These designs leave context change, checkpoint change, normalized quality, and shared cost unmeasured together.
- Contribution: KVShareArena crosses context and producer weights across Retrieved Evidence and Agent Reports, scoring performance-gap recovered between no-context and same-payload ceilings.It also reports compute, KV memory, and same-backend latency frontiers, with cache-building cost separated from per-request cost.
- Findings: Free position alignment is sufficient until sources must interact; in that regime paid repair recovers substantial lost value, while naive assembly can fall below the no-payload floor.The main findings also cover compression losses on assembled reports, replication across three model boards, and checkpoint-specific failures.
2 BENCHMARK SETUP
The benchmark independently varies context and producer weights, evaluates isolated payload KV in two workload tracks, and normalizes quality between shared floor and ceiling anchors. It reports quality alongside separate compute, memory, and latency frontiers under preregistered admission rules.
- Protocol: Payload KV is encoded in an isolated birth context, then tested under concatenated assembly or prefix replacement with changed use contexts.Concatenated blocks are mutually blind and collide in position; prefix replacement changes geometry without cross-source interaction.
- Two-factor design: The four regimes cross context matching with producer–receiver weight matching: exact-prefix control, checkpoint-only handoff, Cross-Context reuse, and their combination.Producer and receiver share architecture and tokenizer, but need not share weights.
- Workloads: Retrieved Evidence uses independently cached raw chunks, while Agent Reports transfers specialist-generated reports produced without seeing the question.The benchmark charges isolated payload encoding uniformly, although deployment may already incur report decode-phase build cost.
- Quality metric: Performance-gap recovered scores each method between a no-payload floor and a same-payload full-prefill ceiling, with 0 at the floor and 1 at the ceiling.PGR may exceed 1 or become negative when reused context outperforms or underperforms the anchors.
- Efficiency: Efficiency is reported through separate Pareto frontiers for prefill work saved, KV bytes saved, and same-backend TTFT saved.The benchmark measures actual tensors and per-sample raw sums, while one-time cache-building cost remains separate from per-request accounting.
- Admission and discipline: Admission is preregistered and keeps the tested method as the only independent variable, excluding regimes with insufficient measurable context value.Inapplicable methods retain explained rows but are not granted private settings or ranked across incompatible settings.
3 MAIN-BOARD RESULTS
Free position alignment is sufficient for largely independent sources, but tasks requiring joint reasoning need paid repair; assembled agent reports are especially unforgiving.
- Free position alignment is sufficient on scientific-paper QA, while repair benefits concentrate on single-document and multi-hop questions.
- Compression never significantly beats free alignment and falls significantly below it in four cells, especially at r=.5 and on multi-hop.
- On assembled reports, only CacheBlend (+.11) and RelayCaching (+.08) clear the free-alignment bar, leaving one third of the gap unclaimed.
- Thirteen of sixteen rows recompute nothing; CacheBlend recomputes 17%, LegoLink under 0.5%, and cached-request TTFT is approximately .9 below dense prefill.
- Compression loses KV information after misaligned assembly, with SnapKV falling from .77/.69/.82 in its birth context to .57/.17/−.08.
- Position correction restores .93/.98/1.05 in single-source handoff, whereas repair is needed when sources are mutually blind and jointly required.
4 CROSS-MODEL CONFIRMATION AND CROSS-CHECKPOINT REUSE
Confirmation boards reproduce the main qualitative pattern, while cross-checkpoint reuse usually changes scores little; producer-specific training is the notable exception.
- On a second model family, paid repairs help single-document QA, scientific-paper QA is dominated by free alignment, and multi-hop QA separates methods.
- The same qualitative patterns hold across the confirmation boards, although boards are not compared numerically and method ordering can reverse across models.
- Foreign producer weights lower every Cross-Context score, but the drop depends on both task and producer.
- Most methods shift by ≤.06 and usually nonsignificantly across producers, whereas KVPacket drops significantly in four of six cells because it fits the producer’s representation.
5 LIMITATIONS
The benchmark’s conclusions are bounded by its multi-source QA setting, limited Agent Reports roster, single-model rankings, and constrained checkpoint comparison.
- The benchmark measures multi-source QA, and the Agent Reports track includes fewer methods by construction.
- Per-row rankings rely on one 8B model, while confirmation boards replicate patterns rather than rankings and same-backend latency covers only shared engines.
- The report protocol is author-constructed, inputs are capped at 29K tokens, and paired differences below roughly .10 at N=100 appear as ties.
- The checkpoint factor covers matched architecture, tokenizer, and RoPE base, so it does not address producers differing in any of those properties.
6 RELATED WORK
The paper distinguishes four forms of KV reuse and focuses on reuse in a new context with repair. It positions KVShareArena against prior evaluations that vary in task, model, and cost accounting.
- Reuse categories: In-place reuse preserves exact-prefix KV caching, while new-context reuse requires repairing caches after relocation.Relocated storage changes where caches reside, and input rewriting manufactures common prefixes; both are outside the paper’s repair focus.
- Repair methods: Repair methods include selective recomputation, calibration, boundary recomputation, and anchor-offset reuse, alongside latent communication approaches.The related methods come from RAG serving, multi-agent systems, and a third line that replaces the medium with latent communication.
- Evaluation landscape: Prior evaluations differ in tasks, metrics, cost accounting, and whether they include training costs or time-to-first-token.The paper notes that earlier studies report absolute-accuracy deltas and TTFT, often without a shared cost framework.
- Evaluation landscape: SCBench tests only exactly matching prefixes, while the closest chunk-level comparison reports absolute scores without cost.These evaluations do not jointly test misaligned contexts, cross-model reuse, and efficiency.
REPRODUCIBILITY STATEMENT
The paper freezes querysets and evaluation decisions before runs, releases independently recomputed fact packs, and publishes raw cost numerators and denominators.
- Reproducibility: Frozen SHA256 manifests and precommitted admission, method, and reading rules establish the evaluation inputs before runs.The paper states that every reported number is backed by an independently recomputed fact pack.
- Reproducibility: Per-sample raw numerators and denominators for all three cost axes are released alongside the scores.This supports independent recomputation of the reported efficiency quantities.
AI USE STATEMENT
The paper documents author responsibility for experimental claims and supplements official metrics with audited LLM-based scoring. It reports companion tables while omitting FRAMES from the second instrument because FRAMES already uses an LLM-based official metric.
- AI use: AI assistance supported engineering and writing, while authors retained responsibility for experimental decisions, claims, and final text.
- Scoring audit: LLM-scored verdicts were audited by blind second scoring and a cross-family scorer panel, with agreement rates and disagreements released.The main-board judge validation included a second LLM that re-judged 40 outputs blind, reaching 92.5% agreement and κ = .84.
- Companion tables: Tables 4 and 5 provide judge-protocol companion boards with the same rows, partitions, and baselines as the main tables.FRAMES is omitted because its official metric is already LLM-based and therefore lacks a second instrument for comparison.
- Official metrics: Table 2 reports Retrieved Evidence PGR under official dataset metrics, including paired differences versus free alignment and measured recomputation percentages.The table caption identifies significant differences at 95%, ties, and the absence of a FRAMES-wave entry where applicable.
B.3 WHERE THE TWO INSTRUMENTS DISAGREE
The official and judge instruments disagree on individual method rankings and measurement headroom, but these disagreements do not change class-level claims. The report track’s official winners are significant while judge ties flatten those differences.
- Instrument disagreement: Re-judging reversed method orders on all three subsets, with sharp rank changes for 4-bit quantization and Block-Attention.The paper attributes these differences to token F1 penalizing verbose correct answers and judge verdicts tying many short-answer pairs.
- Instrument disagreement: The report track’s two official winners are significant under the official metric but not under the judge because binary verdicts tie 46–70% of short-answer pairs.This illustrates how the judge can flatten differences even when class-level conclusions remain unchanged.
- Instrument disagreement: FRAMES shows instrument-dependent headroom: the lexical net gap collapses to .069, whereas the LLM-scored gap is .417.The paper retains FRAMES’s official autorater because the collapse is attributed to the instrument rather than the methods.
- Interpretation: Across the disagreement patterns, individual rows move but no class-level claim changes.
C.1 PREFIX REPLACEMENT: A DOSE SCAN WITH NO REPAIR SPACE
The dose scan shows that prefix replacement is primarily a geometry problem: direct reuse is already near the ceiling, and free position correction leaves no measurable repair space. More demanding handoffs show why this result does not generalize to cross-source assembly.
- Dose scan: PGR 1.03 at the middle dose shows direct reuse already reaches the ceiling without repair.The free position fix remains within .02 of dense recomputation across all tested dose cells.
- Handoff comparison: Free position correction restores .93/.98/1.05 PGR after naive KV handoff, leaving all admission margins below the .25 bar.The corresponding naive handoff scores are −.22/ −.37/ −.98, below the no-context floor.
- Handoff comparison: Latent-state transfer scores −.23/ −.52/ −1.08 PGR, below the floor on all three subsets.The matched text-handoff control scores high, isolating the medium replacement as the failed design choice.
- Handoff comparison: Position-fixed KV handoff reaches text-handoff quality at PGR .86–1.13 while consuming only 102–219 prefilling tokens.That is about 20–33% of the text route’s 498–655 tokens.
- Compression: Compression falls from .772/.691/.817 in place to .570/.172/−.082 after misaligned assembly.Knorm shows the same degradation pattern, indicating that compression damage becomes severe after independent birth contexts are assembled.
- Application boundary: On retrieved evidence, position alignment reaches the ceiling, while on reports it recovers 87.1% of the gap and leaves only .023 residual.This summarization candidate therefore has little or no cross-source repair space on either track.
F DATASET CARDS AND ADMISSION RECORDS
The benchmark fixes separate workload tracks, anchors, admission rules, and cost-accounting conventions so methods are compared under reproducible conditions. Its records also document declined candidates, implementation provenance, and the distinction between serving-layer caching and cross-context repair.
- Dataset cards: Retrieved Evidence uses three LongBench QA subsets with N=100 frozen samples each and context values .395/.329/.308.Natural passage boundaries are used where available, with 512-token windows otherwise and a 29K-token input cap.
- Cost accounting: The runtime frontier reports about 90% per-request savings with the cache in hand, while charging one-time cache construction separately.The build-inclusive panel bounds a single read rather than estimating deployment-level amortization.
- Cost accounting: Only compression saves KV memory, while other methods store the full cache and KVPacket / MiniPIC add about 3%.Measured 4-bit quantization saves 71.9% rather than its nominal 75%.
- Dataset cards: Agent Reports reuse question-blind specialist reports byte-identically, with dense prefill over the same report text defining the ceiling.Reading original documents is treated as a separate pipeline-design question.
- Admission records: Admission failures are published: a cross-session-memory candidate failed C3, the summarization candidate failed C2 on both tracks, and two multi-hop candidates were dropped for thin context value.The benchmark does not silently omit negative results.
- Reference rows: Prefix caching is lossless but saves nothing on these workloads beyond static instruction prefixes, with served-replay hit rates of 0.6% to 21.4%.It is retained as a reference origin rather than a competing repair method.
K IMPLEMENTATION PROVENANCE
Implementation provenance distinguishes official implementations, ports, and reference rows, while checkpoint testing injects alternate producers without modifying method libraries. The trained adapter can fail silently under foreign producers even when outputs remain fluent.
- Method provenance: Table 6 separates selective recomputation, calibration, alignment, anchor reuse, medium replacement, trained repair, adapters, handoff repair, and compression classes.It also records whether each row is official, ported, or reference.
- Method provenance: Table 7 reports raw official F1 for direct-reuse factorial cells with a fixed receiver and different-weight producer conditions.HotpotQA uses n=40 and Qasper n=20.
- Checkpoint testing: The cross-checkpoint roster keeps method libraries unmodified and requires same-model regression drivers to reproduce frozen rows bit-identically.All 15 of 15 modified-driver rows pass this gate; a known cross-site numeric difference explains the two naive-row divergences.
- Checkpoint testing: Foreign-producer degradation in the trained adapter is silent: fluent answers can name the wrong entity, reverse yes/no, or misattribute a year.Recomputation-based repairs show no such flips under the same producers.
L.3 CHECKPOINT ROBUSTNESS SUMMARY
Checkpoint robustness is summarized over six cells per method, while preset scores assess quality-cost trade-offs under different axis weights. The paper treats producer-fitted repairs as the distinctive robustness risk and shows that quality-led rankings are broadly stable across presets.
- Checkpoint robustness: Table 8 measures each method’s PGR shift across six cells, reporting mean, worst cell, and counts of paired 95% confidence intervals below zero.With 84 method cells, about four false positives are expected at the 5% level.
- Checkpoint robustness: Two producer-fitted repairs stand out across every Table 8 column, unlike isolated significant cells among compression rows.The table therefore separates systematic producer sensitivity from expected individual false positives.
- Preset summary: The same three rows lead under every quality-led preset, although their order changes with β.They are the trained adapter, boundary-recomputation port, and selective recomputation method paying 17% of payload compute.
- Preset summary: The balanced preset is the only quality-led setting that separates the paying method from the two near-free methods.The memory-bound preset rewards only compression and assigns every full-cache row zero by construction.
- Preset summary: Agent Reports are summarized with the compute axis alone, while unmeasured axes appear as “–” rather than being imputed.Runtime is measured for eight rows, and reference rows lack memory measurements in this release.