PaperlayerView exact revision on arXiv

Source-linked AI summary

KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

Taiga Ikeda, Daisuke Miyashita, Jun Deguchi

arXiv:2610.13060v1cs.CL

TL;DR

Long-context prompts are costly, and exact-prefix caching cannot reuse documents that appear after different prompt content. KV-Repurpose provides a vLLM-based testbed for segment-wise non-prefix KV reuse. The evaluation reports quality boundaries and serving benefits in long-context workloads.

  • Problem

    RAG requests can share substantial content without an exact shared prefix, while simple, reproducible baselines for evaluating non-prefix reuse remain limited.

    Source: §1 · pp. 1–2

  • Method

    The paper integrates segment-wise non-prefix KV reuse into vLLM and develops an evaluation system covering cache generation, prompt construction, eviction, and runtime monitoring.

    Source: §1 · p. 2

  • Results

    Naive reuse preserves quality for a meaningful subset of long-context workloads but has substantial limitations in cross-document reasoning, while the system shows cache-capacity efficiency and latency improvements in long-context, Zipfian RAG-style workloads.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The testbed supports systematic evaluation of both serving performance and output quality for non-prefix KV reuse in long-context inference.

    Source: §1 · p. 2, §5 · p. 6

    The implementation uses only a single LLM and simple positional correction, and it neither combines nor directly compares KV-Repurpose with more sophisticated prior PIC techniques.

    Source: Limitations · p. 6

Abstract

from arXiv · show

KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating segment-wise KV-cache reuse beyond exact prefixes. The system combines cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments on long-context benchmarks show that naive segment reuse largely preserves output quality for single-document question answering and summarization, while exposing clear limitations in cross-document reasoning. Under a Zipfian RAG-style workload in which 20% of requests share a fixed prefix, KV-Repurpose achieves a substantially higher token-level cache hit rate than prefix caching at the same 30 GB cache capacity, 63.9% versus 3.3%, resulting in lower TTFT. Even with 1 TB of cache capacity, prefix caching reaches only a 17.9% hit rate in this workload. Source code is available under the Apache-2.0 license at https://github.com/taiga333/eval-kv-repurpose.

1 Introduction

Long prompts make LLM serving costly, while exact-prefix caching misses reusable content when retrieved documents appear in different prompt contexts. KV-Repurpose provides a reproducible testbed for evaluating non-prefix reuse, its quality limits, and serving benefits.

1 Introduction

  • RAG requests can share substantial content without sharing an exact prefix, leaving current serving engines unable to reuse the corresponding KV states and potentially increasing redundant computation, TTFT, and cache pressure.Retrieved-document composition and ordering often vary across requests.Source: §1 · p. 1
  • Simple, reproducible baselines for non-prefix KV-cache reuse in modern serving engines remain limited, making practical assessment and comparison difficult.The paper develops a testbed to evaluate serving performance and output quality in long-context inference.Source: §1 · p. 2
  • The work integrates segment-wise, non-prefix KV-cache reuse into vLLM and builds an end-to-end evaluation system for cache generation, prompt construction, eviction, and runtime monitoring.The system is intended as a research baseline rather than a state-of-the-art method for mitigating quality degradation.Source: §1 · p. 2
  • KV-Repurpose is evaluated on a model with more than 100 billion parameters and shows substantial cache-capacity efficiency and latency improvements under long-context, Zipfian RAG-style workloads.The introduction also frames the work as an empirical characterization of when naive reuse preserves quality and when it fails.Source: §1 · p. 2

2 System Implementation

The system pairs an extended vLLM inference server with a proxy that builds prompts and manages KV caches. Its segment-wise reuse and RAG-style workload support enable controlled evaluation of cache behavior and serving performance.

2 System Implementation

  • The system combines an extended vLLM server for inference with a proxy for request preprocessing, KV-cache management, evaluation, and monitoring.The monitoring tracks cache hit rates and storage utilization, while workload emulation measures TTFT and throughput.Source: §2 · p. 2, Figure 2 · p. 2

2.1 Extensions to the vLLM Server

  • The vLLM extension checks cache availability independently for each delimited segment, reusing cached states when available and prefilling misses without blocking reuse of later segments.It stores keys unrotated and applies RoPE at their current positions upon reuse, without recomputing attention with the preceding prefix.Source: §2.1 · p. 2

2.2 Proxy Server

  • The proxy emulates RAG with pseudo-retrieval from a preconstructed question–context database, enabling controlled and reproducible request patterns without evaluating retrieval quality.It builds prompts around cached documents and optional perturbations, such as timestamps, that emulate cases where exact-prefix caching cannot apply.Source: §2.2 · p. 3
  • To avoid user-specific preceding content influencing reusable document KV representations, the proxy generates absent document caches through a dedicated request and evicts cached entries with an LRU policy when capacity is exceeded.The stated reason for avoiding generation during a full user request is the risk of cross-segment information leakage.Source: §2.2 · p. 3

3 Related work

The related work contrasts exact-prefix cache management with reuse of later segments after an earlier miss, and distinguishes the paper’s serving focus from position-independent caching research. It also situates the evaluation at a larger model scale and with longer documents than prior PIC studies discussed here.

3.1 KV-Cache Management for LLM Inference

  • LMCache checks chunks from the start of the sequence and stops checking after a miss, whereas later chunks may still be reusable in this paper’s setting.The authors describe their implementation as fewer than 900 lines of Python, intended to keep it simple to analyze and extend for research.Source: §3.1 · pp. 3–4

3.2 Position-Independent Caching

  • Prior PIC studies focus mainly on preserving quality under positional mismatch, while this work emphasizes serving benefits from increased reuse opportunities; the directions are described as complementary.The paper focuses on a model with more than 100 billion parameters and one or a few longer documents, such as full research papers or entire web pages, unlike the smaller-model, many-short-segment workloads described for prior studies.Source: §3.2 · p. 4

4 Evaluation

The evaluation tests KV-Repurpose against prefix caching and KV recomputation on long-context tasks and RAG-style serving workloads. It reports comparable average benchmark performance, higher cache hit rates, lower TTFT, and improved high-load throughput in evaluated settings.

4.1 Experimental Setup

  • The testbed extends vLLM and evaluates GPT-OSS-120B on a single NVIDIA H200 GPU, using NVMe SSD storage for the KV cache.The evaluation uses LongBench and LongBench Pro for accuracy, and English documents of 8K–128K tokens for system-level workloads.Source: §4.1 · p. 4

4.2 Task Performance

  • Across 25 LongBench Pro tasks, KV-Repurpose and KV recomputation achieve comparable average performance.Scores use standard task metrics and are averaged over three runs.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 Evaluation under a Zipfian Workload

  • At 30 GB, KV-Repurpose reaches a 63.9% token-level cache hit rate versus prefix caching’s 3.3% with 20% fixed-prefix requests, and achieves lower p50 and p95 TTFT.The hit-rate comparison is from the Zipfian workload; the TTFT comparison uses Poisson arrivals at the same cache capacity.Source: Figure 3 · p. 5, §4.3 · pp. 5–6, Figure 4 · p. 5

4.4 Context-Length Scaling Microbenchmark

  • When context length exceeds roughly 16K tokens, KV-Repurpose achieves lower TTFT than prefix caching; prefix caching saturates at high request rates, while KV-Repurpose sustains higher throughput across most evaluated high-load settings.Source: §4.4 · p. 6

5 Conclusion

The conclusion presents KV-Repurpose as a testbed for flexible KV-cache reuse and reports serving benefits in controlled RAG-style scenarios.

5 Conclusion

  • The testbed supports flexible KV-cache reuse in an inference engine and is evaluated under controlled RAG-style serving scenarios.The stated aim is to investigate the potential to reduce LLM inference prefill cost.Source: §5 · p. 6
  • The authors report that reducing redundant KV-cache storage improves storage efficiency, while higher cache hit rates improve response performance and reduce GPU computational load.Source: §5 · p. 6
  • Evaluating deployments with much larger KV-cache repositories is left for future work.The conclusion notes that storage-efficiency advantages may become increasingly important as repositories grow.Source: §5 · p. 6

Limitations

The study is a simple testbed rather than an exhaustive evaluation of KV-Repurpose techniques, and its results may not generalize beyond the evaluated model.

Limitations

  • The study targets a simple testbed and does not exhaustively explore possible KV-Repurpose techniques.Source: Limitations · p. 6
  • The implementation uses only simple positional correction and neither combines KV-Repurpose with nor directly compares it against more sophisticated PIC techniques.Source: Limitations · p. 6
  • Experiments use one LLM, so results may not generalize to other architectures, model sizes, or training settings.Evaluation across broader models and workloads remains future work.Source: Limitations · pp. 6–7

A.1 Database

The database construction treats each question’s gold context as its retrieved and reranked result for prompt processing.

A.1 Database

  • Each question’s gold context is used as the retrieved and reranked result and fed into the prompt processing tool.Source: §A.1 · p. 9
  • Algorithm 1 constructs a pseudo-retrieval database from those inputs.Source: §A.1 · p. 9
  • Algorithm 1 takes a question set, gold-context mapping, and context pool as inputs.Source: §A.1 · p. 9

A.2 Empirical Observation of Cross-Segment Information Leakage

The leakage experiment found that information in a prompt segment before a cached context could reappear after cache reuse, motivating context-only cache generation.

A.2 Empirical Observation of Cross-Segment Information Leakage

  • 35.0% of the 100 tested family names leaked into an output at least once across three trials, despite being absent from the cached context segment.The names appeared only in the preceding prompt during cache generation.Source: §A.2 · p. 9
  • Information preceding a cached context may be reflected in its KV representations and later reproduced even when that information is absent from the cached segment.The reported control setting showed no name leakage.Source: §A.2 · p. 9
  • Reusable document caches are therefore generated from context alone in a dedicated preprocessing step, excluding user-specific prefixes and other preceding segments.This strategy is intended to avoid potential security and privacy risks from inherited information.Source: §A.2 · p. 9

A.3 End-to-End Workflow for Subsection 4.3 and 4.4

The proxy emulates request arrivals and retrieval, selects a cache-reuse mode, and manages cache entries; the workflow evaluation also uses repeated fixed-context requests.

A.3 End-to-End Workflow for Subsection 4.3 and 4.4

  • For each request, the proxy uses Poisson-sampled arrivals, pseudo-retrieves a context-question pair, sends the request to vLLM, and updates cache metadata.When necessary, it evicts cached entries.Source: §A.3 · p. 9
  • The workflow distinguishes prefix caching from repurposing and, for repurposing, checks whether a context cache exists before constructing a context-only prompt when needed.The algorithm specifies separate server request paths for the two modes.Source: §A.3 · p. 10
  • Cache eviction follows the least-recently-used policy.The proxy evicts least-recently-used entries from its cache metadata store.Source: §A.3 · pp. 9–10
  • For each evaluated context length, the microbenchmark processes one fixed context 100 times.The request time is prepended as a perturbation prefix, and arrivals follow a Poisson distribution.Source: §A.3 · p. 10

A.4 Illustrative Capacity Extrapolation

The capacity extrapolation estimates cache-hit and prefill-saving benefits under stated assumptions, while the LongBench Pro results show comparable average performance and the multi-document diagnostic leaves a remaining limitation.

A.4 Illustrative Capacity Extrapolation

  • At 0.2% and 1% of total KV-cache capacity, the extrapolation estimates 14.55% and 24.2% cache hit rates, with 23.74% and 42.94% prefill savings, respectively.The estimates use Wikipedia pages sorted by access frequency; Figure 6 presents storage, hit ratio, and prefill saving.Source: §A.4 · p. 10, Figure 6 · p. 10
  • Even under favorable prefix-caching assumptions, the extrapolation estimates approximately 2.8 PB of required cache capacity.It assumes perfect prediction of reusable requests and ten-turn conversations, with reuse probability capped at 20%.Source: §A.4 · p. 10
  • The extrapolation’s 15 KB-per-byte storage ratio and 20% reuse probability illustrate scale rather than precisely characterize a production workload.The authors state that the qualitative conclusion does not depend on the exact values.Source: §A.4 · p. 10

A.5 Evaluation with Selected Supporting Documents

  • Restricting multi-document QA inputs to selected supporting documents partially reduces the performance degradation, but a gap remains and the limitation persists.The oracle-assisted intervention reduces document count and removes potentially irrelevant content, so the recovery cannot be attributed to either factor individually.Source: §A.5 · p. 11, Table 3 · p. 11

A.6 Detailed Results on LongBench Pro

  • KV-Repurpose and KV recomputation achieve comparable average performance across LongBench Pro, with minor task-level variations.The without-context variant generally scores substantially lower.Source: §A.6 · p. 11, Figure 7 · p. 11

Related papers

Loading 2610.13060v1…