PaperlayerView exact revision on arXiv

Source-linked AI summary

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi

arXiv:2609.11744v1cs.DCcs.LG

TL;DR

External KV caching for vLLM lacks evidence about when loading cached prefixes beats recomputation, especially across hardware and workloads. The paper characterizes this tradeoff and introduces py-kvcache with asynchronous I/O, bounded staging, and scheduler-aware preloading. It reports faster TTFT than LMCache in evaluated long-context settings, while showing that external caching remains setup-specific.

  • Problem

    The paper studies when external KV-cache lookup pays for itself because loading costs can exceed recomputation for short prefixes or fast GPUs.

  • Method

    The authors characterize vLLM KV caching across tiers and workloads, then build py-kvcache with asynchronous direct I/O, bounded staging, and scheduler-aware preload.

  • Results

    In evaluated configurations, py-kvcache halves disk-only query TTFT relative to LMCache at 80k tokens and remains within 4% of native vLLM in the multi-tier comparison.

  • Takeaways & Limitations

    External KV caching should use setup-specific admission and scheduling decisions that account for break-even points, overlap, and queueing.

  • Takeaways & Limitations

    The study evaluates only two hardware classes, a limited set of models and storage configurations, FP16 KV data, one output token, and a single 256-token block size.

Abstract

from arXiv · show

Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.

1 Introduction

Long-context prefix reuse can reduce TTFT, but external KV caching helps only when transfer and scheduling costs are lower than recomputation. This work characterizes those tradeoffs in vLLM and develops py-kvcache with hardware- and workload-aware policies.

  • Motivation: External KV caching replaces GPU recomputation with lookup, storage reads, and CPU-to-GPU transfer, so short prefixes, low hit rates, or fast GPUs can make caching slower.The paper frames cache admission as a break-even decision rather than treating every prefix hit as beneficial.
  • Evaluation: 1.23× faster than LMCache at 80k tokens in the tiered configuration, py-kvcache remains within 1.04× of native vLLM KV Offload.In disk-only operation, preload halves TTFT relative to LMCache and improves TTFT by 1.34× over py-kvcache without preload at 80k tokens.
  • Characterization: The characterization shows that scheduling, copy granularity, and copy-compute overlap can substantially change TTFT beyond the effect of device bandwidth alone.The study traces KV movement across vLLM scheduling, connector interfaces, CPU memory, GPU memory, and storage.
  • py-kvcache: py-kvcache is a Python external KV-cache engine for vLLM and shared filesystems, using efficient I/O, bounded intermediate CPU memory, and hardware-aware admission.Its design targets large contiguous KV objects while limiting staging growth.
  • Motivation: KV-cache size grows linearly with cached tokens, and an 80k-token Llama 3.2 3B context occupies 8.75 GiB, quickly exceeding GPU memory.KV blocks can therefore be moved to CPU DRAM, NVMe, shared filesystems, or remote stores.
  • KV-cache background: Prefix reuse is organized through fixed-size KV blocks and chained hashes, so only matching blocks from the beginning of a prompt can be reused.A partially matching block must still be recomputed, making reuse dependent on block size.

3 Systems Compared

The paper compares py-kvcache with LMCache, llm-d, and vLLM’s native KV Offload across different interfaces, storage tiers, and transfer designs.

  • Systems compared: py-kvcache is evaluated alongside LMCache, llm-d, and native vLLM KV Offload as an external KV-cache system.Table 1 summarizes the compared systems, interfaces, and storage designs.
  • LMCache: LMCache uses vLLM’s KV Transfer connector and supports CPU DRAM plus slower disk or remote storage tiers.The evaluated disk tier uses a small thread pool and standard POSIX file operations.
  • Execution sequences: The compared designs differ in whether transfers block after each pass or are deferred to overlap a subsequent forward pass.Figure 2 contrasts synchronous Transfer behavior with Offload’s deferred stores.
  • llm-d: llm-d provides a filesystem cache with staged CPU memory, separate read/write workers, and a direct-I/O fork used to avoid page-cache effects.Its filesystem tier was later deprecated in favor of vLLM’s integrated secondary filesystem tier, which retained the same on-disk format and direct-I/O thread-pool design.
  • vLLM KV Offload: vLLM KV Offload uses a CPU primary tier as a gateway to secondary filesystem, object-store, or P2P tiers and asynchronously propagates stored blocks.Its native implementation supports ARC or LRU eviction and overlaps transfer work with model execution.

4 Experimental Setup

The evaluation combines tracing with synthetic, benchmark, and trace-driven workloads to measure TTFT, cache reuse, scheduling, and the break-even point for external caching.

  • Methodology: The study combines code tracing with benchmarks to identify where TTFT latency arises across several vLLM versions, with final py-kvcache comparisons using a v0.22 fork.The version range spans v0.16 through v0.22 because vLLM’s interfaces changed during the work.
  • Evaluation goals: The experiments separate when cached KV reuse beats recomputation, how vLLM cache interfaces perform, and where current caches leave performance unexploited.These questions motivate the combined benchmark and tracing methodology.
  • Synthetic workloads: Synthetic workloads vary document size, prefix reuse, prefix size, concurrency, arrival rate, and warmup requests to control cache-reuse conditions.The long-document benchmark is the main synthetic workload and includes a pre-warmup phase to absorb initialization costs.
  • Scheduling and overlap: Mixed workloads vary concurrency, reuse fraction, and traffic gaps to test whether preloading moves disk I/O off the TTFT critical path during concurrent serving.The experiments include both fresh requests and prefix-reuse requests.
  • Break-even analysis: The break-even analysis compares cold GPU computation with GPU, CPU, and cache-enabled runs across document sizes to determine when SSD-backed caching is worthwhile.The study interpolates TTFT values between measured points and uses Pareto-front analysis to estimate the minimum beneficial transfer bandwidth.
  • Trace-driven workloads: ShareGPT represents mostly cold-cache traffic because independently sampled conversations provide little intentional prefix reuse, so it is used to measure external-cache overhead rather than establish a definitive baseline.The benchmark uses one generated output token and samples realistic short and medium requests.
  • Trace-driven workloads: Bailian traces provide realistic request sizes, prefix reuse, and arrivals, while SCBench and LongBench supply shared-prefix datasets for multi-turn and long-context evaluation.SCBench supports round-robin replay under working sets larger than available GPU VRAM and CPU DRAM; LongBench is used for prefix reuse rather than output quality.

5 Characterization of Existing KV Cache Systems

The characterization shows that external KV-cache performance depends on transfer granularity, scheduling overlap, memory behavior, and workload break-even conditions rather than bandwidth alone. Cache benefits increase with document length, but short or low-reuse workloads can favor recomputation.

  • 5.1 GPU ↔CPU Transfer Path: 32.8× lower cache-hit TTFT at 80k tokens shows that external caches increasingly outperform recomputation as documents grow.Across the benchmark, the improvement ranges from 2.2× at 1k tokens to 32.8× at 80k tokens.
  • 5.1 GPU ↔CPU Transfer Path: 24–26 GB/s GPU ↔CPU bandwidth is similar across systems, so Offload’s advantage comes mainly from fewer asynchronous transfers overlapping model execution.Offload used 21 transfers per warmup request, whereas LMCache used 640 smaller transfers.
  • 5.2 NVMe SSD KV Caching: External caching requires a reusable-prefix threshold: below roughly 8k tokens, prefill can beat SSD caching, while a mid-range drive moves the threshold to 32k.The threshold depends on the model, GPU, SSD, and reusable prefix length.
  • 5.2 NVMe SSD KV Caching: A 4.4 GB KV workload caused roughly 10 GB of llm-d traffic because of oversized staging allocation, and fixing it reduced TTFT by approximately 50% below LMCache.The result demonstrates that intermediate memory allocation can materially inflate transfer work.
  • 5.2 NVMe SSD KV Caching: Large-I/O throughput, rather than small-request IOPS or large worker pools, limits end-to-end SSD caching performance.Engines converge near drive bandwidth with 1 MiB requests, motivating few asynchronous workers with bounded queue depth.

6 Design of py-kvcache

py-kvcache is a filesystem-backed vLLM KV Offload connector designed around asynchronous direct I/O, bounded staging, and shared operation across instances. Its scheduler and worker components separate transfer planning from storage execution while pipelining block-level transfers.

  • 6 Design of py-kvcache: py-kvcache targets shared filesystem deployment, NVMe bandwidth, bounded intermediate CPU memory, and scheduler-aware transfer timing.The design is based on the KV Offload API and asynchronous GPU transfers.
  • 6.1 Selecting the Storage Interface: Ordinary files with direct I/O and io_uring preserve asynchronous submission and bounded queue depth while keeping the implementation in Python.The implementation uses liburing Python bindings rather than requiring a detached NVMe device.
  • 6.2 System Architecture: The scheduler-side Manager creates cache and preload plans, while the worker-side Handler and TransferCoordinator execute block-level file jobs through one I/O reactor.Only compact transfer plans cross between scheduler and worker; KV tensors move through GPU memory, staging, and storage.
  • 6.3 Filesystem Layout: A hierarchical hash-addressed filesystem layout shares prefixes across vLLM instances and supports the tested metadata workload even with one million files.Atomic hard-link publication prevents duplicate concurrent writers from publishing the same hash.
  • 6.4 Block Transfer Engine: Fixed CPU staging slots and configurable I/O depth bound memory while pipelining each storage block from disk through CPU staging to GPU.Store operations reverse the pipeline and use per-slot CUDA streams without device-wide synchronization.
  • 6.5 Scheduler-Aware Preloading: Preloading begins disk reads for queued requests before scheduling, using bounded lookahead and stopping speculative work when demand loads arrive.This required scheduler callbacks and connector hooks because the original API exposed no waiting-queue visibility to workers.

7 Evaluation of py-kvcache

Across controlled, benchmark, and trace-driven workloads, py-kvcache improves TTFT by combining asynchronous transfers with scheduler-aware preloading and bounded staging. The gains are largest for long or irregular reusable contexts, while smaller prompts and powerful GPUs can make external caching unnecessary.

  • 7.1 Isolated prefix cache tests: 2.0–2.5× lower disk-only TTFT than LMCache demonstrates py-kvcache’s main controlled-workload advantage with preload enabled.Without preload, py-kvcache was consistently about 1.5× faster than LMCache.
  • 7.1 Isolated prefix cache tests: At 80k tokens, the complete tiered configuration is 1.23× faster than LMCache and within 1.04× of native vLLM offloading.At 1k tokens, configurations fall within 0.12 s and the ordering reverses below break-even.
  • 7.2 Preloading: 1.34× preload improvement at 80k tokens shows that scheduler lookahead reduces TTFT even when large CPU-to-GPU transfers hide part of the gain.The preload improvement is 1.66× at 40k tokens.
  • 7.3 LongBench: On LongBench, py-kvcache is 2.77–3.64× faster than LMCache and 1.22–1.79× faster than native vLLM offloading across two irregular long-context domains.It also outperforms recomputation and GPU prefix caching in both domains.
  • 7.4 SCBench: On SCBench, py-kvcache is 3.11× faster than GPU prefix caching, 2.48× faster than native CPU-and-disk offloading, and 1.26× faster than LMCache.Bounded staging avoids native-offload occupancy spikes; py-kvcache reads 85 GB versus 3.4 TB and finishes at approximately 480 s.
  • 7.5 Evaluation Takeaways: Bailian traces improve TTFT on the RTX 4000 Ada, but on the H100 GPU-only caching matches py-kvcache because requests average below the 6,203-token SSD break-even point.The result makes external-cache usefulness dependent on workload reuse, GPU memory, and hardware.

8 Discussion

External KV caching is a critical-path optimization whose benefit depends on transfer timing, granularity, staging resources, and recomputation cost—not storage bandwidth alone. The evaluated comparisons support conservative, configuration-specific admission and preloading policies.

  • Large 1 MiB storage requests converge near device bandwidth, so improving small-I/O operation rate or changing the I/O engine alone does not improve end-to-end TTFT.Asynchronous I/O mainly maintains enough outstanding large operations to use SSD bandwidth.
  • Transfer granularity and overlap, rather than substantially faster physical copies, explain why native Offload can outperform LMCache despite comparable GPU ↔CPU bandwidth.The Offload path issues fewer, larger transfers and hides store work alongside model forward passes.
  • Preloading helps only when a request is likely to execute and remains queued long enough for useful I/O to complete; unlimited preloading can waste bandwidth, staging memory, and transfer capacity.SCBench shows that concurrent native promotions can reserve most of the CPU tier and displace recently promoted blocks.
  • py-kvcache performs similarly to native vLLM Offload in tiered tests and better on LongBench and SCBench, while its conservative preloading policy handles demanding conditions more robustly.LMCache provides broader engine, backend, and deployment support, so the comparison is configuration-specific rather than a universal replacement claim.
  • GPU-resident reuse remains preferable when the working set fits in VRAM because vLLM avoids transfers by swapping pointers.External caching is complementary when reusable prefixes exceed GPU capacity.
  • External caching is most attractive when reusable prefixes are long, exceed GPU capacity, and requests wait long enough for transfers to overlap with other work.It is less attractive for short prefixes, low reuse, faster GPUs, or lightly loaded systems; thresholds vary by model, hardware, datatype, and tier.

GPU-resident Prefix Reuse

GPU-resident prefix reuse is enabled by non-contiguous KV organization and automatic prefix caching, avoiding external transfers when reusable data fits in GPU memory. py-kvcache addresses the complementary case where reusable prefixes exceed GPU capacity.

  • PagedAttention makes non-contiguous KV blocks practical and underpins vLLM’s automatic prefix caching.
  • py-kvcache complements GPU-resident caching by targeting reusable prefixes that exceed GPU memory capacity.SGLang’s RadixAttention likewise integrates reusable-prefix state with request scheduling.

External and Disaggregated KV Caches

External and disaggregated KV-cache systems separate reusable state from inference execution across engines, memory pools, networks, and storage backends. They coordinate cache movement and serving resources around KV-centric data paths.

  • LMCache provides reusable KV storage across serving engines and multiple local or remote backends.
  • MemServe separates KV storage from inference workers through an elastic memory pool and uses a cost model based on request and cache state.
  • Mooncake treats KV transfer as central to disaggregated serving, coordinating prefill, decode, and cache resources around a KV-centric data path.

SSD KV Caching

SSD KV-caching research focuses on deciding what to load or retain, reducing transfer volume, and choosing suitable transfer granularity and storage paths. These approaches address the fact that disk reuse may not beat recomputation.

  • IMPRESS selectively loads important tokens, while py-kvcache decides whether loading a matched prefix is worthwhile against recomputation.The two approaches are compatible because selective loading reduces the bytes moved.
  • Bidaw selects what to retain by weighing storage footprint against computational savings, whereas py-kvcache gates whether an existing cache load belongs on the critical path.
  • Several systems treat transfer granularity as the dominant SSD concern through larger blocks, shared pruning and prefetch granularity, or distribution across devices.
  • KV compression increases effective tier capacity and shifts the measured break-even frontier, while DUAL-BLADE uses an NVMe-direct path when page-cache behavior becomes unpredictable under severe memory pressure.

Prefetching and Lookahead

Prefetching moves KV transfers off the critical path by loading data before requests are scheduled, using scheduler-aware or speculative selection strategies.

  • Prefetching and Lookahead: Prefetching moves KV transfers off the critical path before a request is scheduled.The cited systems use scheduler-aware fetching, queue inspection, or speculation to select data in advance.

10 Limitations and Future Work

The study’s conclusions are bounded by its hardware, model, storage, I/O, block-size, and workload settings, while future systems could adapt caching decisions dynamically.

  • 10 Limitations and Future Work: The evaluation covers two hardware classes, few models, limited storage configurations, FP16 KV data, one output token, and 256-token blocks.It excludes networked and RAID configurations and does not establish sustained decode-throughput or multi-GPU effects.
  • 10 Limitations and Future Work: Trace replay broadens evaluation beyond fixed prefixes but does not reproduce every aspect of live production deployment.The trace workloads provide broader workload variation without fully representing production behavior.
  • 10 Limitations and Future Work: The measured break-even frontier applies to CPU-dependent I/O with host-memory staging, while faster direct-access paths could move it.A faster path does not remove the loading-versus-recomputation tradeoff, but it can change its boundary.
  • 10 Limitations and Future Work: Comparisons describe evaluated versions and configurations rather than permanent rankings because LMCache and vLLM are rapidly evolving.The authors frame the findings as evidence about system design.
  • 10 Limitations and Future Work: Future serving engines could update store, load, and defer decisions using estimates of prefill time, bandwidth, queue delay, and promotion success.Preload selection could also use a priority queue constrained by explicit memory and I/O budgets.

11 Conclusion

The work frames external KV caching as a setup-specific comparison between moving reusable prefixes and recomputing them, and introduces py-kvcache to improve that decision and overlap transfers with computation.

  • 11 Conclusion: External caching pays off when reusable prefixes move sooner than recomputation and admission selects useful reuse.The comparison depends on storage bandwidth and overlap between compute and copy operations.
  • 11 Conclusion: py-kvcache combines asynchronous direct I/O, bounded shared staging, scheduler-aware preload, and setup-specific break-even decisions.The connector is designed to remove transfer work from the request’s critical path.
  • 11 Conclusion: At 80k tokens, py-kvcache halves disk-only query TTFT versus LMCache and remains within 4% of native vLLM in the multi-tier comparison.These results are reported for the evaluated configurations.
Loading 2609.11744v1…