PaperlayerView exact revision on arXiv

Source-linked AI summary

LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, Junchen Jiang

arXiv:2510.09665v2cs.LG

TL;DR

KV caches increasingly exceed GPU capacity and need efficient movement for reuse across queries and inference engines. LMCACHE addresses this with optimized cache movement, modular engine connectors, and first-class orchestration APIs. Across diverse settings, it reports higher throughput and lower latency, while deployment experience identifies remote-storage and context-truncation boundaries.

  • Problem

    Growing KV-cache volumes exceed GPU memory, but efficient offloading and transfer across queries and inference engines remain unavailable.

  • Method

    LMCACHE is a KV-cache layer that uses optimized movement and pipelining, modular inference-engine connectors, and APIs for cache orchestration across storage and network tiers.

  • Results

    LMCACHE consistently improves throughput and latency versus open-source baselines and commercial APIs, delivering up to 15× higher throughput and at least 2× lower latency.

  • Takeaways & Limitations

    Enterprise adoption shows that remote storage can benefit prefill delay, while context truncation can reduce prefix-cache hit rate by half.

  • Takeaways & Limitations

    Remote-storage caching can increase prefill delay when contexts are short or models are small, requiring adaptive loading-versus-prefilling decisions.

Abstract

from arXiv · show

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.

1 Introduction

LLM inference increasingly requires KV caches to move beyond individual GPUs for cross-query reuse and prefill–decode disaggregation. LMCACHE provides an efficient, modular layer for storing, transferring, and orchestrating these caches across inference engines and heterogeneous devices.

  • Motivation: KV cache is moving beyond single-engine GPU memory to support cross-query reuse and prefill–decode disaggregation.Cross-query caching avoids recomputing shared prefixes, while PD disaggregation transfers cache from prefill GPUs to decode GPUs.
  • Motivation: Real-world usage shows stored KV cache growing beyond GPU capacity, requiring frequent eviction, offloading, and reload for reuse.The observed growth motivates persistent storage and movement across cache tiers.
  • LMCACHE: LMCACHE efficiently extracts and reloads KV caches, stores them across CPU, disk, and Redis tiers, and transfers them over Ethernet, RDMA, and NVLink.It is designed as an open-source implementation of new KV cache semantics for inference engines.
  • Contributions: Batched movement, compute–I/O pipelining, configurable chunking, and minimized copies improve the practicality of KV cache storage and loading.Larger configurable chunks use storage-to-GPU bandwidth more effectively than inference engines’ small native pages.
  • Contributions: Modular connectors decouple LMCACHE from changing inference-engine backends, while first-class APIs let operators locate, move, pin, and compress KV caches.These APIs support higher-level decisions such as KV cache-aware query routing.
  • Evaluation: Up to 15× higher throughput and at least 2× lower latency are reported across local caching, distributed reuse, and PD disaggregation settings.The evaluation compares LMCACHE with built-in open-source mechanisms and commercial inference APIs.

2 Motivation and Real-world Usage Statistics

Growing KV-cache size and reuse beyond GPU memory motivate moving caches across storage tiers and supporting cross-query reuse and PD disaggregation. Efficient extraction and loading remain necessary for these uses.

  • KV cache stores attention states for input and generated tokens, accelerating subsequent token generation within a query.
  • Growing contexts and background knowledge have made cross-query KV-cache sharing popular for reducing redundant computation.
  • The portion of KV cache exceeding GPU memory has increased significantly over five weeks, making GPU memory alone insufficient.
  • Reuse per token has grown significantly over recent weeks, indicating increasingly frequent reuse of tokens stored beyond GPU memory.
  • More than 19% of users reuse stored tokens more than 1.5 times, showing repeated access after storage.
  • Context caching reuses shared prefix segments across queries, while PD disaggregation transfers KV caches across inference stages and devices.

3 Challenges of Efficient KV Caching and Related Work

Practical KV caching is constrained by inefficient movement, rapidly changing inference-engine interfaces, and incomplete existing solutions. These gaps limit efficient, interoperable cache storage, transfer, and management.

  • Prefix caching and PD disaggregation face three interrelated systems challenges in practical adoption.
  • Serialization and primitive tensor copying typically transfer KV caches at sub-1GB/s and introduce delay and extra CPU-GPU copies.
  • Paged attention creates many small, non-contiguous I/O operations that underutilize network bandwidth and reduce throughput.
  • Frequent model and hardware updates can change GPU allocation and KV-cache formats, requiring caching libraries to repeatedly adapt.
  • Related Work and Existing Solutions: Inference-engine-native caching supports single-node transfers but lacks cross-node optimization and hierarchical storage, while storage layers lack an efficient glue layer.

4 Overview of LMCACHE

LMCACHE is a standardized KV-cache layer between inference engines and heterogeneous storage and network devices. It supports storing, retrieving, and locating caches while remaining compatible with evolving engines.

  • LMCACHE provides efficient KV-cache storage, movement, and explicit management for paged-memory inference engines.
  • The layer sits between inference engines and heterogeneous devices while maintaining compatibility with rapidly evolving vLLM and SGLang frameworks.
  • Store: During storage, the KV connector prepares metadata, the token processor identifies uncached tokens, and the storage manager saves them through a transfer channel.
  • Retrieve: During retrieval, the connector and token processor identify matching prefixes, while the event manager coordinates tracked or newly looked-up cache addresses.
  • Lookup: Higher-level routers query the cache controller, whose token pool tracks tokens stored or evicted across LMCACHE instances.

5 Performance Optimizations

LMCACHE improves KV-cache movement by batching transfers, overlapping computation with I/O, and minimizing redundant copies across storage tiers. Dynamic offloading trades lower duplication against possible allocation stalls.

  • Data movement: LMCACHE addresses small-page transfer inefficiency and concurrent-inference overhead with configurable chunks, parallel operations, and pipelined movement.It groups pages into larger chunks and supports concurrent migration across CPU memory, disks, and object storage.
  • Compute-I/O overlapping: Layer-wise pipelining overlaps KV-cache transfers with inference by using separate CUDA streams and a fixed-size GPU buffer.The next layer’s cache is fetched while the current layer is processed.
  • Compute-I/O overlapping: Asynchronous prefetch uses scheduler idle time to move queued queries’ caches from slower storage into faster tiers before inference begins.Users can configure the target tier according to latency objectives and resource constraints.
  • Minimum data copy: Reference-counted zero-copy transfers share data across simultaneous destinations instead of creating redundant copies.This reduces memory pressure during concurrent reads and writes.
  • Dynamic offloading: Dynamic offloading duplicates only a subset of free GPU pages, balancing duplication ratio against the likelihood of allocation stalls.A smaller duplication window saves memory but may delay queries needing pages that are not yet copied.

6 Standardized Interface for Connecting the KV Caching Layer and Inference Engine

LMCACHE uses a modular connector interface to decouple KV-cache management from rapidly changing inference-engine internals. The interface supports engine integration while preserving flexible, low-overhead operation.

  • Motivation: Rapidly evolving inference engines can change KV-cache layouts, making direct integration difficult for LMCACHE.The paper notes that supporting new architectures often requires substantial engine modifications.
  • Connector design: LMCACHE’s standardized connector decouples KV-cache management from the inference-engine backend, preserving compatibility as upstream APIs evolve.The API design was initiated by LMCache and maintained collaboratively with vLLM.
  • Design objectives: The connector prioritizes flexibility, vLLM-native integration, out-of-tree connector support, and minimal API-level overhead.These objectives guide the interface design rather than prescribing a single backend implementation.
  • Interface structure: The connector divides responsibilities between scheduler interfaces that prepare metadata and model-runner interfaces that execute cache transfers.This separation supports cache loading and storage across lower-tier storage systems.
  • End-to-end interaction: At query arrival, scheduler calls determine matched cache tokens and loading decisions, while model-runner calls coordinate layer-wise transfers.Layer-wise execution waits for each layer’s cache before inference proceeds and synchronizes storing of generated cache.
  • Adoption: The KV connector API has seen open-source adoption in NVIDIA Dynamo, llm-d, AIBrix, and vLLM production-stack projects.The paper also reports proprietary connectors from multiple companies.

7 Controller Interfaces

LMCACHE’s controller provides centralized metadata management and distributed APIs for locating, moving, sharing, clearing, pinning, and compressing KV caches. These interfaces support routing, migration, and cross-node cache reuse.

  • Controller architecture: LMCACHE separates external operator APIs from internal per-instance APIs within a distributed caching system.A centralized controller manages global metadata, cache manipulation, and request routing.
  • Capabilities: The controller enables cross-node cache sharing, cache-aware request routing, and dynamic KV-cache migration.These capabilities are presented as applications of the controller interfaces.
  • Cache-aware routing: Cache-aware routing consults a global in-memory cache view to identify instances, storage devices, and hit-token counts.Instances report admissions and evictions through batched interfaces, while routers query the aggregated state.
  • KV-cache migration: The migration API transfers specified KV cache from a source instance and device to a destination when scaling down or balancing load.The source establishes a connection to the destination when needed.
  • P2P sharing: Peer-to-peer sharing lets an instance retrieve cache chunks from another peer after a local cache miss.The controller returns peer locations and hit-chunk counts for selection.
  • Cache management: Additional APIs let applications clear, compress, decompress, pin, and unpin KV caches at specified locations.These operations expose explicit cache management to user applications.

8 Evaluation

LMCACHE is evaluated across CPU offloading, real-trace, and industry-oriented workloads using TTFT and ITL alongside throughput. It consistently outperforms the evaluated baselines, including under high-QPS real traces.

  • Setup: The evaluation covers three representative scenarios using industry-adopted open-source models and workloads including multi-round QA, LongBench, and random benchmarking.Experiments use single-node, multi-node, and prefilling-decoding-disaggregation configurations.
  • Metrics: TTFT measures prefill delay, while ITL measures the average delay between consecutive generated tokens.Both metrics are reported for each experiment.
  • Single-node CPU offloading: 1.9–8.1× smaller TTFT and 2.3–14× higher throughput at the same TTFT are achieved by LMCACHE versus the strongest baseline across five evaluated models.At QPS=1, LMCACHE also achieves 7%–92% smaller ITL than the best baseline.
  • Sources of improvement: LMCACHE’s gains over basic vLLM are attributed partly to CPU offloading, which stores more KV cache than GPU memory and increases cache-hit ratios.Its movement implementation also addresses the bandwidth limitations of per-layer and per-16-token transmission.
  • Real-trace evaluation: LMCACHE consistently outperforms basic vLLM on real traces, reducing TTFT by at least 3.7–6.8× and ITL by 19–58% across five models at high QPS.The traces use input and output distributions from companies F and G with up to 500 GB of CPU DRAM.

8.4 Centralized Storage Server

The centralized-storage evaluation shows that LMCACHE improves throughput and PD-disaggregation latency through larger-capacity caching, efficient transfer granularity, and overlapped loading and computation.

  • Centralized storage: 1.3–3× improvement in inference throughput is achieved over basic vLLM across QPS levels with centralized remote storage.The larger remote backend stores more KV cache than CPU memory, increasing cache hit ratios.
  • PD disaggregation: Significantly better tail latency and 1.5–1.8× lower mean TTFT are achieved than vLLM’s native PD disaggregation.The comparison uses 8K-token inputs and 200-token outputs.
  • Transfer efficiency: LMCACHE reduces transmission latency in PD disaggregation by avoiding the finer-grained page-by-page transfers used by vLLM’s native design.The prefill and decode computation times are the same for both systems; the difference arises in KV-cache transmission.
  • PD disaggregation: 1.1–1.7× lower mean ITL is achieved than vLLM’s native PD disaggregation.LMCACHE’s chunk-based transfer design improves PD-disaggregation efficiency.
  • Asynchronous compute: 1.46× lower end-to-end delay results when query asynchronization overlaps KV-cache loading with prefill or decode computation.Without asynchronization, loading and computation occur sequentially.

8.7 Sensitivity Study

The sensitivity study shows that KV-cache loading is beneficial when network bandwidth and context length make loading faster than naive prefilling, and that LMCACHE remains effective with SGLang.

  • Context length and bandwidth: At 32 Gbps, KV-cache loading outperforms naive prefilling only beyond 256K input tokens.The crossover depends on context length under low bandwidth.
  • Context length and bandwidth: At 64 or 128 Gbps, KV-cache loading achieves lower delay than naive prefilling across all context lengths.Higher bandwidth removes the low-bandwidth crossover limitation.
  • Context length and bandwidth: Adaptive loading decisions are needed under low bandwidth, enabling loading only after its delay becomes lower than prefilling.The recommended decision boundary is the context-length crossover point.
  • SGLang integration: On Qwen3-32B with two H100 GPUs, LMCACHE achieves higher throughput and lower mean TTFT and end-to-end latency than SGLang without CPU offloading.The experiment enables LMCACHE CPU offloading.
  • SGLang integration: LMCACHE achieves comparable performance to SGLang’s native CPU offloading on the same Qwen3-32B evaluation.This supports compatibility with another inference engine beyond vLLM.

9 Real-World Lessons and Experience

Real-world deployments reveal benefits and constraints of remote KV-cache storage, context truncation, production deployment practices, and community-driven expansion across hardware and inference engines.

  • Remote storage: 22–32% lower TTFT than full prefill was achieved when Company C loaded KV caches from its remote object store.The deployment suggests remote backends can improve cache hit ratios while reducing TTFT.
  • Context truncation: Prefix cache hit ratios dropped from roughly 85% to 45% when Company F truncated inputs to retain only the latest tokens.Truncation prevents truncated inputs from matching previously cached prefixes.
  • Production deployment: Containerized deployment through Docker images has become standard among industry users operating Kubernetes-managed GPU clusters.Many users rely on official images without examining LMCACHE’s source code.
  • Production cache reuse: A 50% prefix-cache hit rate was observed for Company G, reflecting dynamically reusable contexts in production applications.Examples include conversation histories in coding assistants, chat applications, and retrieval-augmented generation.
  • Industry and academia: LMCACHE deprioritized flexible APIs for specialized attention mechanisms, making it less popular in academia despite its industry-focused performance, stability, and compatibility.The authors identify more flexible APIs as a next step.
  • Community adoption: Community contributions expanded LMCACHE from three backends integrated with vLLM on NVIDIA GPUs to eight additional backends across four processor types and two inference engines.Industry partners upstreamed their contributions to remain aligned with ongoing development.

10 Conclusion and Outlook

The paper presents LMCACHE as a production-ready KV-cache layer that improves inference efficiency and supports a broader shift toward persistent, cache-aware LLM infrastructure.

  • Conclusion: LMCACHE treats KV cache as a first-class data structure rather than an internal inference byproduct.This design supports distributed compute and storage across enterprise-scale deployments.
  • Conclusion: Across diverse workloads and models, LMCACHE delivers significant throughput improvement and latency reduction relative to open-source baselines and commercial inference APIs.The conclusion also cites CPU offloading, hierarchical storage, and PD disaggregation in production.
  • Outlook: Production deployments reveal opportunities including KV-cache reuse in recommendation systems and lossy compression in open-ended chatbots.These examples extend the system’s use beyond the evaluated inference settings.
  • Outlook: The outlook positions KV caches as a standardized storage and communication medium for scaling LLM inference and agentic workloads.The proposed direction treats AI-native data as a core infrastructure primitive rather than merely an optimization.
Loading 2510.09665v2…