Source-linked AI summary
KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, Haibo Chen
TL;DR
LLM serving needs low latency and high throughput, but its KV$ caching behavior and eviction policies are poorly understood across real workloads. This paper characterizes production KV$ reuse patterns and develops a workload-aware eviction policy, improving cache hits and mean response time on production traces.
Problem
Production LLM serving requires low latency and high throughput, while existing synthetic workloads miss important real-world KV$ cache patterns.
Method
The paper systematically characterizes production KV$ workloads and uses workload-specific reuse probabilities, spatial locality, and lifespan to design an eviction policy integrated into vLLM.
Results
The workload-aware policy improves cache hits by 3.9% and mean response time by up to 41.4% on production traces.
Takeaways & Limitations
KV$ caching should account for workload-specific reuse patterns; small caches can suffice for API-dominated workloads, while workload-aware eviction improves over LRU and LFU.
Takeaways & Limitations
The characterization uses one week of production traces and does not cover newer workloads such as reasoning; other serving components remain future work.
Abstract
from arXiv · showhide
Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV\$) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV\$ caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV\$ workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV\$ reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity.
1 Introduction
Real-world LLM serving workloads exhibit diverse but category-predictable KV$ reuse patterns, with skewed reuse contributions, meaningful single-turn reuse, and moderate cache-capacity requirements. These findings motivate workload-aware eviction, which improves cache performance on production traces.
- Motivation: Caching KV$ reduces repeated computation when requests share input prefixes, improving serving latency and throughput.The cache can reuse intermediate K and V values instead of recomputing them for subsequent requests.
- Motivation: The study addresses missing real-world evidence about reuse prevalence, reuse distributions, and KV$ lifespan needed for cache-policy and capacity decisions.The authors characterize two representative production workloads collected from a large cloud provider under privacy constraints.
- Workload characterization: 10% of KV$ blocks contribute to 77% of reuses, while single-turn requests dominate 97% of KV$ reuses in to-B workloads.Reuse contributions are skewed across requests, and single-turn reuse is substantial despite common emphasis on multi-turn interactions.
- Workload characterization: Reuse time and probability vary across API, chat, and multi-turn request categories but are predictable within each specific category.Because serving systems know request categories, estimated reuse probabilities can inform cache policies.
- Workload characterization: P99 KV$ lifespan is 97 seconds in to-B workloads, making a small cache typically sufficient.A cache with capacity 2 × the per-GPU HBM is sufficient to approach the ideal hit rate on common GQA models using standard policies such as LRU.
- Design and evaluation: A workload-aware eviction policy using profiled reuse probabilities improves cache hits by 3.9% and mean response time by up to 41.4% over workload-agnostic LRU and LFU policies.The policy was integrated into vLLM and evaluated on production traces.
- Scope: The study is based on one week of production traces and leaves newer workloads such as reasoning, along with other serving components, for future work.The authors specifically identify global scheduling and characterization of evolving workloads as outside the present scope.
2 Background: LLM serving, KVCache, KVCache cache and Multi-turn Serving
LLM serving uses prefill and autoregressive decoding, with KV$ caching avoiding repeated computation within and across requests. Multi-turn serving reuses prior conversation context by concatenating earlier inputs and outputs into later requests.
- LLM serving: LLM serving first processes a prompt during prefill, then generates remaining tokens autoregressively during decoding.Decoding repeatedly uses the original prompt together with previously generated tokens until an end-of-sequence token is produced.
- KVCache: KV$ stores the K and V matrices generated during prefill so decoding can reuse them instead of recomputing them.These matrices arise from computation-intensive attention steps.
- KV$ cache: Requests sharing a token prefix can reuse cached KV$, reducing the next request’s time to first token and increasing serving throughput.The shared prefix lets a later request compute only the K and V values for its unmatched tokens during prefill.
- KV$ cache: KV$ caches use configurable token blocks because checking prefix matches at per-token granularity can be costly.vLLM groups 16 tokens per block by default.
- Multi-turn serving: Multi-turn serving concatenates previous turns’ inputs and outputs with a new prompt so the model retains conversation context.The resulting expanded input is then fed to the LLM for the next request.
3 Characterize KV$ Workload in the Wild
Real-world traces show substantial but highly uneven KV$ reuse: reuse patterns vary across workload and request categories, while specific categories exhibit predictable temporal behavior. These findings indicate that cache effectiveness depends strongly on workload characteristics rather than multi-turn activity alone.
- Reuse volume: 62% and 54% ideal cache hit rates occur on Trace A and Trace B, respectively, below synthetic-workload reports exceeding 80%.The ideal rates assume infinite cache capacity; each hit represents a KV$ block reused from a prior request.
- Reuse sources: 97% of Trace B cache hits come from single-turn requests despite a multi-turn ratio below 0.1%, enabled by shared system prompts in API calls.Trace A instead contains substantial chatbot traffic, with request types generally showing 47–51% multi-turn ratios.
- Reuse skew: KV$ hits are concentrated: 19% of Trace A user requests and 4% of Trace B user requests account for more than 90% of block hits.The skew reflects uneven request distributions and substantial variation in per-user hit behavior.
- Temporal locality: Reuse timing is typically short, with 80% of reuse times under 10 minutes in Trace A and under 10 seconds in Trace B.The contrast is associated with human-in-the-loop versus computer-in-the-loop request generation.
- Temporal locality: Single-turn reuse differs by workload: fewer than 30% of Trace A requests are reusable, compared with more than 50% in Trace B.Request types also differ: chat has longer reuse times than multimodal requests but shorter times than file understanding.
- Workload awareness: Reuse-time probabilities fit exponential distributions within request categories, but distributions differ across categories and turns, supporting workload-aware caching.Spatial locality is also workload-dependent: text and multimodal workloads show it, whereas file and search workloads show little.
4 Improved KV$ cache system
The paper replaces workload-agnostic KV$ cache policies with a workload-aware eviction design based on reuse probability, locality, and KV$ lifespan. The policy improves hit rate and queued TTFT, especially when cache capacity is limited, while adding negligible serving overhead.
- Existing cache systems: Existing KV$ systems use hierarchical GPU/CPU caching with block-granular storage and policies such as LRU or FIFO.Blocks are first cached in GPU HBM and evicted to CPU when GPU capacity is insufficient.
- Design motivation: LRU can evict long-reuse-time blocks from multi-turn requests when many single-turn requests arrive.For the same elapsed time since last access, 10-turn text requests are more likely to receive a subsequent turn than 1-turn requests.
- Workload-aware policy: The proposed policy assigns each KV$ block a priority based on workload-specific reuse probability, spatial locality, and expected lifespan, evicting the lowest-priority blocks first.Reuse probability is estimated from sampled reuse data using fitted exponential distributions; locality is represented through prefix length.
- Performance optimization: The policy reduces eviction-candidate search from O(N) to O(W), where W is the number of workloads, with 79 µs per-eviction delay and 1.2 % scheduling overhead.Per-workload priority queues expose the least-recently-used block from each workload before final candidate selection.
- Evaluation: The evaluation reports a fairness limitation shared by existing cache systems: clients can monopolize prefix cache with many requests sharing long identical prefixes.The paper treats fairness as orthogonal to cache policy and points to serving-strategy co-design as a possible remedy.
5 Related Work
Related work covers prefix-based and non-prefix KV$ reuse, KV$ compression and deletion, general caching-policy optimization, and broader LLM-serving optimization. This paper focuses specifically on characterizing production KV$ workloads and using their reuse properties to optimize caching policies.
- KV$ reuse: Existing production systems primarily reuse KV$ through prefix matches, while other studies investigate non-prefix KV$ caching.Prefix matching preserves the original inference algorithm without accuracy loss.
- KV$-related optimizations: KV$ compression, deletion, token selection, and quantization reduce per-request KV$ size, sometimes with accuracy degradation.The characterization is compatible with these techniques when compressed or deleted representations remain reusable.
- Caching policies: Caching-policy research spans general-purpose policies and domain-specific systems, and this work extends that line by exploiting characterized KV$ reuse properties.The paper uses workload observations to guide KV$ cache-policy optimization.
- LLM serving: The paper addresses LLM-serving optimization through production-workload characterization for KV$ caching and is orthogonal to optimizations outside KV$ cache.Its focus is the workload and policy layer rather than other serving-system optimizations.
6 Conclusion
The paper presents a systematic characterization of production KV$ workloads and derives a workload-aware eviction policy from the observed reuse patterns. The resulting approach improves on workload-agnostic policies and offers a workload-driven direction for LLM-serving systems.
- Conclusion: The study provides the first systematic and in-depth characterization of production serving workloads for KV$ cache systems.Its findings address patterns not captured by synthetic workloads used in previous studies.
- Conclusion: The proposed workload-aware eviction policy improves upon workload-agnostic policies such as LRU and LFU.The paper presents the work as a starting point for improving current and future LLM-serving systems through workload-driven design.
A Appendix
Reuse-probability prediction is performed by using historical reuse logs to estimate the next day’s probabilities for each request category. Image requests are harder to predict than other categories.
- Reuse prediction: Historical reuse logs from the previous day can predict next-day reuse probabilities for each request category.The prediction is described for workdays and uses category-specific reuse behavior.
- Reuse prediction: Image-type reuse probabilities are harder to predict than those of other categories.The paper attributes this to coarser reusability granularity and greater variability among users.
A.1 Trace results across different days
Figure 29 examines how reuse-probability distributions vary across request categories and future time windows over five continuous workdays. It specifically compares the 9:00–10:00 a.m. reuse probability across days.
- Figure 29 plots the cumulative distribution of reuse probability for each request category in a future time window.The figure tracks these distributions across five continuous workdays.
- The analysis spans five continuous workdays to examine reuse-probability evolution across days.
- The 9:00–10:00 a.m. reuse probability is compared across the observed workdays.