Source-linked AI summary
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
Jiantong Jiang, Peiyu Yang, Rui Zhang, Feng Liu
TL;DR
LLM serving remains memory-intensive and costly, motivating system-aware optimization of KV caches. This survey organizes sKis by system behavior and analyzes their objective impacts, co-design patterns, progress, and open challenges.
Problem
Efficient LLM serving remains difficult because KV caches are central to low-latency, high-throughput inference while serving remains memory-intensive and costly.
Method
The survey defines sKis and organizes prior work into temporal, spatial, and structural system behaviors, then analyzes behavior-objective links and co-design patterns.
Results
The survey reveals progress and gaps in sKis research through behavior-objective impacts, behavior-behavior co-design patterns, and identified open challenges.
Takeaways & Limitations
The taxonomy and cross-analysis provide a foundation for understanding and further exploring efficient and trustworthy LLM serving infrastructure.
Takeaways & Limitations
The survey conducts no new experiments and synthesizes results primarily from public papers and open-source implementations under mainstream platforms and common configurations.
Abstract
from arXiv · showhide
Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.
1 Introduction
LLM serving relies on KV caches to avoid redundant autoregressive computation, but growing contexts make them a major memory bottleneck. This survey therefore examines serving-time KV infrastructure through a system-behavior lens spanning temporal, spatial, and structural dimensions.
- KV caches store intermediate tensors to avoid recomputing attention, but can grow to millions of tokens as prompts and outputs lengthen.This growth creates memory bottlenecks and motivates KV-centric optimization for serving efficiency.
- The survey defines sKis as system-aware, serving-time KV-centric optimization methods and organizes them by execution, placement, and representation behaviors.The three dimensions cover when KV data is accessed or scheduled, where it is placed or moved, and how it is represented or retained.
- Prior surveys generally treat KV caches as a minor component or organize KV-specific work by lifecycle stages or optimization layers.This survey instead focuses exclusively on sKis and adopts a behavior-oriented perspective.
- The survey frames KV cache optimization as a temporal-spatial-structural behavior space that is decoupled from model and kernel details.This framing is intended to support principled analysis, future directions, and stable placement of new techniques.
2 Foundations, Scope and Taxonomy
This section defines sKis as serving-time KV-cache infrastructure for off-the-shelf models and organizes the literature by system behaviors. It covers inference phases, taxonomy categories, and representative methods while limiting emphasis to primary contributions.
- Foundations: LLM inference has prefill and decode phases, with KV caching enabling reuse of previously computed tensors during autoregressive generation.Prefill processes the initial input and generates the first output token, while decode generates subsequent tokens.
- Scope: sKis covers serving-time methods that primarily optimize KV caches and improve system metrics without retraining model weights or modifying the Transformer architecture.The scope targets off-the-shelf models rather than approaches that redesign attention, KV mechanisms, or backbone structure.
- Taxonomy: The taxonomy organizes sKis by low-level system behaviors, while acknowledging that individual methods may span multiple dimensions.For clarity, the survey presents one or two primary categories per work and omits minor associations from the main taxonomy.
- Taxonomy: The taxonomy includes KV-centric scheduling and pipelining or overlapping among its temporal system behaviors.These categories are represented alongside the broader behavior-oriented organization of sKis.
- Taxonomy: The survey associates representative methods with the taxonomy while annotating each method by its primary contributions.Minor category associations are omitted from the figure and listed separately in Appendix D, Table 9.
3 KV Execution and Scheduling
Temporal sKis methods optimize when KV-related work runs and how it overlaps with computation, I/O, and communication. They include KV-aware scheduling, pipelining, and hardware-aware execution, but robustness and portability remain concerns.
- KV-centric Scheduling: KV-centric scheduling integrates KV characteristics into runtime decisions, prioritizing reusable, latency-sensitive, or resource-intensive work.Request-level methods balance load and contention, while finer-grained methods select token-level attention work or schedule kernels using query and KV lengths.
- Pipelining and Overlapping: Pipelining and overlapping hide KV-related latency by coordinating computation with data movement and communication.OVLP targets idle-time reduction by overlapping compute, I/O, or collective communication at different granularities.
- Takeaways and Limitations: KVS and OVLP directly target runtime stalls, while HAE improves throughput and utilization by decoupling compute and specializing kernels.The surveyed methods often use lightweight predictors or cost models, but HAE can depend on low-level primitives that limit portability.
- Hardware-Aware Execution: Hardware-aware execution adapts KV operations to heterogeneous hardware through disaggregation, specialized kernels, and compute offloading.Systems separate prefill and decode resources or move partial attention computation to CPUs, CSDs, and PIMs.
- Takeaways and Limitations: Temporal methods are usually evaluated on controlled workloads, leaving robustness under bursty traffic and multi-tenant settings insufficiently analyzed.The limitation is stated for KVS and OVLP, while HAE introduces additional portability concerns.
4 KV Placement and Migration
Spatial sKis manage where KV caches reside and how they move across memory hierarchies and compute devices. The surveyed methods use tiering, importance-aware migration, prefetching, and distributed placement, while contention and joint budget optimization remain open issues.
- Memory Hierarchy KV Orchestration: Memory hierarchy orchestration distributes KV caches across GPU HBM, CPU DRAM, SSD, and other memory tiers.These methods trade limited fast memory against larger but slower storage, often using importance scores to retain critical entries on GPUs.
- Memory Hierarchy KV Orchestration: Proxy-based systems offload full KV caches to CPUs while retaining lightweight GPU signals to estimate importance and guide prefetching.ArkVale, OmniKV, ClusterKV, PQCache, and SpeCache follow this pattern.
- Memory Hierarchy KV Orchestration: System-cost approaches optimize placement and migration under bandwidth and latency constraints, including GPU–CPU–disk placement and CXL-based memory pools.FlexGen uses a cost model to maximize throughput, while later systems extend the hierarchy with shared or disaggregated CXL memory.
- Memory Hierarchy KV Orchestration: Intra-GPU methods move KV entries between on-chip caches and HBM, using asynchronous prefetching or selective insertion to improve cache locality without pollution.Dong et al. prefetches blocks into L2, while PRESERVE inserts fetches through graph-level optimization.
- Compute Device KV Orchestration: Compute-device orchestration distributes KV across devices, including disaggregated prefill and decode workers, remote networks, and heterogeneous accelerators.Examples use pull-based KV transfer, adaptive bitstream streaming, or attention offloading to PIMs and CSDs.
- Takeaways and Limitations: Spatial methods target interconnect bottlenecks by overlapping KV transfers with computation, but concurrent-transfer contention and joint offload–prefetch optimization remain under-modeled.These gaps make tail behavior difficult to analyze under shared memory and interconnect budgets.
5 KV Representation and Retention
Structural sKis reduce KV memory or manage which KV states exist at runtime through compression, representation changes, layout redesign, and eviction. Their practical benefits depend on preserving quality and integrating overheads into the serving system.
- KV Cache Compression: KV cache compression directly reduces memory usage through quantization, low-rank approximation, and structural changes to cache organization.The survey covers quantization, low-rank methods, pruning, and merging as distinct representation strategies.
- KV Cache Compression: Quantization commonly uses asymmetric granularity, assigning different treatment to keys and values while handling outliers with higher precision or dedicated techniques.Keys may be quantized per-channel and values per-token; outliers become harder to preserve at very low bitwidths.
- KV Cache Compression: Vector quantization captures inter-element correlation with codebooks, while low-rank methods reduce memory by constraining cached tensors, weights, or attention subspaces.The compared methods also differ in kernel overhead, projection cost, and calibration robustness.
- KV Cache Compression: Lower bitwidth does not guarantee end-to-end gains because dequantization, kernel boundaries, and fusion or overlap determine realized serving performance.Memory savings are more likely to improve latency or throughput when workloads are memory- or I/O-bound and quantization is integrated into the system stack.
- KV Cache Retention Management: Retention methods redesign allocation, sharing, and layout, or evict less critical KV states under a memory budget.PagedAttention uses fixed-size pages and logical-to-physical mapping, while prompt-sharing indices and structured kernel formats support reuse and execution.
- KV Cache Retention Management: Eviction policies differ in timing and selection, using static or dynamic updates and often retaining recent or attention-sink tokens alongside lightweight importance signals.Budget policies can be treated as plug-ins to existing eviction rules.
- Takeaways and Limitations: Structural savings may not translate into system gains without co-design, while retention policies remain workload-sensitive and require fast, stable utility estimation.The survey identifies compression overhead, kernel constraints, tail behavior, and robustness in complex serving environments as practical challenges.
6 Observations and Open Challenges
The survey analyzes sKis through behavior–objective links and cross-behavior co-design patterns, revealing strong attention to memory efficiency and gaps in tail control, energy, trustworthiness, and coordination. These observations motivate system-level challenges spanning multiple KV behaviors.
- Key Observations: The behavior–objective matrix tracks direct or indirect effects on latency, throughput, GPU memory, interconnect I/O, energy, and quality degradation.Stars identify direct objectives for which at least 70% of papers report gains, while the affinity network represents observed co-design patterns rather than validated performance gains.
- Key Observations: Structural work is most studied and dominates memory savings, whereas temporal behaviors act most directly on latency and throughput.The survey attributes temporal alignment to scheduling stalls, pipeline bubbles, and device under-utilization.
- Key Observations: Quality loss is universal across temporal, spatial, and structural behaviors, although the specific risks differ by behavior.The survey associates risks with inconsistent request handling, missed KV data, and reduced KV precision.
- Key Observations: HAE–CDO is identified as the strongest co-design pattern, while KVCC remains relatively isolated despite its popularity.The survey links heterogeneous compute layouts with KV placement or transfer and identifies missed opportunities for broader co-design.
- Open Challenges: Tail control requires attributing SLO violations to concrete KV behaviors and paths because long contexts and bursts can trigger violations through cross-behavior interference.The proposed direction includes standardized preemption and degradation semantics for controllable tail outcomes.
- Open Challenges: The survey calls for energy-aware runtime decisions and joint optimization of energy, latency, and quality under power constraints.It also recommends reporting per-request or per-token energy under serving traces and budgets.
- Open Challenges: Future systems should co-optimize eviction, offload, prefetch, compression, and overlap under shared resource budgets.The survey specifically connects KVCC with OVLP when decompression overhead could otherwise negate memory-related gains.
7 Conclusion
The survey presents a systematic overview of system-aware KV infrastructure for serving LLMs, organizing it across temporal, spatial, and structural behaviors. It cross-analyzes behavior-objective impacts and behavior co-design patterns to identify open challenges.
- The survey provides a system behavior-oriented taxonomy covering temporal, spatial, and structural dimensions of sKis.
Limitations
The survey’s conclusions are limited by its broad and rapidly changing literature base, lack of new experiments, and reliance on results reported under particular platforms and configurations. Reported gains are not aggregated because they depend on model, hardware, workload, and baseline choices.
- The authors may have overlooked valuable contributions because the related work is extensive and rapidly evolving.
- The survey conducts no new experiments and primarily synthesizes public papers and open-source implementations under mainstream platforms and common configurations.
- The conclusions’ generality may be constrained by the platforms and configurations used in the surveyed evidence.
- Raw speedup and memory gains are not aggregated because they depend closely on model, hardware, workload, and baseline choices.
D Supplementary Paper Categorization
The supplementary categorization maps surveyed methods to a fine-grained taxonomy of sKis categories and records primary and secondary associations. It also shows that sKis spans machine learning, NLP, systems, architecture, and interdisciplinary venues, while synthesizing cross-domain takeaways.
- Supplementary categorization: The supplementary mapping covers seven subcategories across temporal, spatial, and structural optimization dimensions.The categories include scheduling, pipelining and overlapping, hardware-aware execution, memory and compute-device orchestration, compression, and retention management.
- Supplementary categorization: Methods are assigned one or two primary categories, while additional associations may be omitted from the main discussion and marked as secondary.Splitwise is given as an example of a method whose hardware-aware execution design also involves device-level KV transfer and pipelining.
- Venue diversity: The surveyed methods span machine learning, artificial intelligence, NLP, systems, architecture, and interdisciplinary publication venues.
- Cross-domain takeaways: The review identifies takeaways including multi-objective KV scheduling, prediction-enhanced scheduling, and overlap strategies that target runtime stalls and pipeline bubbles.
E.1 Scheduling and Overlapping
The survey characterizes scheduling and overlapping as runtime mechanisms for prioritizing reusable, latency-sensitive KV work and filling pipeline bubbles. Prediction, asymmetric pipelines, recomputation, and KV prefetching are highlighted as ways to improve these behaviors, especially under bandwidth bottlenecks.
- Scheduling: KV scheduling prioritizes limited resources for reusable and latency-sensitive work, making it a multi-objective optimization problem.
- Scheduling: KV reuse-driven scheduling is a default paradigm, and lightweight predictors with robust policies can outperform traditional FCFS or SJF schemes.
- Overlapping: Asymmetric pipelines overlap compute-bound prefill on GPUs with memory-bound decode attention and KV operations involving I/O or collective communication.
- Overlapping: Recomputing instead of transferring or prefetching KV caches into L2 during collectives can reduce pipeline bubbles, particularly when bandwidth is the bottleneck.
E.2 Hardware-aware Execution
Hardware-aware execution aligns KV-related computation with device capabilities, while placement, migration, compression, and eviction manage locality, transfer cost, and memory pressure. The survey emphasizes co-design across these behaviors, alongside unresolved concerns about quality robustness, tail behavior, and system integration.
- Hardware-aware Execution: HAE maps execution to hardware capabilities, improving throughput, reducing mean and tail latency, and extending servable context without retraining.Phase-specific mappings exploit compute-bound prefill and memory-bound decode, while device-specialized kernels, tiling, and layouts improve locality.
- Placement and Migration: MHO and CDO keep useful KV caches near compute, offload less useful state to CPU or SSD, and overlap transfers with computation.Cost models select memory paths, while reload decisions can follow attention cues and transfer paths can be co-adapted under interconnect bottlenecks.
- Takeaways and Limitations: Current sKis rarely model contention among concurrent KV transfers, jointly optimize prefetch and eviction, or measure worst-case quality and trustworthiness.Average metrics can conceal failures from dropping rare critical tokens, especially under workload distribution shift; the survey also notes inconsistent metric definitions across tools.
- KV Cache Compression: KV cache compression reaches 1-2 bits with modest quality loss, but its memory savings often fail to improve latency or throughput without system-level co-design.Integration with execution, migration, and runtime control can fuse compression overhead, align transfer units, and tune bitwidth, rank, or sparsity under SLOs.
- KV Cache Eviction: Eviction retains selected recent, sink, or important tokens under memory and bandwidth budgets, with heterogeneous budgets and similarity-based recall or merging preserving salient context.Dynamic policies can track changing importance, while merge or recall is stronger than hard deletion for long-context consistency.
G.3.2 Benchmark Design Principles
The survey proposes benchmark design principles to improve comparability across sKis without defining a complete reproducible suite. The principles cover trustworthy and KV-specific metrics, stress-oriented workloads, and reporting standards for scalability, trade-offs, and hardware context.
- Metrics: Future sKis benchmarks should report trustworthy metrics such as tail latency, SLO violation rate, goodput, recall SLO, and semantic violation rate.These metrics measure reliability in satisfying service-level objectives beyond standard aggregate performance.
- Metrics: Benchmarks should report KV resource metrics including memory footprint, effective bitwidth, interconnect I/O, tier hit rate, KV stalls, bandwidth utilization, and energy efficiency.The proposed measures expose resource use and transfer behavior alongside serving performance.
- Workloads: Workloads should include bursty multi-tenant serving, long-context tasks, and heterogeneous or domain-shifted workloads to stress temporal, spatial, and structural KV behaviors.These workload types target concurrency stability, memory and I/O bottlenecks, and robustness beyond standard settings.
- Reporting standards: Reporting should include graduated context lengths, accuracy-versus-memory curves, and detailed hardware and topology configurations.These standards reveal scalability, structural trade-offs, and the hardware conditions affecting temporal and spatial methods.