Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

1 Introduction

• KV caches avoid redundant computation by storing intermediate tensors, but can reach millions of tokens as prompts and outputs grow, creating memory bottlenecks.KV-cache techniques target memory savings and throughput and latency efficiency.Source: §1 · p. 1
• The survey organizes sKis by when KV data is accessed or scheduled, where it is placed or moved, and how it is represented or retained.It also analyzes cross-behavior co-design patterns and behavior–objective effects to identify overlooked regions and open challenges.Source: §1 · p. 1
• Unlike general inference surveys that treat KV caches as a minor component, this survey focuses exclusively on sKis and offers a behavior-oriented perspective.Closest KV-specific surveys typically organize methods by lifecycle stages or optimization layers (Shi et al., 2024; Li et al., 2024b; Liu et al., 2025c).Source: §1 · p. 2
• The authors present the temporal-spatial-structural behavior space as a first framing of KV-cache optimization and a stable lens for situating new techniques.They argue that this design space is decoupled from model and kernel details and supports principled analysis and actionable future directions.Source: §1 · p. 2

Foundations

• Inference has prefill and autoregressive decode phases; KV caches store previously computed intermediate tensors so attention need not be recomputed across the sequence.Prefill processes the initial input and generates the first output token; decode generates subsequent tokens autoregressively.Source: §2 · p. 2

Scope

• A method falls within sKis if it optimizes KV caches during inference to improve system metrics without retraining base-model weights or modifying Transformer architecture.sKis means system-aware KV infrastructure for serving LLMs.Source: §2 · p. 2
• The survey excludes attention or KV-mechanism redesigns (DeepSeek-AI, 2024) and backbone changes (Ashkboos et al., 2024; Li et al., 2024d).Its focus is serving-time KV-cache optimization for off-the-shelf models, without model alteration or adaptation.Source: §2 · p. 2

Taxonomy

• The taxonomy organizes sKis literature by low-level system behavior, assigns each work 1–2 primary categories based on its main contributions, and leaves minor associations to Appendix D.Like operating systems, LLM serving techniques can span scheduling, memory, and I/O dimensions; the taxonomy’s methods are summarized in Figure 4.Source: §2 · p. 2, Figure 4 · p. 3

3.1 KV-centric Scheduling

• KVS integrates KV characteristics into runtime decisions across request, token, and kernel levels.Request-level methods use predicted KV usage to mitigate prefill-decode interference or prioritize reuse; token-level methods select entries by estimated contribution, while kernel-level scheduling maps attention work by query and KV lengths. Examples include TetriInfer (Hu et al., 2024b), Zheng et al. (2024), Srivatsa et al. (2024), Qin et al. (2024), Xu et al. (2025a), and FlashInfer (Ye et al., 2025).Source: §3.1 · p. 3

3.2 Pipelining and Overlapping

• OVLP hides KV-related latency by overlapping compute, KV-data movement, and communication, reducing idle time and improving efficiency.The surveyed methods are summarized by overlap mode, operations, and granularity.Source: §3.2 · pp. 3–4, Table 2 · p. 4

3.3.1 Disaggregated Inference

• Disaggregated inference separates compute across hardware resources to reduce contention and improve utilization, including operator-level attention splitting and prefill-decode separation.Infinite-LLM (Lin et al., 2024) splits attention across distributed instances; Mooncake (Qin et al., 2024) combines prefill-decode disaggregation with KV-centric scheduling and a distributed cache pool, while MuxServe (Duan et al., 2024) colocates prefill-decode jobs through SM partitioning.Source: §3.3.1 · p. 4

3.3.2 Compute Offloading

• Compute offloading moves partial compute to auxiliary devices to ease GPU bottlenecks by exploiting hardware heterogeneity and workload features.CPU offloading often follows a compute-near-cache principle; examples include FastDecode (He and Zhai, 2024) and Neo (Jiang et al., 2025c), while other methods use computational storage drives (Pan et al., 2024) or processing-in-memory (He et al., 2025; Park et al., 2024).Source: §3.3.2 · p. 4

3.3 Hardware-aware Execution

• KVS and OVLP target KV reuse and stall hiding, but controlled-workload evaluations leave robustness under bursty traffic and multi-tenant settings under-analyzed.HAE can improve throughput and hardware utilization through compute decoupling and kernel specialization, though reliance on low-level primitives can hinder portability.Source: §3.3.2 · p. 4

4.1 Memory Hierarchy KV Orchestration

• MHO uses importance-aware tiering to retain critical KV entries in GPU HBM, with some systems keeping proxy signals on GPU to guide prefetch from CPU-resident caches.Examples include ArkVale, OmniKV, ClusterKV (Liu et al., 2025b), PQCache (Zhang et al., 2025a), and SpeCache (Jie et al., 2025).Source: §4.1 · pp. 4–5
• MHO also places caches across GPU, CPU, disk, or CXL pools using cost models and runtime signals, while intra-GPU methods prefetch KV from HBM into L2 selectively.Runtime signals include queueing state, memory pressure, compute and I/O costs, and future reuse; graph-level insertion can avoid cache pollution.Source: §4.1 · p. 5

4.2 Compute Device KV Orchestration

• CDO distributes KV across prefill and decode GPUs or disaggregated workers, including pull-based fetching and CXL shared memory transfer.DistServe places prefill and decode across clusters with different node affinities; TraCT (Yoon et al., 2025) uses CXL shared memory.Source: §4.2 · p. 5
• CDO also supports remote KV transfer through bandwidth-adaptive bitstream streaming and heterogeneous offload to PIMs or CSDs.CacheGen (Liu et al., 2024c) encodes KV tensors into bitstreams and adapts streaming to runtime bandwidth.Source: §4.2 · p. 5
• MHO and CDO target interconnect bottlenecks and often overlap transfers with compute, but contention among concurrent transfers and joint offload-prefetch optimization remain gaps.The survey notes that bandwidth contention makes tail behavior hard to analyze, and that shared-memory and interconnect budgets complicate joint optimization.Source: §4.2 · p. 5

5.1 KV Cache Compression

• KV cache compression is a central research thrust because it directly reduces memory usage.Source: §5.1 · p. 5

5.1.1 KV Cache Quantization

• Quantization lowers KV precision, often using per-channel keys and per-token values, handling outliers at higher precision, while vector quantization captures inter-element correlation.CQ (Zhang et al., 2024c) learns centroids for 1-bit KV; VQ-LLM (Liu et al., 2025d) and CommVQ (Li et al., 2025a) reduce overhead with fused kernels and RoPE-commutative codebooks.Source: §5.1.1 · pp. 5–6
• At very low bitwidths, preserving outliers and heterogeneous sensitivities makes compression quality difficult, and memory savings yield system gains only when runtime costs are integrated effectively.Quantization and dequantization costs, extra kernel boundaries, and workload conditions matter; gains are more likely for memory- or I/O-bound workloads when quantization is fused or overlapped.Source: §5.1.1 · p. 6

5.1.2 KV Cache Low-rank Approximation

• Low-rank approaches target cached KV (xKV (Chang et al., 2025a)), projection weights (Palu), or the QKV attention subspace, trading cache-level plug-and-play use for projection cost, kernel coupling, or training overhead.Some methods learn extra parameters and require training, placing them outside the survey’s sKis scope.Source: §5.1.2 · p. 6

5.1.3 KV Cache Structural Compression

• Structural compression changes cache organization by pruning structural units or merging them, with decisions often guided by attention or similarity and sometimes by clustering.Examples of clustering-based approaches include Wang et al. (2024a), Agarwal et al. (2024), and Zhang et al. (2025c).Source: §5.1.3 · pp. 6–7

5.2.1 KV Cache Allocation and Reuse

• Allocation and reuse methods use mapped fixed-size pages to reduce fragmentation (PagedAttention (Kwon et al., 2023)), prompt-sharing indices such as radix trees, standardized kernel layouts, and semantic selection or reuse.Semantic approaches compute KV only for critical tokens (Fu et al., 2024) and extend reuse to disaggregated serving with the elastic Mem-Pool system (Hu et al., 2024a).Source: §5.2.1 · p. 7

5.2.2 KV Cache Eviction

• Eviction discards less critical token states under a budget, using static prefill-time or dynamic decoding-time updates, recent or sink-token retention, lightweight importance signals, and layer- or head-aware budgets.Source: §5.2.2 · p. 7
• KV-cache compression provides direct memory relief, but system gains require co-design, while retention management faces challenges in fast, stable utility estimation and robustness across workloads.The survey notes that retention policies are often workload-sensitive and their robustness in complex serving environments remains under-studied.Source: §5.2.2 · p. 7

6.1 Key Observations

• Structural methods dominate study and memory savings; temporal methods target latency and throughput but rarely report tail latency, while spatial methods target interconnect I/O and often pair with overlap.Energy remains under-explored. The affinity network reflects observed co-design patterns, not validated performance gains.Source: §6.1 · p. 8
• Quality loss is a risk across temporal, spatial, and structural methods; HAE–CDO is the strongest co-design pattern, while popular KVCC remains isolated.The section links temporal risks to inconsistent request handling, spatial risks to missed KV data, and structural risks to reduced KV precision.Source: §6.1 · p. 8

6.2 Open Challenges

• Tail latency dominates user experience (Wang et al., 2024b), yet most systems omit tail metrics, making behavior-attributed, controllable SLO violations a challenge.Long contexts and bursts can trigger violations through KV generation, migration, compression, or cross-behavior interference; standardized preemption and degradation semantics are proposed.Source: §6.2 · pp. 8–9
• Energy-aware sKis remains a gap, motivating runtime power profiling, serving-time energy models, and joint energy–latency–quality optimization under power constraints.The section also proposes measuring per-request or per-token energy under serving traces and budgets, and studying energy-friendly KV granularities and layouts.Source: §6.2 · p. 9
• Trustworthy and efficient sKis should expose worst-case failures and quality lower bounds under stress, making trustworthiness behavior-attributed and SLO-aware.Structural policies can evict or compress crucial low-salience context and cause errors under workload shifts; sKis techniques may also serve as trust mechanisms, as Jiang et al. (2025b) show with KV eviction as a jailbreak defense.Source: §6.2 · p. 9
• Generalizing HAE–CDO requires portability across heterogeneous topologies and adaptation to multi-tenant settings, beyond policies tailored to specific fabrics or single tenants.The temporal–spatial–structural lens is hardware-agnostic, but concrete hierarchies, interconnects, and kernel or runtime support vary.Source: §6.2 · p. 9
• sKis needs shared-budget, behavior-aware co-optimization with intermediate semantics, alongside unified benchmarks because inconsistent metrics prevent apples-to-apples comparisons.Directions include jointly deciding eviction, offload, and prefetch; coupling KVCC with OVLP at the edge; and standardizing trust and KV-resource metrics, stress workloads, and reporting.Source: §6.2 · p. 9

• The survey provides a systematic, system behavior-oriented taxonomy of sKis spanning temporal, spatial, and structural dimensions.Source: §7 · p. 9
• Its cross-analysis of behavior-objective impacts and behavior-behavior co-design patterns reveals overlooked regions and open challenges.Source: §7 · p. 9
• The survey aims to inspire continued exploration toward efficient and trustworthy LLM serving.Source: §7 · p. 9

Limitations

• The review may have overlooked valuable contributions because the related work is extensive and the research area is rapidly evolving.The authors describe the work as a comprehensive review and summary of current methods.Source: Limitations · p. 9
• The survey reports no new experiments and synthesizes public-paper and open-source results, primarily from mainstream platforms and common configurations, which may constrain conclusion generality.It avoids aggregating raw speedup or memory numbers because gains depend closely on model, hardware, workload, and baseline choices.Source: Limitations · pp. 9–10

A Preliminaries on LLMs

• LLMs use sequential Transformer blocks with multi-head self-attention and feed-forward networks; attention produces head outputs that are concatenated before the output is passed to the FFN.The FFN applies two linear transformations with a nonlinear activation, and the modules support contextualized autoregressive modeling.Source: A Preliminaries on LLMs · p. 19

B Design of Our Taxonomy

• The taxonomy classifies methods by operational impact across temporal, spatial, and structural dimensions, rather than by selection techniques themselves.This behavior-oriented view is intended to reflect how serving systems are built and optimized in practice.Source: B Design of Our Taxonomy · p. 19
• For selected KV entries, permanent GPU-memory removal is eviction, offloading for possible retrieval is spatial orchestration, and retaining entries while skipping computation is temporal scheduling.These categories correspond respectively to structural, spatial, and temporal dimensions.Source: B Design of Our Taxonomy · p. 19

C Related Surveys

• General efficient-inference surveys cover serving architectures, Roofline analysis, optimization levels, serving scenarios, or resource efficiency, but typically treat KV-cache optimization as a minor component.The cited surveys include Miao et al. (2023), Yuan et al. (2024), Zhou et al. (2024b), Li et al. (2024a), Zhen et al. (2025), Bai et al. (2024), and Xu et al. (2024b).Source: C Related Surveys · p. 20
• Earlier KV-specific surveys use lifecycle, abstraction-level, or compression-focused taxonomies, leaving serving-time KV-cache behaviors unexamined.The cited works are Shi et al. (2024), Li et al. (2024b), and Liu et al. (2025c).Source: C Related Surveys · p. 20
• The survey focuses exclusively on serving-time, KV-centric system metrics without retraining or architecture changes, and uses temporal, spatial, and structural behavior to support cross-behavior and behavior×objective analysis.The authors say this complements prior surveys and clarifies actionable research gaps.Source: C Related Surveys · p. 20

D Supplementary Paper Categorization

• Table 9 maps surveyed methods across seven subcategories spanning execution and scheduling, placement and migration, and representation and retention.The listed subcategories are KV-centric scheduling, pipelining and overlapping, hardware-aware execution, memory hierarchy KV orchestration, compute device KV orchestration, KV cache compression, and KV cache retention management.Source: D Supplementary Paper Categorization · p. 20, Table 9 · p. 21
• Methods are assigned one or two primary categories reflecting their main contributions, with additional, less-emphasized associations also mapped in Table 9.The table marks secondary categories as G# when they are omitted or only briefly mentioned to keep the paper’s classification focused.Source: D Supplementary Paper Categorization · pp. 20, 22, Table 9 · p. 21
• Splitwise (Patel et al., 2024) combines layer-wise KV transmission with computation in the prefill phase, linking its design to device-level KV transfer and pipelining.The supplementary mapping records these enabling cross-category associations even though they are not elaborated in the main sections.Source: D Supplementary Paper Categorization · pp. 20, 23
• Table 9 chronologically orders representative surveyed methods and lists their publication venues.It provides the supplementary mapping of those methods to their sKis categories.Source: Table 9 · p. 21
• The surveyed methods appear across machine-learning, AI, NLP, systems, architecture, and interdisciplinary venues, as well as impactful arXiv preprints.The paper presents this spread as evidence of the cross-cutting nature of KV cache optimization and its recognition across research communities.Source: D Supplementary Paper Categorization · p. 23

E Takeaways

• The takeaways section states that the comprehensive sKis literature review identified takeaways across several domains.The supplied passage does not specify those takeaways.Source: E Takeaways · p. 23

E.1 Scheduling and Overlapping

• KVS is a multi-objective scheduler that prioritizes reusable, latency-sensitive work and typically favors KV usage over FLOPS.KV reuse-driven scheduling is described as the default paradigm.Source: §E.1 · p. 23
• Lightweight predictors paired with robust policies outperform traditional FCFS or SJF schemes (Hu et al., 2024b; Qin et al., 2024; Shahout et al., 2024).The passage identifies prediction as an enhancement to KVS.Source: §E.1 · p. 23
• OVLP aligns compute with data transfer at the true bottleneck using asymmetric pipelines, such as GPU prefill overlapped with decode attention and KV I/O or collective communication.This approach aims to fill pipeline bubbles by overlapping compute-bound prefill with memory-bound decode work.Source: §E.1 · p. 23
• Preferring recompute to transfer (Jiang et al., 2025a) or prefetching KV caches into L2 during collectives (Dong et al., 2025; Yüzügüler et al., 2025) can reduce pipeline bubbles, especially when bandwidth is the bottleneck.Source: §E.1 · p. 23

E.2 Hardware-aware Execution

• Hardware-aware execution improves throughput, mean and tail latency, and servable context without retraining by matching phase-specific execution to device capabilities.This includes device-specialized kernels, tiling, and memory layouts; keeping hot KV local to compute; and adapting to device access granularity and parallelism.Source: §E.2 · p. 23

E.3 Placement and Migration

• Placement and migration manage hierarchy residency and interconnect bottlenecks by retaining future-useful KV on GPU, tiering the rest, and overlapping transfers with compute.Cost models can select CPU, GPU, or SSD paths (Sheng et al., 2023; Jin et al., 2025); co-adapting paths, precisions, or decoding can reduce TTFT and SLO violations versus static schemes (Zhong et al., 2024; Liu et al., 2024c; Shen et al., 2024). Migration should match attention patterns and device access units, while joint prefetch-deadline and eviction-risk objectives remain an opportunity.Source: §E.3 · p. 23

E.4 KV Cache Compression

• KV cache compression can reduce memory pressure, but algorithm-level compression alone may not improve latency or throughput without system-level co-design.Outliers can dominate performance at low bitwidths or ranks, and isolating them can prevent worst-case error explosions; recent vector quantization methods reach 1–2 bits with modest quality loss (Zhang et al., 2024c; Liu et al., 2025d; Son et al., 2025; Li et al., 2025a). Co-design opportunities include fusing or overlapping compression with attention, aligning packing with transfer units, and exposing tunable parameters to runtime control under SLOs; DeltaKV (Hao et al., 2026) is cited in this context.Source: §E.4 · p. 24

E.5 KV Cache Eviction

• KV-cache eviction serves long contexts under tight memory and bandwidth budgets in both prefill and decode, trading memory and transfer cost against attention utility; heterogeneous layer/head budgets and similarity-based recall or merge preserve salient context better than hard deletion.Source: §E.5 · p. 24

F Behavior-behavior Co-design Affinity Computation

• The co-design affinity method weights primary behavior labels as 1 and secondary labels as 0.5, then uses Tanimoto normalization to control for marginal popularity.It represents relative weighted co-occurrence on a 0–1 scale and draws undirected edges only when affinity exceeds θ = 0.14, with thickness proportional to affinity.Source: F Behavior-behavior Co-design Affinity… · pp. 24–25

G.1 Trustworthy sKis

• sKis optimizations can undermine trustworthiness by dropping rare critical tokens or exposing stale or plaintext KV, while average metrics can hide worst-case failures.The extended challenge discussion supplements § 6.2; proposed directions include quality-lower-bound, recall-SLO, and semantic-violation metrics plus runtime violation detection, recovery, and targeted rectification (Yang et al., 2026).Source: G Extended Discussion on Challenges · p. 25, §G.1 · p. 25

G.2 Intermediate Semantics for Behaviors

• Intermediate semantics model KV as states such as compressed or reclaimable, enabling co-optimization through transitions rather than binary retain-or-evict decisions.Compress-then-offload trades precision for I/O bandwidth, while lazy eviction allows cheap recovery during a grace period before permanent eviction.Source: §G.2 · p. 26

G.3 Benchmarking for sKis; G.3.1 Review of sKis Benchmarking Practices

• sKis benchmarking should measure serving-system performance rather than treating task accuracy benchmarks as its primary objective, but available tools still use inconsistent metric definitions.Framework-specific scripts are systems under test; client-side tools define metrics, while benchmark suites standardize workloads, procedures, and reporting for reproducible comparisons. Examples include MLPerf Inference (Reddi et al., 2020), LLM-Inference-Bench (Chitty-Venkata et al., 2024), and BALI.Source: §G.3 · p. 26, §G.3.1 · p. 26

Metrics

• Report trustworthy SLO-reliability metrics and KV-resource metrics to capture serving quality and resource use.Trustworthy metrics include P90, P95, and P99 latency, SLO violation rate, goodput, recall SLO, and semantic violation rate. KV-resource metrics include cache footprint, effective KV bitwidth, interconnect I/O, tier hit rate, KV-related stalls, effective bandwidth utilization, and energy efficiency.Source: §G.3.2 · pp. 26–27

Workloads

• Benchmark multi-tenant or bursty online, long-context, and heterogeneous workloads to stress temporal, spatial, and structural KV behaviors.Heterogeneous cases include agent workloads (Zeng et al., 2025c; Zhou et al., 2024a), structured editing (Zeng et al., 2025d), and domain-shifted workloads (Tian et al., 2024, 2026; Li et al., 2023, 2025b; Yan et al., 2025a; Zeng et al., 2025a).Source: §G.3.2 · p. 27

Reporting standards

• Report performance across graduated context lengths, accuracy-versus-memory curves for structural methods, and detailed hardware and topology setups.The recommendations supplement basic reporting of model, hardware, and configuration; topology details are especially important for temporal and spatial methods.Source: §G.3.2 · p. 27