PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Large Language Model Acceleration based on KV Cache Management

Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, Lei Chen

arXiv:2412.19442v3cs.AIcs.DC

TL;DR

LLM inference remains difficult to scale because of high computational and memory demands, particularly for long-context and real-time applications. This survey synthesizes KV cache management through token-, model-, and system-level taxonomies, alongside datasets and benchmarks. It identifies architectural, memory-management, scheduling, and future-research directions for improving KV reuse and efficiency.

  • Problem

    High computational and memory demands make LLM inference difficult to scale to real-world, long-context, and real-time applications.

  • Method

    The survey categorizes KV cache acceleration techniques into token-level, model-level, and system-level approaches, and reviews relevant datasets and benchmarks.

  • Results

    The survey presents a comprehensive overview of KV cache management, including token selection and quantization, attention grouping, memory management, scheduling, and hardware-aware designs.

  • Takeaways & Limitations

    KV cache management offers a framework for reducing redundant computation and improving memory utilization across text and multimodal LLM inference scenarios.

  • Takeaways & Limitations

    Cross-category integration, production case studies, and domain-specific optimization remain important underexplored directions for KV cache research.

Abstract

from arXiv · show

Large Language Models (LLMs) have revolutionized a wide range of domains such as natural language processing, computer vision, and multi-modal tasks due to their ability to comprehend context and perform logical reasoning. However, the computational and memory demands of LLMs, particularly during inference, pose significant challenges when scaling them to real-world, long-context, and real-time applications. Key-Value (KV) cache management has emerged as a critical optimization technique for accelerating LLM inference by reducing redundant computations and improving memory utilization. This survey provides a comprehensive overview of KV cache management strategies for LLM acceleration, categorizing them into token-level, model-level, and system-level optimizations. Token-level strategies include KV cache selection, budget allocation, merging, quantization, and low-rank decomposition, while model-level optimizations focus on architectural innovations and attention mechanisms to enhance KV reuse. System-level approaches address memory management, scheduling, and hardware-aware designs to improve efficiency across diverse computing environments. Additionally, the survey provides an overview of both text and multimodal datasets and benchmarks used to evaluate these strategies. By presenting detailed taxonomies and comparative analyses, this work aims to offer useful insights for researchers and practitioners to support the development of efficient and scalable KV cache management techniques, contributing to the practical deployment of LLMs in real-world applications. The curated paper list for KV cache management is in: \href{https://github.com/TreeAI-Lab/Awesome-KV-Cache-Management}{https://github.com/TreeAI-Lab/Awesome-KV-Cache-Management}.

1 INTRODUCTION

LLM inference faces substantial computational and memory demands, especially for real-world scaling, while KV caching reuses previously computed values during autoregressive generation. This survey organizes KV cache management research into a comprehensive taxonomy spanning token-, model-, and system-level optimizations.

  • LLMs achieve broad success across language, vision, multimodal, and other domains, but their inference demands challenge real-world scaling.
  • Autoregressive generation creates a KV-cache opportunity because previously computed intermediate results can be stored and reused across subsequent tokens.
  • Existing efficiency surveys examine broad data-, architecture-, and system-level techniques, whereas this work specializes in KV cache management.
  • The survey provides a detailed taxonomy covering token-level, model-level, and system-level optimization approaches for text-based and multimodal LLMs.
  • It also reviews datasets, evaluation metrics, and benchmark effectiveness across tasks and applications.

2 PRELIMINARY

This section introduces decoder-only Transformers, autoregressive generation, and KV caching as the basis for efficient LLM inference. It explains how cached keys and values reduce repeated computation while creating memory-management trade-offs that motivate optimization.

  • Transformer Decoder: Decoder-only Transformers process sequential data through stacked blocks containing multi-head self-attention and feed-forward networks.The blocks pass outputs sequentially so the model progressively refines its representation of the input sequence.
  • Transformer Decoder: Positional encodings add token-order information before the sequence enters the Transformer blocks.The survey describes RoPE as a relative positional embedding applied within the Transformer layers.
  • Auto-regressive Generation Mechanism: Autoregressive generation predicts each next token from the preceding sequence and continues until an EOS token or maximum length is reached.The next-token distribution is produced from the current hidden state through an output projection and softmax.
  • Key-Value Cache: KV caching stores previously computed keys and values so subsequent decoding reuses them instead of recomputing the entire history.At each step, the new token’s key and value are appended to cached matrices and used in attention across heads and layers.
  • KV Cache Complexity: O(L · h · tc · t · (dk + dv) + L · h · tc (△1 + △2)) is the total saved computation for tc cached tokens across h heads and L layers.The saved time grows with the number of cached tokens and becomes especially significant for longer sequences.
  • KV Cache Complexity: O(L · h · tc · (dk + dv) · sizeof(Float16)) is the additional space required to store cached keys and values in Float16 precision.Reducing cached-token counts or lowering storage precision are presented as ways to reduce this overhead.

3 TAXONOMY

The survey organizes KV cache optimization into token-level, model-level, and system-level strategies, each targeting a distinct aspect of inference efficiency.

  • Token-Level Optimization: Token-level optimization selects, organizes, and compresses token KV pairs without changing the model architecture.Its methods include selection, budget allocation, merging, quantization, and low-rank decomposition.
  • Model-Level Optimization: Model-level optimization redesigns model structures and attention mechanisms to improve KV cache sharing and efficiency.Approaches include attention grouping and sharing, architectural alterations, and non-transformer memory-efficient designs.
  • System-Level Optimization: System-level optimization addresses KV cache management through memory management and scheduling.Examples include virtual-memory adaptation, prefix sharing, layer-aware allocation, prefix-aware scheduling, preemption, and layer-specific cache control.

4 TOKEN-LEVEL OPTIMIZATION

Token-level optimization improves KV cache management by exploiting token-level characteristics and sequential-input patterns, without relying on architectural or parallelization changes.

  • Token-Level Optimization: Token-level methods focus exclusively on KV-pair characteristics and sequential-input patterns rather than model architecture or system parallelization.The survey groups them into five categories: selection, budget allocation, merging, quantization, and low-rank decomposition.

4.1 KV Cache Selection

KV cache selection exploits sparse or uneven attention to retain important tokens and reduce cache costs, using static, permanently evicting dynamic, and retrieval-oriented approaches.

  • Static KV Cache Selection: Static selection compresses the cache once after prefilling, leaving the selected tokens fixed during subsequent decoding.FastGen uses attention-pattern-specific policies such as proximity retention, critical-token preservation, frequency filtering, and complete retention.
  • Dynamic Selection with Permanent Eviction: Dynamic selection with permanent eviction repeatedly selects during decoding and permanently removes unselected tokens from memory.Sliding-window methods evict tokens outside the window, while StreamingLLM preserves initial tokens that maintain model performance.
  • Selection Principles: Attention-based selection methods retain high-impact tokens because attention computations are driven primarily by a select group of tokens.H2O formulates retention around cumulative attention scores and identifies these tokens as Heavy Hitters.
  • Limitations and Retrieval: Permanent eviction can impair long-sequence performance and multi-turn adaptation, while decoding-time selection adds overhead that can reduce end-to-end acceleration.These limitations motivate hierarchical caching, indexing, and system-level retrieval enhancements.
  • Retrieval-Oriented Selection: Index-based and heterogeneous retrieval methods organize or estimate salient KV entries at block, cluster, component, or hash-table granularity.Examples include hierarchical CPU-GPU storage, semantic clustering, approximate top-k attention, asynchronous prefetching, and CPU-based locality-sensitive hashing.
  • Comparison and Open Questions: Static selection is generally more decoding-efficient, whereas dynamic selection is adaptive but incurs additional decoding computation.Both approaches require further validation for multi-turn dialogue, extended decoding, and rapid accurate retrieval under latency constraints.

4.2 KV Cache Budget Allocation

KV cache budget allocation distributes memory according to heterogeneous layer- and head-level information patterns rather than using uniform cache sizes.

  • Motivation: Budget allocation addresses layer heterogeneity by assigning cache capacity according to each component’s contribution to prediction accuracy.Uniform compression across layers may be suboptimal because layers extract different information patterns.
  • Layer-wise Budget Allocation: PyramidKV and PyramidInfer allocate larger budgets to lower layers and progressively smaller budgets to upper layers.The strategy reflects more uniform lower-layer attention and more concentrated upper-layer attention; PyramidInfer additionally selects high-attention tokens per layer.
  • Head-wise Budget Allocation: Head-wise methods allocate cache based on distinct attention concentrations and retrieval roles across heads.AdaKV optimizes preserved attention information, while RazorAttention, HeadKV, and DuoAttention distinguish retrieval-related heads from heads focused on recent tokens or attention sinks.
  • Open Challenges: Budget-allocation research lacks comprehensive comparisons between competing allocation patterns and compatibility with systems such as vLLM and FlashAttention.The survey also identifies real-time, task-specific budget adjustment based on input characteristics, task complexity, or downstream requirements as a future direction.

4.3 KV Cache Merging

KV cache merging compresses or consolidates redundant cache entries to reduce memory use while preserving information needed for accurate attention. The survey organizes merging methods by whether they learn compression, merge token representations, or share information across heads and layers.

  • Overview: KV cache merging reduces cache size by consolidating redundant entries while aiming to preserve attention accuracy.The survey describes merging as a way to optimize memory utilization without significantly degrading model accuracy.
  • Token merging: Learned methods compress accumulating past KV pairs into compact memory spaces, often using indicator tokens or dedicated compression modules.CCM inserts [COMP] indicators and compresses attention KV pairs between them; the survey also identifies learned compression modules in CCM, LoMA, and DMC.
  • Token merging: Training-free methods separate important and unimportant tokens, then merge selected unimportant Keys and Values with retained important tokens.CaM, KVMerger, ZeroMerge, and D2O use rule-based or heuristic strategies to retain potentially useful information.
  • Token merging: KVMerger clusters adjacent tokens with high cosine similarity before merging each strongly related group.The clustering step restricts merging to consecutive tokens with strong contextual relevance.
  • Cross-head and cross-layer merging: Cross-head and cross-layer methods exploit redundancy by computing attention for representative heads or sharing KV representations across similar layers.CHAI clusters correlated attention heads, while MiniCache and KVSharer merge or share cache information across layers.
  • Future directions: Current merging methods are broadly designed across tasks, leaving task- or domain-specific strategies as a future direction.The survey identifies specialization for particular tasks or domains as a way to potentially improve efficiency.

4.4 KV Cache Quantization

KV cache quantization lowers numerical precision to reduce storage and computational overhead during autoregressive decoding. The survey covers fixed- and mixed-precision schemes, outlier handling, and future adaptive extensions for multimodal and multitask settings.

  • Overview: Quantization converts full-precision KV values into lower-bit representations, reducing cache size and memory bandwidth requirements.The survey states that compression from FP32 to INT8 or INT4 can achieve up to 4x or more memory savings.
  • Quantization challenges: Outliers in Keys and Values can cause substantial performance degradation under low-bit quantization, motivating fixed- and mixed-precision approaches.Fixed precision uses one bit-width, whereas mixed precision assigns more precision to critical tokens or components.
  • Fixed-precision quantization: Fixed-precision methods quantize token Keys and Values individually or uniformly, with ZeroQuant dynamically computing each token’s min-max range during inference.Per-token ranges adapt quantization to each token and aim to reduce quantization error.
  • Mixed-precision quantization: Mixed-precision methods retain higher precision for important or sensitive components while compressing less important KV entries more aggressively.Examples include retaining recent KV entries in full precision, assigning 8-bit precision to high-variance components and 4-bit precision elsewhere, and using importance-aware compression.
  • Outlier handling: Outlier redistribution either stores extreme activations in appended virtual tokens or applies equivalent transformations that smooth Keys and Values before quantization.MassiveActivation uses virtual tokens, while SmoothQuant, OS+, and AffineQuant use transformations to facilitate quantization.
  • Future directions: Future work includes real-time adaptive quantization and extensions to multimodal and multitask models with diverse attention patterns and memory demands.The survey identifies token importance, outlier presence, and sequence length as possible adaptation signals.

4.5 KV Cache Low-rank Decomposition

Low-rank decomposition compresses KV caches by exploiting the observation that a small number of components retain most cache information. The survey covers direct cache decomposition, weight-matrix approximation, tensor decomposition, and learned low-rank attention approximations.

  • Overview: KV cache low-rank methods reduce memory requirements by retaining essential information in a small number of singular values or low-rank components.The survey presents low-rank decomposition as a compression strategy intended to preserve accurate attention computations.
  • Direct KV decomposition: Direct cache methods apply SVD or related decompositions to KV representations, often retaining top singular values or selecting keys in reduced spaces.ECKVH groups attention heads before SVD, EigenAttention approximates attention components, and Loki ranks keys in a reduced-dimensional space before exact scoring.
  • Direct KV decomposition: ZDC adapts compression across layers by assigning higher compression to less important tokens in shallower layers and preserving more important tokens in deeper layers.This strategy leverages similarity in token characteristics across adjacent layers.
  • Weight-matrix approximation: Weight-matrix methods decompose Key and Value parameter matrices rather than the cached KV pairs, with LoRC applying progressively stronger compression in deeper layers.Progressive compression is conservative in shallow layers to limit error amplification and more aggressive in deeper layers.
  • Tensor decomposition: Tensor decomposition factorizes large matrices into sequential local tensors, reducing storage while preserving structural information needed for attention.The survey describes MPO as factorizing matrices into smaller local tensors and notes its suitability for KV cache compression.
  • Learned low-rank approximation: Learned low-rank approximation can replace softmax with a separable similarity metric, while projection-based methods compress KV caches along the feature dimension.LESS uses row-wise functions ϕ and ψ, and MatryoshkaKV uses trainable orthogonal projection matrices.
  • Future directions: Current methods commonly use fixed ranks, motivating dynamic rank adjustment based on token importance, sequence length, or layer-specific properties.The survey also identifies incremental decomposition for streaming inference as a future direction.

5 MODEL-LEVEL OPTIMIZATION

Model-level optimization modifies transformer architectures or attention mechanisms to reuse KV information more efficiently. The survey organizes these methods around intra-layer grouping, cross-layer sharing, augmented architectures, and non-transformer processing designs.

  • Attention grouping and sharing: Intra-layer grouping shares key and value representations among query-head groups to reduce redundancy, while cross-layer sharing reuses KV or attention components across layers.These approaches generally require retraining or fine-tuning, although some transformation pipelines support faster deployment.
  • Intra-layer grouping: MQA shares one key and value across all attention heads, whereas GQA divides query heads into groups with separate shared keys and values.GQA uses mean-pooling uptraining to convert MHA models and achieves performance close to MHA with inference time comparable to MQA.
  • Cross-layer sharing: Cross-layer methods reduce cache requirements by sharing KV or attention information across layers, with MLKV reducing cache size to almost 1% of normal GQA strategies while retaining comparable performance.CLA reports an additional 2× KV-cache reduction compared with MQA without changing computational complexity.
  • Cross-layer sharing: Further compression combines cross-layer sharing with head or dimension reduction and quantization, as CLLA compresses KV cache to less than 2% of the original model size with comparable performance.LISA instead aligns attention heads using small feed-forward networks and approximates layer-wise variations with low-rank matrices.
  • Summary and future directions: The survey identifies generalization, temporal and contextual attention variation, retraining cost, and downstream fine-tuning impacts as unresolved challenges for model-level optimization.Static grouping and sharing may not capture changing attention patterns, while some methods struggle with emerging or non-standard architectures.

6 SYSTEM-LEVEL OPTIMIZATION

System-level KV-cache optimization targets memory management, scheduling, and hardware-aware execution across computing environments. The survey highlights paging and virtual memory, prefix sharing, cache-aware scheduling, and layer-specific allocation as complementary strategies.

  • Overview: System-level approaches are categorized into memory management, scheduling strategies, and hardware-aware designs for server systems containing GPUs, CPUs, and memory storage.These directions account for communication and data exchange across system components.
  • Memory management: Architectural memory designs such as vLLM and vTensor apply paging and virtual-memory abstractions to dynamically allocate physical memory for KV caches.PagedAttention stores fixed-size cache blocks non-contiguously, while vTensor separates computation from defragmentation through scheduling, CUDA VMM operations, and virtual-tensor mappings.
  • Memory management: Prefix-aware designs such as ChunkAttention organize cache chunks in prefix trees to detect and share common prefixes across requests, reducing redundancy and accelerating inference.LeanKV combines unified paging with heterogeneous quantization and per-head dynamic sparsity based on token importance.
  • Scheduling: Scheduling methods span prefix-aware, preemptive and fairness-oriented, and layer-specific strategies, addressing cache reuse, latency, fairness, and allocation granularity.Echo coordinates scheduling with KV-cache management for offline throughput and online task SLOs, while LayerKV uses layer-wise allocation, offloading, and SLO-aware scheduling to reduce queuing delays and TTFT pressure.
  • Summary and future directions: Future scheduling systems should become adaptive and predictive while addressing automated tuning, context awareness, coherence, hardware-software co-design, and privacy risks in multi-user serving.The survey specifically identifies potential privacy leaks from sharing and scheduling across users and queries.

6.3 Hardware-aware Design

Hardware-aware KV-cache optimization adapts computation, memory movement, and scheduling to GPU architectures and heterogeneous memory hierarchies. The survey covers shared-prefix, distributed, phase-aware, I/O-focused, and CPU-GPU collaborative designs.

  • Overview: Hardware-aware methods optimize memory access, GPU kernels, parallel load balancing, data movement across memory tiers, and CPU-GPU workload coordination.The survey organizes these designs around single or multi-GPU, I/O-based, heterogeneous, and storage-oriented constraints.
  • GPU-oriented designs: Shared-prefix systems such as HydraGen reuse one KV cache for common prefixes while separately handling unique suffix caches across requests.This design uses batched prefix-cache access and tree-structured attention patterns to improve GPU memory utilization.
  • Distributed processing: Distributed systems such as vLLM and ORCA coordinate KV-cache access across GPUs through block tables, tensor or layer parallelism, and reduced CPU-GPU synchronization.Their designs use GPU workers or threads to process attention while maintaining cache coherence through coordinated communication.
  • Phase-aware and parallel designs: DistServe separates prefill and decoding across GPUs because prefill has bursty, growing-cache access patterns whereas decoding has steadier fixed-cache generation.Multi-Bin Batching groups requests by length, while gLLM uses token throttling and asynchronous message passing for pipeline parallelism.
  • I/O-aware designs: I/O-focused methods reduce data movement through tiling, split attention, prefetching, and granular context-switching policies across HBM, SRAM, PCIe, and other storage tiers.Bifurcated Attention maintains the same computational FLOPs while reducing memory I/O in shared-context batch decoding.
  • Heterogeneous designs: Heterogeneous systems distribute attention computation and KV-cache operations between GPUs and CPUs using asymmetric pipelining, load-aware scheduling, and distributed CPU resources.NEO and FastDecode use CPUs for offloaded attention or memory-bound KV operations to expand available capacity.

7 LONG-CONTEXT TEXT AND MULTI-MODAL BENCHMARKS

The survey assembles long-context text and multimodal benchmarks covering diverse tasks, languages, domains, and input modalities. It also summarizes metrics for measuring generation quality, reasoning, retrieval, classification, and similarity.

  • Text benchmarks: Long-context text benchmarks cover question answering, summarization, reasoning, retrieval, generation, aggregation, and multi-turn dialogue across varied input lengths and domains.The survey includes collections such as SCROLLS, ZEROSCROLLS, LooGLE, LongEval, StreamingEval, and MultiTurnBench.
  • Multimodal benchmarks: Multimodal benchmarks combine text, images, audio, or video and evaluate conversation, description, reasoning, perception, prediction, and summarization tasks.The listed benchmarks include LLaVA-Bench, MMBench, MileBench, MLVU, LongVideoBench, Video-MME, NExT-QA, MVBench, MSVD-QA, and MSRVTT-QA.
  • Multimodal benchmarks: Video-focused evaluations vary in scale and temporal coverage, from MVBench’s 4,000 QA pairs to Video-MME’s 900 videos spanning 11 seconds to 1 hour.NExT-QA contains 5,440 videos and approximately 52K manually annotated question-answer pairs.
  • Evaluation metrics: Evaluation metrics include overlap and similarity measures such as Rouge, METEOR, BERTScore, Edit Similarity, BLEU, and SacreBLEU, alongside EM, PM, accuracy, recall, precision, F1, Pass@k, and Exponential Similarity.These metrics assess output overlap, exactness, retrieval, classification, solution success, and similarity under different task requirements.

8 CONCLUSION

The survey organizes KV cache management into token-level, model-level, and system-level strategies for improving LLM inference efficiency. It identifies cross-category integration, real-world evaluation, domain adaptation, privacy, and implementation as priorities for future work.

  • 8 CONCLUSION: KV cache management spans token-level selection and compression, model-level architectural changes, and system-level memory, scheduling, and hardware optimizations.Token-level methods include selection, budget allocation, merging, quantization, and low-rank decomposition.
  • Future directions: Cross-category integration remains underexplored because evaluating combinatorial configurations is complex.
  • Future directions: Real-world case studies are needed to expose production trade-offs, domain-specific priorities, and implementation challenges.
  • Future directions: Future research should tailor KV cache strategies to domain requirements while addressing privacy, secure isolation, scalability, and framework integration.Examples include retaining critical healthcare tokens and structure-aware strategies for legal and scientific applications.

ACKNOWLEDEGMENTS

The authors acknowledge reviewers and editors for constructive comments and recognize institutional, governmental, and industry support for the research.

  • ACKNOWLEDEGMENTS: The authors thank TMLR reviewers and editors for their constructive comments.
  • ACKNOWLEDEGMENTS: The research received support from government programs, research councils, universities, laboratories, and industry collaborators.The acknowledgments list grants and support involving China, Hong Kong, Microsoft Research Asia, HKUST, and other organizations.
Loading 2412.19442v3…