PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

Michael Wang, Keith Li, Roozbeh Bostandoost

arXiv:2609.07966v1cs.LG

TL;DR

Existing KV cache compression methods trade accuracy, latency, and peak memory differently, making a fixed configuration unsuitable across prompts and resource budgets. MetaKV predicts these outcomes and selects a configuration for each prompt under user-specified latency and memory limits. Across tested constraints, it improves constrained success rate over the best static configuration by approximately 0.07 on average and up to 0.135.

  • Problem

    Existing KV cache compression methods make different accuracy, latency, and peak-memory trade-offs, so a fixed configuration is unsuitable across prompts and resource constraints.

    Source: §1 · p. 2

  • Method

    MetaKV predicts candidate accuracy, end-to-end latency, and peak KV cache memory, then selects a configuration for each prompt under user-specified latency and memory limits.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Approximately 0.07 average and up to 0.135 higher constrained success rate than the best static configuration across tested latency and peak-memory constraints.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the tested settings, prompt-specific selection outperforms relying on the best static configuration.

    Source: §1 · p. 2

    The evaluation covers one LLM, one hardware platform, four datasets, and three KV cache compression methods.

    Source: §4 · p. 8

Abstract

from arXiv · show

Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight prediction models to estimate the end-to-end latency, peak memory, and probability of a correct response for each candidate configuration, and selects the configuration that best satisfies the latency-memory constraints while preserving accuracy. We evaluate MetaKV across ten configurations from three representative KV cache compression methods, KVQuant, H$_2$O, and RocketKV, together with an uncompressed FP16 configuration, on four datasets covering mathematics, science, commonsense reasoning, and reading comprehension. Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135. These results demonstrate the benefit of adapting KV cache compression to individual prompts and latency-memory constraints. Code is available at https://github.com/MichaelWang0505/MetaKV.git

1 Introduction

Long-context inference makes KV cache memory a major concern, while compression methods trade accuracy against latency and memory differently. MetaKV adapts configuration choice to each prompt and deployment budget, outperforming the best static configuration in constrained success rate across the tested settings.

1 Introduction

  • The KV cache can consume up to 80% of GPU memory in long-context inference, motivating compression to reduce memory overhead and serve more requests on limited GPU resources.Lossless methods preserve KV values and accuracy but offer limited savings; this work focuses on lossy compression.Source: §1 · p. 1
  • Quantization can lower peak memory but add latency, whereas token eviction and sparse attention often reduce latency while retaining higher-precision KV values and using more memory.These differences mean no single compression approach is consistently superior across efficiency and accuracy objectives.Source: §1 · p. 2
  • Compression preferences vary with deployment constraints and task: token eviction can be more accurate on math prompts, while quantization can be more accurate on scientific-reasoning prompts.Memory-limited settings may favor quantization, while latency-limited settings may favor token eviction; sparse attention can have the least latency on shorter-context prompts.Source: §1 · p. 2
  • MetaKV predicts candidate accuracy, end-to-end latency, and peak KV cache memory, then selects a configuration that satisfies user constraints while preserving accuracy.Its selection adapts to both the input prompt and deployment environment.Source: §1 · p. 2
  • Approximately 0.07 average and up to 0.135 higher constrained success rate than the best static configuration across tested latency and peak-memory constraints.Constrained success rate is the fraction of prompts answered correctly while satisfying both constraints.Source: §1 · p. 2

2 Designing METAKV

The section motivates prompt-specific selection by showing that compression configurations trade accuracy, latency, and peak memory differently. MetaKV predicts resource use, filters candidates against budgets, and chooses among survivors to favor accuracy.

2.1 Baseline KV cache compression methods

  • The evaluated baselines span quantization (KVQuant), token eviction (H2O), and token pruning plus sparse attention (RocketKV), with settings defined by bits, retained-token percentage, or compression ratio.KVQuant uses 2, 3, or 4 bits; H2O retains 20%, 40%, or 60% of prompt tokens; RocketKV evaluates post-prefill ratios of 8×, 16×, and 32×.Source: §2.1 · pp. 2–3

2.2 Motivating METAKV

  • KVQuant has approximately 8.0× the latency of H2O and 5.7× the latency of RocketKV, while using about 29% less peak KV cache memory than H2O and 71% less than RocketKV on average.Table 1 compares accuracy, average end-to-end latency, and average peak memory for each configuration.Source: Table 1 · p. 3, §2.2 · p. 3
  • The best trade-off varies by task: FP16 has the highest overall accuracy, RocketKV does not outperform the other methods on average, and the fastest method changes across datasets.H2O is more accurate than KVQuant on math problems under aggressive compression, while KVQuant is more accurate on multiple-choice scientific reasoning.Source: §2.2 · pp. 3–4

2.3 METAKV

  • Given a prompt and latency and memory limits, METAKV predicts resource use for each configuration and filters out candidates exceeding either budget.For eligible candidates it prefers FP16 if available; otherwise it keeps the least-compressed setting per method and uses predicted correctness to choose among multiple survivors.Source: §2.3 · p. 4, Figure 1 · p. 4

3 Evaluation

The evaluation profiles ten compression configurations on four datasets, then tests prompt-adaptive selection against a fixed configuration across latency and peak-memory budgets. MetaKV improves constrained success rate across tested constraint pairs, with gains varying by budget and prompt-level configuration choices.

3.1 Experimental setup

  • Evaluation used Meta Llama 3.1 8B across four reasoning and comprehension datasets, profiling ten candidate configurations on 2,048 prompts per dataset, with an 85/15 split yielding 308 test prompts per dataset.The evaluation used a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB VRAM.Source: §3.1 · pp. 4–5

3.2 Evaluating METAKV

  • The static baseline typically uses H2O 20% or 40% at 5–30 MB, while MetaKV's gain at 80 MB grows from 0.01 at 50 s to 0.052 at 2 s.The paper attributes the latter increase to greater flexibility to choose faster configurations while preserving accuracy on other prompts.Source: §3.2 · pp. 5–6
  • Approximately 0.07 average ∆CSR improvement over STATICBASELINE appears across tested constraint pairs, with gains of 0.10–0.13 at the 30 MB peak-memory threshold.Figure 2 reports the improvement as a proportion.Source: §3.2 · p. 5, Figure 2 · p. 6
  • With tight latency and loose memory, CSR rose from 0.314 to 0.366 and violations fell from 45.9% to 35.8%; with tight memory and loose latency, accuracy rose 5.3 points as violations rose 3.1 points.The fixed baseline used H2O 60% in the first setting and H2O 20% in the second.Source: §3.2 · p. 6
  • Under moderate constraints, MetaKV raised accuracy by 10.6 percentage points with only a 0.9-point increase in constraint violations; under loose constraints, violations were 0% and accuracy rose 1.2 points.The static baseline used H2O 20% in the moderate setting and H2O 60% in the loose setting.Source: §3.2 · pp. 6–7
  • Violations reached 38%, but ORACLE also violated up to 35% when no configuration was feasible; MetaKV's violation rate was 0–6 points higher and its accuracy 0.2–1.8 points below ORACLE.Figure 3 compares CSR, violation rate, and accuracy across four representative constraint settings.Source: Figure 3 · p. 7, §3.2 · p. 7
  • Across settings, the reported takeaway is that MetaKV adapts to the active resource bottleneck, reducing violations under tight constraints and improving accuracy when more configurations are feasible.Source: §3.2 · p. 7
  • Selections shift with budgets: MetaKV favors FP16 when resources allow, RocketKV for some latency-limited prompts, and H2O or KVQuant under tight memory constraints.For example, FP16 was selected for 74.9% of prompts at 50 s/80 MB, while H2O 20% and KVQuant 2-bit accounted for 29.2% and 28.6% at 50 s/5 MB.Source: §3.2 · pp. 7–8

4 Discussion and Limitations

MetaKV’s predictors are tied to the deployment environment, and constraint violations can occur when no available configuration meets the budgets or because of prediction error. The discussion also identifies evaluation scope and possible extensions to methods and deployment metrics.

4 Discussion and Limitations

  • New compression methods can enter the candidate pool through offline profiling and predictor retraining without changing METAKV’s core selection logic.Source: §4 · p. 8
  • Using another LLM or hardware platform requires reprofiling candidate configurations and retraining the latency and peak-memory predictors.Transfer learning or lightweight online calibration is identified as future work to reduce this recalibration cost.Source: §4 · p. 8
  • Most constraint violations occur when no available configuration can satisfy the constraints, while a smaller fraction is caused by prediction error.The discussion lists expanding the candidate pool, using conservative prediction margins, relaxing constraints, or provisioning more resources as ways to reduce violations.Source: §4 · p. 8
  • METAKV focuses on end-to-end latency and peak KV cache memory, but can incorporate other metrics as constraints or objectives.Examples include time-to-first-token, throughput, inference cost, and energy consumption.Source: §4 · p. 8
  • The evaluation covers one LLM, one hardware platform, four datasets, and three KV cache compression methods; evaluating additional settings remains future work.The authors caution that these results do not establish that any individual compression method is universally better.Source: §4 · pp. 8–9

A Per-Dataset Results

Table 3 reports accuracy, average end-to-end latency, and average peak KV-cache memory for each configuration on each dataset’s held-out test split.

A Per-Dataset Results

  • Table 3 covers 308 held-out test prompts per dataset and reports accuracy, average end-to-end latency, and average peak KV-cache memory for each configuration.Source: Table 3 · p. 13
Loading 2609.07966v1…