PaperlayerView exact revision on arXiv

Source-linked AI summary

ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics

Chin Ting Hsu, Yu-Syuan Xu, Ling Zou, Hsien-Kai Kuo, Wen-Huang Cheng

arXiv:2609.06663v1cs.CLcs.AIcs.CV

TL;DR

MLLM KV caches create substantial memory and computation costs, while cosine-based diversity measures can overestimate similarity by discarding magnitude information and reflecting anisotropy. ECOKV combines Euclidean distance with cosine similarity, adapts diversity weighting using head-level redundancy, and reduces the observation window; experiments report consistent improvements and state-of-the-art performance across benchmarks and compression ratios. The main geometric trade-off is that Euclidean distance captures richer information but is unbounded and varies in scale across layers.

  • Problem

    KV-cache growth makes MLLM inference costly, while cosine-based diversity can overestimate redundancy because normalization discards magnitude information and hidden representations are anisotropic.

  • Method

    ECOKV combines Euclidean distance and cosine similarity for token diversity, estimates redundancy per attention head, and adaptively balances diversity with importance scores.

  • Results

    ECOKV consistently improves existing eviction methods and achieves state-of-the-art performance across benchmarks and compression ratios, while smaller observation windows improve performance.

  • Takeaways & Limitations

    Complementary geometric metrics and reduced observation windows can improve KV-cache token selection while integrating with existing eviction frameworks.

  • Takeaways & Limitations

    Euclidean distance captures magnitude and direction but is unbounded and can vary substantially across layers, complicating direct cross-layer diversity comparisons.

Abstract

from arXiv · show

Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead of KV cache storage. Recent KV cache eviction approaches incorporate a cosine similarity-based diversity metric with importance metrics to selectively retain critical key-value pairs. However, cosine similarity involves normalization that discards magnitude information, and it often yields uniformly high similarity values across layers due to the anisotropy property of hidden representations. In our study ECOKV, we rigorously deconstruct the capabilities of existing diversity metrics. Moving beyond simple measurement, we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity to capture token diversity from complementary perspectives. Furthermore, we use these two metrics to estimate the redundancy level of each attention head, allowing adaptive weighting between diversity and importance scores during token selection. Finally, we demonstrate that the observation window commonly employed to preserve recent tokens can be substantially reduced, thereby allocating more cache capacity to informative tokens and yielding consistent improvements. Extensive experiments demonstrate that ECOKV achieves state-of-the-art performance under various compression ratios and can be seamlessly integrated with existing KV cache eviction methods. We further analyze the relationship between importance and diversity, and examine redundancy patterns across layers and attention heads.

1. Multimodal Large Language Model Efficiency

MLLMs face growing memory and computation costs as contexts lengthen, motivating training-free KV cache eviction. ECOKV addresses cosine similarity’s geometric limitations with complementary metrics, adaptive redundancy estimation, and smaller observation windows.

  • Longer and more complex multimodal inputs increase inference memory and computation costs, making efficiency a critical concern.
  • Existing diversity methods can overestimate redundancy because anisotropic representations produce uniformly high cosine similarity, while normalization discards magnitude information.
  • ECOKV combines Euclidean distance and cosine similarity to capture complementary magnitude-aware and direction-aware information about token diversity.
  • Smaller observation windows improve performance by allocating more cache budget to ECOKV’s token-selection strategy.
  • Extensive experiments report state-of-the-art performance across multimodal benchmarks and compression ratios, with integration into existing eviction frameworks.
  • The method uses these metrics to estimate attention-head redundancy and adapt the balance between diversity and importance during token selection.

2. Related Work

Prior work uses importance, diversity, and dynamic allocation strategies to compress KV caches, but cosine similarity remains common despite anisotropy. ECOKV argues that Euclidean and cosine metrics provide complementary information for assessing token diversity in MLLMs.

  • Early eviction methods retain top-K key-value pairs using accumulated attention, observation-window attention, Key Norm, or Value Norm importance scores.
  • Later methods allocate cache budgets dynamically across layers and attention heads to reflect differing information density.
  • Diversity-based approaches include key similarity, redundancy-importance balancing, and Euclidean distance to a key centroid, with some omitting importance or directional information.
  • Cosine similarity is widely used for semantic similarity, but transformer anisotropy can make it consistently high across settings.
  • ECOKV combines Euclidean and cosine similarity because the metrics complement each other in assessing token diversity for MLLM KV-cache eviction.

3. Preliminary

During inference, projected queries, keys, and values support self-attention, while the KV cache avoids recomputation but grows with context length. Eviction scores retain important or diverse pairs while preserving a recent observation window.

  • During prefilling, the input is projected into queries, keys, and values through learned matrices for self-attention.
  • The KV cache retains key and value representations across decoding, but its size grows linearly with sequence length and increases memory and computation costs.
  • Eviction assigns scalar scores to key-value pairs and retains the top-K pairs with the highest scores.
  • Importance scoring uses recent key-value pairs, whereas diversity scoring favors outlier tokens to promote semantic diversity.
  • The preserved cache combines selected non-window pairs with the observation window retained in its entirety.
  • Cosine similarity can remain high across layers even when PCA shows nonidentical, spatially spread token representations.

4. Diversity Metrics Analysis

The section compares cosine similarity and Euclidean distance as token-diversity metrics. Cosine similarity captures directional differences but can overestimate similarity, whereas Euclidean distance preserves magnitude information while introducing normalization and cross-layer comparability challenges.

  • Motivation: Diversity metrics are used to reduce information loss and redundancy when selecting tokens for retention.The analysis examines commonly used metrics and their strengths and limitations.
  • Cosine Similarity: Cosine similarity measures angular alignment and remains invariant to vector magnitudes.It is commonly used as a proxy for semantic similarity between text tokens.
  • Cosine Similarity: Cosine similarity discards magnitude information, providing only partial geometric information about token diversity.Because it operates on normalized vectors, it cannot represent magnitude differences.
  • Cosine Similarity: High-dimensional transformer embeddings can yield consistently high cosine similarity, causing the metric to overestimate similarity between embeddings.Average similarity exceeds 0.4 across all Qwen-7B-Instruct layers and approaches 1.0 in the first layer, despite token representations remaining spread in PCA space.
  • Euclidean Distance: Euclidean distance measures differences in both magnitude and position using unnormalized vectors.Its squared-distance decomposition jointly encodes vector magnitudes and directional relationships.
  • Euclidean Distance: Euclidean distance captures richer geometric information than cosine similarity alone but is unbounded and varies in scale across layers.These properties require mapping distance to a normalized similarity measure and complicate direct cross-layer diversity comparisons.

5. Method

ECOKV combines Euclidean and cosine-based diversity to capture complementary magnitude and directional information, then adapts diversity weighting to estimated redundancy for each attention head. It also reduces the observation window so more cache capacity remains available for dynamic token selection.

  • 5.1. Key Diversity Score Assessment: ECOKV measures each key’s Euclidean distance from the centroid, treating larger deviations as more diverse and potentially less redundant.Min-max normalization maps distances to comparable diversity scores without changing token ordering.
  • 5.1. Key Diversity Score Assessment: The unified diversity score combines normalized Euclidean diversity with cosine-based diversity using coefficient α.Euclidean distance contributes magnitude-aware information, while cosine similarity contributes direction-aware information.
  • 5.2. Head-adaptive Redundancy Estimation: ECOKV combines diversity scores with importance scores because high cosine similarity alone does not necessarily indicate high redundancy.The approach uses separate heuristics to better approximate semantic-level diversity.
  • 5.2. Head-adaptive Redundancy Estimation: A head-adaptive redundancy estimator jointly uses Euclidean distance and cosine similarity to determine each attention head’s diversity weight λ.More redundant heads receive larger λ, whereas less redundant heads receive smaller λ.
  • 5.2. Head-adaptive Redundancy Estimation: The final score combines scaled diversity and importance scores, retaining the top-k key-value pairs independently for each attention head.Scaling aligns diversity-score magnitude with importance-score magnitude before combination.
  • 5.3. Minimized Observation Window: Reducing the observation window, even to only the most recent token, gives dynamic selection more budget and can improve performance.The smaller window allows more flexible token-retention decisions under a limited KV budget.

6. Experiment

Experiments show that ECOKV integrates with existing eviction methods and improves performance across multimodal and long-context benchmarks. Ablations support complementary metric weighting, smaller observation windows, robust normalization, and low importance-diversity correlation.

  • 6.2. Evaluation: ECOKV integrates with four eviction methods and consistently improves performance across benchmarks, model architectures, and cache budgets.At budget 256, it matches or surpasses FullKV; under tighter budgets, it outperforms all baselines.
  • 6.2.2. Long-context VQA: ECOKV consistently outperforms baselines on MP-DocVQA across compression ratios, supporting its effectiveness for long-context VQA.
  • 6.3. Ablation Studies: β ≈0.6 consistently achieves strong performance, while fixed α = 0.8 and β = 0.6 remains near-optimal across models and benchmarks.The optimal combination outperforms pure Euclidean and pure cosine configurations, reflecting complementary geometric properties.
  • 6.3. Ablation Studies: Window sizes of 4 or 1 consistently improve performance across benchmarks and model architectures.Reducing the observation window leaves more budget for diversity- and importance-based selection.
  • 6.3. Ablation Studies: All three normalization strategies achieve comparable performance, suggesting that outliers do not significantly affect ECOKV.
  • 6.4. Analysis: Importance-based and diversity-based rankings show generally low to moderate Spearman correlation across layers, indicating that the signals are not fully redundant.

7. Conclusion

ECOKV combines complementary diversity metrics to characterize token geometry, guide token retention, and estimate redundancy for adaptive selection. The conclusion emphasizes consistent improvements, reduced observation windows, and empirical analysis of the method’s components and correlations.

  • 7. Conclusion: ECOKV integrates Euclidean distance and cosine similarity to provide a richer geometric characterization of token diversity.
  • 7. Conclusion: The composite metric prioritizes critical key-value pairs and estimates layer-wise redundancy for adaptive token selection.
  • 7. Conclusion: Reducing the observation window can further improve performance under tight budget constraints.
  • 7. Conclusion: Experiments show consistent improvements over existing eviction methods across benchmarks and compression ratios.
  • 7. Conclusion: The analysis examines component effectiveness and the correlation between importance and diversity across layers.

A.2. Benchmarks

The evaluation covers multimodal question answering, recognition, chart reasoning, captioning, and long-context settings using standardized benchmark metrics. Experiments use LMMs-Eval with deterministic evaluation and report configuration-dependent runtime and hyperparameters.

  • A.2. Benchmarks: The main experiments cover DocVQA, OCRBench, TextVQA, ChartQA, and TextCaps across document, recognition, reasoning, and captioning tasks.
  • A.2. Benchmarks: DocVQA reports Average Normalized Levenshtein Similarity on 5.35k validation cases.
  • A.2. Benchmarks: OCRBench reports accuracy over 1000 question-answer pairs spanning text recognition, scene-text, document VQA, information extraction, and handwritten mathematics.
  • A.2. Benchmarks: TextVQA uses 5k validation cases, ChartQA uses 2.5k test cases, and TextCaps reports CIDEr on 6.5k validation cases.
  • A.2. Evaluation Protocol: LMMs-Eval provides the unified evaluation framework, while hyperparameters and benchmark runtime are reported separately.Runtime is measured in GPU-hours under the stated RTX 4090 configuration and may vary with hardware and workload.

B.1. Observation Window Analysis

Reducing the observation window consistently improves performance under tight cache budgets because eviction methods already tend to preserve recent tokens. ECOKV retains its advantage over baselines even when both use the same window size.

  • Reducing the window size consistently improves performance under tight budget constraints at budgets 64 and 128.
  • Recent-token selection patterns differ across benchmarks, indicating that recent-token importance varies across tasks and modalities.The analysis averages retention frequency across layers and attention heads using 100 randomly sampled test cases from four benchmarks at budget 64.
  • Eviction algorithms naturally preserve recent tokens, making a large fixed observation window often unnecessary and reducing budget for dynamically selected informative tokens.
  • Smaller observation windows consistently improve baseline performance on OCRBench, while ECOKV still outperforms baselines at matched window sizes.
  • ECOKV achieves comparable efficiency to baselines with a smaller cache and lower peak memory than FullKV, although total inference-time reduction is modest for short-output benchmarks.

B.5. Analysis of Diversity Weight

ECOKV uses Euclidean distance alongside cosine similarity to estimate diversity and attention-head redundancy. The two metrics show similar overall layer trends but differ in finer details, while a standard-deviation alternative performs comparably overall.

  • Cosine similarity and Euclidean distance show similar overall trends across benchmarks but differ in finer details, indicating layer-specific variation.
  • Redundancy is relatively high in early layers, low in intermediate layers, and gradually increases toward final layers.
  • Computing the global average over all keys and only non-window tokens yields similar behavior, with the current implementation performing slightly better.
  • Algorithm 1 outputs per-key diversity scores S and a scalar redundancy λ quantifying the overall redundancy of an attention head.
  • The framework combines Euclidean distance with cosine similarity and evaluates additional heuristics for comparative performance.

D.1. Standard Deviation-based Approach

The standard-deviation-based alternative estimates redundancy from key dispersion and uses the resulting weight for diversity. It achieves comparable overall performance to the Euclidean approach, with a slight degradation at budget 64.

  • Standard deviation estimates average distance to the mean, making it comparable to ECOKV’s Euclidean implementation.
  • The standard-deviation method scales dispersion using a temperature parameter, inverts the scaled value into redundancy, and uses rstd as the diversity weight.
  • A larger standard deviation indicates greater dispersion, corresponding to higher diversity and lower redundancy.
  • The standard-deviation-based approach achieves comparable overall performance on ChartQA, with slight degradation at budget 64.
Loading 2609.06663v1…