PaperlayerView exact revision on arXiv

Source-linked AI summary

VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference

Lyuke Wang, Zhuo Li, Guangxu Zhu

arXiv:2608.24063v1cs.CVcs.AI

TL;DR

Long-context VLLM inference is expensive because visual tokens create large KV caches and costly attention, while uniform compression can lose important information. VisCache addresses this with prompt-aware frame filtering and PruneKV, achieving strong accuracy and efficiency under aggressive cache retention, with stated limitations in scout alignment and prefilling memory overhead.

  • Problem

    Long-context VLLM inference faces substantial latency and memory bottlenecks, while naive visual-token removal can severely degrade performance.

  • Method

    VisCache is a training-free, plug-and-play framework that filters prompt-relevant keyframes and applies PruneKV for parabolic layer-wise allocation with asymmetric key-value compression.

  • Results

    VisCache achieves the highest average accuracy at 28% retention on the 3B model, reaching 45.64% versus 44.16% for the full cache, and maintains best average accuracy on the 32B model at 56.86%.

  • Takeaways & Limitations

    VisCache consistently achieves superior efficiency-performance trade-offs over existing baselines under aggressive KV-cache compression for long-context VLLM inference.

  • Takeaways & Limitations

    Scout-based filtering may select suboptimal keyframes when its visual representations differ substantially from those of the main model, while PruneKV stores all-layer attention scores during prefilling.

Abstract

from arXiv · show

While Vision Large Language Models (VLLMs) have achieved remarkable success in multimodal reasoning, their long-context inference remains prohibitively expensive due to the massive computation and memory overhead of visual Key-Value (KV) caches. Existing KV compression methods often apply uniform pruning across visual tokens and layers, leading to substantial information loss and degraded performance.To address this challenge, we propose \textbf{VisCache}, a plug-and-play framework for coarse-to-fine \textbf{Vis}ual KV \textbf{Cache} pruning without training, which consists of two synergistic stages. First, a lightweight VLM filters temporal redundancy by selectively forwarding semantically informative keyframes. Second, we introduce {PruneKV}, a surgical KV compression algorithm tailored to the attention dynamics of VLLMs. Unlike rigid pruning strategies, PruneKV adopts a parabolic layer-wise budget allocation together with an asymmetric update mechanism that selectively prunes keys while fusing values, thereby preserving critical contextual information. Extensive experiments demonstrate that VisCache substantially improves inference efficiency, achieving up to {2.35$\times$ speedup} and significant memory reduction while maintaining competitive performance with only {19--28\%} KV cache retention. VisCache consistently outperforms existing baselines, establishing a new Pareto frontier between efficiency and performance for long-context VLLM inference. Code is available at https://github.com/Wlklk/VisCache

1 Introduction

Long-context VLLM inference is limited by visual-token redundancy, KV-cache memory pressure, and quadratic attention cost. VisCache addresses these bottlenecks through prompt-aware frame filtering and layer-aware visual KV compression.

  • Motivation: Longer videos create latency and memory bottlenecks because visual tokens inflate KV caches and quadratic attention computation.The KV cache also intensifies memory-bandwidth contention during long-context inference.
  • Motivation: Naively discarding visual tokens reduces computation but can severely degrade performance, motivating principled KV-cache compression.
  • VisCache: VisCache is a training-free, plug-and-play coarse-to-fine framework that filters redundant frames before inference and compresses visual KV caches during inference.
  • VisCache: A lightweight vision-language scout selects prompt-relevant, diverse keyframes using prompt-aware reasoning and Maximal Marginal Relevance.The selected subset is intended to preserve semantic coverage while reducing temporal visual-token redundancy.
  • PruneKV: PruneKV allocates parabolic layer-wise budgets, retaining more visual tokens in early layers and fewer in deeper layers as visual representations become more abstract.Visual KV entries beyond a truncation threshold are entirely evicted.
  • Contributions: VisCache combines prompt-aware frame filtering with asymmetric key-value updates to improve efficiency-performance trade-offs under aggressive KV-cache compression.

2 Preliminary

VLLM prefilling embeds visual frames and text, concatenates them into one sequence, and computes layer-wise attention projections. During decoding, the model reuses and incrementally updates the prefilling KV cache to generate tokens autoregressively.

  • VLLM Prefilling: Each video frame is encoded into visual tokens, and all frame representations are aggregated into a visual embedding matrix.The visual encoder processes MV frames with NV visual tokens each in a shared d-dimensional space.
  • VLLM Prefilling: Text tokens are embedded separately, then visual and textual embeddings are concatenated along the token dimension into the unified input.The resulting sequence contains MV NV + NT tokens.
  • VLLM Prefilling: At every transformer layer, projection matrices compute query, key, and value representations for the unified input.
  • VLLM Prefilling: The layer-wise key and value representations are cached during prefilling and reused during subsequent decoding.
  • VLLM Decoding: During autoregressive decoding, each newly generated token incrementally updates the KV cache and produces the next token through attention.Generation continues until termination or the maximum generation length.

3 Method

VisCache is a training-free, plug-and-play coarse-to-fine framework that removes temporal redundancy before inference and compresses visual KV caches during generation. Its PruneKV stage uses attention-aware token scoring, parabolic layer budgets, and asymmetric key-value updates to retain information while reducing cache size.

  • Overview: VisCache combines prompt-aware frame filtering before inference with layer-aware visual KV compression during generation.The framework targets temporal redundancy among frames and structural redundancy across visual tokens and layers.
  • Prompt-Aware Scout: MMR selects a compact, diverse, query-relevant keyframe subset by balancing prompt relevance against inter-frame redundancy.A lightweight vision-language scout iteratively adds the highest-scoring frames until the target retention ratio is reached.
  • Token Scoring: PruneKV aggregates attention across layers to score visual-token importance before retaining the top-scoring tokens.The attention-derived score is computed from attention received by each visual token across queries and layers.
  • Parabolic Budget Allocation: PruneKV allocates larger visual-token budgets to early layers and smaller budgets to deeper layers through parabolic decay, with tokens beyond a truncation threshold evicted.This schedule preserves more tokens where fine-grained visual details are encoded and compresses more aggressively in deeper layers.
  • Asymmetric Key-Value Update: PruneKV removes low-ranked keys while redistributing their values to retained tokens through similarity-based weighted aggregation.The redistribution matrix captures semantic affinity, and the aggregated dropped values are fused with the original kept values.
  • Asymmetric Key-Value Update: The resulting compressed KV pair replaces the original visual cache for subsequent decoding steps.This asymmetric update preserves information carried by dropped values while reducing the retained key-value structure.

4 Experiment

Experiments evaluate VisCache across VQA, video summarization, memory, latency, and ablation settings. VisCache delivers strong accuracy and efficiency under aggressive KV-cache retention, with parabolic allocation, value fusion, and shared ranking contributing to its performance.

  • Evaluation Setup: VisCache is evaluated on VQA and video summarization benchmarks using matched retention ratios and Qwen2.5-VL models from 3B to 32B.The evaluation includes ActCap, DREAM1K, NExTQA, ActivityNet-QA, EgoSchema, and MVBench.
  • Efficiency: 40% retention reduces FLOPs to 9% of full-cache computation on 3B and 15% on 32B, while 28% and 19% retention reduce 3B FLOPs to 7% and 6%.VisCache records the lowest FLOPs across the controlled-retention settings reported in Table 1.
  • Main Results: At 28% retention, VisCache reaches 45.64% average accuracy on the 3B model and 56.86% on the 32B model, outperforming the full cache or baselines in the reported comparisons.The 3B result exceeds the full-cache average of 44.16%; the 32B result discards 72% of the KV cache.
  • Main Results: 52.6% on 3B and 55.4% on 32B are VisCache’s highest average accuracies on MVBench at the matched retention setting.On the 32B model, VisCache leads on 11 of 20 tasks and exceeds PDrop by 0.5 points in average accuracy.
  • Efficiency: 2.35× is the maximum reported E2E speedup at 19% retention, increasing from up to 1.93× at 28% retention while maintaining performance close to baseline.VisCache’s TPOT decreases substantially, whereas TTFT changes minimally; the resulting decoding savings reduce overall E2E latency.
  • Ablation Studies: Parabolic allocation consistently performs best among the four ablated budget strategies, and value fusion improves accuracy for Parabola, Fixed, and Arithmetic allocations.Geometric allocation degrades substantially, and fusion slightly harms it when too few tokens remain for meaningful redistribution.
  • Ablation Studies: Shared global token ranking improves average VQA accuracy from 44.26 to 45.64 at 28% retention and from 43.95 to 44.85 at 19% retention.The shared ranking separates token identity selection from each layer’s retention cardinality and achieves the best result on most individual benchmarks.

5 Related Work

Existing efficient VLLM inference methods reduce visual-token redundancy through query-aware frame localization, token reduction, or KV-cache compression.

  • SeViLA performs query-aware keyframe localization and question answering through a self-chained design.
  • LongVU is identified among methods targeting efficient VLLM inference through visual-token redundancy reduction or KV-cache compression.
  • The related methods address efficiency by localizing relevant video clips, reducing visual tokens, or compressing KV caches.

6 Conclusion

VisCache is a training-free framework for efficient visual KV-cache compression in long-video VLLMs, combining prompt-aware filtering with layer-aware PruneKV to preserve hierarchical context.

  • VisCache provides a training-free framework for efficient visual KV-cache compression in long-video VLLMs.
  • The framework combines prompt-aware filtering with layer-aware PruneKV to improve long-context video efficiency.
  • VisCache preserves hierarchical context while improving efficiency in long-video VLLM inference.

Limitations

The paper identifies limitations in scout-backbone alignment and the memory overhead of storing attention scores during prefilling.

  • The scout-based temporal filtering stage may select suboptimal keyframes when the lightweight VLM and main LLM use substantially different visual representations.
  • PruneKV currently requires storing attention scores from all layers during prefilling, introducing additional memory overhead.
  • Future work includes scout-backbone co-adaptation and memory-efficient attention-score computation.

Statement of Impacts

The paper describes VisCache’s impact, PruneKV mechanics, and supporting ablations. It reports efficiency benefits from combining keyframe filtering with layer-aware KV compression while examining ranking consistency and truncation trade-offs.

  • Statement of Impacts: VisCache aims to lower the computational barrier to deploying VLLM video understanding by reducing memory consumption and latency.
  • Statement of Impacts: The authors caution that efficient VLLM inference may facilitate automated surveillance and mass media monitoring, requiring attention to privacy, consent, and fairness.
  • PruneKV: PruneKV aggregates visual-token attention scores, retains tokens according to layer-wise budgets, and compresses visual KV entries asymmetrically.
  • Keyframe Selection: The visual-token sets selected by the MMR-guided small VLM consistently show high Jaccard similarity with tokens receiving higher VLLM attention across compression ratios.
  • Truncation Ablation: Increasing the truncation layer generally improves accuracy but slows inference, whereas earlier truncation accelerates inference with noticeable performance degradation.
  • Truncation Ablation: The study selects truncation layer 48, or 3/4 of the 64-layer model, as the balance between reasoning capability and latency.

E Impact of Fusion Strength

Fusion strength controls the balance between preserving useful visual information and introducing redundant or noisy V-cache content. Experiments select µ = 0.7 as the default because it achieves the highest accuracy with stable repeated-run behavior.

  • µ = 0.7 achieves the maximum inference accuracy on EgoSchema with Qwen2.5-VL-32B-Instruct.Fusion strengths from 0.1 to 0.9 were evaluated.
  • Below µ = 0.7, insufficient fused V-cache contribution can lose useful semantic features and reduce accuracy.
  • Above µ = 0.7, excessive fusion introduces noisy or mismatched representations that contaminate retained semantic information.
  • µ = 0.7 provides the best average performance while maintaining relatively low variance across three independent runs.It is therefore adopted as the default fusion strength for subsequent experiments.
  • VisCache is evaluated beyond Qwen2.5-VL on Qwen3-VL-4B-Instruct and LLaVA-OneVision, which use different visual encoders and fusion pipelines.

F.2 Generalization to LLaVA-OneVision

VisCache maintains competitive performance across LLaVA-OneVision and other VLLM architectures while substantially reducing computation under aggressive visual KV-cache compression. Its compatibility extends to 4-bit quantization, though scout choice and model architecture affect the trade-off.

  • F.2 Generalization to LLaVA-OneVision: With 28% retained visual KV cache on LLaVA-OneVision, VisCache reduces FLOPs from 32.73T to 8.62T while keeping performance degradation relatively limited.The reduced computation is 26% of the original cost.
  • F.2 Generalization to LLaVA-OneVision: On ActCap, VisCache slightly improves ROUGE-L, while on DREAM1K and ActQA it preserves most original performance with lower computational overhead.
  • F.2 Generalization to LLaVA-OneVision: VisCache generalizes across VLLM families with different visual encoders and multimodal fusion pipelines.
  • F.2 Generalization to LLaVA-OneVision: Combining VisCache with 4-bit KIVI or FlatQuant generally preserves competitive performance, with some slight degradation on the larger backbone.
  • F.2 Generalization to LLaVA-OneVision: At matched retention ratios on Qwen2.5-VL-3B-Instruct, VisCache achieves 45.64 average VQA accuracy at 28% retention, exceeding the strongest baseline by 2.72 points.
  • F.2 Generalization to LLaVA-OneVision: At 19% retention, VisCache uses 15% of full-cache FLOPs and reaches 44.85 average VQA accuracy, 2.32 points above the strongest baseline.
  • F.2 Generalization to LLaVA-OneVision: VisCache maintains more stable performance than several baselines as retention ratios decrease.

H.2 Results on Qwen2.5-VL-32B-Instruct

On Qwen2.5-VL-32B-Instruct, VisCache preserves a strong efficiency–performance trade-off at both 28% and 19% retention. Its advantage persists at larger scale, although one DREAM1K comparison favors Q-Frame on ROUGE-L at 19%.

  • H.2 Results on Qwen2.5-VL-32B-Instruct: At 28% retention, VisCache uses 12% of full-cache FLOPs and achieves the best DREAM1K ROUGE-L of 7.29 and EgoSchema accuracy of 65.80.
  • H.2 Results on Qwen2.5-VL-32B-Instruct: At 19% retention, VisCache uses 10% of original FLOPs and achieves the highest EgoSchema accuracy of 65.00.Q-Frame and PDrop score 56.00 and 57.00, respectively, on EgoSchema.
  • H.2 Results on Qwen2.5-VL-32B-Instruct: At 19% retention on DREAM1K, Q-Frame reaches ROUGE-L 7.89 versus VisCache’s 7.35 but requires 2.5× more FLOPs and drops more on EgoSchema.
  • H.2 Results on Qwen2.5-VL-32B-Instruct: VisCache achieves a substantially better computational-efficiency and performance trade-off under matched retention budgets across model scales.
  • H.2 Results on Qwen2.5-VL-32B-Instruct: Across retention ratios, DREAM1K ROUGE-L rises with more cache and reaches 10.78 at 90% retention, surpassing full cache around 60%.
  • H.2 Results on Qwen2.5-VL-32B-Instruct: EgoSchema accuracy peaks at 58.60% and declines after 70% retention, showing that optimal retention ratios depend on the task.

K Specific Inference Time Analysis

Lower visual KV-cache retention consistently reduces inference latency. At 19% retention, VisCache reaches up to 2.35× end-to-end speedup while reducing time per output token on both evaluated benchmarks.

  • K Specific Inference Time Analysis: Reducing KV-cache retention consistently decreases overall inference latency on DREAM1K and EgoSchema.
  • K Specific Inference Time Analysis: 2.35× E2E speedup is achieved on DREAM1K at 19% retention, reducing TPOT from 118.81 ms to 45.60 ms.
  • K Specific Inference Time Analysis: On EgoSchema, E2E latency falls from 13.92 s to 8.30 s and TPOT decreases from 59.37 ms to 47.70 ms.
  • K Specific Inference Time Analysis: Scout selection affects downstream VQA accuracy, with CLIP and OpenCLIP outperforming BLIP on several benchmarks while remaining comparable on video summarization tasks.
  • K Specific Inference Time Analysis: The pull-up example shows that VisCache can produce a different video description from Full Cache after temporal compression.
Loading 2608.24063v1…