PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding

Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang

arXiv:2506.15745v2eess.IVcs.LG

TL;DR

Streaming video understanding is constrained by KV caches that grow with time, while existing methods assume offline video access, known queries, or full-cache materialization. InfiniPot-V continually compresses the cache with TaR and VaN under a fixed memory threshold, achieving comparable or better accuracy while supporting efficient long-video and edge-device operation.

  • Problem

    Streaming video requires query-agnostic processing under fixed memory, but prior compression methods assume offline access or allow memory usage to grow with stream length.

  • Method

    InfiniPot-V is a training-free framework that continually compresses the KV cache at a user-defined threshold using TaR for temporal redundancy and VaN for semantic importance.

  • Results

    Across multiple MLLMs and long-video settings, InfiniPot-V matches or exceeds full-cache accuracy while reducing memory and sustaining real-time performance; on Jetson AGX Orin, peak memory remains 9.2–10.7 GB as Full KV grows from 13.8 to 39.0 GB.

  • Takeaways & Limitations

    InfiniPot-V enables practical on-device streaming video assistants by enforcing fixed memory without retraining or query knowledge.

  • Takeaways & Limitations

    The current approach focuses primarily on vision tokens, leaving unified compression of speech, text, and video for future work.

Abstract

from arXiv · show

Modern multimodal large language models (MLLMs) can reason over hour-long video, yet their key-value (KV) cache grows linearly with time-quickly exceeding the fixed memory of phones, AR glasses, and edge robots. Prior compression schemes either assume the whole video and user query are available offline or must first build the full cache, so memory still scales with stream length. InfiniPot-V is the first training-free, query-agnostic framework that enforces a hard, length-independent memory cap for streaming video understanding. During video encoding it monitors the cache and, once a user-set threshold is reached, runs a lightweight compression pass that (i) removes temporally redundant tokens via Temporal-axis Redundancy (TaR) metric and (ii) keeps semantically significant tokens via Value-Norm (VaN) ranking. Across four open-source MLLMs and four long-video and streaming-video benchmarks, InfiniPot-V cuts peak GPU memory by up to 94%, sustains real-time generation, and matches or surpasses full-cache accuracy-even in multi-turn dialogues. By dissolving the KV cache bottleneck without retraining or query knowledge, InfiniPot-V closes the gap for on-device streaming video assistants.

1 Introduction

Streaming video understanding must process incrementally with fixed device memory, but existing compression methods either depend on offline access or allow memory to grow with stream length. InfiniPot-V addresses this with training-free, query-agnostic continual KV-cache compression using TaR and VaN.

  • KV-cache demand grows linearly in streaming video, with a 15-minute 10-fps clip requiring approximately 100 GB for LLaVA-Next-Video-7B.This exceeds the memory available on mobile and robotic platforms.
  • Prior frame sampling, IVC, and KVC methods either degrade temporal coverage, require the full cache before compression, or remain query-dependent and length-scaled.These limitations prevent them from jointly providing fixed memory and query-agnostic retention for streaming video.
  • InfiniPot-V provides training-free, query-agnostic streaming compression that maintains a length-independent KV-cache memory cap.Compression begins when a user-defined threshold is reached and operates continuously during inference.
  • Temporal-axis Redundancy removes tokens similar to recent frames, while Value-Norm ranks remaining tokens by Value-vector norm and applies layer-adaptive pooling.The two metrics target temporal redundancy and semantic salience respectively.
  • Across four open-source MLLMs and six long-video benchmarks, InfiniPot-V reduces contexts to as low as 6K for 50K-token inputs while matching or exceeding full-cache accuracy.It also reaches 14 frames per second with 0.5% compression overhead and supports multi-turn dialogue without query dependence.

2 Background

Streaming video understanding must answer questions over arbitrarily long streams under fixed memory, but existing compression methods assume offline access, unconstrained compression memory, or query knowledge. InfiniPot-V addresses these requirements with continual KV-cache compression that processes frame blocks and compresses at a fixed limit.

  • Streaming video understanding: Streaming video understanding answers questions at any time using only frames observed so far, unlike offline understanding, which assumes the entire video is available.The stream may contain arbitrarily many frames, making bounded-memory processing essential.
  • MLLM video processing: MLLMs encode frames into visual tokens, construct a KV cache during prefill, and update it incrementally during decoding.The visual token count scales with sampled frames, while cached keys and values support autoregressive generation.
  • Prior compression methods: Prior offline methods reduce memory through frame sampling, input-vision compression, or post-prefill KV-cache compression, but each has distinct limitations.Frame sampling reduces frames, IVC prunes vision tokens after encoding, and KVC evicts cache entries after prefill.
  • Limitations for streaming: Existing methods assume unconstrained memory for compression or a known query, and their memory usage grows with stream length in SVU.KVC must materialize the full vision-token sequence and rebuild the memory-intensive prefill when the query changes.
  • Design requirements: SVU therefore requires fixed memory independent of stream length and query-agnostic token retention, which existing methods fail to provide simultaneously.These requirements motivate continual KV-cache compression with InfiniPot-V.
  • InfiniPot-V: InfiniPot-V processes frames in blocks and, when the memory budget is reached, retains recent tokens while combining TaR-selected and VaN-selected entries into a smaller cache.Algorithm 1 allocates retained entries between Temporal-axis Redundancy and Value-Norm selection before continuing the stream.

3 InfiniPot-V: Memory-Constrained Streaming Video Understanding

InfiniPot-V continually compresses the KV cache within a fixed budget by combining temporal redundancy removal with query-agnostic semantic token selection. TaR preserves temporally distinctive tokens while VaN retains semantically important regions, and the combined approach outperforms memory-constrained baselines.

  • 3.1 Temporal-axis Redundancy (TaR) Reduction via Patch-wise Similarity: Key embeddings reveal stronger temporal similarity for static patches across adjacent frames, motivating TaR's patch-wise redundancy reduction.TaR reshapes Key embeddings to compare corresponding spatial patches across recent and past frames.
  • 3.1 Temporal-axis Redundancy (TaR) Reduction via Patch-wise Similarity: TaR assigns higher importance to less temporally similar patches, selects the least redundant past tokens, and always retains the latest frames.The latest-frame tokens preserve newly introduced or rapidly changing content, while distinctive past elements are selectively retained.
  • 3.2 Spatial Semantic Importance Preserving with Value Norm (VaN): VaN ranks tokens by Value-vector ℓ2 norm because higher-norm tokens exhibit greater representation entropy and are treated as more semantically informative.VaN uses Value embeddings for query-independent semantic importance estimation.
  • 3.2 Spatial Semantic Importance Preserving with Value Norm (VaN): VaN applies layer-adaptive spatial pooling, using larger kernels where locality is stronger and smaller kernels where fine-grained detail must be preserved.The pooling kernel is assigned inversely to each layer's coefficient of variation.
  • InfiniPot-V: Memory-Constrained Streaming Video Understanding: InfiniPot-V combines TaR and VaN to balance temporal distinctiveness and semantic importance during KV-cache compression.TaR allocates tokens to low-redundancy content, while VaN fills the remaining budget using pooled Value-Norm scores.
  • 3.3 Design Space Exploration: InfiniPot-V outperforms query-agnostic TTC and STC baselines under memory-constrained VideoMME and MLVU evaluation.Table 2 compares Qwen-2-VL-7B methods at 3K and 6K memory budgets.

4 Experiments

InfiniPot-V is evaluated across offline and streaming video benchmarks, models, baselines, and hardware settings. It preserves or improves accuracy while reducing memory use and maintaining real-time operation under constrained budgets.

  • Offline Video Understanding: InfiniPot-V reduces LLaVA-Next usage to 25% and Qwen-VL usage to 12.5% with minimal performance loss, matching or exceeding LongVU.The evaluation spans four open-source MLLMs and offline and streaming benchmarks, with additional comparisons against commercial and public models.
  • Comparison with IVC under Memory Constraints: 6K memory delivers superior average accuracy to IVC baselines under an 88% lossless compression rate.The comparison uses unified memory-constrained settings with sliding-window attention for competing IVC methods.
  • Comparison with KVC under Memory Constraints: InfiniPot-V consistently outperforms Uniform Select, SnapKV, and InfiniPot across constrained offline tasks, including at a 1/16 compression ratio.Its query-agnostic TaR–VaN selection remains effective when query-dependent methods cannot access the query.
  • Streaming Video Understanding: 44.5 → 47.6 on OVO-BW, 61.1 → 65.9 on OVO-Real, and 75.2 → 76.4 on StreamingBench versus uniform selection.These gains cover backward recall and real-time visual understanding under a 4K memory budget.
  • Ablation Study: TaR and VaN together achieve the highest ablation accuracy, while patch-wise TaR similarity exceeds frame-level similarity at 64.5 versus 62.9.VaN alone also surpasses the baseline, and adaptive pooling improves it from 63.0 to 64.1.
  • Edge Device Deployment Results: Peak memory stays at 9.2–10.7 GB while Full KV grows from 13.8 to 39.0 GB, yielding a 3.6× reduction at 500 seconds.On Jetson AGX Orin, generation throughput reaches 9.1 tok/sec versus 1.2 tok/sec, and 600-second streams run without out-of-memory failure.

5 Conclusion

The paper concludes that InfiniPot-V provides training-free, query-agnostic KV-cache control for streaming video under strict memory constraints. Its continual TaR–VaN compression preserves useful context while supporting the framework’s reported efficiency goals.

  • 5 Conclusion: InfiniPot-V is a training-free KV-cache control framework designed for streaming video in memory-constrained environments.It addresses unavailable queries and strict memory budgets through TaR and VaN token-eviction criteria.
  • 5 Conclusion: Continual compression combines temporal redundancy removal with semantic token selection during streaming inference.TaR compares recent and past-frame patches, while VaN-based adaptive pooling selects spatially important tokens.
  • 5 Conclusion: TaR performs best with r ≤0.25f, while continual compression ratios of at least 0.5 avoid the degradation seen at 0.25.The main experiments standardize α = 0.5, r = 0.125, and |M|/|C| = 0.75.

A.2 Hyper-Parameter Exploration

The hyper-parameter studies identify balanced TaR–VaN allocation, a small recent-frame fraction, and moderate continual compression as the strongest configurations. Excessive recent-frame allocation or compression degrades benchmark performance.

  • TaR and VaN Budget Ratio: Performance peaks for α between 0.4 and 0.6, exceeding VaN-only and TaR-only configurations under a 6K memory budget.This supports jointly allocating memory to temporal redundancy and semantic importance.
  • Recent Frames and Compression Ratio: Setting r = 0.5f lowers scores to 61.3 versus 63.4 on MLVU and 57.2 versus 59.0 on VideoMME.The larger recent-frame set limits temporal redundancy reduction; optimal performance occurs at r ≤0.25f.
  • Recent Frames and Compression Ratio: A compression ratio of 0.25 causes noticeable degradation, whereas ratios of 0.5 or higher are appropriate for CKV.The standard main-experiment setting is |M|/|C| = 0.75.
  • Experimental Configuration: The reported main configuration uses α = 0.5, r = 0.125, and |M|/|C| = 0.75 across experiments.The study also uses uniform frame sampling across benchmarks, with model-specific input settings described for Qwen-2-VL.

B.2 Long Video Understanding Benchmark Details

The benchmark suite covers offline multiple-choice video QA and timestamped streaming QA, with comparisons implemented under common memory constraints. Evaluation also addresses positional encoding and continual streaming behavior.

  • Offline Video Understanding: Offline evaluation uses VideoMME, MLVU, EgoSchema, and LongVideoBench, with MLVU and EgoSchema evaluated on development sets.VideoMME results exclude subtitles because complete subtitle context is unavailable in realistic streaming settings.
  • Streaming Video Understanding: Streaming evaluation uses timestamped open-ended questions from RVS-Ego and EVS-Movie, scoring generated answers with GPT-3.5-turbo-0125.Questions are presented when their timestamps occur, using the compressed KV cache accumulated up to that point.
  • Input Video Compression Details: IVC baselines are compressed to |C| and combined with sliding-window attention to respect the same memory limit |M|.LongVU and DyCoke are adapted under identical CKV constraints for comparison with InfiniPot-V.
  • KV Cache Compression Details: KVC baselines include Uniform Select and SnapKV, with SnapKV using a last-32-token observation window for eviction scoring.These methods are evaluated within continual KV-cache compression settings.
  • Positional Encoding: Streaming positional indices are reassigned within |M| after caching pre-positional-encoding hidden states, enabling arbitrarily long video processing.This handling discards the original positional information of vision tokens.

C Multi-Turn Video Understanding Analysis

The multi-turn case study contrasts query-guided and query-agnostic cache compression, showing that query-specific retention can fail on later questions while InfiniPot-V preserves answers with the same compressed cache.

  • Multi-Turn Comparison: SnapKV matches Full-KV for the initial query because it compresses using Q1, but that query-guided cache does not generalize to Q2 and Q3.The comparison uses Full-KV at 16K and compressed caches at 3K for InfiniPot-V and SnapKV.
  • Multi-Turn Comparison: InfiniPot-V answers all three multi-turn questions accurately with a 3K compressed KV cache, whereas SnapKV makes critical errors on later queries.SnapKV misidentifies a kettle and miscounts bread, while InfiniPot-V recognizes the kettle and counts all four bread pieces.
  • Query-Guided Compression: Attention-based eviction retains top-M tokens using scores aggregated from an observation window containing the user instruction.The compressed Key and Value caches are formed by extracting the selected token indices.
  • Query-Guided Compression: These methods require the full KV cache before compression and require the user query at the end of the context, conflicting with streaming use.The paper characterizes them as query-guided or attention-based cache compression methods.

D.2 Case Study: Towards Streaming Video Understanding with CKV

The CKV case study tests cache compression under full-context, query-unavailable, and streaming conditions. It identifies fixed memory and query-agnostic retention as the two requirements previous attention-based approaches fail to satisfy together.

  • Case 1: Case 1 can retain query-relevant frames because the full input and user query are available before compression.Attention scores identify critical information in the memory-unconstrained setting.
  • Case 2: 60.32 vs 68.75 shows significant degradation when generic or last-vision-token queries replace the unavailable future user query, even with unconstrained memory.The comparison is reported for the query-agnostic Case 2 study.
  • Case 3: Streaming Scenario: In the streaming Case 3, continual compression is required as new frames arrive and the fixed KV-cache capacity is reached.The constrained budget is shared during both prefill and generation.
  • Case 3: Streaming Scenario: The study frames SVU around two requirements: memory that remains fixed with stream length and token retention that is query-agnostic.It reports that existing methods fail to meet at least one requirement and motivates continual KV cache compression.
  • Attention Scoring Analysis: Attention-selected token sets vary across distinct questions, as evaluated by Jaccard similarity across model layers.The analysis uses three questions associated with the same VideoMME video sample.

E Memory and Latency Measurement Results

Memory-unconstrained approaches scale in both peak memory and TTFT as context grows, whereas the memory-constrained approach remains nearly constant across the tested lengths.

  • Memory and Latency: Peak memory for memory-unconstrained methods rises from 21.29 GB at 5K tokens to 79.38 GB at 100K, while TTFT increases from 0.98 to 3.27 seconds.The measurements were collected during prefill on one NVIDIA A100-80GB GPU.
  • Memory and Latency: Memory-constrained continual compression increases peak memory only from 20.93 GB to 22.85 GB and TTFT from 1.08 to 1.20 seconds between 5K and 100K tokens.The reported measurements indicate near-constant resource utilization as context length increases.
  • Comparison with Prior Compression: Prior methods reduce memory at different stages but still require storage or processing that grows with video length, or incur transfer costs through offloading.Frame sampling can reduce temporal coverage, IVC stores the full vision-token set, and KVC materializes the full cache before selection.

G.1 Comparison between InfiniPot-V and KVC

Across offline long-video comparisons, InfiniPot-V is strongest when both prefill and decoding operate under constrained memory, matching or exceeding alternatives under tighter budgets.

  • Query-Dependent versus Query-Agnostic: SnapKV performs robustly when the full context and final query are available, but degrades notably in the query-agnostic setting.Its query-agnostic performance is comparable to uniform selection across both evaluated models.
  • Memory-Constrained Comparison: InfiniPot-V significantly outperforms all three baselines across compression ratios when the constrained budget covers both prefill and decoding.This corresponds to the CKV framework scenario evaluated on Qwen-2-VL and LLaVA-Next.
  • Comparison with IVC: InfiniPot-V matches or slightly exceeds IVC methods at a 6K decoding budget while constraining both vision encoding and decoding to 6K tokens.The IVC methods use the full 50K-token vision encoding budget in this comparison.
  • Comparison with IVC: At a 3K decoding budget, InfiniPot-V achieves higher accuracy than IVC methods that use the full vision encoding budget while keeping both budgets constrained to 3K.LongVU’s STC is the highest-performing IVC approach in this setting.

H Limitation and Future Work

InfiniPot-V currently focuses on vision-token compression, leaving unified handling of speech, text, and video for future work. Its fixed TaR–VaN budget could also be adapted to input characteristics, while end-to-end learning may enable more aggressive compression.

  • H Limitation and Future Work: The current framework primarily compresses vision tokens, whereas future work could address unified multimodal compression across speech, text, and video.Such an extension would target streaming systems that manage diverse modalities under fixed memory constraints.
  • H Limitation and Future Work: InfiniPot-V uses a fixed budget allocation between TaR and VaN, motivating adaptive compression ratios based on input characteristics.The proposed direction allocates more temporal-redundancy reduction to static scenes and more spatial-importance capacity to content-rich frames.
  • H Limitation and Future Work: End-to-end learning could optimize continual compression and potentially support more aggressive ratios through learned token-importance estimation.This contrasts with InfiniPot-V’s training-free design and is presented as a future direction.
Loading 2506.15745v2…