Source-linked AI summary
StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding
Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, Mengye Ren
TL;DR
Long-video MLLMs are constrained by visual-token context limits and KV-cache memory costs, while existing compression may lose information or require unavailable queries. StreamMem provides training-free, query-agnostic streaming KV compression with bounded memory. It consistently retains high utility across offline and streaming benchmarks, surpasses streaming models, and competes with larger-memory methods.
Problem
Long-video understanding is limited by visual-token context growth and KV-cache memory costs, while existing compression can lose information or require questions before processing.
Method
StreamMem continuously compresses KV caches using visual-token attention to chat-template tokens, input-frame compression, frame-wise merging, and positional consistency through YaRN.
Results
StreamMem consistently retains high utility across three offline and two streaming benchmarks, surpasses state-of-the-art streaming models, and competes with methods using larger memory budgets.
Takeaways & Limitations
StreamMem supports continuous, scalable, memory-efficient video understanding without fine-tuning in streaming and open-world settings.
Takeaways & Limitations
The method is evaluated under a bounded total memory budget distributed across MLLM layers, and earlier streaming compression can forget prior video content when that bound is reached.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have made significant progress in visual-language reasoning, but their ability to efficiently handle long videos remains limited. Despite recent advances in long-context MLLMs, storing and attending to the key-value (KV) cache for long visual contexts incurs substantial memory and computational overhead. Existing visual compression methods require either encoding the entire visual context before compression or having access to the questions in advance, which is impractical for long video understanding and multi-turn conversational settings. In this work, we propose StreamMem, a query-agnostic KV cache memory mechanism for streaming video understanding. Specifically, StreamMem encodes new video frames in a streaming manner, compressing the KV cache using attention scores between visual tokens and generic query tokens, while maintaining a fixed-size KV memory to enable efficient question answering (QA) in memory-constrained, long-video scenarios. Evaluation on three long video understanding and two streaming video question answering benchmarks shows that StreamMem achieves state-of-the-art performance in query-agnostic KV cache compression and is competitive with query-aware compression approaches.
1 Introduction
Long-video MLLMs face visual-token and KV-cache memory constraints, while existing compression can lose temporal information or require queries unavailable during streaming. StreamMem addresses this with query-agnostic, bounded KV memory and reports strong performance across offline and streaming benchmarks.
- Long videos generate too many visual tokens for the LLM context and make KV-cache storage and attention computationally burdensome.
- Existing temporal, spatial, and hybrid compression methods can lose information, while query-dependent methods require questions that may be unavailable during video processing.
- Earlier streaming compression can completely forget earlier video content when its memory bound is reached, motivating StreamMem’s retention strategy.
- StreamMem continuously compresses the KV cache after each incoming clip, maintaining bounded memory without costly offloading regardless of video length.
- StreamMem combines query-agnostic attention-based retention, input-frame redundancy reduction, and frame-wise KV merging to create compact visual memory.
- Across three offline and two streaming benchmarks, StreamMem retains high utility with compact KV caches and surpasses streaming baselines while competing with larger-memory methods.
2 Related Work
Related work spans streaming video systems, long-video token compression, and KV-cache compression. StreamMem is positioned as a query-agnostic method that supports streaming encoding without full-context encoding.
- Streaming video understanding with MLLMs: Streaming systems incrementally encode frames without knowing the video length or downstream query, but ReKV stores all KV cache and LiveVLM discards earlier tokens at its memory limit.
- Long video understanding with MLLMs: Long-video methods reduce visual tokens through training-based compression, but limited context length and frame-level token reduction can cause information loss.
- KV cache compression in LLMs: LLM KV-cache compression commonly uses cross-attention weights to identify important cache entries, a strategy also applied to efficient visual understanding.
- KV cache compression in LLMs: StreamMem belongs to the query-agnostic category and does not require full-context encoding, making it suitable for streaming long-video encoding.
3 Preliminaries
Offline processing uniformly samples a fixed number of frames, whereas streaming processing incrementally encodes incoming clips without knowing the video length. In streaming settings, accumulated KV caches grow linearly unless compressed under a global memory budget.
- Offline video understanding: Offline video understanding uniformly samples frames and encodes their visual tokens before processing the resulting context.The sampled frame count is determined by context-length, computational, or memory constraints.
- Streaming video understanding: At each streaming time step, a new video clip is encoded and forwarded through the LLM, producing layer-wise key and value matrices from accumulated visual tokens.
- KV cache compression: Naively retaining all streaming keys and values causes linear memory growth, making KV-cache compression necessary for long videos.
- KV cache compression: Compressed key and value matrices are maintained at each time step and transformer layer under a global memory constraint.The total budget M is distributed across L transformer layers.
- Design challenge: Effective streaming compression must preserve essential temporal information while bounding memory usage, because existing ReKV and LiveVLM caches grow linearly over time.
4 Method
StreamMem processes streaming video incrementally while keeping a bounded KV memory. It combines redundancy filtering, attention-guided token pruning, frame-wise KV merging, and position-preserving context extension.
- Streaming KV construction: Each incoming clip is filtered for temporal redundancy, encoded by the vision encoder and MLLM, then merged with the previous compressed KV memory.Consecutive frames whose visual-embedding cosine similarity exceeds threshold δ are merged by averaging.
- KV compression: The compression module combines attention-based pruning using proxy query tokens with frame-wise KV merging into compact prototype representations.The updated memory mixes salient visual tokens with frame prototypes to retain both fine-grained and global information.
- Query-agnostic saliency: Chat-template tokens approximate a generic query, allowing cross-attention scores to rank visual-token importance without access to the user’s question.Scores are aggregated across query tokens, and the top-k tokens are retained with the memory budget distributed across transformer layers.
- KV merging: Frame prototypes are computed as attention-weighted averages of visual keys and values, complementing token pruning with compact spatial summaries.The normalized importance score of each visual token determines its contribution to the prototype representation.
- Positional consistency: YaRN extends the visual context window and preserves positional consistency across streaming segments instead of naively reassigning position embeddings.The scaling factor is chosen based on the MLLM’s visual context window length, and the resulting approach improves performance over naive reassignment.
5 Experiments
Experiments evaluate StreamMem across offline and streaming long-video benchmarks, comparing it with query-agnostic, online, compressed, and uncompressed-memory baselines. Results show strong performance under constrained KV budgets, while ablations analyze proxy queries, merging, and frame filtering.
- Main Results: StreamMem outperforms baselines on all offline benchmarks except VideoMME’s “long” subset for Qwen2-VL-7B.It also narrows the full-versus-compressed KV gap on MLVU with Qwen2.5-VL-3B.
- Main Results: At 24K tokens, less than half the full KV-cache size, StreamMem surpasses the full-KV setting.Its margin over InfiniPot-V increases with a larger KV-size budget.
- Main Results: StreamMem outperforms ReKV without offloading and is competitive with InfiniPot-V and Flash-VStream on streaming open-ended QA.The evaluation covers RVS-Ego and RVS-Movie under constrained memory settings.
- Ablation Studies: Generic text and chat-template proxy queries perform similarly, but ground-truth queries remain significantly better, especially for multi-detail tasks.This exposes the challenge of retaining every question-relevant detail without access to the user query.
- Ablation Studies: All frame-wise KV merging methods outperform no merging, while attention-weighted merging with middle insertion improves over simple averaging.Input frame filtering reduces redundancy, with a reported similarity threshold of 0.95 as a sweet spot.
6 Conclusion
The paper presents StreamMem as a training-free, query-agnostic KV-cache compression framework for streaming video understanding under bounded memory. Across offline and streaming benchmarks, it reports state-of-the-art performance while retaining salient visual information without future-query access.
- Conclusion: StreamMem continuously compresses the KV cache using visual-token and chat-template attention scores without requiring future queries.The framework targets continuous video processing under bounded memory for open-source MLLMs.
- Conclusion: StreamMem achieves state-of-the-art performance across diverse offline and streaming long-video benchmarks.The paper also analyzes input frame filtering, KV merging strategies, and positional embedding techniques.
A Experiment Details
The appendix specifies model-dependent context scaling, fixed streaming and compression settings, video-tokenization details, and benchmark-specific MLVU accuracy conventions.
- Hyper-parameter details: YaRN scaling uses λ = 8 for LLaVA-OneVision, λ = 2 for Qwen2-VL, and λ = 1 for Qwen2.5-VL.The difference follows the models’ different default context lengths.
- Hyper-parameter details: All models use a 0.95 frame-filtering threshold, 8-frame chunks, weighted frame-wise KV merging, and middle insertion.These settings are kept consistent across models except for the YaRN scaling factor.
- Video sampling details: Each frame is encoded into up to 130 tokens for Qwen2-VL and Qwen2.5-VL, or 196 tokens for LLaVA-OneVision.The vision processor resizes images so width and height are divisible by 28.
- MLVU evaluation details: MLVU accuracy is computed over the entire benchmark for LLaVA-OneVision and averaged across tasks for Qwen2-VL and Qwen2.5-VL.The two reporting methods are not identical, with whole-benchmark accuracy often somewhat higher.
B Additional Experiments
Additional experiments examine context-window scaling and KV-memory budgets. Results show sensitivity to the YaRN factor and strong performance from StreamMem across different memory sizes.
- KV-size experiments: StreamMem performance is evaluated on MLVU with different KV sizes using Qwen2.5VL-3B.The appendix reports this experiment in Table 8.
- Ablation on YaRN scaling factor: YaRN visual context-window extension significantly improves video understanding, with λ = 4 favoring holistic tasks and λ = 8 favoring detail tasks.The reported sensitivity differs across holistic, single-detail, and multi-detail task groups.