Source-linked AI summary
FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding
Yiweng Xie, Bo He, Junke Wang, Xiangyu Zheng, Ziyi Ye, Zuxuan Wu
TL;DR
Streaming video understanding must retain long-term context while processing sequential inputs in real time and producing causal responses. FluxMem addresses this with a training-free hierarchical memory that compresses temporal and spatial redundancy using adaptive thresholds. It achieves strong online and offline results while substantially reducing visual-token and system costs.
Problem
Streaming video understanding requires real-time causal responses while preserving useful long-term temporal context, despite substantial redundant visual tokens.
Method
FluxMem uses hierarchical memory with TAS for adjacent-frame redundancy, SDC for within-frame spatial redundancy, and Otsu-based adaptive thresholds.
Results
FluxMem achieves strong performance across online and offline benchmarks, including 76.4 on StreamingBench, 67.2 real-time on OVO-Bench, and 73.1 on MLVU with 65.6% fewer visual tokens.
Takeaways & Limitations
A single training-free hierarchical memory framework supports efficient online and offline video understanding while discarding 60–70% of visual tokens.
Abstract
from arXiv · showhide
This paper presents FluxMem, a training-free framework for efficient streaming video understanding. FluxMem adaptively compresses redundant visual memory through a hierarchical, two-stage design: (1) a Temporal Adjacency Selection (TAS) module removes redundant visual tokens across adjacent frames, and (2) a Spatial Domain Consolidation (SDC) module further merges spatially repetitive regions within each frame into compact representations. To adapt effectively to dynamic scenes, we introduce a self-adaptive token compression mechanism in both TAS and SDC, which automatically determines the compression rate based on intrinsic scene statistics rather than manual tuning. Extensive experiments demonstrate that FluxMem achieves new state-of-the-art results on existing online video benchmarks, reaching 76.4 on StreamingBench and 67.2 on OVO-Bench under real-time settings, while reducing latency by 69.9% and peak GPU memory by 34.5% on OVO-Bench. Furthermore, it maintains strong offline performance, achieving 73.1 on MLVU while using 65% fewer visual tokens.
1. Introduction
FluxMem addresses the difficulty of retaining useful long-term context while producing causal responses from streaming video. Its training-free hierarchical memory adaptively compresses visual tokens and achieves strong online and offline benchmark performance.
- Streaming video understanding requires real-time processing, long-term temporal memory, and causal responses to user queries.
- A global compression policy can under-prune long-term context while over-pruning short-term details needed for causal reasoning.
- FluxMem uses hierarchical short-, mid-, and long-term memory with progressive token reduction through TAS and SDC.
- Its adaptive thresholds derive from Otsu’s method rather than manually tuned retention ratios or similarity thresholds.
- FluxMem reaches 53.3 on OVO-Bench, 76.4 on StreamingBench, 65.3 on VideoMME, 73.1 on MLVU, and 61.1 on LongVideoBench.
- The framework discards 60–70% of visual tokens while reducing latency and GPU memory usage, and adaptive thresholds outperform fixed-rule methods.
2. Related Work
Related work frames streaming video understanding as the problem of processing sequential inputs in real time while managing historical information. Visual token reduction targets redundancy through temporal compression, adaptive merging or pruning, and language-guided selection.
- Offline MLLMs process pre-loaded videos, whereas streaming systems must handle sequential frames and produce real-time, temporally coherent responses.
- Streaming applications require efficient historical-information management while focusing on present context for timely and accurate reactions.
- Visual token reduction addresses repetitive or static regions that add computational and memory overhead while contributing little semantic content.
- Existing token-reduction research includes long-term memory compression, adaptive spatial-temporal merging or pruning, and language-guided visual selection.
3. Method
FluxMem is a training-free, causal memory framework that progressively compresses streaming visual tokens across short-, mid-, and long-term memory. TAS selects temporally changing tokens, SDC consolidates spatially redundant regions, and distribution-adaptive thresholds adjust compression to scene dynamics.
- Hierarchical Memory: FluxMem processes encoded frame tokens through cascaded short-, mid-, and long-term memories with capacities that trigger progressive compression.Tokens first enter short-term memory, then move through TAS into mid-term memory and through SDC into long-term memory as capacities overflow.
- Temporal Adjacency Selection: TAS retains tokens showing significant semantic changes between adjacent frames using local 3×3 comparisons and a union rule over temporal novelty.The procedure is single-pass, strictly causal, robust to local motion or camera jitter, and O(HW) per overflow event.
- Spatial Domain Consolidation: SDC builds a sparse graph over TAS-retained tokens, links locally similar neighbors, and replaces each connected component with a mean anchor.This removes spatial redundancy while preserving necessary information, with union-find operating in near-linear time.
- Proactive Response Triggering: FluxMem reuses TAS similarity statistics as a zero-cost trigger for proactive responses when incoming frames indicate scene changes.The same statistics are computed when tokens enter short-term memory, avoiding an additional similarity computation for activation triggering.
- Adaptive Thresholding: Both TAS and SDC use Otsu-derived thresholds computed from runtime similarity distributions rather than fixed manually tuned values.The adaptive policy partitions scores into groups for keeping and dropping, allowing retention strength to respond to scene dynamics.
4. Experiments
Experiments show that FluxMem improves online and offline video understanding while substantially reducing visual-token, latency, and memory costs. Ablations indicate that hierarchical memory, adaptive thresholds, and the TAS–SDC design jointly support this accuracy–efficiency trade-off.
- Online Video Understanding: 67.2 real-time and 53.3 overall on OVO-Bench improve from 63.3 and 49.8, respectively.Gains include +6.5 in Prospective Reasoning and +3.3 in Spatial Understanding on StreamingBench, plus +6.4 in Action Recognition and +5.9 in Object Recognition on OVO-Bench.
- Offline Video Understanding: 65.3 on VideoMME, 73.1 on MLVU, and 61.1 on LongVideoBench exceed their corresponding baselines despite using significantly fewer tokens.
- Efficiency: 69.9% lower latency and 34.5% lower memory on OVO-Bench accompany a +3.5 accuracy improvement.On MLVU, latency and memory decrease by 44.3% and 31.2%, with +5.2 accuracy; online updates add 4.1 ms per frame.
- Ablation Studies: 73.1 accuracy with 65.6% fewer visual tokens is achieved by combining mid- and long-term memory on MLVU.The full short-, mid-, and long-term hierarchy reaches 71.6 overall accuracy with 64.3% token reduction, while S+L reaches 77.0 on StreamingBench.
- Ablation Studies: 73.1 accuracy at a 64% drop ratio and 70.1 at an 85% drop ratio show FluxMem outperforming competing reduction strategies in the practical 50–70% range.
- Ablation Studies: 65.6 accuracy at a 42.8% drop ratio for mid-term memory and 65.4 at an 84.5% drop ratio for long-term memory demonstrate adaptive-threshold compression.The best fixed thresholds achieve 65.5 at 29.4% and 65.6 at 63.8% drop ratios, respectively.
5. Conclusion
FluxMem is a training-free, plug-and-play framework for efficient long-stream video processing. Its hierarchical memory and TAS and SDC components reduce spatiotemporal redundancy while preserving causality, and experiments show strong performance across online and offline benchmarks.
- FluxMem combines hierarchical memory with Temporal Adjacency Selection and Spatial Domain Consolidation to mitigate spatiotemporal redundancy while respecting causality.
- Experiments show FluxMem consistently outperforming existing training-free methods and surpassing training-based methods across online and offline benchmarks.