Source-linked AI summary
StreamForest: Efficient Online Video Understanding with Persistent Event Memory
Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, Yi Wang, Limin Wang
TL;DR
Streaming video MLLMs remain limited by historical-feature storage constraints and insufficient real-time spatiotemporal reasoning. StreamForest combines persistent event memory, fine-grained local temporal perception, and streaming-specific instruction tuning, and reports strong online and offline benchmark performance with resilience to extreme token compression. The paper also identifies imprecise semantic event-boundary detection as a limitation.
Problem
Streaming video MLLMs face limited historical visual-feature storage and insufficient real-time spatiotemporal reasoning.
Method
StreamForest combines a Persistent Event Memory Forest, a Fine-grained Spatiotemporal Window, and the OnlineIT instruction-tuning dataset for streaming video understanding.
Results
StreamForest reports 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench, retaining 96.8% of average accuracy across eight benchmarks at 1024 tokens.
Takeaways & Limitations
The results support StreamForest’s robustness and efficiency for continuously processing streaming video across online and offline understanding settings.
Takeaways & Limitations
Event-boundary detection based on inter-frame similarity captures coarse scene changes and often misses true semantic event boundaries.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in video understanding. However, their effectiveness in real-time streaming scenarios remains limited due to storage constraints of historical visual features and insufficient real-time spatiotemporal reasoning. To address these challenges, we propose StreamForest, a novel architecture specifically designed for streaming video understanding. Central to StreamForest is the Persistent Event Memory Forest, a memory mechanism that adaptively organizes video frames into multiple event-level tree structures. This process is guided by penalty functions based on temporal distance, content similarity, and merge frequency, enabling efficient long-term memory retention under limited computational resources. To enhance real-time perception, we introduce a Fine-grained Spatiotemporal Window, which captures detailed short-term visual cues to improve current scene perception. Additionally, we present OnlineIT, an instruction-tuning dataset tailored for streaming video tasks. OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction. To evaluate generalization in practical applications, we introduce ODV-Bench, a new benchmark focused on real-time streaming video understanding in autonomous driving scenarios. Experimental results demonstrate that StreamForest achieves the state-of-the-art performance, with accuracies of 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench. In particular, even under extreme visual token compression (limited to 1024 tokens), the model retains 96.8% of its average accuracy in eight benchmarks relative to the default setting. These results underscore the robustness, efficiency, and generalizability of StreamForest for streaming video understanding.
1 Introduction
StreamForest addresses streaming video understanding by combining persistent long-term visual memory with fine-grained real-time perception. Its reported results show strong performance across online and offline benchmarks, including under severe visual-token compression.
- Motivation: Streaming video understanding must retain historical frame information while supporting fine-grained real-time spatiotemporal reasoning.Prior compression during sampling limits detailed current-scene reasoning, while storage compression reduces historical visual information.
- Approach: StreamForest organizes long-term visual information into an event-based tree memory using temporal-distance, content-similarity, and merge-frequency penalties.The architecture processes ultra-long streaming video at a constant rate of 1 fps while adaptively merging segments.
- Approach: The Fine-grained Spatiotemporal Window captures local temporal context, while OnlineIT improves present-moment perception and future-event anticipation.OnlineIT uses recent observations and long-term historical cues for streaming video instruction tuning.
- Results: 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench are reported for StreamForest under the default 8192-token setting.The model significantly outperforms previous state-of-the-art streaming video understanding MLLMs.
- Results: 96.8% of average accuracy across eight benchmarks is retained with a 1024-token limit relative to the default setting.The result demonstrates resilience under extreme visual-token compression.
2 Related work
Related work develops video MLLMs for long-range modeling and efficient long-video processing, while streaming-video research increasingly targets online real-time interaction. Existing streaming approaches largely focus on dense video captioning and semantic summarization.
- Multimodal Large Language Model: Video MLLMs extend multimodal modeling to dynamic sequences, requiring long-range dependency modeling and event understanding.Recent approaches introduce strategies for efficiently processing and reasoning over long video inputs.
- Streaming Video Understanding: Streaming video understanding has gained attention as users expect online processing and real-time interaction from MLLMs.The task has been explored by multiple recent works across streaming-video settings.
- Streaming Video Understanding: Most existing streaming approaches focus on dense video captioning and semantic summarization rather than broader streaming-video tasks.This related-work passage identifies the dominant focus of prior approaches without specifying their performance.
3 Methodology
StreamForest combines fine-grained short-term perception with a persistent event-level memory that adaptively compresses long-term streaming video information. OnlineIT further supports streaming-task training through expanded, diverse instruction data.
- Fine-grained Spatiotemporal Window: Fine-grained Spatiotemporal Window retains second-level, high-resolution features near the query to support real-time spatiotemporal perception.It combines direct current-frame perception with short-term spatiotemporal memory as older frames arrive.
- Persistent Event Memory Forest: Persistent Event Memory Forest organizes long-term video segments into event-level tree structures while processing streaming video at a constant rate of 1 fps.It adaptively stores and manages historical visual information based on event boundaries.
- Persistent Event Memory Forest: Similarity, merge-count, and temporal-distance penalties jointly determine which event-node pairs receive priority for adaptive merging.Similarity favors merging visually similar events, merge-count regularizes repeated participation, and temporal distance preserves recent information with higher fidelity.
- Persistent Event Memory Forest: Adjusting penalty weights enables trade-offs among similarity-based compression, uniform downsampling, and FIFO-like retention strategies.The resulting behavior can adapt to different streaming-memory requirements.
- OnlineIT: OnlineIT addresses limited streaming-data volume, quality, and task diversity through refined existing data and two new datasets totaling 32K general streaming instances.OnlineIT-drive adds 89K autonomous-driving streaming QA instances spanning localization, traffic entities, and risk or accident assessment.
4 ODV-Bench
ODV-Bench evaluates online video understanding in autonomous-driving scenarios through task categories spanning static targets, dynamic targets, and multi-agent events. Its construction combines curated driving videos, detailed annotations, automated QA generation, and quality control.
- Benchmark motivation: ODV-Bench targets a gap in existing streaming benchmarks by evaluating dynamic, high-stakes driving scenarios and prediction tasks.Its scope includes rapidly changing scenes, multi-agent interactions, risk assessment, and motion planning.
- Task formulation: The benchmark organizes autonomous-driving evaluation into static-target, dynamic-target, and event-oriented task scenarios.These cover stationary traffic elements, vehicle and pedestrian behavior or trajectories, and multi-agent interactions, risks, or accidents.
- Benchmark construction: ODV-Bench construction uses a four-stage process intended to ensure the quality of generated questions and includes examples across driving scenarios.The pipeline includes video selection, meta-annotation generation, multiple-choice construction, and quality control.
- Evaluation: Table 1 reports that StreamForest outperforms state-of-the-art offline and online video MLLMs in zero-shot testing and improves further after driving-domain fine-tuning.The supplied caption states the comparison and fine-tuning pattern without listing individual table values.
5 Experiments
Experiments evaluate StreamForest across online, offline, ablation, token-budget, runtime, and training-data settings. Results show strong benchmark performance, complementary architectural benefits, robustness under compression, bounded memory use, and gains from OnlineIT.
- Online Benchmark Results: 71.2% on ODV-Bench after OnlineIT-drive training improves over 59.9% without it in autonomous-driving streaming video understanding.ODV-Bench tests fine-grained perception, spatial relations, and future trajectory prediction.
- Online Benchmark Results: 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench demonstrate strong performance across online streaming video QA benchmarks.The evaluation uses ODV-Bench, StreamingBench, OVBench, and OVO-Bench, with models restricted to video content available before the current timestamp.
- Offline Benchmark Results: 61.4% on VideoMME, 70.0% on MLVU, 70.2% on MVBench, and 73.1% on PerceptionTest show strong offline generalization across long and short videos.The method outperforms recent online Video MLLMs and leading offline models on most evaluated benchmarks.
- Ablations: PEMF outperforms alternative memory strategies on long-horizon tasks, including +2.2% over Similarity Merge on OVO-Bench and +2.0% on MLVU.FIFO performs worst on MLVU, while fixed-capacity pyramidal memory also trails PEMF on OVO-Bench and MLVU.
- Ablations: FSTW and PEMF provide complementary benefits: FSTW improves near-query perception, while PEMF preserves efficient long-term memory and event-level representations.Removing both modules causes the largest performance drop, and event-level construction prevents over-merging within events.
- Robustness and Efficiency: Under a 1K visual-token budget, StreamForest reaches up to 99.8% compression on long-video benchmarks while retaining competitive performance.Across token budgets, PEMF consistently exceeds Similarity Merge by 2–3% average accuracy and has a +1.8% relative retention advantage on VideoMME at 1K tokens.
- Robustness and Efficiency: PEMF caps visual memory at 8K tokens, keeping memory usage near 17 GB regardless of processed-frame count.The runtime analysis assumes frame-level visual features are already extracted by the vision encoder in real time.
- Training Data: Combining OnlineIT with existing offline VideoQA datasets significantly improves streaming video understanding performance.OnlineIT targets real-time perception and future prediction while mitigating hallucinations from inconsistent historical and current context.
6 Conclusions
The paper combines StreamForest, OnlineIT, and ODV-Bench to address long-term memory, fine-grained perception, training, and evaluation for streaming video understanding. OnlineIT covers general and autonomous-driving tasks, while ODV-Bench organizes driving scenarios across perception modes and target categories.
- 6 Conclusions: StreamForest manages long-term visual information through adaptive event merging guided by temporal distance, content similarity, and merge frequency.Its Persistent Event Memory Forest organizes video segments into event-level tree structures.
- 6 Conclusions: The Fine-grained Spatiotemporal Window maintains precise understanding of the current scene by preserving short-term visual detail.The conclusion presents it as complementary to persistent historical memory.
- OnlineIT: OnlineIT targets real-time perception, future prediction, and event continuity through OnlineIT-general and OnlineIT-drive.OnlineIT-general contains over 400k instances across spatial, temporal, spatiotemporal, and event perception, while OnlineIT-drive contains 89k autonomous-driving instances.
- OnlineIT-drive: OnlineIT-drive covers static-target memory and perception, dynamic-target location and movement prediction, and event-oriented risk and accident reasoning.These task groups address traffic targets, future trajectories, and contextual traffic risks.
- ODV-Bench: ODV-Bench categorizes autonomous-driving tasks by static targets, dynamic targets, and multitarget interaction events across 12 perception tasks.The benchmark taxonomy and examples are presented as a framework for real-time streaming video understanding.
C.2 Dataset Statistics
ODV-Bench evaluates streaming video understanding using diverse first-person driving clips and time-indexed question-answer pairs. Its videos span routine conditions, hazards, and accidents across countries and varied durations.
- C.2 Dataset Statistics: ODV-Bench comprises 1,190 first-person driving video clips and 6,322 question-answer pairs.The average query timestamp is 18.9 seconds.
- C.2 Dataset Statistics: The benchmark videos range from 5 to 90 seconds and cover driving scenarios across different countries.The collection includes routine driving conditions, potential hazards, and accidents.
D More Implementation Details
StreamForest uses staged training that combines offline long-form video preparation with online fine-tuning for streaming inputs. The evaluation compares the model on three online video understanding benchmarks.
- Training Strategy: The training configuration uses three offline pre-training stages followed by fourth-stage online fine-tuning and fifth-stage driving fine-tuning.The parameter settings are documented in Tables 10 and 11.
- Online Fine-Tuning: Online fine-tuning feeds continuous frame sequences so the model learns current-scene perception while retaining long-term event memory under compression constraints.This phase targets realistic streaming inputs.
- Evaluation: The experiments evaluate StreamForest on StreamingBench, OVBench, and OVO-Bench.These are the three online video understanding benchmarks used for comprehensive comparison.
E.1 StreamingBench
StreamForest achieves leading performance across StreamingBench, OVBench, and OVO-Bench while supporting real-time streaming evaluation. Qualitative examples attribute its strengths to fine-grained detail capture, persistent memory, and spatiotemporal grounding.
- StreamingBench: 77.26% average on StreamingBench is the highest score among evaluated open-source and proprietary models at 1 fps.It exceeds GPT-4o at 73.28%, Gemini 1.5 Pro at 75.69%, and Dispider at 67.63%.
- OVBench: 60.5% average on OVBench surpasses Videochat-Online at 54.9%, Flash-VStream at 31.2%, and Gemini-1.5-Flash at 50.7%.The benchmark covers six task categories.
- OVO-Bench: 55.57% overall on OVO-Bench leads open-source online video MLLMs, exceeding Dispidier-7B at 41.78% and Flash-VStream-7B at 33.15%.Reported strengths include real-time visual perception, backward tracing, and forward active responding.
- Qualitative Comparison: The qualitative comparison highlights persistent memory, fine-grained visual detail capture, and spatiotemporal grounding across examples.The bottom example also illustrates potential in intelligent driving.
G More Ablations
Ablations vary the three penalty weights governing long-term memory, and the results indicate that balanced weighting is most effective.
- Penalty-weight ablations: Figure 8 reports accuracy under different combinations of the three penalty weights.
- Penalty-weight ablations: Balanced penalty weights outperform imbalanced settings across VideoMME and OVBench accuracy.The ablation varies similarity, merge count, and temporal distance penalties.
- Penalty-weight ablations: The best reported weighting assigns 0.4 to similarity, 0.4 to merge count, and 0.2 to temporal distance.
H Efficiency of Multi-round Inference
StreamForest is evaluated for multi-round inference on a 600-second stream at 1 FPS, isolating processing throughput from response-generation latency.
- Throughput evaluation: 9.9 FPS average processing speed is achieved while streaming a 600-second video at a constant 1 FPS.The model produced a single-token response per frame to isolate throughput from text-generation latency.
- Throughput evaluation: 9.9 FPS is competitive with VideoLLM-Online’s 12.3 FPS in the reported comparison.
- Throughput evaluation: Table 15 compares multi-round inference speed across models.
I.1 Limitations
The paper identifies limitations in event-boundary detection and highlights deployment risks related to bias and environmental variation.
- Method limitations: Inter-frame similarity detects coarse scene changes but often misses true semantic event boundaries.The method identifies local minima in similarity scores to decide when outputs should be produced.
- Method limitations: A lightweight auxiliary MLLM is proposed as a possible way to provide more precise, context-aware output guidance.
- Deployment scope: Domain-specific fine-tuning is needed to adapt the method to downstream tasks requiring continuous visual processing.
- Broader impacts: Deployment may amplify training-data biases and produce performance disparities across weather, lighting, or geographic conditions.