Source-linked AI summary
A Simple Baseline for Streaming Video Understanding
Yujiao Shen, Shulin Tian, Jingkang Yang, Ziwei Liu
TL;DR
Streaming video understanding has increasingly adopted complex mechanisms for managing long-range history, but it remains unclear whether that complexity is necessary for strong performance. The paper introduces SimpleStream, which feeds only recent frames to an off-the-shelf VLM, and finds strong benchmark performance alongside a backbone-dependent context trade-off. It argues that added memory mechanisms should be evaluated against strong recency baselines with perception and memory reported separately.
Problem
Complex memory mechanisms increasingly manage long streaming-video histories, but whether they are necessary for strong streaming understanding remains underexplored.
Method
SimpleStream answers each query by feeding only the last N frames from the causal prefix to an off-the-shelf VLM, without additional memory, retrieval, compression, or training.
Results
SimpleStream achieves state-of-the-art performance on OVO-Bench and StreamingBench while maintaining the lowest peak GPU memory and competitive latency among compared streaming methods.
Takeaways & Limitations
Added memory, retrieval, or compression should demonstrate clear gains over SimpleStream under matched protocols, with perception- and memory-oriented capabilities evaluated separately.
Takeaways & Limitations
The paper is a strong-baseline study rather than a solution to long-horizon video understanding, and does not introduce a new long-term memory, retrieval, or compression architecture.
Abstract
from arXiv · showhide
Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.
1 Introduction
The paper challenges the assumption that strong streaming video understanding requires complex memory mechanisms. It introduces SimpleStream, a recent-frame baseline that performs strongly while exposing a perception–memory trade-off and motivating stricter evaluation.
- Complex memory-centric streaming designs often deliver modest or uneven gains, motivating a test of whether such complexity is necessary.
- SimpleStream feeds each query and only the last N observed frames directly to an off-the-shelf VLM, without additional memory, retrieval, compression, or training.
- SimpleStream achieves state-of-the-art performance on OVO-Bench and StreamingBench, with the lowest peak GPU memory and competitive latency among compared streaming methods.
- Controlled analyses show that longer context is not uniformly beneficial: its utility depends on backbone and model scale, while memory gains often reduce present-scene perception.
- Future evaluations should compare added complexity against strong recency baselines using separate perception- and memory-oriented metrics.
2 Related Work
Related streaming video understanding work spans proactive interaction, streaming-oriented training, and memory-centric context management. Benchmarks evaluate causal reasoning under observed-only constraints while requiring both current-scene perception and prior-context use.
- Streaming video understanding research covers proactive response, streaming-oriented training, and memory-centric context management.
- SimpleStream represents a contrasting approach by retaining only a small recent frame window rather than expanding context through historical-information mechanisms.
- Proactive systems address response timing and interaction policy through answer-readiness prediction, perception–decision separation, or external response triggers.
- Memory methods differ in how explicitly they model history, including compression or pruning of tokens and KV states, retrieval, and adaptive KV selection.
- Streaming benchmarks evaluate causal online reasoning under observed-only constraints while combining current-scene perception with prior-context use.
3 From Complex Streaming Methods to SimpleStream
SIMPLESTREAM tests whether strong streaming performance requires explicit long-range memory by feeding an off-the-shelf VLM only a bounded window of recent frames and the current query. It omits memory, retrieval, compression, architectural changes, and additional training, keeping inference costs bounded by the window size.
- SimpleStream baseline: SIMPLESTREAM answers each query with the last N observed frames and the query text sent directly to the base VLM.It preserves only a short recent window of visual evidence.
- SimpleStream baseline: Unlike prior systems, SIMPLESTREAM introduces no additional mechanism for managing long-range history.Frames outside the sliding window are discarded.
- Design principles: SIMPLESTREAM is an inference-time input policy without architectural modification, memory modules, or additional training.The baseline isolates performance obtained from recent visual context while minimizing confounding system changes.
- Efficiency rationale: Per-query memory and computation remain bounded by N because the retained frame window does not grow with stream length.This fixed-window design contrasts with systems that maintain accumulating historical state.
4 Experiments
The experiments evaluate SIMPLESTREAM against offline and streaming video LLMs on OVO-Bench and StreamingBench, then analyze model scale, window size, efficiency, and Visual-RAG. SIMPLESTREAM performs strongly with recent frames while using favorable latency and memory, but longer context is not uniformly beneficial.
- Experimental setup: SIMPLESTREAM is evaluated on OVO-Bench and StreamingBench alongside six offline and seven representative streaming video LLMs.The unified comparison uses reported inference settings and frame budgets from the original methods where applicable.
- Benchmark performance: 67.7% average accuracy on OVO-Bench exceeds HERMES by 8.5 percentage points, while 80.59% on StreamingBench surpasses HERMES at 79.44%.The OVO-Bench result is achieved by Qwen3-VL with 4 frames.
- Efficiency: SIMPLESTREAM-4f maintains the lowest and flattest peak GPU-memory curve as observed frames increase.Its fixed recent-frame window prevents memory usage from accumulating with stream length.
- Efficiency: SIMPLESTREAM-4f remains latency-competitive, achieving the second-lowest TTFT at 16, 64, and 256 observed frames.HERMES is the only method reported as consistently faster.
- Window-size ablation: 4 recent frames produce the highest Real-Time accuracy in the controlled window ablation, while overall accuracy does not improve monotonically with wider windows.The ablation varies only the number of consecutive recent frames under otherwise fixed settings.
- Visual-RAG analysis: Visual-RAG appends five retrieved historical chunks to a matched recent-window baseline to test whether targeted history changes the context finding.The analysis compares matched base and +V-RAG conditions within Table 4.
- Model scale effects: The preferred window size varies across model scales and backbone families rather than increasing uniformly with parameter count.Larger windows can help some higher-capacity checkpoints, while smaller and mid-sized models often plateau or decline.
5 Analysis
The analysis tests whether more visual history consistently helps and finds that its value depends on the backbone, while memory gains commonly trade against present-scene perception.
- Ablation design: Longer context is evaluated through recent-window, model-scaling, and Visual-RAG probes, testing contiguous history, backbone capacity, and retrieved distant frames.The probes hold other factors fixed where applicable and examine whether additional history improves streaming answers.
- Recency-window ablation: 66.4 → 67.7 Overall accuracy and 79.3 → 81.4 Real-Time accuracy result from expanding the recent window from 2 to 4 frames.The controlled recency ablation varies N ∈{2, 4, 8, 16} while keeping retrieval, prompting, and decoding fixed.
- Model scaling: The optimal window is non-monotonic and checkpoint-dependent: Qwen2.5-VL-72B prefers 16 frames, while Qwen2.5-VL-32B peaks at 4 frames.Qwen3-VL-32B prefers 8 frames, whereas Qwen3-VL-30B-A3B peaks at 4 frames; Figure 5 also shows several checkpoints preferring 8f or 16f.
- Visual-RAG ablation: Visual-RAG also fails to produce uniform gains, indicating that selectively retrieving distant history does not reliably improve performance.The broader pattern is that gains saturate quickly and longer context helps only when the backbone can use it.
- Perception-memory trade-off: Every evaluated external baseline falls below SIMPLESTREAM on ∆P, while memory gains become common when measured using EPM and ASI.StreamForest reaches ∆M = +8.9 with ∆P = −13.8, and HERMES reaches ∆M = +2.4 with ∆P = −6.0.
- Perception-memory trade-off: Visual-RAG improves EPM and ASI by 6.6 points on average but reduces real-time perception by 4.9 points relative to the matched recent window.The metrics define memory using the mean of EPM and ASI, while ∆P measures change in OVO-Bench Real-Time accuracy.
6 Why Does a Simple Baseline Win?
SIMPLESTREAM wins because it preserves clear recent visual evidence, which current VLM backbones use effectively, while added history can dilute present-scene perception and benchmarks overweight perception.
- Recent context matters most: Current VLMs are strong at short-horizon perception, including text reading, object recognition, local action tracking, and query-conditioned questions.SIMPLESTREAM protects this capability by keeping recent visual evidence dense and uncompressed.
- Complex memory can hurt present perception: Additional memory can interfere with current-scene understanding through compression, retrieval noise, abstract latent states, or large memory injection.Attention dilution is offered as a plausible hypothesis rather than an established empirical mechanism.
- Perception-memory trade-off: Figure 6 shows many methods improving memory while external baselines incur substantial perception cost relative to SIMPLESTREAM Qwen2.5-VL + 2f.Green bars represent changes in OVO-Bench Real-Time average, and blue bars represent changes in the mean of EPM and ASI.
- Implications for future research: Future models should preserve recent context by default and access history on demand, while benchmarks should separate perception, memory recall, and related capabilities.The paper recommends evaluating recall gains together with any damage to real-time perception.
7 Conclusion
The conclusion argues that SIMPLESTREAM is a strong reference point for streaming systems and that progress claims should be tested against it using disaggregated capability and efficiency measures.
- Conclusion: SIMPLESTREAM exceeds recently published complex-memory streaming systems on OVO-Bench and StreamingBench while remaining latency-competitive.The conclusion positions this result as evidence that additional memory, retrieval, or compression requires clear gains over the simple baseline.
- Conclusion: Additional memory, retrieval, or compression should be justified by clear gains over SIMPLESTREAM on relevant capability slices.The recommended reporting includes strong simple baselines, disaggregated perception-versus-memory metrics, and transparent efficiency statistics.
- Conclusion: The optimal recent window does not grow monotonically with model scale, and memory injection often improves recall-oriented cases at the cost of current-scene perception.These findings reinforce that more history is not always better and that aggregate benchmark scores may not faithfully reward long-term memory.
8 Limitations
The paper’s conclusions are scoped by its use of strong Qwen backbones and by its deliberate role as a baseline study rather than a new streaming architecture.
- Backbone scope: SIMPLESTREAM is evaluated specifically with Qwen2.5-VL and Qwen3-VL, so its competitiveness may not transfer automatically to broader backbone families.The paper calls for comparisons across models with different pretraining data, visual encoders, and temporal reasoning characteristics.
- Study scope: The study establishes a strong baseline and analyzes trade-offs, but does not introduce a new memory-centric architecture or long-term memory mechanism.Its scope is intentionally a strong-baseline paper rather than an architectural proposal.