Source-linked AI summary
StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding
Xichen Zhang, Guankai Li, Yinghao Zhu, Shijian Wang, Sitong Wu, Shaozuo Yu, Meng Chu, Yuan Lu, Jiaya Jia
TL;DR
Existing streaming-video evaluations underspecify hour-scale, interactive understanding by relying on isolated capabilities, short clips, and multiple-choice questions. StreamArena addresses this gap, while StreamMind uses decoupled frontend interaction and asynchronous multimodal memory; it ranks first across four capabilities, improves over corresponding baselines by 53.7% to 228.1%, and reduces pooled response latency by 66.2%.
Problem
Existing evaluations often assess streaming capabilities in isolation using short clips and multiple-choice questions, limiting evidence for sustained hour-scale understanding.
Method
StreamMind decouples latency-sensitive frontend interaction from backend persistent multimodal memory, retrieval, historical recall, and external search.
Results
StreamMind ranks first among streaming systems across all four capabilities, improving over corresponding baselines by 53.7% to 228.1% and reducing pooled response latency by 66.2% under a shared backbone.
Takeaways & Limitations
StreamArena exposes the tension between responsive interaction and persistent multimodal understanding, while StreamMind improves all four capabilities over existing streaming baselines.
Takeaways & Limitations
Because StreamMind uses a larger backbone than several streaming baselines, reported gains reflect both model capacity and system design.
Abstract
from arXiv · showhide
Deploying autonomous multimodal agents in continuous, real-world environments requires them to ingest unbounded audio-visual streams and maintain hour-scale memory. However, current evaluations predominantly rely on brief clips and multiple-choice formats. This design allows minimal baselines that process only the last four frames to match or surpass complex streaming models, while answer options also expose language shortcuts. We introduce StreamArena, a benchmark for hour-scale, interactive streaming video understanding. StreamArena contains 243 full-length videos averaging 88.8 minutes and 3,646 rigorously annotated, open-ended question-answer pairs that evaluate real-time perception, historical retrospection, proactive interaction, and multimodal tool utilization. Evaluation across diverse systems exposes a tension between continuous interaction and long-horizon multimodal comprehension. Methods that retain only recent frames cannot recover distant events, methods that convert past observations into text lose visual evidence, and methods that repeatedly compress visual memory struggle to preserve fine-grained details over time. We address this tension with StreamMind, a two-tier architecture that assigns latency-critical interaction and proactive monitoring to independently scheduled frontend workers, while backend workers asynchronously construct persistent multimodal memory and perform historical recall and external search. StreamMind outperforms existing streaming baselines across all four capabilities and reduces query-to-answer latency by reusing persistent state.
1 Introduction
StreamArena addresses gaps in streaming-video evaluation with 243 full-length videos and 3,646 open-ended tasks spanning four capabilities. StreamMind uses a two-tier worker architecture to decouple interaction from persistent multimodal memory, ranking first among streaming systems while reducing latency.
- StreamArena: 243 videos average 88.8 minutes, exposing models to hour-scale temporal horizons rather than isolated clips.The videos are full-length and designed to test genuinely long temporal dependencies.
- Limitations of existing systems: Existing systems expose capability trade-offs: recent-frame methods respond promptly but lose distant events, while text summaries discard visual evidence.AURA’s historical accuracy falls from 25.4% within five minutes to 10.5% beyond thirty minutes, while VST achieves 21.2% on retrospection.
- StreamMind: StreamMind decouples interaction from memory and retrieval through independently scheduled, function-specific frontend and backend workers.Frontend workers handle dispatch and monitoring, while backend workers asynchronously write hierarchical multimodal memory and perform evidence-driven recall and search.
- StreamArena: StreamArena jointly evaluates real-time perception, historical retrospection, proactive interaction, and multimodal tool utilization over continuous audio-visual streams.The benchmark uses causal queries and evidence timestamps with open-ended generation.
- StreamMind: 53.7% to 228.1% improvement over the strongest corresponding baselines and 66.2% lower pooled response latency establish StreamMind as the leading streaming system.The latency reduction is reported under a shared backbone.
2 Methodology
This section presents StreamArena, an hour-scale causal benchmark with rigorously curated streaming-video data, and StreamMind, a two-tier architecture that separates responsive interaction from persistent long-horizon memory. It also defines open-ended accuracy, tool-enabled performance, latency, and proactive triggering metrics for evaluation.
- StreamArena Dataset: Videos are at least 60 minutes long, recorded at 1080p or higher, and contain English or Chinese audio, while sensitive, inappropriate, and political content is excluded.Thirty PhD-level annotators draft approximately 20 multi-turn question-answer pairs per video; two validators correct drafts and a third annotator audits them.
- StreamArena Dataset: StreamArena comprises 243 videos and 3,646 question-answer pairs spanning seven domains, with videos averaging 88.8 minutes and lasting 60–134.2 minutes.The dataset includes 263 perception, 877 retrospection, 1,732 tool-use, and 774 proactive tasks.
- StreamArena Dataset: Historical-retrospection questions have a median evidence-to-query gap of 12.1 minutes, spanning an interquartile range of 5.7–25.7 minutes.The passage also reports 49 questions whose nearest annotated evidence lies more than one…
- Evaluation Metrics: Gemini 3.1 Pro judges open-ended responses with a strict binary decision based on whether each answer contains the factual core of its ground truth.Accuracy is the primary objective for explicitly triggered tasks, while tool-utilization evaluation measures end-to-end answers rather than answer correctness alone.
- Evaluation Metrics: Proactive triggering accuracy accepts event detections occurring from −0.5 s to 2.0 s relative to the target time, independent of subsequent text-generation speed.Reactive latency includes query-triggered routing, recall, external search, and inference, excluding stream processing completed before the query.
- StreamMind Architecture: StreamMind uses a two-tier architecture whose frontend handles interaction and monitoring while its backend maintains persistent memory and performs retrieval-intensive reasoning.The frontend can respond immediately, request backend retrieval for historical or external evidence, or create independent Monitor Workers for future conditions.
3 Experiments
Experiments compare five architecture groups across StreamArena’s four capabilities and show that StreamMind leads streaming systems while improving tool use, proactive interaction, and response latency. Diagnostics further demonstrate that multimodal inputs are essential for perception, with visual evidence and ASR providing complementary information.
- Evaluation setup: Five architecture groups are evaluated: offline turn-based MLLMs, recent-window methods, text-summary VST, internal-compression methods, and StreamMind.The evaluated systems include named representatives such as Qwen3.5-397B-A17B, AURA, MiniCPM-o-4.5, StreamForest, and ThinkStream.
- Overall capability results: StreamMind ranks first among streaming systems across all four target capabilities, while alternative memory designs degrade as evidence becomes more distant.Human reference accuracy reaches 91.8% on RTP, 95.2% on Tool, and 91.5% on Proactive; historical retrospection falls from 80.7% with rewatching to 63.4% without it.
- Tool use and proactive interaction: 228.1% relative improvement over the strongest streaming baseline on Tool reflects explicit coordination among Router, Recall, and Search Workers.Tool questions require visual grounding, identifying information gaps, retrieval, and integrating external evidence; proactive interaction must detect target events without a new prompt.
- Latency: 66.2% relative latency reduction, from 81.4 to 27.5 seconds, is achieved by StreamMind while retaining 89.7% of pooled accuracy.Using Qwen3.5-397B-A17B as the shared backbone, latency decreases by 84.6% on RTP, 73.9% on HR, and 55.3% on Tool.
- Input diagnostics: 32.4% accuracy from visual input plus ASR exceeds visual-only input by 5.6 percentage points, whereas ASR-only input reaches 4.2%.The diagnostic subset contains 616 questions, and the result indicates that visual evidence is essential while speech contributes complementary information.
4 Discussion and Future Work
The discussion argues that long-horizon memory needs future-aware retention rather than simply greater capacity. StreamMind’s HR drops 63.4% from L2 to L4, motivating adaptive multimodal evidence preservation guided by retrieval outcomes.
- Long-horizon memory requires future-aware retention: StreamMind’s HR decreases 63.4% from L2 to L4, indicating that long-horizon memory is not merely a capacity problem.The decline occurs across temporal strata despite StreamMind’s overall HR improvement.
- Long-horizon memory requires future-aware retention: Because future queries are unknown during ingestion, future systems should learn from retrieval outcomes to estimate evidence utility.This learning is intended to guide what information should be retained before queries are available.
- Long-horizon memory requires future-aware retention: Future work should preserve uncertain multimodal details at adaptive fidelity to support long-horizon retrieval.The proposal targets retention that adjusts fidelity according to uncertainty rather than applying uniform compression.
5 Conclusion · A Related work · B StreamArena details
StreamArena is introduced as an hour-scale benchmark for causal, interactive streaming video understanding, while related work has mainly addressed longer-video comprehension through post hoc memory organization and retrieval. The benchmark targets real-time perception, historical retrospection, multimodal tool use, and proactive interaction while exposing shortcuts in conventional evaluations.
- 5 Conclusion: StreamArena jointly evaluates real-time multimodal perception, historical retrospection, multimodal tool utilization, and proactive interaction under causal access.Its protocol is designed for continuous streaming rather than offline clip processing.
- 5 Conclusion: Its open-ended questions, timestamped evidence, and continuous protocol expose shortcuts hidden by short-clip or multiple-choice evaluations.
- 5 Conclusion: Experiments identify a central tension between responsive interaction and persistent multimodal understanding.
- A Related work: Traditional video language models generally perform offline, turn-based question answering over pre-segmented clips.
- A Related work: MemDreamer, M3-Agent, EgoRAG, and EGAgent extend comprehension to longer recordings using hierarchical summaries, episodic or semantic memories, and entity-centric graphs.
- A Related work: These related methods primarily study memory representation and retrieval after video processing.
B.1 Data collection and annotation pipeline
StreamArena’s data collection uses a three-stage pipeline involving 30 PhD-level annotators, who source and screen long, high-resolution YouTube videos before generating timestamp-grounded question-answer pairs.
- Video sourcing and filtering: 30 PhD-level annotators collect YouTube videos spanning seven domains, each lasting at least 60 minutes and having at least 1080p resolution.Screening excludes sensitive, inappropriate, and political content.
- Initial generation: Annotators draft approximately 20 question-answer pairs per video during initial generation.Each task includes a question, query timestamp, reference answer, and timestamped audio-visual evidence.
- Annotation pipeline: The pipeline is organized into three annotation stages.
B.2 Task taxonomy
StreamArena organizes questions into four capabilities that test short-window audio-visual grounding, long-horizon recall, external tool use, and autonomous monitoring. Each capability specifies its evidence or interaction protocol and scoring rule, with HR additionally covering distinct multi-segment reasoning patterns.
- Real-time multimodal perception (RTP): Real-time multimodal perception (RTP) tests audio-visual grounding from a short window around the query time.Questions target on-screen objects, actions, ambient sounds, or spoken utterances, and scoring uses Eq. (1).
- Historical retrospection (HR): Historical retrospection (HR) requires recalling factual information from timestamped earlier evidence, with evidence-to-query gaps exceeding one hour.HR stresses persistent observation storage, content indexing, and open-ended retrieval; questions may use single-point, counting, recall, comparison, or temporal-range reasoning.
- Multimodal tool utilization (Tool): Multimodal tool utilization (Tool) evaluates end-to-end answers requiring external image and text search for entities, facts, or attributes absent from the video.The canonical workflow grounds a visual referent, uses image search to obtain a textual handle, then issues a text query; accuracy requires tool-enabled end-to-end performance.
- Proactive interaction (Pro): Proactive interaction (Pro) requires continuously monitoring the stream after a user instruction and autonomously alerting when the target event occurs, without another prompt.Immediate acknowledgment is excluded, and Proactive-Acc in Eq. (2) gives no credit to agents that reason only after explicit prompting.
B.3 Complete dataset statistics
StreamArena comprises 243 hour-scale videos with predominantly Mandarin audio-visual content and includes substantial historical-retrospection reasoning complexity. Its videos average 88.8 minutes, while 877 historical-retrospection questions span multiple reasoning patterns and evidence requirements.
- Video-level metadata: 243 videos each last at least 60 minutes, averaging 88.8 minutes and reaching a maximum of 134.2 minutes.The dataset includes 17 live streams and 226 non-live long-form recordings.
- Video-level metadata: 189 videos carry Chinese audio, 25 English, 26 mixed Chinese-English, and 3 another language.Mandarin dominates the audio-visual content.
- Historical-retrospection evidence complexity: 877 historical-retrospection questions are annotated with answer-supplying video segments and reasoning patterns.The annotations support complementary analyses of reasoning-pattern distributions and distinct evidence-segment counts.
C Detailed experimental setups
The appendix documents the evaluation protocol and implementation details for every system, supporting reproducible cross-paradigm comparison. It separates the evaluation harness from model design and organizes configurations by system group, StreamMind, and infrastructure.
- Evaluation protocol: Table 3 documents the evaluation protocol for every system to support reproducibility and fair cross-paradigm comparison.The appendix explicitly frames this documentation as an evaluation-protocol resource.
- Evaluation protocol: Appendix C.1 distinguishes the evaluation harness from model design, while C.2–C.4 detail Groups (A)–(D).The group-specific setups are distributed across Appendices C.2, C.3, and C.4.
- Implementation details: Appendix C.5 specifies StreamMind’s full configuration, and C.6 records computing infrastructure and reproducibility notes.These sections complete the implementation and reproducibility documentation.
C.1 Evaluation harness and model design · C.2 Offline turn-based MLLMs
The evaluation separates video delivery from temporal attention, distinguishing true continuous ingestion from query-triggered replay while preserving each model’s intended inference mechanism. Offline turn-based MLLMs follow a standardized per-question protocol for visual, audio/subtitle, sampling, generation, and tool inputs.
- C.1 Evaluation harness and model design: StreamArena distinguishes query-triggered replay, which reconstructs causal inputs on demand, from continuous ingestion, which consumes sequential frames once and preserves information in internal state.Replay can revisit designated intervals independently for each question, whereas continuous ingestion must retain useful information before future queries are known.
- C.1 Evaluation harness and model design: The model dimension independently classifies visual processing as non-causal or causal, with causal attention enforcing a prefix-only mask.This dimension is separate from how the harness delivers video.
- C.1 Evaluation harness and model design: StreamMind uses continuous ingestion and satisfies both evaluation recurrences, while every baseline reconstructs a causal prefix or bounded recent window without hidden video state across questions.Table 10 classifies systems by harness, model, and design rather than conflating these categories.
- C.1 Evaluation harness and model design: The harness adapts each released interface with its prescribed causal interval, sampling policy, and memory mechanism while preventing reactive queries from receiving frames after tq_i.Multi-turn textual history is preserved within each video, enabling a common causal, open-ended comparison without forcing a shared continuous-ingestion implementation.
- C.2 Offline turn-based MLLMs: Offline turn-based MLLMs uniformly sample frames from S0:tq_i, resize them so no side exceeds 1280 × 720, and apply a short-window cap to avoid near-duplicates.The short-window cap applies when tq_i < Nmax seconds.
- C.2 Offline turn-based MLLMs: Omnicapable models receive raw 16 kHz mono audio prefixes, while other models use native YouTube captions by priority language with Qwen3-ASR-1.7B as a .vtt fallback.The omnicapable set is MiMo-V2.5, Qwen3.5-Omni, and Gemini 3.5 Flash; the caption-language priority list is zh-CN, zh-Hans, en, zh-Hant.
- C.2 Offline turn-based MLLMs: All offline MLLMs use T = 0.6, top-p = 0.95, and K = 65,536 tokens; Gemini additionally uses high reasoning effort and high media resolution.Tool-utilization tasks share a common loop allowing at most Rmax = 5 rounds and supporting text and image search.
C.3 Recent-window and text-summary methods … C.6 Computing infrastructure
Sections C.3–C.6 specify how recent-window, text-summary, internal-compression, and StreamMind systems process causal video, interact proactively, retrieve information, and persist multimodal memory. They also document StreamMind’s runtime configuration and the shared computing infrastructure used for evaluation.
- C.3 Recent-window and text-summary methods: AURA reconstructs at most the latest 30 one-second chunks per query, while proactive checks independently rebuild that recent window until two seconds after the event.Reactive queries use available captions through the query time; responses before the event are premature, while the event time and next two sampled seconds are candidate alerts.
- C.3 Recent-window and text-summary methods: VST samples each causal prefix at 2 fps, reduces it to at most 384 frames, and converts earlier temporal segments into textual memory before a final visual pass.Its visual budget is equivalent to 8,192 tokens per segment; intermediate and final generations are capped at 5,000 and 32,768 tokens, respectively, with state discarded after each query.
- C.5 StreamMind (ours): StreamMind continuously ingests video at 2 fps without rewinding, pauses only non-proactive queries, and combines query-driven, monitoring, routing, recall, search, and memory-writing workers.Workers share a frame buffer, bounded conversation log, and persistent memory bank containing a typed entity-relation graph and key frames.
- C.5 StreamMind (ours): The Monitor Worker checks 4 recent frames every 2 seconds, fires at confidence 0.6, and supports up to 8 active monitors with persistent visual and textual trigger records.One-time monitors terminate after firing; repeating monitors enforce a 3-second minimum trigger gap and retain 60 visual records plus 200 textual summaries.
- C.5 StreamMind (ours): StreamMind’s Router executes up to 4 ReAct rounds, while Search and Recall workers perform bounded concurrent external retrieval and historical memory operations.Recall uses up to 4 rounds and can search time windows, entities, micro-events, relations, and associated frames; a final visual pass checks up to 8 recalled frames.
- C.5 StreamMind (ours): StreamMind’s Memory Writer runs every 4 seconds, reads 30 frames and 15 seconds of ASR, and organizes records into micro-, macro-, and super-events.The launch configuration uses a shared Qwen3.5-397B-A17B vLLM endpoint, with non-thinking Front Worker inference and thinking backend workers.
C.7 Reproducibility artifacts
The release provides structured StreamArena annotations, licensing and availability safeguards, and the evaluation and StreamMind code needed to reproduce reported results.
- Data release: StreamArena identifies 243 videos by YouTube ID and stores each question as a structured JSON record.Records include domain categories, capability labels, questions, query timestamps, answers, and evidence annotations.
- Data release: Evidence records specify temporal spans for real-time perception, historical retrospection, and proactive interaction annotations.RTP uses a single short window, HR uses one or more segments before the query, and Pro records the annotated trigger time.
- Data release: HR annotations encode reasoning patterns as single_point, multi_count, multi_recall, multi_compare, or temporal_range.The reasoning_pattern field applies only to historical retrospection questions.
- Data license and fallback for expired videos: Annotations are released under CC BY 4.0, with documented archived-copy substitutions for unreachable YouTube videos and no redistribution of original video content.The availability index is periodically refreshed, and release notes list every substitution.
- Code release: The Apache-2.0 code release includes the evaluation harness, Table 3 baseline adapters, and the complete StreamMind worker implementation.Released workers include Front, Monitor, Router, Search, Recall, and Memory Writer; each module also maps its implementation role to documented hyperparameters.