Source-linked AI summary
StreamReady: Learning What to Answer and When in Long Streaming Videos
Shehreen Azad, Vibhav Vineet, Yogesh Singh Rawat
TL;DR
Streaming video models must answer when supporting evidence appears, yet existing approaches and benchmarks insufficiently evaluate this timing. The paper introduces ARS, StreamReady, and ProReady-QA, and reports superior readiness-aware performance across streaming and offline long-video benchmarks.
Problem
Existing streaming evaluation emphasizes answer correctness, while proactive scenarios require determining when sufficient future evidence has appeared; current benchmarks lack annotated evidence durations for verifying timing.
Method
The paper combines ARS's asymmetric early and late timing penalties with StreamReady's lightweight readiness mechanism and ProReady-QA's annotated evidence windows and proactive multi-turn tasks.
Results
StreamReady achieves the highest accuracy and ARS across five ProReady-QA tasks, surpassing the best model by ~3% in accuracy and ~9% in ARS on average, while outperforming prior methods across additional streaming and offline benchmarks.
Takeaways & Limitations
Readiness-aware streaming understanding evaluates answer content and timing together, supporting models that answer both accurately and on time.
Takeaways & Limitations
Existing proactive benchmarks lack annotated answer evidence durations, limiting verification of whether responses occur at evidence-supported times.
Abstract
from arXiv · showhide
Streaming video understanding often involves time-sensitive scenarios where models need to answer exactly when the supporting visual evidence appears: answering before the evidence reflects speculation, answering after it has passed reduces real-time utility. To capture this behavior, we introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS), a timing-aware objective with asymmetric early and late penalties. When combined with correctness, ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment. Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering through a lightweight readiness mechanism that decides if sufficient evidence has been observed before responding. To evaluate this capability, we further introduce ProReady-QA, a benchmark with annotated answer evidence windows and proactive multi-turn questions across local and global contexts. StreamReady achieves superior performance on ProReady-QA, and consistently outperforms prior methods across eight additional streaming and offline long-video benchmarks, demonstrating robust and broadly generalizable video understanding capability.
1. Introduction
Streaming video understanding requires models to reason online and answer when supporting evidence appears, not merely produce correct answers with full-video access. StreamReady formalizes this timing requirement, introduces ARS and ProReady-QA, and reports stronger readiness-aware performance.
- Streaming models receive frames sequentially without full-video access, making timely, context-aware reasoning important for real-world applications.
- Existing streaming work primarily evaluates correctness when evidence is already available, leaving proactive answer timing underexplored.
- Existing proactive benchmarks lack annotated answer evidence durations, so they cannot verify whether chosen response times are supported by evidence.
- StreamReady introduces ARS with asymmetric early and late penalties and uses a lightweight readiness token and head to decide when evidence is sufficient.
- ProReady-QA evaluates proactive multi-turn questions across local and global temporal contexts using annotated answer evidence windows.
- StreamReady outperforms existing methods in readiness-aware streaming settings and generalizes to other streaming and offline long-video benchmarks.
2. Related Works
Prior offline long-video methods model extended temporal context using memory- or agent-based approaches, but their full-video access and query-specific memory rebuilding do not fit sequential streaming settings.
- Offline long-video methods use memory-based or agent-based approaches, with some adopting query-conditioned storage.
- These methods rebuild memory for every query and rely on full-video access, making them unsuitable when frames and questions arrive sequentially.
3. Method
StreamReady combines hierarchical visual and contextual memories with query-aware reasoning and a lightweight readiness mechanism. It retrieves relevant evidence across temporal scales, monitors readiness, and answers only after sufficient support appears.
- Memory and retrieval: StreamReady stores streaming video in hierarchical visual memory and retrieves temporally relevant context when questions arrive.
- Memory and retrieval: The visual memory tree uses multi-granular clustering and abstraction to preserve long-range details while reducing redundancy.
- Memory and retrieval: A contextual memory bank stores prior question embeddings and answer representations for semantic history and multi-turn context reuse.
- Query-aware reasoning: Dual-branch reasoning combines short-term recent context with coarse-to-fine retrieval from long-term prototypes and centroids.
- Query-aware reasoning: Prototype selection identifies broad relevant regions, while centroid selection isolates fine-grained evidence for focused long-term reasoning.
- Readiness mechanism: A learnable <RDY> token within long-term reasoning tracks the transition from diffuse retrieval to an answer-bearing state as evidence arrives.
- Readiness mechanism: The readiness head outputs a score at each timestep and triggers the LLM only when that score exceeds a threshold; otherwise observation continues.
- Readiness mechanism: Readiness training updates the readiness head and token separately from the reasoning module, with temporal coherence regularization reducing unstable signals.
4. ProReady-QA Benchmark and Evaluation
ProReady-QA evaluates proactive streaming video understanding with annotated evidence windows, multi-turn questions, and tasks spanning local and global temporal contexts. Its readiness-aware evaluation combines correctness with answer timing through ARS and effective accuracy.
- Benchmark Design: ProReady-QA provides annotated answer evidence windows for systematic evaluation of correctness and temporal appropriateness.The benchmark identifies when sufficient evidence first appears and when it ceases to be valid.
- Benchmark Design: The benchmark contains proactive multi-turn questions covering five tasks that require models to track future evidence and make temporally aware decisions.Tasks include sequential steps recognition, repetitive event counting, clues reveal responding, causal trigger detection, and goal-state detection.
- Dataset Construction: ProReady-QA includes 10 one-hour Ego-4D videos, 22 half-hour MovieNet videos, and 5k proactive QA pairs with annotated evidence-window timestamps.Questions were created through semi-automatic generation with human refinement, while complex tasks included manually authored questions.
- Dataset Construction: The dataset supports local and global multi-turn dependencies, with later questions referencing earlier entities or events for long-horizon reasoning.Evidence windows were aligned using visual frames, subtitles, action boundaries, and native annotations.
- Readiness Evaluation: ARS penalizes answers given before or after the evidence window, using harsher early penalties and milder late penalties to assess readiness.Effective accuracy combines ARS with answer accuracy, while multiple valid answers are scored per turn and averaged.
- Readiness Evaluation: Table 2 compares readiness-aware streaming methods using accuracy, Answer Readiness Score, and effective accuracy, with best and second-best results highlighted.These metrics jointly distinguish answer correctness from whether the response was appropriately timed.
5. Experiments
Experiments evaluate StreamReady on readiness-aware, streaming, and offline long-video benchmarks, then analyze component contributions, penalty settings, and computational cost.
- Streaming Video Understanding: StreamReady improves streaming performance by up to ~5% on proactive tasks, while stronger retrieval and long-horizon reasoning account for most accuracy gains.
- Offline Long Video Understanding: StreamReady also outperforms prior models on offline long-video understanding, where temporal awareness is irrelevant.
- Ablation Studies: The readiness mechanism provides modest ARS gains alone, while stronger memory and reasoning modules produce the major combined improvements in accuracy and timing.
- Penalty Sharpness: Larger γe suppresses premature answers, lower γℓ tolerates slight delays, and γe=6 with γℓ=1 provides a balanced trade-off.
- Computation Cost and Inference Latency: StreamReady maintains stable latency and memory as video length increases through fixed-size compact memory, while its readiness head adds no extra inference overhead.
6. Conclusion
The conclusion presents readiness-aware streaming understanding as evaluating both answer content and timing relative to visual evidence. It combines ARS, StreamReady, and ProReady-QA to support accurate and timely responses.
- Readiness-aware streaming understanding evaluates not only what a model answers but also when it answers relative to visual evidence.
- StreamReady integrates long-horizon temporal reasoning with a lightweight readiness mechanism that determines when sufficient evidence has appeared.
- ProReady-QA evaluates this capability with proactive multi-turn questions and annotated answer evidence windows.
Supplementary Material
The supplementary material adds ablations, quantitative and qualitative analyses, benchmark-generation details, implementation details, and future research directions.
- The supplementary material provides additional ablation studies and quantitative and qualitative analyses.
- It also documents the ProReady-QA generation pipeline and additional implementation details.
- The supplement concludes by outlining directions for future research.
A. Additional Ablation
Additional ablations examine generalization across proactive and non-proactive streaming tasks and offline long-video understanding, focusing on memory construction and query-aware reasoning.
- Additional Ablation: Additional ablations cover three ProReady-QA tasks, two StreamingBench tasks, and VideoMME-Long across proactive, non-proactive, and offline settings.
- Contribution of Each Memory Bank: Similarity-based clustering outperforms caption-based summarization, while adaptive hierarchical clustering preserves temporal structure with compact memory.
- Contribution of Each Memory Bank: Adaptive hierarchical clustering yields the strongest accuracy and ARS on ProReady-QA and the best performance on StreamingBench and VideoMME-Long.
- Design Choice of Query-Aware Reasoning: Without query-aware retrieval, the model cannot focus on the correct memory region or judge evidence sufficiency, while long-term centroid-level reasoning produces larger improvements.
B.1. Model Behavior Analysis
StreamReady’s model behavior depends on balancing hierarchical memory, selective retrieval, contextual filtering, and readiness supervision. These design choices preserve reasoning quality while improving timing-aware performance across streaming tasks.
- Memory capacity and depth: Each visual-memory level has an optimal capacity; larger buffers can plateau accuracy while reducing ARS through slower retrieval and timing penalties.The effect is especially sharp for raw-frame buffers and centroid levels, where excessive capacity adds latency or noise.
- Memory capacity and depth: A three-level visual memory with 24 frames, 96 centroids, and 12 prototypes balances abstraction, temporal detail, retrieval speed, and ARS.Two-level designs lose mid-level cues, whereas four-level designs over-fragment memory and introduce timing penalties.
- Query-aware retrieval: Query-aware retrieval performs best with a selective number of prototypes and centroids, since too few miss relevant contexts while too many add noise and overhead.The coarse-to-fine strategy therefore trades evidence coverage against retrieval efficiency.
- Contextual reasoning: Similarity-guided filtering of prior QA history improves reasoning over naive attention to all previous turns, which reduces StreamingBench accuracy by roughly 4 −5%.Unfiltered histories introduce unrelated reasoning traces, while relevance gating preserves semantically aligned context and temporal alignment.
- Readiness and robustness: Readiness supervision changes timing behavior rather than correctness, while StreamReady remains robust across 7B-scale backbones and dense ET-Bench tasks.The readiness pathway learns an evidence-driven window, and the framework’s gains remain consistent across streaming benchmarks and task types.
B.2. ARS Metric Behavior Analysis
ARS evaluates whether answers align with the evolving availability of visual evidence, using smooth asymmetric penalties for premature and delayed responses. Its design supports stable interpretation across varied evidence durations and boundary cases.
- Metric design: ARS uses median evidence duration to normalize timing penalties across short and long evidence windows.This avoids overly harsh penalties for short windows and overly lenient penalties for long windows.
- Metric design: The metric applies asymmetric penalties: early answers are penalized more harshly, while slight delays after evidence ends receive milder treatment.Softmin and softmax transitions prevent abrupt scoring changes near evidence-window boundaries.
- Edge cases: ARS assigns full grounding when the answer occurs within the evidence window, while late responses decay gradually rather than collapsing immediately.Far-early answers approach zero early support, whereas delays beyond the window remain partially credited.
- Metric scope: ARS can evaluate evidence sufficiency even when precise evidence-window annotations are unavailable.Its underlying requirement is sufficient evidence rather than exact temporal boundaries.
B.3. Qualitative Analysis
Qualitative examples show that StreamReady links readiness decisions to the same evidence used for reasoning, enabling accurate answers within evidence windows. The analysis also exposes a counting-specific delay failure under ambiguous evidence.
- Readiness alignment: StreamReady answers accurately and on time because its readiness signal is tied to hierarchical visual memory shared with reasoning.StreamBridge’s decoupled activation module produces mistimed outputs and ARS penalties.
- Readiness alignment: In CTD and CRR examples, StreamReady detects cues within the evidence window, whereas StreamBridge either waits too long or triggers after evidence has ended.These timing differences affect both answer correctness and ARS.
- Counting behavior: In the combined REC+CRR example, StreamBridge hallucinates a count before the true clue appears, while StreamReady waits and counts only evidence-supported events.The premature response and later ungrounded answer incur ARS penalties despite a correct count.
- Failure case: A counting failure occurs when the readiness signal remains high beyond the evidence window while the model waits to confirm that no further event occurs.This ambiguity produces delayed responses and a late ARS penalty.
- Overall behavior: Across the qualitative cases, the combined retrieval, reasoning, and readiness design supports precise evidence tracking and on-time answering.The benchmark includes proactive multi-turn questions built from long-form Ego4D and MovieNet videos with evidence annotations.
D. Implementation Details
The implementation trains the reasoning and readiness components while keeping the visual encoder and language decoder frozen. The paper also identifies multi-view, multimodal, and embodied streaming as future extensions.
- Training setup: Fine-tuning trains the reasoning module and readiness mechanism while the visual encoder and language decoder remain frozen.Training uses five epochs, a 2e −5 learning rate, cosine annealing, AdamW, and 1 FPS video sampling.
- Future directions: Future work includes extending the framework from single-view video to multi-camera and multi-agent settings with asynchronously appearing evidence.The paper also proposes multimodal streaming inputs such as audio and readiness-aware embodied agents.
- Training setup: The dataset statistics table documents the training data used separately for the reasoning module and readiness mechanism.These statistics correspond to the components fine-tuned in the implementation.