Source-linked AI summary
STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding
Junho Kim, Hosu Lee, James M. Rehg, Minsu Kim, Yong Man Ro
TL;DR
Video-LLMs are strong offline reasoners, but streaming systems must decide when to respond as frames arrive and existing activation methods can make unstable point-wise decisions. STRIDE models activation as a structured sequence over a temporal window using lightweight masked diffusion, and experiments report more reliable, temporally coherent proactive responses with improved when-to-speak decisions.
Problem
Existing Video-LLMs mainly operate offline, while point-wise activation decisions in streaming can produce flickering, fragmented spans, and poorly resolved transition boundaries.
Method
STRIDE jointly predicts and progressively refines span-structured activation signals over a temporal window using a lightweight masked diffusion module.
Results
STRIDE produces more reliable and temporally coherent proactive responses and significantly improves when-to-speak decisions across streaming benchmarks and downstream models.
Takeaways & Limitations
Jointly modeling activation over a temporal window improves event boundary localization and reduces unstable triggering with minimal streaming-pipeline overhead.
Takeaways & Limitations
At 1 FPS, brief events or rapid visual transitions may disappear before sufficient temporal context is accumulated, limiting reliable localization.
Abstract
from arXiv · showhide
Recent progress in video large language models (Video-LLMs) has enabled strong offline reasoning over long and complex videos. However, real-world deployments increasingly require streaming perception and proactive interaction, where video frames arrive online and the system must decide not only what to respond, but also when to respond. In this work, we revisit proactive activation in streaming video as a structured sequence modeling problem, motivated by the observation that temporal transitions in streaming video naturally form span-structured activation patterns. To capture this span-level structure, we model activation signals jointly over a sliding temporal window and update them iteratively as new frames arrive. We propose STRIDE (Structured Temporal Refinement with Iterative DEnoising), which employs a lightweight masked diffusion module at the activation interface to jointly predict and progressively refine activation signals across the window. Extensive experiments on diverse streaming benchmarks and downstream models demonstrate that STRIDE shows more reliable and temporally coherent proactive responses, significantly improving when-to-speak decision quality in online streaming scenarios.
1 UIUC, 2
The passage identifies equal contribution, corresponding-author, and advisory-role acknowledgments, and provides a Hugging Face resource link.
- The paper marks some authors as equal contributors.
- The paper identifies a corresponding author.
- The paper provides a Hugging Face link for interlive.
1 Introduction
The introduction frames proactive streaming activation as structured sequence modeling and proposes STRIDE to refine span-level activation over temporal windows. It reports more stable, temporally coherent triggering and improved when-to-speak decisions in online settings.
- Existing Video-LLMs mainly process prerecorded videos offline, while streaming systems continuously ingest frames but remain reactive.
- Point-wise binary activation decisions can produce flickering, fragmented spans and poorly resolved transition boundaries.
- STRIDE reformulates when-to-speak control as structured sequence prediction over a temporal activation window.
- STRIDE uses a lightweight masked diffusion model to jointly predict and iteratively refine activation sequences with span-level structure.
- STRIDE shows more stable proactive triggering, improved temporal consistency, and significantly better when-to-speak decisions across streaming benchmarks and downstream backbones.
2 Related Work
Related work spans offline vision-language reasoning, streaming video understanding, proactive response generation, and masked diffusion language models. These lines of work motivate separating proactive triggering from generation and modeling temporal context explicitly.
- LVLMs introduced strong multimodal conversation capabilities, while video-specialized models improved video question answering and captioning.
- Most LVLMs remain offline, limiting their applicability when video frames arrive continuously in real time.
- Streaming video research adapts objectives, data formats, memory, and architectures for continuous inputs and state maintenance.
- Proactive-response methods determine timing through EOS tokens, explicit task-relevant intervals, or query-aware visual pruning.
- Masked diffusion language models iteratively denoise masked sequences with bidirectional conditioning and parallel token updates.
3 Proposed Method
STRIDE treats proactive streaming activation as span-level sequence denoising over a sliding temporal window. Its two-stage framework combines masked-diffusion activation modeling with downstream Video-LLM response generation under partial observability.
- 3 Proposed Method: Masked diffusion predicts all masked activation positions jointly and iteratively refines them through progressive unmasking.The diffusion process uses bidirectional context and accepts subsets of predictions at successive denoising steps.
- 3.1 STRIDE: Proactive Streaming Framework: STRIDE processes incoming frames online under partial observability, using only observed frames and interaction context at each time step.The streaming framework interleaves incoming visual frames with user queries and generated responses.
- 3.1 STRIDE: Proactive Streaming Framework: A lightweight activation model monitors the stream, and triggered responses forward accumulated visual context to a downstream Video-LLM.Generated responses are appended to interaction context to maintain dialogue coherence across activation events.
- 3.1 STRIDE: Proactive Streaming Framework: Activation is represented as a window-level binary sequence, enabling the model to learn contiguous spans and 0→1, 1→1, and 1→0 transitions.This replaces isolated point-wise decisions with sequence-level prediction over a temporal window.
- 3.2 Training: Activation as Sequence Denoising: STRIDE trains with boundary-anchored span masking, span unmasking, and full masking to encourage temporal reasoning over activation spans and boundaries.The three structured corruption strategies are selected with equal probability.
- 3.2 Training: Activation as Sequence Denoising: Sequence duplication restores bidirectional conditioning for masked activation prediction without changing the underlying causal attention layers.The method appends a copy of the activation region so masked positions can use full-sequence context.
4 Experiments
Experiments show that STRIDE improves streaming activation quality, temporal coherence, and proactive response timing across benchmarks and ablations. The method also provides efficient online inference with limited latency and memory overhead.
- Benchmark Evaluation: STRIDE outperforms Baseline-AR across OVO-Bench and StreamingBench, with gains especially pronounced in proactive response-timing tasks.Improvements extend across downstream models and StreamingBench evaluation dimensions.
- Activation Evaluation: 27.1 on TVG and 8.3 on average: STRIDE substantially outperforms Baseline-AR on ET-Bench online activation accuracy.The 2B-parameter activation model also surpasses larger temporal-localization MLLMs on the overall average.
- Masking Strategy: Diverse span corruption strategies significantly improve activation performance over independent masking across all evaluated metrics.Boundary-anchored span, full, and span-unmasking patterns target contiguous activation regions and boundary refinement.
- Sequence Context and Re-masking: Removing sequence duplication reduces the average score from 32.6, while restricting re-masking to the newest position reduces it to 22.6.These ablations support using full-window context and revising uncertain prior decisions.
- Temporal Stability: STRIDE produces smoother activation patterns than Baseline-AR around event boundaries, avoiding frequent on/off oscillations.The analysis measures activation-transition frequency relative to ground-truth events.
5 Conclusion
STRIDE’s activation-span modeling is supported by curated multi-dataset training data that converts temporal annotations into unified binary activation sequences. The resulting corpus covers single- and multi-event samples at substantial scale.
- Training data: Seven video-understanding datasets are curated to train activation span modeling across dense captioning, temporal localization, grounded QA, and procedural understanding.The datasets provide temporal event or action annotations reorganized for activation training.
- Training data: Each training sample pairs a video and event query with a temporal span, labeling positions inside the span active and all others inactive.This unified representation converts event boundaries into binary activation signals over the timeline.
- Training data: Training samples distinguish single-event queries from multi-event queries when constructing activation segments.Multi-event samples represent multiple relevant events, while single-event samples generally correspond to one queried event.
- Training data: The curated corpus contains 141.7K videos, 379.9K annotations, and 654.7K training segments.It includes 310.3K single-event and 69.6K multi-event samples.
A.2 Training Hyperparameters
STRIDE is trained with bounded streaming inputs and a temporal activation window matched to the maximum processed frame sequence. Its sampling procedure varies context placement and handles events inside or outside the observable window.
- Input configuration: STRIDE processes video at 1 FPS for up to 256 frames, using a temporal activation window size W of 256.The configuration supports bounded temporal processing for the streaming setup.
- Context sampling: Training samples use visual context windows uniformly sampled between max(L, 8) and min(L, 256) seconds.The window is randomly positioned along the timeline, where L denotes source-video length.
- Context sampling: Positions for previously completed events remain inactive after masking, while events outside the context window produce an all-zero activation sequence.Overlapping positions for the target event are set active during training.
B Detailed Architecture and Inference of STRIDE
STRIDE represents event activity as a binary sequence and refines masked positions with a diffusion-style denoising process. During streaming inference, a sliding window selectively remasks uncertain history before producing span-based triggers.
- Activation representation: The activation target is a length-W binary sequence whose positions indicate whether the queried event is active.Annotated event-span positions are labeled 1 and all other positions 0.
- Masked diffusion: Forward corruption progressively replaces activation tokens with [M] at noise level t, creating partially masked sequences.The process is defined over the discrete activation sequence.
- Masked diffusion: Structured masking preserves contiguous temporal context while hiding selected portions, encouraging coherent activation spans instead of isolated token predictions.This is the practical masking strategy used during training rather than independent token-wise masking.
- Masked diffusion: Reverse denoising predicts masked tokens from observed context while keeping revealed tokens fixed and progressively unmasking the sequence.The process reconstructs the clean activation sequence iteratively.
- Training objective: Training minimizes cross-entropy only on masked activation positions, using unmasked tokens as context.A 1/t weighting balances loss contributions across noise levels.
- Streaming inference: During streaming inference, low-confidence carried-forward positions are remasked and jointly refined with newly appended positions before triggering when the active span reaches ratio γ.The window shifts over recent positions and applies K-step denoising after selective remasking.
- Streaming inference: Context accumulation is bounded at 256 frames; once exceeded, STRIDE retains the most recent 128 frames and rebuilds the context.This limits overhead while preserving the latest visual evidence.
C.1 Additional Benchmark Explanation
The benchmarks evaluate streaming understanding under partial observability, temporal reasoning, and event-boundary localization. The reproduced autoregressive baseline provides a point-wise comparison against STRIDE’s span-level activation predictions.
- Benchmarks: OVO-Bench evaluates timestamp-conditioned online video understanding across 644 videos, 2,814 QA pairs, and 12 tasks.Its scenarios cover backward tracing, real-time perception, and forward active responding.
- Benchmarks: StreamingBench contains 900 videos and 4,500 QA pairs spanning 18 tasks across three streaming-comprehension dimensions.Questions are presented at different temporal positions to simulate real-time interaction.
- Benchmarks: ET-Bench contributes five temporal-boundary tasks used to measure activation timing independently of the end-to-end streaming pipeline.The broader benchmark contains 7,300 samples across 12 tasks and eight domains.
- Benchmarks: ET-Bench temporal localization is evaluated with F1, which measures alignment between predicted and ground-truth event boundaries.The metric is applied across the five adopted tasks.
- Baseline: Baseline-AR is retrained with the same backbone, training data, and input configuration as STRIDE for a fair autoregressive comparison.Its score head independently predicts a trigger probability for each frame and applies a fixed threshold of 0.35.
D.1 Sensitivity Analysis for τ
The sensitivity analysis evaluates how the retention threshold τ affects performance across five ET-Bench temporal understanding tasks. Performance is lowest with unconditional inheritance, peaks broadly at intermediate thresholds, and motivates using τ = 0.75.
- The evaluation uses score differences relative to each task’s average, with task-wise average scores reported in the legend.
- τ = 0 produces the lowest scores, including a −19.7 pt drop on TVG.The comparison is relative to each task’s average score.
- Performance peaks broadly at τ ∈[0.75, 0.85] across most tasks.The evaluated tasks are TVG, EPM, TAL, DVC, and SLC.
- Tightening the retention criterion beyond the peak range causes a gradual performance decline.The threshold controls confidence-based retention of prior activation decisions as the window slides.
D.2 Scalability Analysis for Activation Backbone
The scalability analysis tests larger activation backbones across downstream Video-LLMs on OVO-Bench and StreamingBench. STRIDE-4B consistently outperforms STRIDE-2B, indicating that increased activation-backbone capacity transfers across downstream models.
- The 4B activation backbone is trained with the same data and configuration as the 2B model.
- STRIDE-4B consistently achieves higher overall scores than STRIDE-2B across downstream backbones.The comparison covers OVO-Bench and StreamingBench across multiple downstream Video-LLMs.
- The improvement transfers regardless of the downstream Video-LLM, supporting scalability of the plug-in activation design.
- Qualitative examples visualize query arrival time, ground-truth event spans, and STRIDE-predicted activation spans on three streaming benchmarks.The timelines illustrate progressive identification of relevant event segments and alignment with temporal boundaries.
E.1 Failure Cases and Discussion
The discussion identifies two practical failure modes for proactive activation: sparse temporal sampling can miss brief events, while broad queries can yield ambiguous multi-span activations. Qualitative examples span ET-Bench, OVO-Bench, and StreamingBench, including proactive-output and scene-understanding cases.
- Temporal sampling limitations: 1 FPS sampling can miss extremely short-lived events or rapid visual transitions before sufficient temporal context accumulates.A brief event between sampled frames may therefore be impossible to localize reliably.
- Query ambiguity: Broad or loosely defined queries may match multiple candidate segments, producing dispersed or multi-span activations.
- Query ambiguity: Multiple visually plausible moments can make the appropriate triggering point ambiguous.The discussion links this sensitivity to query specificity and temporal sampling granularity.
- Qualitative examples: Qualitative examples cover ET-Bench tasks including TVG, EPM, TAL, DVC, and SLC.
- Qualitative examples: Additional qualitative examples cover OVO-Bench and StreamingBench recognition, understanding, proactive-output, and scene-understanding cases.