PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning

Ziheng Huang, Yicheng Bao, Xueheng Li, Zhenkun Gao, Bangwei Liu, Kunquan Li, Yuxiang Shen, Bangyan Li, Xuejiao Wang, Changbo Wang, Gaoqi He

arXiv:2609.37035v1cs.AI

TL;DR

In streaming video, an online state formed before future questions are known can omit details that later questions make relevant. Watch-Think-Interact combines time-indexed memory with selective recall, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.

  • Problem

    In causal streaming, retained state is formed before future questions are known, so visual details excluded by compression may be unavailable when later questions reveal their relevance.

    Source: Introduction · p. 1

  • Method

    WTI uses time-indexed natural-language memory to support direct reasoning or selective recall of relevant source-video evidence, and trains multi-turn behavior with trajectory-level reinforcement learning.

    Source: Introduction · p. 2

  • Results

    83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench establish WTI's state of the art among compared open-source streaming models.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Ablations support complementary contributions from memory and source-video recall to WTI's performance.

    Source: Conclusion · p. 7

    WTI assumes a causal stream: future chunks and the time when sufficient evidence becomes available are unavailable to the model at inference.

    Source: Problem Definition · p. 3

Abstract

from arXiv · show

Streaming video assistance requires models to answer asynchronous questions from an observed prefix under a fixed context budget. Existing approaches model response timing or compress history, but an online state formed before future questions are known can omit visual details before later questions reveal their relevance; the retained state alone cannot recover them. We introduce Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning. WTI maintains compact natural-language memory entries tagged with source-video time ranges; these entries support direct reasoning when sufficient and otherwise anchor selective recall of finer visual evidence. For each question, WTI answers when current context and memory suffice, continues watching when required evidence has not appeared, or recalls a relevant past interval and decides again after incorporating the returned chunks, without replaying the full observed history. To train this behavior, we construct WTI-82K, comprising 82,335 timed questions across 4,812 causally aligned trajectories, and develop Stream-GDPO to optimize complete multi-question streaming rollouts using trajectory-level feedback for response timing, source-video recall, and memory updates. WTI achieves state-of-the-art aggregate performance among the compared open-source streaming baselines, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.

Introduction

Streaming video reasoning must answer questions from an incomplete, evolving context, where forward-only compression can lose evidence that later questions make relevant. WTI addresses this with time-indexed memory and selective source-video recall, trained for multi-turn interaction.

Motivation

  • Because streaming state is formed before future questions are known, forward-only compression can omit visual details that later become relevant, and reasoning over that state cannot recover them.Streaming assistance must answer from the observed prefix and respond once the active visual context and retained history provide sufficient evidence; otherwise it must wait or revisit observed evidence.Source: Introduction · p. 1

Watch-Think-Interact

  • WTI answers when evidence suffices, waits for not-yet-observed evidence, or recalls a relevant past interval and decides again without replaying the full history.Its compact memory entries pair semantic summaries for direct reasoning with source time ranges that localize finer visual evidence.Source: Introduction · p. 1

Training data

  • WTI-82K contains 82,335 timed questions across 4,812 causally aligned multi-turn trajectories linking query times, answerable moments, evidence, state updates, and actions.Relative to the 8-second active window, 56.9% of trajectories span at least 120 seconds and 21.7% span at least 240 seconds.Source: Introduction · pp. 1–2

Results

  • WTI sets an open-source streaming-model state of the art, scoring 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench while leading its Real-Time, Backward, and Forward groups.It also retains strong offline long-video performance.Source: Introduction · p. 2

Training method

  • Stream-GDPO optimizes complete multi-turn rollouts for response timing, source-video recall, and memory updates, with separate normalization of outcome, format, recall, and memory rewards.Masked SFT initializes the interaction protocol before trajectory-level reinforcement learning.Source: Introduction · pp. 1–2

Related Work

Related work spans online streaming systems, long-stream memory, and agentic multimodal reasoning. WTI combines these concerns by enabling later video questions to recover source-linked evidence.

Streaming Video Understanding

  • Streaming systems model answer timing and interaction, while WTI addresses later turns after source-linked evidence leaves the active visual window.Prior systems add response-state modeling, feedback policies, proactive interaction, streaming thinking, or instruction tuning (Chen et al. 2024; Qian et al. 2025; Wang et al. 2026; Guan et al. 2026; Xia et al. 2025).Source: Related Work · p. 2

Agentic Multimodal Reasoning

  • WTI brings tool-based evidence acquisition and compact-state revisiting to streaming video through wait, answer, or recall decisions.Agentic multimodal models acquire evidence through tools or visual inspection, while memory agents update or revisit compact state (Zheng et al. 2026; Hong et al. 2026; Gao et al. 2026; Yu et al. 2026; Shi et al. 2026; Wang et al. 2026).Source: Related Work · p. 2

Method

WTI frames streaming video assistance as causal, multi-question reasoning over observed prefixes, with compact time-indexed memory and selective recall supporting bounded-context interaction. Masked SFT cold-starts this protocol, and Stream-GDPO trains complete rollouts with component-wise feedback for answers, format, recall, and memory.

Problem Definition

  • At each question, the model must answer no earlier than sufficient evidence is available and ground its response in the observed prefix.Question timestamps and answerable times support data construction and evaluation; answerable time is hidden at inference.Source: Problem Definition · p. 3

Watch-Think-Interact Framework

  • WTI chooses silence, response, or recall after a thinking update; recall returns past evidence for a new decision without advancing the stream.Recall intervals use absolute source-video seconds and must precede the current chunk; the protocol permits one recall per chunk, returning at most four observed chunks.Source: Watch-Think-Interact Framework · p. 3
  • Compact memory entries pair semantic notes with source-time ranges, enabling direct reasoning when sufficient or selective inspection of finer evidence.The fixed memory budget and active visual window keep model-visible context bounded, while the controller-managed archive grows with the observed stream.Source: Watch-Think-Interact Framework · pp. 3–4

Masked Supervised Fine-Tuning

  • Masked SFT trains on causal, bounded inference state using stream-causal and label masks, balancing sparse action targets against thinking and memory-update targets.The stage supervises teacher action and state-update targets while stream inputs, environment observations, and recall returns remain conditioning context.Source: Masked Supervised Fine-Tuning · p. 4

Stream-GDPO Reinforcement Learning

  • Stream-GDPO assigns trajectory-level credit across online decisions and component-wise normalizes outcome, format, recall-use, and memory rewards before combining advantages.Recall reward is gated on a correct, causally valid response; persistent memory quality is scored at rollout level, while the clipped objective retains trajectory-level credit without separate turn- or token-level advantages.Source: Stream-GDPO Reinforcement Learning · pp. 4–5

WTI-82K: Causally Aligned Streaming · Interaction Data

WTI-82K turns streaming video questions into causally aligned, stateful interaction trajectories rather than isolated video-QA pairs. Its construction links temporally grounded evidence to questions, answerable moments, and interaction decisions under the visible prefix.

Interaction Data

  • 82,335 timed questions form 4,812 trajectories aligning observed chunks, query times, answerable moments, supporting evidence, state updates, and actions under causal context.Trajectories average 17.1 questions; 56.9% span at least 120 seconds and 21.7% at least 240 seconds, compared with an 8-second active window.Source: Interaction Data · p. 5
  • 49.9% of questions address historical, future, multistate, or temporal reasoning, requiring repeated decisions over an evolving stream rather than current-frame answers.The dataset is organized around shared trajectories, not isolated questions.Source: Interaction Data · p. 5
  • Temporal evidence extraction filters for clear visuals and locatable events, then links chunk-level facts that persist or change to minimal supporting intervals.These intervals support evidence-chain selection and time-anchored question generation.Source: Interaction Data · p. 6
  • Qwen3.5-397B generates candidate questions and reference answers from local facts and cross-chunk evidence chains, covering immediate, historical, temporal, and proactive interaction.This produces both current-prefix questions and questions requiring long-range state maintenance.Source: Interaction Data · p. 6
  • Query and answerable timestamps are assigned from supporting evidence intervals following timestamped online-video protocols (Niu et al. 2025; Yang et al. 2025; Wang et al. 2026).Real-Time questions are answerable from the visible prefix; Backward Tracing questions require past evidence; Proactive questions require waiting for future evidence.Source: Interaction Data · p. 6
  • The resulting samples expose answernow, recall-before-answer, and wait-for-evidence behaviors under evaluation visibility constraints.Automated checks verify temporal bounds, answer availability, action grammar, and evidence localization before serialization.Source: Interaction Data · p. 6

Experiments

Experiments use a shared streaming setup for training and evaluation, compare WTI across streaming and long-video benchmarks, and track training-time diagnostics.

Experimental Setup

  • WTI streams 1-second chunks at 2 FPS with an 8-chunk active window, and recall returns at most four observed chunks once per current chunk.SFT, RL, and evaluation share the chunking, memory budget, recall tool, and action grammar; inference triggers compact-and-reprefill at the active-context boundary.Source: Experimental Setup · p. 6
  • Evaluation covers OVO-Bench, StreamingBench, Video-MME, MLVU, and LongVideoBench, while comparisons include proprietary, offline/static, and online/streaming Video-LLMs.OVO-Bench reports Real-Time, Backward, Forward, and weighted overall accuracy; StreamingBench covers ten real-time visual-understanding categories.Source: Experimental Setup · p. 6
  • Figure 4 tracks outcome accuracy, BT recall triggering, post-recall outcome accuracy, and memory quality during GSPO/Stream-GDPO training.The caption distinguishes raw curves from rolling values but provides no numerical results.Source: Figure 4 · p. 7

Config. Stream. OVO-RT OVO-BT OVO-FT OVO Avg. V-MME MLVU LVB

WTI leads open-source streaming baselines on the reported aggregate benchmarks while retaining strong offline long-video performance. Ablations show that rollout optimization and complementary memory and visual recall contribute to its reasoning gains.

Config. Stream. OVO-RT OVO-BT OVO-FT OVO Avg. V-MME MLVU LVB

  • Table 4 separates protocol initialization from trajectory-level policy optimization; masked SFT raises OVO-RT from 64.8% to 74.2% but reaches only 60.3% on OVO-BT and 60.9% on OVO-FT.Its OVO-FT score remains below the base model’s 63.5%, indicating imitation initializes recent-prefix behavior but is weaker on Backward and Forward reasoning than rollout optimization.Source: Table 4 · p. 7, Ablations · p. 7

Main Results

  • WTI scores 73.6% overall on OVO-Bench, leading open-source streaming baselines, and achieves the best reported Real-Time, Backward, and Forward scores: 76.9%, 73.4%, and 70.0%.Its overall score exceeds StreamBridge-7B (62.6%) and ViSpeak-7B (61.1%).Source: Main Results · p. 7
  • WTI reaches 83.3% on StreamingBench, ahead of VST-7B (79.5%), StreamForest-7B (77.3%), and StreamBridge-7B (77.0%), while scoring 66.9% on Video-MME, 69.9% on MLVU, and 62.9% on LongVideoBench.It leads reported streaming baselines on Video-MME and LongVideoBench and is within 0.1 point of StreamForest-7B on MLVU (69.9% vs. 70.0%).Source: Main Results · p. 7

Ablations

  • Stream-GDPO raises OVO-BT from 65.3% to 73.4% and OVO-FT from 60.7% to 70.0% versus GSPO, lifting the overall score to 73.6%.It sustains stronger Backward recall triggering, post-recall accuracy, and memory quality, and separately normalizes outcome, format, recall, and memory signals before aggregation.Source: Ablations · p. 7
  • Removing memory lowers OVO-BT from 73.4% to 62.8%, while removing recall lowers it to 55.5%; the memory-only variant reaches 48.2%.Offline, removing recall reduces Video-MME from 66.9% to 48.3%, while removing memory reduces MLVU from 69.9% to 44.7%; text memory indexes history but cannot replace visual evidence restored by recall.Source: Ablations · p. 7, Table 5 · p. 7

Conclusion

WTI combines active-window perception, time-indexed memory, source-video recall, and response timing for long-horizon streaming video reasoning. Its training and evaluation show strong performance on streaming and offline long-video tasks.

  • WTI sets a new open-source state of the art on StreamingBench and OVO-Bench while retaining strong offline long-video performance.Ablations verify complementary contributions from memory and recall.Source: Conclusion · p. 7
  • WTI combines bounded active-window perception, time-indexed memory, source-video recall, and response timing in a causal framework for long-horizon multi-turn streaming video reasoning.Source: Conclusion · p. 7
  • WTI-82K supplies causally aligned trajectories, and Stream-GDPO optimizes complete online rollouts in the same chunk-level environment used at inference.Source: Conclusion · p. 7

Related papers

Loading 2609.37035v1…