Source-linked AI summary
Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
Lu Wang, Zhuoran Jin, Yupu Hao, Yubo Chen, Kang Liu, Yulong Ao, Jun Zhao
TL;DR
Continuously arriving video makes causal multi-turn reasoning difficult because existing interleaved systems can lose early memory and serialize perception with generation. Think While Watching preserves segment-level memory through staged streaming-aligned training and causal architecture, while overlapping watching and thinking at inference. It improves single-round accuracy on two streaming benchmarks and maintains multi-round performance with substantially fewer output tokens.
Problem
Existing video MLLMs are mainly evaluated offline, while streaming multi-turn interaction requires causal answers that retain earlier visual evidence as new segments arrive.
Method
Think While Watching uses persistent segment-level memory, three-stage streaming-aligned CoT training, causal masking with separate streaming positional encodings, and an overlapping inference pipeline.
Results
2.6% higher accuracy on StreamingBench and 3.79% higher accuracy on OVO-Bench were achieved for single-round streaming, while multi-round performance was maintained with 56% fewer output tokens.
Takeaways & Limitations
The framework supports online multi-turn video reasoning while preserving performance and reducing generated output under the evaluated streaming protocols.
Takeaways & Limitations
The formal setting assumes strict causality: each answer uses only the observed video prefix, the current question, and prior dialogue history.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/
1 Introduction
Think While Watching addresses the difficulty of multi-turn reasoning over continuously arriving video by preserving segment-level memory and decoupling perception from generation. Its streaming design and training strategy improve single-round accuracy while maintaining multi-round performance with fewer output tokens.
- Streaming video assistants must answer questions in real time using visual evidence observed so far, including earlier memories needed by later turns.The setting covers live broadcasting, monitoring, robotic assistants, and other continuously arriving streams.
- Interleaved perception and generation serializes processing, causing memory erosion and a bottleneck where text decoding blocks further video ingestion.The method targets both degraded long-range recall and increasing latency as streaming rounds accumulate.
- Think While Watching maintains persistent segment-level memory, answers queries by retrieving relevant memories, and overlaps visual ingestion with text decoding.Its inference design uses a dual KV cache to support parallel perception and generation.
- The framework combines a three-stage multi-round CoT dataset with stage-matched training, streaming causal masking, and streaming positional encoding.These components are designed to enforce causal segment-level modeling during streaming reasoning.
- 2.6% higher single-round accuracy on StreamingBench and 3.79% higher accuracy on OVO-Bench were reported, while multi-round performance was maintained with 56% fewer output tokens.The results cover both single-round and multi-round streaming protocols.
2 Related Work
Prior work improves offline video understanding, streaming efficiency, or context reuse, but interleaved streaming systems remain coupled across perception and generation. Think While Watching focuses on stable segment-level memory and decoupled inference for multi-turn streaming.
- Most offline video methods assume the full video is available before answering, leaving causal online multi-turn interaction less explored.
- Streaming systems commonly alternate visual ingestion and text decoding, limiting input-output parallelism and long-horizon dependency modeling across turns.
- Efficiency research reduces redundant visual tokens or reuses historical context through KV-cache retrieval and compression.
- Think While Watching emphasizes persistent segment-level memory together with an inference design that keeps perception and generation decoupled.
3 Preliminary
The paper models video as an ordered stream of segments with questions arriving at segment boundaries, and defines both single-round and multi-round causal protocols. Received units and aligned generated units provide a unified notation for online processing.
- 3.1 Streams and Turns: A video stream is represented as an ordered sequence of contiguous frame segments that arrive and are processed online.
- 3.1 Streams and Turns: At each turn, a question arrives after an observed stream prefix, and its answer is conditioned only on that prefix and prior dialogue history.The latest observed segment indices are nondecreasing across turns.
- 3.2 Streaming Protocols: Single-round streaming uses one question and one online output, whereas multi-round streaming uses multiple questions at different times and answers each from the available prefix.
- 3.3 Streaming Unit Notation: The interaction is serialized as received units containing segments or questions and one aligned generated unit for each received unit.A segment produces a memory note, while a question produces a rationale and final answer.
- 3.3 Streaming Unit Notation: Figure 2 illustrates causal attention over received and generated units while separate input and output position indices prevent future-segment leakage.
4 Method
Think While Watching combines persistent segment-level memory with causally constrained streaming architecture, stage-matched training, and parallel inference. The design preserves online causality while decoupling perception from generation for multi-turn video reasoning.
- Segment-level Memory: The framework writes one persistent memory note per arriving video segment, preserving reusable evidence throughout multi-turn interaction.Notes capture entities, attributes, actions, interactions, scene changes, and short-range temporal relations.
- Streaming Causality: The segment-level streaming mask prevents future received units from leaking into earlier outputs and restricts each generated unit to the available received prefix and prior generated units.For example, C1 attends to S1, C2 to ⟨S1, Q1⟩, and C3 to ⟨S1, Q1, S2⟩.
- Streaming Positional Encoding: Streaming MRoPE assigns input positions from the received prefix and output positions independently from generated tokens, allowing new segments to receive correct positions during decoding.This supports parallel reasoning even when the eventual output length is unknown.
- Three-stage Training: A three-stage training strategy progresses from single-round memory writing to multi-round dialogue and long-range evidence recall with uncertainty handling and distractors.The dataset includes short-video single-round and multi-round examples plus long-video, multi-round instances from YouTube.
- Parallel Inference: At inference, a dual KV cache decouples source ingestion from autoregressive decoding while adaptive attention selects Flash Attention or an explicit memory-efficient masked backend.The backend choice depends on whether the streaming pattern is standard causal and whether query and key lengths differ.
5 Experiments
Think While Watching is evaluated across streaming and offline video-understanding settings, with analyses of long-range attention, memory, segmentation, and latency. The method improves streaming accuracy and efficiency while transferring benefits to offline long-video reasoning.
- Evaluation Setup: StreamingBench and OVO-Bench evaluate single-round and multi-round online video understanding, alongside offline and efficiency metrics.The evaluation reports accuracy, average output tokens, token reduction, average frames, and time to first token.
- Main Results: Naive multi-turn streaming reaches only 21.47% and 18.13% overall on StreamingBench, versus 56.67% and 58.52% offline for Qwen3-VL-4B Instruct and Thinking.OVO-Bench shows the same pattern, with online accuracies of 21.45% and 16.21% versus offline accuracies of 50.32% and 50.70%.
- Main Results: 56% fewer output tokens are achieved in multi-round streaming while performance is maintained.For the 4B backbone, TWWmulti-turn,S3 achieves 57.40% on StreamingBench; Stage 3 generally improves over Stage 2.
- Offline Generalization: 73.41% on Video-MME and 57.68% on LV-Bench show that streaming training transfers to offline video understanding.TWWsingle-turn,S3 improves Video-MME from 68.89% to 73.41% and LV-Bench from 53.47% to 57.68%.
- Analysis: Stage 3 shifts attention from near-history toward distant segments, especially through accumulated MEMORY tokens.Removing the memory bank drops accuracy from 57.40% to 52.35%; shorter 30s/15s segments preserve accuracy but raise decoding length to 380.50 tokens (+25.8%).
- Latency: 92.6% lower TTFT reduces processed tokens before the first answer token from 31203.69 to 2304.28 while maintaining comparable accuracy.The method decouples ingestion from decoding, avoiding decoder-induced backlog growth as ρ approaches 1.
6 Conclusion
Think While Watching maintains segment-level memory and supports causal multi-turn streaming through a practical inference pipeline. The reported evaluation validates effectiveness, while future work targets broader robustness and evaluation.
- The framework maintains segment-level memory, enforces strict causality, and supports real-time deployment through a dual KV cache with adaptive attention backends.
- The evaluation reports improved online accuracy and strong efficiency on StreamingBench and OVO-Bench.
- Future work includes adaptive segmentation, audio and speech integration, robustness on very long streams, and evaluation combining accuracy, latency, and resource usage.
B Implementation Details
The implementation fine-tunes Qwen3-VL instruction models with full-parameter supervised learning on eight NVIDIA RTX A6000 GPUs. Detailed training settings are summarized in Table A1.
- Experiments use 8× NVIDIA RTX A6000 GPUs with 48GB per GPU.
- Qwen3-VL-2/4/8B-Instruct models are fine-tuned using full-parameter supervised fine-tuning.
- Table A1 summarizes the main training settings and hyperparameters.
C Theoretical Latency Derivation
The latency analysis models interleaved decoding as ingestion downtime, showing that stream lag amplifies as utilization approaches saturation. Dual KV caching reduces this coupling, though practical overheads leave residual backlog possible.
- Queueing model: The queueing model defines utilization as ρ ≜ λ/µ, where λ is segment arrival rate and µ is segment processing rate.
- Interleaved baseline: Interleaved decoding pauses ingestion while new segments arrive, creating a backlog that later drains at net rate µ − λ when µ > λ.
- Latency amplification: Tcatch = ρ/(1 − ρ) Tdec, so each decoding second adds ρ/(1 − ρ) seconds of future catch-up delay.
- Decoupled design: Dual KV caching allows ingestion to proceed during decoding, substantially reducing decoder-induced backlog growth.
- Practical boundary: Residual backlog can arise from scheduling, synchronization, cache maintenance, and overlap overheads, so zero backlog is not guaranteed.
- Latency amplification: Increasing the generated-token count L increases decoding time and therefore future stream lag under interleaving.
- Related systems: Flash-VStream, VideoLLM-online, Dispider, StreamForest, and StreamAgent represent related approaches using memory, streaming pipelines, disentanglement, or selective recall.
E.1 Benchmark Details
The benchmark suite evaluates streaming understanding across timestamped question answering, temporal awareness, adapted offline video tasks, and extreme long-video capabilities.
- StreamingBench: StreamingBench contains 18 tasks over 900 videos and 4,500 human-curated QA pairs tied to specific video timestamps.
- StreamingBench: StreamingBench covers real-time visual understanding, omnisource understanding, and contextual understanding.
- OVO-Bench: OVO-Bench evaluates backward tracing, realtime understanding, and forward responses requiring sufficient future evidence.
- Video-MME: Video-MME is adapted by grouping QA pairs by video and presenting each video as an ordered stream of temporal segments.
- LV-Bench: LV-Bench measures entity recognition, event understanding, key information retrieval, temporal grounding, reasoning, and summarization in extreme long videos.
E.2 Overall Dataset Composition
The dataset combines three stage-aligned training objectives with strict pseudo-streaming generation constraints for causal, segment-level video reasoning. Its prompt and data design enforce chronological processing, video-grounded updates, and question-aware answers without future information leakage.
- Three-stage composition: Stage 1 targets single-round memory writing and answering, Stage 2 multi-round consistency, and Stage 3 long-horizon recall, uncertainty handling, and distractor robustness.Stage 1 and Stage 2 draw from VideoChatOnline-IT, while Stage 3 uses long YouTube videos retrieved across tutorial, lecture, and longform categories.
- Stage 1: Stage 1 trains segment-level memory notes and single-question answers grounded only in the observed video prefix.Its source subsets are HiREST, DiDeMo, and QuerYD, with statistics reported in Table A2.
- Stage 2: Stage 2 trains later answers to reuse earlier segment-level memory notes without accessing future segments.Table A3 reports segment and question counts, including average segments and questions per sample.
- Stage 3: Stage 3 uses long videos spanning procedural, explanatory, and continuous-recording content to target long-horizon streaming behaviors.More than 500 retrieval keywords are used to collect candidate YouTube videos.
- Synthesis constraints: For S segments and Q questions, pseudo-streaming CoT synthesis produces exactly S + Q generated items in chronological interleaved order.The prompt uses delimiters for segment and question units and requires one reasoning chunk per input unit.
- Streaming prompt: The prompt enforces causal, video-grounded reasoning by restricting each segment and question to evidence available through the current time and prohibiting future information or answer leakage.Segment reasoning tracks observed evidence and state continuity, while question reasoning uses only prior units and copies the reference answer verbatim in the Answer field.
G Error Analysis
The error analysis identifies residual multi-turn failures despite improved long-horizon reasoning. These failures involve losing fine-grained early details, committing before decisive evidence, and allowing salient distractors to corrupt retrieval.
- Residual failure patterns: Long-range recall can preserve the coarse event trace while losing an early fine-grained identity, object, or side attribute.Figure A6 depicts an incorrect later retrieval of the person referenced in an earlier question.
- Residual failure patterns: Compact memory notes support long-range access but may over-compress details across long temporal gaps.This limitation links the memory representation to residual fine-grained recall errors.
- Residual failure patterns: Under incomplete evidence, the model may commit to a specific hypothesis before decisive visual evidence appears.This reflects a residual limitation of the uncertainty-handling objective targeted in Stage 3.
- Residual failure patterns: Visually salient but task-irrelevant later segments can override earlier evidence and contaminate memory retrieval.The analysis characterizes this as distractor-induced memory contamination in streaming multi-turn reasoning.