PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, Xiang Bai

arXiv:2603.12262v2cs.CV

TL;DR

Online VideoLLMs need explicit reasoning under causal, real-time, and limited-context constraints, but post-query Chain-of-Thought can add substantial latency. VST reasons over incoming clips while watching, supported by causal post-training and knowledge-graph-based supervision. It achieves strong performance across online and offline benchmarks while reducing QA latency relative to offline CoT approaches.

  • Problem

    Online video understanding requires strict temporal causality, real-time processing, and finite context, while existing streaming methods lack explicit analytical reasoning.

  • Method

    VST interleaves proactive reasoning with video consumption and uses VST-SFT, VST-RL, and knowledge-graph-based data synthesis to adapt VideoLLMs for causal streaming reasoning.

  • Results

    VST achieves strong performance across multiple online and offline video understanding benchmarks and significantly reduces QA latency compared with offline CoT VideoLLMs.

  • Takeaways & Limitations

    VST provides a practical streaming test-time scaling approach that combines logical inference with real-time responsiveness across model scales.

  • Takeaways & Limitations

    Streaming thoughts add non-negligible LLM token consumption, while VST primarily focuses on text-guided memory management rather than combining it with visual memory mechanisms.

Abstract

from arXiv · show

Online Video Large Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose Video Streaming Thinking (VST), a novel paradigm for streaming video understanding. It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM reasoning latency over video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence reasoning and sustained attention to the video stream. Extensive evaluations show that VST-7B performs strongly on online benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or reasoning benchmarks. Compared with Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse video understanding tasks. Code, data, and models will be released at https://github.com/1ranGuan/VST.

1 Introduction

Video Streaming Thinking (VST) addresses the tension between explicit reasoning and real-time responsiveness in online video understanding by reasoning during video consumption. It combines streaming reasoning, specialized post-training, and knowledge-graph-based data synthesis, achieving strong online and offline benchmark performance with lower QA latency.

  • Online video understanding must satisfy strict temporal causality, real-time processing, and finite context constraints.
  • Existing streaming VideoLLMs improve visual-context efficiency but provide limited LLM involvement and no explicit analytical reasoning.
  • VST interleaves explicit Chain-of-Thought generation with incoming video, amortizing reasoning cost before queries arrive while preserving responsiveness.
  • VST-SFT and VST-RL adapt an offline VideoLLM to causal streaming reasoning through a dedicated post-training pipeline.
  • A knowledge-graph-based synthesis pipeline generates streaming QA pairs and intermediate reasoning traces from temporally grounded evidence chains.
  • Extensive evaluations show state-of-the-art online performance, competitiveness offline, and significantly lower QA latency than offline CoT VideoLLMs.

2 Method

VST interleaves streaming video clips with autoregressive thoughts before a query, using dual memory to preserve temporal context and amortize reasoning cost.

  • Streaming inference: The system maintains short-term visual memory for current context and long-term textual memory for past events.The textual memory is updated recursively with a FIFO eviction strategy.
  • Streaming inference: VST processes incoming video clips incrementally, generating a streaming thought from each current clip and accumulated memory.The model samples z_k conditioned on c_k and m_{k−1}.
  • Query-time response: When a query arrives, VST generates the final response from the accumulated streaming thoughts, current clip, and textual memory.Reasoning continues through step K before the user query triggers the answer.
  • Design rationale: VST amortizes Chain-of-Thought computation before interaction and aligns sequential thought generation with streaming video causality.This design provides test-time scaling without adding latency at user interaction.
  • Training pipeline: The method combines VST-SFT and VST-RL to progressively adapt an offline VideoLLM to streaming thinking.The pipeline uses supervised adaptation followed by reinforcement learning.

Stage 1: VST-SFT.

VST-SFT adapts offline VideoLLMs to causal streaming reasoning by training on interleaved clip-thought sequences under a fixed visual attention window.

  • Training sequence: VST-SFT represents training examples as interleaved video clips and streaming thoughts, ending with a query and ground-truth response.The sequence begins with initial memory and concludes with the final clip, query, and answer.
  • Causal attention: The streaming attention mask restricts visual attention to the latest L visual tokens while preserving access to non-visual tokens.The mask mirrors the fixed-size visual buffer used during inference.
  • Long-video handling: Temporal segmentation slices long-form sequences into consecutive segments and recursively carries memory across segment boundaries.The memory state is updated using the streaming thoughts generated within each segment.
  • Optimization: SFT applies next-token prediction exclusively to streaming thoughts and the final response, conditioning on visual tokens and historical memory.Visual tokens and memory serve as inputs rather than prediction targets.

Stage 2: VST-RL.

VST-RL uses on-policy agentic rollouts and verifiable final-answer rewards to improve streaming thoughts across complete video-interaction trajectories.

  • Optimization: VST-RL transitions from off-policy imitation to on-policy self-improvement through trajectory rollout and policy-gradient optimization.The two phases form the core reinforcement-learning procedure.
  • Trajectory rollout: The rollout agent interacts with the streaming environment, sequentially sampling streaming thoughts and the final response from the policy.Each trajectory follows the joint thought-and-answer generation process.
  • Reward assignment: Rewards are computed solely from the final answer, while the resulting advantage is assigned to every generated token in the trajectory.This propagates answer-based supervision across both thoughts and response tokens.
  • Optimization: GRPO optimization uses clipped probability ratios and a KL penalty relative to a reference policy.The objective averages token-level clipped losses across sampled trajectories.

2.3 Data Synthesis Pipeline for VST

The VST data pipeline builds temporally consistent video knowledge graphs, samples evidence chains, and synthesizes filtered streaming QA examples with grounded reasoning.

  • Knowledge-graph construction: The pipeline extracts entities and temporal relations from sliding video windows to construct a timeline-aligned knowledge graph.An entity bank retains recent overlapping clips to preserve temporal continuity.
  • Evidence-chain sampling: DFS over the completed graph extracts evidence chains containing entities, relations, timestamps, and scene descriptions.These chains are designed to capture long-term causal dependencies.
  • QA synthesis: Gemini generates streaming rationales, queries, and final answers conditioned on the video knowledge graph and sampled evidence chains.The synthesis requires multi-evidence reasoning across visual events and relations.
  • Quality control: Automatic filtering checks world knowledge, format alignment, logical consistency, repetition, and thought validity.Filtering is applied after generation to improve data fidelity.
  • Dataset scale: The training corpus includes 100K streaming-thought examples, 50K open-ended SFT instances, and 11K questions for VST-RL.Videos and questions are drawn from LLaVA-Vid, Video-Marathon, Onethinker, and RepCount.

3 Experiment

VST is evaluated across online and offline video benchmarks, training ablations, model scales, latency, and failure cases. It achieves strong benchmark performance while reducing post-query latency, with remaining errors centered on memory construction.

  • Online Video Benchmark Results: 79.5% on StreamingBench and 59.3% on OVO-Bench, outperforming prior open-source streaming models.VST-7B also surpasses GPT-4o and Gemini 1.5 Pro on StreamingBench and matches GPT-4o on OVO-Bench.
  • Offline Video Benchmark Results: 55.3% on VideoMME-long, 58.0% on LongVideoBench, and 41.9% on VideoHolmes, exceeding TimeChat-Online or Video-R1 on each reported comparison.The reported gains are +6.9% on VideoMME-long, +2.6% on LongVideoBench, and +5.4% on VideoHolmes.
  • Ablation on Training Schedule: +4.8% on OVO-Bench from mixing 20K LLaVA-Vid and 30K VST data, while combining VST-SFT and VST-RL reaches 59.3% on OVO-Bench and 64.9% on VideoMME.VST-SFT primarily improves backward memory capacity by +9.2%, whereas VST-RL improves forward prediction by +12.7%.
  • Ablation on Base Model Size: +7.7%, +7.8%, and +9.2% on StreamingBench over 3B, 7B, and 32B base models, respectively.VideoHolmes gains are +5.4%, +9.0%, and +5.0%, demonstrating consistent improvements across model sizes.
  • Inference Latency: VST adds no post-query response latency because asynchronous streaming thoughts finish within the clip inter-arrival interval and are amortized over playback.Streaming thoughts take 7.0 seconds on average, with 11.2 seconds P99 latency, and the latest completed memory state is used when interrupted.
  • Failure Cases: VST errors mainly arise from imperfect memory construction, including irrelevant details, overly aggressive compression, missed temporal spans, and weak cross-event associations.The paper identifies finer and more faithful memory construction as a direction for future work.

4 Related Work

Streaming video understanding must operate under strict temporal causality, real-time processing, and limited context. Existing approaches mainly manage visual-token memory, while streaming logical reasoning remains underexplored.

  • Streaming Video Understanding: Online video understanding processes continuous inputs without global video access, making temporal causality, real-time processing, and finite context central challenges.Offline methods can access the entire video post hoc, unlike online systems.
  • Post-Training Research: Recent SFT and R1-style RL approaches improve VideoLLM reasoning, but post-training research remains predominantly focused on offline video understanding.Reasoning in streaming contexts, particularly for long-horizon understanding, remains insufficiently explored.

5 Conclusion

VST introduces synchronized logical inference during streaming and reports strong performance across online and offline video-understanding benchmarks. Its main limitation is that streaming thoughts still consume additional LLM tokens, while combining text-guided memory with visual memory remains future work.

  • VST synchronizes logical inference with real-time responsiveness through thinking while watching incoming video clips.The paradigm includes VST-SFT, VST-RL, and automated video-knowledge-graph-based data synthesis.
  • VST delivers strong performance across multiple online and offline video-understanding benchmarks and generalizes across VideoLLMs from 3B to 32B parameters.
  • Additional LLM token consumption remains non-negligible despite scheduling streaming-thought computation alongside incoming video clips.
  • Combining VST’s text-guided memory management with existing streaming visual-memory mechanisms is identified as future work.

6 Details of VST Inference

VST performs intermittent streaming inference for each incoming video clip before the user query arrives, then generates the final answer from accumulated memory. This scheduling uses video playback time to reduce query-to-response latency.

  • 6. Details of VST Inference: VST generates intermittent streaming thoughts as video clips arrive, conditioning them on prior memory and the current clip.The memory contains historical streaming outputs, while the video clip denotes the incoming stream.
  • 6. Details of VST Inference: After receiving a user query, VST generates the final answer using the accumulated memory and current video context.
  • 6. Details of VST Inference: Streaming thinking is completed before the subsequent clip arrives, using the natural waiting time in real-world video streams.
  • 6. Details of VST Inference: VST reduces QA latency by performing reasoning before query submission rather than deferring the computation until the query arrives.QA latency is measured from user-query submission to the LLM response.

7 Details of VST-SFT / RL Training

VST training uses separate supervised fine-tuning and reinforcement-learning stages with reported settings for video-token capacity, optimization, prompting, and rollout generation.

  • 7. Details of VST-SFT / RL Training: VST-SFT samples up to 384 frames, caps video tokens at 24K, and reserves 8K context for language and reasoning.The stage uses 1 epoch, a 5e-6 learning rate, and 8 gradient accumulation steps.
  • 7. Details of VST-SFT / RL Training: VST-RL uses GRPO with a maximum 11,000-token prompt, 1,000 tokens reserved for generated responses, and 8 candidate responses per prompt.Rollouts use vLLM with temperature 1.0 and top-p 0.98.
  • 7. Details of VST-SFT / RL Training: VST-RL is trained for 1 epoch with global batch size 256 and PPO mini-batch size 64.The passage also reports a 5e-7 learning rate.

8 Details of VST Data Generation

VST’s data-generation pipeline uses structured prompts to create video knowledge graphs, intermediate chain-of-thought reasoning, and question-answer annotations from video segments.

  • 8. Details of VST Data Generation: The data-generation process produces video knowledge graphs, intermediate chain-of-thoughts, and question-answer pairs through three distinct prompts.The templates are applied consistently with segment-specific content substituted into curly-brace placeholders.
  • 8. Details of VST Data Generation: Video knowledge graph generation maps physical relationships and object states from the current video segment and known entities.
  • 8. Details of VST Data Generation: Intermediate chain-of-thought and question-answer generation use the structured outputs of the data-synthesis pipeline to produce streaming supervision.
Loading 2603.12262v2…