PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking in Streaming Video

Zikang Liu, Longteng Guo, Handong Li, Ru Zhen, Xingjian He, Ruyi Ji, Xiaoming Ren, Yanhao Zhang, Haonan Lu, Jing Liu

arXiv:2603.12938v1cs.CVcs.AI

TL;DR

Continuous video reasoning must operate causally with low latency and bounded computation instead of waiting for the full context. ThinkStream uses Watch–Think–Speak updates, Reasoning-Compressed Streaming Memory, and streaming RLVR to incrementally reason and time responses. It substantially outperforms existing online video models while maintaining low latency and memory usage over long streams.

  • Problem

    Existing video reasoning is predominantly batch-based, deferring reasoning until the full context is observed despite streaming requirements for causal, low-resource, timely understanding.

    Source: §1 · pp. 1–2

  • Method

    ThinkStream incrementally updates reasoning after each video chunk, uses reasoning traces as semantic memory while evicting outdated visual tokens, and trains response timing with streaming RLVR.

    Source: §1 · pp. 2–3

  • Results

    ThinkStream substantially outperforms existing online video models while maintaining low latency and memory usage over long streams and retaining competitive performance on offline video tasks.

    Source: §1 · p. 3, §7 · p. 15

  • Takeaways & Limitations

    Streaming video models can combine incremental reasoning, bounded memory, and timely interaction while preserving strong video-understanding performance.

    Source: §1 · p. 3, §7 · p. 15

Abstract

from arXiv · show

Real-time understanding of continuous video streams is essential for interactive assistants and multimodal agents operating in dynamic environments. However, most existing video reasoning approaches follow a batch paradigm that defers reasoning until the full video context is observed, resulting in high latency and growing computational cost that are incompatible with streaming scenarios. In this paper, we introduce ThinkStream, a framework for streaming video reasoning based on a Watch--Think--Speak paradigm that enables models to incrementally update their understanding as new video observations arrive. At each step, the model performs a short reasoning update and decides whether sufficient evidence has accumulated to produce a response. To support long-horizon streaming, we propose Reasoning-Compressed Streaming Memory (RCSM), which treats intermediate reasoning traces as compact semantic memory that replaces outdated visual tokens while preserving essential context. We further train the model using a Streaming Reinforcement Learning with Verifiable Rewards scheme that aligns incremental reasoning and response timing with the requirements of streaming interaction. Experiments on multiple streaming video benchmarks show that ThinkStream significantly outperforms existing online video models while maintaining low latency and memory usage. Code, models and data will be released at https://github.com/johncaged/ThinkStream

1 Introduction

Streaming video understanding must reason causally and efficiently as observations arrive, rather than waiting for a complete video. ThinkStream addresses this with incremental Watch–Think–Speak updates, compressed semantic memory, and verifiable-reward training for timely responses.

1 Introduction

  • Batch video reasoning defers multi-step inference until after consuming long contexts, causing latency and compute costs that conflict with continuous streaming.Streaming also requires strict causality, bounded resources, and timely interaction.Source: §1 · pp. 1–2
  • The Watch–Think–Speak paradigm updates understanding after each video chunk, then emits a response or remains silent until sufficient evidence arrives.Each update integrates new evidence with accumulated context and produces structured <think> and interaction actions.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • RCSM retains reasoning traces as semantic anchors while evicting outdated visual tokens, preserving historical context with stable inference cost over long streams.This replaces dense perceptual memory with compressed semantic memory as the stream exceeds the visual window.Source: §1 · p. 3
  • Streaming RLVR uses automatically verifiable rewards to train structured reasoning, evidence-sensitive response timing, and answer correctness throughout the stream.The framework is supported by a streaming rollout backend and time-grounded reasoning-trace data for cold-start and RLVR training.Source: §1 · pp. 3–4
  • ThinkStream substantially outperforms existing online video models while maintaining low latency and memory usage, including with a compact 3B-scale model.It also preserves strong performance on standard offline video benchmarks.Source: §1 · pp. 3–4

2 Related Work

Prior streaming video systems use direct perception-response cycles or visual memory compression, while video reasoning adaptations remain largely offline. ThinkStream targets the missing combination of incremental multimodal reasoning, semantic memory, and streaming interaction.

2 Related Work

  • Early online video frameworks rely on direct perception-response cycles without intermediate reasoning, while later methods manage visual token growth through memory, cache compression, or pruning.These memory mechanisms remain visual rather than providing semantic compression and explicit logical deduction.Source: §2.1 · p. 4
  • Video-domain Chain-of-Thought and reinforcement-learning methods remain confined to offline batch processing, while text-only streaming reasoning does not address concurrent visual think-and-respond demands.The related-work gap concerns multimodal streaming reasoning under dense visual-token bottlenecks.Source: §2.2 · p. 4

3 Streaming Watch–Think–Speak Paradigm

ThinkStream models streaming video understanding as a Watch–Think–Speak loop, incrementally updating reasoning with each video chunk and responding only when sufficient evidence has accumulated.

Watch–Think–Speak Loop

  • Each new video segment triggers a short reasoning update that integrates current evidence with previously accumulated understanding.The process supports summarizing events, updating activity hypotheses, and refining causal or temporal relations under strict latency constraints.Source: §3.1 · pp. 5–6
  • After each reasoning update, the model either emits a response when evidence is sufficient or remains silent and continues observing.The action space consists of silent or response actions, with response content attached to the latter.Source: §3.1 · p. 6, §3.2 · p. 6

Formal Definition

  • The model maintains a historical state summarizing prior context while observing only the stream prefix available at the current step.This formulation enforces strict streaming causality by conditioning each reasoning segment on observations available up to time t.Source: §3.2 · p. 6
  • Each streaming step outputs a structured reasoning segment followed by an interaction decision.The format is ⟨think⟩r_t⟨/think⟩a_t, where the action determines whether to respond or continue watching.Source: §3.2 · p. 6

4 ThinkStream

ThinkStream combines incremental streaming reasoning with Reasoning-Compressed Streaming Memory, RLVR training, and specialized inference to preserve long-horizon understanding under real-time resource constraints.

4 ThinkStream

  • ThinkStream addresses the central trade-off between coherent long-horizon understanding and real-time inference efficiency.Continuous video otherwise causes visual-token count, KV-cache memory, and attention complexity to grow with time.Source: §4 · pp. 6–7, §4.1 · pp. 7–8

4.1 Reasoning-Compressed Streaming Memory

  • Reasoning-Compressed Streaming Memory replaces outdated dense visual tokens with compact reasoning traces that preserve semantic context.Recent visual tokens remain available while accumulated reasoning tokens serve as long-term semantic anchors.Source: §4.1 · p. 7
  • The retained visual context is bounded by a sliding window, while reasoning tokens grow more slowly, keeping effective context length stable over long streams.When the stream exceeds window size W, the oldest visual chunk is evicted from the KV cache.Source: §4.1 · p. 7

4.2 Streaming-Context RLVR Training

  • Streaming RLVR uses accuracy, format, and time rewards to train structured reasoning, appropriate response timing, and answer correctness.The time reward penalizes both premature responses and excessive latency within a tolerance window.Source: §4.2 · pp. 8–9

Streaming Inference

  • A CUDA-Graph backend uses eager prefill for new visual tokens and replayable decode-and-prune execution for customized KV-cache updates.This design targets the high-throughput chunk-level decoding and cache manipulation required by RCSM.Source: §4.3 · p. 9

5 ThinkStream Dataset

The ThinkStream dataset is generated through timestamped video grounding, diverse instruction synthesis, and time-grounded chain-of-thought generation.

Video Segmentation and Dense Captioning

  • The pipeline segments videos into coherent scenes and densely captions each segment with timestamped semantic descriptions.These captions provide factual grounding for later instruction and reasoning generation.Source: §5 · pp. 9–10

Diverse Instruction Synthesis

  • Instructions span interaction modes, temporal scopes, and seven content-semantic categories formed through their Cartesian product.The dimensions include real-time dialogue, event triggers, continuous output, and past, current, or future temporal scopes.Source: §5 · p. 10

Time-Grounded CoT Generation

  • Time-grounded CoT generation constrains each thought and response to the timestamp associated with its evidence.The resulting traces are designed to form a dense, continuous stream of reasoning.Source: §5 · p. 10

Dataset Scale

  • The resulting corpus contains 110K cold-start instances and 9K RLVR instances with detailed, verifiable reasoning traces.The RLVR instances follow strict formatting for reward optimization.Source: §5 · p. 10

6 Experiments

ThinkStream is evaluated on streaming and offline video benchmarks, ablations, and efficiency profiles. It improves benchmark performance while maintaining bounded latency and memory usage for long streams.

Streaming Video Benchmarks

  • 59.66 on OVO-Bench and 75.00 on StreamingBench Real-Time show ThinkStream-3B outperforming open-source online baselines and competing with GPT-4o.ThinkStream-3B exceeds Qwen2.5-VL-3B at 51.00 and Streamo-3B at 51.64 on OVO-Bench, and exceeds Dispider-7B at 67.63 while approaching GPT-4o at 73.28 on StreamingBench Real-Time.Source: §6.1 · p. 11, Table 1 · p. 11, Table 2 · p. 12

Offline Video Benchmarks

  • 59.4 average performance across offline benchmarks exceeds Qwen2.5-VL-3B’s 54.4 while retaining 61.9 on VideoMME and 56.4 on Long VideoBench.The comparison indicates that aggressive visual-token eviction preserves competitive offline video understanding.Source: §6.1 · p. 11, Table 3 · p. 13

Ablation Study

  • 20 reasoning tokens per second improve OVO-Backward from 41.8 to 52.3, while larger budgets provide only marginal gains.The ablation identifies 20 tokens per second as the useful balance between reasoning capacity and efficiency.Source: §6.2 · p. 12, Table 4 · p. 13
  • A 20-second visual KV cache window peaks at 75.0 on StreamingBench Real-Time and 52.3 on OVO-Backward.Narrower windows reduce OVO-Backward scores to 46.7 and 50.1, while a 30-second window slightly lowers it to 51.6.Source: §6.2 · p. 12, Table 5 · p. 13

Efficiency and Real-Time Analysis

  • 154.07 tokens/s versus 30.06 at batch size 1 gives the custom engine more than 5× decoding speedup, while batch size 8 reaches 766.87 versus 276.10 tokens/s.The CUDA Graph backend preserves flexible KV-cache control for the eviction strategy.Source: §6.3 · pp. 13–14, Figure 3 · p. 14
  • Latency remains below the 0.5s real-time threshold as video length grows, whereas the baseline fluctuates between 1.0s and 1.4s per second of video.Algorithmic KV-cache eviction and engineering optimizations keep ThinkStream’s end-to-end latency flat for 2 FPS inputs.Source: §6.3 · p. 14, Figure 4 · p. 14

7 Conclusion

ThinkStream addresses streaming video reasoning under strict latency and memory constraints through Watch–Think–Speak, RCSM, and streaming RLVR. Experiments show strong streaming performance, low latency, bounded memory, and competitive offline capability.

7 Conclusion

  • ThinkStream combines Watch–Think–Speak, Reasoning-Compressed Streaming Memory, and streaming reinforcement learning with verifiable rewards for long-horizon video streams.The framework incrementally reasons over incoming video while maintaining bounded memory and coherent understanding.Source: §7 · p. 15

A More Implementation Details

The implementation uses CUDA Graphs and optimized attention and sampling components to support flexible KV-cache eviction with low decoding overhead. These choices target efficient training and real-time streaming deployment.

A More Implementation Details

  • Static CUDA graphs capture decoding and eviction operations, eliminating per-step kernel-launch overhead while preserving explicit KV-cache manipulation.The design supports flexible cache updates required by RCSM and high-throughput RL rollouts.Source: §A.1 · p. 19
  • The streaming algorithm evicts the oldest visual chunks once the active window reaches W, then pre-fills and decodes each new video chunk.The procedure uses captured eviction and decode kernels alongside eager execution for variable-length visual tokens.Source: §A.1 · p. 19
  • A static KV cache with flash-attn kvcache and FlashInfer top-k/top-p sampling reduces memory-access, compute, and token-selection latency.These components accelerate autoregressive generation within the streaming backend.Source: §A.1 · p. 19
  • FlexAttention enables customizable RCSM attention masks during training, while Liger Kernel reduces training time and peak GPU memory consumption.Together they support more efficient training over extended video sequences.Source: §A.2 · pp. 19–20

B Dataset Information

The ThinkStream dataset is built from diverse video sources and synthesized across interaction modes, temporal scopes, and content semantics. Its statistics show broad scenario coverage and varied reasoning-trace lengths.

B Dataset Information

  • The dataset samples dynamic video content from LLaVA-Video-178K and Tarsier2-Recap-585K.These subsets provide the visual foundation for streaming video understanding.Source: B Dataset Information · p. 20
  • Cartesian products of interaction modes, temporal scopes, and content semantics produce 39 meaningful scenario combinations.The combinations are filtered for practical validity before instruction synthesis.Source: B Dataset Information · p. 20
  • Interaction modes include Real-time Dialogue, Event Trigger, and Continuous Output, covering immediate answers and monitoring-style responses.Event Trigger scenarios require the assistant to remain silent until a specified event or prediction condition occurs.Source: B Dataset Information · p. 20
  • Temporal scopes cover Past, Current, and Future queries, requiring long-term recall, immediate-context understanding, or prediction.The scopes define the target time frame relative to the current stream timestamp.Source: B Dataset Information · p. 20
  • Seven fine-grained semantics span entity attributes, actions, spatial relations, causal reasoning, procedural states, global scenes, and OCR.This composition broadens the visual and logical behaviors represented in the dataset.Source: B Dataset Information · pp. 20–21
  • The dataset statistics show diversity across hierarchical scenario composition and thinking-token lengths.These distributions expose the model to varied streaming situations and reasoning-process durations.Source: B Dataset Information · p. 21, Figure 5 · p. 21

C More Results

ThinkStream-7B extends the framework to the Qwen2.5-VL-7B architecture and improves performance across OCR, real-time visual perception, and long-term memory retrieval on OVO-Bench.

  • 87.92 OCR score versus 67.79 for the base model, an improvement of over 20 points.Source: C More Results · p. 21
  • 69.12 versus 59.90 in Real-Time Visual Perception and 60.68 versus 52.28 in Backward Tracing.These results indicate gains in both real-time perception and long-term memory retrieval.Source: C More Results · p. 22

D Qualitative Analysis

The qualitative examples illustrate Watch-Think-Speak responding incrementally to event triggers and real-time queries, while the training prompts enforce time-grounded, concise reasoning and dialogue generation.

Qualitative Examples

  • For an “EXIT” trigger, the model outputs <silent> during irrelevant scenes and produces a <response> when the target appears.A separate time query is answered immediately from visual evidence.Source: D Qualitative Analysis · p. 22

Instruction Data Synthesis

  • The instruction-data prompt restricts generated dialogue to provided video descriptions and specifies timestamp-ordered JSON output.It also requires exactly one user message and constrains assistant-message length and spacing.Source: E Prompt Used · pp. 22–24

CoT Data Synthesis

  • The CoT synthesis prompt enforces temporal causality by allowing each thought to reference only captions or messages at or before its timestamp.It also requires at least two seconds between consecutive thoughts and continued thinking when no question has been asked.Source: E Prompt Used · p. 24

Related papers

Loading 2603.12938v1…