PaperlayerView exact revision on arXiv

Source-linked AI summary

StreamScout: Learning When to Look Deeper for Streaming Video Understanding

Ce Zhang, Jing Bi, Jinxi He, Jianshu Zhang, Jingyang Lin, Yunzhong Xiao, Minghao Fu, Yaqi Xie, Zhentao Xie, Weicong Chen, Katia Sycara, Ming Zhou

arXiv:2609.00291v1cs.CV

TL;DR

Streaming video systems typically access bounded memory with the same fixed-cost procedure despite questions requiring different amounts of evidence. StreamScout adaptively escalates through increasingly informative visual views, learns its stop policy through self-distillation and reinforcement learning, and consistently improves performance while reducing inference cost across benchmarks and backbones.

  • Problem

    Streaming systems must answer questions over unbounded streams while deciding both what information to retain and how deeply to inspect it for each query.

  • Method

    StreamScout maintains a lightweight textual timeline and adaptively stops or escalates across recent-frame, uniform-look-back, and query-salient visual views, with StreamScout-S and StreamScout-R learning the decision policy.

  • Results

    StreamScout variants consistently surpass prior streaming systems across three benchmarks and three backbones; StreamScout-S improves Qwen3-VL-8B by 14.65 points on OVO-Bench while using 59% fewer tokens than uniform sampling.

  • Takeaways & Limitations

    Query-adaptive evidence acquisition improves streaming video understanding while substantially reducing token consumption and answer latency.

  • Takeaways & Limitations

    The textual timeline and cached frame embeddings grow linearly with stream duration, so truly unbounded streams require future consolidation or eviction of old entries.

Abstract

from arXiv · show

Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access memory for each query is as important as deciding what the memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer tokens than uniform sampling and answering in 1.04 s on average.

1. Introduction

Streaming video systems must answer questions over unbounded streams while deciding both what to retain and how deeply to inspect evidence. StreamScout addresses the overlooked access decision with an adaptive visual cascade, and trained variants improve accuracy while reducing inference cost.

  • Motivation: Unbounded streams and arbitrary query times require systems to compress incoming video into bounded memory without knowing which details will later matter.At 0.5 fps, one hour produces 1,800 frames, and the stream may continue indefinitely.
  • Motivation: Prior systems primarily optimize what to retain, but read their memories with fixed context, retrieval rules, and costs despite unequal query demands.Recent questions may need only a few frames, whereas distant fine-grained details require broader access.
  • StreamScout: StreamScout maintains a lightweight textual timeline and progressively adds recent-frame, uniform look-back, and query-salient retrieval views at query time.The model answers when evidence suffices or emits an escalation decision for deeper inspection.
  • StreamScout: The zero-shot cascade allocates visual evidence according to query difficulty, using inexpensive inspection for easy questions and deeper access only when needed.The framework explicitly makes query-time evidence acquisition a decision of the answering model.
  • Results: StreamScout-S and StreamScout-R use roughly 40% of the tokens and one quarter of the latency of uniform sampling while consistently surpassing prior streaming systems.The reported comparison spans OVO-Bench, StreamingBench, and StreamBench, and includes three widely used backbones.
  • Results: Up to 20.26 percentage points of accuracy improvement, 59% lower token consumption, and 72% lower answer latency are reported across three benchmarks and three backbones.The trained variants sharpen the stop decision, with StreamScout-S using self-distilled supervision and StreamScout-R using reinforcement learning.

2. Related Work

Prior streaming video work improves model training or memory design, but typically uses fixed-cost memory access. StreamScout instead makes query-dependent inspection depth explicit through a progressively richer evidence cascade.

  • Long-Form Video Understanding: Long-form video methods compress frames into sparse tokens, visual summaries, or adaptive selections to fit inputs exceeding current multimodal context limits.Examples include latent memory merging, visual summary distillation, and spatiotemporal redundancy removal.
  • StreamScout: StreamScout operationalizes adaptive access with a cascade from recent-frame glance to uniform look-back and query-salient retrieval.Each stage either commits to an answer or requests more evidence.
  • Streaming Video Understanding: Streaming benchmarks evaluate unbounded video with questions arriving at arbitrary timestamps, including real-time perception, backward tracing, and proactive responding.Existing approaches either train streaming behavior or design memories such as KV caches, episodic-semantic memories, and adaptive updates.
  • Streaming Video Understanding: Despite varied memory designs, existing systems generally read memory at the same cost for every question, leaving query-dependent inspection depth largely overlooked.OASIS provides a related coarse-to-fine pipeline, but its evidence stages and transition policy are manually specified.

3. Method

StreamScout separates lightweight stream maintenance from query-conditioned visual inspection, progressively adding visual evidence only when needed. StreamScout-S learns stopping labels from the earliest successful view, while StreamScout-R further optimizes the trajectory-level correctness–cost trade-off with reinforcement learning.

  • StreamScout: StreamScout maintains a timestamped textual timeline and bounded visual cache while processing the stream in one pass.The cache stores at most 64 raw frames with embeddings and evicts frames from temporally dense regions when full.
  • StreamScout: At query time, StreamScout progressively augments the timeline with three nested views: recent frames, uniform look-back, and query-salient retrieval.Escalation strictly adds evidence because each frame set is a subset of the next.
  • StreamScout: The woven context inserts selected frames at their timestamps, keeping visual evidence aligned with the textual timeline and observed history.The model either answers from the current context or emits an escalation marker requesting the next view.
  • StreamScout-S: StreamScout-S probes every cascade stage on an auxiliary labeled set and labels each question by its earliest successful view.Questions for which no view succeeds are discarded, preventing supervision from rewarding unsupported commitments.
  • StreamScout-S: The StreamScout-S LoRA is trained only on stop-or-escalate decisions, with answer-content tokens masked from the loss.Ground-truth answers determine probe outcomes but are not used as answer-generation targets.
  • StreamScout-R: StreamScout-R initializes from StreamScout-S and uses reinforcement learning to optimize correctness against escalation cost at the trajectory level.Its reward penalizes wrong commitments and charges each escalation through λ; GRPO updates retained trajectory groups, while StreamScout-S serves as the reference policy.

4. Experiments

Experiments across three backbones and streaming benchmarks show that StreamScout and its trained variants outperform prior streaming systems, while adaptive evidence acquisition reduces inference cost. Ablations and offline tests examine view complementarity and transfer.

  • Experimental setup: Three backbones and three streaming benchmarks evaluate StreamScout against uniform sampling and representative streaming systems.The evaluation uses Qwen3-VL-8B, Qwen2.5-VL-7B, and LLaVA-OneVision-2-8B across OVO-Bench, StreamingBench, and StreamBench.
  • Streaming benchmarks: 72.21 and 76.63 are StreamScout-R’s best overall OVO-Bench accuracies, exceeding GPT-5.1 with 8B-scale models.StreamScout-S adds 7.15, 5.04, and 4.97 points across the three backbones before reinforcement learning further raises the best overall accuracy.
  • Streaming benchmarks: 72.7 and 76.5 are StreamScout-R’s best StreamingBench averages on Qwen2.5-VL-7B and Qwen3-VL-8B, with Qwen3-VL exceeding GPT-5.1.The zero-shot cascade raises question-weighted averages to 71.3 and 71.9, while ACU improves by up to 24.8 points.
  • Streaming benchmarks: 61.5 and 66.1 are StreamScout-R’s StreamBench averages with Qwen2.5-VL-7B and Qwen3-VL-8B, surpassing OASIS and other streaming systems.StreamBench uses free-form answers scored by an LLM judge, testing transfer beyond multiple-choice benchmarks.
  • Ablations: 69.69 is the best OVO-Bench accuracy from combining glance, look-back, and retrieval views.Glance provides the strongest perception foundation, while retrieval contributes most to backward tracing by recovering distant details.
  • Efficiency: 1.04 s and 6,559 tokens are StreamScout-S’s average OVO-Bench costs, with 59% fewer tokens than uniform sampling.A full three-view traversal costs 2.77 s, while stream-time maintenance uses roughly 17.6 s of GPU time per streamed minute.

5. Conclusion

StreamScout maintains a lightweight textual timeline and adaptively grounds it in progressively richer visual views, stopping when evidence suffices. Across streaming benchmarks, this design improves performance while reducing token use.

  • Conclusion: StreamScout progressively adds richer visual views to a lightweight textual timeline and stops when the available evidence suffices.StreamScout-S distills the model’s empirical competence boundary, while StreamScout-R refines stopping through reinforcement learning.
  • Conclusion: Figure 3 compares per-question latency and token consumption for the zero-shot cascade, StreamScout-S, and StreamScout-R.Bars show worst-case full-pipeline costs by stage, while vertical pins show averages across questions on a single A100.
  • Conclusion: Query-adaptive evidence acquisition offers a favorable accuracy–efficiency trade-off for streaming video understanding.The framework and variants consistently surpass prior streaming systems while consuming substantially fewer tokens per question.

Appendix

The appendix provides additional experimental results and implementation details for StreamScout.

  • Appendix: The appendix contains additional experimental results and implementation details of StreamScout.No further appendix-specific findings are supplied here.

A.1. Qualitative Examples

StreamScout adapts evidence acquisition to query difficulty: it stops after a recent-frame glance when the answer is visually evident, but escalates through richer views when earlier evidence is needed.

  • Questions about earlier events can require escalation from the glance and uniform look-back to query-salient retrieval, which locates relevant frames.The qualitative example shows the model progressively escalating before producing the correct answer.
  • Visually evident questions can be answered correctly from the recent-frame glance, avoiding unnecessary retrieval.

A.2. More Discussions

The cascade’s stopping behavior reveals a correctness–cost trade-off shaped by training, question source, and evidence layout. Woven visual evidence improves both accuracy and efficiency, while naive stop-decision reinforcement learning can collapse toward shallow answers.

  • Stopping Layers: 82.4% of OVO-Bench questions resolve at the glance with StreamScout-S, versus 47.6% training-free and 74.5% for StreamScout-R.Retrieval usage is 16.8% for -S and 23.4% for -R, compared with 46.2% training-free.
  • Stopping Layers: StreamScout-R reopens escalation where extra visual evidence improves correctness, gaining accuracy over StreamScout-S at a modest cost increase.StreamScout-S follows distilled shallowest-sufficient labels, whereas -R optimizes correctness rather than label matching.
  • Question-source adaptation: 89.0% of StreamingBench questions resolve at the glance for StreamScout-R, compared with 35.0% retrieval escalation on MLVU.The same policy escalates for 20.3% on StreamBench and 35.0% on MLVU, reflecting different question demands.
  • Woven Views: 67.6 to 69.7 accuracy improvement results from weaving frames into the timeline rather than appending them as a chronological block.The woven layout also reduces retrieval escalation from 54% to 46% and generations from 2.07 to 1.99.
  • RL stability: Naive GRPO stop-decision training collapses to always answering at the glance, reducing on-policy accuracy from 0.69 to 0.51 within 24 steps.KL anchoring and a wrong-commit penalty only delay the collapse, while degenerate question groups distort the advantage signal.

B.1. Training Details

Training constructs a timestamped textual timeline and applies a three-view cascade whose visual coverage increases from recent frames to uniform history sampling and query-salient retrieval. The prompts require either answering or explicitly requesting the next view.

  • Streaming Memory: The backbone captions consecutive 16-second stream windows to form the textual timeline, without an auxiliary captioner.The timeline is maintained as the stream unfolds and supports later query-time visual grounding.
  • Cascade views: View 1 weaves 4 recent frames into the timeline, View 2 adds up to 8 frames at 32-second stride, and View 3 adds 12 CLIP-retrieved salient frames.The views progressively expand visual coverage while retaining the same textual timeline.
  • RL training: StreamScout-R trains a fresh same-configuration LoRA with GRPO using 8 trajectories per question, batches of 40 questions, 300 steps, and learning rate 5 × 10^-6.The merged -S adapter is used in the backbone, with a frozen KL reference and cached views reused during rollouts.
  • QA-summary folding: The folding prompt updates the QA summary using only the existing summary and new answer, preserving key changes while removing duplicates under a 300-word limit.
  • Decision policy: Each non-final stage either emits an answer or requests more context, while the final richest view requires a best-effort answer.Escalation replaces the prior visual view rather than retaining all previous views.
  • Prompt behavior: The prompts use captions for history, temporal, and counting reasoning while woven frames verify the captions’ visual claims.Open-ended prompts permit complete answers using reasoning and general knowledge, whereas multiple-choice prompts require a single option letter.

C. Limitations and Broader Impact

The paper identifies boundaries in when StreamScout acts, how long its memory can grow, and what its backbone can perceive. It also connects adaptive perception to lower deployment cost.

  • Limitations: StreamScout decides how deeply to inspect after a question arrives but does not decide when to speak.Adding response initiation as a third action is left as future work.
  • Limitations: The textual timeline and cached frame embeddings grow linearly with stream duration, so truly unbounded streams require consolidation or eviction.The reported benchmarks cap videos at tens of minutes; handling longer streams is left to future work.
  • Limitations: Self-distilled supervision is bounded by backbone competence because questions unanswered from every view yield no stop label.StreamScout-S and -R therefore allocate existing perception rather than expand what the backbone can perceive.
  • Broader Impact: Adaptive perception reduces tokens and latency, lowering deployment energy costs and making real-time assistants on wearable or embedded devices more practical.The paper also identifies accessibility applications such as live scene description.
Loading 2609.00291v1…