StreamVibe: Proactive Video Agents for Evolving User Intent

1. Introduction

• StreamVibe-Bench provides 4.1K evaluation and 85K training samples spanning explicit, implicit, and dynamic intents.Source: §1 · p. 2
• StreamVibe-VL jointly learns when and how to assist under implicit and evolving user intents.Source: §1 · p. 2
• StreamVibe-Harness maintains updatable intent states with flexible lifecycles and hierarchical memory for sustained interaction.Source: §1 · p. 2

Streaming Video Benchmark

• Earlier online and proactive benchmarks often focus on one intent type or environment and evaluate responses independently; StreamVibe-Bench covers three intent types across physical and digital tasks with pointwise and trajectory evaluation.Source: §2 · p. 3

Proactive Interaction Models

• Most proactive systems center on explicit requests or objectives, with limited support for tracking interaction state and adapting to evolving intent over time (Tao et al., 2026).Source: §2 · p. 3

Memory and Agent Harness

• Memory systems organize or retrieve historical information, while StreamVibe-Harness explicitly tracks evolving user intents and task progress as updatable states.Source: §2 · p. 3

3.1. Task Formulation

• At each timestep, the agent uses only past and current video, available intents, and prior history to choose silence or a response; intents remain active until their evidence points are covered.Responses must occur within valid evidence windows and avoid premature, missed, repeated, or irrelevant assistance.Source: §3.1 · p. 4

3.2. Hierarchical Difficulty

• The benchmark distinguishes explicit requests, implicit high-level goals requiring inference from observations and task progress, and dynamic intents that may be revised, composed, or interrupted.Source: §3.2 · p. 4

3.3. Benchmark Construction

• The benchmark collects physical activity videos and digital interaction trajectories, then verifies response windows, reference answers, and timestamped multi-turn dynamic-intent annotations with four reviewers.Source: §3.3 · p. 5

3.4. Data Statistics

• StreamVibe-Bench contains 4,100 evaluation samples, 20,546 help points, and 183.4 hours of video; StreamVibe-85K includes 70K SFT samples and 15K RL tasks.SFT also incorporates 768K general streaming-video samples.Source: §3.4 · p. 5

3.5. Evaluation Protocol

• Pointwise evaluation tests independent decisions under Gold or Self History, while trajectory evaluation runs continuously to measure error propagation, requirement tracking, and repeated or missed responses.Timing uses Precision, Recall, F1, and FPR; answer quality uses Answer Score and Content F1.Source: §3.5 · pp. 5–6

4. StreamVibe: Grounding Evidence, Tracking Intent

• StreamVibe-Harness tracks evolving intents and task progress, while the policy grounds response decisions in emerging visual evidence; the Harness also supplies persistent state and retrieved history as context.This separates state tracking from response decisions and generation.Source: §4 · p. 6, Figure 3 · p. 7

4.1. StreamVibe-Harness Framework

• The Harness updates an explicit interaction state from supported evidence, tracking active intents, task and environment progress, and previously delivered responses.Each intent has an independent lifecycle; persistent intents can remain active across events, while one-time intents close once satisfied.Source: §4.1 · p. 6
• Hierarchical memory retains recent information in detail, compresses older history, and retrieves relevant context for each decision before writing the resulting interaction back.Retrieved history supplements the current decision without directly modifying persistent state.Source: §4.1 · p. 6

4.2. Evidence-Grounded Policy Learning for Streaming Interaction

• Adaptive streaming SFT adjusts decision-token weights to the training distribution while separately optimizing response generation, balancing silence and response patterns.The objective distinguishes response and silence decision classes and preserves response-generation learning.Source: §4.2 · pp. 6–7
• Localized trajectory RL uses short causal rollouts around help points, matching responses by timing and judged content while balancing missed evidence against over-response and rewarding sequence coherence.Its global reward also discourages redundant or conflicting responses; the final reward weights alignment at 0.7.Source: §4.2 · pp. 7–8

5.1. Setup

• The experiments evaluate proactive interaction on StreamVibe-Bench and external timing benchmarks, alongside general video understanding benchmarks, using official protocols for external suites.StreamVibe-Bench uses Claude Sonnet 4.6 as a shared judge and scores response quality on a 0–4 scale.Source: §5.1 · p. 8

5.2. Analysis

• StreamVibe-VL consistently outperforms open-source baselines on StreamVibe-Bench, with RL improving Content F1 and response quality under trajectory evaluation.It also achieves the highest Content F1 in Pointwise evaluation while remaining competitive with proprietary models in response quality; gains transfer to external benchmarks.Source: §5.2 · p. 8, Table 2 · p. 9
• Implicit and Dynamic Intents are harder than Explicit Intents; higher Recall often coincides with more false triggers, and sustained interaction worsens these challenges.Timing F1 declines with interaction length; Revision has a sharper late-stage Recall drop, while Composition has weaker content quality.Source: §5.2 · p. 9, Figure 5 · p. 10, Figure 6 · p. 10
• With the policy fixed, adding the Harness suppresses unnecessary responses and improves content quality in most settings, with clearest gains on Dynamic Intents.SFT raises Recall but also unnecessary triggers; RL improves selectivity by reducing FPR while improving answer quality.Source: §5.2 · p. 10, Figure 6 · p. 10
• In representative physical and digital interactions, StreamVibe-VL grounds assistance in current visual evidence and task state, while streaming baselines can respond prematurely or fail to update.The cases show response timing and content remaining consistent across sustained interaction.Source: §5.2 · pp. 10–11, Figure 7 · p. 11

6. Conclusion

• StreamVibe-Bench covers explicit, implicit, and dynamic intents with pointwise and trajectory evaluation.The paper also presents StreamVibe-VL, trained with adaptive streaming SFT and localized trajectory RL, and StreamVibe-Harness for persistent intent and task-progress states.Source: §6 · p. 11
• Experiments show improved response timing and quality while preserving general video understanding; explicit state tracking further reduces outdated and redundant responses..Source: §6 · p. 11
• The current setting does not capture richer co-evolving interactions in which observations, outcomes, and prior assistance jointly reshape intents and the environment.The paper identifies complex intent–environment feedback over longer horizons as future work.Source: §6 · p. 11

A.1. Results Under the Three Evaluation Protocols

• Supplementary model-comparison tables cover Pointwise evaluation with gold or self-generated history and Trajectory evaluation with model-generated history.The appendix reports matching tolerances of ±2, ±1, and ±0 seconds; tighter tolerances rescore the same predictions.Source: A. Additional Evaluation Results · p. 16, §A.1 · p. 16

A.2. General Timing and External Benchmarks

• The additional external-benchmark results include general timing, EgoProactive, IPIBench, and OVO-Bench, each using its stated benchmark protocol.The supplied table captions identify results on EgoProactive and IPIBench, OVO-Bench, and general timing.Source: §A.2 · p. 18, Table 11 · p. 19, Table 10 · p. 19

A.3. StreamVibe-Harness

• Harness comparisons report Trajectory results on StreamVibe-Bench and compare with and without the Harness on general timing, IPIBench, and EgoProactive.The external-benchmark comparison also includes a matched SFT pair with and without the Harness.Source: §A.3 · p. 20, Table 13 · p. 20, Table 14 · p. 20

A.4. Category Difficulty Across Intent Types

• The analysis examines subtype difficulty, performance across evaluation settings, and RL-related shifts in response behavior and trade-offs.Figures 8–10 report subtype results under Pointwise and Trajectory evaluation with a ±2 s matching tolerance.Source: §A.4 · p. 20, Figure 8 · p. 21, Figure 9 · p. 22
• Trajectory evaluation is markedly harder across subtypes: Timing F1 drops from 63.5–75.6 in Pointwise evaluation to 12.1–28.2, while Trajectory Content-F1 is 2.7–11.1.The authors relate sustained performance to updating active intents, tracking task state, and regulating responses using evolving interaction history.Source: §A.4 · p. 21
• Subtype difficulty shifts under sustained interaction: Explicit I has the highest Pointwise Timing F1 at 75.6 but the lowest Trajectory Timing F1 at 14.5, while R reaches 28.1.Guidance is more challenging than Tool/GUI among Implicit intents; Revision and Composition underperform Interruption among Dynamic intents.Source: §A.4 · p. 21
• StreamVibe-VL performs best on Tool/GUI, with Timing F1 37.9 and Content-F1 13.0, while Monitoring scores 17.2 and 4.1, respectively.For Dynamic intents, Revision and Composition have Timing F1 around 10, versus 19.0 for Interruption.Source: §A.4 · p. 21
• RL improves Timing F1 and Content-F1 while reducing FPRnear across subtypes; P has the largest Timing F1 gain (+7.2), and Monitoring the largest FPR reduction (−15.3).The authors describe this as more selective intervention, particularly in settings prone to over-triggering.Source: §A.4 · p. 22, Figure 10 · p. 22

B.1. Overview of Evaluation Datasets

• StreamVibe-Bench evaluates sustained interaction across 4,100 samples and ten subtypes, using pointwise decisions and trajectories with model-generated history.The subtypes range from specified events to inferred assistance and revised or interleaved requests.Source: §B.1 · p. 23
• The evaluation also tests response timing and content, egocentric procedural assistance, evolving interaction requirements, and preservation of general temporal video understanding.General-Timing is held out from training; EgoProactive and IPIBench use their official protocols, as do MVBench, Video-MME, OVO-Bench, and OVBench.Source: §B.1 · p. 23

B.2. Task Taxonomy and Evaluation Sources

• Each example pairs a stream and initial instruction with optional timestamped turns and evidence points specifying an intervention window, preferred time, and gold response.Examples are serialized at 1 FPS, expose only observations available by each time, and undergo review for request fidelity, visible evidence, timing, and assistance content.Source: §B.2 · pp. 24–25

B.3. Human Review Criteria and Inter-Annotator Agreement

• Samples are retained only after all four reviewers approve the same final version; unresolved ambiguity or insufficient visual evidence requires exclusion rather than a guessed answer.The agreement audit uses original independent decisions before discussion, correction, or acceptance filtering.Source: §B.3 · pp. 25–26
• Human–human Fleiss’ κ ranges from 0.82 to 0.87, while Gemini–human Cohen’s κ ranges from 0.73 to 0.79, supporting the reported annotation-pipeline reliability.Table 17 reports the agreement measures; Dynamic has the lowest human–human agreement at 0.82 and Explicit the highest at 0.87.Source: Table 17 · p. 26, §B.3 · p. 26

B.4. Training Mixture and Evaluation Cohorts

• The training mixture combines 70K task-aligned SFT examples with 768K general-streaming examples, alongside a separate 15K localized-RL seed pool.The table describes 85K task-oriented training examples when task-aligned SFT and the RL pool are combined.Source: Table 18 · p. 27, §B.4 · p. 27

• Predictions and evidence points within ±τ are matched chronologically and one-to-one; precision, recall, and F1 use the resulting true positives.FPRnear measures responses on designated nearby negative probes.Source: §B.5 · p. 27
• For matched responses, Claude Sonnet 4.6 assigns a 0–4 content score normalized to [0, 1], and ANS averages those scores.Source: §B.5 · p. 27
• Content-F1 penalizes missed intents and extra interventions instead of dropping them from a conditional answer average.For Dynamic Intents, ANS and Content-F1 score proactive responses only; reactive answers remain in interaction history.Source: §B.5 · p. 27

C. StreamVibe-Harness Implementation Details

• The Harness maintains persistent interaction state and organizes causal evidence for the interaction policy; the auxiliary visual-memory model summarizes and consolidates memory without additional training.The evaluated StreamVibe-VL checkpoint remains the interaction policy, not one of four independent models.Source: C. StreamVibe-Harness Implementation… · p. 28

C.1. Interaction State

• State S_t tracks active intents, observed progress, and delivered responses, applying typed events deterministically and ignoring duplicate updates.Requests may introduce, revise, or retire intents; visual observations establish task-relevant events, while delivery records are created only after a response is emitted.Source: §C.1 · pp. 28–29

C.2. Intent Lifecycle

• Each intent has its own criterion, evidence reference, and delivery status, so addressing one event does not complete other active intents.A standing request remains active after reporting an occurrence; cancellation retires the request, and neither an active intent nor a matched event deterministically triggers a response.Source: §C.2 · p. 29

C.3. Hierarchical Memory

• Hierarchical memory retains recent visual context, timestamped mid-term summaries, and consolidated older text blocks; retrieval ranks facts and transitions by request-focused lexical overlap and recency.Retrieval is bounded, uses no embedding model or learned retriever, and current observations take precedence over superseded historical states.Source: §C.3 · pp. 30–31

C.4. State Integration

• Before each policy call, the Harness projects relevant state and retrieved evidence into compact context alongside causal video input, while the policy alone decides whether to respond and generates its text.Delivered responses are recorded for later context but are not treated as visual evidence; response verification, cadence gating, and post-generation rewriting are disabled.Source: §C.4 · p. 31

D. Qualitative Examples and Case Studies

• The section presents annotated examples from all ten task subtypes, followed by additional comparisons of model responses during sustained interaction.Source: D. Qualitative Examples and Case Studies · p. 31

D.1. Representative Samples from Every Subtype

• Figures 11–30 show two verified samples per subtype, with chronological six-frame filmstrips and timestamped reference assistance.Three interventions are annotated per sample except single-event I, which has one; outlined frames mark interventions, and dynamic examples include applicable user updates.Source: §D.1 · p. 31
• The examples span one-time events, state transitions, repeated events, guidance, learning, monitoring, GUI tasks, revisions, compositions, and interruptions.The displayed instructions and evidence illustrate varied tasks, including monitoring a deck’s state and guiding capsule-coffee preparation.Source: §D.1 · pp. 32, 33, 37

D.2. Additional Qualitative Case Studies

• In these case studies, StreamVibe uses task progress, interface state, and user goals for timely concrete guidance, while baselines more often mistime, miss, repeat, or mismatch advice.Source: §D.2 · p. 39
• Figures 31–36 compare model responses at two annotated evidence points, matched within ±2 seconds, across six additional cases: GIMP drawing and saving, Filmora editing/export, reminders, shopping, coffee-making, and shelf assembly.Source: Figure 31 · p. 39, Figure 32 · p. 40, Figure 33 · p. 40