PaperlayerView exact revision on arXiv

Source-linked AI summary

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

Boyu Cui, Junhao Luo, Zhihao Wang, Linghan Chen, Zejun Li, Jiwen Zhang, Siyuan Wang, Zhongyu Wei

arXiv:2610.21568v1cs.LG

TL;DR

Existing video benchmarks mainly test interpretation of observed content, while proactive assistance also requires anticipating risks and deciding when intervention is justified. P3IBENCH evaluates these capabilities in real-life streaming scenarios and benchmarks 12 models; results reveal response-calibration difficulties, while targeted training improves several tasks unevenly.

  • Problem

    Existing benchmarks mainly evaluate perception, while proactive assistance also requires anticipating emerging risks and intervening at justified, actionable times.

    Source: §1 · p. 1

  • Method

    P3IBENCH evaluates daily perception, risk alert, and event intervention using temporally grounded targets, and the authors train P3I-Assistant-4B with supervised fine-tuning and reinforcement learning.

    Source: §1 · pp. 1–3

  • Results

    No evaluated model performs reliably across all three capabilities, and the strongest system reaches only 46.59 PRS-B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective proactive assistance requires well-calibrated decisions about whether and when to respond, beyond visual understanding alone.

    Source: §1 · p. 2

    P3IBENCH is intended for research evaluation, not deployment in safety-critical or surveillance applications, because models may produce incorrect or delayed responses.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce \textsc{P$^3$IBench}, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train \textsc{P$^3$I-Assistant-4B}, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.

1 INTRODUCTION

Proactive assistants must anticipate what may happen and decide when guidance is justified, while existing benchmarks largely emphasize perception and require precise temporal boundaries. P3IBENCH evaluates perception, prediction, and intervention in real-life streaming scenarios; tests of current models and targeted training show persistent calibration challenges and uneven gains.

Motivation

  • Existing benchmarks mainly assess perception, whereas real-life assistance also requires predicting emerging risks and intervening at justified, actionable times.Delayed responses can be useless, while premature responses may lack sufficient evidence.Source: §1 · p. 1

Benchmark contribution

  • P3IBENCH combines 533 video-query samples from 429 videos with temporally grounded targets and hybrid multi-LLM and human annotation and review.The benchmark covers Daily Perception, Risk Alert, and Event Intervention, evaluating response correctness and timeliness.Source: §1 · p. 2

Evaluation and training

  • No evaluated model performs reliably across all three capabilities, and the strongest system reaches only 46.59 PRS-B.Adapted offline models tend to over-respond, while native proactive models often miss warranted responses or respond late.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors construct proactive training data and train P3I-ASSISTANT-4B using supervised fine-tuning and reinforcement learning, reporting clear gains on P3IBENCH.The abstract characterizes gains as uneven across capabilities.Source: §1 · pp. 2–3

2 RELATED WORKS

Prior work includes online and proactive video models with varied mechanisms for streaming context and response control. Existing benchmarks often cover only some proactive capabilities and face challenges in maintaining consistent response validity and temporal boundaries; P3IBENCH combines all three capabilities under a unified causal protocol with targeted verification and human inspection.

2.1 ONLINE AND PROACTIVE VIDEO MODELS

  • Online and proactive video models use approaches including streaming dialogue, memory banks, separated perception and reaction, and learned response-or-silence decisions.Examples include VideoLLM-online, MMDuet2, VideoChat-Online, Dispider, and Em-Garde.Source: §2.1 · p. 3

2.2 STREAMING AND PROACTIVE VIDEO BENCHMARKS

  • Prior benchmarks typically cover only a subset of perception, prediction, and intervention or combine them in specialized settings.Their construction pipelines vary in automation and human review, and may inherit heterogeneous annotations or automatically generated labels that complicate consistent temporal boundaries.Source: §2.2 · p. 3
  • P3IBENCH combines automated annotation, targeted verification, and human inspection with unified causal evaluation across everyday monitoring, risk warning, and goal-conditioned assistance.Its response targets are intended to be more consistent and high-confidence than approaches relying on heterogeneous source annotations or limited post-hoc review.Source: §2.2 · p. 4

3 P3IBENCH

P3IBENCH evaluates streaming models that must decide what and when to respond using only observations available so far, and remain silent when no response is warranted. Its three categories cover monitoring, hazard warning, and goal-conditioned assistance, with temporally distributed targets and response scoring that accounts for correctness, timing, missed targets, and redundant outputs.

3.1 BENCHMARK CONSTRUCTION

  • P3IBENCH defines streaming decisions over incrementally observed video: models respond with task-relevant content or remain silent when no response is warranted.Daily Perception covers event and object monitoring; Risk Alert covers imminent collisions and detectable criminal or suspicious behavior; Event Intervention provides goal-conditioned reminders, corrections, or guidance.Source: §3.1 · p. 4
  • Daily Perception uses timestamped narrations and visual boundary checks for event monitoring, and segmentation masks plus inspection for object disappearance.Risk Alert uses accident annotations and manually refined crime intervals; Event Intervention uses scripted first-person videos with manually verified intervention windows and reference responses.Source: §3.1 · p. 4

3.2 BENCHMARK STATISTICS

  • 836 ground-truth response targets span early, middle, and late video thirds, which contain 37.8%, 39.7%, and 22.5% of events, respectively.Event Intervention is especially balanced across those thirds, with 87, 87, and 84 events.Source: §3.2 · p. 5

3.3 EVALUATION PROTOCOL

  • Proactive Response Score combines response correctness with temporal alignment, applying exponential decay as emission time departs from the target time.Correctness is rule-based for structured outputs and LLM-judged for open-ended intervention responses.Source: §3.3 · p. 5
  • PRS aggregates matched response scores per target, penalizing redundant outputs and assigning zero to missed targets; PRS-U and PRS-B differ in how responses are matched to targets.PRS-U assigns each response to the most recent target at or before emission, while PRS-B assigns it to the temporally nearest target.Source: §3.3 · pp. 5–6

4 EXPERIMENT

Across five proactive tasks, most models score below 30 on average, and even the strongest model struggles with timely event perception. The analysis identifies distinct response-frequency and timing failures, sensitivity to streaming context, and preliminary gains from targeted training.

4.2 MAIN RESULTS

  • Offline models often generate excessive repeated replies; on Accident Warning, Qwen2.5-VL-3B produces at least three responses in 53 of 100 target windows.Those windows score 21.27 PRS-B, versus 71.39 without the redundancy penalty.Source: §4.2 · pp. 6–7
  • Most models average below 30 in PRS, while GPT-5.6-Luna performs strongest overall but still struggles to promptly perceive events and objects in streaming contexts.The challenge is especially pronounced on complex intervention tasks.Source: §4.2 · p. 6, Table 2 · p. 7
  • Proactive models can overcorrect into silence: MMDuet2 misses 79% and JoyVL misses 100% of Accident Warning ground-truth windows.The authors hypothesize that current training may imbalance supervision for silent and content-bearing responses.Source: §4.2 · p. 7
  • Event-level perception requiring continuous, complex context is substantially weaker than object perception, affecting prediction and intervention timing.PRS-B is generally 6–15 points higher than PRS-U, suggesting difficulty identifying when responses are actually required despite some appropriate early warnings.Source: §4.2 · pp. 7–8

4.3 UNDERSTANDING THE CHALLENGES BEHIND STREAMING VIDEO CONTEXTS

  • Longer videos and later events challenge both tested models, while Qwen3-VL is less sensitive to information presented early in a video.The authors suggest accumulated-context complexity may contribute to the difficulty.Source: §4.3 · p. 8
  • Higher FPS can facilitate object perception but makes risk prediction more challenging, whereas retaining more historical visual information consistently improves performance.The benchmark therefore retains the default 1-FPS setting.Source: §4.3 · p. 9, Table 3 · p. 9, Table 4 · p. 9

4.4 PRELIMINARY EXPLORATION TO ENHANCE PROACTIVE CAPABILITIES

  • The preliminary training study adds 1.3K samples and fine-tunes Qwen3-VL-4B with two-stage SFT and RL, using dynamic weighting to address excessive silence.The training set includes 600 Accident Warning and 768 Event Perception samples.Source: §4.4 · p. 9
  • Post-training raises average PRS-B by +8.1 with SFT and +13.5 with RL, and the final model outperforms JoyVL despite using approximately 300× fewer training samples.Gains also generalize to Object Perception and Crime Alert without corresponding training data, while Intervention remains a target for more sophisticated strategies.Source: §4.4 · p. 9, Table 5 · p. 9

5 CONCLUSION

P3IBENCH evaluates proactive video understanding across three categories, and P3I-Assistant-4B improves performance on several tasks. The authors describe the benchmark as intended for research evaluation rather than safety-critical or surveillance deployment.

5 CONCLUSION

  • P3IBENCH evaluates proactive video understanding across Daily Perception, Risk Alert, and Event Intervention.Source: §5 · p. 9
  • P3I-Assistant-4B, trained through targeted post-training, improves performance on several tasks.Source: §5 · p. 10
  • The authors report that AI-assisted content was reviewed and verified, and that they take responsibility for the final paper.Source: AI USE STATEMENT · p. 10
  • The benchmark is intended for research evaluation, not safety-critical or surveillance deployment, because models may respond incorrectly or late.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Upon publication, the authors plan to release the benchmark and training data, construction pipeline, checkpoints, code, prompts, and configurations.Source: REPRODUCIBILITY STATEMENT · p. 10

A TASK DEFINITIONS AND DATA EXAMPLES

The appendix defines a causal streaming setting in which the model sees frames chronologically and responds only when the instructed condition is reliably identified.

A TASK DEFINITIONS AND DATA EXAMPLES

A.1.1 EVENT MONITORING

Event Monitoring comprises three task forms that vary in the scope of the monitored event and the required response, with queries supplied at the start of the video.

A.1.1 EVENT MONITORING

  • Atomic Event Monitoring counts visually supported occurrences satisfying all query constraints and outputs the target index when that occurrence becomes identifiable.Source: §A.1.1 · p. 13, Figure 6 · p. 14
  • Event Type Monitoring counts occurrences in a semantic category, allowing different objects or manners, and outputs the target index when identifiable.Source: §A.1.1 · pp. 13–14, Figure 7 · p. 14
  • Compositional Process Monitoring reports start and end once for a complete, goal-related activity, including its initiating, advancing, adjusting, and completing actions.Source: §A.1.1 · p. 14, Figure 8 · p. 15

A.1.2 OBJECT MONITORING

The appendix specifies response rules for object monitoring, risk alerts, and user-goal interventions, then describes the streaming inference procedure used to evaluate models.

A.1.2 OBJECT MONITORING

  • Object Monitoring tracks a described object and issues one Warning per confirmed disappearance, including a new warning if it reappears and disappears again.Source: §A.1.2 · p. 15, Figure 9 · p. 15

A.2.1 ACCIDENT WARNING

  • Accident Warning requires one Warning at the earliest reliable identification of imminent collision risk, before the collision occurs.Source: §A.2.1 · p. 16, Figure 10 · p. 16

A.2.2 CRIME ALERT

A.3 EVENT INTERVENTION

  • Event Intervention links observed progress to the user’s goal and provides relevant guidance within an annotated response window; multiple interventions may be required.Source: §A.3 · p. 17, Figure 12 · p. 18

B STREAMING INFERENCE PROCEDURE

  • Streaming inference samples video at a default 1 FPS and prompts a response decision at each step, retaining instructions and dialogue history while handling silence tokens.Source: B STREAMING INFERENCE PROCEDURE · p. 18

C.1 TRAINING DATA

Training data combine broad proactive QA examples with benchmark-targeted annotations. SFT trains response-or-silence decisions with imbalance-aware weighting, while RL uses incremental video rollouts and trajectory rewards that favor timely correct responses and penalize redundant or invalid outputs.

C.1 TRAINING DATA

  • The SFT corpus contains 12,000 broadly scoped proactive QA examples and 1,368 benchmark-targeted examples; RL uses 863 rollout windows across Accident and Event tasks.The SFT data include 10,000 PVQ examples from Shot2Story, 2,000 CRR examples from LLaVA-Video, and 600 Accident plus 768 Event examples; Event RL windows comprise 100 Atomic, 100 Event Type, and 363 Compositional windows.Source: §C.1 · p. 19

C.2 SUPERVISED FINE-TUNING

  • SFT starts from Qwen3-VL-4B-Instruct, freezes the vision encoder, and trains on preceding observations and interaction history with silence or response targets.Focal and frequency-balancing factors weight state-token decisions, while ordinary response tokens retain standard cross-entropy under shared token-count normalization.Source: §C.2 · pp. 19–20

C.3 REINFORCEMENT LEARNING

  • RL initializes from the final SFT checkpoint and reveals video incrementally, optimizing only responses generated within each rollout window.Accident and Compositional rollouts retain a bounded preceding video prefix; Atomic and Event Type use context from the video start.Source: §C.3 · p. 20
  • Accident rewards score the closest correct warning by temporal proximity and penalize other non-silent outputs, while Atomic and Event Type use bidirectional first-only trajectory scores.Explicit additive penalties are used for Accident and Compositional because the SFT checkpoint shows a stronger tendency toward redundant or invalid outputs on those tasks.Source: §C.3 · pp. 21–22
  • Compositional rewards average bidirectional first-only scores over active boundaries and subtract a 0.2 penalty for invalid outputs; repeated predictions lower the score.Silent outputs receive no independent positive reward, so silence throughout a rollout does not earn the temporal reward associated with target responses.Source: §C.3 · p. 21

D ERROR ANALYSIS

The error traces illustrate failures in response control, warning timing, and goal-conditioned intervention. Correctly recognizing an event or location does not ensure a timely or useful response.

Repeated triggering

  • Qwen3-VL-8B-Instruct outputs 1 at 13 timestamps for a request requiring one response to the first clay-mix placement, violating the one-response constraint.Later molding actions were not additional requested targets, and the first response slightly preceded the annotated onset.Source: D ERROR ANALYSIS · p. 22

Delayed warning

  • GPT-5.6-Luna issues its only Warning at 40.200 s, 21.333 s after the 18.867 s target and after the close approach and abrupt camera motion.The valid warning token therefore arrives too late to anticipate the depicted risk.Source: D ERROR ANALYSIS · p. 22

Missing corrective guidance

  • JoyAI-VL-Interaction announces the second-floor arrival but repeats that notice during the correction window instead of telling the descending user to stop or turn back.The trace distinguishes recognizing a floor from identifying an action that deviates from the requested destination and correcting it.Source: D ERROR ANALYSIS · p. 22

E BEST-REPLY EVALUATION WITHOUT REDUNDANCY PENALTY

The best-reply variant retains PRS matching, correctness, and temporal decay, but scores each target by its highest-scoring matched reply without dividing by matched-response count. Results are averaged by video and task.

E BEST-REPLY EVALUATION WITHOUT REDUNDANCY PENALTY

  • Target scores are averaged within each video and then across videos; Event averages Atomic, Compositional, and Event Type, while Overall equally averages the five tasks.Scores are reported on a 0–100 scale.Source: E BEST-REPLY EVALUATION WITHOUT… · p. 22
  • Figure 13 groups response-trace failures into repeated outputs, a delayed warning, and a repeated floor-arrival notice that omits correction.Compare the traces for response control, warning timeliness, and intervention guidance; the caption states no best-reply evaluation outcome.Source: Figure 13 · p. 23
  • Table 8 reports PRS-U’ and PRS-B’ best-reply scores, retaining the original matching and temporal-decay rules while scoring each target by its highest-scoring matched reply.The variant does not divide by the number of matched responses; targets requiring silence retain the original scoring rule.Source: E BEST-REPLY EVALUATION WITHOUT… · p. 22, Table 8 · p. 24

Related papers

Loading 2610.21568v1…