PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveProBench: Can Streaming Video Models Really Interact Like Humans?

Kaixuan Du, Xin Wan, Hang Zhang, Meng Cao, Dai Guan, Ming Chen, YuKun Wang

arXiv:2609.12658v2cs.LG

TL;DR

Timestamp-selected benchmarks do not measure whether or when a streaming model should respond to a standing request. LiveProBench evaluates one-second stream intervals without response cues; early responses outnumber missed ones for three named systems, and the best model trails human performance by over 40 points.

  • Problem

    Timestamp-selected streaming benchmarks omit preceding silence and premature responses, while proactive interaction requires deciding when to respond to a standing request.

    Source: §1 · p. 1

  • Method

    LiveProBench evaluates standing requests at one-second intervals without response cues, pairing response and silence scoring and testing sustained silence and repeated-event counting.

    Source: §1 · p. 2, §2.2 · p. 2, §2.4 · p. 3

  • Results

    Early responses outnumber missed responses for ROMA, MOSS-VL-Realtime, and JoyAI-VL-Interaction by factors of 2.1–28.2.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    Effective streaming interaction requires deciding when to speak and when to remain silent; the best model trails human performance by over 40 points.

    Source: §5 · p. 4, §3.2 · p. 4

    The benchmark couples response policy with perception, and whether its error patterns generalize beyond the six evaluated systems remains open.

    Source: §4 · p. 4

Abstract

from arXiv · show

Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.

1. INTRODUCTION

Existing streaming-video benchmarks typically ask questions at evaluator-selected timestamps, leaving response timing and warranted silence unmeasured. LiveProBench evaluates standing requests throughout streams and distinguishes premature, timely, and missed responses across six subtasks.

  • Timestamp-selected benchmarks assess partial-stream question answering, omitting whether a response is warranted before the query time.Standing requests instead require silence until sufficient evidence appears, followed by a response within an acceptable interval.Source: §1 · p. 1
  • LiveProBench evaluates every response opportunity at one-second resolution without explicit response cues, preserving each model’s native silence mechanism.Source: §1 · p. 2
  • Six subtasks vary trigger clarity and timing tolerance, pairing response and silence rates while penalizing premature, missed, omitted, or repeated responses.Source: §1 · p. 2
  • Comparative evaluation identifies premature responses as the predominant error and quantifies how paired scoring affects model rankings.Source: §1 · p. 2

2. THE LIVEPROBENCH SUITE

LiveProBench evaluates streaming models’ response decisions at one-second intervals under standing requests, with six subtasks varying trigger clarity and timing tolerance. Its metrics assess timely responding, silence, and repeated-event counting across a dataset of distinct recordings.

2.1. Protocol

  • Models receive a standing request and a stream in one-second intervals, deciding after each interval whether to respond or remain silent, with at most 30 seconds of preceding context.The initial interval establishes the request and is excluded; labels remain independent of earlier model outputs.Source: §2.1 · p. 2

2.2. Two axes, six subtasks

  • The six subtasks vary timing tolerance and trigger clarity, from a single decisive second to extended or ambiguous activities; Onset Response uses a fixed three-second window after the earliest unambiguous second.The subtasks are Event Sensitivity, Onset Response, Window Response, Anomaly Alert, Sustained Silence, and Duplicate Counting.Source: §2.2 · p. 2, Table 1 · p. 2
  • Sustained Silence tests silence during a long pre-event prefix and recognition afterward, while Duplicate Counting scores whether predicted and reference occurrence totals match exactly.Sustained Silence examples average 102 evaluated decisions, compared with 6 to 32 for other subtasks.Source: §2.2 · p. 2

2.3. Data

  • Each example uses a distinct video, and videos are not shared across subtasks; stream durations span 4–207 seconds, totaling 5.4 hours and 18,146 evaluated one-second decisions.Requests undergo independent review, and examples with an earlier interval satisfying the request are excluded.Source: §2.3 · pp. 2–3

2.4. Metrics

  • For window-based subtasks, the first response is classified as early, in-window, or missed; accuracy is the fraction classified in-window, while Duplicate Counting uses exact-match accuracy.A response after the window or no response is missed.Source: §2.4 · p. 3
  • Event Sensitivity geometrically combines response-at-alert and silence-through-segment indicators, requiring both components to be high and discouraging indiscriminate responding or uniform silence.Pairing response and silence on the same recording controls for content, wording, and recording difficulty.Source: §2.4 · p. 3

3. EXPERIMENTS

The experiments train and compare streaming models under a one-second evaluation protocol, retaining their established prompting and silence mechanisms. In the audio-disabled comparison, Qwen3-Omni-SFT leads AVG, but different systems lead individual subtasks and premature responses remain a prominent error.

3.1. Setup

  • The training corpus contains 196k sequences; silence decisions and explicit-query responses are supervised throughout, while proactive trajectories retain only the final non-silent response.Silent-target weighting normalizes their aggregate contribution per sequence and remains defined when no silent target occurs.Source: §3.1 · p. 3
  • Qwen3-Omni-SFT uses Qwen3-Omni-30B-A3B as its base, and none of the evaluated models is further fine-tuned on LiveProBench.The primary comparison disables audio; prompting and silence criteria are retained where architectures permit.Source: §3.1 · p. 3
  • M4’s prompted silence scores are not directly comparable, and ROMA’s gate-head silence rate depends on its threshold.Treating M4’s “nothing to report” text as silence would conflate a textual response with no response.Source: §3.1 · p. 3

3.2. Results

  • Qwen3-Omni-SFT achieves the highest AVG at 51.37, leading AA, SS, and DC; MOSS-VL-Realtime leads ES, while AURA leads OR and WR.No system dominates all evaluated behaviors.Source: §3.2 · pp. 3–4, Table 2 · p. 4
  • The Human Agent achieves 92.99% AVG, while the best model trails by over 40 points, with deficits concentrated in SS and DC.The Human Agent scores 85.9% on SS and 83.6% on DC.Source: §3.2 · p. 4
  • Temporal errors are dominated by premature responses: in Fig. 2, early responses outnumber missed responses for ROMA, MOSS-VL-Realtime, and JoyAI-VL-Interaction by factors of 2.1–28.2; Qwen3-Omni-SFT instead yields 57.6% in-window, 17.7% early, and 24.7% missed.Source: §3.2 · p. 4
  • Audio shifts response timing differently across models: enabling it raises Qwen3-Omni-SFT’s early responses from 17.7% to 48.4%, while removing it raises ROMA’s from 66.2% to 74.2%.Qwen3-Omni-SFT’s ES score remains comparable with and without audio (59.4% vs. 61.8%).Source: §3.2 · p. 4
  • For M4, response and silence rates of 3.7% and 95.5% produce an arithmetic mean of 49.6 but a geometric mean of 18.8; ROMA’s 49.5% and 48.0% yield 48.7.The geometric aggregation exposes M4’s imbalance while reflecting ROMA’s more balanced rates.Source: §3.2 · p. 4

4. LIMITATIONS

The benchmark evaluates end-to-end behavior, so response policy and perceptual capability are coupled. Its error-pattern findings are based on six systems, with broader generalization still open.

6. Limitations

  • End-to-end scores couple response policy with perception, and error-pattern generalization beyond the six evaluated systems remains open.Model modality and silence-mechanism differences, fixed temporal parameters, and delay statistics conditioned on successful in-window responses also bound interpretation.Source: §4 · p. 4

5. CONCLUSION

LiveProBench evaluates response timing in streaming multimodal models under standing requests. In the audio-disabled comparison, premature responses outnumber missed responses for three systems, highlighting the need to decide when to speak and when to remain silent.

5. Conclusion

  • Across six systems, premature responses outnumber missed responses for three in the audio-disabled comparison, underscoring the role of timely speaking and silence decisions in streaming interaction.The benchmark provides a time-resolved framework for measuring response policy under continuous streams.Source: §5 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The work reports computational evaluation on existing video recordings and involves no human or animal intervention. Institutional ethics approval was not required.

6. Compliance with Ethical Standards

  • The evaluation uses existing video recordings and involves no human or animal intervention; no institutional ethics approval was required.Source: §6 · p. 5

Related papers

Loading 2609.12658v2…