Source-linked AI summary
OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
Ruixiang Zhao, Jie Yang, Zijie Xin, Tianyi Wang, Fengyun Rao, Jing LYU, Xirong Li
TL;DR
Existing proactive streaming benchmarks inadequately test audio-visual perception, autonomous response timing, and diverse video understanding. OMNIPRO introduces a 2,700-sample benchmark with Probe and Online protocols, finding consistent but heterogeneous audio gains, severe late-video degradation, and weakest non-speech perception.
Problem
Existing benchmarks rely mainly on visual signals, polling or fixed timestamps, and limited task coverage, hindering comprehensive evaluation of omni-proactive streaming models.
Method
OMNIPRO combines 2,700 samples across 9 sub-tasks and 3 cognitive levels with 84% audio dependency, modality-isolation labels, and Probe and Online evaluation modes.
Results
Audio-visual input improves performance by +2.4 to +11.1 across models, while models retain 37% of early-segment performance later in videos and non-speech perception remains weakest.
Takeaways & Limitations
OMNIPRO differentiates models' audio utilization and reveals long-horizon temporal degradation and non-speech sound perception as concrete research challenges.
Takeaways & Limitations
English-only questions and annotations limit evaluation of multilingual or non-English proactive streaming models.
Abstract
from arXiv · showhide
Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-modal large language models. Existing benchmarks fall short in three key aspects: they rely primarily on visual signals, adopt polling or fixed-timestamp protocols instead of true proactive evaluation, and cover only a limited range of tasks, preventing reliable assessment and differentiation of omni-proactive streaming models. We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks. It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities. Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis. We further introduce a dual-mode evaluation protocol: Probe mode assesses content understanding by querying the model before and after each ground-truth trigger, while Online mode evaluates full proactive ability by requiring models to autonomously decide when to respond in streaming input. Evaluating 11 representative models reveals three key findings: (1) audio provides consistent gains but with highly variable utilization across models, (2) performance degrades significantly over time, indicating limited long-horizon robustness, and (3) non-speech audio perception remains the weakest dimension.
1 Introduction
OMNIPRO addresses shortcomings in existing proactive streaming benchmarks by jointly testing multimodal perception, autonomous response timing, and diverse video understanding. It combines broad audio-visual coverage with Probe and Online evaluation modes, and exposes heterogeneous audio use, temporal degradation, and weak non-speech perception.
- Motivation: Existing benchmarks underrepresent audio, proactive timing, and diverse video understanding, limiting unified assessment of omni-proactive models.Prior protocols rely on visual cues, polling or preset queries, and narrow task coverage.
- Benchmark: OMNIPRO contains 2,700 human-verified samples across 9 sub-tasks and 3 cognitive levels covering 6 basic video understanding capabilities.Figure 1 presents representative frames, time-aligned triggers, instructions, and expected responses.
- Benchmark: 84% of samples depend on speech or non-speech audio, with modality-isolation labels supporting fine-grained multimodal ablation.The benchmark is designed to distinguish audio-visual reasoning from vision-only performance.
- Evaluation: Probe mode queries models around ground-truth triggers, whereas Online mode requires autonomous responses during continuous video streams.Probe supports models without streaming inference; Online evaluates full proactive behavior.
- Findings: Audio-visual input outperforms video-only input by +2.4 to +11.1 across models, but models retain only 37% of early-segment performance on average later in videos.Non-speech sound perception remains the weakest dimension across evaluated models.
2 Related Work
Prior proactive streaming benchmarks incompletely test multimodal perception, autonomous timing, and broad video understanding. OMNIPRO addresses these gaps with audio-inclusive, multi-response online evaluation across all six capabilities.
- Proactive Modeling: Token-driven and classification-head approaches represent two established strategies for deciding when a streaming model should respond.Token-driven timing is embedded in generation, while classification heads decouple response timing from content generation.
- Benchmark Criteria: Existing benchmarks are compared by audio requirements, autonomous response timing, and coverage of the 6 basic video understanding capabilities.These criteria distinguish genuine omni-proactive evaluation from visual or offline question answering.
- Existing Benchmarks: StreamingBench-Pro is visual-only, polled every second, limited to one response, and covers only Alert (1/6 capabilities).Its sports and gaming questions require no audio.
- Existing Benchmarks: OVO-Bench-Pro queries preset time points and remains effectively offline, covering Counting and weak Monitoring (2/6) without audio.The model never initiates responses autonomously.
- Existing Benchmarks: OmniMMI-Pro permits genuine online responding, but only one response per question, speech-only audio, no non-speech sound, and Alert-only video understanding.Its Proactive Turn-Taking subset is unrelated to video understanding.
- OMNIPRO: OMNIPRO requires or benefits from audio in 84% of samples, supports multiple online responses with over-trigger penalties, and covers all 6 capabilities across 9 sub-tasks.Thus, it simultaneously addresses the three benchmark criteria.
3 Proposed Benchmark
OMNIPRO is constructed as a nine-task benchmark spanning three cognitive levels and six video-understanding capabilities, with audio-visual source data, human quality control, and dual evaluation modes. Its design emphasizes multimodal triggers, extended temporal coverage, and separate assessment of content understanding and proactive responding.
- Task Taxonomy: OMNIPRO organizes 9 sub-tasks into Perception, Comprehension, and Reasoning levels covering 6 basic video understanding capabilities.
- Task Taxonomy: The taxonomy includes alerting, monitoring, snapshot and cumulative counting, target grounding, narration, semantic condition detection, deduplicated counting, and sequential step instruction.
- Source Video Collection: Source videos comprise 1,171 LongVALE videos and 600 COIN videos, selected to provide broad audio-visual content and instructional procedures.
- Data Generation: The generation pipeline creates temporally aligned multimodal captions and structured question-answer samples, prioritizing audio and speech while enforcing trigger-time and streaming constraints.
- Human Quality Control: Two rounds of annotation review verify naturalness, trigger timing, response faithfulness, and modality labels; retaining approximately 30% yields 2,700 samples across 1,262 videos.
- Dataset Statistics: Dataset statistics show audio-triggered tasks, dominant visual-plus-speech triggers, and trigger times averaging 54.1 seconds initially and 126.2 seconds finally.
- Evaluation Protocol: Probe queries models before and after each ground-truth trigger using cumulative frames, while correctness requires both responses to be correct.
4 Experiments
Experiments evaluate 11 representative models in Probe and Online modes, showing that audio helps but models differ in modality use, long-horizon performance declines, and non-speech audio remains difficult.
- 11 representative models are evaluated across Probe and Online modes, including open-source, proprietary, omni-modal, and vision-only systems.
- 40.4% average accuracy makes Gemini-3-Flash nearly twice as strong as the best open-source model at 22.1%.
- Audio-visual input improves over video-only input by +2.4 for Qwen3-Omni to +11.1 for video-SALMONN 2+.
- Models use modalities differently: audio dominates Event-Alert, whereas vision dominates Dedup.-Count and Step-Inst. for the cited examples.
- 37% of Short-term performance remains on average for Long-term triggers, while Gemini-3-Flash retains 46% and MiniCPM-o 4.5 falls from 29.1 to 0.3.
- All models perform weakest on visual+sound triggers, with scores ranging from 15.3 to 22.3.
5 Conclusions
The conclusion presents OMNIPRO as a benchmark for jointly assessing omni-modal perception, proactive responding, and diverse video understanding. Its evaluation finds persistent modality, temporal, and model-capability gaps.
- OMNIPRO contains 2,700 human-verified samples across 9 sub-tasks and 3 cognitive levels, with 84% audio dependency and Probe and Online evaluation modes.
- Proprietary systems outperform open-source systems substantially, with reported average performance of 40.4% versus 22.6%, especially on reasoning-level tasks.
- Audio and video provide complementary cues, but models exhibit divergent modality-utilization patterns.
- All models struggle with late events in long videos, while online streaming models nearly fail beyond 180 seconds.
- Non-speech audio perception remains the weakest dimension across models.
A.1 Tolerance Window Ablation
The tolerance-window ablation varies temporal matching tolerances for Online-mode evaluation and adopts ±3 seconds as the default.
- Performance is evaluated under temporal matching tolerances ranging from ±1 s to ±5 s for three Online-mode models.
- ±3 s is adopted as the default tolerance window for all Online-mode evaluations.
B More Details of Data Construction
Data construction combines dense multimodal video captions with task-specific prompts to synthesize structured proactive QA samples across multiple event-monitoring tasks.
- Gemini 3 Flash generates temporally aligned dense captions integrating visual, audio, and speech descriptions for each source video.
- Captions divide videos into fine-grained segments, typically 5–30 seconds, with new segments for scene, action, speech, people, audio, or visual changes.
- Task-specific prompts use the original video and dense captions to synthesize structured QA pairs with questions, trigger times, responses, trigger modalities, and audio dependency.
- Instant Event Alert: Instant Event Alert asks the assistant to issue natural responses when a specified event occurs, preferably using audio triggers and accurate timestamps.
- Explicit Target Grounding: Explicit Target Grounding tests real-time trigger detection and target localization using a 3x3 frame grid.
- Realtime State Monitor: Realtime State Monitor requires reporting discrete state transitions, prioritizing audio dimensions and excluding initial-state reports.
C.1 Limitations
OMNIPRO is limited to English-language questions and ground-truth annotations, constraining evaluation of multilingual or non-English proactive streaming models.
- All OMNIPRO questions are written in English.
- All OMNIPRO ground-truth annotations are written in English.
- The benchmark’s English-only design limits applicability to multilingual or non-English proactive streaming models, with additional languages left for future work.
C.2 Broader Impacts
OMNIPRO supports standardized comparison and capability-gap analysis while presenting general benchmark risks and restricting commercial use through its license.
- OMNIPRO provides standardized evaluation across omni-modal perception, proactive responding, and diverse video understanding tasks.
- It facilitates fair model comparison and identifies concrete capability gaps that guide future research directions.
- The benchmark introduces no domain-specific risks beyond those inherent to the underlying models, although improved capabilities could be misapplied in unintended contexts.
- The dataset is released under CC BY-NC 4.0, prohibiting commercial use, and contains publicly available YouTube videos without personally identifiable information in annotations.
C.3 Licenses
OMNIPRO uses CC BY-NC 4.0, and its licensing is compatible with the source datasets’ licenses while respecting their terms of use.
- OMNIPRO is licensed under CC BY-NC 4.0.
- The evaluation code is released under the MIT License.
- OMNIPRO’s license is compatible with the source dataset licenses, whose terms of use are respected and whose datasets are properly cited.