Source-linked AI summary
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
Xudong Lu, Xueying Li, Annan Wang, Yang Bo, Jinpeng Chen, Zengliang Li, Nianzu Yang, Rui Liu, Xue Yang, Jingwen Hou, Hongsheng Li
TL;DR
Real-time omnimodal assistants must recognize multimodal triggers, decide when to respond, and answer without future context, but existing benchmarks do not fully test this interaction loop. OmniInteract introduces a native online audio-visual benchmark with temporally grounded interaction slots and evaluates models across response quality, timing, interruptions, and context resumption. Current models struggle especially with long-horizon monitoring, nested resumption, and full-duplex-oriented reasoning, showing that offline capability does not necessarily transfer to reliable online interaction.
Problem
Existing benchmarks do not fully evaluate the coupled process of recognizing spoken intent, timing responses, and answering during continuous audio-visual streams.
Method
OmniInteract benchmarks native online inference over continuous audio-visual streams using interaction slots defined by triggers, response windows, and target answers.
Results
Current models show substantial gaps in real-time interaction, especially for continuous monitoring, nested query resumption, and full-duplex-oriented reasoning.
Takeaways & Limitations
Offline multimodal understanding or reasoning does not necessarily translate into robust real-time interaction.
Takeaways & Limitations
The benchmark covers four representative models, limits online degradation analysis to MiniCPM-o mathematical reasoning, and has restricted language and domain coverage.
Abstract
from arXiv · showhide
We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.
1 Introduction
Existing benchmarks underrepresent real-time omnimodal interaction because they remove response timing, spoken intent, or native online inference. OmniInteract preserves continuous audio-visual interaction and evaluates temporally grounded responses across localized, nested, and long-horizon tasks.
- Motivation: Offline QA grants future access, while text-prompted streaming QA bypasses spoken intent recognition and native online interaction.These settings therefore only partially evaluate deciding whether, when, and what to respond during an evolving stream.
- Benchmark: 1,430 temporally grounded response slots span 250 videos, including 1,062 1Q1A slots and 368 1QnA slots for localized interaction and continuous task monitoring.The 1Q1A split includes real-time, proactive, and nested scenarios, while 1QnA covers multiple responses to one instruction.
- Evaluation: 192 interrupted slots and 240 nested slots test interruption handling and context resumption alongside quality and timeliness.The benchmark uses IA-QTF1, IDS, and NCCS to evaluate these interaction-control dimensions.
- Findings: Current models show substantial streaming-interaction gaps, particularly in continuous task monitoring and full-duplex-oriented reasoning.The reported comparison finds that offline reasoning quality degrades when models listen and generate simultaneously.
- Benchmark: OmniInteract evaluates omnimodal LLMs through native online inference over continuous audio-visual streams containing spoken queries, visual events, and background sounds.Models process the stream without lookahead to future content rather than answering converted video-text pairs.
- Evaluation: Each interaction slot specifies a trigger, expected response window, and target answer, making response opportunity, timing, and content jointly measurable.The formulation preserves the temporal and multimodal structure of continuous interaction.
2 Related Work
Related work advances online video understanding, unified omnimodal modeling, and full-duplex speech interaction. These directions motivate evaluation of synchronized perception, decision-making, and response in richer real-time settings.
- Streaming Video Understanding: Streaming video understanding moves from offline post-hoc analysis toward online interaction requiring synchronized perception, decision-making, and response.Recent approaches use temporally aligned long-context modeling and streaming token management.
- Omnimodal Modeling: Omnimodal LLMs integrate vision, audio, speech, and text, allowing spoken intent, background sounds, and visual events to shape interaction context.These developments extend multimodal interfaces beyond visual-language inputs.
- Full-Duplex Interaction: Full-duplex studies target low-latency speech-to-speech interaction without explicit turn segmentation and improve native audio interaction through dedicated training.This line of work motivates real-time interaction while models process incoming input and generate output.
3 OmniInteract Benchmark
OmniInteract benchmarks native online omnimodal interaction by preserving continuous audio-visual streams and organizing response opportunities into temporally grounded slots. Its evaluation jointly measures response content, timing, invalid outputs, interruption handling, and context resumption across 1Q1A and 1QnA scenarios.
- Benchmark Design: OmniInteract evaluates omnimodal LLMs during native online inference, preserving spoken queries, visual events, and ambient sounds without future-content lookahead.This avoids converting interactions into offline video-text question-answer pairs or externally supplied textual prompts.
- Data Composition: 1Q1A covers real-time, proactive, and nested interactions, whereas 1QnA targets continuous monitoring with multiple responses to one instruction.The 1Q1A data is curated from 210 self-recorded videos, while 1QnA uses 40 videos from procedural and task-oriented benchmarks.
- Slot Construction: Each interaction slot contains a trigger, expected response window, and target answer, making response opportunity, timing, and content measurable.For continuous streams, slots discretize evaluation around temporal interaction events and valid-answer times.
- Slot Construction: Generated chunks are assigned to slots by start time, overlap is resolved using the latest context, and chunks crossing the valid-answer time are split into early and core segments.Unassigned chunks become unmatched outputs and are penalized during metric computation.
- Interaction-Aware Evaluation: IA-QTF1 combines soft timing- and quality-sensitive true positives with penalties for missing or unwarranted behavior, including unmatched chunks, early hallucinations, low-quality responses, and spill.The framework also introduces targeted diagnostics for interruption handling and nested context management.
4 Experiments
Experiments evaluate four omnimodal models under native online streaming, measuring localized, nested, long-horizon, interruption, and offline–online reasoning behavior. Performance varies substantially, with long-horizon monitoring and context resumption remaining especially difficult.
- Evaluation setup: Four representative models were evaluated using native real-time pipelines and audio-visual streams, with GPT-4o judging open-ended answers against ground truth.The protocol requires models to handle spoken intents, visual evidence, and response timing jointly.
- 1Q1A interaction: AURA achieved the highest global 1Q1A IA-QTF1 of 0.467, narrowly ahead of MiniCPM-o at 0.456.Gemini led explicit real-time queries at 0.553, while MiniCPM-o led proactive interactions at 0.607.
- Nested interaction: MiniCPM-o achieved the best nested NCCS at 0.284, while Gemini and Qwen3.5-Omni failed to resume outer queries in 119 and 116 of 120 cases.NCCS captures chain-level completion beyond local inner- and outer-answer validity.
- 1QnA interaction: AURA achieved the highest 1QnA IA-QTF1 at 0.052, but every model performed substantially worse than on 1Q1A.Models often missed intermediate response opportunities or answered at inappropriate times during continuous monitoring.
- Overall performance: MiniCPM-o achieved the highest overall Global IA-QTF1 of 0.368, followed by AURA at 0.363.The uniformly low 1QnA scores indicate weak general-purpose performance across localized and long-horizon settings.
- Interruption analysis: Interruption diagnostics exposed a trade-off: MiniCPM-o had the best PAQ of 0.571 but severe conditional spill, whereas Gemini had the highest NOR of 85.94% and best CSM.Gemini’s CSM was 40.74% and 0.312 s, while MiniCPM-o’s was 83.15% and 10.067 s.
- Offline–online transfer: MiniCPM-o’s mathematical reasoning quality fell from 0.6833 offline to 0.3475 online, an absolute decrease of 0.3358.The comparison uses pure quality scores that exclude time decay and false-positive/false-negative penalties.
5 Conclusion
The paper introduces OmniInteract for native online evaluation of omnimodal interaction while preserving spoken queries, visual events, ambient sounds, and response timing. Results show that current models struggle especially with long-horizon monitoring and nested-query resumption, underscoring a gap between offline understanding and reliable full-duplex interaction.
- Contribution: OmniInteract evaluates omnimodal LLMs in native online streaming audio-visual interaction rather than offline or pre-segmented question answering.The benchmark jointly evaluates answer quality, timeliness, interruption handling, and context resumption.
- Conclusion: Current models struggle with robust real-time interaction, especially long-horizon 1QnA monitoring and nested-query resumption.The results identify a gap between offline multimodal understanding and reliable full-duplex-oriented interaction.
Limitations
The benchmark’s evidence is limited by model coverage, a single-model online capability study, speech-generation differences between splits, and narrow language and domain coverage.
- Scope boundaries: The study evaluates four representative models, limits online capability degradation analysis to MiniCPM-o on mathematical reasoning, and mixes TTS-synthesized and naturally recorded speech across splits.The benchmark covers Chinese daily-life interactions and English mathematical reasoning, leaving broader language and domain coverage for future work.
Ethical Considerations
The benchmark uses consented author-recorded videos and publicly licensed datasets, while recognizing privacy and surveillance risks from always-on multimodal assistants.
- Data and deployment: OmniInteract does not release unauthorized personal data: self-recorded videos were made with informed consent, and 1QnA uses public datasets under their original licenses.The authors note that always-on multimodal systems still require deployment safeguards because of privacy and surveillance concerns.
A.1 Data Licenses and Annotation Details
Human annotation for OmniInteract was compensated at US$20 per hour, while external data sources and generation tools are documented in Table A.1.
- US$20 per hour was the compensation rate for human annotators.
A.2 Detailed Scoring Definitions
OmniInteract scores responses by combining early interaction behavior, core-answer quality and timeliness, false-positive handling, and interruption-specific diagnostics.
- Early-stage scoring rewards timely acknowledgments and penalizes substantive answers produced before sufficient evidence.The early segment is [tstart, ta).
- Core-stage scoring combines semantic answer quality with a timeliness factor that decays as the semantic anchor moves toward the response-window end.Semantic quality is judged against the ground-truth answer, while timeliness decreases linearly from ta to tend.
- Soft true positives combine acknowledgment and core scores while clamping the result so it cannot exceed 1.
- False positives include unmatched chunks, early hallucinations, low-quality core responses, and spillover beyond the slot boundary.
- Non-interrupted slots receive FN = 1 when no valid core answer is produced, whereas interrupted slots do not incur false negatives.Acknowledgments alone do not satisfy completion for non-interrupted slots.
- Interrupted slots are additionally evaluated with No-Output Rate, Partial Answer Quality, and Conditional Spill Metrics.These diagnostics distinguish silence, useful partial responses, and spillover behavior.
A.3 Detailed TP/FP/FN Breakdown
The appendix reports category-level TP, FP, and FN values for IA-QTF1, with mutually exclusive 1Q1A categories and a global aggregation over all 1,430 slots.
- 1Q1A categories use mutually exclusive response slots in the detailed TP/FP/FN breakdown.
- Table A.1 is identified as the license and access-term summary for external data sources and generation tools.
- 1,430 slots are aggregated in the global score, whose false positives include unmatched chunks not assigned to individual categories.
A.4 LLM Judge Evaluation Protocol
The LLM-judge protocol evaluates early behavior, interrupted partial answers, and core responses separately, while case studies illustrate timing, nested resumption, and interruption-control failures.
- LLM Judge Evaluation Protocol: Core-stage judging scores correctness and coverage against the current answer and extracts the earliest answer-bearing substring as the trigger phrase.For 1QnA, future-step spoilers and skipped current steps are penalized.
- LLM Judge Evaluation Protocol: Early-stage judging treats acknowledgments as neutral but flags premature substantive answers, guesses, future-step revelations, and definitive unsupported claims.
- LLM Judge Evaluation Protocol: Interrupted partial answers are scored for relevance, correctness, and usefulness without penalizing incompleteness.The interruption prompt uses the question, ground-truth answer, and already spoken assistant output.
- Case Study: In the proactive book-title case, AURA and MiniCPM-o answer after the book appears, while premature responses from Gemini and Qwen3.5-Omni incur early-hallucination penalties.AURA and MiniCPM-o obtain TP scores of 0.9343 and 0.8664, respectively.
- Case Study: In the nested case, MiniCPM-o and AURA resume the outer task after answering an inserted query, whereas Qwen3.5-Omni fails to resume it.Their NCCS scores are 0.7845 and 0.7593; Qwen3.5-Omni receives zero despite a valid inner answer.
- Case Study: The interruption case shows that useful partial content and stopping control must be considered together, because no spill can coexist with low partial quality.Gemini has PAQ 0.20 with no spill, while MiniCPM-o continues for about 23 seconds after interruption.
- Case Study: The breakfast-burrito case shows that continuous task assistance requires timely decisions across multiple slots, making early over-generation or missed events costly.All models struggle across the first four response slots, and the only nonzero shown performance is delayed and partial.