Source-linked AI summary
Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction
Chaoqun He, Mingyang Xiang, Yingjing Xu, Bokai Xu, Junbo Cui, Jie Zhou, Yuan Yao, Lijie Wen
TL;DR
Most MLLMs and benchmarks remain centered on offline video processing, leaving comprehensive evaluation of real-time duplex interaction unresolved. Omni-DuplexEval introduces two open-ended scenarios, a 660-video benchmark across 9 tasks, and an LLM-as-a-Judge framework for content and timing. Experiments find substantial limitations: the best model scores 39.6% overall and 20.0% on Proactive Reminder.
Problem
Most MLLMs are evaluated offline, and existing benchmarks do not comprehensively assess continuous responses, temporal alignment, and response timing in real-time duplex interaction.
Method
Omni-DuplexEval combines Real-Time Description and Proactive Reminder across 660 videos and 9 tasks, using open-ended questions, human annotations, and LLM-as-a-Judge evaluation.
Results
39.6% overall is the best model performance, while Proactive Reminder reaches only 20.0%; models also struggle with completeness–timeliness balance and response timing.
Takeaways & Limitations
Current duplex models remain far from human-level real-time interaction and struggle particularly with deciding when to respond.
Takeaways & Limitations
The benchmark focuses mainly on short streaming interactions, its LLM-as-a-Judge may exhibit open-ended evaluation biases, and few duplex models are evaluated.
Abstract
from arXiv · showhide
Real-time duplex interaction is essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existing multimodal large language models (MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensive benchmark or automatic evaluation method for this setting. To address this gap, we propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. The benchmark consists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce an automatic evaluation framework based on LLM-as-a-Judge, which enables systematic assessment by jointly evaluating response-content alignment and response timing through timestamp-aware and sequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.
1 Introduction
Existing MLLM benchmarks largely evaluate offline video understanding, leaving real-time duplex interaction—continuous adaptation and appropriate response timing—insufficiently measured. Omni-DuplexEval addresses this gap with two scenarios, an open-ended benchmark, and an automatic evaluator, while experiments expose substantial model limitations.
- Most MLLMs process the entire video before responding, unlike real-world interaction where perception and response are coupled continuously.
- Existing streaming benchmarks often use multiple-choice questions or simple responses, limiting assessment of temporal alignment, continuous adaptation, and realistic response quality.
- Omni-DuplexEval evaluates continuous responses to evolving video content and proactive decisions about when to respond and what to say.
- The benchmark contains 660 samples with open-ended questions, detailed human annotations, and 9 tasks spanning entertainment, lifestyle, education, and other real-world domains.
- Its LLM-as-a-Judge framework jointly evaluates semantic correctness and response timing beyond traditional final-answer-based evaluation.
- Models show a completeness–timeliness trade-off in Real-Time Description, while Proactive Reminder performance remains low because models struggle to respond at appropriate moments.Models remain silent for approximately 50–60% of the video duration in Real-Time Description, and the best Proactive Reminder model achieves only 20.0%.
2 Related Works
Prior work has advanced streaming, interactive, proactive, and omni-modal video evaluation, but existing benchmarks do not jointly assess continuous generation, temporal alignment, and autonomous response timing. Omni-DuplexEval unifies these dimensions around evaluating both what to say and when to say it.
- Recent MLLMs and streaming systems have moved from offline processing toward incremental generation and full-duplex multimodal interaction.
- Traditional video benchmarks cover increasingly complex perception and reasoning tasks, including long-form, multi-task, ego-centric, and activity understanding.
- Streaming and real-time benchmarks have introduced sequential inputs, interactive tasks, event-driven reasoning, and proactive evaluation, but often emphasize final response quality.
- Omni-modal benchmarks evaluate tightly coupled visual-audio reasoning, while hallucination and spoken-interaction benchmarks target specialized evaluation problems.
- Existing benchmarks largely use discrete question answering and separate response timing from content correctness, leaving continuous temporally aligned generation insufficiently evaluated.
- Omni-DuplexEval unifies evaluation of response content and timing for real-time duplex interaction.
3 Omni-DuplexEval
Omni-DuplexEval organizes real-time duplex evaluation around continuous description and event-driven reminders, using a 660-video dataset and an LLM-as-a-Judge pipeline that assesses content and timing.
- Benchmark Scenarios: The benchmark evaluates two complementary response patterns: continuous, time-aligned description and event-driven reminders triggered at appropriate moments.Real-Time Description tracks evolving video content, while Proactive Reminder identifies relevant events and determines when to respond.
- Benchmark Scenarios: Real-Time Description contains six subtasks covering counting, interaction relations, omni-modal synthesis, world knowledge, OCR, and fine-grained movement.These tasks require models to track changing visual and auditory information over time.
- Benchmark Scenarios: Proactive Reminder contains Event Reminder, Post-Event Reminder, and Correction subtasks for future events, repeated past events, and incorrect descriptions.The scenario requires deciding both when to respond and what to say.
- Dataset Construction: 660 videos provide human-curated open-ended question–answer annotations across diverse domains, with videos under one minute and an average duration of 34 seconds.The dataset emphasizes videos with clear temporal dynamics and omni-modal signals, while excluding static or low-information content.
- Evaluation Pipeline: Iterative evaluation-framework design achieved strong agreement with human judgments.The paper reports detailed ablations and comparisons with human annotations in Appendix B.
4 Experiments
Experiments evaluate duplex models and humans on Omni-DuplexEval, showing substantial performance gaps and distinct weaknesses in reasoning, holistic consistency, and response timing.
- Experimental setup: Model outputs are recorded under native duplex protocols, enabling evaluation of response timing and interaction behavior during streaming inference.Experiments include LiveCC, MMDuet2, StreamingVLM, and MiniCPM-o 4.5; all experiments use a single NVIDIA A100 GPU.
- Main results: 39.6 overall versus 81.8 for Human-Duplex, with performance higher on Real-Time Description than Proactive Reminder.MiniCPM-o 4.5 consistently outperforms other models, but all systems remain far from human-level real-time interaction.
- Fine-grained analysis: 68.6 on OCR contrasts with 51.4 for Counting for MiniCPM-o 4.5, revealing stronger perception than structured reasoning.Interaction Relationships and World Knowledge also receive lower scores, suggesting difficulty integrating dynamic context into coherent reasoning.
- Response behavior: Sparse, intermittent outputs preserve local temporal alignment but underperform in Content Consistency and fail to capture continuous video context.Models remain silent for large portions of videos, producing fragmented responses that limit holistic understanding.
- Proactive Reminder: 20.0 is the best Proactive Reminder score, with models often failing to trigger responses at appropriate times.MiniCPM-o 4.5 and MMDuet2 are dominated by No Answer cases, whereas LiveCC and StreamingVLM mainly produce Wrong outputs.
5 Conclusion and Future Work
Omni-DuplexEval introduces a benchmark for real-time full-duplex omni-modal interaction and identifies persistent challenges in timely, coherent responses. Future work may extend the benchmark to longer, more complex interactions and richer modalities.
- Conclusion: Omni-DuplexEval includes two scenarios, nine tasks, 660 videos, and human-curated timestamp-level annotations for real-time full-duplex evaluation.Real-Time Description contains six tasks, while Proactive Reminder contains three.
- Conclusion: Models struggle to balance global content consistency with local temporal sensitivity and to determine when to respond in Proactive Reminder.These findings highlight the importance of real-time duplex interaction capabilities.
- Future Work: Future work may extend Omni-DuplexEval toward longer and more complex interaction settings.Future benchmarks may also cover richer modalities and broader forms of real-time interaction.
A Detailed Evaluation Protocols
The evaluation protocols define separate measures for content fidelity and temporal alignment in Real-Time Description and Proactive Reminder.
- Evaluation dimensions: Content Consistency measures factual consistency between model responses and video content while checking alignment with user instructions.
A.1.1 Evaluation Process
Content Consistency uses a deduction-based scoring process, while its evaluation prompt supplies the instruction, response, and reference annotations to a fine-grained evaluator.
- Scoring procedure: The evaluator starts at 3.00 and deducts a penalty for each identified error according to Table 5.
- Scoring procedure: The final score is at least 0.01, except completely empty or entirely irrelevant responses receive 0.00.
- Evaluation prompt: The Content Consistency prompt evaluates response accuracy using the original instruction, generated response, and reference annotations.It requests a fine-grained decimal score from 0.00 to 3.00 and JSON containing the score and reasoning.
- Temporal evaluation: Temporal Sensitivity evaluates alignment between generated text and the video’s temporal windows.
A.2.1 Evaluation Process
The evaluation process extracts timestamped responses around events, filters and scores temporal alignment, and uses task-specific LLM judgments for proactive reminders and corrections.
- Temporal Sensitivity: Temporal Sensitivity evaluates timestamped sentences through semantic filtering, multi-window sampling, multimodal context extraction, and LLM-based alignment scoring.Irrelevant sentences are excluded before scoring, and the sentence score is the maximum alignment across candidate windows.
- Proactive Reminder: Proactive Reminder extracts model responses within a fixed 10-second window after each annotated event and evaluates event identification and instruction consistency.Extracted responses are concatenated into an event-specific response segment before judgment.
- Proactive Reminder: Reminder judgments require explicit reference to the target event and a clear indication that it occurred; vague, ambiguous, or unrelated outputs fail.Successful reminders receive 1, while unsuccessful reminders receive 0.
- Correction: Correction judgments check whether outputs fix all user-specified errors while preserving consistent context, scoring complete corrections as successful.Missing errors, inconsistent corrections, or context mismatches are unsuccessful.
- Calibration: The framework was calibrated against human annotations using structured score combinations, with Spearman correlation as the alignment metric.The calibration set contains 7 video-question instances and 63 annotated answer samples.
B.3.1 Content Consistency
The authors iteratively tune Content Consistency and Temporal Sensitivity configurations, then report strong alignment between the resulting automatic scores and human judgments.
- Content Consistency: Content Consistency experiments vary frame sampling, the number of ground-truth references, and prompt refinements.The tested reference counts are 0, 1, and 2, while sampling compares 0.5 FPS with 0.3333 FPS.
- Content Consistency: The best Content Consistency alignment uses 2 ground-truth references with 0.3333 FPS sampling.This configuration is reported as the best result of the iteration.
- Temporal Sensitivity: Temporal Sensitivity experiments compare window strategies, sentence- versus action-level segmentation, video-frame versus oracle-text context, prompts, penalties, sampling rates, and window offsets.The explored sampling rates are 2 FPS and 3 FPS, and the penalty uses λ = 1.
- Final Configuration: The final framework adopts 2 ground-truth references at 0.3333 FPS for Content Consistency and four-window sentence-level Temporal Sensitivity with irrelevant-sentence penalties.Temporal prompts are refined against human re-annotations.
- Final Configuration: Spearman correlation exceeds 0.9 for Content Consistency and approaches 0.8 for Temporal Sensitivity.These correlations measure alignment between automatic evaluation and human judgments.
C.2 Implementation Details.
Inference experiments evaluate duplex models under streaming conditions, using model-specific generation settings and a single A100 GPU per run.
- Model Settings: MiniCPM-o 4.5 processes synchronized video and audio chunks in full-duplex streaming with a 20-token generation limit per chunk.Its average inference time is approximately 150–200 ms per multimodal chunk.
- Model Settings: LiveCC processes video at 2 FPS with roughly 400–500 ms inference latency per step for real-time commentary.The reported settings include a repetition penalty of 1.05 and an EOS base threshold of 0.0.
- Compute Resources: All inference experiments use one NVIDIA A100-SXM4 80GB GPU per evaluation run.The experiments are conducted on an internal cluster.
D Limitations
The benchmark is limited in interaction length, automatic-judge reliability, and model coverage, while broader deployment may create misuse and mistiming risks.
- Scope: The benchmark focuses mainly on short streaming interactions and does not fully capture long-term conversations requiring persistent memory or planning.This limits coverage of extended conversational scenarios.
- Evaluation: LLM-as-a-Judge evaluation may still exhibit bias in open-ended settings despite reference annotations and carefully designed prompts.The framework relies on automatic judgments for evaluation.
- Model Coverage: The number of evaluated duplex models remains limited because publicly available real-time multimodal systems are scarce.The authors expect future streaming MLLM advances to expand evaluation scope.
- Broader Impacts: Real-time multimodal systems could be misused for misleading live content, impersonation, or automated interaction at scale.Temporal decision failures may also produce inappropriate or mistimed responses in sensitive scenarios.
- Broader Impacts: The work focuses on evaluation rather than deployment and filters potentially unsafe or high-risk content during dataset construction.The authors report avoiding collection of personal sensitive information.