PaperlayerView exact revision on arXiv

Source-linked AI summary

SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum, Deval Pandya, Shaina Raza

arXiv:2601.21666v2cs.AIcs.CV

TL;DR

Existing benchmarks provide limited evidence about sequential native audio-video understanding and demographic variation in real-world interactions. SONIC-O1 addresses this gap with a human-verified benchmark spanning three tasks and demographic metadata, finding that temporal localization is especially difficult and disparities persist across groups.

  • Problem

    Existing audio-video benchmarks underuse native audio and largely omit group-wise demographic analysis, limiting evaluation of accurate and fair multimodal performance.

  • Method

    SONIC-O1 is a fully open-source, human-verified benchmark combining native audio-video input, summarization, MCQ reasoning, temporal localization, and demographic metadata across real-world interactions.

  • Results

    Closed-source models mostly outperform open-source alternatives, temporal localization is the most challenging task, and demographic disparities reach up to 21.4% R@0.5 across racial groups.

  • Takeaways & Limitations

    SONIC-O1 supports task-specific fairness evaluation and future work on stronger temporal reasoning and more equitable multimodal systems.

  • Takeaways & Limitations

    The English-focused, limited-scale benchmark may not generalize fully across cultures, languages, domains, modalities, or future model generations.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process sequential audio-video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified benchmark of 60 hours (231 clips) spanning 13 real-world conversational domains with 4,958 annotations and demographic metadata. SONIC-O1 evaluates three capabilities: open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Across closed- and open-source models, we find that the MCQ accuracy shows the smallest gap between model families, but the best closed-source model outperforms the best open-source model by 22.6% on temporal localization. We further observe accuracy gaps of up to 21.4% on temporal localization across demographic groups, indicating persistent disparities in model behaviour. SONIC-O1 provides an open evaluation suite for temporally grounded and demographically robust multimodal understanding. SONIC-O1 is publicly available for research: Project page (https://vectorinstitute.github.io/sonic-o1/), Dataset (https://huggingface.co/datasets/vector-institute/sonic-o1), GitHub (https://github.com/vectorinstitute/sonic-o1), Leaderboard (https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard).

1 Introduction

Existing audio-video benchmarks underuse native audio and rarely assess demographic variation, motivating SONIC-O1 as a human-verified benchmark for real-world omnimodal understanding across tasks and groups.

  • Existing benchmarks often treat audio as optional or replace it with transcripts, underexploring paralinguistic cues that shape speaker meaning and emotion.
  • Group-wise analysis across demographic groups is largely absent, limiting assessment of systematic variation in model performance with user characteristics.
  • SONIC-O1 combines native audio-video input, open-ended summarization, fine-grained MCQ reasoning, temporal localization, and demographic metadata.The benchmark contains 4,958 QA instances from approximately 60 hours of real-world interactions across 13 topics and five domains.
  • The benchmark evaluates global conversation comprehension, fine-grained reasoning, and identification of event timing through three complementary tasks.
  • SONIC-O1 is open-source, human-verified, and designed to reveal where omnimodal models succeed, fail, and diverge across demographic groups.

2 Related Work

Prior video benchmarks increasingly study temporal reasoning and longer videos, but commonly underuse native audio and omit systematic demographic analysis. SONIC-O1 addresses both gaps through integrated audio-video evaluation and group-wise assessment.

  • Its benchmark design also combines temporal understanding, audio-visual reasoning, social cue analysis, open-source access, and automatic and human annotations.
  • Video benchmarks have expanded from short-form temporal tasks to long-form understanding, including videos up to one hour.
  • Audio is often underutilized because evaluations default to frame- or subtitle-based setups when model support is limited.
  • Group-wise analysis is largely absent from existing benchmarks, preventing systematic measurement of demographic performance differences.
  • SONIC-O1 combines native audio-video evaluation, temporal localization, and demographic metadata for group-wise analysis.

3 SONIC-O1: Dataset and Benchmark Design

SONIC-O1 is a real-world, human-verified audio-video benchmark spanning diverse domains, durations, and demographics, with three tasks designed for summarization, reasoning, and temporal grounding.

  • 3.1 Data Collection: The dataset samples five high-stakes domains and 13 topics across short, medium, and long videos, while targeting demographic representation.Videos range from under five minutes to 20–60 minutes, and collection began from 2,237 candidates before licensing and quality filtering.
  • 3.1 Data Collection: SONIC-O1 uses real-world audio-video content with perceived demographic annotations to support fairness-oriented slice analysis.
  • 3.2 Tasks Annotation: Video summarization requires detailed narratives of key events, actions, and outcomes, using hierarchical chunking for videos longer than 10 minutes.The task contains 231 audio-video instances paired with human-verified reference summaries.
  • 3.2 Tasks Annotation: MCQs test fine-grained local understanding on segments up to three minutes using four candidate answers plus a Not enough evidence option.
  • 3.2 Tasks Annotation: Temporal localization requires start and end timestamps plus reasoning grounded in salient audio-visual evidence.Experts label times relative to segment starts, which are converted to absolute timestamps using each segment’s offset; the dataset contains 3,392 instances.
  • 3.3 Quality Assurance: Domain experts review every task, while AI-assisted drafts are verified, corrected, and adjudicated after removing ambiguous or insufficiently evidenced items.
  • 3.4 Evaluation Suite: The released evaluation suite unifies direct audio-video inputs, long-video segmentation and sampling policies, inference, and metric computation for reproducible extensions.

4 Experiments

Experiments evaluate commercial and open-source MLLMs across three tasks, demographic groups, video durations, input modalities, and frame sampling. Closed-source models generally lead, while temporal localization and longer videos expose the largest weaknesses and disparities.

  • Gemini 3.0 Pro leads overall with 81.4% MCQ accuracy, a 7.07 summarization judge score, and 25.4% R@0.5 temporal localization.
  • 23% separates Gemini 3.0 Pro from the best open-source model on temporal localization, while Qwen3-Omni is the strongest open-source model overall.
  • Temporal localization is the most challenging task, whereas summarization and MCQ are comparatively easier; smaller open-source models show up to 41.6% lower summarization scores and 6.7× lower R@0.5 than Qwen3-Omni.
  • Temporal localization shows the largest demographic variation, including racial-group R@0.5 values of 40.9% for Indigenous participants versus 19.5% for Black participants, while MCQ accuracy is relatively stable.
  • Performance declines as videos lengthen, with temporal localization most affected and MCQ relatively stable; smaller models show limited robustness to longer, more complex videos.
  • Adding audio benefits Qwen3-Omni by +10.3% MCQ accuracy, +2.38 summarization score, and +2.7% R@0.5, although gains vary by model.
  • Increasing sampled frames does not consistently improve performance, with summarization and MCQ remaining largely unchanged across tested frame counts.
  • Generated summaries exhibit distinct styles: Gemini 3.0 Pro is neutral and clinical, whereas Qwen3-Omni and UniMoE-2.0 are warmer and more supportive.

5 Conclusion

SONIC-O1 benchmarks real-world audio-video understanding through summarization, MCQ, and temporal localization. Results show closed-source advantages, task-specific demographic disparities, and a need for stronger temporal reasoning and fairer multimodal evaluation.

  • SONIC-O1 evaluates MLLMs on summarization, MCQ, and temporal localization through a real-world audio-video benchmark.
  • Closed-source models mostly outperform open-source alternatives, while temporal localization is the most challenging task.
  • Demographic gaps reach 21.4% R@0.5 across racial groups, while MCQ remains stable, showing that fairness evaluation must be task-specific.

Limitations

The study’s scope and evaluation design constrain how broadly its findings can be interpreted. Limitations include English-only interactions, restricted benchmark scale and task scope, annotation subjectivity, and several model-evaluation constraints.

  • SONIC-O1 focuses on English-language interactions, limiting cultural and linguistic coverage.
  • The benchmark is restricted in scale and task scope, so findings may not generalize fully to other domains, modalities, or future model generations.
  • Human-verified annotations retain some subjectivity, particularly for open-ended tasks.
  • Long audio is often truncated, context limits require video segmentation, and unequal demographic sample sizes may increase variance for underrepresented groups.

Ethical Considerations

SONIC-O1 frames demographic analysis as a way to identify and reduce potential harms, while warning that benchmark results may be misused when limitations are ignored.

  • Benchmark results could be taken out of context or selectively used in adversarial, reputational, or legal situations.
  • Demographic-gap analysis is intended to reduce potential harms rather than reinforce stereotypes, offend communities, or target any group.

A Data Collection Details

SONIC-O1 constructs a licensed, human-screened, demographically and topically broad video dataset through staged retrieval, filtering, sampling, and media processing. The collection emphasizes real-world interactions while imposing quality, safety, language, and licensing constraints.

  • Licensing: CC BY 4.0 licensing is required, while Standard License videos are excluded because redistribution is restricted.
  • Retrieval and coverage: Candidate videos are retrieved with topic and demographic query expansion, but final demographic annotations are produced later through human-verified review.
  • Coverage requirements: Videos typically range from 30 seconds to 60 minutes and are stratified across short, medium, and long duration ranges.
  • Coverage requirements: The dataset targets 13 topics across five domains, primarily English recordings, and speakers across specified race, gender, and age groups.
  • Filtering and selection: Two-stage filtering removes metadata outliers before domain experts screen candidates for relevance, quality, safety, and licensing.
  • Sampling: Balanced sampling caps topics at up to 25 videos when possible and stratifies selections across duration categories, while retaining all eligible videos for undersupplied topics.
  • Media processing: Accepted videos are capped at 1080p, paired with extracted stereo audio and available English captions, and packaged with JSON metadata for reproducibility.
  • Observed distribution: The resulting exposure distribution is moderately skewed toward male-presenting speakers and participants aged 40+, partly reflecting source-content composition and group co-occurrence.

D.3 Robustness and Adaptive Fallback Protocols

SONIC-O1 uses adaptive visual and audio fallback procedures to preserve evaluation under hardware and context constraints. Its protocols also distinguish segment-relative annotation from absolute-time inference for temporal localization.

  • Adaptive fallback: Models begin with architecture-specific maximum frame capacities and fall back to smaller budgets when memory or context limits cause failure.
  • Adaptive fallback: Audio fallback samples non-overlapping chunks uniformly across the full recording instead of truncating or cropping long tracks.
  • Adaptive fallback: The default audio limit is 64 ten-second chunks, with chunk limits halved alongside visual frames during retries.
  • Adaptive fallback: Ola and Baichuan-Omni 1.5 use 30-second chunks to match their Whisper-based audio encoders.
  • Evaluation protocol: The start-high, fall-back protocol aims to report each model’s maximum capability under specified GPU hardware constraints.
  • Temporal localization: Temporal-localization inference provides absolute segment boundaries, requiring predictions within the stated range rather than relative offsets.
  • Temporal localization: Human annotations use segment-relative timestamps that are converted by adding the segment start offset before evaluation.

E.1 Task 1: Video Summarization Metrics

Video summarization is evaluated with lexical, semantic, and judge-based metrics, alongside task-specific metrics for MCQ and temporal localization. Scores are aggregated across samples and topics, with failed predictions excluded.

  • Automatic metrics: ROUGE-L measures longest-common-subsequence F1 overlap between generated and reference summaries, reported as a percentage in [0, 100].
  • Automatic metrics: Text Similarity measures cosine similarity between sentence embeddings and is reported in [0, 1].
  • Aggregation: Summarization metrics are computed separately for detailed and short summaries, then averaged across the dataset.
  • Task metrics: MCQ evaluation reports answer accuracy and rationale quality, while temporal localization reports mIoU, Recall@θ, and temporal error metrics.
  • LLM judging: An LLM judge scores semantic correctness, factual completeness, and relevance on a 0–10 scale, accepting meaning-preserving paraphrases.
  • Aggregation: Metrics are first computed within topics and then averaged across topics to produce aggregate scores.
  • Evaluation handling: Samples without a generated prediction are excluded from metric computation, with successful evaluation counts reported.

F Detailed Results

SONIC-O1 reports broad benchmark results across summarization, MCQ answering, and temporal localization, with Gemini 3.0 Pro leading overall. Rationale quality can diverge from task execution, especially for temporal grounding.

  • Gemini 3.0 Pro leads summarization, MCQ accuracy, and temporal localization, scoring 7.07, 81.4%, and 26.6% mIoU respectively.It also achieves 25.4% R@0.5 on temporal localization.
  • Qwen3-Omni is the strongest open-source model for summarization, while VITA 1.5 produces the highest-quality open-source temporal rationales.VITA 1.5 receives a 3.91 judge score for temporal rationales.
  • Open-source models except Gemini show MAE values of 60.0–685.8 seconds, indicating systematic temporal reference frame hallucination.Models may process long videos in segments but fail to ground predictions in absolute timestamps.
  • Rationale quality does not perfectly track task performance, as models can produce coherent explanations while making substantial temporal localization errors.Qwen3-Omni performs competitively on temporal rationale similarity despite weaker execution than Gemini.
  • Per-topic breakdowns report model capabilities across 13 conversational domains and all three tasks using detailed automatic and judge-based metrics.Metrics include summarization, MCQ, temporal localization, and rationale quality measures.

G.1 Key Observations from Per-Topic Analysis

Per-topic results show substantial variation by conversational domain and expose a recurring temporal-grounding failure in open-source models. Models often understand events semantically while misplacing them on the absolute timeline.

  • Topic-specific performance variation: Summarization and MCQ performance varies substantially across topics, including Gemini’s 8.33 on job interviews versus 5.04 on Olympics.Qwen3-Omni’s MCQ accuracy similarly ranges from 77.0% on job interviews to 54.9% on transportation.
  • Temporal hallucination in open-source models: Gemini maintains temporal grounding with 7.8–23.5 seconds MAE and 15.9–33.5% R@0.5, while open-source models often treat each segment as starting at 0 seconds.This persists despite explicit prompting with absolute timestamps.
  • Model hierarchy: Temporal performance forms clear tiers: Gemini has 7.8–23.5 seconds MAE, Qwen3-Omni 38.5–95.7 seconds, and UniMoE-2.0/MiniCPM-o-2.6 often 300–1500 seconds.The pattern indicates weak temporal coherence across segments in most open-source models.
  • Rationale-accuracy disconnect: High rationale scores do not ensure accurate localization, as VITA 1.5 reaches 3.37–4.45 rationale scores but only 0.0–2.3% R@0.5.UniMoE-2.0 and MiniCPM-o-2.6 also produce coherent explanations alongside severe localization errors.
  • Professional vs. dynamic scenes: Models perform better on structured interactions than dynamic scenarios, with Gemini’s MAE increasing from 8.9–12.5 seconds to 13.6–22.6 seconds.Structured topics include patient-doctor, job interviews, and parent-teacher interactions; dynamic topics include emergency, transportation, and Olympics.
  • Temporal error analysis: The benchmark evaluates absolute temporal predictions using segment boundaries, IoU, MAE, fixed tolerances, and an error taxonomy for localization failures.The taxonomy includes relative-to-absolute mismatch, hallucination, timing shifts, wrong event at the right time, and other errors.
  • Formatting-error correction: Coordinate alignment improves UniMoE-2.0 by +1.14% R@0.5, correcting 7.6% of predictions, while most failures remain genuine temporal reasoning limitations.UniMoE-2.0’s dominant relative-to-absolute error rate is 39.4%.
  • Model-dependent failure patterns: Failure patterns differ by model: UniMoE-2.0 has 39.4% relative-to-absolute mismatches, while MiniCPM-o-2.6 has 26.8% hallucination and 37.5% late shifts.Qwen3-Omni, VITA-1.5, and Baichuan-Omni 1.5 are dominated by timing shifts.

I.1 Qualitative Analysis

Demographic analyses identify systematic performance disparities across race, gender, and age groups, with the largest failures appearing in temporal localization. Statistical tests support several recurring gaps while also qualifying their magnitude by task.

  • Demographic disparities are systematic, with the same groups consistently underperforming across models and tasks.The benchmark stratifies results by race, gender, and age.
  • Race disparities: Black participants receive lower summarization scores than Asian or Arab participants, including Gemini’s 6.02 versus 7.05 and Qwen’s 4.39 versus 5.95.Indigenous-participant results vary sharply across models.
  • Gender disparities: Female-participant videos yield higher performance across models and tasks, with summarization advantages ranging from +0.17 to +0.66.The passage attributes the uniform directionality to differences in processing voices and interaction styles.
  • Age disparities: Participants aged 40+ outperform those aged 18–24 in summarization, including Gemini’s 6.91 versus 6.28.The passage links this pattern to training data skewed toward formal, professional contexts.
  • Temporal localization failures: Temporal localization shows the most severe disparity: open-source models reach 0.0% R@0.5 for Indigenous participants while Gemini reaches 40.9%.For Black participants, Gemini reaches 19.5% R@0.5 rather than complete collapse.
  • Implications: The authors caution that demographic slice sizes are uneven, which can inflate variance for smaller groups.They also report larger robustness failures for reasoning-heavy MCQ and temporal localization than for generation-oriented summarization.
  • Statistical significance: The Black–Asian summarization gap is significant across all models, with p < 0.001 and rank-biserial effect sizes of |r| = 0.19–0.36.Within-topic controls confirm the gap in 5/8 models.
  • Statistical significance: Female participants outperform male participants in T1 summarization, while no meaningful gender effects appear in T2 or T3.Reported effects range from |r| = 0.09–0.19 across five models.

J.1 Key Observations

Models express empathy through distinct combinations of emotional vocabulary, perspective-taking, prosocial language, and warmth. These dimensions do not consistently align, and the authors report limited situational adaptation.

  • Gemini emphasizes emotional validation, OLA leads prosocial, affiliation, and insight language, and Baichuan-Omni 1.5 has the warmest tone.Their reported values are 4.35% Total Emotion, 2.72% Prosocial, and 60.09 Tone, respectively.
  • Cognitive vs. affective empathy: OLA and VITA-1.5 show strong insight scores but moderate or minimal negative-emotion language, separating cognitive perspective-taking from emotional validation.OLA reaches 3.84% Insight and VITA-1.5 3.51%, with negative emotion at 1.15% and 0.60%.
  • Small model limitations: Smaller models show weak empathic performance, with VideoLLaMA2 producing 0.62% prosocial language, 2.02% total emotion, and near-neutral tone.The passage characterizes these outputs as detached clinical descriptions.
  • Tone-emotion independence: Tone and emotional vocabulary are not correlated: Gemini combines high emotion with 46.16 tone, whereas Baichuan-Omni 1.5 combines 60.09 tone with moderate emotion.The authors describe warmth and emotional validation as orthogonal dimensions.
  • Implications: The authors conclude that current models lack nuanced empathic control and often use fixed strategies rather than adapting to situational demands.They propose separately controlling emotional validation, warmth, and perspective-taking.
  • Release and reproducibility: The released dataset removes personally identifiable information and supports academic non-commercial research under the Vector Institute License.The authors also release processing scripts for reproducibility.
Loading 2601.21666v2…