PaperlayerView exact revision on arXiv

Source-linked AI summary

See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models

Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, Yong Jae Lee

arXiv:2512.02231v2cs.CVcs.AIcs.LG

TL;DR

Existing video benchmarks often leave fine-grained human-speech reasoning underassessed because questions can be solved visually or use coarse speech labels. AV-SpeakerBench addresses this with 3,212 expert-curated, speaker-centric audiovisual questions whose semantics require temporal cross-modal alignment. Gemini models lead the evaluation, with Gemini 2.5 Pro strongest overall and Qwen3-Omni-30B approaching Gemini 2.0 Flash among open models.

  • Problem

    Existing benchmarks provide limited evidence about whether models can align who speaks, what is said, and when it occurs in natural audiovisual scenes.

  • Method

    AV-SpeakerBench uses expert-reviewed, speaker-centered multiple-choice questions whose construction requires audiovisual fusion and temporal grounding.

  • Results

    Gemini 2.5 Pro achieves the best overall performance, while Qwen3-Omni-30B reaches parity with Gemini 2.0 Flash but remains behind Gemini 2.5 Pro.

  • Takeaways & Limitations

    AV-SpeakerBench provides a foundation for advancing fine-grained audiovisual reasoning toward jointly contextualized language, vision, and speech understanding.

  • Takeaways & Limitations

    Strong models may sometimes infer answers from visual cues such as mouth movement or gestures without explicitly using audio.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.

1. Introduction

Existing multimodal benchmarks often permit vision-only answers and rarely test fine-grained audiovisual speaker reasoning. AV-SpeakerBench addresses this gap with speaker-centered, fusion-driven, expert-reviewed evaluation and shows a substantial performance gap between current models and human accuracy.

  • Motivation: Current benchmarks rarely evaluate whether models jointly determine who is speaking, what is said, and the visual context.Existing speaker datasets use closed-set or frame-level labels, while video QA often remains single-modality solvable or coarsely evaluates speech.
  • Benchmark design: AV-SpeakerBench treats human speakers as the core reasoning unit, requiring identity-based reasoning about who speaks, when, and under which visual context.Its real-world videos include multiple visible individuals and diverse conversational settings.
  • Benchmark design: Each four-choice question jointly encodes auditory and visual cues, requiring associations between spoken phrases, visible speakers, visual events, and multiple voices.The design makes cross-modal fusion part of the question and answer semantics rather than an optional strategy.
  • Dataset and annotation: 3,212 question–answer pairs across 12 task types evaluate speaker-centric temporal localization, identification, speech-content reasoning, counting, and paralinguistic comparison.Clips are manually selected and segmented, and questions undergo multi-stage expert review for temporal precision and cross-modal validity.
  • Experimental findings: Gemini 2.5 Pro leads overall, while Qwen3-Omni-30B reaches parity with Gemini 2.0 Flash but remains behind the strongest model.The reported analysis attributes Gemini 2.5 Pro’s advantage primarily to stronger audiovisual fusion, with 10–20% gains from audio inputs.

2. Related Work

Prior multimodal benchmarks cover broad video, audio, and audiovisual capabilities, but commonly use scene-level, event-level, matching, or low-level speech annotations. AV-SpeakerBench differs by binding spoken content to visible speakers through open-ended speaker-centric reasoning.

  • Multimodal understanding benchmarks: Existing video benchmarks span narrative, temporal, procedural, egocentric, world-modeling, and broad capability tasks, but their coverage is heterogeneous.Table 1 compares multimodal video QA benchmarks by modality and annotation type.
  • Audiovisual understanding benchmarks: Audiovisual speech datasets separately target speaker identity, transcription, or frame-level speaking and audibility labels rather than open-ended multimodal reasoning.These resources rely on closed-set labels or low-level annotations.
  • Audiovisual understanding benchmarks: Recent audiovisual benchmarks emphasize audio-event classification, audio–visual matching, narrated-image reconstruction, or high-level scene understanding instead of speaker identity and spoken utterances.Their tasks differ fundamentally from speaker-centric reasoning about human speech.
  • Benchmark gap: Across prior benchmarks, audio generally characterizes scenes, events, or background context rather than binding speech to the people who produce it.AV-SpeakerBench adopts the contrasting requirement of aligning spoken content with visible speakers.
  • Multimodal large language models: Omni-modal MLLMs unify audio, video, and language through shared cross-modal alignment, but existing evaluations largely emphasize broader multimodal capabilities.This leaves speaker-level audiovisual reasoning insufficiently assessed.

3. AV-SpeakerBench

AV-SpeakerBench is a 3,212-item benchmark designed around temporally grounded audiovisual fusion in natural human-speech scenes. Its questions, curated clips, expert annotations, and diverse task distribution target speaker identity, speech content, and temporal coordination, while recognizing that visual cues can sometimes suffice.

  • Fusion-Driven Question Design: Each question requires audiovisual evidence to reason over temporally ordered actions and utterances, with fusion embedded in the question and answer options.The benchmark links spoken phrases to visible identities and uses events in one modality to localize reasoning in the other.
  • Fusion-Driven Question Design: Figure 2 illustrates spoken-phrase grounding, visual-event conditioning, cross-modal temporal localization, and multi-speaker coordination, alongside distributions of duration, task type, and visual complexity.Visual complexity is measured by the number of unique visible people.
  • Data curation and annotation: Annotators select 5–30-second clips with meaningful audiovisual variation, compose four-option questions, and refine them through expert review using realistic within-clip distractors.Sampling avoids monologues or static dynamics that would make speaker- or temporal-grounded questions nearly deterministic.
  • Dataset summary: AV-SpeakerBench contains 3,212 multiple-choice pairs across 12 speaker-centric task types requiring coordinated multimodal reasoning.Tasks include temporal localization, speaker identification, speech-content retrieval, utterance counting, and paralinguistic comparison.
  • Dataset Statistics: The dataset maintains at least 200 validated examples per task and spans dyadic exchanges through multi-party conversations with varied interaction density.Speaker-related tasks contain more examples because temporal reasoning about who speaks and when can use cues from either modality.
  • Interpretation and Scope of Evaluation: Strong models may occasionally answer from mouth movements or gestures alone, so the benchmark evaluates audiovisual integration when audio is available without penalizing visual-only success.The authors report that audio consistently simplifies reasoning and boosts accuracy.

4. Experiments

AV-SpeakerBench evaluates native audio-video models on speaker-centric audiovisual reasoning, revealing a substantial human–model gap and a clear advantage for Gemini 2.5 Pro. Results attribute this advantage mainly to stronger multimodal fusion and temporal grounding, while errors increase with visual complexity and remain concentrated in audio and temporal reasoning.

  • Results: 73.04% overall accuracy leaves Gemini 2.5 Pro more than 20 percentage points below humans, who reach 93.74% on AV-SpeakerBench.Gemini 2.5 Pro is the strongest model overall, but current multimodal LLMs remain substantially below human performance.
  • Results: Gemini 2.5 Pro outperforms all other models on 11 of 12 tasks, while Qwen3-Omni-30B is the strongest open model and slightly surpasses Gemini 2.0 Flash.Earlier open-source models often approach random-guessing accuracy, whereas recent Omni models show non-trivial performance.
  • Analysis: Gemini 2.5 Pro gains roughly 10–20 percentage points from audio across tasks, whereas Qwen3-Omni-30B shows smaller or sometimes negative audio contributions.The comparison indicates more reliable audiovisual fusion for Gemini than for Qwen3-Omni-30B.
  • Analysis: Vision-only cues such as mouth motion and gestures can sometimes identify speakers, but audio resolves ambiguities that visual evidence alone can misinterpret.Gemini succeeds on some vision-only cases yet fails when gestures misleadingly suggest speech rate; audio corrects the ambiguity in the paired example.
  • Analysis: Most Gemini 2.5 Pro failures arise from audio perception and temporal grounding, including missed overlapping speakers, incorrect temporal relations, and misaligned segments.The error analysis distinguishes unimodal perception, cross-modal attribution, temporal grounding, and temporal localization errors.
  • Analysis: Models generally become less accurate as the number of visible people increases, creating challenges for speaker identification, temporal association, and audiovisual grounding.Visual complexity is operationalized by the number of visible people.

5. Conclusion

AV-SpeakerBench is introduced as a benchmark for fine-grained, speaker-centric audiovisual reasoning in real-world video. The experiments show strong fusion and temporal grounding for Gemini 2.5 Pro, while other models still have clear multimodal-integration limitations.

  • 5. Conclusion: AV-SpeakerBench centers evaluation on human speakers and explicitly requires integrating auditory and visual cues in real-world video.
  • 5. Conclusion: Gemini 2.5 Pro exhibits strong audiovisual fusion and temporal grounding, whereas other models retain clear limitations in multimodal integration.

Supplementary Material

The supplementary material describes a structured annotation workflow for creating clear, consistent audiovisual multiple-choice questions. It specifies the interface, clip-selection process, annotator guidelines, and constraints on video content and identity visibility.

  • Annotation Process: Annotators receive standardized guidelines and are selected for experience to promote consistent, high-quality audiovisual question creation.
  • Annotation Process: Annotators use a web interface containing video, metadata, a question, four answer options, and fields for recording the reasoning span.
  • Annotation Process: Each annotation uses a 5-to-30-second segment, one assigned-task multiple-choice question, the answer, and a brief justification.
  • Video Selection and Clip Duration: Selected videos must avoid persistent captions, extreme violence, and non-English content while keeping queried identities visibly distinguishable.

A.2.2. Task-specific Guideline

The annotation guidelines train experienced researchers to construct precise audiovisual questions and remove items solvable without localized multimodal evidence.

  • A.2.2. Task-specific Guideline: Experienced researchers trained on the full task taxonomy annotate clips to distinguish genuine joint audiovisual inference from trivial cases.Their prior expertise supports meaningful clip selection and precise application of task definitions.
  • A.2.2. Task-specific Guideline: The annotation interface records the selected answer, reasoning time span, and brief justification to validate temporal grounding.Rate-comparison examples include lowest- and highest-speech-rate judgments over specified intervals.
  • A.4. Eliminating Trivially Solvable Questions: Questions are filtered when persistent speaker patterns or constant scene composition make moment-specific answers globally obvious.Examples include one person speaking continuously or an unchanged number of visible people across the clip.
  • A.4. Eliminating Trivially Solvable Questions: Burned-in captions are removed because they allow speech-content questions to be answered by reading rather than audio-based reasoning.The same principle excludes items that fail to require the intended audiovisual capabilities.

B. Detailed Evaluation

The evaluation uses a standardized four-option video-and-audio prompt for models and an independent, transcript-free protocol for measuring human performance.

  • B.1. Model Evaluation: Models receive uniformly sampled frames, the full audio track, a fixed instruction, and four natural-language answer options labeled A–D.Predictions are parsed by option letter, and invalid responses count as incorrect.
  • B.1. Model Evaluation: A single letter-only instruction is applied across all sub-benchmarks to produce deterministic, standardized model evaluation.Responses must contain a valid A–D option.
  • B.2. Human Evaluation: Human evaluation is decoupled from annotation: ten English-proficient students independently answer roughly 300 questions each.Their assignments provide broad, non-overlapping coverage of the test set.
  • B.2. Human Evaluation: Human evaluators answer without transcripts or subtitles in the same four-choice setting, making performance directly comparable to model predictions.Optional issue tags support sanity checks but are excluded from human-accuracy computation.

C. Qualitative Analysis by Error Pattern

Qualitative analyses identify four failure patterns in Gemini 2.5 Pro: cross-modality attribution, perception, temporal grounding, and temporal localization.

  • Cross-modality attribution: Gemini 2.5 Pro can misattribute speech to the wrong visible person and hallucinate an additional speaker despite full audiovisual access.The failure reflects incorrect alignment between acoustic evidence and visual identity.
  • Audio and visual perception: The model may combine incorrect visual perception with incorrect acoustic comparison when identifying a speaker’s voice pitch.One example includes a nonexistent clothing description and an incorrect lowest-pitch judgment.
  • Temporal grounding: Temporal grounding errors occur when the model identifies the anchor moment but counts people appearing before or after the requested interval.The example concerns visibility at the moment a man begins speaking.
  • Temporal localization: Temporal localization errors shift the cutoff point for counting utterances, changing which speech events fall within the requested time range.The model localized the action around 00:10 instead of approximately 00:08.
  • Task coverage: AV-SpeakerBench spans speaker identity, actions, counting, speech content, duration, and prosodic attributes within a unified audiovisual QA framework.These tasks require associating speakers with speech, visual context, and fine-grained acoustic properties.

E. Performance of Gemini 3 Pro on AV-SpeakerBench

Gemini 3 Pro improves substantially over Gemini 2.5 Pro on AV-SpeakerBench, but counting, speech-attribute reasoning, and the human-performance gap remain unresolved.

  • E. Performance of Gemini 3 Pro on AV-SpeakerBench: 77.62% average accuracy lets Gemini 3 Pro outperform Gemini 2.5 Pro at 73.04% by +4.6 points.Table 4 reports broad improvements, especially for fine-grained audiovisual grounding and speech-related tasks.
  • E. Performance of Gemini 3 Pro on AV-SpeakerBench: Counting and speech-attribute reasoning remain far from solved, while human performance reaches 93.74%.The remaining gap limits the claim of robust, generalizable audiovisual reasoning.

F. Evaluation beyond default frame sampling.

Additional frame-sampling experiments show that fixed 8- or 16-frame inputs slightly reduce accuracy while preserving model rankings.

  • The additional experiments evaluate models with non-trivial AV-SpeakerBench accuracy under different frame-sampling densities.
  • Figure 10 compares performance under visual-only (V) and audiovisual (A+V) settings.
  • Slightly lower accuracy results from fixed 8- or 16-frame sampling, while the model ranking remains unchanged.The evaluated frame counts are widely adopted in audiovisual large language models.

G. Audio necessity vs. vision-only solutions.

AV-SpeakerBench is designed so that audiovisual input is generally necessary for its speaker-centered questions, rather than allowing reliable vision-only shortcuts. Its task suite combines speech, speaker, temporal, activity, and paralinguistic reasoning under curated evaluation conditions.

  • 72.7% on the audio-required subset versus 73.04% overall shows Gemini 2.5 Pro performs similarly when vision alone cannot answer.Across all 12 tasks, audiovisual input yields near-ceiling human performance, whereas vision-only solutions are generally insufficient.
  • Questions pair publicly available YouTube videos with precise timestamps and human-written annotations for evaluation rather than model training.
  • The CC BY-NC-SA 4.0 release restricts use to non-commercial research and prohibits facial recognition, surveillance, and biometric identification.Individuals in referenced videos may request removal, with corresponding segments withdrawn from future releases.
  • The benchmark spans speaker identification, speech content, temporal localization, counting, activity recognition, and speech attributes such as pitch, rate, and intensity.Task requirements use audio or visual anchors to condition questions on speakers, events, timing, or utterance properties.
Loading 2512.02231v2…