PaperlayerView exact revision on arXiv

Source-linked AI summary

See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models

Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, Yong Jae Lee

arXiv:2512.02231v2cs.CVcs.AIcs.LG

TL;DR

Fine-grained reasoning about who speaks, what is said, and when remains rarely tested by video benchmarks. AV-SpeakerBench evaluates this through speaker-centered, fusion-grounded questions, and Gemini 2.5 Pro achieves the benchmark’s best results.

  • Problem

    Existing video benchmarks rarely test whether models jointly identify who is speaking, what is said, and the visual context, often relying on vision-solvable questions or coarse speech categories.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    AV-SpeakerBench centers questions on speakers, embeds temporal audiovisual dependencies in their wording, and uses expert-reviewed clips and annotations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    73.04% overall accuracy was achieved by Gemini 2.5 Pro (thinking), which outperformed all other models on 11 of 12 tasks.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Speaker-centered multimodal reasoning remains a core barrier to human-level audiovisual understanding.

    Source: §1 · p. 2

    Some questions may be answerable from visual cues alone, which the benchmark treats as in-scope rather than penalizing.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.

1. Introduction

AV-SpeakerBench addresses the gap in evaluating whether multimodal models can jointly reason about who speaks, what is said, and when it occurs in real-world video. Its speaker-centered, fusion-dependent questions and expert-reviewed annotations target fine-grained audiovisual understanding.

Motivation

  • Existing speaker datasets use closed-set labels or frame-level supervision, while video QA often permits single-modality solutions or tests coarse matching rather than speaker-level audiovisual reasoning.Consequently, benchmarks do not systematically test whether models jointly identify who is speaking, what is said, and the visual context.Source: §1 · p. 1

Speaker-centric task formulation

  • Each question treats speakers as the reasoning unit and asks models to resolve who speaks, when, and in what visual context amid multiple visible individuals and varied conversations.This shifts evaluation from scene-level understanding to human-centered audiovisual grounding.Source: §1 · p. 2

Fusion-driven question design

  • Four-choice questions jointly encode auditory and visual cues, requiring cross-modal fusion to link speech with visible speakers, visual events, or other voices in a shared scene.The design makes audiovisual coordination necessary to solve each item.Source: §1 · p. 2

High-quality human annotation

  • Clips are manually selected and temporally segmented around speech-driven interaction, and questions undergo multi-stage expert review for temporal precision and cross-modal validity.Annotators select segments requiring audiovisual reasoning, such as aligning an utterance with its visible speaker.Source: §1 · p. 2

Dataset summary and evaluation scope

  • AV-SpeakerBench contains 3,212 curated question–answer pairs across 12 task types spanning temporal localization, speaker identification, speech-content reasoning, utterance counting, and paralinguistic comparison.Together, the tasks evaluate recognition, grounding, and temporal reasoning across audio and vision rather than reliance on static or unimodal cues.Source: §1 · p. 2

Experimental findings and implications

  • Gemini 2.5 Pro is strongest overall, while Qwen3-Omni-30B reaches parity with Gemini 2.0 Flash but gains less from audio; a human-accuracy gap remains.Gemini 2.5 Pro gains 10–20% from audio inputs, whereas Qwen3-Omni’s improvements are modest or sometimes negative; audio perception and temporal grounding dominate errors.Source: §1 · p. 2

2. Related Work

Existing multimodal and audiovisual benchmarks emphasize broad video abilities, low-level speech labels, or scene and event matching, leaving fine-grained reasoning that binds spoken content to visible speakers under-evaluated; AV-SpeakerBench targets this gap.

Multimodal Understanding Benchmarks

  • Existing video benchmarks cover varied abilities but rarely require fine-grained visual–speech integration or speaker-centric reasoning.AV-SpeakerBench is introduced as a dedicated benchmark for speaker-level audiovisual understanding.Source: §2 · p. 2

Audiovisual Understanding Benchmarks

  • Speech resources target speaker identity, transcription, or speaking and audibility labels, but their closed-set labels or low-level annotations do not test open-ended MLLM reasoning.The cited examples include VoxCeleb [7] [41], LRS/LRS3 [2], and AVA-ActiveSpeaker / ASW [24] [45].Source: §2 · p. 2
  • VGGSounder [66], AV-Odyssey [18], WorldSense, OmniBench, and Daily-Omni emphasize event classification, audiovisual matching, temporal reconstruction, or scene understanding rather than speaker identity and spoken utterances.These tasks do not center human speech or bind it to visible speakers.Source: §2 · pp. 2–3
  • Across these benchmarks, audio characterizes scenes, events, or background context rather than binding speech to visible speakers, which AV-SpeakerBench explicitly requires.Its speaker-centric formulation aligns spoken content with the people who produce it.Source: §2 · p. 3

Multimodal Large Language Models

  • Despite progress from image–text and video-oriented MLLMs to omni-modal systems, evaluations largely test visual QA or coarse audiovisual matching, not who speaks, what is said, and when.AV-SpeakerBench assesses these speaker-centric capabilities in natural multi-speaker video.Source: §2 · p. 3

3. AV-SpeakerBench

AV-SpeakerBench addresses the limits of visually solvable or coarse speech benchmarks by testing speaker-centric audiovisual reasoning about who speaks, what is said, and when. Its questions and curated clips are designed to require temporally grounded integration of visual and auditory cues.

3. AV-SpeakerBench

  • AV-SpeakerBench targets a gap in prior benchmarks, which often allow visual-only answers or represent speech only through coarse categories rather than speaker-specific and linguistic content.The benchmark instead centers questions on reasoning about who speaks and what is said in natural audiovisual scenes.Source: §3 · p. 3

3.1. Fusion-Driven Question Design

  • Questions use a unified multiple-choice format and embed dependencies linking speech to visible identities, using one modality to localize the other, or coordinating cues across speakers.Successful answers require aligning voices, appearances, and contextual events over time, rather than relying on static visual or textual shortcuts.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Dataset Construction

  • Researchers select 5–30-second clips with meaningful speaker and temporal variation, then expert annotators create and multi-stage review questions to remove ambiguous or trivially solvable items.Videos come from YouTube, chiefly movie clips, as well as game shows, interviews, podcasts, and vlogs; a separate researcher reviews each item, a language model polishes it, and at least two additional researchers verify it.Source: §3.2 · pp. 4–5

3.3. Dataset Statistics

  • 3,212 MCQ pairs span 12 speaker-centric task types, with at least 200 validated examples per task; speaker-related tasks are more numerous, and evaluations report per-task accuracy.Speaker-related tasks support a wider range of reasoning patterns, while Figure 2 summarizes question-type, video-duration, and speaker-count distributions.Source: Figure 2 · p. 4, §3.3 · p. 5

3.4. Interpretation and Scope of Evaluation

  • The benchmark evaluates audiovisual integration when both modalities are available but does not penalize correct answers inferred from visual cues alone.The authors regard such inference, for example from mouth movement or gestures, as natural; they report that audio information consistently simplifies reasoning and boosts accuracy.Source: §3.4 · p. 5

4. Experiments

AV-SpeakerBench evaluates fine-grained audiovisual speaker reasoning across native audio–video models, and results show that even the strongest system remains well below human accuracy. Performance differences point to audiovisual fusion and temporal reasoning as persistent challenges, especially in visually complex multi-person scenes.

4.1. Experiment Setup

  • The evaluation compares proprietary and open-source models using native audio–video interfaces, with frame sampling following each model’s default temporal policy.The proprietary set includes Gemini 2.5 Flash thinking and non-thinking variants and Gemini 2.5 Pro thinking; the open-source set spans omni-modal and video-first architectures. Sampling ranges from fixed 5- or 8-frame inputs to 1-fps policies, with uniform sampling unless otherwise noted.Source: §4.1 · pp. 5–6

4.2. Results

  • Human evaluation reaches 93.74% overall accuracy, while Gemini 2.5 Pro reaches 73.04%, leads on 11 of 12 tasks, and remains over 20 percentage points below humans.The full evaluation reports MCQ accuracy; the results indicate substantial room to improve fine-grained, temporally grounded speaker-centric reasoning.Source: Table 2 · p. 6, §4.2 · p. 6
  • Qwen3-Omni-30B is the strongest open-source model, slightly surpassing Gemini 2.0 Flash but remaining far behind the Gemini 2.5 family.Recent Omni-series models show substantial improvement over earlier open models, which often approach random-guessing accuracy; Qwen3-Omni-30B is reported in its non-thinking variant because its thinking mode takes about five minutes per query.Source: §4.1 · pp. 5–6, §4.2 · pp. 6–7

4.3. Analysis

  • With audio and vision, Gemini 2.5 Pro gains roughly 10–20 percentage points across tasks, while Qwen3-Omni-30B gains less or even declines on some tasks.The analysis attributes Gemini’s advantage primarily to stronger temporal alignment and cross-modal integration, and argues that progress depends on fusion robustness rather than only scaling unimodal perception.Source: §4.3 · p. 7, §4.2 · p. 7
  • Some questions can be answered from visual cues such as mouth motion, gaze, and gestures, but audio resolves ambiguities that vision alone can leave unresolved.The paper treats these cues as natural properties of communication rather than dataset bias and does not penalize unimodal solutions; its examples show both a misleading gesture-based inference and correction after audio is provided.Source: §4.3 · p. 7, Figure 4 · p. 8
  • Model accuracy decreases as the number of visible people increases, challenging speaker identification, temporal association, and audiovisual grounding.Questions are regrouped by the number of visible faces for this analysis.Source: Table 3 · p. 7, §4.3 · p. 8
  • Most failures arise from audio perception and temporal grounding, including difficulty with overlapping speech and anchoring reasoning to the requested time.The analysis distinguishes visual/audio perception, cross-modal attribution, temporal grounding, and temporal localization errors; Figure 3b summarizes their distribution.Source: §4.2 · p. 7, §4.3 · p. 8

5. Conclusion

AV-SpeakerBench evaluates fine-grained, speaker-centric audiovisual reasoning in real-world video by requiring models to integrate auditory and visual cues. Experiments show strong fusion and temporal grounding in Gemini 2.5 Pro, while other models show multimodal-integration limitations.

5. Conclusion

  • AV-SpeakerBench centers human speakers and requires integrating auditory and visual cues, unlike prior multimodal datasets focused on scene- or event-level understanding.It targets fine-grained audiovisual reasoning in real-world video.Source: §5 · p. 8
  • Gemini 2.5 Pro exhibits strong fusion and temporal grounding, while other models still face clear limitations in multimodal integration.The conclusion underscores the need for more robust fusion mechanisms in future multimodal systems.Source: §5 · p. 8
  • AV-SpeakerBench is intended to provide a foundation for advancing models toward jointly reasoning about language, vision, and speech within the same conversational context.Source: §5 · p. 8

Supplementary Material … A.2.1. General Guidelines

The supplementary material describes how audiovisual questions are collected and standardized, including the annotation interface, workflow, and constraints on video selection, question design, and clip duration.

A. Annotation Details

  • The annotation-details section describes the interface, consistency guidelines, experienced-annotator selection criteria, and a trivially solvable example intended to clarify the task and set a lower bound on difficulty.The supplied passage outlines these topics but does not provide the annotator-selection criteria or example itself.Source: A. Annotation Details · p. 13

A.1. Annotation Interface

  • The web interface presents each example as a card with a video player, metadata, approval status, question, and four answer options.Annotators enter start and end times, select an answer, and explain it using visual or auditory evidence; reviewers use the explanation and question materials to decide whether to refine or approve it.Source: §A.1 · p. 13

A.2.1. General Guidelines

  • Annotators select a contiguous 5–30-second segment and write one task-specific multiple-choice question with an answer and brief justification.The workflow uses English-language YouTube videos and requires a single question for each selected clip segment.Source: §A.2.1 · p. 13
  • Videos must be in English, avoid persistent subtitles that reveal speech, and exclude extreme or explicit violence or gore.Mild or non-gratuitous physical conflict is allowed, while highly graphic or disturbing content should be avoided.Source: §A.2.1 · p. 13
  • The queried person must be clearly visible in the foreground and visually distinguishable, with attributes chosen to separate them from others in the clip.Suggested discriminative attributes include clothing, position, and actions.Source: §A.2.1 · p. 13
  • Answer options must draw on entities, actions, or speech events from the same clip, using plausible, balanced attribute combinations when candidates are scarce.Distractors may recombine visible or auditory attributes such as clothes, positions, or phrases.Source: §A.2.1 · p. 13
  • Each question is localized to a contiguous 5–30-second window, trimmed to the shortest span that still supports correct reasoning.The appropriate duration depends on the task’s requirements.Source: §A.2.1 · p. 13

A.2.2. Task-specific Guideline · A.3. Annotator Selection · A.4. Eliminating Trivially Solvable Questions

The guidelines use task-specific templates and trained expert annotators, then filter questions that can be answered without localized audiovisual evidence. Annotation also records temporal grounding and justifications to support validation.

A.2.2. Task-specific Guideline

  • Task-specific templates guide multiple-choice question construction, with a small initial batch soliciting feedback before annotation scales up.The requirements are provided in Table 6.Source: §A.2.2 · p. 13

A.3. Annotator Selection

  • Experienced researchers with backgrounds in video understanding, speech processing, or multimodal LLMs select informative clips and provide labels.The paper argues that this expertise supports label quality and reliability beyond what crowd-sourced annotation could provide.Source: §A.3 · p. 13
  • Researchers are trained on the full task taxonomy so they can distinguish trivial cases from questions requiring joint audiovisual inference.Training is intended to help annotators apply task definitions precisely.Source: §A.3 · p. 13

A.4. Eliminating Trivially Solvable Questions

  • Questions answerable without localized temporal or spatial cues are removed because they do not test fine-grained audiovisual reasoning.Such items may be obvious from a property that holds throughout the clip or from question phrasing.Source: §A.4 · pp. 13–14
  • The rate-comparison annotation interface shows the clip, metadata, question, choices, and selected response, while requiring a judgment window and brief justification.The examples demonstrate temporal grounding for different speech-rate comparisons.Source: Figure 5 · p. 14
  • Filtering targets persistent speaker patterns, constant scene composition, and globally obvious answers that make moment-specific questions solvable without checking the referenced event.Examples include a continuously speaking person or an unchanged count of visible people throughout a segment.Source: §A.4 · p. 14
  • Always-on burned-in captions make speech-content questions trivial because answers can be read rather than inferred through speech recognition or audiovisual alignment.These items therefore fail to evaluate the intended modality.Source: §A.4 · p. 14
  • Figure 6 illustrates removed items where a single visible person or verbatim burned-in captions reveal answers without grounding to the utterance or listening to the audio.Both examples bypass the intended audiovisual reasoning.Source: §A.4 · p. 14, Figure 6 · p. 15

B. Detailed Evaluation · B.1. Evaluation Prompt · B.2. Human Performance

The appendix details a unified model-evaluation prompt and a separate human evaluation designed to provide a comparable accuracy reference. The human protocol separates evaluation from dataset annotation and relies on audiovisual information alone.

B. Detailed Evaluation

  • The detailed evaluation appendix specifies model prompts and answer parsing, then describes human performance measurement as a reference ceiling for benchmark accuracy.The human protocol covers recruitment, instructions, and response aggregation.Source: B. Detailed Evaluation · p. 15

B.1. Evaluation Prompt

  • Models receive uniformly sampled video frames, the full audio track, a fixed instruction, and a four-option multiple-choice question under one prompt across tasks.Figure 7 illustrates the prompt format.Source: §B.1 · p. 15, Figure 7 · p. 15
  • Models must answer with only A–D; invalid-option responses count as incorrect, and standardized instructions support deterministic results across sub-benchmarks, following prior work [14].Predictions are mapped to their corresponding option.Source: §B.1 · p. 15

B.2. Human Performance

  • Ten English-proficient undergraduate and master’s students, none involved in dataset authoring or review, each answered roughly 300 questions for broad, non-overlapping test-set coverage.This decoupling is intended to measure task difficulty without annotator familiarity.Source: §B.2 · p. 15
  • Human evaluators watched clips and answered the model’s multiple-choice questions without transcripts, subtitles, or other auxiliary text; the interface also asked for the number of visible people.Playback controls and a visible-person count prompt were provided.Source: §B.2 · p. 15, Figure 8 · p. 16
  • Questions appeared in random order, with no opportunity to revisit answers or access ground truth or other participants’ responses; optional issue tags were unused and excluded from accuracy.The tags supported sanity checks for question or video mismatches; no evaluator submitted one.Source: §B.2 · pp. 15–16
  • The human protocol is independent of annotation, uses only audiovisual information, and keeps the multiple-choice setting directly comparable to model predictions.The interface did not show transcripts or subtitles.Source: §B.2 · pp. 15–16

C. Qualitative Analysis by Error Pattern · D. Question Examples

The qualitative analysis uses Gemini 2.5 Pro reasoning traces to expose four audiovisual failure patterns, while the question panels show the benchmark’s varied task formulations and paired audio–visual input. Together, the sections illustrate speaker identity, visual context, speech content, and acoustic attributes as targets of reasoning.

C. Qualitative Analysis by Error Pattern

  • The model misattributes speech to a visible man and hallucinates an off-screen speaker, showing failure to align voice evidence with visual identity.The error leads to an incorrect speaker attribution and an over-count of speakers.Source: C. Qualitative Analysis by Error Pattern · p. 16
  • The model misdescribes a woman’s clothing and selects the wrong woman as having the lowest pitch, with errors in vision and audio reinforcing each other.It describes a nonexistent woman in a dark green top instead of the woman in a blue dress, then identifies the khaki-sweater woman as lowest-pitched.Source: C. Qualitative Analysis by Error Pattern · p. 16
  • The model finds the requested speaking moment but counts people appearing before and after the specified frame, aggregating visual evidence beyond the temporal window.The question anchors the count to the frame when a man holding a cup starts a phrase.Source: C. Qualitative Analysis by Error Pattern · p. 16
  • For speech counting up to an action, the model places the action around 00:10 instead of approximately 00:08, shifting the cutoff and changing which utterances count.The example counts “I remember it” from the clip’s beginning until a man in a blue sweater puts on headphones.Source: C. Qualitative Analysis by Error Pattern · p. 16
  • The four representative failures are cross-modality attribution, audio and visual perception, temporal grounding, and temporal localization.Figure 9 examines Gemini 2.5 Pro’s reasoning traces when it has access to the full video and audio segment.Source: C. Qualitative Analysis by Error Pattern · p. 16, Figure 9 · p. 17

D. Question Examples

  • Question panels pair video frames, an audio waveform, a natural-language question, and four options; evaluation uses accuracy in choosing the correct answer.The examples make the task definitions concrete across the benchmark’s task groups.Source: D. Question Examples · p. 17
  • Speaker Detection, Recognition, and Counting test whether models associate speech segments with visible speakers, including who speaks, when, and how many distinct speakers are active.These speaker-centric examples are grouped in Figure 11.Source: D. Question Examples · p. 17
  • Other panels cover Activity Recognition and visually grounded counting alongside speech recognition, counting, duration, and rate, pitch, and intensity comparisons.These examples couple speech with a person’s actions or context, or require recovering phrases, comparing speech durations, and using audio to compare how fast, high, or loud people speak.Source: D. Question Examples · pp. 17–18

E. Performance of Gemini 3 Pro on AV-SpeakerBench

Gemini 3 Pro improves on Gemini 2.5 Pro on AV-SpeakerBench, especially on fine-grained audiovisual tasks, but persistent weaknesses and a human-performance gap limit evidence of robust, generalizable reasoning.

E. Performance of Gemini 3 Pro on AV-SpeakerBench

  • 77.62% average accuracy for Gemini 3 Pro exceeds Gemini 2.5 Pro’s 73.04% by +4.6 points on AV-SpeakerBench.Table 4 reports the comparison.Source: E. Performance of Gemini 3 Pro on… · p. 18
  • Gemini 3 Pro’s improvements are broad and most pronounced in speaker-centric understanding, visual reasoning, speech recognition, and prosody-related perception.Source: E. Performance of Gemini 3 Pro on… · p. 18
  • Counting and speech-attribute reasoning remain far from solved, and Gemini 3 Pro’s 77.62% accuracy remains below human performance of 93.74%.Despite meaningful progress, Gemini 3 Pro does not yet demonstrate robust and generalizable audiovisual reasoning.Source: E. Performance of Gemini 3 Pro on… · p. 18

F. Evaluation beyond default frame sampling.

Under fixed 8- and 16-frame sampling, accuracy was slightly lower, but model rankings were unchanged. The strongest models already use a similar default policy of 1 fps with a frame cap.

F. Evaluation beyond default frame sampling.

G. Audio necessity vs. vision-only solutions. · H. Ethical Statement

AV-SpeakerBench is designed to require audiovisual fusion: humans generally cannot solve its questions from vision alone, while audiovisual input yields near-ceiling performance. The paper also describes safeguards for its evaluation-only YouTube dataset and specifies audio-anchored task designs.

G. Audio necessity vs. vision-only solutions.

  • 72.7% versus 73.04%: Gemini 2.5 Pro performs nearly as well on audio-required questions as overall, while human evaluation finds vision-only insufficient and audiovisual input near-ceiling.The same human-evaluation trend holds across all 12 tasks; some questions may still be inferred from visual conversational cues.Source: G. Audio necessity vs. vision-only… · p. 18

H. Ethical Statement

  • The dataset uses public YouTube videos with precise timestamps and human annotations for evaluation only; its CC BY-NC-SA 4.0 license restricts use and permits removal requests.The license limits use to non-commercial research and prohibits facial recognition, surveillance, or biometric identification; requested segments will be withdrawn from future releases.Source: H. Ethical Statement · pp. 18–19
  • Activity recognition links speakers’ actions before, after, or during an utterance, while visual counting counts visible people relative to an audio-anchored utterance.Questions use short utterances as temporal anchors and require clearly visible entities or specified activity choices.Source: H. Ethical Statement · p. 25
  • Speech recognition tests spoken content around a visual event, while speech counting measures phrase or keyword mentions within event-bounded intervals.Speech-recognition options include one matching transcript; counting questions use numerical choices and countable, separated utterances.Source: H. Ethical Statement · pp. 25–26
  • Speech-duration questions compare speakers’ relative speaking time over event-anchored intervals, requiring a perceptible duration difference of at least 1–2 seconds.The task uses clips where at least two people speak for clearly different durations.Source: H. Ethical Statement · p. 25
  • Speech-attribute tasks compare speakers’ pitch, rate, and loudness, using clips where these differences are clear and avoiding loudness confounded by microphone distance.Questions ask which speaker has the highest or lowest pitch, speech rate, or voice intensity.Source: H. Ethical Statement · pp. 25–26

Related papers

Loading 2512.02231v2…