PaperlayerView exact revision on arXiv

Source-linked AI summary

MSU-Bench: Towards Speaker-Centric Understanding in Conversational Multi-Speaker Scenarios

Zhaokai Sun, Shuai Wang, Zhennan Lin, Chengyou Wang, Dehui Gao, Yuang Cao, Chunjiang He, Pan Zhou, Lei Xie

arXiv:2606.22868v1eess.AS

TL;DR

Existing speech benchmarks provide limited evaluation of speaker-centric understanding in realistic multi-speaker conversations. MSU-Bench combines a two-tier, 16-task benchmark with verified QA and diagnostic analyses. Closed-source systems lead overall, yet temporal grounding and wrong-speaker attribution remain challenges.

  • Problem

    Existing speech benchmarks focus mainly on single-speaker settings or isolated subtasks, leaving performance across speaker attribution and higher-order multi-speaker reasoning insufficiently characterized.

    Source: §1 · p. 1

  • Method

    MSU-Bench evaluates 16 speaker-centric tasks across two tiers using QA instances generated from speaker-aware annotations and verified by human annotators.

    Source: §1 · p. 2, §2.2 · p. 2

  • Results

    Closed-source systems lead overall, but temporal grounding remains difficult and strong models still make wrong-speaker attribution errors.

    Source: §3 · p. 3, §4.1 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    Speaker-referencing schemes and diagnostic error types provide a basis for improving robust multi-speaker audio understanding.

    Source: §5 · p. 4

Abstract

from arXiv · show

Spoken Language Understanding (SLU) is moving from task-specific pipelines toward large audio language models (LALMs) that generate natural-language responses. However, existing speech benchmarks mainly focus on single-speaker settings or isolated subtasks, leaving speaker-centric understanding in realistic multi-speaker conversations insufficiently evaluated. We introduce MSU-Bench, a diagnostic benchmark for multi-speaker conversational understanding, covering 16 speaker-centric tasks and 2,300 QA instances in a two-tier framework from speaker grounding to dialogue reasoning. We build a Gemini-assisted annotation and QA generation pipeline with human-in-the-loop verification, achieving high QA validity and strong agreement between human answers and verified labels. We further analyze speaker-referencing schemes and diagnostic error types to reveal bottlenecks in speaker grounding and reasoning. Experiments reveal clear gaps across model families, with closed-source systems leading overall but all models still facing challenges in complex speaker grounding and multi-speaker reasoning. The benchmark annotations, metadata, and evaluation scripts will be available at the GitHub repository: https://github.com/ASLP-lab/MSU-Bench.

1. Introduction

SLU is shifting toward end-to-end audio language models, but realistic multi-speaker conversations require speaker identity and interaction tracking beyond acoustic perception or transcript content alone. MSU-Bench addresses the evaluation gap with a two-tier hierarchy, 16 tasks, and 2,300 QA instances.

1. Introduction

  • Recent LALM work strengthens speaker grounding, but interaction-level reasoning about speaker relations, motivations, and evolving states remains comparatively underexplored.Grounding approaches include speaker-attributed transcripts, speaker registration, and time-aware representations [7] [8] [9] [10].Source: §1 · p. 1
  • Existing benchmarks mainly target single speakers or isolated subtasks, while conversation-level evaluations offer limited diagnosis of speaker-centric understanding in realistic multi-speaker settings.The resulting gap leaves performance across speaker attribution through higher-order reasoning unclear.Source: §1 · p. 1
  • MSU-Bench covers 16 speaker-centric tasks and 2,300 QA instances in a two-tier framework, with human-in-the-loop verification.The benchmark also analyzes speaker-referencing schemes and diagnostic error types to identify current bottlenecks.Source: §1 · p. 2

2. MSU-Bench: Hierarchical Design for Multi-Speaker Understanding

MSU-Bench evaluates speaker-centric understanding through a two-tier hierarchy and diagnostic multiple-choice QA. Its construction combines diverse dialogue data, speaker-aware annotation, QA generation, and human verification.

2. MSU-Bench: Hierarchical Design for Multi-Speaker Understanding

  • MSU-Bench uses diagnostic multiple-choice QA to evaluate speaker-centric understanding in realistic multi-speaker conversations.Its design spans Chinese and English, eight corpora, and conversational and media-style audio across telephone calls, meetings, podcasts, and movies.Source: §2 · p. 2, §2.1 · p. 2, §2.2 · p. 2

2.1. Benchmark Design and Task Hierarchy

  • Tier 1 grounds attributes, identities, and speaker-specific information to particular speakers, while Tier 2 reasons over relations, roles, motivations, and interaction dynamics.Five speaker-referencing schemes vary grounding difficulty, from audio or time cues to transcript, appearance-order, and combined references.Source: §2.1 · p. 2

2.2. Data and QA Construction Pipeline

  • The construction pipeline selects coherent dialogue, adds speaker-aware annotations, generates speaker-referenced QA candidates, and uses trained annotators to verify and retain qualified items.It produces 2,300 verified QA instances.Source: §2.2 · p. 2

2.3. Evaluation Protocol

  • Four-option questions have one ground-truth answer supported by audio evidence, and exact-match accuracy is reported by task, capability, tier, and referencing scheme.Wrong-speaker, hallucination, and unknown distractors map incorrect choices to diagnostic error types.Source: §2.3 · p. 2

3. Experimental Setup and Results

Closed-source systems outperform open-source systems overall, but performance varies across models and speaker-centric capabilities. The results identify persistent challenges in complex grounding and multi-speaker reasoning.

3. Experimental Setup and Results

  • Overall accuracy ranges from 0.19 for Qwen2.5-Omni to 0.77 for Gemini-3-Flash; MiMoAudio leads open-source models at 0.56 overall.Attribution, viewpoint aggregation, and dialogue reasoning remain challenging for most open-source models, while clearer acoustic or semantic cues tend to yield better performance.Source: §3 · p. 3
  • Gemini-3-Flash leads overall at 0.77, with Tier 1 accuracy of 0.73 and Tier 2 accuracy of 0.84; closed-source systems consistently outperform open-source systems.Gemini-2.5-Pro and Gemini-2.5-Flash score 0.70 and 0.69 overall, respectively, both above the strongest open-source model.Source: §3 · p. 3
  • Qwen-Omni systems do not consistently outperform audio-oriented models, suggesting broad multimodal capability does not necessarily translate into robust speaker-centric understanding.MiMoAudio is the strongest open-source system, especially on Tier 2 reasoning tasks.Source: §3 · p. 3

4. Analysis and Discussion

The analysis examines how speaker-referencing schemes affect grounding, how error types vary with model capability, and whether human verification supports QA quality.

4.1. Speaker-Referencing Scheme Analysis

  • Time Index generally yields the lowest accuracy, while Complex Index often improves performance by adding complementary grounding cues.Gemini-3-Flash scores 0.71, 0.64, and 0.84 on Tier 1 and 0.83, 0.76, and 0.92 on Tier 2 under No Index, Time Index, and Complex Index, respectively.Source: §4.1 · pp. 3–4, Table 4 · p. 4

4.2. Diagnostic Error-Type Analysis

  • Unknown errors rise for Qwen3-Omni from 0.27 in Tier 1 to 0.40 in Tier 2, while stronger models’ errors center on wrong-speaker attribution.Tier 2 wrong-speaker rates are 0.53 for MiMoAudio and 0.67 for Gemini-3-Flash; hallucination and instruction-following failures remain secondary overall.Source: Table 5 · p. 4, §4.2 · p. 4

4.3. Question Quality and Human Verification

  • Human reviewers judged initial QA validity at 95% in Tier 1 and 86% in Tier 2; agreement with verified labels was 98% and 96%, respectively.Invalid or ambiguous questions were revised or removed before inclusion.Source: Table 6 · p. 4, §4.3 · p. 4

5. Conclusion

MSU-Bench provides a diagnostic benchmark for multi-speaker conversations, and evaluations reveal persistent grounding and reasoning bottlenecks across models.

5. Conclusion

  • MSU-Bench covers 16 speaker-centric tasks in a two-tier hierarchy and contains 2,300 verified QA instances for realistic multi-speaker conversations.The hierarchy spans speaker grounding to multi-speaker reasoning.Source: §5 · p. 4
  • Across nine speech-language models, temporal grounding remains especially difficult, and strong models still make wrong-speaker attributions.Weaker models often default to unknown under higher reasoning demands.Source: §5 · p. 4
  • Speaker-referencing schemes and diagnostic error types expose persistent bottlenecks, providing a diagnostic basis for improving robust multi-speaker audio understanding.Source: §5 · p. 4

7. Generative AI Use Disclosure

The disclosure describes AI use in both benchmark construction and manuscript preparation, with human review of generated annotations and QA items.

7. Generative AI Use Disclosure

  • Gemini supported benchmark construction through dialogue quality assessment, paralinguistic annotation, and QA generation.Source: §7 · p. 5
  • Trained annotators reviewed all AI-generated annotations and QA items for metadata correctness, question validity, answer determinacy, and format compliance.Items were reviewed before inclusion in the benchmark.Source: §7 · p. 5
  • AI writing assistants were used to improve clarity and grammar in some manuscript passages.Source: §7 · p. 5
Loading 2606.22868v1…