PaperlayerView exact revision on arXiv

Source-linked AI summary

MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant

Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin, Kai-Wei Chang, Siddhant Arora, Shu-wen Yang, Abdelrahman Mohamed, Shinji Watanabe, Hung-yi Lee, David Harwath

arXiv:2609.13076v1eess.AScs.AIcs.CL

TL;DR

Existing benchmarks largely overlook active participation in multiparty conversations, despite the importance of understanding turn-taking and contextually appropriate responses. MP-Bench evaluates voice agents in controlled multiparty scenarios, and finds that realtime agents lag non-realtime counterparts while both struggle with implicit and negative turn-taking.

  • Problem

    Existing benchmarks largely overlook multiparty conversations, leaving limited evaluation of voice agents as active participants in a prevalent setting with complex turn-taking.

    Source: §1 · p. 2

  • Method

    MP-Bench evaluates turn-taking, response behavior, and comprehension in controlled Discussion and Turn-based Game scenarios.

    Source: §1 · p. 2

  • Results

    Realtime agents lag behind non-realtime counterparts in multiparty conversation, and both perform near-randomly on implicit and negative turn-taking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations identify multiparty awareness as a foundational missing capability in current voice agents.

    Source: §8 · p. 9

    MP-Bench tests a single decision point at the end of pre-recorded clips, not multi-turn engagement, barge-in, or backchanneling.

    Source: §9 · p. 9

Abstract

from arXiv · show

Conversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multi-party conversations. Evaluating agents in these settings is fundamentally more challenging than in dyadic interactions due to the exponentially greater conversational complexity. For voice agents to integrate seamlessly into human group dynamics, they must not only generate contextually appropriate responses but also demonstrate a nuanced understanding of open turn-taking. To address this gap, we introduce Multiparty Bench (MP-Bench), the first benchmark specifically designed to objectively evaluate conversational speech systems as active participants within multi-party contexts. MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness. Additionally, we incorporate comprehension-based question-answering tasks as a complementary evaluation. By benchmarking 12 voice agents, we find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking, exposing an open challenge for real-time voice agents under multiparty scenario.

1 Introduction

Existing voice-agent benchmarks largely overlook active participation in multiparty conversations, a prevalent setting with more complex turn-taking than dyadic interaction. MP-Bench evaluates understanding and behavior in controlled multiparty scenarios and finds substantial challenges for current agents.

1 Introduction

  • The paper motivates voice agents as a natural speech-based interaction modality and describes cascaded and end-to-end architectures.Cascaded systems use ASR, a text-based LLM, and TTS; end-to-end systems directly process and generate speech.Source: §1 · p. 1
  • Multiparty conversations are prevalent in daily communication, but existing benchmarks largely overlook them.The paper envisions voice agents contributing naturally to human group interactions.Source: §1 · p. 2
  • MP-Bench evaluates turn-taking and response behavior under explicit, implicit, and negative turn-taking conditions using synthetically generated recordings.Synthetic recordings enable precise control over conversational state.Source: §1 · p. 2
  • The benchmark includes discussion and turn-based game scenarios, plus comprehension tasks testing speaker attribution.The evaluation also compares realtime agents with non-realtime systems and text LLMs.Source: §1 · p. 2
  • Realtime agents lag behind non-realtime counterparts, while both perform near-randomly on implicit and negative turn-taking.The evaluation also reports that 12 state-of-the-art voice agents struggle with appropriate multiparty interaction.Source: §1 · p. 2
  • The benchmark is presented as a first step toward evaluating voice agents as active participants in multiparty conversations.Its assessment includes both understanding and behavior.Source: §1 · p. 2

2 Related Works

Prior voice-agent benchmarks cover several conversational dimensions, but multiparty work has focused mainly on passive understanding, while active turn-taking work is limited to text. MP-Bench evaluates understanding, turn-taking, and response appropriateness directly from audio.

2 Related Works

  • Existing multiparty benchmarks mainly assess passive understanding, while concurrent active turn-taking benchmarks are confined to text.MP-Bench evaluates understanding, turn-taking, and response appropriateness directly from audio, combining acoustic and contextual cues.Source: §2 · p. 2

3 MP-Bench Overview

MP-Bench comprises understanding and behavioral tasks across two multiparty scenarios, with evaluation metrics for comprehension, turn-taking, response appropriateness, and latency. Its controlled scenarios and task definitions specify when an agent should speak, remain silent, or respond appropriately.

3 MP-Bench Overview

  • MP-Bench contains 927 tasks across Discussion and Turn-based Game scenarios, covering understanding and behavioral task families.Discussion includes both task families; Turn-based Game includes behavioral tasks only.Source: §3 · p. 2

3.1 Scenario

  • Discussion tests debate comprehension and response to an invitation, while Turn-based Game uses team roles and rules to determine who responds next.Game variants minimize spoken-content complexity to focus on multiparty turn-taking.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2 Task

  • Understanding tasks test comprehension of multiparty conversation, while behavioral tasks assess whether the agent speaks or stays silent and whether its response is appropriate.Discussion behavioral tasks target explicit invitations; Turn-based Game tasks target implicit and negative turn-taking.Source: §3.2 · p. 3

3.3.1 Evaluation Pipeline

  • The evaluation streams audio to agents, uses word-level ASR timestamps to detect response onsets, and assesses spoken content with a GPT-5 LLM judge.A response is detected when a generated word begins after the input audio ends.Source: §3.3.1 · pp. 3–4

3.3.2 Understanding Task Metrics

  • Understanding metrics include overall answer accuracy and speaker-name accuracy, which is compared with a 33.3% random-guessing baseline..Source: §3.3.2 · p. 4

3.3.3 Behavioral Task Metrics

  • Behavior metrics assess correct speaking or silence and count a response as appropriate only when turn-taking and response validity are both correct.The Turn-based Game labels are balanced between speaking and silence, yielding a 50% random-guessing baseline.Source: §3.3.3 · p. 4

3.3.4 Response Latency

  • Response latency is measured as the interval from the end of the input audio to the agent's first generated word.The reported value is the mean response latency for realtime voice agents on each task.Source: §3.3.4 · p. 4

4 MP-Bench Construction

MP-Bench construction combines controlled multiparty dialogue generation with evaluation tasks designed to require auditory cues. Its pipeline covers speaker attribution, dialogue filtering, comprehension questions, and behavioral tasks.

4 MP-Bench Construction

  • Construction targets objective automatic evaluation and auditory-cue dependence, using LLM-generated dialogue and ElevenLabs conversational audio.Generated dialogue instances are then used to construct specific evaluation tasks, which undergo human verification.Source: §4 · p. 4

4.1 Speaker Attribution

  • Four participants balance multiparty dynamics against audio length and context limits; curated distinct voices support attribution, with similar-voice conditions used as a harder acoustic ablation.The authors describe three speakers as near-dyadic and more than four as risking long-context memory failures.Source: §4.1 · p. 4

4.2.1 Discussion Scenario

  • Discussion dialogues sample from 83 topics and valid speaker combinations; generated turns include addressee and coreference metadata.Addressees need not be the next speaker, and coreferences link expressions to participants or prior topics.Source: §4.2.1 · pp. 4–5

4.2.2 Turn-based Game Scenario

  • Turn-based Game dialogues use game-specific prompts and valid speaker combinations, without generated addressee or coreference metadata.The passage attributes that omission to the game setup.Source: §4.2.2 · p. 5

4.2.3 Dialogue Instance Filtering

  • Dialogue filtering keeps instances solvable with anonymized speaker labels but not from spoken content alone, while requiring each participant to be mentioned or addressed.The labeled condition is described as simulating a cascaded system with flawless ASR and speaker diarization.Source: §4.2.3 · p. 5

4.3.1 Understanding Tasks

  • Discussion dialogues produce QA tasks about speaker attributes and content details; questions are retained only when solvable with speaker names but not from transcript alone.The filtering is intended to make retained questions logical and solvable with full context while requiring spoken-conversation comprehension.Source: §4.3.1 · p. 6

4.3.2 Behavior Tasks

  • Behavior tasks append an agent-directed agreement question to Discussion dialogues and game-playing turns to Turn-based Game dialogues.The game turns follow the rules introduction and team assignment.Source: §4.3.2 · p. 6

5 Benchmark Results

MP-Bench results show a sharp contrast between explicit and implicit or negative turn-taking, alongside weaker multiparty understanding for realtime than non-realtime agents. Behavioral performance and the realtime gap vary by scenario.

5.1 Evaluated Models

  • Evaluation includes 10 realtime systems, two non-realtime audio models, and text-LLM references with transcript-only or anonymized speaker labels.The labeled reference is described as an upper bound for a cascaded system with perfect diarization.Source: §5.1 · p. 6

5.2 Main Results

  • Discussion behavioral tasks show higher scores than understanding across nearly all settings, while the ASR-only versus SpkD-ASR gap is narrower for appropriateness than understanding.The reported gaps are 66.67 percentage points on understanding and 14.06 on behavioral appropriateness accuracy.Source: §5.2 · p. 6
  • Nearly all voice agents perform at or below random in Turn-based Game, while multiple realtime agents exceed 90% accuracy in Discussion; SpkD-ASR is near-perfect across both.The table defines the Turn-based TT random baseline as 50% and notes that Discussion TT ground truth is always to respond.Source: Table 3 · p. 7, §5.2 · p. 7
  • Non-realtime agents significantly outperform realtime counterparts on understanding, and most realtime models fall below the 33.3% speaker-identification random baseline.Even the best realtime system, Covo-Audio, lags Gemini-3.1-Pro in overall and speaker-name accuracy.Source: §5.2 · p. 7, Table 3 · p. 7
  • The realtime versus non-realtime gap narrows for Discussion behavior and vanishes in Turn-Based Games, where all models perform near-randomly.The authors characterize difficulty with implicit and negative turn-taking as orthogonal to latency constraints.Source: §5.2 · p. 7

6 Ablation Results

The ablations separate multiparty turn-taking difficulty from game-rule following and test whether explicit multiparty prompting changes performance. Single-speaker results are above chance, while prompt effects differ by scenario.

6.1 Disentangling Game-Rule Following from Multiparty Capabilities

  • 76.0% GPT-Realtime and 68.0% Gemini-2.5 Live TT Acc. exceed the 50% random baseline in single-speaker games, unlike their near-chance multiparty counterparts.Their single-speaker appropriateness accuracies are 74.0% and 63.0%, versus 28.13% and 20.00% in the multiparty setting.Source: Table 4 · p. 8, §6.1 · p. 8
  • The single-speaker ablation does not isolate turn-taking from speaker attribution, but text references show Turn-Based TT at chance with ASR-only and near-ceiling with oracle speaker labels.The reported accuracies are 49.22 for ASR-only and 98.44 for SpkD-ASR.Source: §6.1 · p. 8

6.2 Multiparty-aware Prompting

  • Multiparty-aware prompts yield substantive gains for several systems in Discussion, but Turn-Based Games stagnate or degrade while TT accuracy remains near the 50% random baseline.The authors attribute lower appropriateness to weaker game-rule compliance when agents do speak.Source: §6.2 · p. 8, Table 5 · p. 8

7 Human Evaluation

A human evaluation of Discussion behavior-task appropriateness shows reasonably strong alignment between annotators and the LLM judge. The evaluation sampled 280 entries across 14 settings.

7 Human Evaluation

  • Krippendorff’s α = 0.745 and Cohen’s κ = 0.646 indicate reasonably strong alignment between human ratings and the LLM judge.The 280 sampled entries were equally drawn across 14 behavior-task settings from the Discussion scenario.Source: §7 · pp. 8–9

8 Conclusion

MP-Bench is presented as a first step toward benchmarking voice agents as active participants in multiparty conversations. The evaluation finds broad capability gaps, with failures attributed to multiparty dynamics rather than task-specific nuances or general instruction-following deficits.

8 Conclusion

  • MP-Bench is a first step toward benchmarking voice agents as active participants within multiparty interactions.The authors describe multiparty awareness as a foundational missing capability for conversational AI.Source: §8 · p. 9
  • Non-realtime models show robust comprehension but near-random active participation, while realtime agents struggle across understanding and behavioral metrics.The authors attribute these failures to multiparty dynamics, not task-specific nuances or general instruction-following deficits.Source: §8 · p. 9
  • Explicit system prompting does not circumvent the reported shortcomings in multiparty conversation.The authors describe this as a trivial intervention that cannot resolve the severe shortcomings.Source: §8 · p. 9

9 Limitation

MP-Bench is a controlled first step, not a full test of open-ended multiparty participation. Its scope and synthetic audio simplify real meetings, while the dataset uses four-participant conversations and deliberately varied speaker profiles.

9 Limitation

  • A single decision at the end of a prerecorded clip excludes multi-turn engagement, barge-in, and backchanneling; within-clip full-duplex speech is ignored by the metrics.A response is registered only when ASR detects a word onset after the input audio concludes.Source: §9 · p. 9
  • Synthetic audio omits overlapping speech, background noise, reverberation, long pauses, and disfluencies, making the setup a conservative lower bound on model limitations.The authors note that real or actor-voiced recordings remain an important direction for future work.Source: §9 · p. 9

A.1 Statistics

  • Tasks average 45.6 ± 11.5 seconds and 8.4 ± 2.1 turns; Figures 3–6 show duration, speaker, and turn distributions.Each participant contributes an average of 2.8±1.1 turns per task.Source: Figure 3 · p. 11, Figure 4 · p. 11, Figure 5 · p. 11
  • Across the dataset, each unique speaker contributes a mean of 647.3 ± 124.7 turns, about 8.3%±1.6% of all dialogue turns.The authors characterize speaker participation as well-balanced.Source: §A.1 · p. 12

A.2 Speaker Attribution

  • Each dialogue pairs three simulated human speakers with the agent; 12 voices are selected using gender and pitch constraints and described acoustically.The design aims to make the sampled speakers within a conversation highly distinct.Source: §A.2 · p. 12

A.3 Audio Quality Analysis

MP-Bench uses clean, intelligible synthesized audio and naturalistic timing for active turn transitions, while deliberately omitting several features of real meetings. Its appendix illustrates the benchmark’s discussion and game tasks and describes the human annotation protocol.

A.3 Audio Quality Analysis

  • For active transitions with gaps under one second, MP-Bench’s 0.40 s and AMI’s 0.43 s are closely matched; MP-Bench deliberately omits long pauses, overlap, noise, and reverberation.AMI’s long silences make up 56.6% of gaps and raise its overall mean gap to 1.8 s.Source: §A.3 · p. 12
  • WER is 6.6% on MP-Bench versus 21.7% on real AMI recordings; speaker separation is 0.412 versus AMI’s 0.299.The authors interpret these comparisons as evidence of clean audio and clearly distinguishable speakers, not greater naturalness than real conversation.Source: §A.3 · p. 12, Table 6 · p. 13

A.4 List of Topics for Discussion Scenario

  • Discussion topics range across general knowledge and everyday categories, including capitals, arithmetic, food, sports, and entertainment.The topic list also includes subjects such as weather, natural disasters, and common idioms.Source: §A.4 · p. 13

A.5.1 Understanding Task for Discussion Scenario

  • An Understanding example asks the agent to identify the relatively low-pitched male speaker after a multiparty discussion.The clip ends with a direct question asking for that speaker’s name.Source: §A.5.1 · p. 13

A.5.2 Behavior Task for Discussion Scenario

  • A Behavioral Discussion example asks the agent to answer a dispute and identify whose information is correct.Source: §A.5.2 · p. 14

A.5.3 Behavior Task for Turn-Based Game

  • In the Turn-based Game, group followers respond with the next integer when the opposing group’s leader speaks.The example assigns the agent a follower role in one of the two groups.Source: §A.5.3 · p. 14

B Details of Human Evaluation

  • Thirteen graduate-student annotators from two universities worked in batches of 30 items; a response counted as consistent only if it stated the answer and identified the agreeing speaker.Source: B Details of Human Evaluation · p. 14

C Robustness of LLM Judge across model families

The authors re-judged a stratified sample with three additional judges to test whether GPT-5 labels were idiosyncratic. Agreement was substantial across task scenarios, and the judges did not systematically change model rankings.

C Robustness of LLM Judge across model families

D Ablation for high voice similarity speakers for MP-Bench

This section reports how similar voices affect acoustic attribution and turn-based turn-taking, and documents the prompts and LLM judging procedures used for evaluations. It also discloses the authors’ limited use of generative AI tools.

D Ablation for high voice similarity speakers for MP-Bench

  • MP-Bench uses distinct voices to separate core multiparty ability from diarization difficulty and enable objectively gradable questions about speaker characteristics.The authors give questions about the higher-pitched speaker as an example of objective question design that becomes difficult when voices are too similar.Source: D Ablation for high voice similarity… · p. 15
  • Understanding fell 40% to 4%, speaker-name accuracy fell 45% to 0%, and Discussion appropriateness fell 75%→45% with similar voices; turn-based turn-taking remained near/below the 50% chance baseline.The ablation used Covo-Audio, the strongest realtime understanding model, and excluded questions explicitly referencing pitch; the authors report that weak understanding and near-chance implicit/negative turn-taking persist, with understanding worsening as voices become harder to distinguish.Source: D Ablation for high voice similarity… · p. 15

E.1 Inference-Time System Prompts for Realtime Models

  • The default realtime system prompt is used for main results, while a variant adding one sentence about multiple speakers is reserved for the prompt ablation.Both prompts ask for concise, conversational responses; the variant says to be aware that multiple speakers may be present.Source: §E.1 · p. 15

E.2 LLM Prompt for Scenario Generation

  • Scenario-generation prompts create three-person factual discussions and four-person turn-based games, specifying speaker roles, conversation progression, and output structure.In discussion scenarios, speakers answer a factual question before asking the AI Assistant to judge who is correct; game scenarios establish groups and leader/follower roles, and stop when the game begins.Source: §E.2 · pp. 15–16

E.3 LLM Prompt for Validating Generated Main Dialogues

  • Dialogue-validation prompts either predict speaker names from an unnamed transcript or map anonymized speaker labels back to real names.The first outputs speaker-labeled utterances; the second uses Speaker1, Speaker2, and Speaker3 labels and outputs their real-name mappings.Source: §E.3 · pp. 16–17

E.4 LLM-as-a-Judge Prompts

  • Four GPT-5 judge prompts produce main-table and ablation scores for Understanding, Discussion consistency, Turn-Based rule compliance, and speaker-name accuracy.The speaker-name judge uses fuzzy matching on the speaker-name slice of the Understanding QA pool.Source: §E.4 · p. 17
  • The turn-based game judge identifies game rules, group roles, and the triggering leader, then checks whether the assistant responded or stayed silent as required.For cross-group reaction games, a leader triggers the other group’s follower; the judge evaluates the final response rather than factual correctness outside the game.Source: §E.4 · p. 18
  • The speaker-name judge extracts an intended valid name, allowing phonetic or ASR spelling variants and names mentioned alongside refusal language.It compares the extracted name with the ground truth and uses null only when no valid name or phonetic variant appears.Source: §E.4 · p. 19

F Use of Generative AI Tools

  • The authors used generative AI to polish writing, support implementation and debugging, and prepare figures and tables, but not to originate the research, design the benchmark, or interpret results.Authors reviewed AI-assisted outputs and take responsibility for the paper’s motivation, methods, results, and errors.Source: F Use of Generative AI Tools · p. 20
Loading 2609.13076v1…