PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

Puneet Mathur, Dinesh Manocha

arXiv:2610.20172v1cs.SDcs.AIcs.CL

TL;DR

Existing evaluations do not jointly test genuine floor relinquishment and acquisition against confusable non-events in controlled multi-party settings. MultiPartyTurnBench introduces two event-conditioned detection tasks across synthetic conversations, and finds a pronounced gap between turn completion and selective floor acquisition that extends to full-duplex agents.

  • Problem

    Existing evaluations study dyadic turn-taking or agent behavior, but joint controlled evaluation of genuine floor relinquishment and successful acquisition against confusable non-events in multi-party settings remains underexplored.

    Source: §1 · p. 1

  • Method

    MultiPartyTurnBench evaluates EOT against temporary holds and INT against non-floor-taking speech in 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types.

    Source: §1 · p. 1

  • Results

    At qualified operating points, systems recover up to 92.6% of turn endings but only 56.2% of successful interruptions, and full-duplex agents perform near chance on selective yielding.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    Selective interruption handling remains an unresolved challenge for reliable multi-party voice interaction.

    Source: §8 · p. 8

    Synthetic speech cannot fully capture spontaneous conversation's prosody, disfluencies, timing adaptations, and ambiguous floor negotiations, so MPTB scores are controlled diagnostics rather than natural-conversation error-rate estimates.

    Source: §9 · p. 9

Abstract

from arXiv · show

Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.

1 Introduction

Multi-party floor control requires distinguishing genuine floor changes from pauses and backchannels, a combination existing evaluations have not jointly tested under controlled conditions. MultiPartyTurnBench evaluates those decisions and finds that selective floor acquisition remains difficult for both detection systems and full-duplex agents.

Motivation

  • Existing evaluations cover dyadic turn-taking, multi-party next-speaker prediction, or agent behavior, but not joint controlled evaluation of floor relinquishment and acquisition.Misreading pauses or backchannels can prompt premature responses or unnecessary abandonment of an agent's response.Source: §1 · p. 1

Benchmark and evaluation

  • MultiPartyTurnBench evaluates End-of-Turn detection against temporary holds and Interruption detection against non-floor-taking speech in 240 conversations spanning two to five speakers and six interaction types.The benchmark uses exact event-conditioned annotations and evaluates causal detection with false-positive-constrained recall and signed latency.Source: §1 · pp. 1–2, Figure 1 · p. 2

Main Results

  • At qualified operating points, systems recover up to 92.6% of turn endings but only 56.2% of successful interruptions, while agents remain near chance on selective yielding.Four of five agents achieve 86.1–95.8% balanced accuracy on turn entry, while all five reach only 46.9–54.7% on selective yielding.Source: §1 · p. 2

Contributions

  • The benchmark contributes a systematic comparison of acoustic, semantic, and turn-projection systems, plus testing of five full-duplex agents that reveals near-chance selective yielding.The agents show opposing tendencies: yielding indiscriminately or rarely yielding to genuine interruptions.Source: §1 · p. 2

2 Related Work

Prior work evaluates dyadic turn events, natural multi-party speaker prediction, and full-duplex agent behavior, but these approaches do not jointly score multi-party EOT and INT events against event-specific hard negatives.

Multi-party turn-taking resources

  • Natural multi-speaker resources study future speaker activity, shift/hold, or floor shifts, but do not score end-of-turn and interruption detection.Examples include TEIDAN, AVCC, and VoxConverse.Source: §2 · pp. 2–3

Benchmark comparison

  • MultiPartyTurnBench jointly evaluates multi-party EOT and INT detection with event-specific hard negatives, causal timestamped decisions, FPR-constrained scoring, controlled speaker-count comparisons, and agent validation.The comparison distinguishes this event-level protocol from benchmarks centered on dyadic events, next-speaker prediction, or conversational outcomes.Source: §2 · pp. 2–3, Table 1 · p. 3

Full-duplex agent benchmarks

  • Full-duplex agent benchmarks assess conversational behavior and outcomes, whereas MultiPartyTurnBench isolates floor-event discrimination and separately validates its manifestation in agents.Related benchmarks include the Full-Duplex-Bench series, FD-Bench, Duplex-MPE, MultiTalkBench, and MSI-Bench.Source: §2 · p. 3

3 MultiPartyTurnBench

MultiPartyTurnBench operationalizes floor relinquishment and acquisition as separate causal detection tasks, using controlled conversation construction and exact event boundaries. Its benchmark spans multiple speaker counts and interaction styles.

3 MultiPartyTurnBench

  • EOT detects whether the current speaker relinquishes the floor at speech offset, while INT detects successful floor acquisition by another speaker at the interrupting speech onset.Mid-turn pauses and backchannels are hard negatives; unsuccessful interruption attempts are excluded because their outcome may only become apparent through subsequent interaction.Source: §3 · p. 3

3.1 Benchmark Construction

  • MPTB constructs conversations from an explicit floor-state representation and controls speaker count, interaction type, listener activity, and acoustic conditions.Its five-stage pipeline establishes an event log and interaction timeline before synthesis, then renders speakers, adds acoustic conditions, and assembles channels; the orchestrator resolves floor ownership and timing.Source: §3.1 · pp. 3–5, Figure 2 · p. 4
  • The event taxonomy defines TURN, INTERRUPTION, NON-FLOOR-TAKING INTERRUPTION, BACKCHANNEL, and NON-CONTENT activity according to whether floor ownership is retained, relinquished, or transferred.The controlled interaction axes cross two to five speakers with six styles: Casual, Task-Oriented, Instructional, Collaborative, Argumentative, and Narrative.Source: §3.1 · p. 4

3.2 Benchmark Composition

  • MPTB contains 240 conversations totaling 6.89 hours, with 48 development and 192 test conversations and disjoint speaker voices.The corpus includes 3,422 EOT positives and 241 hold negatives, alongside 367 INT positives and 1,481 non-floor-taking negatives; generated floor events also receive independent human annotation validation.Source: §3.2 · p. 5

4 Evaluation

The evaluation scores causal, timestamped EOT and INT decisions against reference events and hard-negative spans. Development-selected thresholds are frozen for testing, with recall, false-positive rate, and signed latency reported.

Prediction and Event Matching

  • Causal predictions are matched one-to-one to reference events, while detections within hard-negative spans count as false positives.A matched prediction must fall within the interval from 0.25 seconds before to 3.0 seconds after the gold event.Source: §4 · p. 5

Operating Points and Metrics

  • For each model, the development threshold maximizes recall subject to FPRdev ≤0.10 and is frozen across test conversations, speaker counts, and interaction types.Models with FPRtest >0.15 are marked out-of-budget; the reported metrics include recall, FPR, and signed decision latency.Source: §4 · p. 5

Full-Duplex Behavioral Diagnostics

  • For full-duplex agents, balanced accuracy measures discrimination between genuine transitions and hard negatives, while action bias indicates excessive responsiveness or reluctance to act.Balanced accuracy is (Recall + 1 − FPR)/2; action bias is Recall + FPR − 1.Source: §4 · p. 6

5 Experiments

The experiments evaluate offline EOT and INT detection across three system families, then test five full-duplex agents on interactive floor-control scenarios.

Offline EOT and INT Evaluation

  • Twelve systems across acoustic, semantic, and turn-taking families are evaluated on EOT and INT detection, with performance broken down by speaker count and interaction type.VAP is adapted to multi-party audio through speaker-versus-rest inference; operating points are selected on development data.Source: §5 · p. 6

Full Duplex Agent Evaluation

  • Five full-duplex agents are tested on 120 interactive floor-control scenarios, repeated three times per system.The evaluation distinguishes offline event detection from agents’ observable decisions to enter or relinquish the floor.Source: §5 · p. 6

Evaluation systems

  • The offline evaluation covers RMS Energy, WebRTC VAD, Silero VAD, Smart Turn v2, Kyutai SVAD, Mimi Endpointer, VAP, MuVAP, DualTurn, WavLM-Large, Turn-1-Mini, and ESPnet Turn-Taking.The five agents tested are GPT-Live-1, Gemini-3.8-Live, MiniCPM-o-4.5, Realtime-Venus, and Step-Audio3-Realtime.Source: §5 · p. 6

6 Results

Results show that EOT detection is more reliable than detecting successful floor takeovers, and that the gap varies with speaker count and conversational dynamics. Full-duplex agents likewise recognize turn endings more reliably than they selectively yield.

RQ2: Does Speaker Count Make Floor Control Harder?

  • At five speakers, no plotted detector reaches 50% INT recall, while EOT recall is broadly stable; VAP stays at 92–93% and Kyutai rises from 84.6% to 96.6%.The figure reports recall by speaker count from two to five with 95% conversation-bootstrap confidence intervals.Source: Figure 4 · p. 6, RQ2: Does Speaker Count Make Floor… · p. 6

RQ1: Can Models Reliably Identify Floor Transitions?

  • 92.6% EOT recall contrasts with 56.2% INT recall at the strongest qualified operating points, while acoustic detectors’ higher recall comes with excessive false positives.VAP and Kyutai SVAD reach 92.6% and 92.5% EOT recall at 7.5% and 4.3% FPR, but only 11.7% and 56.2% INT recall; Silero VAD’s 97.2% EOT and 85.4% INT recall come with 93.5% and 89.2% FPR.Source: RQ1: Can Models Reliably Identify… · p. 6, Table 3 · p. 7

RQ4: Can Voice Agents Yield Selectively?

  • Four agents achieve 86.1–95.8% START balanced accuracy, but all five are near chance on STOP at 46.9–54.7%, with opposite yielding failures at the extremes.Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions.Source: Table 4 · p. 7, RQ4: Can Voice Agents Yield Selectively? · pp. 7–8

RQ3: Which Interaction Types Expose Failures?

  • EOT recall remains comparatively stable across interaction types, but INT recall swings by up to 30 points and its difficulty varies with conversational dynamics.The reported dynamics include argumentative settings for VAP and Mimi EP, collaborative for Kyutai, narrative for MuVAP, and task-oriented for DualTurn; † marks recall over the false-positive budget.Source: RQ3: Which Interaction Types Expose… · p. 7, Figure 5 · p. 8

7 Discussion

The discussion frames multi-party floor control as a problem of identifying who is claiming the floor, not merely detecting speech. It also highlights diagnostic breakdowns and real-speech evidence for the interruption challenge.

Floor control and speaker roles

  • Reliable interruption handling requires distinguishing competing floor claims from backchannels and other non-floor-taking speech, conditioned on the current speaker and conversational context.The discussion argues for event-conditioned floor-transition modeling rather than independent speech-activity decisions.Source: §7 · p. 8

Evaluation diagnostics

  • Breakdowns across speaker counts and interaction regimes expose failure modes that aggregate scores can obscure and provide targets for more selective systems.High event recall can conceal excessive false activations, while conservative yielding can appear reliable despite missed genuine interruptions.Source: §7 · p. 8

Synthetic to real evaluation gap

  • On real four-party AMI meetings, VAP recovers only 35.9% of interruptions at 5.3% FPR, supporting the interruption challenge’s relevance beyond synthetic speech.Source: §7 · p. 8

8 Conclusion

The conclusion presents MultiPartyTurnBench as a controlled benchmark for EOT and interruption detection in multi-party spoken dialogue. Its results identify selective interruption handling as an unresolved challenge for reliable multi-party voice interaction.

Conclusion

  • MultiPartyTurnBench is a controlled benchmark for end-of-turn and interruption detection in multi-party spoken dialogue.Source: §8 · p. 8
  • Existing systems detect turn completion reliably but struggle to recognize floor takeovers without excessive false activations, with failures varying across conversational regimes and speaker counts.Source: §8 · p. 8
  • All tested full-duplex agents perform near chance on selective yielding, establishing interruption handling as an unresolved challenge for reliable multi-party voice interaction.Source: §8 · p. 8

9 Limitations

MPTB provides controlled evaluation of floor control, but its synthetic and limited real-speech validation constrain how its findings generalize to spontaneous conversation.

9 Limitations

  • Synthetic speech cannot fully capture spontaneous conversation’s prosody, disfluencies, timing adaptations, or ambiguous floor negotiations.The authors frame MPTB scores as controlled diagnostics, not estimates of natural-conversation error rates.Source: §9 · p. 9
  • The 15-meeting AMI validation uses manually segmented floor events rather than labels independently annotated under MPTB’s taxonomy.Corpus differences and in-slice tuning mean cross-corpus differences cannot be attributed causally to speaker count or synthetic speech, nor establish direct performance transfer.Source: §9 · p. 9
  • MPTB is English-only, and its synthesized accents do not represent real-world speaker populations.The study does not systematically test generalization across languages, regional accents, or demographics, so its findings are not universal patterns of floor control.Source: §9 · p. 9

Ethics Statement

The paper describes the corpus’s privacy and deployment boundaries and details human validation procedures and agreement measures.

Ethics Statement

  • The released corpus contains synthetic speech and fictional LLM-generated dialogue, not newly collected private conversations.MPTB is a diagnostic benchmark, not a safety certification; the authors advise considering performance alongside broader evaluations of recognition, safety, fairness, privacy, and task correctness.Source: Ethics Statement · p. 9

A.1 Human Annotation

  • Human validation covers all 240 conversations and 6,116 candidate events, with three independent annotations per event.Native or near-native English annotators receive guidelines, examples, and calibration; they inspect synchronized speaker channels with 10 seconds of surrounding context, label event and floor-holder changes, and timestamp boundaries. A blind repeat of 25% of examples checks consistency; future context means the study validates perceptual realization, not causal predictability at onset.Source: §A.1 · p. 11

A.2 Consensus and Agreement Measures

  • The study distinguishes categorical consensus, chance-corrected Fleiss’ κ, temporal boundary F1 within 500 ms, and generator–human agreement.Generator–human agreement is conditional on events with human consensus, so it is not accuracy over all generated events; unresolved candidates lack categorical consensus.Source: §A.2 · p. 11

A.3 Human Validation Results

Human validation finds that generated floor events are recognizable but vary in ambiguity, while the benchmark’s tracks, event anchors, and generation pipeline define a controlled evaluation of floor transitions.

A.3 Human Validation Results

  • 81.4% consensus and boundary F1 = 0.89 for successful interruptions mark them as less consistently identified than backchannels, which reach 93.6% consensus and boundary F1 = 0.95.Successful interruptions also have 91.2% generator–human agreement among consensus events; high conditional agreement does not establish that every generated event is unambiguous.Source: §A.3 · pp. 11–12, Table 6 · p. 13
  • Categorical consensus falls from 87.8% at N = 2 to 80.1% at N = 5, while Fleiss’ κ drops from 0.84 to 0.76 and boundary F1 from 0.96 to 0.90.The decline is descriptive, not an isolated causal effect of speaker count, because event composition and acoustic conditions may differ.Source: §A.3 · p. 12, Table 7 · p. 13

B.1 Anchor Derivation

  • EOT positives anchor at the offset of a yielding TURN, while INT positives anchor at the onset of a successful floor claim; holds and non-floor-taking vocalizations supply hard-negative spans.Interrupted TURN events and unsuccessful claims are excluded from their respective scoring windows.Source: §B.1 · p. 12

B.2 Matching, Metrics and Operating Points

  • Predictions match positives within [t−0.25, t+3.0] seconds, and thresholds are selected on development data under FPRdev ≤ 0.10 then frozen for test and subgroup analyses.A system is in budget when FPRdev ≤ 0.10 and FPRtest ≤ 0.15; signed latency is prediction time minus gold time.Source: §B.2 · pp. 12–13

C.1 Role-relative event taxonomy

  • The taxonomy defines events relative to the current floor holder, and labels come from a generator that proposes symbolic acts and resolves their outcomes rather than inferring them from audio.The taxonomy table reports each class’s scoring role and corpus frequency.Source: §C.1 · p. 13, Table 13 · p. 15

C.2 Interaction-Type Definitions

  • The six interaction types condition dialogue generation on intended conversational regimes, but do not enforce deterministic event patterns.Realized backchannels, interruptions, overlap, and turn durations are measured separately.Source: §C.2 · p. 13, Table 14 · p. 16

C.3 Generation pipeline

  • The five-stage pipeline generates dialogue, orchestrates floor states, synthesizes separate speaker channels, adds controlled noise, and assembles audio with an event log.All 240 conversations pass acoustic checks; 2,369 of 2,436 overlap events (97.2%) are realized as simultaneous speech.Source: §C.3 · pp. 13, 15

B MultiPartyTurnBench Evaluation Tracks

  • The two tracks contrast floor-changing events with conversationally similar events that leave ownership unchanged, preventing success through indiscriminate silence or speech-activity detection.The human study validates plausibility of generator-selected events, not independent event discovery or real-time detectability, and does not establish that synthetic speech captures natural interaction variability.Source: §A.3 · p. 12, Table 11 · p. 15

D.1 Benchmark Analysis

MPTB balances speaker count and interaction type while evaluating causal floor-control decisions on synthetic and real audio. Results vary by task and setting: interruption detection is especially sensitive to interaction type and party count, and real-audio comparisons show domain sensitivity.

D.1 Benchmark Analysis

  • MPTB contains 240 conversations across four speaker counts and six interaction types, with 48 development and 192 test conversations and disjoint speaker voices.The 24 conditions contain 10 conversations each; the corpus totals 6.89 hours of speech.Source: §D.1 · p. 16, Table 20 · p. 20
  • Interaction type shifts conversational dynamics more than speaker count: collaborative dialogue has the highest interruption rate at 1.33/min, while narrative turns average 9.22 s.Casual dialogue has the fastest exchanges (19.4 events/min) and shortest turns (4.81 s).Source: §D.1 · pp. 17–18

E.1 Offline Baseline Configuration

  • Offline evaluation uses causal timestamped scores, with learned INT scores generally gated by activity on another speaker’s channel and thresholds frozen from development data.VAP is excepted from the other-speaker gate; thresholds are not adjusted by party count or interaction type.Source: §E.1 · p. 19

E.3 Full-Duplex Behavioral Evaluation

  • Full-duplex trials balance four event conditions across 120 scenarios, repeated three times on five agents; START scores entering after genuine EOT versus falsely during a hold, and STOP scores yielding to interruption versus backchannel.Agents receive streaming speech without scenario labels or reference event times.Source: §E.3 · p. 20

F.1 Synthetic vs Real-world Turn Taking in Multi-party Setting

  • VAP’s INT recall ranges from 94.5% on official dyadic TurnBench to 35.9% on four-party AMI and 11.7% on MPTB, while FPR stays between 2.8% and 10.7%.The sources differ in audio, labels, and threshold selection, so absolute values are not comparable across columns.Source: §F.1 · p. 21, Table 24 · p. 22

F.3 Performance by party count

  • As speaker count rises, Kyutai’s EOT recall increases from 84.6% to 96.6%, while its INT recall falls from 63.3% to 48.7%; acoustic INT recall rises alongside 82–90% FPR.The reported acoustic trend is attributed to more speakers producing more overlap onsets of every kind.Source: §F.3 · p. 22, Table 26 · p. 24

F.4 Operating Points; F.5 Latency Distributions; F.6 Paired Comparisons; F.7 Robustness to Acoustic Conditions

  • WavLM Anchor and Turn-1-Mini exceed the test FPR ceiling despite meeting the development constraint, while VAP is the only in-budget system with a sub-200 ms median EOT decision.Paired comparisons find Kyutai and VAP statistically indistinguishable in EOT recall; Kyutai, DualTurn, and MuVAP significantly exceed VAP on INT recall with small significant FPR increases. Across the rendered 20–38 dB range, learned EOT recall shows no monotonic loss at lower SNR.Source: §F.4 · p. 22, §F.5 · p. 23, §F.6 · p. 23

F.2 EOT and INT detection recall-FPR breakdown by interaction type

  • Across interaction types, EOT recall is comparatively stable, while INT recall varies substantially; Kyutai ranges from 47.0% in collaborative talk to 77.5% in instructional talk.Per-type cells contain 32 test conversations; their EOT false-positive rates are based on only 21–38 negative spans and are not reliable for type comparisons.Source: §F.2 · p. 22, Figure 9 · p. 23

F.8 Error Anatomy

Error analyses identify distinct sources of false floor decisions, including vocalization duration, question-shaped pauses, and overlapping failed floor claims. Full-duplex behavior and offline–online comparisons further show that yielding, latency, and detector correctness do not map uniformly onto agent behavior.

F.8 Error Anatomy

  • Kyutai’s INT advantage over VAP comes largely from vocalization duration: its recall rises from 37.2% for 1–3-word claims to 62.6% for claims of 9+ words, while long negative vocalizations trigger false alarms.For negatives with four or more words, Kyutai fires on 72.3%; DualTurn and MuVAP show the same direction with flatter slopes.Source: §F.8 · p. 24
  • Question-ended hold pauses attract false EOT triggers, while every learned system detects true yields after questions more reliably than after statements.The pattern is consistent with reliance on clause completion; VAP’s true-yield recall is 96.9% after questions versus 90.8% after statements.Source: §F.8 · pp. 24–25
  • Contested holds drive speaker-versus-rest false alarms: VAP triggers on 39.3% of contested holds versus 1.9% of uncontested holds, as competing speech in the pooled rest channel resembles floor release.Contested holds are 28 of 186 test hold spans; channel-isolated Kyutai is nearly unaffected.Source: §F.8 · p. 25

G.1 Per-family counts and accuracy

  • MiniCPM-o ignores 73 of 74 valid interruptions with perfect consistency, while Gemini yields to 86 of 88 backchannels.GPT-Live-1 is less consistent on STOP trials, and its yield rate rises with party count alongside backchannel accuracy.Source: §G.1 · p. 26

G.2 Latency; G.3 Barge-in instrumentation

  • Gemini ends its ongoing utterance a median 230 ms after interruption onset, whereas MiniCPM-o’s 8.6 s median reflects finishing its answer before stopping.Gemini’s stops on true interruptions and 95.5% of stops on backchannels coincide with provider barge-in events; GPT-Live-1 often briefly yields and reclaims the floor.Source: §G.2 · p. 26, §G.3 · p. 26, Figure 11 · p. 26

H Offline–Online Association

  • Offline INT correctness is associated with higher STOP behavioral correctness (.584 versus .422; OR 1.91 [1.34, 2.73]), but START shows no such association.Model-level rank associations are weak, and the authors do not interpret the pooled association because START and STOP have different base rates.Source: H Offline–Online Association · p. 27

H.1 Sim2Real Gap

  • The real-audio evaluation uses compatible detectors on dyadic TurnBench and four-party AMI, with corpus-specific operating points where development data are available.Because recording conditions, interaction structure, and speaker composition differ, cross-corpus comparisons characterize domain sensitivity rather than isolating a causal effect of speaker count.Source: §H.1 · p. 27

I Qualitative Analysis; J Reproducibility Checklist

  • Qualitative figures show frozen-point detector decisions and repeated agent scenarios alongside selected scored events; non-generative release stages are deterministic given released seeds.The examples are selected to illustrate patterns quantified in the error analysis.Source: I Qualitative Analysis · p. 27, Figure 12 · p. 29, Table 38 · p. 29
Loading 2610.20172v1…