PaperlayerView exact revision on arXiv

Source-linked AI summary

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

Chengqian Ma, Wenhao Feng, Weixuan Jin, Gaole Dai, Tianyu Xie, Yuexiao Ma, Zhaolu Kang, Xiangyu Zhao, Xiawu Zheng, Fei Chao

arXiv:2609.31948v1cs.SDeess.AS

TL;DR

Full-duplex assistants in shared conversations must decide when to speak, remain silent, or stop, but existing benchmarks offer limited evidence for this setting. Duplex-MPE evaluates those choices using paired multi-party scenarios and separate capability scores. MiniCPM-o 4.5 leads on three capabilities, while frequent speech does not ensure appropriate participation.

  • Problem

    Existing benchmarks largely center on a designated user, leaving open how well an assistant participates when any speaker may request help, provide relevant evidence, or resolve a request.

    Source: §1 · pp. 1–2

  • Method

    Duplex-MPE evaluates five speech systems on continuous audio from 2,000 multi-party scenarios, paired across explicit and implicit addressing, using four separate capability scores.

    Source: §1 · p. 2, §3.5 · p. 6

  • Results

    MiniCPM-o 4.5 leads on three scored capabilities, while frequent speech does not ensure appropriate participation; Gemini 3.1 Pro responds 64.3 percentage points more often to explicit than implicit requests.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Separate scores expose distinct failure profiles because speech presence alone does not determine whether systems answer, remain silent, or stop appropriately.

    Source: §5 · p. 10

    The synthetic English scenes do not represent real populations, room acoustics, or social norms, so results apply only to benchmark scenarios and do not support claims about particular speaker groups.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Real-time full-duplex speech models can listen while speaking, enabling natural interaction without rigid turn boundaries. Existing benchmarks evaluate turn-taking, interruption handling and multi-round dialogue, but largely centre on a designated user rather than an assistant participating in a shared conversation among several people. We introduce Duplex-MPE to evaluate when such an assistant should answer, remain silent or stop speaking. The benchmark contains 2,000 scenarios with three or four human speakers and one assistant, each paired across explicit and implicit addressing of the same request. Models receive continuous conversation audio without transcripts or supplied turn boundaries. Four scores measure fresh response initiation, answer accuracy, silence preservation and stopping when a human resolves a request. We evaluate five open-weight speech systems: MiniCPM-o 4.5, Moshi, FLM-Audio, Voila and Freeze-Omni. MiniCPM-o 4.5 leads on three scored capabilities, while frequent speech from other systems can coexist with inaccurate answers or failures to remain silent. A transcript-based Gemini 3.1 Pro reference responds 64.3 percentage points more often to explicit than implicit requests; paired tests detect no significant response-rate difference for the speech systems.

1 INTRODUCTION

Full-duplex assistants in shared conversations must track what people say while deciding whether to answer, stay silent, or stop. Duplex-MPE tests these decisions in continuous multi-party audio and finds that frequent speech alone does not indicate appropriate participation.

1 INTRODUCTION

  • In shared conversations, other speakers may address someone else, provide context for a later answer, or resolve a request while the assistant is speaking.The assistant must follow the conversation while deciding separately whether its participation is needed.Source: §1 · p. 1
  • Existing benchmarks leave open whether an assistant can participate in a shared conversation where any speaker may request help, provide evidence, or resolve a request.Prior benchmarks often expect an answer to isolated inputs, or test third-party speech while serving a designated user.Source: §1 · pp. 1–2
  • Duplex-MPE pairs explicit and implicit versions of 2,000 scenarios with three or four humans, testing requests that require an answer, silence, or stopping after human resolution.Systems receive a spoken preamble and continuous room audio without transcripts, speaker labels, turn boundaries, or candidate endpoints.Source: §1 · p. 2
  • MiniCPM-o 4.5 leads on three scored capabilities, while other systems show inaccurate answers, missed requests, or speech during turns requiring silence.Gemini 3.1 Pro responds 64.3 percentage points more often to explicit than implicit requests; the five speech systems show no statistically significant paired response-rate difference.Source: §1 · p. 2
  • Four separate scores assess fresh response initiation, answer correctness, silence preservation, and stopping after a human resolves a request.The evaluation pipeline uses generated scripts, synthesized audio, timing checks, semantic judgments, and human validation of sampled data and outputs.Source: §1 · p. 2

2 RELATED WORK

Prior work studies full-duplex interaction, addressee recognition, and selective response, but these remain distinct evaluation problems. Duplex-MPE combines them in a shared multi-party protocol where the assistant’s emitted speech is evaluated across turns.

Spoken dialogue and full-duplex evaluation

  • Full-duplex benchmarks assess when a system should take, hold, or yield the floor, while many spoken-language benchmarks expect an answer on every item.Table 1 compares benchmark task interfaces rather than model scores.Source: §2 · p. 3, Table 1 · p. 4

Addressee recognition and device-directed speech

  • Addressee-recognition and device-directed-speech studies typically predict an explicit label for a supplied utterance, whereas Duplex-MPE treats addressee recognition as a latent decision controlling speech.Source: §2 · p. 3

Selective responding and non-addressed speech

  • Selective-response benchmarks make silence valid, but Duplex-MPE also tests whether other participants provide context or resolve a request already addressed to the assistant.Existing audio benchmarks often test rejecting speech outside an interaction with a designated user.Source: §2 · p. 3

RELATED WORK

  • Duplex-MPE combines full-duplex interaction, addressee recognition, and response rejection in a protocol where every speaker contributes to dialogue state and the assistant’s waveform is evaluated.Each scenario is paired across two addressing forms while keeping the scenario and gold answer fixed.Source: §2 · p. 3

3 THE DUPLEX-MPE BENCHMARK

Duplex-MPE tests whether an assistant selectively participates in continuous multi-party audio, using paired addressing conditions and separate measures for responding, answering, silence, and stopping. Its protocol specifies turn-level actions and evaluates speech systems without transcripts or supplied turn boundaries.

3 THE DUPLEX-MPE BENCHMARK

  • 2,000 scenario pairs yield 4,000 evaluated conversations in which an assistant must respond selectively to continuous multi-party audio.Each scenario has three or four human speakers, and each pair holds the task and gold answer fixed across addressing conditions.Source: §3 · p. 4, §3.3 · p. 5

3.1 TURN TYPES AND EXPECTED ASSISTANT BEHAVIOUR

  • T is the sole ordinary-turn label requiring a response, while N1, N2 and N3 require silence.N4 separately tests whether the assistant responds to a question and stops when a human resolves it.Source: §3.1 · p. 4

3.2 DATASET CONSTRUCTION

  • Scripts specify speakers, utterances, turn labels and gold answers; T is not first and is followed by further conversation.Human turns are synthesized separately using a deterministic speaker-to-voice map, providing construction-level gold boundaries.Source: §3.2 · p. 4

3.3 PAIRED EXPLICIT AND IMPLICIT CONDITIONS

  • Each pair changes whether T names Aria or uses contextual cues, while keeping the scenario task and gold answer fixed.Some implicit rewrites also revise the preceding turn; final versions may differ in wording, event counts and audio, so comparisons concern complete paired scenarios.Source: §3.3 · p. 5

3.4 A SINGLE AUDIO-ONLY PROTOCOL

  • Speech systems receive continuous scenario audio without transcripts, speaker labels, turn boundaries, addressee labels or gold decisions.The benchmark restricts scored evaluation to full-duplex systems because segmented invocation would make fresh-onset scoring partly dependent on the harness and leave answering-window yield undefined.Source: §3.4 · pp. 5–6

3.5 METRICS

  • Four separate success rates measure fresh response initiation, answer accuracy, silence preservation and answering-window yield, with no aggregate score.Separate denominators prevent success on one capability from compensating for failure on another.Source: §3.5 · p. 6
  • Reviewers found 98%–100% pass rates on sampled construction checks and agreement with Opus 5 on 99/100 answer-correctness judgments.Reviewers also agreed with Opus 5 on 98/100 acknowledgement-versus-intrusion judgments.Source: §3.5 · p. 7

4 RESULTS

Across the evaluated systems, response presence alone does not reveal whether speech is newly initiated, correct, or appropriately absent. Addressing, stopping, request timing and scoring choices expose additional differences, while tests find no significant response-rate difference between explicit and implicit requests for the speech systems.

4.1 SEPARATE METRICS EXPOSE DISTINCT BEHAVIOURAL PROFILES

  • 0.9955 response presence but 0.2525 fresh-onset response rate for Freeze-Omni contrasts with MiniCPM-o’s 0.9545 and 0.9500, respectively.Similar response presence also masks conditional answer accuracy (0.4730 for MiniCPM-o versus 0.0011 for FLM-Audio) and silence preservation (0.9242 for MiniCPM-o versus 0.0002 for Freeze-Omni).Source: §4.1 · p. 8

4.2 EXPLICIT VERSUS IMPLICIT ADDRESSING

  • Gemini 3.1 Pro responds 0.9470 to named requests and 0.3040 to unnamed requests, while all five speech systems lack a significant response-rate difference.The speech-system explicit-minus-implicit differences range from −1.75 to +0.60 percentage points, and all five tests have p values above 0.05.Source: §4.2 · p. 9

4.2 EXPLICIT VERSUS IMPLICIT ADDRESSING; 4.3 FLOOR RELEASE IS A COVERAGE-CONDITIONED OUTCOME

  • MiniCPM-o produces speech after 95.45% of explicit T requests addressed to Aria and fails to preserve silence on 216/4,704 requests addressed elsewhere.For explicit N4 events, answering-window yield is 0.596 for MiniCPM-o, 0.192 for Moshi, 0.008 for FLM-Audio and 0.848 for Voila; Freeze-Omni has only one eligible event, so its rate is not displayed.Source: §4.2 · p. 9, §4.3 · p. 9

4.3 FLOOR RELEASE IS A COVERAGE-CONDITIONED OUTCOME

  • 968 of FLM-Audio’s 977 explicit scored events remain active 2 s after N4R begins, while 97 of Moshi’s 177 failed events resume speaking before the observation window ends.A brief pause alone does not meet the stopping criterion; silence must persist through the end of the observation window.Source: §4.3 · p. 9

4.4 RESPONSE PERFORMANCE BY REQUEST TIME

  • Voila’s response presence falls from 71.64% to 59.55% across earliest-to-latest explicit request groups, while MiniCPM-o’s explicit answer accuracy falls from 50.69% to 46.82%.FLM-Audio’s fresh-onset rate also declines across request-time groups while its response presence remains above 94% in every group; the other four models’ answer accuracy stays below 2%.Source: §4.4 · pp. 9–10

4.5 SENSITIVITY AND UNCERTAINTY

  • Changing from response presence to fresh-onset response rate changes model rankings, with Spearman correlation −0.4303 across ten condition entries.The silence-preservation ranking is unchanged across speech-duration thresholds, and bootstrap intervals support its ordering for all five models under both conditions.Source: §4.5 · p. 10
  • MiniCPM-o has significantly higher conditional answer accuracy than every other model on shared response-present requests; Moshi exceeds FLM-Audio under both conditions.Moshi’s comparison with Freeze-Omni is significant for implicit requests (28/1,640 versus 2/1,640; pcorr = 1.74 × 10−5), but not explicit requests (20/1,623 versus 5/1,623; pcorr = 0.0815).Source: §4.5 · p. 10

5 CONCLUSION

Duplex-MPE evaluates selective speech control in paired multi-party audio scenarios using four separate capability scores. The results show why a single response rate can conceal distinct behaviours, while the synthetic benchmark setting limits the scope of conclusions.

5 CONCLUSION

  • Duplex-MPE tests when an assistant should begin speaking, answer, remain silent or release the floor across 2,000 addressing-inverted scenario pairs.It reports four capability scores on separate denominators and provides a pipeline for larger or fresh evaluation draws.Source: §5 · p. 10
  • Across five systems, speech presence does not determine fresh initiation, answer accuracy, silence preservation or stopping behaviour.The evaluation exposes missed requests, low-content responding, intrusion and near-continuous speech as distinct profiles.Source: §5 · p. 10
  • The transcript-conditioned reference shows a large explicit-versus-implicit response-rate effect, whereas the speech systems show no significant paired effect.The conclusion describes the reference contrast as a paired intervention and does not claim the same effect for the speech systems.Source: §5 · p. 10
  • Synthetic English scenes do not represent real populations, room acoustics or social norms, so results are limited to benchmark scenarios.The scenes may inherit biases from the generating and speech-synthesis models; they do not support claims about particular speaker groups.Source: ETHICS STATEMENT · p. 11

A1 RELATED WORK AND EVALUATED SYSTEMS

Duplex-MPE combines questions studied separately in turn-taking and addressee recognition: selective speech control in an audio-only, multi-party interaction. It is positioned alongside adjacent understanding and proactivity evaluations, and the evaluated systems were selected for continuous-audio speech control.

A1.1 FULL-DUPLEX DIALOGUE AND TURN-TAKING

  • Turn-taking work studies floor-taking, while addressee-recognition work identifies whom an utterance addresses; Duplex-MPE combines these decisions in audio-only interaction.The related-work discussion cites turn-taking research and addressee-recognition work (Jovanovic & op den Akker, 2004; Ouchi & Tsuboi, 2016; Gu et al., 2021; Inoue et al., 2025; Fukuda et al., 2026).Source: A1.1 FULL-DUPLEX DIALOGUE AND… · pp. 15–16

A1.2 ADJACENT EVALUATION TASKS

A1.3 EVALUATED SYSTEMS

  • The five systems were selected for public code and weights, continuous audio input and an interface allowing speech emission while input continues.Moshi uses the male-voice moshiko checkpoint, and Voila denotes the autonomous-preview checkpoint.Source: A1.3 EVALUATED SYSTEMS · p. 16

A2 DATASET CONSTRUCTION AND ACCOUNTING

The dataset pairs explicit and implicit versions of the same request in multi-party scenarios, with additional controls on request placement, speech synthesis, paired edits, and audio accounting.

A2.1 A COMPLETE SCENARIO AND ITS PAIRED REQUEST

A2.2 SCENARIO COMPOSITION AND REQUEST PLACEMENT

A2.3 SPEECH SYNTHESIS

  • Qwen3-TTS synthesises human utterances separately, mapping within-scenario speaker identifiers A–D to four fixed voices; manifest records include turn labels and T's gold answer.The speaker identifiers are not personal names.Source: A2.3 SPEECH SYNTHESIS · pp. 16–17

A2.4 PAIRED EDITS AND AUDIO ACCOUNTING

  • The addressing rewrite changes only T in 1,958 pairs, while final scripts differ only at T in 1,661; all pairs retain the same T gold answer.The 4,000 conversations contain 110.93 hours including reused clips and inter-turn gaps, and 61.10 hours of distinct clips.Source: A2.4 PAIRED EDITS AND AUDIO ACCOUNTING · p. 17

A2.5 DURATION STATISTICS

A3 RUNTIME PROTOCOL AND SCORING

The protocol presents continuous audio under a shared scheduler and scores response initiation, answer quality, silence, and stopping-related behavior. Human checks and duration-threshold analyses assess the scoring procedure and its robustness.

A3.1 SHARED SCHEDULER AND DUTY INSTRUCTION

  • The shared scheduler gives models 5 s after T to start speaking and applies the same timing settings to all five systems.The spoken instruction directs Aria to answer only direct, answerable requests and stay silent otherwise.Source: A3.1 SHARED SCHEDULER AND DUTY… · p. 18

A3.2 FRESH ONSET AND RESPONSE PRESENCE

A3.3 SEMANTIC SCORING

  • VAD-empty windows pass directly; detected speech is transcribed and classified, brief non-informative backchannels are allowed, and detected speech without a valid classification fails scoring.Treating Voila's 171 empty explicit-condition transcripts as passes would raise its silence preservation from 83.51% to 84.35%.Source: A3.3 SEMANTIC SCORING · p. 19

A3.4 HUMAN VALIDATION OF DATA AND SEMANTIC SCORING

A3.5 SENSITIVITY TO DETECTED-SPEECH DURATION

A3.6 REQUEST TIMING AND RESPONSE QUALITY

  • Request-time groups use input timing, not model scores; answer accuracy is conditional on response presence, while response rates use all requests in each group.The 4,000 requests are divided into approximately equal earlier, middle, and later groups.Source: A3.6 REQUEST TIMING AND RESPONSE QUALITY · p. 21

A4 N4 STOPPING ANALYSIS

The stopping analysis defines eligibility around the N4 question and human resolution, then measures whether an eligible assistant stops by a fixed deadline. Eligibility counts qualify interpretation of the reported outcomes.

A4.1 ELIGIBILITY AND STOPPING RULE

  • Only models first speaking during the 3 s after N4Q and still speaking at N4R enter the stopping denominator; passing requires silence at N4R+2 s.If the human resolution and following silence extend beyond the deadline, the model must remain silent through their end; Freeze-Omni has one eligible event per condition.Source: A4.1 ELIGIBILITY AND STOPPING RULE · pp. 21, 23, Table 14 · p. 23

A4.2 STOPPING OUTCOMES

  • FLM-Audio was speaking at the deadline in 968 of 977 explicit events; Moshi resumed before observation ended in 97 of its 177 failures.Table 15 separates successful stopping, speech spanning the deadline, and later resumed speech.Source: Table 15 · p. 23, A4.2 STOPPING OUTCOMES · p. 23

A5 ADDRESSING AND TRANSCRIPT-REFERENCE CONTROLS

The section compares paired explicit and implicit request versions and uses transcript-conditioned Gemini evaluations as a reference. It also examines silence-preservation failures by turn type and addressee.

A5.1 PAIRED ADDRESSING COMPARISON

  • The 2,000 scenarios were evaluated in paired explicit and implicit versions, with response defined as detected speech in the response window and correctness scored separately.For MiniCPM-o, the paired comparison yielded 90 explicit-only versus 78 implicit-only responses, a net difference of 0.60 percentage points.Source: A5.1 PAIRED ADDRESSING COMPARISON · pp. 23–24, Table 16 · p. 24

A5.2 TEXT-BASED EVALUATION OF RESPONSE DECISIONS AND ANSWERS

  • Gemini’s transcript evaluation made independent RESPOND-or-SILENT decisions from each utterance and preceding speaker-attributed dialogue, without future turns or prior generated answers.It scored response presence on T requests, silence preservation on N1–N3 utterances, and window response on N4Q questions; it separately checked addressee identification.Source: A5.2 TEXT-BASED EVALUATION OF RESPONSE… · p. 24
  • The text evaluation cannot measure fresh-onset response rate or stopping after an N4R turn begins.Across two Gemini text-evaluation runs, decisions agreed on 97.2% of 48,542 cases, and the large paired addressing contrast was reproduced.Source: A5.2 TEXT-BASED EVALUATION OF RESPONSE… · pp. 24–25
  • Gemini’s explicit-versus-implicit response-rate difference was 0.6430, while paired tests found no significant response-rate difference for any of the five speech systems.The paired tests do not establish that the speech systems’ response rates are equivalent.Source: A5.2 TEXT-BASED EVALUATION OF RESPONSE… · p. 24, A5.1 PAIRED ADDRESSING COMPARISON · p. 24

A5.3 SILENCE PRESERVATION BY TURN TYPE

A6 QUALITATIVE EXAMPLES

The section presents selected model outputs from silence-requiring windows alongside the judge’s reasons for allowing or penalising speech.

A6 QUALITATIVE EXAMPLES

A7 STATISTICAL COMPARISONS

Paired statistical comparisons support a clear ordering for silence preservation but only a partial ordering for conditional answer accuracy. The analyses use matched evaluation items and corrected tests.

A7.1 CONFIDENCE INTERVALS AND PAIRED TESTS

  • All model pairs differed significantly in silence preservation under both addressing conditions, supporting the ordering MiniCPM-o, Voila, FLM-Audio, Moshi, then Freeze-Omni.For explicit addressing, the scenario-level bootstrap placed MiniCPM-o 8.25 to 9.58 percentage points ahead of Voila.Source: A7.1 CONFIDENCE INTERVALS AND PAIRED… · p. 26, Table 21 · p. 27
  • MiniCPM-o exceeded all other models in conditional answer accuracy, but the tests did not establish a complete ranking for that metric.Moshi exceeded FLM-Audio under both conditions; Moshi–Freeze-Omni was significant only under implicit addressing, where both answered fewer than 2% of shared requests correctly.Source: A7.1 CONFIDENCE INTERVALS AND PAIRED… · p. 26
  • Pairwise McNemar tests compare models on common items, with answer-accuracy comparisons restricted to T requests where both models produced speech.The tests use Bonferroni correction across 20 comparisons per addressing condition; silence-preservation intervals resample scenarios while keeping each scenario’s windows together.Source: A7.1 CONFIDENCE INTERVALS AND PAIRED… · pp. 26–27, Table 21 · p. 27
Loading 2609.31948v1…