PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?

Jiaheng Dong, Shihong Tan, Gongping Huang, Jean Honorio, Abhirup Ghosh, Ting Dang

arXiv:2610.17738v1cs.AI

TL;DR

Existing benchmarks lack structured action-level coverage and often assess whether an interaction behavior occurs rather than whether it succeeds. DUPLEXACT evaluates FD-SLMs using a hierarchical taxonomy, two-stage success criteria, and a real-world-grounded dataset; evaluations reveal uneven capabilities and frequent failures to follow through after immediate success.

  • Problem

    Existing FD-SLM benchmarks lack a structured action-level formulation, often measure action occurrence rather than success, and provide limited coverage of diverse real-world conversations.

    Source: §1 · pp. 1–2

  • Method

    DUPLEXACT defines a three-level taxonomy of eight actions and 24 conditions, evaluates immediate execution and subsequent follow-through, and provides 4,691 annotated instances across four scenarios.

    Source: §1 · p. 2

  • Results

    Evaluations of 14 models show uneven action-taking capabilities, weak follow-through beyond immediate action-taking, and limited adaptation to conversational norms across scenarios.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Current models perform unevenly across the action space, and successful immediate execution frequently does not carry through to the behavior an action requires.

    Source: §5.2 · p. 8, §6 · p. 9

    Some conditions with limited natural coverage are complemented by targeted synthesis, and Simulator Replay currently requires compatible model interfaces.

    Source: §6 · p. 10

Abstract

from arXiv · show

End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriate interaction actions as conversations unfold. However, existing benchmarks lack a structured action-level formulation, often evaluate action occurrence rather than success, and provide limited coverage of diverse real-world conversations. We introduce DuplexAct, a benchmark for evaluating interaction action taking in E2E FD-SLMs. DuplexAct defines a three-level action taxonomy spanning eight interaction actions and 24 corresponding interaction conditions. We further characterize successful action-taking through immediate execution and subsequent follow-through, and develop action-specific evaluation metrics for both. DuplexAct contains 4,691 test instances spanning casual conversation, meetings, customer service, and interviews, including 2,531 instances derived from real-world conversational recordings. Together, DuplexAct provides a structured and real-world-grounded framework for systematically assessing how E2E FD-SLMs participate in ongoing spoken interaction.

1 INTRODUCTION

Existing E2E full-duplex spoken-language benchmarks lack a principled action-level framework and measures of whether actions succeed both immediately and through follow-through. DUPLEXACT addresses these gaps with a structured taxonomy, two-stage success criteria, a real-world-grounded dataset, and evaluation of current models.

1 INTRODUCTION

  • Existing benchmarks cover only subsets of actions, often score behavior occurrence rather than success, and rely heavily on synthetic or scripted interactions.Source: §1 · p. 2
  • DUPLEXACT organizes interaction into a three-level taxonomy spanning eight actions and 24 conditions.Source: §1 · p. 2
  • Its action-specific success criteria assess both immediate execution and subsequent follow-through.Source: §1 · p. 2
  • The dataset contains 4,691 annotated instances across 24 conditions and four conversational scenarios, with 54% drawn from real-world interactions.Source: §1 · p. 2
  • Evaluating 14 models reveals uneven performance across actions, weak subsequent reasoning beyond immediate action-taking, and limited adaptation to conversational norms across scenarios.Source: §1 · p. 3

2 RELATED WORK

Prior FD-SLM controls and benchmarks use inconsistent granularities and limited action coverage, while common occurrence-based metrics do not assess whether actions are successfully carried through.

2 RELATED WORK

  • FD-SLMs use model-specific control tokens or signals at different granularities, leaving no common structured action space for systematic evaluation.Source: §2 · p. 3
  • Existing benchmarks cover arbitrary subsets of actions and sometimes mix model actions with the conversational conditions calling for them.Source: §2 · p. 3
  • Occurrence and timing metrics cannot distinguish disruptive from appropriate backchannels or establish whether an action is followed through.Source: §2 · p. 3

3 CONSTRUCTING DUPLEXACT

DUPLEXACT combines a hierarchical action-and-condition taxonomy, a dataset curated from real conversations and targeted synthesis, two-stage success measures, and controlled replay pipelines.

3.1 INTERACTION ACTION SPACE AND CONDITION CONSTRUCTION

  • The taxonomy separates observable participation state, interaction actions, and the conditions under which each action is required.Source: §3.1 · p. 4
  • Eight actions adapt human interaction categories: four for turn management, three for interactive listening, and Recover for breakdown repair.Source: §3.1 · p. 4

3.2 BENCHMARK DATA CURATION

  • The benchmark covers 24 action–condition pairs across casual conversation, meetings, customer service, and interviews, using synthesis when natural data are insufficient.Source: §3.2 · p. 4
  • Real-world segments are selected using action-specific signals and retained after GPT-5.4 verifies the intended action and condition label.Source: §3.2 · p. 4
  • Human validation on 10% of test instances yields 97.9% majority-vote agreement for action labels and 95.5% for condition labels.Source: §3.2 · p. 5

3.3.1 ACTION SUCCESS DEFINITION AND METRICS

  • Success assessment pairs rule-based immediate-execution checks with GPT-5.4 evaluation of whether later behavior remains consistent with the action.Source: §3.3.1 · p. 5
  • Immediate success rate measures instances meeting the immediate criterion; follow-through success rate measures immediate successes that also meet the subsequent criterion.Source: §3.3.1 · p. 5

3.3.2 INTERACTION PIPELINE

  • Direct Replay uses fixed audio prompts, while Simulator Replay adapts user-cue timing to the model’s ongoing response for bidirectional interaction.Source: §3.3.2 · pp. 5–6

4 EXPERIMENTAL SETUP

The experiments evaluate 14 recent FD-SLMs under two replay protocols, with simulator-based evaluation limited to systems supporting live bidirectional interaction and aligned streaming signals.

4 EXPERIMENTAL SETUP

  • The study evaluates 14 models, with all 14 tested on four Direct Replay actions and 10 interface-compatible models tested on simulator-based actions.Source: §4 · p. 6
  • Open-source systems generally run on one NVIDIA A100 GPU, and input audio is 16-kHz mono PCM, resampled as needed.Source: §4 · p. 6
  • Table 4 reports immediate-execution and subsequent-follow-through success rates plus action-specific timing measures.Source: Table 4 · p. 7

5 RESULTS

Across actions, conditions, and scenarios, model capabilities are uneven; successful immediate execution often does not carry through to the behavior an action requires. Performance also varies with conversational setting, especially for follow-through.

5.1 IMMEDIATE EXECUTION PERFORMANCE

  • 94.0% vs 75.4% median immediate success rates for Wait vs Ignore; close-source models score 77.7–97.0% on Recover vs 27.5–41.3% on Take Turn.These contrasts show that similar immediate behaviors do not imply equivalent capability across actions.Source: §5.1 · p. 7
  • No model performs consistently well across the full action set; targeted supervision is associated with strengths on the actions it covers.Raon-SpeechChat leads open-source Backchannel at 31.0%, while BayLing-Duplex reaches 98.8% on Stop and 49.0% on Recover.Source: §5.1 · p. 7
  • 60.9% success for delayed-entry Take Turn contrasts with 8.0% for immediate entry and 20.9% for slight overlap, while Recover stays at 49.3–55.1% across conditions.Condition-level variation reveals differences that action-level scores alone do not capture.Source: §5.1 · p. 8, Figure 3 · p. 8

5.2 SUBSEQUENT FOLLOW-THROUGH BEYOND IMMEDIATE EXECUTION

  • Successful immediate execution often fails to carry through: Moshi has 67.5% immediate success on Interrupt but nearly 100% subsequent failure, and Wait can fail to incorporate information heard while silent.Follow-through is evaluated only after successful immediate execution.Source: §5.2 · p. 8, Figure 4 · p. 8
  • 90.5%, 85.4%, and 88.9% immediate success on Hold Floor conditions contrast with subsequent success of 73.3% for continuing and 4.5% and 5.7% for shortening and simplifying.Close-source models also achieve 78.6–83.0% subsequent success on Take Turn and 78.6–90.3% on Recover, but only 39.3–57.3% on Hold Floor; ongoing-turn actions remain difficult.Source: §5.2 · pp. 8–9

5.3 PERFORMANCE ACROSS INTERACTION SCENARIOS

  • Wait, Ignore, and Stop remain strong across scenarios, but Wait is weaker in meetings and follow-through for Wait and Take Turn varies more by setting.The authors suggest broader scenario coverage in full-duplex training to expose models to varied interaction norms and contexts.Source: §5.3 · p. 9, Figure 5 · p. 9

6 LIMITATIONS AND CONCLUSION

DUPLEXACT frames full-duplex interaction around a shared action space and evaluates both immediate execution and subsequent follow-through. The authors identify uneven model capabilities and two boundaries in benchmark coverage: sparse natural examples for some conditions and interface requirements for Simulator Replay.

6 LIMITATIONS AND CONCLUSION

  • Current models show uneven action-taking, and successful immediate execution frequently fails to produce the required subsequent behavior.The findings point to a gap between immediate interaction control and the semantic consequences of an action.Source: §6 · p. 9
  • DUPLEXACT provides a shared action space for learning what an action requires as interaction continues and for tracking action-level progress.The authors present the framework as guidance for future model development.Source: §6 · p. 10
  • Conditions with limited natural coverage are supplemented by targeted synthesis, and Simulator Replay currently requires compatible model interfaces.The authors identify additional conversational corpora and more flexible simulator designs as ways to broaden coverage.Source: §6 · p. 10

A INTERACTION-ACTION TAXONOMY DETAILS

This appendix section details the interaction-action taxonomy introduced in Section 3.1, covering construction of the L1 action space and definitions of its 24 L2 conditions.

A INTERACTION-ACTION TAXONOMY DETAILS

A.1 CONSTRUCTION OF THE L1 ACTION SPACE

DUPLEXACT adapts human interaction-analysis categories into an eight-action L1 taxonomy and specifies 24 L2 conversational conditions in which those actions are expected. The conditions distinguish both interactional circumstances and the response appropriate to them.

A.1 CONSTRUCTION OF THE L1 ACTION SPACE

  • The L1 action space draws on human interaction analysis across turn management, interactive listening, and breakdown repair, adapting and consolidating categories for E2E FD-SLMs.The mapping is not one-to-one: multiple breakdown-repair forms are consolidated into Recover.Source: §A.1 · p. 13, Table 5 · p. 13
  • Wait and Ignore distinguish retaining silence while the current speaker holds the floor from withholding a response to speech that does not require model participation.Both can produce silence at the L0 level, but represent different interactional decisions.Source: §A.1 · p. 13

A.2 L2 INTERACTION CONDITIONS

  • L2 conditions specify conversational circumstances for each L1 action; the taxonomy covers eight actions and 24 conditions.Table 6 provides definitions and illustrative examples for the conditions.Source: §A.2 · p. 13, Table 6 · p. 13, Figure 6 · p. 15

L1 Action L2 Condition Definition Illustrative Example

  • Take Turn conditions distinguish immediate response, delayed entry after a late shift or constraint, and slight overlap when remaining speech cannot change the response.The examples contrast a final key entity completing a request with a final shift that requires re-planning.Source: L1 Action L2 Condition Definition… · p. 14
  • Other conditions cover stopping for a substantive interruption, continuing through background or other-directed speech, and proactive interruption to help or address safety.Brief affirmative or uncertainty signals are also defined for cases where the user retains the floor.Source: L1 Action L2 Condition Definition… · pp. 14–15
  • Hold Floor conditions distinguish continuing after a backchannel, shortening when the user understands, and simplifying when the user is uncertain or struggling to follow.The examples show the model retaining the floor while adapting its remaining response.Source: L1 Action L2 Condition Definition… · p. 14
  • Recover conditions specify how to resume after interruption: wrap up, repeat missed information, ask a follow-up, enrich, shift topic, or revise a response after correction.The condition definitions tie each response to the information or change conveyed by the interruption.Source: L1 Action L2 Condition Definition… · p. 14

A.3 DATA CURATION DETAILS

The curation combines naturally occurring interaction segments with benchmark-composition reporting. Real-world extraction uses structural cues followed by semantic verification, with timing and overlap constraints tailored to interaction structures.

A.3 DATA CURATION DETAILS

  • Figure 6 summarizes real-world and synthesized instances by scenario, taxonomy coverage, and overall dataset and audio statistics.The section introduces detailed curation procedures after this composition overview.Source: §A.3 · p. 15

A.3.1 REAL-WORLD INTERACTION EXTRACTION

  • Real-world segments are identified from speaker turns, timestamps, pauses, overlaps, and transcript boundaries, then semantically checked for the intended condition and conversational completeness.The source corpora are AMI, CANDOR, CORAAL, and Loqui.Source: §A.3.1 · p. 15
  • Extraction retains substantive speech segments rather than short backchannels as primary turns, using timing and overlap constraints suited to each interaction structure.Corpus-specific thresholds, prompts, and implementation details are released with the benchmark code.Source: §A.3.1 · p. 15

A.3.2 CONTROLLED INTERACTION SYNTHESIS

Controlled synthesis fills action–condition pairs lacking sufficient or reliably identifiable natural examples, while varying conversational context and rendering speech with diverse voices. Human annotation separately checks action and condition performance, and rates naturalness for synthesized instances.

A.3.2 CONTROLLED INTERACTION SYNTHESIS

  • 2,160 synthesized instances cover 11 action–condition pairs for which sufficient or reliably identifiable real-world examples are unavailable.Table 8 summarizes the synthesized conditions and the contextual dimensions varied during generation.Source: §A.3.2 · p. 15, Table 8 · p. 17
  • Transcripts use hierarchical scenario, action, and condition prompts, with varied topics or physical settings to diversify instances within action–condition pairs.Safety-based Interrupt examples span household and driving hazards.Source: §A.3.2 · pp. 15–16
  • For safety-based Interrupt examples, a second deterministic LLM pass truncates the user utterance when the hazard first becomes identifiable and the interruption becomes appropriate.The original wording before that point is preserved; the examples span five household and five driving hazard categories.Source: §A.3.2 · pp. 16–17
  • Synthesized transcripts are rendered with five realistic personas, while other-directed and background speech receive condition-specific acoustic processing.The five personas vary speaker characteristics, accents, speaking styles, and prosody; addressed speech remains unprocessed.Source: §A.3.2 · pp. 17–18

A.4.1 ANNOTATORS AND RECRUITMENT

  • The final annotator pool comprised 20 university students recruited through Wuhan University, aged 18–26 years.English proficiency was self-reported, with most participants reporting CET-6 scores.Source: §A.4.1 · p. 18

A.4.2 ANNOTATION DESIGN

  • Five annotators independently judge target-action performance and condition fit for each instance, and additionally rate content and voice naturalness for synthesized instances.Naturalness uses five-point scales, as specified among the human-validation questions.Source: §A.4.2 · p. 18, Table 10 · p. 18

A.4.3 VALIDATION RESULTS

Human judgments strongly support the assigned action and interaction-condition labels, with validity also reported across individual condition pairs. The validation analyzes 468 instances, including naturalness ratings for 216 synthesized examples.

A.4.3 VALIDATION RESULTS

  • 468 benchmark instances, or 10% of the test set, were analyzed, each with five ratings.Three action–condition cells contain fewer than 20 instances: 19, 16, and 13.Source: §A.4.3 · p. 18
  • Human judgments strongly support the assigned action and interaction-condition labels, with condition-level validity reported across all 24 action–condition pairs.Table 11 reports overall label validation, while Table 12 provides the condition-level breakdown.Source: §A.4.3 · p. 19, Table 11 · p. 19, Table 12 · p. 19
  • For 216 synthesized instances, annotators rated content and voice naturalness on five-point scales, summarized using MOS and the Non-negative Rate.The Non-negative Rate is the proportion of ratings of 3–5; results are summarized in Table 13.Source: §A.4.3 · p. 19

B DETAILED ACTION DEFINITIONS AND EVALUATION

This appendix details the evaluation of interaction actions through immediate execution and, when immediately successful, subsequent follow-through, using rule-based assessment, action-specific checks, and timing diagnostics. It specifies evaluations for Interrupt, Backchannel, Ignore, and Recover, alongside the Direct Replay and simulator-based interaction pipelines and model-output transcription setup.

B DETAILED ACTION DEFINITIONS AND EVALUATION

  • The evaluation separates immediate execution from conditional follow-through, using time-aligned rules for immediate success and action-specific checks for later success.Backchannel execution additionally requires an audio-capable judge to verify the specified function; follow-through generally uses a single-item checklist with an LLM judge.Source: B DETAILED ACTION DEFINITIONS AND… · p. 20

B.1 TIMING DIAGNOSTICS

  • Timing diagnostics characterize when Stop, Interrupt, and Recover occur without changing their binary success criteria.Yield latency, interruption timing difference, and recovery latency indicate whether the action occurs before or after its relevant interruption point.Source: §B.1 · p. 20

B.2 WAIT

  • Wait tests withholding a substantive response during pauses or continuous speech, then reflecting at least some of the user’s continuing contribution in a later response.A speech segment of at most three words counts as a backchannel, not a substantive takeover; waiting through a pause does not require waiting until the user’s full turn ends.Source: §B.2 · pp. 20–21

B.3 TAKE TURN

  • Take Turn requires a substantive response within the condition’s interval and subsequent attention to the user’s intent, information, and constraints.The intervals are 0–0.5 s for immediate, 0–3.0 s for delayed, and −1.0–0.5 s for slight overlap; overlap is permitted but not required.Source: §B.3 · p. 22

B.4 STOP

  • Stop requires terminating the active turn after an interruption, then remaining yielded until the user intervention is nearly over.A new substantive turn before the intervention’s final 1.0 s fails follow-through; brief backchannels are allowed, and both stages are assessed automatically from time-aligned speech segments.Source: §B.4 · p. 23

B.5 HOLD FLOOR

  • Hold Floor requires maintaining speech through a cue without yielding, then preserving the response trajectory while making the condition-required adjustment.The longest silence gap across the cue interval and the following 1.0 s must not exceed 0.50 s; follow-through may require continuing, shortening, simplifying, or ignoring background or other-directed speech.Source: §B.5 · pp. 23–24

B.6 INTERRUPT

  • Interrupt requires a substantive intervention while the user is speaking and at least 1.0 s before the utterance ends, with follow-through serving the help or safety condition.The authored reference point informs the timing diagnostic only; it does not determine immediate-execution success.Source: §B.6 · p. 25
  • Interrupt follow-through succeeds only when the intervention provides condition-specific help or safety guidance, rather than merely announcing the interruption or giving a generic response.For safety, the response must identify the risk and explain why it is dangerous or what the user should do instead; for help, it must provide the assistance or constraint the situation requires.Source: §B.6 · pp. 25–26

B.7 BACKCHANNEL

  • Backchannel requires brief, function-appropriate listener feedback while the user retains the floor, followed by later speech consistent with its expressed stance or understanding.Direct Replay evaluates affirmative and uncertainty conditions; a candidate must be a genuine backchannel of at most three words, and its judged function must match the condition. A later stance change is acceptable when justified by intervening user speech; no later model speech fails follow-through.Source: §B.7 · pp. 26–27

B.8 IGNORE

  • Ignore succeeds when the model gives no substantive response during background or other-directed distractor speech, then keeps its later reply focused on the addressed interaction.Responses of at most three words count as backchannels, not substantive responses; later speech must not quote, echo, answer, or otherwise treat ignored speech as relevant.Source: §B.8 · pp. 27–28

B.9 RECOVER

  • Recover requires re-entry after an interruption and a condition-specific adjustment, such as wrapping up, repeating, following up, enriching, shifting topic, or revising.All six Simulator Replay conditions share an immediate criterion: a substantive recovery onset within the observation interval; up-to-three-word segments count as backchannels, and Recovery Latency is also reported.Source: §B.9 · pp. 28–29

C.1 DIRECT REPLAY

  • Direct Replay streams fixed user audio in real time, records output on a time-aligned timeline, and appends a 15-second observation period after user content.Timing begins at the first streamed input sample, not connection establishment, and scored speech onsets come from the recorded output timeline rather than provider turn-detection events.Source: §C.1 · p. 29

C.2 SIMULATOR-BASED INTERACTION

  • Simulator-based interaction preserves fixed prerecorded user audio while adapting cue timing to model behavior; Interrupt reverses the interaction so the user holds the floor until model takeover.For Stop, Hold Floor, and Recover, the simulator uses a lead-in and inserts a fixed cue while the model speaks, force-injecting if needed; transcript playback alignment prevents decisions based on unheard content. Its Interrupt decision only controls interaction progression, while immediate execution is scored independently from the recorded speech timeline.Source: §C.2 · p. 30

D MODEL SETUP; D.1 MODEL OUTPUT TRANSCRIPTION

  • Evaluated models use official implementations and recommended inference settings, with whisper-1 transcribing most outputs and native streaming transcripts used for BayLing-Duplex and Raon-SpeechChat.Native transcripts are used for those two models because temporal jitter in rendered audio makes post-hoc ASR less reliable for transcript-dependent evaluation.Source: D MODEL SETUP · p. 30, §D.1 · p. 30

E COMPREHENSIVE CONDITION-LEVEL RESULTS

This section reports comprehensive condition-level results for evaluated models and separate timing diagnostics for Stop, Recover, and Interrupt. It also identifies the model versions and checkpoints used in the experiments.

E COMPREHENSIVE CONDITION-LEVEL RESULTS

  • Close-source evaluations used the model versions served by their APIs during September 6–13, 2026; grok-voice-latest is a floating API model identifier.Source: E COMPREHENSIVE CONDITION-LEVEL RESULTS · p. 30
  • Table 14 lists the model versions and checkpoints used in the experiments.Source: Table 14 · p. 31
  • Table 15 reports Stop, Recover, and Interrupt timing as median (IQR, nT) seconds; smaller Stop values indicate faster yielding, Recover is relative to interruption end, and negative Interrupt values precede the authored target point.Source: Table 15 · p. 31, E COMPREHENSIVE CONDITION-LEVEL RESULTS · p. 31
  • Table 16 reports each model’s condition-level immediate-execution / subsequent-follow-through percentages, with entering-instance counts in parentheses; follow-through is conditional on successful execution and applicability.“n/a” indicates that no instance entered the corresponding subsequent evaluation.Source: E COMPREHENSIVE CONDITION-LEVEL RESULTS · p. 30, Table 16 · p. 32
Loading 2610.17738v1…