PaperlayerView exact revision on arXiv

Source-linked AI summary

Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents

Yunqi Lu, Tyler Baumgartner, Nikhil Johri, Brandon Tai, Candice Fan, Luc Debaupte, Ruben Aguilar, Bill Wang, Yi Zhong

arXiv:2609.13117v1cs.CLcs.SD

TL;DR

Existing frameworks do not cleanly isolate adapting while retaining the floor, although a listener’s contribution may shape an ongoing utterance without requiring a handoff. Duplex Cue labels listener intent and speaker response separately, then compares recorded responses with conditional PersonaPlex continuations on human-confirmed cues. PersonaPlex adapted on 34.8%, with remaining responses divided between unchanged continuation and yielding.

  • Problem

    Existing frameworks do not cleanly isolate adapting while retaining the floor, although a listener’s contribution may shape an ongoing utterance without requiring a handoff.

    Source: §1 · p. 1

  • Method

    Duplex Cue labels listener intent and speaker response separately, then compares recorded responses with conditional PersonaPlex continuations on human-confirmed cues.

    Source: §1 · pp. 1–2

  • Results

    PersonaPlex adapted on 34.8%, with remaining responses divided between unchanged continuation and yielding.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Evaluating full-duplex conversation should measure how a listener’s contribution shapes the utterance already underway alongside whether the speaker continues or yields.

    Source: §7 · p. 8

    The results cover one checkpoint and one generation per trial in English, with repeated participants, and only pairs meeting activity and scorability criteria.

    Source: §6 · p. 7

Abstract

from arXiv · show

Full-duplex evaluation often emphasizes whether an agent keeps speaking or stops. That binary cannot express a third response humans use routinely: continuing to speak while incorporating what the listener just contributed. The contribution may be a missing word, a correction or a clarification. We introduce Duplex Cue, an evaluation of this \emph{in-turn adaptation} in full-duplex voice agents. Duplex Cue separates listener intent (backchannel, collaboration, or interruption) from speaker behavior: continuing unchanged, adapting within the turn, or yielding. Adaptation includes acknowledgment as well as content revision. In a single-model case study using 300 human-confirmed cues from unscripted English conversations, we compare recorded human responses with PersonaPlex continuations generated while replaying the listener's audio. We retain 208 pairs with the ongoing speaker active at cue onset and a scorable response in each condition. On the 66 collaborative pairs, recorded speakers adapt in 68.2\% of cases, compared with 34.8\% for PersonaPlex. The model otherwise continues unchanged (42.4\%) or yields (22.7\%). These findings show why evaluating natural voice interaction requires measuring how an agent responds to a listener's contribution as well as whether it keeps speaking.

1 Introduction

Duplex Cue makes in-turn adaptation visible as a response distinct from continuing unchanged or yielding, addressing a gap in full-duplex evaluation. It separates listener intent from speaker behavior and pairs an evidence-linked cue inventory with a PersonaPlex case study.

1 Introduction

  • A listener can shape an ongoing utterance while the speaker retains the floor, a response that a continue-or-stop distinction cannot express.The paper calls this in-turn adaptation and includes acknowledgment, rewording, and substantive revision.Source: §1 · p. 1
  • Existing frameworks do not cleanly isolate adaptation while retaining the floor; a full-duplex survey [19] includes Repair on its intent axis but not in its diagnostics or response options.Its response axis has no category for continuing with revision.Source: §1 · p. 1
  • Duplex Cue separates listener intent—backchannel, collaboration, or interruption—from speaker response: continued, adapted, or yielded.This matrix distinguishes the contribution from the behavior it receives.Source: §1 · p. 1
  • The paper contributes an evidence-linked inventory of 2,591 cues from 20.32 hours of unscripted conversation and a paired PersonaPlex case study of collaborative cues.The study compares unchanged continuation, adaptation, and yielding against recorded responses.Source: §1 · p. 2

2 Related work

The related work spans conversation-analytic accounts of overlapping contributions, full-duplex benchmarks, and conditional dialogue continuation. Duplex Cue builds on these areas to measure how an ongoing speaker responds to different kinds of listener contributions.

Listener contributions and overlapping talk

  • Conversation-analytic work distinguishes listener contributions and overlap practices, motivating operational labels for cue intent and speaker response [19].The cited work covers continuers, collaborative completions, correction, and competitive versus noncompetitive overlap.Source: §2 · p. 2

Full-duplex and turn-taking evaluation

  • Prior benchmarks evaluate turn behavior, response quality, interruptions, corrections, and other events, while Duplex Cue separates unchanged continuation, in-turn adaptation, and yielding by cue type [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11].Its categories cover backchannels, collaborations, and interruptions in unscripted conversation.Source: §2 · p. 2

Conditional dialogue continuation

  • Duplex Cue adopts conditional continuation with recorded partner audio, selecting cutovers before listener contributions and comparing PersonaPlex generations with recorded responses [22] [23] [24] [25].The case study uses PersonaPlex, which adds voice and role conditioning to the Moshi speech-text architecture.Source: §2 · p. 2

3 Definitions

Duplex Cue describes overlapping conversation using two labels: the listener’s intent and the ongoing speaker’s response. Its definitions distinguish supportive contributions, collaborative shaping, floor claims, uptake, and handoff.

3 Definitions

  • A cue is Speaker B’s contribution while Speaker A holds the floor, and a response is A’s subsequent behavior; each receives a separate label.Source: §3 · p. 2

3.1 Cue intent

  • Backchannels support ongoing speech, collaborations help shape the utterance without claiming the floor, and interruptions attempt to take or retain it.Source: §3.1 · p. 2

3.2 Observed response

  • Continued means proceeding as if the cue were absent, Adapted means acknowledging or incorporating it within the turn, and Yielded means handing over the floor.Adaptation concerns local uptake, not whether a correction is correct.Source: §3.2 · p. 3

4 Methods

The methods build an inventory of human-confirmed overlapping cues, select a shared set of active and scorable trials, and compare recorded responses with conditional PersonaPlex continuations. Response labels are assigned from timed transcript evidence and checked against human ratings.

4.1 Corpus and cue review

  • The corpus comprises 80 dual-channel English conversations totaling 20.32 hours, with 39 participants; collection instructions requested natural, unscripted conversation.Recordings averaged 15.24 minutes, ranging from 14.99 to 30.36 minutes.Source: §4.1 · p. 3
  • Review screened 10,478 proposals for an ongoing turn at cue onset and a mid-sentence cutover, then retained 2,591 cues: 1,896 backchannels, 361 collaborations, and 334 interruptions.A balanced audio sample received 49/60 human label agreement (81.7%); seven of eleven disagreements crossed the Collaboration–Interruption boundary.Source: §4.1 · p. 3

4.2 Selection and eligibility

  • Multiple human reviewers selected 300 confirmed trials; the primary comparison requires A to be active at cue onset and both responses to be scorable, yielding 208 paired trials (416 responses).The selected cohort is a controlled set of human-confirmed cues, not a prevalence-preserving sample of the corpus.Source: §4.2 · p. 3, Table 1 · p. 4

4.3 Conditional generation

  • PersonaPlex continued A’s channel from synchronized two-speaker audio, with B’s recorded cue remaining audible during a ten-second generation after the cutover.The cutover occurred 80 ms before B’s cue onset; no cue label, task category, or persona text was supplied.Source: §4.3 · p. 4

4.4 Response review and human validation

  • Response reviewers used shared-timeline packets containing preceding transcripts, B’s contribution, and the response; model packets withheld the source continuation.The review classified responses as Continued, Adapted, or Yielded using timed evidence and a rationale.Source: §4.4 · p. 4
  • Among comparable responses, scorer–human agreement was 108/142 (76.1%) with Cohen’s κ = 0.641.Six human not_overlap judgments and two responses deemed unusable by the scorer were excluded from that denominator.Source: §4.4 · p. 4

5 Results

Recorded speakers used all three response types across cue categories, while PersonaPlex adapted less often than people on collaborative cues. The comparison also depends on which cues and responses qualified for the paired analysis.

5.1 Recorded speakers distribute across all three responses

  • Recorded speakers adapted on 68.2% of Collaboration cues; on interruptions, they adapted on 45.9% and yielded on 39.3%.Backchannels most often received continuation (71.6%), though speakers adapted on 23.5% of them.Source: §5.1 · p. 5
  • Interruption cues include unsuccessful floor claims, and the ten-second window may miss later handoffs, understating yielding in both conditions.Restricting the class to completed floor transfers would raise its yield rate by construction.Source: §5.1 · p. 5

5.2 PersonaPlex adapts less often on collaborative cues

  • On 66 active, scorable Collaboration pairs, recorded speakers adapted in 68.2% and PersonaPlex in 34.8%, a difference of 33.3 percentage points.PersonaPlex continued unchanged in 42.4% of cases and yielded in 22.7%.Source: §5.2 · p. 5
  • Twenty of 23 model adaptations occurred where the source also adapted; in trial 0175, both adapted but developed the contribution differently.The four Collaboration examples span response combinations rather than showcasing only success or failure.Source: §5.2 · p. 5, Table 3 · p. 6
  • Backchannel continuation was nearly identical for source and model (71.6% versus 70.4%); on interruptions, PersonaPlex yielded more and adapted less than the source.Interruption yielding was 50.8% for PersonaPlex versus 39.3% for the source; adaptation was 32.8% versus 45.9%.Source: §5.2 · pp. 5–6
  • The comparison rests on the least reliable label cell: Adapted agreement was 68.6%, and Collaboration cue agreement was 14/20.Seven of eleven cue disagreements crossed the Collaboration–Interruption boundary.Source: §5.2 · p. 6

5.3 Coverage

  • The 34.8% model adaptation rate is conditional on active speakers and scorable responses, excluding 86 inactive-at-cue judgments and 46 unusable outputs among 300 reviewed outputs.The coverage table reports all 300 trials; its categories overlap.Source: §5.3 · p. 6, Table 4 · p. 6

6 Discussion and limitations

Duplex Cue makes uptake during an ongoing turn measurable, while the discussion shows that existing overlap metrics miss this distinction. The reported comparison is bounded by annotation dependence, evaluation setup, and narrow sampling.

6 Discussion and limitations

  • The evaluated model also often yields on collaborative cues, while Duplex Cue distinguishes yielding from responding while retaining the floor.The authors suggest evaluating content uptake and floor behavior together; semantic task scoring could further separate acknowledgment from successful incorporation.Source: §6 · pp. 6–7
  • Existing overlap metrics can collapse or mis-score a speaker’s response to a contribution that both answers the listener and continues the prior utterance.In trial 0124, the recorded speaker confirms a proposed stopping time while the listener is still talking, then continues; the discussed benchmarks either miss that distinction or score the human response as an error.Source: §6 · p. 7
  • Cue intent was judged with access to the speaker’s subsequent response, making some of the 68.2% adaptation rate on Collaboration cues definitional.The authors identify blind relabeling without post-cue audio as a check; agreement was reported with one human reviewer.Source: §6 · p. 7
  • The comparison uses original human context and model continuations conditioned on bounded, voice-converted history, so history length and voice conversion are not isolated.The authors propose varying history length and voice conversion, adding no-cue controls, and testing interactive partners and longer response windows.Source: §6 · p. 7
  • Results cover one English checkpoint and one generation per trial, with repeated participants, and describe only human-confirmed pairs meeting activity and scorability criteria.The authors call for broader models, languages, conversations, and accounting for generation failures to connect these conditional results to overall system performance and natural cue frequencies.Source: §6 · p. 7

7 Conclusion

Duplex Cue makes in-turn adaptation a target by separating listener intent from the speaker’s observed response. On collaborative pairs, recorded speakers adapted more often than PersonaPlex; the authors argue that evaluation should measure uptake alongside continuing or yielding.

7 Conclusion

  • 68.2% of recorded speakers adapted on 66 active, scorable collaborative pairs, compared with 34.8% for PersonaPlex.PersonaPlex’s remaining responses divided between unchanged continuation and yielding, patterns an aggregate stop/continue measure would obscure.Source: §7 · pp. 7–8
  • Duplex Cue separates listener intent from observed response to distinguish unchanged continuation, adaptation within the turn, and yielding.It treats how a listener’s contribution shapes an utterance already underway as a question alongside whether the speaker continues or yields.Source: §7 · pp. 7–8
  • The single-model study with a recorded partner demonstrates the diagnostic value of measuring uptake and floor behavior together.The framework is presented for examining that distinction across models and interaction settings.Source: §7 · p. 8

Ethics statement

The paper reports retrospective use of privately licensed recordings and describes operational rules for selecting cues and judging responses. Those rules specify timed evidence, intent distinctions, and criteria for labeling or excluding cases.

Ethics statement

  • The retrospective evaluation used privately licensed recordings with audited acceptance terms but had no formal ethics-board review or exemption determination.Audio and participant materials remain access-controlled, and voice conversion is not claimed to guarantee anonymity.Source: Ethics statement · p. 8

A.1 Cue eligibility and intent

  • A cue qualified only when Speaker A held the floor, remained mid-thought at B’s onset, and was mid-sentence at the cutover 80 ms earlier.The less-than-0.5-second gap tolerance allowed brief pauses; equal onsets, natural transitions, and padding artifacts did not qualify.Source: §A.1 · p. 11
  • Eligible contributions were labeled Backchannel, Collaboration, or Interruption by local intent, with ambiguous cases rejected and timed evidence retained.Collaborations shape A’s utterance without claiming the floor; Interruptions attempt to take or retain it. Of 10,478 proposals, 2,591 events were accepted.Source: §A.1 · p. 11, Table 5 · p. 11

A.2 Response review

  • Responses were labeled Continued, Adapted, or Yielded using cited post-cue words and evidence of whether A incorporated B’s contribution or relinquished the floor.Silent and unusable responses were distinguished; empty ASR or VAD output alone did not establish silence, and floor ownership relied on conversational evidence rather than VAD alone.Source: §A.2 · pp. 11–12

B.1 Cue validation

The validation checks report cue-label agreement on a balanced audio sample and scorer–human response agreement on manually reviewed responses. The two checks use different samples and report distinct agreement figures.

B.1 Cue validation

  • 49/60 cue labels agreed (81.7%) in a balanced sample of 20 cues per model-assigned category.Agreement was 19/20 Backchannels, 14/20 Collaborations, and 16/20 Interruptions; seven of eleven disagreements crossed the Collaboration–Interruption boundary. The sample came from the first ten conversations and was separate from the evaluation-selection queue.Source: §B.1 · p. 12

B.2 Response validation

  • 108/142 comparable response ratings agreed (76.1%; 95% Wilson CI: 68.4–82.3%), with Cohen’s κ = 0.641.The 150-item review used targeted adjudication of prior disagreements, not a fresh independent review; six human not_overlap and two scorer-unusable cases were excluded from the comparable-response denominator.Source: §B.2 · p. 12, Table 6 · p. 12

C Detailed paired results

Mapping agreement is a secondary descriptive statistic that compares each response with the action mapped from its cue type; the overall rates combine different cue behaviors.

D Conversion and reproducibility

The evaluation converted speaker A’s tracks for PersonaPlex inputs, preserved the recorded listener channel, and saved artifacts and settings relevant to reproduction.

Loading 2609.13117v1…