PaperlayerView exact revision on arXiv

Source-linked AI summary

A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversations

Matthew Sun, Vinay Kothapally, Meng Yu, Chao Huang, Hao Zhang, Yixuan Zhang, Steve Yves

arXiv:2609.28806v1eess.AScs.AI

TL;DR

Full-duplex dialogue requires distinguishing floor transfers from pauses, acknowledgments, and other overlapping speech, while existing corpora offer limited intent labels and control. The paper synthesizes intent-labeled two-channel speech from relational events; ablations show complementary diversity effects, and evaluations support learnable, transferable turn-management supervision.

  • Problem

    Existing conversational audio has natural timing but limited intent labels and controlled coverage, although overlapping events with similar activity patterns can require different system actions.

    Source: §I · p. 1

  • Method

    The pipeline authors relational events, renders and aligns them on a shared two-channel clock, derives labels from intent, and uses diversity hubs and probability verbalization.

    Source: §I · pp. 1–2

  • Results

    Ablations show complementary diversity effects, while internal VAD and Moshi evaluations support learnable labels and transfer to full-duplex turn management.

    Source: §VI · p. 11

  • Takeaways & Limitations

    Controlled, intent-labeled synthesis is a practical complement to uncontrolled conversational recordings for training full-duplex dialogue systems.

    Source: §VI · p. 11

    Free-generation boundary scores remain lower because autoregressive speech can diverge from the single reference timeline.

    Source: §VI · p. 11

Abstract

from arXiv · show

Full-duplex dialogue systems, which listen while speaking, must distinguish a completed turn from a pause within a turn and an interruption that requests a turn from a brief acknowledgment or speech addressed to a third party. Yet existing conversational corpora provide limited control over these events and limited labels for their intent. We present a pipeline for synthesizing intent-labeled, two-channel conversational speech from relational event lists. An LLM authors each event's speaker, text, conversational act, and attachment to an earlier event without predicting absolute timestamps. Events are synthesized independently, aligned with their source text, and placed on a shared clock, so turn-taking landmarks are measured from the rendered signal while silence durations are specified or sampled from turn-taking distributions. The pipeline covers 42 phenomena across eight families in English and Mandarin, derives frame-level system actions from authored intent, and promotes diversity using small, diverse sets of prior examples and batch prompts that request alternatives with self-reported probabilities. Ablations show gains in each targeted diversity dimension. On a four-action label space for taking, holding, releasing, and not holding the conversational floor, a semantic voice-activity detector using only current and past audio reaches start-speaking and start-listening F1 scores of 0.819 and 0.802. When generating its own responses, the full-duplex speech model Moshi takes 0.85 of the reference turns after fine-tuning on the generated corpus, compared with 0.44 before fine-tuning. Its frame-level precision for predicting system-floor occupancy rises from 0.46 to 0.88. With reference context at each step, its frame-level floor F1 rises from 0.893 to 0.962. These results show that controlled synthesis can provide learnable and transferable supervision for full-duplex turn management.

I. INTRODUCTION

Full-duplex systems must distinguish floor transfers from pauses, acknowledgments, and side-talk, even when these events sound similar. The paper proposes relational event lists to control two-channel speech and preserve intent in its labels.

I. INTRODUCTION

  • Full-duplex systems must infer whether silence or overlap signals continued floor holding, a turn transfer, acknowledgment, or side-talk, because these cases require different actions.Source: §I · p. 1
  • Existing speech corpora offer natural timing but limited intent labels and construction-time control; text synthesis controls content, not acoustic timelines.Source: §I · p. 1
  • An LLM authors relational events for English and Mandarin, and rendered speech is aligned and placed on a shared clock without requiring absolute timestamps.Source: §I · p. 1
  • The pipeline derives frame-level actions from authored intent, samples overlapping timing distributions, and covers 42 phenomena across eight families in English and Mandarin.Source: §I · p. 2
  • The contributions also include tools for improving lexical diversity and conversational chronemics, alongside intent-derived labels that distinguish barge-ins from backchannels.Source: §I · p. 2

B. Relational Event Representation

  • The event representation records speaker, channel, text, act, and relational timing cue; its acts distinguish turns, pauses, backchannels, barge-ins, and asides.Source: B. Relational Event Representation · p. 2

C. From Events to Training Labels

The approach specifies scenarios through relational events, then realizes them as aligned two-channel audio and frame-level labels. Timing landmarks come from rendered speech, while gaps and overlaps are represented on a shared clock.

III. APPROACH

  • The pipeline separates authoring event content and relations from acoustic realization, avoiding absolute timestamp prediction by either the authoring model or synthesizer.Source: §III · p. 3

A. Scenario and Conversation Specification

B. Render, Align, and Assemble

C. Natural Timing and Intent-Derived Labels

Timing and labels are derived from event intent rather than duration or activity alone. Two complementary few-shot hubs track floor mechanics and conversation meaning to promote structural and semantic diversity.

C. Natural Timing and Intent-Derived Labels

D. Few-Shot Diversity Hubs

  • Two few-shot hubs use diverse prior conversations as negative memory: a structural hub tracks floor mechanics, while a semantic hub tracks topic and meaning.Source: D. Few-Shot Diversity Hubs · p. 5

1) Structural Hub: Floor Fingerprints:

  • The structural hub combines a 27-dimensional floor-mechanics feature vector with ordered act sequences, weighting sequence distance at 0.75 and normalized feature distance at 0.25.Source: 1) Structural Hub: Floor Fingerprints · p. 5

2) Semantic Hub: Meaning Embeddings:

  • The semantic hub encodes concatenated event text as normalized 1024-dimensional BGE-M3 embeddings, keeping paraphrases close while separating conversations about different content.Source: 2) Semantic Hub: Meaning Embeddings · p. 6

3) Online Coreset Maintenance:

The section combines online diversity steering with within-call batch alternatives, then evaluates how these mechanisms change generated conversations. Ablations show complementary gains in floor-pattern coverage, semantic spread, and content diversity, with mixed effects on timing.

3) Online Coreset Maintenance:

  • The streaming max-min cover retains candidates far from their nearest retained neighbor, with capacity bounded between 5 and 32 and update cost O(M^2).Existing records are replayed through the same update at startup; valid conversations enter the corpus whether or not they join a hub, avoiding rejection loops and saturation.Source: 3) Online Coreset Maintenance · p. 6

E. Probability-Verbalized Batch Authoring

  • Probability-verbalized batch authoring requests five conversations with self-reported probabilities and asks for distinct interpretations, wording, and floor-management patterns.The probabilities prompt lower-probability alternatives but are not used as corpus weights or acceptance scores; accepted conversations pass the usual quality checks and update the hubs one at a time.Source: E. Probability-Verbalized Batch… · p. 6

F. Acoustic Variation and Reusable Outputs

  • At render time, voice references and user-channel noise can vary while preserving event structure and labels.A clean parallel copy is retained, and the system channel is unchanged by the noise mixing.Source: F. Acoustic Variation and Reusable… · pp. 6–7

IV. EVALUATION

  • The evaluation tests generation ablations, whether intended floor-action labels are recoverable from audio, and whether fine-tuning improves Moshi.Ablation conditions hold scenario, sample count, and remaining settings fixed while changing one intervention; each arm contains 45 English conversations.Source: §IV · p. 7, A. Generation-Pipeline Ablations · p. 7

A. Generation-Pipeline Ablations

  • The structural hub restores barge-ins and truncations on AF-02; feature-space Vendi rises 64.2%, nearest-neighbor distance 95.0%, and act-sequence EAD-2 185.7%.The no-hub baseline collapses onto turn-plus-backchannel patterns; the reported gains reflect broader floor-mechanics coverage, not only longer conversations or more events.Source: A. Generation-Pipeline Ablations · p. 7, Table I · p. 7
  • On AF-02, the semantic hub raises Semantic Vendi 3.6% beyond the topic-seeded baseline, while lexical evidence is mixed.Unique types and length-adjusted EAD metrics increase, but pairwise ROUGE-L also increases slightly.Source: A. Generation-Pipeline Ablations · p. 7, Table II · p. 7
  • On TT-01, probability verbalization raises lexical EAD-2 37.6%, lowers pairwise lexical ROUGE-L 37.8%, and raises Semantic Vendi 68.7%, while floor-mechanics effects are mixed.Nearest-neighbor distance decreases and gap spread narrows, despite increased structural Vendi.Source: A. Generation-Pipeline Ablations · p. 8, Table III · p. 8
  • The structural hub prevents floor-pattern collapse, the semantic hub adds meaning spread beyond topic seeding, and probability verbalization diversifies content while requiring timing monitoring.On AF-02, feature-space Vendi rises 64.2%, nearest-neighbor distance 95.0%, and act-sequence EAD-2 185.7% with the structural hub; the semantic hub raises Semantic Vendi 3.6%. On TT-01, probability verbalization raises lexical EAD-2 37.6% and Semantic Vendi 68.7%, while timing effects are mixed.Source: A. Generation-Pipeline Ablations · pp. 7–8

B. Internal Semantic-VAD Benchmark

The semantic-VAD benchmark tests whether authored floor-action labels are learnable from causal audio, while the Moshi benchmark tests transfer to an independently designed model. Moshi improves free-generation turn coverage and floor precision, and teacher-forced evaluation shows stronger floor-transition prediction.

B. Internal Semantic-VAD Benchmark

  • The causal benchmark uses 19,575 training clips and 2,756 validation clips, with user-channel noise added only in the noisy condition.The train and validation sets share no speakers or conversations; predictions are made every 80 ms.Source: B. Internal Semantic-VAD Benchmark · p. 8
  • Clean four-token accuracy is 0.9932, start-speaking and start-listening F1 are 0.8186 and 0.8019, and their mean start-speaking latency is 6.2 ms.User-channel noise changes accuracy by only -0.0003; start-speaking latency is 5.2 ms in noise.Source: Table IV · p. 8, B. Internal Semantic-VAD Benchmark · p. 8
  • User-channel noise most reduces reply-gap F1 by 0.0971, continue-listening by 0.0548, and silence by 0.0348, while principal floor transitions remain stable.Start-speaking, start-listening, and yield F1 decrease by 0.0021, 0.0087, and 0.0006, respectively.Source: B. Internal Semantic-VAD Benchmark · pp. 8–9
  • In the 13-token space, clean accuracy is 0.9613, speaking and listening exceed 0.98 F1, and rare barge-in has the lowest F1 at 0.4325 on 209 frames.Contested-floor and hold-backchannel each exceed 0.90 F1.Source: B. Internal Semantic-VAD Benchmark · p. 8, Table V · p. 9
  • Merging user- and companion-originated variants raises clean accuracy from 0.9584 to 0.9613 and hold-backchannel F1 from 0.7120 to 0.9061.On noisy audio, accuracy rises from 0.9506 to 0.9525 and hold-backchannel F1 from 0.7005 to 0.8568.Source: B. Internal Semantic-VAD Benchmark · p. 9

C. Moshi Benchmark

  • In free generation, after 2,000 fine-tuning steps, Moshi’s fraction of reference turns taken rose from 0.44 to 0.85, and floor precision rose from 0.46 to 0.88.Source: C. Moshi Benchmark · p. 9
  • With reference context, floor F1 improves from 0.893 to 0.962, while zero-tolerance start-listening F1 rises from 0.60 to 0.93.Free-generation boundary scores remain lower because autoregressive speech can diverge from the single reference timeline.Source: Table VI · p. 9, C. Moshi Benchmark · p. 9

V. RELATED WORK

  • Existing approaches cover subsets of the targeted requirements, while this work combines two-channel acoustics, realistic timing, authored intent, and controlled floor-management coverage.The related-work comparison spans full-duplex dialogue, turn prediction, simulated conversational audio, and LLM dialogue synthesis.Source: §V · p. 9

A. Full-Duplex Dialogue and Turn Prediction

Prior approaches provide pieces of full-duplex turn management, conversational-audio simulation, and dialogue synthesis, but do not combine controlled coverage with authored intent and realistic timing. The paper reports that its generated supervision transfers to Moshi and identifies limits in current scenario coverage.

A. Full-Duplex Dialogue and Turn Prediction

  • Full-duplex and continuous turn-prediction systems model synchronized streams or floor states, but prior systems use uncontrolled conversations or lack authored intent for ambiguous overlaps.VAP predicts future joint voice activity, while this approach preserves different targets for successful interruptions, failed interruptions, and backchannels.Source: A. Full-Duplex Dialogue and Turn… · pp. 9–10

D. Turn-Taking Benchmarks and Remaining Gap

  • Existing benchmarks probe deployed systems, while real two-channel corpora cannot balance behaviors or identify intent behind each overlap; the pipeline targets those requirements across 42 phenomena.Figure 5 compares approaches by support for conversational-speech data requirements.Source: Figure 5 · p. 10, D. Turn-Taking Benchmarks and… · p. 11

B. Simulated Conversational Audio

  • Diarization simulation places recorded segments on a shared timeline, but overlap shares one speaker-activity interpretation and clips need not form coherent dialogue.This pipeline authors event purpose and relations, then uses TTS and forced alignment to place events with distinct system-side labels.Source: B. Simulated Conversational Audio · p. 10

C. Dialogue Synthesis and Timing

  • Text dialogue synthesis omits acoustic duration, overlap, and floor-transfer timing, while conversational TTS datasets remain sequential and lack intent-resolved labels.The pipeline instead authors relational events and resolves timing after synthesis, using conversational timing distributions as fallback priors while retaining authored gaps.Source: C. Dialogue Synthesis and Timing · p. 10

VI. SUMMARY

  • Generation ablations show distinct diversity effects: structural hubs restore interruptions and varied floor-action sequences, while probability verbalization most clearly improves wording and meaning diversity.Semantic hubs add meaning diversity beyond topic seeding, with gains limited by a strong distinct-topic baseline; probability verbalization has mixed effects on floor mechanics and timing.Source: §VI · p. 11
  • The semantic-VAD results indicate that authored labels are learnable from causal audio, but rare classes and distinctions weakly expressed in waveforms remain difficult.Merging user- and companion-originated variants improves performance under both acoustic conditions.Source: §VI · p. 11
  • Moshi provides evidence that the synthetic corpus transfers beyond the internal classifier, improving turn-taking while free-generation boundary scores remain lower because of divergence from the reference timeline.With correct conversational context, release timing also sharpens.Source: §VI · p. 11
  • Current limitations include underdeveloped live-translation and language-teaching prototypes, insufficient held-floor and within-turn-pause coverage, and language coverage limited to English and Mandarin.Generated multilingual samples do not yet meet the released corpus's quality standard.Source: §VI · p. 11

APPENDIX A CONVERSATION-AUTHORING PROMPTS

The appendix presents prompt templates for authoring two-channel event lists through relational cues rather than absolute timestamps. Runtime metadata and optional examples guide scenario variation, timing, and output structure.

APPENDIX A CONVERSATION-AUTHORING PROMPTS

  • Prompt templates insert runtime values from merged metadata, the scenario registry, canonical examples, diversity hubs, topic seeds, and validators.The metadata block includes event-field definitions, timing policy, reaction-time ranges, and multilingual guidance.Source: Appendix A · p. 13

<EVENT FIELDS, TIMING, MS_POLICY, REACTION-TIME

  • Timing fields use a referenced event or word boundary, with gap_before_ms for signed gaps and offset_ms for timing displacement.For interruptions, truncate.until_word names a word in the interrupting utterance used to time the cut.Source: Appendix A · p. 13

APPENDIX A CONVERSATION-AUTHORING PROMPTS

  • The system prompt asks an expert author to write a two-channel script specifying speakers, utterances, and attachments to earlier events through cues.Placement is expressed with cues and gap_before_ms, never absolute timestamps.Source: Appendix A · p. 13
  • User and companion voices occupy the human channel, while the system voice occupies the AI channel; companion is reserved for asides.The prompt identifies the human channel as channel 1 and the AI channel as channel 2.Source: Appendix A · p. 13

<EVENT FIELDS, TIMING, MS_POLICY, REACTION-TIME

  • The output contract requires one JSON object containing a conversation description and an event list with fields such as voice, act, cue, and text.The prompt also instructs authors to tag each event's language and describe the new topic rather than the fixed phenomenon.Source: <EVENT FIELDS, TIMING, MS_POLICY,… · p. 13
  • The prompt requires unique sequential event IDs and cue references to earlier events, with a null reference only for the first event.The displayed requirements also specify that referenced words must occur verbatim in the relevant text.Source: <EVENT FIELDS, TIMING, MS_POLICY,… · p. 13
  • Barge-in prompts require a matching timing bucket and truncation word, while failed barges set interrupts:false and omit truncation.Same-channel turns must not overlap, and spoken conversations target roughly 30–60 seconds.Source: <EVENT FIELDS, TIMING, MS_POLICY,… · p. 13

<LANGUAGE-SPECIFIC AUTHORING GUIDANCE>

The authoring guidance preserves each scenario's fixed conversational phenomenon while prompting variation in the surrounding conversation. Optional examples and topic seeds provide additional diversity cues.

<LANGUAGE-SPECIFIC AUTHORING GUIDANCE>

  • Each scenario re-authoring prompt names a fixed scenario ID, family, and phenomenon description that the new script must preserve.The scenario's phenomenon is fixed rather than selected by the author.Source: <LANGUAGE-SPECIFIC AUTHORING GUIDANCE> · p. 13

<CANONICAL EXAMPLE AS FORMATTED JSON>

  • Authors study a generic example's acts and cues, then create a different topic, situation, wording, entities, chronemics, and turn structure.The new conversation should demonstrate the same phenomenon and name its new topic in conv_desc.Source: <LANGUAGE-SPECIFIC AUTHORING GUIDANCE> · p. 13

<OPTIONAL SEMANTIC-HUB EXAMPLES, WITH AN INSTRUCTION

  • Optional semantic-hub examples instruct authors to vary situation, topic or meaning, vocabulary, and entities; structural fingerprints target act order and overlap words.The prompt also requests variation in related structural features.Source: <OPTIONAL SEMANTIC-HUB EXAMPLES, WITH… · p. 13

<OPTIONAL SOFT TOPIC SEED AND ITS VOCABULARY

  • An optional soft topic seed supplies additional vocabulary guidance, while prior conversations are to differ genuinely in situation, meaning, vocabulary, and entities.The prompt rejects paraphrases or the same subject expressed in other words.Source: <OPTIONAL SOFT TOPIC SEED AND ITS… · p. 13

<SEMANTIC-CORESET EXAMPLES AS COMPACT JSON>

The examples prompt an LLM to create distinct conversations around a target phenomenon while separating content authoring from turn-taking and timing. Additional prompt blocks encourage alternative outputs and varied floor mechanics.

<SEMANTIC-CORESET EXAMPLES AS COMPACT JSON>

  • Structural-hub prompts present previously produced floor mechanics as ordered act sequences and ask for a noticeably different sample.Turn-taking and timing are handled separately from the content focus.Source: <SEMANTIC-CORESET EXAMPLES AS COMPACT… · pp. 13–14
  • Authors are told to vary act sequences, barge-in openers, backchannel tokens, reaction offsets, pauses or overlaps, turn counts, and overlap extent.The prompt explicitly forbids reusing the same timing numbers or interruption and backchannel words.Source: <SEMANTIC-CORESET EXAMPLES AS COMPACT… · p. 14
  • A probability-verbalized batch variant requests five distinct responses and a self-estimated probability for each response relative to the full distribution.Each response includes a new conversation description and full event list.Source: <SEMANTIC-CORESET EXAMPLES AS COMPACT… · p. 14
  • The content prompt asks for five distinct conversations that vary in situation, wording, entities, and chronemics while preserving the target phenomenon and topic.It also asks for new conversation descriptions and full event lists.Source: <SEMANTIC-CORESET EXAMPLES AS COMPACT… · p. 14

A. Scenario-Judge Prompts

The scenario-judge prompt reviews authored two-channel conversations for semantic coherence and whether event anchors fit their intended conversational roles. It also restricts its remit by excluding checks handled by deterministic tools and by narrowing formatting review to language tags.

A. Scenario-Judge Prompts

  • The prompt documentation notes that the source user template says two axes while the system template lists three: semantics, anchoring choice, and formatting.It also defines an interpret chunk as spoken translation and its ear–voice span as the delay from source to translated speech.Source: A. Scenario-Judge Prompts · p. 14
  • The prompt excludes structure, reference-graph legality, anchor-word presence, and timing legality because separate deterministic tooling validates them.For barge-ins, the judge may assess whether the reaction offset value fits the intended interruption timing, but not its mere presence.Source: A. Scenario-Judge Prompts · p. 14
  • The judge checks whether turns are coherent, on-topic responses and whether event attachments and word anchors fit the intended reaction or translation.It evaluates anchoring choice even when the reference graph and anchor-word presence have already been validated.Source: A. Scenario-Judge Prompts · p. 14
  • Formatting review is limited to language tags because unsupported tags can make the English- and Chinese-only TTS rendering fail.The judge is instructed to flag genuine problems conservatively and otherwise return an empty error list.Source: A. Scenario-Judge Prompts · p. 14

THEIR POINTS AND OCCURRENCE INDICES>

The prompt directs the reviewer to inspect the specified axes and return the JSON verdict required by the system message.

THEIR POINTS AND OCCURRENCE INDICES>

APPENDIX B LABELING EXAMPLES

The labeling examples show the same two-channel waveform at three levels of detail, from four core actions to finer distinctions in system and human-channel states.

APPENDIX B LABELING EXAMPLES

  • The core-4 view retains only the four basic speaking and listening actions.The panels display the same human and system waveforms while changing label granularity.Source: Appendix B · p. 16, Figure 6 · p. 16
  • The granular view adds distinctions such as backchannels, interruptions, and types of silence.For cooperative overlap with backchannels, finer views distinguish additional floor states that collapse into the four core actions.Source: Appendix B · p. 16, Figure 7 · p. 16
  • Granular-all further distinguishes contested-floor speech, system pauses while holding the floor, and whether human-channel speech comes from the user or a companion.It is not the 13-token all_merged benchmark space, which merges user and companion variants of yield and hold-backchannel.Source: Appendix B · p. 16

Related papers

Loading 2609.28806v1…