PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech

Pengcheng Wang, Sheng Li, Jiyi Li, Takahiro Shinozaki

arXiv:2608.16053v1cs.CLeess.AS

TL;DR

Existing dialogue synthesis prescribes conversational timing through handcrafted rules rather than letting it emerge from interaction. DuplexGen decouples content, timing, and acoustics through script-constrained duplex performance, producing dynamics closer to real dialogue than stitching-based synthesis.

  • Problem

    Existing dialogue synthesis pipelines prescribe overlap, interruptions, and backchannels with handcrafted timing rules instead of interaction-driven generation.

  • Method

    DuplexGen separates scripted content, real-time duplex timing, and acoustic re-rendering to preserve lexical fidelity while enabling emergent conversational dynamics.

  • Results

    FTO Wasserstein distance falls from 0.695 to 0.366, while the framework produces dynamics closer to real dialogue than evaluated stitching baselines.

  • Takeaways & Limitations

    The framework yields patient–clinician speech data with aligned transcripts, speaker activity, and interaction events for conversational ASR and speaker diarization research.

  • Takeaways & Limitations

    Interaction remains only partially emergent because overlap frequency and backchannel density require explicit controls, and overlap excludes deep mid-utterance interruptions.

Abstract

from arXiv · show

Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dialogue synthesis pipelines typically generate dialogue content first and then insert interruptions, overlap, and backchannels using handcrafted markers or timing rules, making conversational timing prescribed rather than interaction-driven. We present DuplexGen, a dialogue synthesis framework that explicitly decouples content, timing, and acoustics. An LLM first generates the dialogue script, and then two full-duplex conversational models perform the script while listening to each other in real time. This allows conversational timing to emerge naturally while preserving the scripted content. Finally, a high-fidelity text-to-speech model re-renders the interaction without altering its timing. As a demonstration of the proposed framework, we construct a patient--clinician conversational speech corpus with construction-time annotations, including word timestamps, speaker activity, overlap regions, and interaction events. Experimental results show that the proposed framework produces conversational dynamics closer to real dialogue than conventional stitching-based synthesis.

I. INTRODUCTION · II. METHOD

DuplexGen decouples dialogue content, conversational timing, and acoustic realization so scripts remain controllable while interaction dynamics emerge through real-time full-duplex performance. The framework is demonstrated for patient–clinician speech synthesis and produces interaction-aware corpora with construction-time annotations.

  • I. INTRODUCTION: Existing synthesis pipelines often generate dialogue first and impose timing through handcrafted markers or rules, making conversational timing prescribed rather than interaction-driven.Synthetic conversational speech supports conversational speech recognition and speaker diarization, while LLMs and neural TTS enable large-scale automatic corpus generation.
  • I. INTRODUCTION: Full-duplex conversational models generate interaction through continuous listening while speaking but cannot reliably follow predefined scripts.The combination of controllable content and emergent interaction remains relatively underexplored.
  • I. INTRODUCTION: The framework decomposes synthesis into three independent processes: semantic generation, interaction generation, and acoustic realization.This decomposition explicitly separates content, timing, and acoustics.
  • I. INTRODUCTION: An LLM determines dialogue content and speaker order, two full-duplex models perform the script while listening in real time, and conditional TTS re-renders the interaction.The duplex stage allows conversational dynamics to emerge naturally without altering the scripted dialogue.
  • I. INTRODUCTION: The framework is demonstrated as MedDialSpeech, a patient–clinician corpus with construction-time annotations for transcripts, speaker activity, and interaction events.The resulting corpus is directly applicable to conversational ASR and speaker diarization.
  • I. INTRODUCTION: Compared with evaluated stitching baselines, the proposed approach produces conversational dynamics closer to real clinical conversations while maintaining exact script fidelity and high-quality speech synthesis.The interaction-aware corpus is reported to be closer to the reference distribution than the evaluated stitching baselines.
  • I. INTRODUCTION: Script-constrained duplex decoding enforces exact lexical constraints while allowing overlap, interruptions, and backchannels to emerge through real-time interaction.This strategy combines script fidelity with interaction-driven timing.
  • II. METHOD: The pipeline uses separate stages in which the script fixes content, duplex performance fixes timing, and re-rendering fixes acoustics.These stages never share a decision, and high-fidelity TTS preserves the generated conversational dynamics.

A. Overview and Notation · B. Script-Constrained Duplex Generation

Duplex generation separates dialogue content, conversational timing, and acoustics into distinct stages. Script-constrained duplex models preserve the prescribed transcript while interaction determines when tokens are emitted or speakers remain silent.

  • A. Overview and Notation: The dialogue script is an ordered sequence of turns assigned to speakers i ∈ {A, B}, with each speaker’s turns concatenated into a target token sequence.This notation supports tracking each speaker’s prescribed lexical content during duplex generation.
  • A. Overview and Notation: Each speaker maintains a cursor p_i for the next token, continuously listens to the other speaker’s audio, and independently chooses whether to emit or wait.The script fixes the transcript, while emit-or-wait decisions determine conversational timing.
  • B. Script-Constrained Duplex Generation: The full-duplex model predicts a text token before its audio frame, and the system restricts lexical output to the next script token when the speaker is floor-eligible.This follows the inner-monologue architecture of Moshi [3] and removes lexical uncertainty from duplex generation.
  • B. Script-Constrained Duplex Generation: The allowed candidates include the next script token and a fixed whitelist B of listener backchannels, such as mm-hmm, yeah, and right.Backchannel production is subject to the refractory constraint introduced in Section II-C.
  • B. Script-Constrained Duplex Generation: When the next script token is selected, its audio is emitted and the cursor advances; when PAD is selected, the speaker remains silent for that frame.The cursor update is p_i ← p_i + 1.
  • B. Script-Constrained Duplex Generation: Because each emitted word is uniquely determined by the script, duplex models decide only whether to emit the next token or wait, leaving timing to interaction.Dialogue content is fixed by construction, while conversational timing is determined by interaction between the two duplex instances.

C. Emergent Timing

DuplexGen generates conversational timing through two cross-listening full-duplex models that independently choose whether to emit scripted tokens or remain silent, without a global turn schedule. Control parameters constrain interaction-event frequency, while the models determine transition onsets, overlap duration, and backchannel placement online from each other’s audio.

  • Emergent timing: Two cross-listening full-duplex models independently decide at every frame whether to emit the next script token or remain silent, with no global schedule for turn boundaries.Conversational timing is therefore produced through real-time interaction while preserving the scripted token sequence.
  • Emergent timing: The handoff window Wh defines when a listener may take the floor around the projected end of the current turn.The projected end, τend, is estimated from the current speaker’s remaining script tokens.
  • Emergent timing: max_padding (Pmax) caps consecutive PAD frames before a floor-eligible speaker starts its queued turn, preventing unrealistically long silent gaps.This parameter limits silence duration rather than prescribing a complete turn schedule.
  • Emergent timing: bc_refractory (Rbc) sets the minimum interval between listener backchannels, preventing excessive acknowledgements.It regulates backchannel frequency while leaving exact attachment timing to the duplex interaction.
  • Emergent timing: The duplex models determine floor-transition onset, overlap duration, and backchannel attachment online in response to each other’s audio.The three control parameters regulate only interaction-event frequency, so conversational timing emerges from interaction rather than an explicit schedule.

D. Fidelity Re-rendering

DuplexGen re-renders symbolic interaction scores as high-fidelity speech while transferring conversational timing through relative positions rather than absolute timestamps. This preserves silent transitions, overlap structure, and backchannel placement despite speaking-rate differences.

  • Symbolic-to-speech rendering: The duplex models produce symbolic interaction scores recording active speakers, turn boundaries, overlap regions, and backchannel positions before CosyVoice renders the final waveform.The symbolic score represents the interaction structure, while CosyVoice supplies high-fidelity speech.
  • Relative timing transfer: Timing is transferred using relative positions because the duplex model and TTS system generally speak at different rates.Absolute timestamps therefore cannot be copied directly between the two systems.
  • Relative timing transfer: For overlaps, the next turn begins at eDu, where e is the elapsed fraction of the current turn and Du is its rendered duration; backchannels map identically.This preserves overlap structure regardless of speaking-rate differences between the duplex model and renderer.
  • Rendering implementation: A sliding prompt maintains speaker consistency, interrupted speech is truncated with a short crossfade, and backchannels are independently mixed at mapped positions.These implementation choices support long conversations, acoustically realistic interruptions, and concurrent speech.
  • Automatic annotations: Annotations are generated automatically from the symbolic score and CosyVoice alignment, including word timestamps, RTTM speaker activity, overlap, interruption, gap, and backchannel labels.The score completely specifies the generated conversation, avoiding post-processing annotation.

III. RELATED WORK · A. Scripted conversational speech synthesis · B. Full-duplex conversational modeling

Related work follows two complementary directions: scripted synthesis emphasizes controllable dialogue generation, while full-duplex models emphasize interaction modeling. DuplexGen combines these directions by separating content generation, interaction generation, and acoustic rendering into independent stages.

  • A. Scripted conversational speech synthesis: Recent LLM and neural TTS advances enable large-scale conversational speech synthesis by generating dialogue scripts and synthesizing utterances with neural speech generators.This describes the prevailing scripted synthesis pipeline.
  • A. Scripted conversational speech synthesis: Behavior-SD uses behavior prompts to control conversational events, while PersonaPlex additionally models speaker personas and dialogue roles.These systems extend scripted synthesis with controllable behavioral and speaker-related factors.
  • A. Scripted conversational speech synthesis: Other dialogue synthesis systems introduce overlap, interruption, or backchannel events through dialogue markers or related controls.These approaches explicitly add conversational events to synthesized dialogue.
  • B. Full-duplex conversational modeling: Full-duplex conversational models directly represent bidirectional spoken interaction, beginning with dGSLM’s dualstream spoken dialogue generation.dGSLM established an early full-duplex modeling direction.
  • B. Full-duplex conversational modeling: Moshi, SyncLLM, and SALMONN-Omni advance real-time duplex conversation through inner-monologue or synchronous modeling and continuous listening while speaking.These systems naturally produce interruptions and other conversational dynamics.
  • B. Full-duplex conversational modeling: The two directions are complementary: scripted synthesis emphasizes controllable dialogue generation, whereas full-duplex modeling emphasizes interaction modeling.This distinction motivates combining their respective strengths.
  • B. Full-duplex conversational modeling: DuplexGen combines these strengths by separating content generation, interaction generation, and acoustic rendering into three independent stages.The framework explicitly decouples the three components rather than treating conversational speech generation as a single process.

IV. EXPERIMENTS · A. Experiment 1: Distributional Realism · B. Experiment 2: Fidelity of Decoupled Generation

Experiments show that DuplexGen produces dialogue timing closer to real conversational statistics while preserving scripted content, timing annotations, and speaker identity through acoustic re-rendering. Robustness checks support the realism result within natural overlap ranges, while excessive overlap reduces realism.

  • A. Experiment 1: Distributional Realism: The evaluation compares DuplexGen with two stitching baselines using FTO Wasserstein distance, KS statistic, overlapped-transition ratio, and long-tail gap ratio.The reference distribution contains 5,611 floor transfers from PriMock57.
  • A. Experiment 1: Distributional Realism: 47% relative improvement reduces FTO Wasserstein distance from 0.695 to 0.366, while the KS statistic decreases from 0.432 to 0.197.DuplexGen also reproduces overlapped transitions and the long-tail silence distribution, unlike the stitching-based synthesis.
  • A. Experiment 1: Distributional Realism: Robustness checks preserve the ranking across utterance-end constants of {0.3, 0.4, 0.5} s, with Wfto = 0.435/0.357/0.292 outperforming both baselines.Bootstrap evaluation across six random seeds gives a 95% confidence interval of [0.280, 0.545].
  • A. Experiment 1: Distributional Realism: Natural timing is reproduced within the overlap range observed in real dialogue, whereas excessively increasing overlap makes conversational timing less realistic.The overlap-control result bounds the method’s claim to natural interaction rather than arbitrary overlap patterns.
  • B. Fidelity of Decoupled Generation: The fidelity experiment tests whether content generation, conversational timing, and acoustic rendering remain decoupled through content fidelity, timing preservation, and speaker consistency.Dialogue scripts are generated with DeepSeek-V4, while the reference voice is used only for acoustic consistency and not timing statistics.
  • B. Fidelity of Decoupled Generation: 0% forced WER results from transcripts remaining identical to the fixed input script, while symbolic overlap events transfer exactly to rendered speech.Lexical content is fixed before duplex interaction, preventing content changes during timing generation.
  • B. Fidelity of Decoupled Generation: Acoustic re-rendering introduces at most 1.7% WER while preserving speaker identity with a CAM++ similarity of 0.82.Delivered overlap closely matches annotations; remaining discrepancies arise from separately annotated low-level (−6,dB) backchannel overlays.

C. Experiment 3: Emergence and Controllability

DuplexGen produces interaction that adapts to dialogue style while remaining controllable through timing configurations. With fixed scripts, duplex interaction determines conversational behavior, including variable barge-ins, overlap, backchannels, and silence.

  • Emergence and controllability: Under identical scripts and knob settings, different random seeds produce zero to three barge-ins per dialogue.This shows that overlap placement varies across stochastic interaction realizations despite fixed content and controls.
  • Emergence and controllability: 7.2% overlap in medical conversations is lower than 10.5% in casual conversations and close to 6.3% in real clinical dialogue.The generated interaction therefore adapts to dialogue register under identical timing parameters.
  • Emergence and controllability: 95.5% of backchannels occur at prosodically appropriate positions, exceeding the random baseline of 71.7%.Increasing the handoff window raises overlap frequency, while increasing max_padding produces more long silent gaps.

D. Experiment 4: Downstream Stress Test

The downstream stress test shows that DuplexGen provides a controllable benchmark whose ASR difficulty increases with conversational overlap while clean-region performance remains stable. A matched-overlap ablation attributes downstream difficulty primarily to acoustic rendering rather than timing, and the trend generalizes across speakers and domains.

  • Benchmark construction: The benchmark contains 180 segments spanning 175 minutes and three interaction-difficulty levels, with the natural tier closely matching real clinical dialogue overlap statistics.The tiers were constructed by varying timing parameters and exhibit progressively denser interaction patterns.
  • ASR stress test: Overlap-region WER is consistently 3–7× higher than aggregate WER and increases monotonically with interaction difficulty for Whisper large-v2 and wav2vec2.Clean-region performance remains nearly unchanged, indicating that timing controls regulate ASR difficulty through conversational overlap rather than overall speech quality.
  • Ablation: In a matched-overlap 2 × 2 ablation, timing effects are small while rendering dominates downstream difficulty.The ablation values are not directly comparable to by-tier results because overlap rate and renderer are held fixed and backchannel overlays are excluded.
  • Ablation: With rendering fixed, emergent and stitched timing yield overlapping confidence intervals: 18.8 vs. 23.2 and 44.1 vs. 43.3 at n=30.With timing fixed, replacing truncate-and-fade with equal-volume overlay roughly doubles the error, with disjoint confidence intervals.
  • Robustness: The overlap-region degradation persists across 12 LibriTTS speaker pairs and out-of-domain casual dialogue, while pyannote-3.1 diarization error remains largely unchanged across difficulty levels.This indicates that the benchmark primarily stresses speech recognition rather than speaker attribution.

V. LIMITATIONS · VI. CONCLUSION · ETHICS AND DATA STATEMENT

DuplexGen decouples scripted content, emergent conversational timing, and high-fidelity acoustics, but its interaction dynamics, adaptability, and clinical realism remain limited. The authors also restrict the synthetic corpus to labeled research use without clinical validity or medical-decision-making claims.

  • V. LIMITATIONS: Conversational interaction is only partially emergent because overlap frequency and backchannel density remain controlled by explicit parameters.Overlap is confined to turn-transition regions rather than deep mid-utterance interruptions.
  • V. LIMITATIONS: Fixed dialogue content prevents speakers from revising or abandoning planned utterances after interruptions, limiting repair, self-correction, and content negotiation.This limitation follows from strict decoupling between content and timing.
  • V. LIMITATIONS: The benchmark is a simplified approximation of real clinical conversations because it uses one reference corpus and synthetic rendering.Downstream ASR difficulty is influenced more by acoustic rendering than emergent timing itself.
  • V. LIMITATIONS: Further improvements in overlap rendering and multi-speaker interaction are needed to better match real-world conversations.The benchmark nevertheless provides controllable conversational dynamics.
  • VI. CONCLUSION: DuplexGen is a zero-training pipeline that decouples content, timing, and acoustics, enabling scripted content, emergent timing, and high-fidelity sound.The framework reproduces overlapped transitions and long gaps absent from the evaluated stitching baselines.
  • VI. CONCLUSION: Emergent turn-taking is distributionally closer to real clinical dialogue than stitching and supports controllable densities through three physical knobs.The passage describes a clean separation between control and emergence, although its final wording is truncated.
  • ETHICS AND DATA STATEMENT: The reference corpus PriMock57 is used under its license, and all released audio is synthetic and labeled as such.Use is restricted to research on speech recognition and dialogue robustness.
  • ETHICS AND DATA STATEMENT: The synthetic data carries no clinical validity claim and must not be used for medical decision-making.This restriction accompanies the stated research-only use.
Loading 2608.16053v1…