Source-linked AI summary
ConversationalVoice: Full-Duplex Speech Data from Real Conversations through Source-Faithful Reconstruction and Conversation-Grounded Expansion
Richard Yucheng He, Baodong Cao, Chen Xu, Yihang Liu, Tairan Chen
TL;DR
Full-duplex speech models need conversational timing and speaker interaction preserved, but real recordings are noisy and entangle speakers. ConversationalVoice converts real two-speaker excerpts into separated, reconstructed, and grounded-expanded training artifacts, with strong automatic quality results while leaving downstream model gains unevaluated.
Problem
Full-duplex speech training data must preserve turn-taking, overlap, interruption, backchannels, and speaker streams, yet real conversations are noisy monaural recordings with entangled speakers.
Method
ConversationalVoice screens and separates real two-speaker excerpts, then reconstructs the source exchange and expands it into grounded new dialogue with aligned timing and delivery metadata.
Results
All stages retain positive speaker-discrimination margins and strong predicted acoustic quality; expansion scores 4.94/5 for contextual coherence and 4.80/5 for dialogue naturalness.
Takeaways & Limitations
The pipeline provides three linked training-data views: faithful observed speech, cleaner source-faithful speech, and new interactions constrained by recovered speakers, context, and interaction patterns.
Takeaways & Limitations
The study evaluates data properties only; downstream full-duplex model performance is not measured.
Abstract
from arXiv · showhide
Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordings. We present Conversational Voice, a pipeline that converts real two-speaker excerpts into three complementary training-data artifacts. (1) Separation recovers speaker-specific tracks with stable speaker assignments, a canonical transcript, and naturally observed interaction timing. (2) Reconstruction generates speech in matched voices from a fixed source transcript, reconstructs the source turn order, pauses, and overlaps, and adds word-level alignment and delivery instructions. (3) Expansion generates new dialogue constrained by the source context, speakers, and observed interaction pattern. Automatic speaker-verification metrics remain strong across stages, with same-speaker similarity of 0.983-0.991 and positive discrimination margins of 0.199-0.209. Predicted speech quality (NISQA MOS) is 3.56 for separation, 4.41 for reconstruction, and 4.61 for expansion. A Gemini-based automatic evaluator assigns expansion mean scores of 4.94/5 for contextual coherence and 4.80/5 for dialogue naturalness. Expansion and reconstruction exhibit broadly similar interaction profiles; expansion's turn, overlap-event, backchannel, and interruption rates are 4.6%, 8.0%, 13.2%, and 16.0% lower, respectively. We evaluate data properties only; downstream gains in full-duplex model training remain for future work.
1 Introduction
ConversationalVoice addresses the mismatch between full-duplex training needs and noisy monaural recordings by transforming real two-speaker excerpts into validated, complementary data artifacts. Separation recovers stable speaker-specific evidence, while reconstruction and expansion generate aligned speech that preserves or extends interaction structure.
- Full-duplex training data must preserve words, speaker identity, turn transitions, silence, overlap, feedback, and nonverbal behavior.
- ConversationalVoice converts real-world two-speaker audio into three complementary full-duplex training-data forms.
- Separation combines diarization, dialogue separation, stable speaker assignment, quality checks, and ASR to recover speaker tracks with a canonical transcript.
- Reconstruction regenerates the source exchange with fixed words and speaker identities while rebuilding word timing, turns, pauses, and overlaps around generated durations.
- Expansion generates new dialogue constrained by the source transcript, audio, speaker profiles, context, and observed interaction pattern.
- Reconstruction and expansion provide paired aligned tracks with speaker-attributed transcripts, word timestamps, audio tags, and delivery instructions.
2 Related Work
Prior work established synchronized conversational speech modeling, in-the-wild separation, controllable synthesis, and rich speech annotation as relevant components. ConversationalVoice combines these capabilities into a conversation-level artifact that separates, reconstructs, and expands real dialogue.
- Synchronized-stream models and benchmarks motivate retaining independent channels, lexical structure, turn-taking, pauses, backchannels, interruptions, and simultaneous speech.
- In-the-wild dialogue systems address degraded monaural two-speaker audio, while DuplexChat produces separated full-duplex tracks at scale.
- ConversationalVoice builds on DuplexChat by adding screening, stable speaker assignment, quality verification, canonical transcription, reconstruction, and grounded expansion.
- Separation recovers observed streams, reconstruction regenerates the same linguistic event, and expansion adds a new event constrained by the original context.
- Modern TTS enables multilingual synthesis, voice cloning, and instruction-based control, but does not determine the semantics of training data.
- ConversationalVoice combines speaker identity, emotion, sound events, transcripts, and expressive controls in one conversation-level artifact without introducing a new separator, TTS model, or speaker encoder.
3 ConversationalVoice Pipeline
The pipeline screens real recordings, fixes speaker identities, and creates canonical transcripts before generating source-faithful reconstructions and grounded continuations. Its shared artifacts encode lexical content, timing, interaction structure, provenance, and delivery information.
- 3.1 Screening, diarization, and two-speaker selection: Screening normalizes recordings, detects speech, filters music and low-SNR regions, diarizes speakers, and retains windows with a strict two-speaker structure.
- 3.1 Screening, diarization, and two-speaker selection: Dynamic window selection uses diarization boundaries and activity statistics to preserve enough context for speaker identification and interaction mapping.
- 3.1 Screening, diarization, and two-speaker selection: Speaker-slot mapping uses activity correspondence and embeddings, then remains fixed to prevent local permutation from becoming an identity switch.
- 3.2 Quality verification and transcription: Quality verification rejects silent, weak, leaking, merged, split, or mismatched tracks, while ASR supplies the canonical transcript for reconstruction and expansion.
- 3.3 Source-faithful reconstruction: Reconstruction augments fixed source text with position-preserving audio tags and delivery instructions, validates lexical fidelity, and synthesizes speech from stable and local speaker references.
- 3.3 Source-faithful reconstruction: Forced alignment occurs before timeline assembly, avoiding timestamp drift at speech–silence boundaries.
- 3.3 Source-faithful reconstruction: Scheduling preserves turn order, source gap intent, and overlap relations while adapting timing to synthesized durations and preventing self-overlap.
- 3.4 Conversation-grounded expansion: Expansion creates new dialogue grounded in source transcript, audio, speaker profiles, and interaction style, with explicit event types, overlap status, tags, and delivery instructions.
4 Evaluation
The evaluation measures shared output quality, reconstruction fidelity, expansion interaction statistics, dialogue quality, and audio-tag alignment across the pipeline's artifacts. Results show strong speaker and acoustic quality, substantial preservation of source interaction structure, plausible expansion content, and weaker audio-tag alignment for expansion than reconstruction.
- Shared Output Quality: WER, predicted acoustic quality, and speaker identity are evaluated against stage-specific references without requiring waveform correspondence between stages.WER uses mixed output tracks and transcript-derived references; NISQA, DNSMOS, and speaker-identity metrics operate on active speech.
- Shared Output Quality: Same-speaker Similarity ranges from 0.983 to 0.991, while Speaker Discrimination Margins remain positive across all stages.NISQA MOS rises from 3.563 for separation to 4.405 for reconstruction and 4.608 for expansion.
- Reconstruction Fidelity: 26.3% longer reconstruction preserves source interaction structure with Turn F1 = 0.693, Overlap F1 = 0.664, and Backchannel F1 = 0.860.Events are matched using corresponding source utterances rather than absolute timestamps.
- Expansion Statistics and Analysis: Expansion duration is 2.340 times reconstruction, with turn rate 4.6% lower, overlap-event density 8.0% lower, backchannel rate 13.2% lower, and interruption density 16.0% lower.These statistics describe interaction structure rather than semantic quality, so content coherence and dialogue naturalness are evaluated separately.
- Audio-tag Annotation Quality: Audio-tag alignment scores 4.376 for reconstruction and 3.750 for expansion, with an overall score of 4.221 out of 5.The result indicates stronger correspondence between declared annotations and acoustic realizations in reconstruction than in expansion.
5 Discussion
ConversationalVoice assigns complementary roles to separation, reconstruction, and expansion, distinguishing waveform, content-and-interaction, and context fidelity. Its explicit interaction controls support coherent expansion and broadly preserve interaction profiles, while downstream training benefits remain untested.
- Reconstruction: Reconstruction regenerates source exchanges from fixed words and identities, improving quality while preserving turns, overlaps, and backchannels.It also links generated speech to explicit aligned supervision and delivery information.
- Expansion: Expansion extends content beyond recorded exchanges by conditioning new dialogue on source context, speakers, and observed interaction patterns.It explicitly plans backchannels, paralinguistic events, and sequential or overlapping placement.
- Expansion: Expansion receives coherent and natural automatic evaluations while remaining broadly similar to reconstruction in turn and overlap-event density.Its backchannel and interruption rates are modestly lower, and expressive delivery control remains an improvement area.
- Fidelity roles: The three artifacts represent different fidelity objectives: separation preserves recorded timing, reconstruction preserves content and interaction, and expansion preserves context while adding content.Their measured differences should be interpreted alongside the added control and coverage of each generated stage.
- Training implications: A staged mixture could combine natural interaction evidence, clean aligned supervision, and broader grounded conversational coverage.The compatible representation enables this mixture, but the paper does not evaluate it as a training curriculum.
6 Limitations, Ethics, and Release Considerations
The evaluation and release of ConversationalVoice face limitations involving automatic metrics, unverified detector calibration, English-only model judgment, rights and privacy, bias, and changing third-party components.
- Evaluation limits: Automatic evaluation does not establish absolute transcription accuracy, semantic plausibility, human preference, or reliable speaker fidelity.WER depends on pipeline-generated references, speaker similarity may miss pronunciation or emotion errors, and Gemini scores are descriptive.
- Evaluation limits: The frozen interaction detector has unverified calibration, while Audio-tag Alignment Score measures declared tags rather than annotation recall.These metrics therefore do not fully assess detected interaction quality or unannotated audible events.
- Scope: The study does not measure downstream full-duplex model performance.Whether the artifacts improve model training remains outside the evaluation scope.
- Ethics and release: Releasing real-world audio requires lawful processing, source-license compliance, privacy safeguards, provenance documentation, and controls for voice-cloning misuse.Generated expansions should be labeled synthetic and not presented as statements made by original speakers.
- Bias: Bias may enter through source selection and every major processing stage, potentially excluding accents, dialects, noisy environments, and overlapping styles.The paper calls for separate reporting across language, accent, gender presentation, acoustic condition, and source domain where appropriate.
- Reproducibility: Third-party model and service versions, licenses, and behavior can change, requiring pinned revisions, retained configurations, and stage-level service records.These records support reproducibility and compliance with component conditions.
7 Conclusion
ConversationalVoice converts monaural conversations into three linked artifacts with complementary roles: separation recovers natural evidence, reconstruction adds aligned supervision, and expansion broadens grounded coverage. Automatic results support this division of labor, but downstream effectiveness and independent perceptual validation remain unestablished.
- Pipeline: ConversationalVoice produces separation, source-faithful reconstruction, and conversation-grounded expansion in a shared aligned artifact schema.The schema includes speaker tracks, word-aligned transcripts, audio tags, utterance types, and delivery instructions.
- Evaluation: All stages retain positive speaker-discrimination margins and strong predicted acoustic quality, while expansion receives 4.94 coherence and 4.80 dialogue-naturalness scores.These results characterize data properties across the complementary artifacts.
- Interaction profile: Expansion differs from reconstruction by 4.6% in turn density and 8.0% in overlap-event density, with backchannel and interruption rates 13.2% and 16.0% lower.Audio-tag alignment is stronger in reconstruction, whose overall score is 4.221 out of 5.
- Conclusion: The results support distinct roles: separation provides natural interaction evidence, reconstruction cleaner controllable supervision, and expansion broader grounded conversational coverage.The paper presents this as a practical route to structured supervision for learning conversational timing.
- Scope: The study does not establish downstream effectiveness or independent perceptual validation.Matched training studies and independent validation are required to test those outcomes.