SmoothConv and DuplexConv: Complementary Mandarin Multi-Party Conversational Speech Corpora for Speech Interaction

1. INTRODUCTION

• Multi-party conversations expose turn-taking, response timing, paralinguistic cues, overlaps, and changing interaction states that speech interaction systems need to model.Existing resources often lack synchronized participant-level tracks or comprehensive interaction annotations, limiting their applicability to realistic multi-party interaction.Source: §1 · p. 1
• Existing Mandarin resources remain insufficient for realistic multi-party modeling because they lack synchronized participant tracks or detailed interaction annotations.Switchboard and Fisher are English two-speaker telephone corpora; CHiME-5 targets robust ASR and separation; AMI, AliMeeting, and AISHELL-4 have other recording or annotation limitations.Source: §1 · p. 1
• Manual annotation is reliable but costly to scale, while automatic annotation scales but may introduce recognition, alignment, and semantic errors.This motivates pairing carefully verified data for reliable analysis with automatically constructed data for scalable model training.Source: §1 · pp. 1–2
• SmoothConv and DuplexConv provide 2,100 hours of Mandarin multi-party conversation with synchronized participant tracks and multi-dimensional annotations, pairing verified analysis data with scalable training data.The release also includes the SmoothConv Benchmark and evaluations for speech separation, MSASR, and turn detection.Source: §1 · p. 2

2.1. Construction Pipeline

• The construction pipeline first performs per-channel recognition and speaker labeling, then adds multi-dimensional annotations for turn states, paralinguistic information, and speaker attributes.VAD preserves timeline synchronization; ASR outputs are fused with ROVER, then diarization, forced alignment, and speaker clustering refine labels and boundaries.Source: §2.1 · p. 2
• Human review accepted 82.4% of automatically annotated samples in a randomly selected 20-hour DuplexConv subset.A sample passed when transcription, speaker labeling, and conversational interaction annotations met predefined quality criteria.Source: §2.1 · p. 2
• Turn-state annotation distinguishes complete, incomplete, backchannel, and wait states; automatic labels achieve 91% agreement with human annotations on the expert-reviewed subset.Textual predictions are verified with acoustic information, with audio decisions preferred for incomplete turns and backchannels when predictions disagree.Source: §2.1 · p. 2
• Emotion, nonverbal events, gender, and age are annotated automatically, with SmoothConv using predefined emotion categories and DuplexConv using free-form emotion descriptions.DuplexConv age labels use five groups, while paralinguistic labels include vocal events such as crying, coughing, and sighing.Source: §2.1 · p. 2

2.2. Dataset Statistics

• SmoothConv contains 100 hours from 2,503 human-verified clips, while DuplexConv contains 2,000 automatically annotated hours from 93,709 clips; both cover two-to-four-speaker conversations.SmoothConv receives review by four annotators and three senior annotators; the corpora contain Tutoring and Social Chat conversations with synchronized participant tracks.Source: §2.2 · p. 2

2.3. SmoothConv Benchmark

• The SmoothConv Benchmark provides task-dedicated evaluation subsets, with a shared speaker-disjoint subset for speech separation and MSASR and a separate turn-detection subset.The shared subset contains natural two-to-four-speaker conversations with a 3.33% average speech overlap ratio; turn detection includes 1,000 synthesized wait samples used only for evaluation.Source: §2.3 · pp. 2–3

3. EXPERIMENTS

• Speech separation, MSASR, and turn detection respectively assess speaker-level signal recovery, who-said-what understanding, and conversational dynamics.The experiments describe these as complementary aspects of interactive speech processing.Source: §3 · p. 3

3.1. Speech Separation

• Hybrid training offers a better trade-off across simulated and conversational conditions, while DuplexConv-only training performs competitively on conversational evaluation sets.The comparison reports SI-SDRi and CER for in-domain and out-of-domain evaluations.Source: Table 3 · p. 3, §3.1 · p. 4

3.2. Multi-speaker Automatic Speech Recognition

• DuplexConv adaptation reduces diarization and transcription errors on SmoothConv; combining it with SmoothConv further improves several conditions, while far-field AliMeeting performance degrades.The AliMeeting result suggests acoustic mismatch remains a challenge for cross-domain conversational adaptation.Source: Table 4 · p. 4, §3.2 · p. 4

3.3. Turn Detection

• On EasyTurn-Test, EasyTurn training performs best on wait, DuplexConv strengthens complete-turn recognition, and DC+SC improves incomplete and backchannel detection.On the SmoothConv Benchmark, DC+SC performs best on most states, but wait remains challenging because explicit waiting is rare and evaluation includes synthesized samples.Source: Table 5 · p. 4, §3.3 · p. 4

• SmoothConv provides human-verified data for analysis and evaluation, while DuplexConv supplies large-scale automatically annotated conversations for training.The paper also releases the SmoothConv Benchmark and reports experiments on speech separation, MSASR, and turn detection.Source: §4 · p. 4