Source-linked AI summary
SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity
Hanke Xie, Haopeng Lin, Wenxiao Cao, Dake Guo, Wenjie Tian, Jun Wu, Hanlin Wen, Ruixuan Shang, Hongmei Liu, Zhiqi Jiang, Yuepeng Jiang, Wenxi Chen, Ruiqi Yan, Jiale Qian, Yichao Yan, Shunshun Yin, Ming Tao, Xie Chen, Lei Xie, Xinsheng Wang
TL;DR
Existing TTS systems are largely designed for single-speaker speech, while coherent multi-speaker conversational synthesis remains challenging. SoulX-Podcast uses interleaved text–speech modeling with paralinguistic and dialect support; the paper reports strong results in both multi-turn dialogue and conventional TTS.
Problem
Most previous TTS systems focus on single-speaker generation, while multi-speaker, multi-turn conversation requires fluency and naturalness that those systems struggle to maintain.
Source: §1 · p. 2Method
SoulX-Podcast uses interleaved text–speech modeling with speaker, dialect, and paralinguistic cues, alongside semantic-token prediction, flow matching, and vocoding.
Source: §1 · p. 2, §2.2 · p. 5Results
Takeaways & Limitations
The framework supports long-form multi-speaker speech across Mandarin, English, and several Chinese dialects, with paralinguistic cues and context-dependent prosody.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recent advances in text-to-speech (TTS) synthesis have significantly improved speech expressiveness and naturalness. However, most existing systems are tailored for single-speaker synthesis and fall short in generating coherent multi-speaker conversational speech. This technical report presents SoulX-Podcast, a system designed for podcast-style multi-turn, multi-speaker dialogic speech generation, while also achieving state-of-the-art performance in conventional TTS tasks. To meet the higher naturalness demands of multi-turn spoken dialogue, SoulX-Podcast integrates a range of paralinguistic controls and supports both Mandarin and English, as well as several Chinese dialects, including Sichuanese, Henanese, and Cantonese, enabling more personalized podcast-style speech generation. Experimental results demonstrate that SoulX-Podcast can continuously produce over 90 minutes of conversation with stable speaker timbre and smooth speaker transitions. Moreover, speakers exhibit contextually adaptive prosody, reflecting natural rhythm and intonation changes as dialogues progress. Across multiple evaluation metrics, SoulX-Podcast achieves state-of-the-art performance in both monologue TTS and multi-turn conversational speech synthesis.
1 Introduction
SoulX-Podcast addresses the gap between single-speaker TTS and coherent multi-speaker dialogue with a long-form, multi-speaker, multi-dialect framework. It combines paralinguistic controls and broad language support, and reports strong performance in both dialogue synthesis and conventional TTS.
1 Introduction
- Single-speaker systems that work well in isolated speech tasks can struggle to maintain fluent, natural multi-speaker, multi-turn conversation.The paper frames this as a gap motivating a system designed for seamless multi-speaker dialogue.Source: §1 · p. 2
- Most earlier conversational systems offer limited control over paralinguistic features, constraining the expressiveness and realism of generated dialogue.Prior approaches include parallel speaker channels, integrated dialogue text with speaker labels, and alternating multi-speaker generation.Source: §1 · p. 2
- SoulX-Podcast models long-form, multi-speaker, multi-dialect podcast speech with interleaved text–speech sequences to support consistent quality and speaker similarity.The framework models dialectal variation, paralinguistic cues, and context-dependent prosody.Source: §1 · p. 2
- The framework supports natural, fluent, coherent long-form dialogue with a variety of paralinguistic labels.It also supports Mandarin and English and Chinese dialects including Sichuanese, Henanese, and Cantonese.Source: §1 · p. 2
- SoulX-Podcast supports cross-dialectal zero-shot voice cloning, using one audio prompt to generate speech in any supported dialect.Supported dialects include Sichuanese, Henanese, and Cantonese alongside Mandarin and English.Source: §1 · p. 2
- SoulX-Podcast reports superior performance in multi-turn conversational synthesis and conventional TTS, including voice cloning.The introduction presents this as effectiveness across diverse speech-synthesis scenarios.Source: §1 · p. 2
2 Method
The section frames dialogue-text–speech paired data with speaker identity correspondence as a prerequisite for dialogue speech synthesis, then previews the report’s data-processing and algorithm descriptions.
2 Method
- Dialogue-text–speech paired data with speaker identity correspondence are a necessary prerequisite for dialogue speech synthesis.Source: §2 · p. 3
- The section introduces processing methods for dialogue data, including dialectal and paralinguistic annotation.Source: §2 · p. 3
- The section then presents the specific SoulX-Podcast algorithm.Source: §2 · p. 3
2.1 Data Processing
The data pipeline processes in-the-wild dialogue recordings into speaker-labeled, transcribed, quality-filtered speech and adds dialectal and paralinguistic annotations. The resulting training collection combines conversational and monologue speech at large scale.
2.1 Data Processing
- The dialogue workflow includes speech enhancement, segmentation and diarization, transcription, and quality filtering, with additional dialectal and paralinguistic annotations.The workflow is depicted in Figure 2.Source: §2.1 · p. 3, Figure 2 · p. 3
2.1.1 Basic Processing Workflow
- Long recordings are split into short utterances and roughly five-minute dialogue segments before diarization, addressing speaker-label inconsistency as voices and speaking states change over time.Source: §2.1.1 · p. 3
- Dual-ASR transcripts are checked using Chinese CER or English WER; high-error utterances retain text but lose audio, balancing dialogue completeness with transcription reliability.The pipeline also filters segments using SNR and DNSMOS quality estimates and excludes speaker-embedding outliers from audio while retaining their transcriptions.Source: §2.1.1 · pp. 3–4
2.1.2 Paralinguistic and Dialectal Data Annotation
- Approximately 1,000 hours of speech receive detailed paralinguistic annotations through a two-stage process combining automated detection with model-assisted verification.Source: §2.1.2 · p. 4
- The corpus includes approximately 2,000 hours of Sichuanese, 1,000 hours of Cantonese, and 500 hours of Henanese speech, transcribed with Seed-ASR after the standard pipeline proved suboptimal for dialectal speech.Source: §2.1.2 · p. 4
2.1.3 Corpus Overview
- Approximately 0.3 million hours of conversational speech and 1.0 million hours of monologue data yield a training dataset of about 1.3 million hours.Source: §2.1.3 · p. 4
2.2 SoulX-Podcast
SoulX-Podcast combines semantic-token prediction with flow matching and vocoding, organizing speech as speaker-labeled interleaved text and audio. Its training curriculum and dialect-guided inference address long-form dialogue, dialect generation, and cross-dialect prompting.
2.2 SoulX-Podcast
- SoulX-Podcast predicts semantic tokens with an LLM, converts them to acoustic features through flow matching, and synthesizes waveform audio with a vocoder.Its Qwen3-1.7B backbone has an extended text codebook for speech and special dialectal and paralinguistic tokens.Source: §2.2 · p. 5
2.2.1 Token Organization
- Each speaker’s text tokens precede corresponding speech tokens in temporal order, with speaker tokens marking identity, dialect tokens following them, and paralinguistic cues placed at their sequence positions.Source: §2.2.1 · p. 5
2.2.2 Training
- Training progresses from mixed monologue and dialogue data to Chinese- and English-language multi-speaker dialogue with dialectal and paralinguistic elements, followed by additional dialect-data fine-tuning.Context regularization progressively drops historical speech tokens while retaining text context to encourage semantic continuity and improve extended-dialogue coherence and stability.Source: §2.2.2 · pp. 5–6
2.2.3 Inference
- When Mandarin and a target Chinese dialect share similar text, dialect control can weaken; Dialect-Guided Prompting prepends a dialect-typical sentence so a Mandarin prompt can guide speech in the target dialect.Inference continues the training format by autoregressively generating interleaved speech tokens.Source: §2.2.3 · p. 6, Figure 4 · p. 6
3 Performance of SoulX-Podcast
The evaluation covers monologue TTS, multi-turn podcast generation, paralinguistic control, and three Chinese dialects. SoulX-Podcast performs strongly across these settings, with results also qualified by evaluator ambiguity for subtle paralinguistic events and possible ASR limitations for dialect CER.
3.1 Monologue Speech Generation
- On Seed-TTS-eval, SoulX-Podcast has the lowest Chinese CER; speaker similarity ranks just behind Seed-TTS and MaskGCT on both test sets.The comparison evaluates zero-shot monologue TTS on Chinese and English test sets.Source: Table 1 · p. 7, §3.1 · p. 7
3.2 Podcast Generation
- On ZipVoice-Dia, SoulX-Podcast achieves the lowest WER/CER and highest cpSIM on Chinese and English subsets, while maintaining competitive UTMOS scores.The benchmark comprises natural multi-turn conversations and evaluates intelligibility and cross-speaker consistency.Source: Table 2 · p. 7, §3.2 · p. 7
3.3 Evaluation of Paralinguistic Control
- Paralinguistic-event control reaches 0.82 overall accuracy, with breathing (0.75) and coughing (0.70) identified as harder for the evaluator to distinguish.The test used 20 GPT-5-generated utterances for each of five labels, synthesized in monologue mode and checked by an automated recognizer.Source: §3.3 · pp. 7–8, Table 3 · p. 8
3.4 Dialect Generation
- Across Sichuanese, Henanese, and Cantonese, speaker similarity is consistent with Mandarin and English performance.Dialect CER may partly reflect limitations of the dialect-specific ASR systems; the evaluation includes monologue and dialogue settings.Source: Table 4 · p. 8, §3.4 · p. 8
4 Conclusions
SoulX-Podcast uses interleaved text–speech modeling for long-form, multi-speaker, multi-dialect conversational synthesis. The reported results show effectiveness in dialogue synthesis and zero-shot monologue TTS, alongside support for dialects and paralinguistic cues.
4 Conclusions
- Interleaved text–speech modeling supports long-form, multi-turn conversational speech with consistent quality and coherence, alongside zero-shot monologue TTS, dialect, and paralinguistic capabilities.The conclusion presents SoulX-Podcast as a unified framework for speech generation.Source: §4 · p. 8
5 Ethics Statement
The study used publicly available or synthetically generated datasets and excluded personally identifiable information and private recordings. It also identifies misuse risks and recommends safeguards for downstream applications.
5 Ethics Statement
- All study datasets were public or synthetically generated, with no personally identifiable information or private recordings included.Source: §5 · p. 9
- The authors identify voice spoofing, impersonation, and misinformation risks, and recommend consent, watermarking, and misuse detection for downstream applications.Source: §5 · p. 9