PaperlayerView exact revision on arXiv

Source-linked AI summary

MoonCast: High-Quality Zero-Shot Podcast Generation

Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, Xiangyang Li

arXiv:2503.14345v2eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Short-utterance TTS remains challenged by long, multi-speaker, spontaneous podcast dialogue. MoonCast combines long-context language-model speech modeling with LLM-generated spontaneous scripts, and outperforms concatenate baselines, particularly on spontaneity and coherence.

  • Problem

    TTS systems focused on short individual utterances face challenges generating podcasts that are long, multi-speaker, coherent, and spontaneous.

    Source: §1 · p. 1

  • Method

    MoonCast pairs long-context language-model speech modeling with an LLM module that turns text-only sources into spontaneous podcast scripts.

    Source: §1 · p. 2

  • Results

    MoonCast outperformed concatenate baselines, with subjective gains of 0.40 in Chinese spontaneity and 0.85 in English spontaneity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MoonCast demonstrates text-only podcast generation with unseen speakers for spontaneous podcasts of up to ten minutes.

    Source: §1 · p. 2, §6 · p. 8

    Generated speech sometimes hallucinates speaker identity, causing incorrect attribution of utterances.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Recent advances in text-to-speech synthesis have achieved notable success in generating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These limitations arise from two primary challenges: 1) long speech: podcasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming to synthesize natural podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers. To generate long audio, we adopt a long-context language model-based audio modeling approach utilizing large-scale long-context speech data. To enhance spontaneity, we utilize a podcast generation module to generate scripts with spontaneous details, which have been empirically shown to be as crucial as the text-to-speech modeling itself. Experiments demonstrate that MoonCast outperforms baselines, with particularly notable improvements in spontaneity and coherence.

1. Introduction

Existing TTS systems remain challenged by long, multi-speaker podcasts and the spontaneous flow of conversation. MoonCast combines long-context speech modeling with spontaneous-script generation and outperforms concatenate baselines on podcast-generation evaluations.

1. Introduction

  • Prior TTS research largely targets short individual utterances, while podcasts require long, coherent, multi-speaker speech with spontaneous interaction.Podcast speech may include fillers, hesitations, minor mistakes, and speaker interactions; prior conversational-speech work has difficulty with longer podcast scenarios.Source: §1 · p. 1
  • MoonCast uses language-model-based speech modeling, large-scale long-context speech data, staged curriculum learning, and chunk-wise autoregressive detokenization.The curriculum progressively develops zero-shot, long-context, and spontaneous speech generation capabilities.Source: §1 · p. 2
  • An LLM-powered module converts text-only sources into conversational scripts with spontaneous details, which the authors report significantly influence generated-audio spontaneity.The module provides examples and guidance for generating spontaneous details in scripts.Source: §1 · p. 2
  • MoonCast outperformed concatenate baselines, with subjective gains of 0.40 in Chinese spontaneity and 0.85 in English spontaneity.The reported evaluations also show gains in coherence, intelligibility, speech quality, and speaker similarity for Chinese, and in coherence and intelligibility for English.Source: §1 · p. 2
  • MoonCast is open-sourced with script-generation prompts and the audio-modeling module to support future research.Source: §1 · p. 2

2. Background

The background reviews zero-shot TTS representations, multi-speaker dialogue generation, and spontaneous speech synthesis. It highlights the remaining challenge of generating long, natural, spontaneous conversations.

2.1. Zero-Shot TTS

  • Zero-shot TTS uses brief prompt speech to mimic a target speaker without fine-tuning; this paper combines code- and latent-based methods for two-speaker podcast speech.Prior approaches include codec language models and semantic-code methods, as well as latent diffusion and flow matching.Source: §2.1 · p. 2

2.2. Dialogue and Conversation Generation

  • Prior dialogue systems model turn-taking, backchannels, laughter, and diverse timbres, but many rely on roughly 2,000-hour datasets and generate dialogues under 90 seconds.MoonCast addresses this duration boundary with a long-context text-to-semantic autoregressive architecture for inter-sentence prosody, speaker change, and paragraph-level spontaneity.Source: §2.2 · pp. 2–3

2.3. Spontaneous TTS

  • Spontaneous TTS targets conversational rather than read speech, with fillers, varied rhythms, and natural prosody; prior work models spontaneous style or behavior explicitly.The paper also reports that script text itself significantly influences generated-audio spontaneity.Source: §2.3 · p. 3

3. Method

MoonCast separates source-to-script conversion from multi-speaker speech generation, using long-context audio modeling to synthesize two-speaker podcasts in unseen voices.

3.1. Overall

  • MoonCast converts knowledge sources into spontaneous scripts, then generates two-speaker podcast speech using unseen speakers’ voices.The audio pipeline represents speech as discrete semantic codes before mel-spectrogram and waveform reconstruction.Source: §3.1 · p. 3

3.2.1. LONG-CONTEXT TWO-SPEAKER TEXT-TO-SEMANTIC MODEL

  • A unified language model predicts both speakers’ full speech-code sequence, including transitions, using full-text/full-audio interleaving and a curriculum that progresses toward spontaneous conversational data.Training scales to 40,000-token contexts for multi-turn speech generation.Source: §3.2.1 · pp. 3–4

3.2.2. CHUNK-WISE AUTOREGRESSIVE SPEECH DETOKENIZER

  • Chunk-wise autoregressive detokenization avoids slow, memory-heavy whole-sequence reconstruction and independent-segment boundary discontinuities by processing small chunks with causal access to prior chunks.A flow-matching model conditions on speech codes to generate mel-spectrograms.Source: §3.2.2 · p. 4

3.3. LLM-Powered Script Generation Module

  • The script module analyzes source content, builds a key-point briefing, then prompts a two-speaker conversational script with citations, term explanations, and spontaneous speech cues.The script guidance includes fillers, response words, repetitions, informal grammar, and pause formatting.Source: §3.3 · p. 5

4. Experiments and Results

The experiments compare MoonCast with concatenation baselines on Chinese and English podcast generation and examine how script spontaneity affects output. MoonCast leads on coherence and spontaneity, while written scripts show poorer performance than spontaneous-script variants.

4.1.1. DATA PREPARATION

  • The internal Chinese-English corpus contains 300,000 hours of Chinese audiobook speech, 15,000 hours of Chinese conversational speech, and 200,000 hours of English conversational speech.The source audio pool comprises approximately 1.0 million hours from podcasts, audiobooks, and show clips.Source: §4.1.1 · p. 6

4.1.2. MODEL DETAILS

  • The model details include a 2.5B-parameter text-to-semantic Transformer trained in three curriculum stages and a 0.8B-parameter speech detokenizer.The text-to-semantic model's third stage mixes Chinese and English conversational data; inference uses 3-second chunks.Source: §4.1.2 · p. 6

4.1.3. EVALUATION DETAILS

  • Evaluation uses unseen two-speaker Chinese podcasts, text-only knowledge sources, concatenation baselines, subjective ratings, SIM-O, and CER.The podcast-source set uses PDFs and web URLs; speaker prompts contain 3–10 seconds of speech, and five evaluators rate spontaneity, coherence, intelligibility, speech quality, and speaker similarity.Source: §4.1.3 · p. 6

4.2.1. EVALUATION ON PODCAST GENERATION

  • MoonCast consistently surpasses both concatenation baselines on coherence and spontaneity for Chinese and English podcast generation.The supplied tables are identified as Chinese and English podcast-generation comparisons, but their cell values are not provided.Source: §4.2.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • MoonCast records CER values of 2.15 for Chinese and 1.91 for English despite ASR errors on proper nouns and filler words.The authors present these results as evidence of system robustness.Source: §4.2.1 · p. 7
  • SIM-O and subjective similarity ratings diverge, and the authors identify lower English SIM-O as a potential limitation.They speculate that a single-speaker embedding may miss traits such as prosody, and that the lack of English audiobook sources may contribute to the lower score.Source: §4.2.1 · p. 7

4.2.2. IMPACT OF SPONTANEOUS SCRIPT

  • Removing spontaneous details lowers the spontaneity score by 0.95 versus the GT script; reintroducing them brings it to within 0.13 of GT.Written scripts also show a performance deficit exceeding 0.3 on quality, similarity, and coherence compared with spontaneous-script settings.Source: §4.2.2 · p. 7
  • SIM-O and intelligibility remain commendable across script settings, but generated audio still differs noticeably in quality from GT audio.The authors point to recognition and diarization errors in the GT script and omitted spontaneous non-speech details, such as throat clearings, as possible contributors.Source: §4.2.2 · pp. 7–8

5. Discussions

The discussion contrasts BPE with phoneme-based text representation and describes speaker-attribution hallucinations and a spontaneity–hallucination trade-off.

5.1. Phoneme vs. BPE for Text Representation

  • BPE preserves semantic content for long-form, multi-speaker speech and maintains intelligibility while improving prosody and context-driven paralinguistic effects such as laughter.The authors contrast this with phoneme-based representations, which they say strip semantic information needed for natural speaker transitions, emotion, and prosody.Source: §5.1 · p. 8

5.2. Hallucination Issues

  • Hallucinations can arise from retained timbre information in semantic tokens, speaker-identification or diarization errors, and ambiguous interpretations of the text.The discussion highlights rapid transitions between speakers as a setting where data-pipeline errors may occur.Source: §5.2 · p. 8
  • Ambiguous turn-taking, such as interpreting “um” as either a filler or a response, can confuse speaker attribution even with adequate semantic understanding.Increasing sampling diversity to enhance spontaneity is also associated with aggravated hallucinations, restricting higher spontaneity.Source: §5.2 · p. 8

6. Conclusion

MoonCast combines long-context language-model-based audio modeling with a podcast generation module to produce podcast-style speech from text-only sources using unseen speakers’ voices. The paper reports consistent outperformance of existing baselines across several speech-generation measures.

6. Conclusion

  • MoonCast combines long-context language-model-based audio modeling and a podcast generation module to synthesize podcast-style speech from text-only sources using unseen speakers’ voices.The approach addresses long speech duration and spontaneity, which the authors identify as challenges for traditional text-to-speech systems.Source: §6 · p. 8
  • Experiments show MoonCast consistently outperforms existing baselines in speech quality, speaker similarity, intelligibility, coherence, and spontaneity.Source: §6 · p. 8
  • The authors conclude that the approach advances text-to-speech for long, spontaneous dialogues and may support more realistic and engaging podcast generation.Source: §6 · p. 8

7. Impact Statements

The impact statement identifies potential misuse of high-fidelity voice generation and calls for detection and flagging safeguards.

7. Impact Statements

  • High fidelity to an original speaker’s voice creates a risk of improper application.Source: §7 · p. 8
  • The statement names deceptive voice recognition and mimicking an individual’s speech as examples of potential misuse.Source: §7 · p. 8
  • The authors call for reliable synthetic-speech detection and a mechanism to flag suspected malicious use.Source: §7 · p. 8

A. Speech Semantic Codec Details

The speech semantic codec discretizes self-supervised features into semantic tokens using a VQ-VAE, with reconstruction used to train the representation.

A. Speech Semantic Codec Details

B. Data Preparation

MoonCast builds training data through automated audio cleanup, diarization, transcription, quality scoring, and alignment. It applies different selection rules to single-turn and long-context conversational data.

B. Data Preparation

  • The pipeline suppresses noise, diarizes speaker segments, transcribes them with Paraformer, scores quality with DNSMOS, and retains successfully aligned segments for subsequent processing.Forced alignment uses a narrow beam size of 5.Source: B. Data Preparation · p. 11
  • Single-speaker, single-turn curriculum data retain segments with DNSMOS scores greater than 2.6.Source: B. Data Preparation · p. 11
  • Long-context conversational data require exactly two speakers, more than 10 turns, and average turn durations below 30 seconds.Source: B. Data Preparation · p. 11
  • The resulting dataset contains 300,000 hours of Chinese audiobooks, 15,000 hours of Chinese conversation, and 200,000 hours of English conversation.Source: B. Data Preparation · p. 11
  • Unlike the single-turn selection, long-context data are not filtered by DNSMOS scores or alignment results to preserve contextual information.Source: B. Data Preparation · p. 11

C. Prompts

The prompts define a bilingual brief-to-podcast workflow, specifying how to create a structured source brief and turn it into an accessible, lively two-speaker script. They also constrain script content, pronunciation, length, speaker roles, formatting, and discussion depth.

C. Prompts

  • The authors chose Gemini 2.0 Pro Experimental 02-05 for conversational style, natural dialogue, and topic coverage, and open-sourced prompts for both generation stages.The released prompts cover brief generation and brief-to-script generation.Source: C. Prompts · p. 11

C.1. English Prompt For Brief Generation

  • The English brief prompt requests a comprehensive, formal summary covering title and author, abstract, nonoverlapping themes organized by What, Why, and How, key citations, and a detailed conclusion.It also asks for accessible explanations of potentially confusing terms, faithful evidence and reasoning, and no mathematical formulas.Source: §C.1 · pp. 12–13

C.2. Chinese Prompt For Brief Generation

  • The brief requests a formal written-Chinese summary identifying the document’s title and authors, briefly covering its topic and author background, and organizing each main theme and concept according to a 3W principle.It also asks for a separate lay-friendly explanation of potentially confusing terms, concepts, and methods, with proper-name explanations grounded in the original. The How section should use accessible language and omit mathematical formulas and technical details.Source: §C.2 · pp. 13–14

C.3. English Prompts for Brief-to-Script Generation.

  • The English script prompt requires a JSON-parsable two-speaker dialogue that introduces the topic, develops key themes beyond the brief, and closes with a recap and outlook.The script should be conversational, use fillers and short sentences, avoid copying the brief verbatim, and follow restricted punctuation rules.Source: §C.3 · pp. 15, 17
  • The English prompt balances accessible information density by explaining technical terms and limiting experimental detail while encouraging multi-turn discussion, examples, and added background.Each topic should receive at least four speaker turns; the prompt suggests adding knowledge, new information, contrasting perspectives, and concrete stories or details.Source: §C.3 · pp. 17–18
  • The English prompt assigns Speaker 1 as host and Speaker 2 as guest, with consistent roles, interactive questioning, logical topic transitions, and a gradual, listener-aware knowledge pace.It also requires natural pronunciation of numbers and foreign words, with scripts capped at 3,000 words and 60 turns.Source: §C.3 · pp. 19–20

C.4. Chinese Prompts for Brief-to-Script Generation.

  • The Chinese script prompt likewise calls for an informal, detailed host–guest conversation that explains terms accessibly, develops each topic, and follows the brief’s logical order.It encourages examples and background knowledge while keeping information density manageable, introducing knowledge gradually, and restricting script punctuation.Source: §C.4 · pp. 20–23
  • Convert most foreign words and numbers into Chinese, choosing number readings by context and preserving digit-based English expressions such as a2b as their intended words.Use spaces between letters in uncommon abbreviations read letter by letter, and retain entity names unless they have a common Chinese translation.Source: §C.4 · p. 24
  • Keep the script under 3,000 characters and 60 utterances, while preserving depth in technical details and thematic discussion.The prompts discourage shortening each topic merely to meet the limits or restricting the conversation to a summary of the source.Source: §C.4 · p. 24
  • Assign speaker 1 as host and speaker 2 as guest, without giving either a name.Source: §C.4 · p. 24
  • Restrict script text to commas, periods, and question marks, prohibiting exclamation marks, ellipses, parentheses, quotation marks, and dashes.Source: §C.4 · p. 24
  • Develop each topic in depth and supplement the source with relevant background and examples to help listeners understand.The prompts say not to limit the discussion to summarizing the supplied text.Source: §C.4 · p. 24

D. Limitations and Future Works

The authors identify limits in language coverage, interaction scale, data quality, and annotation scope. They describe expanding these areas as directions for future work.

Language Coverage

  • Language Coverage: The system currently supports only Chinese and English; expanding language coverage is proposed to improve applicability across linguistic contexts.Source: D. Limitations and Future Works · p. 24

Multi-Speaker

Data Quality

  • Data Quality: The preparation pipeline may introduce ASR and diarization errors and filters out overlapping speech; the authors propose human-annotated spontaneous speech for fine-tuning.Source: D. Limitations and Future Works · p. 24

Annotations

Loading 2503.14345v2…