Source-linked AI summary
FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, Yao Hu
TL;DR
Existing dialogue TTS systems can require complete dialogue text and produce inseparable mixed speech, limiting interactive generation. FireRedTTS-2 addresses this with streaming interleaved text–speech modeling and a dual-transformer architecture. It integrates with chat frameworks and outperforms stated dialogue baselines in podcast intelligibility, speaker-turn reliability, and naturalness.
Problem
Existing dialogue TTS approaches can require complete dialogue text, produce inseparable mixed voices, and exhibit unstable synthesis, inaccurate speaker transitions, or incoherent prosody.
Method
FireRedTTS-2 combines a 12.5Hz streaming speech tokenizer with chronological speaker-labeled text–speech interleaving and a dual-transformer TTS model.
Results
FireRedTTS-2 surpasses MoonCast, ZipVoice-Dialogue, and MOSS-TTSD in podcast intelligibility, speaker-turn reliability, and perceived naturalness.
Takeaways & Limitations
The system supports sentence-by-sentence generation for interactive chat and podcast production, with context-consistent prosody and minimal fine-tuning.
Takeaways & Limitations
Dia and Sesame were excluded from comparison because the authors could not generate the test set with those systems due to instability.
Abstract
from arXiv · showhide
Current dialogue generation approaches typically require the complete dialogue text before synthesis and produce a single, inseparable speech containing all voices, making them unsuitable for interactive chat; moreover, they suffer from unstable synthesis, inaccurate speaker transitions, and incoherent prosody. In this work, we present FireRedTTS-2, a long-form streaming TTS system for multi-speaker dialogue generation, delivering stable, natural speech with reliable speaker switching and context-aware prosody. A new 12.5Hz streaming speech tokenizer accelerates training and inference, extends maximum dialogue length, encodes richer semantics to stabilize text-to-token modeling and supports high-fidelity streaming generation for real-time applications. We adopt a text-speech interleaved format, concatenating speaker-labeled text with aligned speech tokens in chronological order, and model it with a dual-transformer: a large decoder-only transformer predicts tokens at the first layer, and a smaller one completes subsequent layers. Experimental results show that FireRedTTS-2 integrates seamlessly with chat frameworks and, with minimal fine-tuning, produces emotionally expressive speech guided by implicit contextual cues. In podcast generation, it surpasses existing systems including MoonCast, Zipvoice-Dialogue, and MOSS-TTSD in objective intelligibility, speaker-turn reliability, and perceived naturalness with context-consistent prosody. Our demos are available at https://fireredteam.github.io/demos/firered_tts_2.
1 Introduction
Existing dialogue TTS systems organize or synthesize turns in ways that limit interactive generation and can produce mixed voices, while FireRedTTS-2 uses interleaved text–speech modeling for flexible sentence-by-sentence synthesis.
- Dialogue TTS systems commonly produce a single mixed speech track, limiting separation of voices for interactive use.
- Interleaving each turn’s text and speech supports flexible sentence-by-sentence generation for interactive chat and podcast production.
- FireRedTTS-2 is a long-form streaming multi-speaker TTS system targeting stable speech, reliable speaker switching, and context-aware prosody.
2 FireRedTTS-2
FireRedTTS-2 combines a low-rate semantic speech tokenizer with interleaved text–speech modeling and a dual-transformer architecture for efficient conversational generation.
- Speech tokenizer: The tokenizer reduces speech representation to 12.5Hz while adding semantic supervision to stabilize long multi-speaker sequence modeling.It also supports high-fidelity streaming generation for real-time applications.
- Speech tokenizer: The tokenizer extracts semantic and acoustic features, downsamples them, quantizes them with an RVQ, and reconstructs waveform audio through decoders.
- Speech tokenizer: The system trains its tokenizer in two stages, first optimizing non-streaming acoustic reconstruction and then replacing the decoder with a streaming variant.
- Text-to-speech model: The TTS model interleaves speaker-labeled text and corresponding speech tokens chronologically, enabling sentence-by-sentence generation.Its dual-transformer uses a large backbone for first-layer prediction and a smaller decoder for subsequent layers.
- Text-to-speech model: The training objective combines backbone, decoder, and textual cross-entropy losses, with λtext = 0.01 and λdecoder = 0.6.The decoder transformer is optimized on one eighth of the speech segments.
- Training: A three-stage curriculum uses monologue pretraining, multi-speaker dialogue post-training, and supervised fine-tuning.
3 Downstream Applications
FireRedTTS-2 supports monologue, interactive chat, and podcast generation by conditioning on text and speech context, enabling coherent prosody and flexible turn-by-turn production.
- Downstream applications: FireRedTTS-2 provides competitive zero-shot voice cloning for monologues and coherent-prosody dialogue generation.
- Monologue generation: Its tokenizer combines semantic and acoustic information, while prompt transcript, target text, and prompt speech tokens condition autoregressive speech-token generation.
- Interactive chat: Existing chat frameworks often use monologue TTS without prior conversational context, causing inconsistent emotion and prosody and motivating explicit emotion instructions.
- Interactive chat: FireRedTTS-2 integrates into existing chat frameworks and is fine-tuned to infer emotion and prosody from implicit context.
- Podcast generation: Compared with segmented monologue synthesis, the system generates dialogue sentence by sentence, simplifying editing and producing contextually coherent prosody.
- Podcast generation: Zero-shot podcast generation uses two dialogue turns as prompt context and generates subsequent turns sequentially for multi-speaker conversations.The described setup supports 3-minute dialogues with 4 speakers.
- Podcast generation: Customized podcast generation delivers stable synthesis, accurate speaker transitions, and prosody matching the hosts’ distinctive speaking styles.
4 Results
FireRedTTS-2 demonstrates strong tokenizer, voice-cloning, interactive-chat, and podcast-generation performance, including stable synthesis, reliable speaker transitions, and naturalness comparable to ground truth in fine-tuned podcast tests.
- 4.1 Speech Tokenizer Evaluation: The 12.5Hz speech tokenizer achieves the highest intelligibility and ranks first or second on speaker-similarity and speech-quality metrics.Its advantages are attributed to semantic injection, explicit supervision, a larger quantizer, and a Vocos-based acoustic decoder, though it trails Mimi on PESQ and Xcodec2 on UTMOS.
- 4.2 Voice Cloning Evaluation: 1.14% CER on Mandarin and 1.95% WER on English closely match the best reported results of 1.12% CER and 1.83% WER.The authors note that objective metrics may not fully reflect expressive TTS quality and therefore emphasize subjective evaluations.
- 4.3 Interactive Chat Evaluation: FireRedTTS-2 produces emotionally expressive speech by inferring target emotions from implicitly contextualized queries and responses.The evaluation covers surprise, sadness, happiness, concern, apology, and anger, with 30 test cases per emotion.
- 4.4 Podcast Generation Evaluation: FireRedTTS-2 achieves the lowest WER/CER, highest speaker similarity, and lowest MCD in zero-shot podcast generation, while CMOS confirms contextually coherent naturalness.The results indicate stable long-sequence synthesis, accurate cross-turn speaker transitions, and minimal deviation from ground truth.
- 4.4 Podcast Generation Evaluation: The podcast comparison excludes Dia and Sesame because the authors were unable to generate their test sets due to instability.This limits the set of competing systems included in that evaluation.
- 4.4 Podcast Generation Evaluation: 56% of fine-tuned podcast trials match or surpass ground-truth naturalness, with synthesis preferred in 28% and judged equally natural in 28%.The comparison uses two podcast speakers and ground-truth dialogue recordings.
5 Conclusions
FireRedTTS-2 combines a low-rate semantic speech tokenizer with context-aware, interleaved text–speech modeling for monologue, interactive chat, and podcast generation. It supports streaming sentence-by-sentence synthesis, competitive voice cloning, implicit emotion control, and stable dialogue generation with reliable speaker transitions.
- 5 Conclusions: FireRedTTS-2 combines a 12.5Hz semantic speech tokenizer with a dual-transformer and interleaved text–speech format for dialogue-centric generation.The tokenizer shortens speech sequences and supports high-fidelity streaming decoding, while the model supports first-packet latency under 100 ms.
- 5 Conclusions: The system supports interactive chat and offline podcast production through sentence-by-sentence generation and integration without modifying other framework modules.It also produces emotionally expressive speech inferred from implicit contextual cues.
- 5 Conclusions: FireRedTTS-2 provides competitive zero-shot voice cloning for monologues and more coherent prosody, stable synthesis, and accurate speaker transitions for dialogues.The conclusion also reports that the system can be tailored to application requirements with minimal data.