FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications

1 Introduction

• The framework targets growing demand for personalized and diverse speech in virtual assistants, chatbots, video dubbing, and other AI products.The paper cites intelligent interaction and AI content creation as TTS use cases.Source: §1 · p. 1
• Dubbing supports zero-shot voice cloning for UGC and few-shot adaptation to studio-level expressive PUGC voices with one hour of recording.The paper frames these capabilities as making video dubbing customizable and contrasts their training-data needs with conventional TTS.Source: §1 · p. 2
• Instruction tuning supports casual chatbot speech with multiple emotions and rich paralinguistic behaviors.The stated aim is human-like speech generation for chatbot interaction.Source: §1 · p. 2
• FireRedTTS combines a high-coverage TTS dataset pipeline, a language-model-based foundation system, and dubbing and chatbot applications.The dataset is designed to cover content, speaking style, and timbre; the system generates discrete semantic tokens from prompt text and audio, then reconstructs audio through a two-stage generator.Source: §1 · p. 2

2 Data Processing Pipeline

• The five pipeline steps are speech enhancement, speech segmentation, speaker clustering, transcription, and data filtering.The pipeline converts massive in-the-wild speech into a large-scale TTS dataset.Source: §2 · pp. 2–3
• Speech enhancement removes music and noise, while segmentation cuts audio into 2–20-second chunks and joins segments separated by less than one second of silence.Segment boundaries are extended by 0.3 seconds to avoid cutting aspiration and trailing sounds.Source: §2 · p. 3
• Speaker clustering assigns speaker IDs using embeddings and merges clusters above 0.8 cosine similarity; multi-speaker and distant-from-centroid segments are filtered out.The pipeline uses spectral clustering and K-Means before iteratively merging similar clusters.Source: §2 · p. 3
• 248k hours of labeled speech were collected from 624k hours of unlabeled audio; training uses 150k hours (110k Chinese, 40k English).Figure 2 presents the amount of data filtered at each step.Source: Figure 2 · p. 3, §2 · p. 3
• A two-pass Transducer ASR model transcribes segments, followed by filtering for speech quality, sampling rate, and transcription confidence.Segments below ASR confidence 0.8 are discarded.Source: §2 · p. 3
• Filtering retains segments above DNSMOS P.835 OVRL 3.3 and roll-off frequency 7 kHz, where roll-off marks the frequency containing 99.5% of spectral energy.The figure caption gives thresholds of 3.3 for αmos, 7 kHz for αrolloff, and 0.8 for αasr.Source: Figure 1 · p. 2, §2 · p. 3

3.1 Semantic-Aware Speech Tokenizer

• SAST quantizes HuBERT-derived semantic features at 40-ms intervals and pairs them with an utterance-level embedding for speaker identity, style, and acoustic environment.Source: §3.1 · p. 4
• Its decoder adds a duplicated global embedding to quantized tokens, then upsamples to reconstruct SSL and acoustic features under weighted VQ and reconstruction losses.Source: §3.1 · pp. 4–5

3.2 Text-To-Speech Language Model

• A decoder-only Transformer predicts semantic tokens from prompt text and a speaker embedding derived from prompt audio, generating tokens autoregressively through in-context learning.Source: §3.2 · p. 5

3.3 Token-to-Waveform Generation

• The two-stage generator converts semantic tokens to 16-kHz Mel spectrograms, then super-resolves them to 48-kHz audio, allowing low-rate training data to contribute.Source: §3.3 · p. 5

3.3.1 Flow-Matching

• Flow matching transforms Gaussian noise into Mel spectrograms conditioned on semantic tokens, while Conformer cross-attention after each self-attention layer extracts timbre from same-speaker reference audio.Source: §3.3.1 · p. 5

3.3.2 Streamable Decoder

• To avoid slow iterative inference, the streamable decoder predicts four-stream acoustic tokens from semantic tokens, enabling synchronous streaming of acoustic tokens and waveform.Source: §3.3.2 · p. 6

3.3.3 Super-Resolution Vocoder

• A BigVGAN-V2 vocoder upsamples 16-kHz Mel features by 480x to 48-kHz audio, trained on a curated 294-hour high-sampling-rate subset.Source: §3.3.3 · p. 6

4.1 Voice Cloning for Dubbing

• For UGC dubbing, zero-shot in-context learning uses speaker embeddings from prompt audio, with prompt enhancement intended to improve embeddings from noisy recordings.For expressive PUGC voices, supervised fine-tuning adapts the foundation model to the target speaker using studio-quality recordings.Source: §4.1 · pp. 6–7

4.2 Human-like Speech Generation for Chatbots

• Chatbot speech generation uses instruction tuning to address limited fine-grained control of emotion and paralinguistic behaviors in the foundation system.Emotion embeddings are inserted into the prompt sequence, while training uses a separate audio sample from the same speaker to separate speaker and emotion representations.Source: §4.2 · p. 7
• Paralinguistic behaviors enter as text tokens for acoustic events or embeddings associated with words for overlapping labels; the implementation includes 13 behaviors.The behaviors include pauses, elongations, repetitions, laughter, and breathing sounds.Source: §4.2 · p. 7, Table 1 · p. 8

5.1.1 Consistency Evaluation

• 4.32 CoMOS for FireRedTTS exceeds CosyVoice’s 4.15, while ground-truth audio scores 4.53 on consistency of content, speaking style, and timbre.The test uses 94 Chinese prompt-target pairs and listener ratings from 1 to 5.Source: §5.1.1 · p. 8, Table 2 · p. 8

5.1.2 Stability Evaluation

• FireRedTTS has better overall sentence-level error performance than CosyVoice, but significant stability issues remain for English and code-switch utterances.English pronunciation errors are primarily insertions and deletions; the authors attribute the discrepancy mainly to limited and insufficiently diverse English and code-switch training data.Source: §5.1.2 · pp. 8–9, Table 3 · p. 9

5.1.3 Streamable Decoding

• The streamable decoder enables real-time generation with a modest synthesis-quality reduction relative to flow matching, whose CoMOS is 4.48.The authors partly attribute the quality degradation to suboptimal Mel spectrogram generation by the Mel codec.Source: §5.1.3 · p. 9, Table 4 · p. 9

5.2.1 Zero-Shot v.s. Few-Shot

• In UGC, zero-shot synthesis is already high quality and 2-minute fine-tuning slightly improves performance; in PUGC, few-shot fine-tuning improves MOS and SIM for the distinctive voice.The PUGC adaptation uses 1 hour of data, while UGC fine-tuning uses 2 minutes.Source: §5.2.1 · pp. 9–10, Table 5 · p. 10

5.2.2 Prompt Enhancement

• Prompt enhancement may hurt cloning with high-SNR audio, but effectively mitigates degradation with lower-SNR prompts, so the authors recommend applying it selectively.The comparison tests prompts with added background noise at 20 dB, 10 dB, and 0 dB SNR.Source: §5.2.2 · p. 10, Table 6 · p. 10

5.3 Human-like Speech Generation

• Most synthesized emotional samples convey the intended emotion, with a small subset of low-intensity samples misclassified as neutral.The test assigns one of four emotions—neutral, happy, sad, or angry—to each of 100 Chinese utterances.Source: §5.3 · p. 10, Figure 5 · p. 11
• Instruction tuning significantly improves emotion classification accuracy over the pretrained model, supporting controllable target-emotion synthesis.The comparison uses an emotion recognition model to classify synthesized speech.Source: §5.3 · pp. 10–11, Table 7 · p. 11
• Synthesized speech with paralinguistic behaviors receives a higher human-likeness preference of 45% than speech without them.The authors identify this preference test as evidence for the role of paralinguistic behaviors in natural-sounding chatbot speech.Source: §5.3 · p. 11, Table 8 · p. 11

• 624k hours of raw audio were transformed into 248k hours of high-quality TTS data spanning content, speaking style, and timbre.Source: §6 · p. 11
• Subjective and objective evaluations show high-quality speech aligned with prompt text and audio in content, speaking style, and timbre.Source: §6 · p. 11
• FireRedTTS supports dubbing voice cloning in UGC through zero-shot use and in PUGC through few-shot approaches.Source: §6 · p. 11
• Instruction tuning enables controllable emotional speech with paralinguistic behaviors for human-like chatbot speech.Source: §6 · p. 12