Source-linked AI summary
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, Xiaobin Zhuang
TL;DR
Seed-TTS addresses the challenge of generating natural, expressive, and controllable speech for arbitrary speakers with little data. It combines large-scale autoregressive TTS with self-distillation, reinforcement learning, and a fully diffusion-based variant, achieving human-matched zero-shot ICL quality and comparable Seed-TTS_DiT performance to autoregressive methods.
Problem
Seed-TTS targets speech generation approaching human-level quality for arbitrary speakers in the wild with little data.
Method
The paper develops a large-scale autoregressive TTS family with self-distillation, reinforcement learning, and a fully diffusion-based Seed-TTS_DiT variant.
Results
Seed-TTS achieves zero-shot ICL subjective performance closely matching real human speech, while Seed-TTS_DiT performs comparably to autoregressive language model-based methods.
Takeaways & Limitations
Seed-TTS provides a foundation model for expressive, controllable speech generation across speech ICL, fine-tuning, controllable TTS, voice conversion, and speech editing.
Takeaways & Limitations
Seed-TTS remains limited in nuanced emotion and contextual understanding, scenario coverage, singing, and prompts containing background music or excessive noise.
Abstract
from arXiv · showhide
We introduce Seed-TTS, a family of large-scale autoregressive text-to-speech (TTS) models capable of generating speech that is virtually indistinguishable from human speech. Seed-TTS serves as a foundation model for speech generation and excels in speech in-context learning, achieving performance in speaker similarity and naturalness that matches ground truth human speech in both objective and subjective evaluations. With fine-tuning, we achieve even higher subjective scores across these metrics. Seed-TTS offers superior controllability over various speech attributes such as emotion and is capable of generating highly expressive and diverse speech for speakers in the wild. Furthermore, we propose a self-distillation method for speech factorization, as well as a reinforcement learning approach to enhance model robustness, speaker similarity, and controllability. We additionally present a non-autoregressive (NAR) variant of the Seed-TTS model, named $\text{Seed-TTS}_\text{DiT}$, which utilizes a fully diffusion-based architecture. Unlike previous NAR-based TTS systems, $\text{Seed-TTS}_\text{DiT}$ does not depend on pre-estimated phoneme durations and performs speech generation through end-to-end processing. We demonstrate that this variant achieves comparable performance to the language model-based variant and showcase its effectiveness in speech editing. We encourage readers to listen to demos at \url{https://bytedancespeech.github.io/seedtts_tech_report}.
1 Introduction
Seed-TTS is introduced as a family of human-like, expressive speech generation models targeting arbitrary speakers with little data. The paper adds self-distillation, reinforcement learning, and a fully diffusion-based variant to improve controllability, robustness, and applicability.
- Scope and impact: The paper reports highly expressive, human-like speech and discusses applications, limitations, deployment behavior, and socially responsible AI challenges.Its stated application scope includes virtual assistants, audiobooks, and video dubbing.
- Motivation: Seed-TTS targets human-level speech generation for arbitrary speakers in the wild with little data, evaluated on zero-shot ICL, speaker fine-tuning, and emotion control.The authors also release the test-dataset configuration for future benchmarking.
- Model extensions: Self-distillation enables speech factorization and high-quality timbre disentanglement without changing the model structure or loss function.The method is presented as an extension intended to enhance controllability.
- Model extensions: Reinforcement learning improves robustness, speaker similarity, and controllability through preference biasing.The paper describes RL as a post-training extension that holistically improves model performance.
- Diffusion variant: Seed-TTS_DiT directly predicts speech latent representations end to end without separate duration prediction and performs comparably to autoregressive language model-based methods.The variant is also demonstrated for speech editing.
2 Method
Seed-TTS combines tokenization, autoregressive token generation, diffusion refinement, and vocoding in a large-scale transformer-based speech-generation system. Its design targets broad generalization, controllability, expressive quality, and robustness across multiple speech tasks.
- Architecture: Seed-TTS uses four main components: a speech tokenizer, token language model, token diffusion model, and acoustic vocoder.The system is trained on substantially more data than previous largest TTS systems.
- Architecture: The inference pipeline tokenizes reference speech, autoregressively generates speech tokens from text and speech conditions, refines representations with diffusion, and produces the waveform with a vocoder.The diffusion stage enhances acoustic details before waveform synthesis.
- Training: Training proceeds through pre-training, speaker and instruction fine-tuning, and reinforcement-learning post-training.These stages respectively target coverage and general modeling, selected-speaker performance and controllability, and holistic improvement.
- Reported behavior: Seed-TTS demonstrates superior naturalness and expressiveness across difficult scenarios, including shouting, crying, and highly emotional speech.The authors report clear advantages over prior state-of-the-art systems in these tests.
- Reported behavior: Seed-TTS achieves better robustness across test sets through token and model design, training and inference strategies, data augmentation, and reinforcement post-training.The paper frames stability as addressing deployment issues in language model-based TTS systems.
- Applications: The report demonstrates speech ICL, speaker fine-tuning, controllable TTS, and voice conversion as foundation-model tasks.Zero-shot ICL generates a new utterance matching a short reference clip’s timbre and prosody.
3 Experiments
Seed-TTS is evaluated on objective and subjective speech-generation tasks spanning zero-shot in-context learning, ASR utility, speaker similarity, and speaker fine-tuning. Results show human-like quality and strong similarity, while exposing limitations for hard speakers, noisy ASR, and long-form prosody.
- Zero-shot in-context learning: Seed-TTS achieves WER similar to ground-truth speech while attaining significantly higher speaker similarity in zero-shot in-context learning.The authors attribute this to reproducing enrollment-clip characteristics more faithfully than ground-truth/reference utterances that differ in style or environment.
- Zero-shot in-context learning: -0.07 and -0.08 CMOS scores for English and Mandarin closely match real human speech in subjective zero-shot evaluation.The diverse test set included expressive speech; prior models scored below -1 in the same evaluation.
- Zero-shot in-context learning: 47.9% of common-speaker test samples favored zero-shot Seed-TTS over traditional speaker-fine-tuned systems trained with roughly 5 hours per speaker.Traditional fine-tuned models performed better for hard speakers with strong accents or highly distinctive speaking styles.
- Speech understanding evaluation: 1.81% and 1.6% absolute WER drops occurred on noisy dev_other and test_other sets when ASR was trained on synthetic rather than real speech.Clean-set ASR performance was very similar, while reduced background noise in generated speech may lower noise robustness.
- Speaker similarity: Ground-truth and synthesized embeddings from the same speaker reliably cluster together, supporting close speaker-similarity preservation.The visualization uses WavLM-based speaker embeddings for 25 VoxCeleb1 speakers and t-SNE projection.
- Speaker fine-tuning: +0.37 CMOS favors speaker-fine-tuned Seed-TTS over the base zero-shot model in subjective evaluation, despite similar objective metrics.Fine-tuning captures additional target-speaker nuances, including subtle prosody changes and distinctive end-of-sentence pronunciation.
- Emotion control: Additional controlling signals significantly improve emotion-control accuracy over speaker fine-tuning without explicit control.The evaluation covered angry, happy, sad, and surprised emotions using 100 utterances per emotion.
4 Model extensions
Seed-TTS is extended with self-distillation for timbre disentanglement, reinforcement learning for broader capability improvements, and a fully diffusion-based variant for end-to-end speech generation. These extensions improve voice conversion, robustness, speaker similarity, emotion control, and duration-conditioned speech generation.
- 4.1 Speech factorization by self-distillation: Self-distillation creates controlled speech pairs differing in selected attributes, enabling high-quality attribute disentanglement with minor architectural changes.The report highlights timbre disentanglement, using pairs that share content and prosody while differing in timbre.
- 4.1 Speech factorization by self-distillation: Speaker perturbation produces speech with shared content and prosody but shifted timbres, supplying training pairs for disentanglement.The diffusion model is retrained to recover the original speaker’s vocoder embeddings from the altered-timbre input and reference.
- 4.1 Speech factorization by self-distillation: Self-distillation significantly improves SIM in zero-shot voice conversion and surpasses pre-existing methods in all other reported dimensions.The evaluation uses a test set with a randomly selected non-matching speaker as the timbre reference.
- 4.2 Preference biasing through reinforcement learning: Reinforcement learning improves stability and speaker similarity in voice in-context learning and increases emotion-controlled accuracy across emotions relative to Seed-TTSICL.REINFORCE variants use SIM and WER rewards for similarity and robustness, or SER accuracy for emotion controllability.
- 4.3 Fully diffusion-based speech generation: Seed-TTSDiT directly converts Gaussian noise to vocoder latent representations from text, removing dependence on a separate duration prediction module and acoustic tokenizer.The fully diffusion-based model is designed for end-to-end non-autoregressive speech generation.
- 4.3 Fully diffusion-based speech generation: Seed-TTSDiT achieves superior SIM and similar WER to Seed-TTSICL on zero-shot TTS, while duration conditioning enables natural timing adjustments.The model can insert silence or stretch vowel pronunciation rather than uniformly changing sentence speaking rate.
5 Model applications, limitations, and safety
Seed-TTS supports expressive, cross-lingual speech applications including voice chats, audiobooks, and content creation. Its limitations include weak performance for nuanced emotion, incomplete scenario coverage, singing, and prompts with background music or excessive noise, while safety procedures address misuse risks.
- Applications: Seed-TTS enables expressive and cross-lingual speech applications such as voice chats, audiobooks, and content creation.The paper also describes potential accessibility benefits across language barriers and for patients with speech impairments.
- Limitations: The model sometimes struggles with nuanced emotion and contextual understanding, and its scenario coverage still has room for improvement.These limitations remain despite training on a vast amount of data.
- Limitations: Seed-TTS performs poorly for singing and prompts containing background music or excessive noise, often generating inconsistent backgrounds.For example, it may ignore music altogether.
- Safety: Related products use multi-step voice verification and multi-level watermarking to reduce misuse risks during development and potential deployment.Verification checks enrollment audio for an authorized user’s voice and timbre.
6 Authors (alphabetical order)
The paper’s authors are listed in alphabetical order.
- Authors (alphabetical order): The author list includes Philip Anastassiou, Jiawei Chen, Jitong Chen, and Yuanzhe Chen.
- Authors (alphabetical order): The author list continues with Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, and Yuchen Liu.
- Authors (alphabetical order): The author list includes Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Qingqing Huang, Zhiying Huang, and Yuanyuan Huo.
- Authors (alphabetical order): The author list concludes with Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, and Yuping Wang.