PaperlayerView exact revision on arXiv

Source-linked AI summary

Seed-TTS: A Family of High-Quality Versatile Speech Generation Models

Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, Xiaobin Zhuang

arXiv:2406.02430v1eess.AScs.SD

TL;DR

Seed-TTS targets speech generation approaching human-level quality for arbitrary in-the-wild speakers with little data. It introduces autoregressive and diffusion-based models, with subjective zero-shot ICL tests reporting CMOS scores of -0.07 for English and -0.08 for Mandarin, closely matching real human speech.

  • Problem

    The central goal is a speech-generation model approaching human-level speech for arbitrary in-the-wild speakers with little data.

    Source: §1 · p. 1

  • Method

    Seed-TTS combines a large-scale autoregressive model with a fully diffusion-based non-autoregressive variant, and adds self-distillation for attribute disentanglement and reinforcement-learning post-training.

    Source: §2 · p. 2, §1 · p. 2, §4 · p. 7

  • Results

    CMOS scores of -0.07 (English) and -0.08 (Mandarin) show Seed-TTS closely matched real human speech in subjective zero-shot ICL tests.

    Source: §3.1 · p. 4

  • Takeaways & Limitations

    The model series has potential applications in voice chats, audiobooks, and content creation.

    Source: §5 · p. 12

    Seed-TTS can struggle with nuanced emotion or contextual understanding, has incomplete scenario coverage, and performs poorly on singing or prompts with background music or excessive noise.

    Source: §5 · p. 12

Abstract

from arXiv · show

We introduce Seed-TTS, a family of large-scale autoregressive text-to-speech (TTS) models capable of generating speech that is virtually indistinguishable from human speech. Seed-TTS serves as a foundation model for speech generation and excels in speech in-context learning, achieving performance in speaker similarity and naturalness that matches ground truth human speech in both objective and subjective evaluations. With fine-tuning, we achieve even higher subjective scores across these metrics. Seed-TTS offers superior controllability over various speech attributes such as emotion and is capable of generating highly expressive and diverse speech for speakers in the wild. Furthermore, we propose a self-distillation method for speech factorization, as well as a reinforcement learning approach to enhance model robustness, speaker similarity, and controllability. We additionally present a non-autoregressive (NAR) variant of the Seed-TTS model, named $\text{Seed-TTS}_\text{DiT}$, which utilizes a fully diffusion-based architecture. Unlike previous NAR-based TTS systems, $\text{Seed-TTS}_\text{DiT}$ does not depend on pre-estimated phoneme durations and performs speech generation through end-to-end processing. We demonstrate that this variant achieves comparable performance to the language model-based variant and showcase its effectiveness in speech editing. We encourage readers to listen to demos at \url{https://bytedancespeech.github.io/seedtts_tech_report}.

1 Introduction

Seed-TTS is a family of speech generation models designed for human-like, expressive, controllable speech, including zero-shot generation from a short enrollment clip. The paper also introduces self-distillation, reinforcement learning, and a fully diffusion-based variant, and discusses deployment, limitations, and societal impact.

1 Introduction

  • Seed-TTS targets human-level naturalness and expressiveness, generating controllable, high-fidelity speech from a short enrollment clip in a zero-shot manner.The paper evaluates it on zero-shot speech in-context learning, speaker fine-tuning, and emotion control.Source: §1 · p. 1
  • The paper releases its test-dataset configuration for future benchmarking and discusses model behavior relevant to product deployment.It also considers potential applications, limitations, development challenges, and socially responsible AI concerns.Source: §1 · pp. 1–2
  • Self-distillation enables Seed-TTS to disentangle timbre without changing its model structure or loss, and the paper reports state-of-the-art voice conversion performance.The method is presented as an alternative to feature engineering, specialized losses, or model-design changes.Source: §1 · pp. 1–2
  • Reinforcement-learning post-training improves Seed-TTS robustness, speaker similarity, and controllability.The paper describes this extension as holistically improving model performance.Source: §1 · p. 2
  • Seed-TTSDiT predicts speech latents end to end without a separate duration module, performs comparably to autoregressive methods, and supports speech editing.The paper describes it as a fully diffusion-based, non-autoregressive variant and claims superior generation quality.Source: §1 · p. 2

2 Method

Seed-TTS is a large-scale autoregressive transformer-based speech-generation model built from a speech tokenizer, token language model, token diffusion model, and acoustic vocoder. Its training combines pre-training, speaker and instruction fine-tuning, and reinforcement-learning post-training.

2 Method

  • Seed-TTS generates speech tokens autoregressively, refines them into continuous acoustic representations with a diffusion model, then produces the waveform through a vocoder.The language model is trained on paired text and speech-token sequences, with the text loss masked for this report’s speech-generation focus; the tokenizer design is crucial to system performance.Source: Figure 1 · p. 2, §2 · pp. 2–3
  • Training proceeds through pre-training for broad scenario and speaker coverage, speaker and instruction fine-tuning, and reinforcement-learning post-training.Speaker fine-tuning targets selected speakers, while instruction fine-tuning improves controllability and interactivity; pre-training uses data and model scale orders of magnitude larger than previous speech-generation models.Source: §2 · p. 3
  • Seed-TTS demonstrates natural, expressive synthesis in challenging emotional scenarios and improves robustness across test sets relative to prior systems.The report attributes stability to token and model design, training and inference strategies, data augmentation, and reinforcement post-training.Source: §2 · p. 3
  • The foundation model supports speech in-context learning, controllable and cross-lingual TTS, voice conversion, timbre generation, and speaking-style transfer.The report demonstrates speech ICL (zero-shot voice continuation), speaker fine-tuning, controllable TTS, and voice conversion; ICL continues audio and text prompts to produce a novel utterance matching a short reference clip’s timbre and prosody [Wang et al., 2018, 2023b, Zalán et al., 2022].Source: §2 · p. 3

3 Experiments

Across experiments, zero-shot in-context Seed-TTS approached human speech quality, while speaker fine-tuning improved subjective ratings and instruction signals strengthened emotion control. The deployed system retained performance comparable to the offline model while reducing latency, computation, and memory use.

3.1 Zero-shot in-context learning

  • Lower WER did not necessarily mean better subjective similarity: more standardized, less accented outputs could lose naturalness and speaker-specific expressiveness.This tradeoff was especially noted when prompts contained strong accents or high expressiveness.Source: §3.1 · p. 4
  • English and Mandarin CMOS scores were -0.07 and -0.08, respectively; Seed-TTS had human-like WER, higher speaker similarity, and same-speaker embeddings clustered together.The subjective test set included diverse, expressive speech. The authors attribute the speaker-similarity result to Seed-TTS capturing prompt characteristics despite differences in style or background between reference and ground-truth utterances.Source: §3.1 · pp. 4, 6, Table 1 · p. 4, Figure 3 · p. 6
  • With 15-second prompts, Seed-TTS was preferred on 47.9% of common-speaker samples, while traditional fine-tuned models performed better for hard speakers.The authors suggest that longer prompts and better training-data coverage could help preserve accents and distinctive speaking styles in hard speakers.Source: §3.1 · p. 5, Figure 2 · p. 5
  • ASR trained on synthetic LibriSpeech performed similarly on clean sets to real-data training, but noisy-set WER drops were 1.81% on dev_other and 1.6% on test_other.The authors attribute the noisy-set drops to Seed-TTS reducing background noise and suggest synthetic speech could support speech-understanding models, potentially with data enhancement to reduce the gap.Source: §3.1 · pp. 5–6, Table 2 · p. 5

3.2 Speaker fine-tuning

  • Speaker fine-tuning preserved similar objective metrics to the base model while achieving a +0.37 CMOS advantage and capturing more target-speaker prosodic and pronunciation nuances.The experiment fine-tuned on 20 hours combined from five speakers.Source: §3.2 · p. 6, Table 3 · p. 6
  • Instruction fine-tuning improved emotion-control accuracy over the moderate accuracy achieved without explicit control signals, with text content potentially helping infer emotion.The evaluation covered angry, happy, sad, and surprised speech, using 100 utterances per emotion.Source: §3.2 · pp. 6–7, Table 4 · p. 7

3.3 Low-latency inference and streaming processing

  • Streaming and inference optimizations yielded performance comparable to the offline model in subjective and objective tests while significantly reducing latency, computation, and memory consumption.The methods included causal diffusion for streaming, consistency distillation and modified flow matching, plus language-model techniques such as grouped-query attention, paged attention, flash attention, and quantization.Source: §3.3 · p. 7, Table 5 · p. 7

4 Model extensions

The section extends Seed-TTS with self-distillation for timbre disentanglement, reinforcement learning for speech attributes, and a fully diffusion-based model designed for end-to-end generation and editing.

4.1 Speech factorization by self-distillation

  • Self-distillation uses controlled speech pairs with shared content and prosody but shifted timbre, enabling high-quality disentanglement and improved SIM over prior methods.The diffusion model is retrained to recover the original speech using its timbre reference while disregarding the altered-timbre token; the results also surpass prior methods in other dimensions.Source: §4.1 · p. 8, Table 6 · p. 9

4.2 Preference biasing through reinforcement learning

  • REINFORCE improves voice-ICL stability and speaker similarity, while SER-rewarded training improves emotion accuracy over Seed-TTSICL despite lower controllability than speaker-fine-tuned Seed-TTSSFT.The paper reports subjective and objective improvements for SIM-WER rewards and improved emotion-controlled accuracy across emotions for SER rewards.Source: §4.2 · p. 9, Table 7 · p. 9, Table 8 · p. 9
  • Reward optimization can cause reward hacking: lowering WER may slow and over-articulate speech at the expense of naturalness, so network tuning must balance the trade-offs.The authors compare external-reward methods such as PPO and REINFORCE with DPO, noting that both approaches are effective, with external rewards offering clearer attribute control and DPO simpler implementation.Source: §4.2 · p. 9

4.3 Fully diffusion-based speech generation

  • Seed-TTSDiT directly generates vocoder latents from noise and text, predicts total duration rather than phoneme durations, and supports content editing through non-streaming generation.Providing total duration also enables speaking-rate editing; language-model approaches retain advantages in streaming and integration with text-based language models.Source: §4.3 · p. 10, Figure 5 · p. 10
  • Seed-TTSDiT achieves superior SIM and similar WER to Seed-TTSICL on zero-shot TTS, while its table caption also reports improved stability and speaker similarity.The prose reports the SIM and WER comparison; the table caption describes stability and speaker similarity without supplying cell values.Source: §4.3 · p. 10, Table 10 · p. 11
  • Seed-TTSDiT adjusts speaking rate by inserting silence or stretching vowels according to the target duration, maintaining naturalness and speaker similarity better than uniform rate alteration.The speaking-rate evaluation re-synthesizes examples using durations scaled from the original utterance; Figure 7 notes slight WER degradation at excessively high speed rates.Source: §4.3 · p. 11, Figure 7 · p. 11

5 Model applications, limitations, and safety

Seed-TTS enables expressive, cross-lingual applications, but remains limited in nuanced emotion, contextual understanding, and some noisy or musical scenarios. The paper describes safety procedures to reduce misuse.

Applications

  • Seed-TTS supports voice chats, audiobooks, and content creation, and may improve accessibility across language barriers for people with speech impairments.The paper also describes Seed-TTS as a potential bridge between speech understanding and generation models.Source: §5 · p. 12

Limitations

  • Seed-TTS can struggle with nuanced emotion, contextual understanding, singing, and prompts with background music or excessive noise, sometimes producing inconsistent backgrounds.The paper notes that scenario coverage still has room for improvement despite training on a vast amount of data.Source: §5 · p. 12

Safety

  • To mitigate misuse, the authors verify that enrollment audio contains only authorized users’ voices and require multi-level watermarks in generated content.Watermarks may appear in video backgrounds and content descriptions.Source: §5 · p. 12

Related papers

Loading 2406.02430v1…