PaperlayerView exact revision on arXiv

Source-linked AI summary

CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens

Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, Zhijie Yan

arXiv:2407.05407v2cs.SDcs.AIeess.AS

TL;DR

LLM-based TTS requires speech tokens that capture semantic information and align with text, but existing tokens are learned unsupervised. The paper introduces supervised semantic tokens and CosyVoice, combining an LLM with conditional flow matching; experiments report improved content consistency, speaker similarity, and synthesis quality with large-scale data.

  • Problem

    Existing unsupervised speech tokens may lack explicit semantic information and alignment with text, leaving their impact on TTS unclear.

  • Method

    The paper derives supervised semantic tokens from a multilingual ASR model and uses them in CosyVoice, an LLM and conditional flow matching TTS system.

  • Results

    Supervised semantic tokens outperform unsupervised counterparts for content consistency and speaker similarity, while scaling model size and data improves performance to human-parity generation quality.

  • Takeaways & Limitations

    CosyVoice supports scalable multilingual zero-shot speech generation with cross-lingual voice cloning, instructed generation, and fine-grained control of emotion and paralinguistic features.

Abstract

from arXiv · show

Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.

1 Introduction

LLM-based TTS depends critically on speech tokens, but unsupervised tokens may lack explicit semantics and text alignment. CosyVoice addresses this gap with supervised semantic tokens and a scalable zero-shot synthesis architecture.

  • 1 Introduction: LLM-based TTS models condition token-sequence generation on text and reconstruct waveforms with a token vocoder.Speech signals are converted into token sequences before waveform reconstruction.
  • 1 Introduction: Unsupervised speech tokens may lack explicit semantic information and alignment with corresponding text.This limitation motivates supervised token representations for TTS.
  • 1 Introduction: CosyVoice derives supervised semantic tokens from a multilingual speech-recognition model by inserting vector quantization into its encoder.The approach is intended to improve semantic representation and text alignment.
  • 1 Introduction: CosyVoice combines an LLM for text-to-token generation with conditional flow matching for token-to-speech synthesis.The system is presented as a scalable and efficient zero-shot TTS synthesizer.
  • 1 Introduction: The system avoids additional phonemizers and forced aligners, providing a more direct pathway from text to speech.This design contrasts with flow-matching TTS systems that rely on phoneme duration prediction and supplementary alignment tools.
  • 1 Introduction: Experiments report that supervised semantic tokens outperform unsupervised counterparts, while large-scale data improves CosyVoice synthesis performance.These findings support the system’s scalable capacity and zero-shot voice-cloning performance.

2 CosyVoice: A Scalable TTS model using Supervised Semantic Tokens

CosyVoice combines supervised semantic speech tokens with autoregressive text-to-token generation and conditional flow matching for token-to-speech synthesis. Its design covers token extraction, sequence construction, zero-shot prompting, and controllable generation.

  • CosyVoice architecture: CosyVoice uses a text encoder, speech tokenizer, LLM, and conditional flow matching model to generate speech from text and tokens.The text encoder aligns text and speech-token semantic spaces; the LLM generates speech tokens autoregressively, and flow matching converts them into a Mel spectrogram.
  • Supervised semantic tokens: The S3 tokenizer inserts vector quantization between two encoder parts of a supervised multilingual ASR model to produce discrete semantic speech tokens.Quantized hidden representations are passed through the remaining encoder layers and ASR decoder, while codebook embeddings are updated with EMA during training.
  • Zero-shot generation: Zero-shot in-context learning constructs prompt-dependent sequences, while cross-lingual cloning omits prompt text and tokens when prompt and target languages differ.Generated tokens are combined with prompt tokens, speaker embeddings, and prompt Mel spectrograms as conditions for flow-matching synthesis.
  • Large language model for TTS: The LLM formulates TTS as autoregressive speech-token generation, conditioning on text encodings, speaker information, and supervised speech tokens.A turn-of-speech token separates text encodings from speech tokens, and training uses teacher forcing with cross-entropy losses for speech tokens and the end token.
  • Conditional flow matching: The token-to-speech module uses optimal-transport conditional flow matching conditioned on speaker embeddings, speech tokens, masked Mel features, and timestep state.The model learns a time-dependent vector field; masked Mel spectrograms zero continuous frames from a random start point to the end, and a cosine scheduler places more generation steps at the beginning.
  • Controllable generation: CosyVoice-instruct adds control over speaker identity, speaking style, and fine-grained paralinguistic features such as laughter, breaths, and emphasis.The system supports controls including emotion, gender, speaking rate, and pitch.

3 Dataset

The experiments use both a 585-hour English LibriTTS corpus and a large multilingual training dataset. The multilingual data are processed with in-house speech-quality and speaker-processing tools, then refined pseudo-text labels support training-data analysis by language and instruction type.

  • 3.1 Single-lingual Dataset: 585 hours from 2,456 English speakers are used in the LibriTTS corpus, following the official training, development, and test partitioning.The training split merges train-clean-100, train-clean-360, and train-other-500; dev-clean supports model selection, while test-clean forms the evaluation set.
  • 3.2 Large-scale Multi-lingual Dataset: The large-scale dataset spans multiple languages and is processed with in-house tools for speech detection, SNR estimation, speaker diarization, and separation.These tools are applied during collection to prepare the multilingual training data.
  • 3.2 Large-scale Multi-lingual Dataset: Table 2 reports CosyVoice training-data hours across languages in the large-scale experiments.The table provides the language-level duration breakdown for the multilingual dataset.
  • 3.2 Large-scale Multi-lingual Dataset: Table 3 reports duration statistics for instruction training data by type.This organizes the instruction data according to its type and duration.
  • 3.2 Large-scale Multi-lingual Dataset: Pseudo-text labels are generated with SenseVoice-Large and Paraformer, then refined using force-alignment models to remove low-quality data and improve punctuation accuracy.The resulting statistics are organized by language and instruction type in Tables 2 and 3.

4 Experimental Settings

The supervised semantic speech tokenizer uses ASR backbones with a vector quantizer inserted after the first six encoder layers. Experiments compare tiny and normal models across single-lingual and multilingual settings, with distinct training resources and optimization schedules.

  • 4.1 Supervised Semantic Speech Tokenizer: The small-scale single-lingual tokenizer uses an ESPNet Conformer ASR backbone with a vector quantizer after the first six encoder layers.It uses one 4,096-code codebook, and the first six encoder layers plus the quantizer form the speech tokenizer.
  • 4.1 Supervised Semantic Speech Tokenizer: The text tokenizer is a sentence-piece model with a 4,000-word vocabulary, and the quantizer-augmented ASR model is trained on Librispeech for 50 epochs from scratch.This setting applies to the small-scale single-lingual dataset.
  • 4.1 Supervised Semantic Speech Tokenizer: The large-scale multilingual tokenizer uses SenseVoice-Large and inserts the same single 4,096-code quantizer after the first six encoder layers.The SenseVoice-Large backbone is initialized from a pre-trained checkpoint; further quantizer-layer and code-count choices are left for future work.
  • 4.2 Model Training: Tiny and normal CosyVoice models are trained in both single-lingual and multilingual experiments, with architecture details provided in Table 4.The table specifies the model architecture settings for these two model sizes.
  • 4.2 Model Training: The tiny model is trained on LibriTTS for 50 epochs using four V100-32M GPUs, while the multilingual model is trained for 800,000 steps on an internal dataset using 64 V100-32M GPUs.Tiny and normal models use learning rates of 10^-3 and 10^-4, respectively, with 10,000 warmup steps.

5 Experimental Results

Experiments evaluate S3 tokens and CosyVoice on recognition preservation, multilingual TTS quality, emotion control, and synthetic-data augmentation. Results show that supervised tokens preserve semantic information, CosyVoice achieves strong content consistency and speaker similarity, and larger or more diverse data improves performance.

  • 5.1 Evaluation on S3 Tokenizer: 3.18% and 7.56% WER show that inserting vector quantization only slightly affects recognition on LibriTTS test-clean and test-other.The VQ-inserted Conformer ASR model retains sufficient semantic information and text alignment.
  • 5.1 Evaluation on S3 Tokenizer: 4.14% relative error-rate reduction shows that S3 tokens surpass Whisper-Large V3 on Common Voice zh-CN.S3 tokens also demonstrate robust recognition on both Chinese and English benchmarks using a single 4,096-entry codebook.
  • 5.2 Comparison with Baselines: CosyVoice achieves comparable content consistency and higher speaker similarity than other TTS models, while tokenizer choices chiefly affect content consistency.Replacing single-lingual tokenizers with multilingual ones degrades performance when training uses only LibriTTS, but large-scale internal data restores and significantly improves both measures.
  • 5.3 Evaluation on Generation Quality of CosyVoice: 1.51% WER is achieved on English after ASR re-ranking, with human-level content recognition and higher speaker similarity than the original utterances.CosyVoice also outperforms ChatTTS in WER, insertion errors, and deletion errors on English.
  • 5.3 Evaluation on Generation Quality of CosyVoice: 1.84% CER is achieved on Chinese after ASR re-ranking, while CosyVoice avoids ChatTTS’s speaker-leaking insertion and deletion errors and shows greater speaker similarity.Without re-ranking, CosyVoice has comparable CER and fewer insertion and deletion errors than ChatTTS.
  • 5.4 Emotion Controllability: Emotional instructions significantly improve CosyVoice-instruct’s emotion-control accuracy over CosyVoice-base and instruction-free CosyVoice-instruct.The evaluation covers six emotions across 100 English utterances per emotion.
  • 5.5 CosyVoice as a Data Generator: Synthesized data alone achieves comparable ASR results to the original 960-hour Librispeech training set, and combining datasets further improves recognition accuracy.Synthetic data using MLS text produces larger gains, suggesting varied linguistic content is more important than speech duration for this ASR experiment.

6 Conclusion

CosyVoice is a scalable multilingual speech-generation model supporting zero-shot in-context learning, cross-lingual voice cloning, instructed generation, and fine-grained control. Experiments show that architecture matters for speaker similarity, tokenizers matter for content consistency, and scaling model size and data improves performance to human-parity quality.

  • 6 Conclusion: CosyVoice supports zero-shot in-context learning, cross-lingual voice cloning, instructed generation, and fine-grained control of emotion and paralinguistic features.The conclusion presents these capabilities as properties of the introduced multilingual speech-generation model.
  • 6 Conclusion: The system architecture is important for speaker similarity, whereas text and speech tokenizers affect content consistency much more.This conclusion summarizes the reported experimental findings.
  • 6 Conclusion: Scaling up model size and data volume significantly improves performance, and CosyVoice achieves human-parity generation quality.The conclusion links scalability with the reported overall quality improvement.
Loading 2407.05407v2…