Source-linked AI summary
CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang, Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, Shiliang Zhang, Wen Wang, Jieping Ye
TL;DR
CosyVoice 3 addresses CosyVoice 2’s limited coverage and underexplored scaling and post-training strategies for in-the-wild zero-shot speech synthesis. It introduces a multi-task speech tokenizer, differentiable reward optimization, expanded data and model scaling, and the CV3-Eval benchmark. The model achieves state-of-the-art results across multiple benchmarks while retaining limitations in textual timbre control and singing-voice generation.
Problem
CosyVoice 2 has limited language coverage, domain diversity, data volume, and text-format variety, while speech-generation scaling laws and suitable post-training techniques remain insufficiently explored.
Method
CosyVoice 3 combines a multi-task-trained speech tokenizer, differentiable reward optimization, scaled data and model capacity, and the CV3-Eval benchmark for in-the-wild zero-shot synthesis.
Results
CosyVoice 3 achieves state-of-the-art results on multiple benchmarks, with reinforcement-learning post-training contributing 12% to 35% relative improvements in content consistency.
Takeaways & Limitations
The reported improvements advance zero-shot multilingual speech synthesis toward more diverse real-world languages, scenarios, and speech characteristics.
Takeaways & Limitations
CosyVoice 3 cannot control timbre through textual instructions and does not perform quite well when generating singing voice.
Abstract
from arXiv · showhide
In our prior works, we introduced a scalable streaming speech synthesis model, CosyVoice 2, which integrates a large language model (LLM) and a chunk-aware flow matching (FM) model, and achieves low-latency bi-streaming speech synthesis and human-parity quality. Despite these advancements, CosyVoice 2 exhibits limitations in language coverage, domain diversity, data volume, text formats, and post-training techniques. In this paper, we present CosyVoice 3, an improved model designed for zero-shot multilingual speech synthesis in the wild, surpassing its predecessor in content consistency, speaker similarity, and prosody naturalness. Key features of CosyVoice 3 include: 1) A novel speech tokenizer to improve prosody naturalness, developed via supervised multi-task training, including automatic speech recognition, speech emotion recognition, language identification, audio event detection, and speaker analysis. 2) A new differentiable reward model for post-training applicable not only to CosyVoice 3 but also to other LLM-based speech synthesis models. 3) Dataset Size Scaling: Training data is expanded from ten thousand hours to one million hours, encompassing 9 languages and 18 Chinese dialects across various domains and text formats. 4) Model Size Scaling: Model parameters are increased from 0.5 billion to 1.5 billion, resulting in enhanced performance on our multilingual benchmark due to the larger model capacity. These advancements contribute significantly to the progress of speech synthesis in the wild. We encourage readers to listen to the demo at https://funaudiollm.github.io/cosyvoice3.
1 Introduction
CosyVoice 3 targets the language, domain, data, text-format, and post-training limitations of CosyVoice 2 for in-the-wild zero-shot speech generation. It combines a new speech tokenizer, differentiable reward optimization, scaled training resources, and a broad evaluation benchmark, achieving state-of-the-art results on multiple benchmarks.
- CosyVoice 2 performs well in general Chinese and English broadcast scenarios but remains limited in language coverage, domain diversity, data volume, text formats, and post-training techniques.
- The new speech tokenizer uses supervised multi-task training to capture paralinguistic information such as emotion and pronunciation style.
- Differentiable reward optimization provides a post-training strategy applicable to CosyVoice and other discrete-token-based speech synthesis models.
- Training data expands from ten thousand hours to one million hours across 9 languages, 18 Chinese accents or dialects, diverse domains, and text formats.
- CV3-Eval evaluates zero-shot speech synthesis using in-the-wild speech spanning languages, dialects, domains, environments, emotions, and styles.
- CosyVoice 3 achieves state-of-the-art results on multiple benchmarks.
2 CosyVoice 3
CosyVoice 3 extends zero-shot speech generation with a supervised multi-task tokenizer, differentiable reward optimization, broader controllability, and expanded multilingual support.
- 2 CosyVoice 3: The model’s training pipeline includes large-scale pretraining, post-training, continual pretraining, and multi-speaker fine-tuning.Figure 2 distinguishes modules used only during training and identifies CFM as the conditional flow matching model.
- 2.1 Speech Tokenizer via Supervised Multi-task Training: A supervised multi-task tokenizer uses ASR, LID, SER, AED, and speaker analysis to capture speech information.It inserts FSQ into MinMo’s voice encoder and predicts corresponding text tokens during training.
- 2.2 Reinforcement Learning with Differentiable Reward Optimization: DiffRO directly optimizes speech tokens using a Token2Text posterior reward, Gumbel-Softmax sampling, back-propagation, and token-level KL regularization.Multi-task rewards additionally use SER, MOS prediction, AED, and other audio-understanding tasks to control voice attributes through instructions.
- 2.4 Pronunciation Control: LLM-based TTS pronunciation control is extended by modeling mixed word-and-phoneme sequences with an expanded tokenizer vocabulary and auxiliary pronunciation data.Chinese characters are replaced with pinyin and English words with phonemes from the CMU pronunciation dictionary.
- 2.3 Text Normalization: Raw text can be synthesized directly after augmenting training with rule-based and LLM-generated text-normalization examples.The extended system shows better robustness and coverage for various special symbols.
- 2.5 Instructed Speech Generation: CosyVoice 3 broadens expressive and multilingual control through more instruction-following data and speaker-language instructions.High-quality instruction-following data grows from 1,500 to 5,000 hours, while multilingual data specifies speaker and language IDs in natural-language instructions.
3 The Multilingual Data Pipeline
The multilingual data pipeline converts heterogeneous in-the-wild audio into filtered training pairs through segmentation, denoising, transcription, punctuation adjustment, normalization, and ratio-based filtering.
- Data Collection and Processing: In-the-wild multilingual audio is collected mainly from Internet audiobooks, videos, and podcasts, then processed into model training data.The pipeline has six stages: speech detection and segmentation, noise reduction, ASR transcription, punctuation adjustment, volume standardization, and filtering.
- Speech Detection and Segmentation: Speaker diarization, VAD, and audio event detection produce speaker-level speech segments shorter than 30 seconds.The in-house modules can be replaced by open-source alternatives to the same effect.
- ASR Transcription: Multiple ASR systems are cross-validated, retaining transcriptions whose average pair-wise WER is below 15%.Faster-Whisper Large-V3 first performs language identification, and several open-source ASR models generate candidate transcriptions.
- Punctuation Adjustment: Montreal Forced Aligner adjusts punctuation using pause durations, adding commas at ≥300 milliseconds and removing pause punctuation at ≤50 milliseconds.The adjustment targets commas, semicolons, colons, full stops, question marks, and exclamation marks.
- Volume Standardization: Volume standardization applies a simple normalization before subsequent filtering.The passage identifies normalization as the volume-standardization operation.
- Length-Ratio Filtering: Utterances in the smallest 1% and largest 5% of speech-token-to-text-token length ratios are discarded as possible abnormal cases.The ratios are calculated after speech and text tokens are extracted for each utterance-text pair.
4 Experimental Settings
CosyVoice 3 is evaluated with expanded training data, larger generation models, and benchmarks designed for multilingual and in-the-wild zero-shot synthesis. The settings measure content consistency, speaker similarity, audio quality, and expressive cloning across challenging conditions.
- Training Data: The speech tokenizer uses 530,000 hours of supervised multi-task data spanning ASR, LID, SER, AED, and speaker analysis.The multilingual ASR portion includes Chinese, English, Japanese, Korean, Russian, French, and German.
- Dataset Scaling: CosyVoice 3 expands data diversity through domain collection, text normalization and inverse text normalization, self-training, and seven additional languages.The added languages are Japanese, Russian, French, German, Spanish, Korean, and Italian.
- Model Scaling: The text-to-speech LM grows from 0.5B to 1.5B parameters, while the conditional flow-matching model grows from 100M to 300M.The conditional flow-matching model uses a diffusion transformer backbone.
- Evaluation Benchmarks: Evaluation uses SEED-TTS-Eval and the new CV3-Eval benchmark to assess zero-shot synthesis under multilingual, cross-lingual, emotional, expressive, and accent conditions.SEED-TTS-Eval includes Mandarin, English, and hard Chinese subsets; CV3-Eval targets broader real-world diversity.
- Baselines: The study compares CosyVoice 3 with ten autoregressive and non-autoregressive speech generation baselines.The baselines include MaskGCT, E2 TTS, F5-TTS, F5R-TTS, Seed-TTS, FireRedTTS, Qwen2.5-Omni, CosyVoice, CosyVoice 2, and Spark TTS.
5 Experimental Results
CosyVoice 3 improves objective and subjective zero-shot speech synthesis performance across standard, multilingual, cross-lingual, emotional, and expressive evaluations. Results also expose remaining weaknesses in rare-word generation and a few Chinese subjective cases.
- SEED-TTS-Eval: 44% and 51% relative content-consistency gains over CosyVoice 2 occur on test-zh and test-en, respectively.On test-hard, CER decreases from 6.83% to 5.09%, a 26% relative improvement.
- SEED-TTS-Eval: CosyVoice 3-1.5BRL records 0.71% CER on test-zh and 1.45% WER on test-en, while CosyVoice 3-0.5BRL reaches 5.09% CER on test-hard.The larger variant reaches 5.66% CER on test-hard, below the smaller variant there.
- SEED-TTS-Eval: CosyVoice 3 outperforms CosyVoice 2 and most baselines in speaker similarity, while RL post-training contributes 12% to 35% relative content-consistency improvements.Seed-TTS remains ahead on speaker similarity in the reported comparisons.
- Multilingual Voice Cloning: CosyVoice 3 is the only evaluated system covering all languages in the CV3-Eval multilingual voice-cloning subset.The subset reports CERs for Chinese, Japanese, and Korean and WERs for English, German, Spanish, French, Italian, and Russian.
- Cross-lingual Voice Cloning: CosyVoice 3 improves cross-lingual voice cloning over CosyVoice 2, and the 1.5B model achieves better WERs across conditions with similar speaker similarity.Converting Japanese characters into kana resolves CosyVoice 2’s Japanese-to-Chinese transfer problem.
- Emotion Cloning: CosyVoice 3 achieves the highest performance on text-related emotion subsets, while emotion accuracy drops on text-unrelated tasks.The happy emotion is easiest for all evaluated systems.
- Limitations: Rare words, tongue twisters, and domain-specific terms remain difficult, and a few low-scoring Chinese outputs contribute to the remaining human-speech gap.The authors identify synthesis stability as a future improvement area.
5.3 Ablation of Speech Tokenizer
The tokenizer ablation shows that supervised semantic tokenizers preserve recognition ability and support stronger downstream TTS as training data scales. Gains become smaller when scaling reaches one million hours.
- Upstream Tokenizer Evaluation: FSQ-MinMo maintains multilingual ASR capabilities while outperforming MinMo on the Fluers CN test set.It performs comparably to MinMo on AIR-Bench tasks including language, speaker, emotion, and sound classification.
- Downstream TTS Evaluation: Downstream TTS comparisons evaluate supervised, self-supervised, and unsupervised tokenizers on 3,000-hour and 170,000-hour datasets.The LM and CFM architectures remain unchanged while tokenizers are replaced.
- Downstream TTS Evaluation: Supervised semantic tokenizers achieve speaker similarity comparable to HuBERT and significantly higher than W2v-BERT 2.0 on 3,000 hours.The comparison attributes this pattern to differences in semantic and acoustic information retained by the tokenizers.
- Dataset Scaling: 63% to 75% relative WER/CER improvements result when training data increases from 3,000 to 170,000 hours, especially for English and challenging scenarios.Scaling to one million hours further improves performance, but the improvement rate begins to plateau.
5.4 Ablation of Reinforcement Learning
DiffRO improves TTS recognition performance across CosyVoice 2 and 3, but gains vary by language, model size, and test difficulty, with trade-offs among reward objectives.
- 20%–50% relative WER improvements are achieved across TTS systems, including CosyVoice 2 and CosyVoice 3.Over 50% relative improvement occurs in half of the evaluated conditions.
- 68.7% relative WER improvement is observed for CosyVoice 3-0.5B in Korean.
- RL training reduces the performance gap between the 0.5B and 1.5B models.
- DiffRO slightly reduces speaker similarity across most datasets, while DiffRO-EMO achieves top emotion accuracy across most emotions for CosyVoice3-1.5B.Improved emotion expression can adversely affect pronunciation, exposing a reward-balancing challenge.
- DiffRO improvements in WER, SS, and DNSMOS are less pronounced on hard-sample test sets than on overall test sets.Hard samples include rare words, tongue twisters, and repeated words, which challenge reward models.
5.5 Pronunciation Inpainting
The pronunciation-inpainting evaluation targets difficult Chinese and English polyphonic cases, comparing replacement strategies and measuring correction rate.
- 100% correction rate is achieved by the best pronunciation-inpainting method.The evaluation focuses on Chinese polyphonic characters and English polyphonic words.
- RepAll replaces all Chinese characters and English words as potential candidates, using internal G2P models for phoneme prediction.Its broad coverage introduces mismatches from G2P predictions.
- RepMono replaces only monophonic characters or words, preserving accuracy in the training set.
- CatPhn retains characters alongside phonemes, whereas MixPhn replaces characters solely with phonemes.CatPhn preserves semantic completeness but requires prioritizing phoneme representations.
5.6 Instructed Generation
CosyVoice 3 is evaluated for instructed expressive speech generation on multi-speaker and internal datasets, showing improved style similarity but unresolved evaluation and coverage limitations.
- 3,000 Expresso samples and 3,600 internal samples are used to evaluate instructed generation.The datasets cover speaking styles, emotions, speeds, dialects, accents, and role-playing styles.
- Approximately 11% relative style-similarity improvement is reported for CosyVoice 3 over its predecessor.
- WER is higher for CosyVoice 3 on Expresso but lower on the internal test set.The discrepancy is attributed largely to ASR bias toward standard rather than emotional pronunciations.
- Objective evaluation of content consistency in emotional speech remains challenging.
- Singing is not included in instructed generation, which currently focuses on emotion, speech, and style rather than timbre.Timbre editing through natural language or other modalities remains underexplored.
5.7 Results on Speaker Fine-tuned Models
Speaker fine-tuning benefits from larger and more diverse training data and upgraded speech tokens, reducing error rates and enabling strong multilingual conversion, with Japanese and Korean remaining harder cases.
- Increasing training-data volume and diversity and upgrading speech tokens reduces error rates in speaker-fine-tuned models.The reduction is particularly noticeable on the test-en and test-hard sets.
- Below 4% CER/WER is achieved for Chinese, English, German, Spanish, French, Italian, and Russian when converting a monolingual speaker into a polyglot.The result demonstrates the effectiveness of the continual-training approach.
- 9% character error rate is observed for Japanese.Kanji-to-kana conversion and multiple character pronunciations introduce additional errors.
- Approximately 6% CER is observed for Korean, mainly because available data are limited in volume and quality.The authors plan to extend Korean data in future work.
6 Conclusion
CosyVoice 3 is an advanced zero-shot speech synthesis model for in-the-wild applications, combining scaled data and model parameters with tokenizer and post-training innovations. It achieves state-of-the-art results across multiple benchmarks while improving consistency, speaker similarity, and prosody naturalness.
- CosyVoice 3 is tailored for zero-shot speech synthesis in diverse real-world applications.
- Scaling data and model parameters helps address earlier limitations in language coverage and synthesis quality.
- A novel speech tokenizer and post-training strategies improve capture of intricate paralinguistic details.
- CosyVoice 3 achieves state-of-the-art results across multiple benchmarks, with superior content consistency, speaker similarity, and prosody naturalness.
7 Limitations
CosyVoice 3 remains limited in textual control of acoustic characteristics and in singing-voice generation. The authors identify both areas as targets for future improvement.
- CosyVoice 3 cannot control acoustic characteristics such as timbre through textual instructions.
- CosyVoice 3 does not perform quite well when generating singing voice.
- The authors suggest adding singing data during training of both the tokenizer and language-model stages.