Source-linked AI summary
CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, Jingren Zhou
TL;DR
Zero-shot TTS has largely lacked established streaming solutions, despite latency being important for interactive speech applications. CosyVoice 2 unifies streaming and non-streaming synthesis with tokenizer, language-model, and flow-matching optimizations, achieving human-parity naturalness and virtually lossless streaming quality. It also expands fine-grained instruction control, within stated language and acoustic-control limitations.
Problem
Recent zero-shot TTS models generally operate offline, while diffusion-based TTS and hybrid systems lack well-established streaming solutions.
Method
CosyVoice 2 combines finite scalar quantization, a simplified pre-trained-LLM-backed text-speech model, and chunk-aware causal flow matching in a unified synthesis framework.
Results
CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in streaming mode.
Takeaways & Limitations
A single model supports streaming and non-streaming synthesis while providing more versatile instructed speech generation with fine-grained control.
Takeaways & Limitations
CosyVoice 2 supports only a limited number of languages, can degrade on languages with overlapping character sets, and cannot control timbre through textual instructions.
Abstract
from arXiv · showhide
In our previous work, we introduced CosyVoice, a multilingual speech synthesis model based on supervised discrete speech tokens. By employing progressive semantic decoding with two popular generative models, language models (LMs) and Flow Matching, CosyVoice demonstrated high prosody naturalness, content consistency, and speaker similarity in speech in-context learning. Recently, significant progress has been made in multi-modal large language models (LLMs), where the response latency and real-time factor of speech synthesis play a crucial role in the interactive experience. Therefore, in this report, we present an improved streaming speech synthesis model, CosyVoice 2, which incorporates comprehensive and systematic optimizations. Specifically, we introduce finite-scalar quantization to improve the codebook utilization of speech tokens. For the text-speech LM, we streamline the model architecture to allow direct use of a pre-trained LLM as the backbone. In addition, we develop a chunk-aware causal flow matching model to support various synthesis scenarios, enabling both streaming and non-streaming synthesis within a single model. By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode. We invite readers to listen to the demos at https://funaudiollm.github.io/cosyvoice2.
1 Introduction
CosyVoice 2 addresses the latency gap in zero-shot TTS by unifying streaming and non-streaming synthesis while improving model flexibility and speech quality. Its systematic changes target tokenizer efficiency, LLM integration, controllable synthesis, and streaming quality.
- Motivation: Zero-shot TTS models commonly operate offline, requiring complete text and full-utterance synthesis before returning audio, which raises latency for voice-chat applications.
- Contributions: CosyVoice 2 unifies streaming and non-streaming synthesis through a text-speech language model and chunk-aware causal flow matching model.
- Contributions: Removing the text encoder and speaker embedding allows a pre-trained textual LLM to serve as the backbone while enhancing context understanding.
- Contributions: Finite scalar quantization replaces vector quantization in the speech tokenizer to improve codebook utilization and capture more speech information.
- Contributions: Instruction and zero-shot capabilities are integrated, supporting emotion, accent, role style, and fine-grained speech control in one model.
- Results: CosyVoice 2 achieves human-parity synthesis quality and nearly lossless streaming synthesis, while its unified framework supports both synthesis modes with one model.
2 CosyVoice 2
CosyVoice 2 separates semantic and acoustic speech modeling, using a supervised tokenizer and distinct language-model and flow-matching stages. The design simplifies text processing and supports speaker-conditioned acoustic generation.
- Architecture: CosyVoice 2 separates semantic and acoustic information, with the text-speech LM decoding semantic speech tokens and Flow Matching introducing acoustic details.
- Text tokenizer: Raw text is tokenized directly with a BPE-based tokenizer, removing phoneme-based frontend processing and allowing end-to-end contextual pronunciation learning.
- Speech tokenizer: Finite scalar quantization is inserted into a SenseVoice-Large encoder to quantize intermediate speech representations before subsequent encoder and ASR-decoder modules.
- Speech tokenizer: FSQ projects representations into a low-rank space, quantizes each dimension with bounded rounding, and projects them back to the original dimension.
- Speech tokenizer: Straight-through estimation approximates gradients through the FSQ module and Encoder1 during training.
- Speech tokenizer: The speech tokenizer operates at 25 Hz, producing 25 speech tokens per second.
2.3 Unified Text-Speech Language Model
A single text-speech language model supports both streaming and non-streaming synthesis through different sequence constructions. Streaming interleaves text and speech tokens, while inference variants accommodate ICL and speaker fine-tuning.
- Unified design: The unified text-speech LM differs between streaming and non-streaming modes only in how the input sequence is constructed.
- Non-streaming mode: In non-streaming mode, start, text, turn-of-speech, speech, and end-of-sequence tokens are concatenated sequentially.
- Streaming mode: In streaming mode, every N text tokens is followed by M speech tokens; experiments set N:M to 5:15.
- Unified design: Training on both sequence types enables one model to generate speech in streaming and non-streaming modes.
- In-context learning: ICL uses reference text and speech tokens to imitate accent, prosody, emotion, and style during generation.
- In-context learning: For streaming ICL, prompt and target text are mixed with prompt speech tokens, with filling tokens and manual text padding when needed.
- Speaker fine-tuning: Speaker fine-tuning without a prompt starts from a sequence containing the start token and text, after which the LM generates speech autoregressively.
2.4 Chunk-aware Flow Matching
CosyVoice 2 makes flow matching causal and chunk-aware so one model supports both streaming and offline synthesis across latency–performance settings.
- Acoustic Processing: The acoustic pipeline upsamples speech tokens by two to match 50 Hz Mel features and adds look-ahead context before causal modules.Mel spectrograms use a 24,000 Hz sampling rate; the look-ahead convolution is right-padded.
- Flow Matching: Flow matching reconstructs Mel spectrograms from speech tokens, reference speech, and speaker embeddings using a conditional flow matching model.The model learns an ODE with a causal convolutional Transformer UNet and minimizes L1 loss against the ground-truth ODE.
- Inference Controls: Classifier-free guidance is enabled by training under both conditional and non-conditional situations.The reported CFG strength β is 0.7 and the number of flow estimations is 10.
- Causal Design: The system unfolds ten repeated UNet flow-estimation steps into a deeper causal network for streaming synthesis.This converts multi-step flow estimation into a causal architecture that can operate incrementally.
- Masking Strategies: Four masks cover offline, extremely low-latency, and intermediate chunked synthesis scenarios.Non-causal attends to all condition frames; full-causal attends only to past frames; Chunk-M and Chunk-2M trade latency for future context.
- Unified Training: Chunk-aware training samples the four masks uniformly, allowing one flow matching model to serve different deployment scenarios.Masks with more context also act as teachers for masks with less context through implicit self-distillation.
2.5 Latency Analysis for Streaming Mode
The latency analysis decomposes streaming first-package delay into text-to-speech generation stages and adds preceding LLM text-generation time for voice-chat applications.
- TTS Latency: First-package TTS latency sums LM, Flow Matching, and vocoder computation for M speech tokens.The terms are M · d_lm, M · d_fm, and M · d_voc.
- Motivation: First-package latency is treated as important because it significantly affects user experience in LLM-based voice-chat applications.The analysis accounts for known-to-synthesize text and separates latency contributions from speech-token, Mel, and waveform generation.
- Chat Latency: Chat first-package latency is bounded by LLM text-token generation time plus first-package TTS latency.The formulation is L_Chat ≤ N · d_llm + L_TTS.
2.6 Instructed Generation
CosyVoice 2 integrates instructed data into base training to support natural-language and fine-grained control over generated speech.
- Instruction Data: The instructed training set contains 1,500 hours of data spanning natural-language and fine-grained instructions.Natural-language instructions cover emotion, speaking rate, role-playing, and dialects.
- Instruction Types: Table 1 presents examples of natural-language instructions and fine-grained instructions.
2.7 Multi-Speaker Fine-tuning
Multi-speaker fine-tuning adapts the pretrained model jointly across speakers to improve coverage while reducing risks associated with single-speaker specialization.
- Multi-Speaker Adaptation: Multi-speaker fine-tuning adapts the pretrained model to multiple speakers simultaneously rather than one speaker.Speaker-prompt tags identify the target speaker during training and help avoid timbre confusion.
- Benefits: Joint fine-tuning provides broader prosody and pronunciation coverage and mitigates potential catastrophic forgetting from pretrained models.
2.8 Reinforcement Learning for SFT
CosyVoice 2 uses speaker similarity and recognition WER to define preference signals for DPO-based reinforcement learning, while introducing a differentiated optimization path for the text-speech LM.
- Speaker similarity and recognition WER serve as rewards for improving speaker similarity and pronunciation accuracy during fine-tuning.
- WER and SS distinguish preferred sample xw from rejected sample xl for direct preference optimization of the TTS system.
- Repeated TTS synthesis makes preference-sample construction time-consuming and computation-consuming, requiring four forward operations per training step.
- The method recovers LM-predicted tokens into quantized low-rank representations and reuses the speech tokenizer’s ASR backend to re-predict the input text.
- Because sampling blocks direct optimization, Gumbel softmax sampling makes the process differentiable for optimizing θLM with LASR.
3 Experimental Settings
CosyVoice 2 uses large-scale, multi-source speech data and evaluates synthesis across English, Chinese, Japanese, and Korean settings, including challenging text cases.
- 200,000 hours of speech train the speech tokenizer using normalized transcriptions from open-source ASR, internal industrial, and TTS-generation datasets.
- The speech tokenizer is trained on Chinese and English data but shows zero-shot capability for Japanese and Korean synthesis.
- CosyVoice 2 shares its predecessor’s training data, augmented with pseudo-text labels from Paraformer and SenseVoice and force-alignment filtering.
- Evaluation uses Librispeech test-clean for limited-domain English content consistency and speaker similarity measurements.
- SEED evaluations use Chinese and English CommonVoice samples across varied text domains, reference speeches, and hard cases such as repetition and tongue twisters.
- Japanese evaluation uses 1,000 CommonVoice samples to measure WER, SS, and MOS with shuffled reference and target utterance pairings.
4 Experimental Results
CosyVoice 2 improves speech-token representations and achieves strong multilingual, instructed, baseline-comparison, and streaming results. Ablations attribute gains to pretrained LLM initialization, speaker-embedding removal, FSQ, and unified streaming training.
- 4.1 Evaluations on Speech Tokenizer: FSQ fully utilizes the codebook while maintaining more effective ASR information than VQ.The tokenizer evaluation also examines speaker visualization and speaker-identification training.
- 4.1 Evaluations on Speech Tokenizer: Quantized speech representations are nearly indistinguishable across three speakers, reducing speaker-identity entanglement in the tokenizer.Before quantization, Encoder1 outputs show different distributions among speakers.
- 4.2 Comparison Results with Baselines: CosyVoice 2 surpasses all evaluated baselines across WER, NMOS, and SS on LibriSpeech test-clean, including human utterances on these metrics.The authors describe this as human-parity synthesis quality.
- 4.2 Comparison Results with Baselines: On SEED test-zh, CosyVoice 2 leads open-source models in CER and SS; offline CosyVoice 2 achieves state-of-the-art performance on test-hard.On test-en it ranks fourth in WER and third in SS, while streaming performance is nearly lossless in typical cases.
- 4.3 Modular Ablation Study: Pretrained LLM initialization improves content consistency by 18.46% on test-zh and 15.40% on test-hard, while FSQ further improves content consistency without changing speaker similarity.Removing speaker embeddings reduces content errors while maintaining speaker similarity.
- 4.3 Modular Ablation Study: Streaming modules minimally affect typical test-zh and test-en cases, with their main impact appearing on challenging test-hard samples.The analysis attributes this difference likely to lost contextual information.
- 4.4 Results on Japanese and Korean Benchmarks: Korean synthesis substantially outperforms Japanese across metrics, while CosyVoice 2 supports both languages and instructed generation covers 29 instruction types.Japanese performance is attributed primarily to character overlap with Chinese; a 290-sample Chinese set evaluates instructed generation.
- 4.7 LM Fine-tuning with Reinforcement Learning: SFT improves speaker similarity and speech quality over the pretrained base model, but WER may worsen; differentiable ASR rewards generalize better out of domain than DPO.DPO benefits Chinese and English SEED subsets but worsens hard samples.
5 Conclusion
CosyVoice 2 is an improved LLM-based streaming speech synthesis model. It unifies streaming and non-streaming synthesis while targeting human-parity naturalness, low latency, and nearly lossless streaming quality.
- CosyVoice 2 combines FSQ, a pretrained-LLM-compatible text-speech architecture, and chunk-aware causal flow matching in one framework.The framework supports both streaming and non-streaming synthesis and adds fine-grained instructed generation.
- The model delivers human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in streaming mode.Its unified design also loosens deployment requirements for streaming and non-streaming applications.
6 Limitations
CosyVoice 2 has language, acoustic-control, and singing limitations that remain open for future work.
- CosyVoice 2 supports only a limited number of languages, and overlapping character sets can degrade synthesis performance.The paper identifies this as an open challenge for future research.
- Textual instructions cannot control acoustic characteristics such as timbre, and CosyVoice 2 performs poorly when tasked with singing.The authors suggest acoustic control through text as a potential direction for role-playing applications.