Source-linked AI summary
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang, Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Siqi Zheng
TL;DR
FunAudioLLM addresses the challenge of making interactions between humans and LLMs more natural through voice. It combines SenseVoice for multilingual speech understanding with CosyVoice for controllable speech generation, achieving low-latency or broad-coverage recognition and expressive multilingual synthesis. The report positions these capabilities for applications such as translation, emotional chat, podcasts, and audiobooks.
Problem
FunAudioLLM targets the need for more natural voice interactions between humans and LLMs by combining speech recognition, emotion understanding, and voice generation.
Method
The report combines SenseVoice, a multi-task speech understanding foundation model, with CosyVoice, a multilingual and controllable speech generation model, for integration with LLMs.
Results
The models provide low-latency or broad-coverage speech recognition and multilingual voice generation with WER below 2%, speaker similarity above 75%, and three-second zero-shot voice cloning.
Takeaways & Limitations
FunAudioLLM supports voice applications including speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration.
Takeaways & Limitations
SenseVoice performs worse on under-resourced languages and is not designed for streaming transcription, while CosyVoice supports limited languages and cannot infer emotion or style from text semantics.
Abstract
from arXiv · showhide
This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.
1 Introduction
FunAudioLLM is introduced as a framework for natural voice interaction between humans and LLMs, combining speech understanding and generation capabilities. Its models support multilingual recognition, expressive synthesis, and applications including translation, emotional conversation, podcasts, and audiobooks.
- 1 Introduction: FunAudioLLM combines voice processing capabilities with LLMs to support more natural human–machine interaction.The framework targets high-precision speech recognition, emotion recognition, and voice generation.
- 1 Introduction: SenseVoice-Small supports five languages with inference more than 5 times faster than Whisper-small and more than 15 times faster than Whisper-large.Its non-autoregressive end-to-end architecture enables the low-latency design.
- 1 Introduction: SenseVoice-Large supports speech recognition in over 50 languages, while SenseVoice also provides emotion recognition and audio event detection.The models target broad multilingual coverage and multiple speech-understanding tasks.
- 1 Introduction: CosyVoice generates multilingual, speaker-tailored voices with zero-shot adaptation, cross-lingual voice cloning, emotional expression, and instruction-based control.It supports Chinese, English, Japanese, Cantonese, and Korean.
- 1 Introduction: Integrating SenseVoice, CosyVoice, and LLMs enables speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook applications.These demos connect speech understanding and generation to interactive use cases.
- 1 Introduction: FunAudioLLM leverages SenseVoice and CosyVoice to enable more natural and seamless communication between humans and LLMs.This is the report’s overall stated direction for voice interaction technology.
2 FunAudioLLM Models
FunAudioLLM comprises SenseVoice for multilingual speech understanding and CosyVoice for multilingual speech generation. The models combine low-latency or high-precision recognition with controllable, multilingual voice synthesis and zero-shot voice cloning.
- 2 FunAudioLLM Models: FunAudioLLM includes SenseVoice and CosyVoice as foundation models for voice understanding and generation.SenseVoice addresses speech understanding, while CosyVoice addresses speech generation.
- 2 FunAudioLLM Models: SenseVoice-Small has recognition latency below 80ms and runs more than 5 and 15 times faster than Whisper-Small and Whisper-large, respectively.SenseVoice is trained on over 300k hours and SenseVoice-Large supports high-precision ASR for over 50 languages.
- 2 FunAudioLLM Models: CosyVoice is trained on over 170k hours across five languages and produces samples with WER below 2% and speaker similarity above 75%.The reported speaker similarity reaches the quality level of human parity.
- 2 FunAudioLLM Models: SenseVoice supports ASR, language identification, speech emotion recognition, and audio event detection through Small and Large variants with different speed, accuracy, and language-coverage goals.The Small model is encoder-only and rapid, whereas the Large model is encoder-decoder and aimed at more precise understanding across more languages.
- 2 FunAudioLLM Models: CosyVoice supports three-second zero-shot voice cloning, cross-lingual reproduction of timbre and style, and text-based control of speaker identity and speaking style.Its in-context learning can reproduce timbre, emotion, prosody, and style within or across languages.
2.2 Voice Understanding Model: SenseVoice
SenseVoice is a multilingual speech foundation model supporting ASR and additional voice-understanding tasks through two architectures tailored for speed or accuracy. Its task conditioning uses prepended embeddings or decoder tokens to select language, emotion, audio-event, and transcription-style outputs.
- SenseVoice supports automatic speech recognition, spoken language identification, speech emotion recognition, and audio event classification or detection.
- SenseVoice-Small is a non-autoregressive encoder-only model designed for rapid multilingual, multi-style speech understanding.It computes speech features, downsamples them, and processes them with a memory-equipped self-attention network.
- Four prepended embeddings condition SenseVoice-Small on language identification, emotion recognition, audio-event classification, and transcription style.The transcription-style embeddings select inverse text normalization with punctuation or unnormalized transcription without punctuation.
- ASR uses CTC loss, while language identification, emotion recognition, and audio-event classification use cross-entropy loss during training.
- SenseVoice-Large uses an autoregressive encoder-decoder architecture with decoder tokens for language, emotion, and timestamped audio-event prediction.Its main advantage over SenseVoice-Small is higher transcription accuracy and support for more than 50 languages.
2.3 Semantic Speech Tokenizer
The supervised semantic speech tokenizer S3 builds on SenseVoice-Large to produce compact, semantically informed speech tokens for speech generation. Its supervised design links tokens to textual and paralinguistic information while improving robustness to noisy data.
- S3 is a supervised semantic speech tokenizer built from SenseVoice-Large with a vector quantizer inserted after the encoder’s first six layers.An additional positional embedding after quantization enhances temporal information.
- Supervised end-to-end training gives S3 tokens strong semantic relationships to textual and paralinguistic information.
- S3 is more robust to data noise and reduces reliance on pristine data collection, allowing a broader range of training data.
2.4 Voice Generation Model: CosyVoice
CosyVoice is a multilingual speech-generation family supporting speaker adaptation, cross-lingual voice cloning, emotional expression, and instruction-based control. Its architecture combines autoregressive speech-token generation, flow matching for Mel-spectrogram reconstruction, and HiFTNet waveform synthesis.
- Capabilities: CosyVoice supports multilingual voice generation, zero-shot adaptation, cross-lingual voice cloning, emotional expression, and instruction-controlled speech output.Released variants target speaker identity and cross-lingual cloning, emotional expressiveness, or deployment across seven multilingual speakers.
- Architecture: The CosyVoice pipeline generates speech tokens with an autoregressive Transformer, reconstructs Mel spectra with flow matching, and synthesizes waveforms using a HiFTNet-based vocoder.The vocoder includes modifications for streaming generation.
- Architecture: Flow matching estimates P(S|X, v, Sref) from speech tokens, speaker embeddings, and reference Mel spectra, typically requiring five to ten inference iterations.Classifier-free guidance and masking 70%∼100% of preceding feature conditions improve in-context learning.
- Voice adaptation: Zero-shot in-context learning replicates an arbitrary voice from a brief reference speech sample by conditioning token generation and flow matching on prompt speech information.The composite condition includes generated speech tokens, speaker embedding, and the prompt Mel spectrogram.
- Instruction fine-tuning: CosyVoice-instruct adds control over speaker identity, emotion, gender, speaking rate, pitch, laughter, breaths, and word emphasis through textual instructions.It is fine-tuned from CosyVoice-base without speaker embeddings in the autoregressive language model.
3 Dataset
The datasets combine large multilingual speech corpora with automatically generated rich labels for SenseVoice and cleaned, multilingual speech data for CosyVoice. Instruction fine-tuning uses separately characterized data organized by instruction type.
- SenseVoice data: SenseVoice-Small uses approximately 300,000 hours across Chinese, Cantonese, English, Japanese, and Korean, while SenseVoice-Large adds 100,000 hours of diverse multilingual data.The training-data distribution is summarized across languages in log scale.
- SenseVoice data: Rich transcription labels include 150 million audio-event-detection entries and 30 million speech-emotion-recognition entries.These labels are generated as pseudo labels for the training data.
- CosyVoice data: CosyVoice training data are collected across multiple languages using speech detection, SNR estimation, speaker diarization, separation, pseudo-label generation, and force-alignment refinement.SenseVoice-Large and Paraformer generate pseudo text labels, while force alignment removes low-quality data.
- Instruction data: CosyVoice-instruct is fine-tuned on instruction data whose durations are reported by instruction type.The fine-tuning procedure does not incorporate speaker embeddings in the autoregressive language model.
4 Experimental Results
Experiments evaluate SenseVoice across multilingual recognition, emotion recognition, and audio-event tasks, and evaluate CosyVoice on speech-generation quality, controllability, and data augmentation. Results show strong recognition and generation performance, with task-specific caveats for event detection.
- Multilingual speech recognition: SenseVoice-Small and SenseVoice-Large outperform corresponding Whisper models on most tested speech-recognition datasets except Librispeech.The comparisons cover AISHELL-1, AISHELL-2, WenetSpeech, Librispeech, and Common Voice.
- Multilingual speech recognition: SenseVoice-Large performs comparably to Whisper-Large-V3 overall and performs significantly better on Cantonese, Catalan, and Marathi.The comparison evaluates a broader range of languages with or without ground-truth language identifiers.
- Speech emotion recognition: SenseVoice-Large achieves the best results on almost all evaluated emotion-recognition datasets, while SenseVoice-Small outperforms other baselines on most datasets.The evaluation spans seven Chinese and English emotion-recognition corpora.
- Audio-event detection: SenseVoice supports audio-event classification and detection, but BEATs and PANNs may achieve higher F1 scores because their thresholds can be adjusted to trade accuracy against recall.SenseVoice-Small predicts event presence with at most one event per utterance, whereas SenseVoice-Large can predict event boundaries.
- Speech generation: 4.14% relative error-rate reduction is achieved by S3 tokens over Whisper-Large V3 on Common Voice zh-CN.The result is reported as evidence that S3 tokens capture semantic content.
- Speech generation: CosyVoice reaches 1.51% WER on English after ASR re-ranking and 1.84% CER on Chinese after ASR re-ranking, with higher speaker similarity than original utterances.The English evaluation reports human-level performance, while the Chinese comparison reports fewer insertion and deletion errors than ChatTTS.
- Instruction control: CosyVoice-instruct significantly improves emotion-control accuracy over CosyVoice-base and instruction-free CosyVoice-instruct.The evaluation supplies content text together with a speaking-style instruction for the instructed condition.
- Data generation: Training ASR with CosyVoice-synthesized data alone achieves performance comparable to the original Librispeech training set, while combining datasets improves recognition accuracy.Synthesized data based on MLS text produces a notable improvement, suggesting text diversity is important for this ASR setting.
5 Applications
FunAudioLLM combines SenseVoice, CosyVoice, and LLMs into applications for translation, emotional conversation, interactive podcasting, and expressive audiobook narration. Each application assigns speech understanding, language reasoning, and speech synthesis complementary roles.
- Speech-to-Speech Translation: Speech-to-speech translation recognizes source speech with SenseVoice, translates it with an LLM, and synthesizes target speech with CosyVoice using cross-lingual voice cloning.Users can speak a foreign language while retaining their own voice.
- Emotional Voice Chat: Emotional Voice Chat combines SenseVoice emotion and event recognition, LLM response generation with speaking-style descriptions, and CosyVoice emotional synthesis.The application is depicted in Figure 11.
- Interactive Podcast: Interactive Podcast uses real-time knowledge retrieval, content-generation and role-matching agents, and CosyVoice voice synthesis for multi-agent discussions.Users can insert themselves into the podcast for interactive dialogue.
- Expressive Audiobook: Expressive audiobook narration uses LLM analysis of narrative structure, dialogue, characters, and fine-grained sentiment before CosyVoice synthesizes the speech.The resulting application is designed for enhanced expressiveness.
6 Limitations
FunAudioLLM’s models have scope and integration limitations, including weaker ASR for under-resourced languages, limited CosyVoice capabilities, and separate training from LLMs.
- SenseVoice generally delivers lower ASR performance for under-resourced languages and is not designed for streaming transcription.Future work may develop streamable voice-understanding models based on SenseVoice.
- CosyVoice supports a limited number of languages and cannot infer appropriate emotion or speaking style from text semantics.Its emotional and stylistic control depends on explicit instructions.
- CosyVoice does not perform well on singing and needs improved expressive emotional changes while preserving the original voice timbre.
- SenseVoice and CosyVoice are not trained end-to-end with LLMs, so the pipeline approach may introduce error propagation affecting overall performance.
7 Authors (alphabetical order of family name)
This section lists the report’s authors in alphabetical order by family name.
- The author list begins with Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, and Xiang Lv.
- The author list continues with Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, and Xian Shi.
- The next authors are Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, and Shengpeng Ji.
- The final listed authors are Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang, Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, and Siqi Zheng.
A Auxiliary Results of SenseVoice on Common Voice.
Table 14 compares model performance with and without a language ID.
- Table 14 presents performance comparisons among different models under conditions with and without language ID.