FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

• SenseVoice-Small recognizes Chinese, English, Cantonese, Japanese, and Korean using a non-autoregressive end-to-end architecture, running more than 5 times faster than Whisper-small and more than 15 times faster than Whisper-large.Source: §1 · p. 1
• SenseVoice-Large supports speech recognition in over 50 languages, with advantages in Chinese and Cantonese, alongside emotion recognition and audio event detection.Source: §1 · pp. 1–2
• CosyVoice (Du et al., 2024a) generates natural speech in five languages with speaker adaptation, cross-lingual voice cloning, emotional expression, and instruction-based control.Its base, instruct, and SFT 300M variants respectively emphasize speaker identity and zero-shot cloning, expressive controllability, and deployment after fine-tuning on seven multilingual speakers.Source: §1 · p. 2
• Integrating SenseVoice, CosyVoice, and Qwen supports demos for Speech-to-Speech Translation (Berard et al., 2018), Emotional Voice Chat (Xue et al., 2024), Interactive Podcast (Laban et al., 2022), and AudioBook.Source: §1 · p. 2

2.1 Overview of FunAudioLLM

• Less than 80ms recognition latency, over 5× faster than Whisper-Small and over 15× faster than Whisper-large, distinguishes SenseVoice-Small’s efficient inference.SenseVoice-Small supports 5 languages.Source: §2.1 · p. 2, Figure 2 · p. 3
• SenseVoice-Large supports high-precision ASR for over 50 languages, while SenseVoice-Small is optimized for rapid understanding across 5 languages.Figure 2 identifies SenseVoice-Small as encoder-only and SenseVoice-Large as encoder-decoder.Source: §2.1 · p. 2, Figure 2 · p. 3
• SenseVoice combines multilingual speech recognition with emotion recognition, audio event detection, inverse text normalization, and punctuation.Its training uses over 300k hours of data.Source: §2.1 · p. 2, Figure 2 · p. 3
• WER below 2% and speaker similarity over 75% give CosyVoice generated samples human-parity quality.Source: §2.1 · pp. 2–3
• CosyVoice is trained on over 170k hours across Chinese, English, Japanese, Cantonese, and Korean for multilingual speech generation.Source: §2.1 · p. 2
• CosyVoice zero-shot in-context learning enables voice cloning from a 3-second prompt and reproduces timbre, emotion, prosody, and style within or across languages.Its instruction model controls speaker identity, speaking style, and other fine-grained paralinguistic features using natural-text instructions.Source: §2.1 · p. 3

2.2 Voice Understanding Model: SenseVoice

• SenseVoice-Small uses an encoder-only architecture for rapid, multilingual, multi-style ASR and other speech-understanding tasks.It extracts 80-dimensional log-mel features, stacks and downsamples frames by a factor of 6, then encodes them with memory-equipped self-attention (SAN-M).Source: §2.2 · p. 3
• Prepended task tokens select language, emotion, audio-event, or transcription-style outputs; ITN adds inverse text normalization and punctuation, while NoITN omits both.Language, emotion, and audio-event tasks use cross-entropy loss, while ASR uses CTC loss.Source: §2.2 · pp. 3–4
• SenseVoice-Large is an autoregressive encoder-decoder for multilingual ASR and other speech-understanding tasks, supporting 50+ languages.It uses decoder tokens to request language, emotion, and timestamped audio-event predictions.Source: §2.2 · p. 4
• SenseVoice-Large differs from SenseVoice-Small by using an autoregressive encoder-decoder architecture for greater transcription accuracy and support for more languages.Its task tokens specify language, emotion, and audio-event prediction with timestamps.Source: §2.2 · pp. 3–4

2.3 Semantic Speech Tokenizer

• Prior unsupervised or self-supervised tokenizers can have tenuous semantic association, unstable synthesis, and sensitivity to noise, requiring carefully curated clean datasets.The section discusses residual-quantization, multi-grouped-quantization, and HuBERT-based semantic tokenizers.Source: §2.3 · p. 4
• S3 (Du et al., 2024a) builds on pre-trained SenseVoice-Large, placing a vector quantizer after the encoder’s first six layers and adding positional embedding after quantization.Encoder1 plus the vector quantizer forms the tokenizer; speech tokens are indices of the closest code vectors.Source: §2.3 · pp. 4–5
• A single codebook with 4,096 entries produces 50 Hz token sequences, reducing the computational load of text-to-token generation in language models.Source: §2.3 · p. 5
• S3 is trained end to end to minimize rich-text recognition errors, producing tokens strongly linked to textual and paralinguistic information while improving noise robustness and reducing reliance on pristine data.This allows a broader spectrum of data to be used for training.Source: §2.3 · p. 5

2.4 Voice Generation Model: CosyVoice

• CosyVoice supports five languages and offers speaker-specific multilingual generation, zero-shot voice adaptation, cross-lingual cloning, emotional speech, and text-instruction control.CosyVoice-base-300M targets speaker identity and zero-shot or cross-lingual generation; CosyVoice-instruct-300M targets expressive, instruction-controlled speech; CosyVoice-sft-300M is fine-tuned on seven multilingual speakers for deployment.Source: §2.4 · p. 5

2.4.1 System Overview

• Speech generation combines an autoregressive Transformer LM for speech tokens, an ODE-based flow-matching model for Mel reconstruction, and a HiFTNet-based vocoder for waveform synthesis.Some components are optional for cross-lingual cloning and speaker-fine-tuned inference.Source: §2.4.1 · p. 6

2.4.2 Model Training

• Training uses teacher forcing for token prediction, while flow matching estimates conditional speech distributions with an optimal-transport vector field and typically needs five to ten inference iterations.Classifier-free guidance masks 70%∼100% of proceeding feature conditions to improve in-context learning; HiFTNet modifications support streaming waveform generation.Source: §2.4.2 · p. 6

2.4.3 Zero-shot In-context Learning

• Zero-shot in-context learning replicates a voice from a brief reference sample, using different prompt-sequence construction depending on whether prompt speech and target text share a language.For cross-lingual generation, prompt text and tokens are omitted; generated tokens, speaker embeddings, and reference Mel features condition flow matching to preserve timbre and environment.Source: Figure 5 · p. 6, §2.4.3 · p. 7

2.4.4 Instruction Fine-tuning

• CosyVoice-instruct adds control over speaker identity, emotion, gender, speaking rate, pitch, and fine-grained features such as laughter, breaths, and word emphasis.The instruction-finetuned variant extends CosyVoice-base.Source: §2.4.4 · p. 7

3.1 Training Set for SenseVoice

• SenseVoice-Small used approximately 300,000 hours across Chinese, Cantonese, English, Japanese, and Korean; SenseVoice-Large added 100,000 hours of diverse multilingual data.Figure 6 overviews SenseVoice training hours across languages.Source: §3.1 · p. 7, Figure 6 · p. 8
• Audio event detection and speech emotion recognition models generated pseudo-labels for 150 million AED entries and 30 million SER entries.These labels contributed to an extensive richly transcribed dataset.Source: §3.1 · pp. 7–8

3.2 Training Set for CosyVoice

• CosyVoice training data was collected across multiple languages, processed with speech detection, SNR estimation, diarization, and separation, then refined using pseudo-text labels and force alignment.SenseVoice-Large and Paraformer generated pseudo-text labels; force alignment helped remove low-quality data and improve punctuation. Table 4 presents training duration by language.Source: §3.2 · p. 8, Table 4 · p. 8
• CosyVoice-instruct was fine-tuned from CosyVoice-base on instruction data without speaker embeddings in the autoregressive language model.Table 5 presents instruction-training duration by type.Source: §3.2 · p. 8, Table 5 · p. 8

4.1 Multilingual Speech Recognition

• SenseVoice-S and SenseVoice-L outperform their Whisper counterparts on most popular benchmark test sets, except Librispeech.SenseVoice-Large is generally comparable to Whisper-Large-V3 and performs significantly better on Cantonese, Catalan, and Marathi; SenseVoice-S is over 5 times faster than Whisper-small and over 15 times faster than Whisper-Large-V3.Source: §4.1 · p. 9, Table 6 · p. 10, Table 7 · p. 10

4.2 Speech Emotion Recognition

• SenseVoice-Large achieves the best results on almost all SER datasets, while SenseVoice-Small outperforms other baseline models on most datasets.SenseVoice also performs well across the reported test sets and metrics without target-domain fine-tuning.Source: §4.2 · p. 10, Figure 8 · p. 10, Table 8 · p. 11

4.3 Audio Event Detection

• SenseVoice performs well on audio event classification and detection, although BEATs and PANNs may achieve higher F1 scores.The authors attribute this partly to the baselines’ adjustable detection thresholds and SenseVoice’s training on ASR data with AED pseudo-labels rather than AED-specific data; SenseVoice and Qwen-Audio also show higher accuracy than recall.Source: §4.3 · p. 11, Figure 9 · p. 11

4.4 Preserving Semantic Information by S3 Tokenizer

• On Common Voice zh-CN, S3 tokens reduce error rate by 4.14% relative to Whisper-Large-V3, while showing robust recognition performance on Chinese and English test sets.The S3 tokenizer uses a single codebook with 4,096 entries.Source: §4.4 · p. 12, Table 9 · p. 12

4.5 Evaluation on Generation Quality of CosyVoice

• CosyVoice achieved human-level English synthesis performance; ASR re-ranking reduced WER to 1.51%, while Chinese CER reached 1.84%.On Chinese, CosyVoice had comparable CER to original utterances and fewer insertion and deletion errors than ChatTTS, which showed speaker leakage; CosyVoice also had higher speaker similarity than the original utterances in both languages.Source: §4.5 · pp. 12–13, Table 10 · p. 12, Table 11 · p. 13

4.6 Evaluation on Emotion Controllability of CosyVoice

• CosyVoice-instruct with emotional instructions significantly improves emotion-control accuracy over CosyVoice-base and CosyVoice-instruct without emotional instructions.Evaluation used 100 English utterances for each of six emotions, with emotion predictions assessed by emo2vec.Source: §4.6 · p. 13, Table 12 · p. 13

4.7 CosyVoice as a Data Generator

• SenseVoice-generated training data alone yields ASR results comparable to original Librispeech training, and combining the data improves recognition accuracy.Synthesized data using MLS text significantly improves performance, suggesting text diversity may matter more than speech duration; the authors attribute improvement to varied linguistic content.Source: §4.7 · pp. 13–14, Table 13 · p. 14

Speech-to-Speech Translation

• SenseVoice recognizes source speech, an LLM translates it, and CosyVoice synthesizes target-language speech with cross-lingual voice cloning, letting users speak foreign languages in their own voice.Figure 10 illustrates the speech-to-speech translation workflow.Source: §5 · p. 14, Figure 10 · p. 14

Emotional Voice Chat

• SenseVoice recognizes speech, emotion, and audio events; an LLM generates response content and a speaking-style description; CosyVoice produces speech following that style.Figure 11 depicts the emotional voice chat application.Source: §5 · p. 14, Figure 11 · p. 14

Interactive Podcast

• An LLM plugin fetches real-time daily knowledge for a content-generation agent to script a podcast; a multi-agent system assigns roles, and CosyVoice synthesizes the voices.Users can join the podcast in interactive dialogues with the multi-agent system.Source: §5 · p. 14, Figure 12 · p. 15

Expressive Audiobook

• An LLM analyzes narrative, dialogue, characters, and fine-grained sentiment in books, while CosyVoice synthesizes speech with enhanced expressiveness.Figure 13 illustrates the expressive audiobook application.Source: §5 · p. 15, Figure 13 · p. 15

6 Limitations

• SenseVoice performs worse on under-resourced languages and is not designed for streaming transcription.Future work may develop streamable voice understanding models based on SenseVoice.Source: §6 · p. 15
• CosyVoice supports limited languages, cannot infer emotion or speaking style from text meaning, and performs poorly on singing.It also needs improvement in expressing emotional changes while maintaining the original voice timbre.Source: §6 · p. 15
• Because the two FunAudioLLM models are not trained end-to-end with LLMs, the pipeline may propagate errors and affect overall performance.Source: §6 · p. 15

A Auxiliary Results of SenseVoice on Common Voice.

• The table compares model performance with and without language ID.The supplied caption gives no metric values or comparative outcomes.Source: Table 14 · p. 21