Source-linked AI summary
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
Guanrou Yang, Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang, Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, Fan Yu, Zhihao Du, Zhifu Gao, ShiLiang Zhang, Xie Chen
TL;DR
Emotion-controllable TTS still lacks fine-grained natural-language control, high-quality labeled data, and reliable evaluation aligned with human perception. EmoVoice addresses these gaps with an LLM-based freestyle-prompting model, a parallel phoneme-audio variant, EmoVoice-DB, and metric analysis. It achieves state-of-the-art performance on the paper’s English and Chinese test sets, while evaluation results show that existing metrics and multimodal LLMs remain imperfect for fine-grained assessment.
Problem
Emotion-controllable TTS lacks fine-grained natural-language control, high-quality emotion datasets, and reliable evaluation metrics aligned with human perception.
Method
EmoVoice uses an LLM for freestyle natural-language emotion prompting, while EmoVoice-PP predicts phoneme and audio tokens in parallel and the paper introduces EmoVoice-DB and evaluates emotion metrics.
Results
EmoVoice achieves state-of-the-art performance on the English EmoVoice-DB and Chinese Secap test sets, while existing metrics show weak sentence-level alignment with human perception.
Takeaways & Limitations
Fine-grained emotion-controlled TTS can be trained with synthetic data, but reliable fine-grained emotion evaluation remains an open need.
Takeaways & Limitations
On the Chinese test set, GPT-series models show unnatural prosody and accented timbre and lag across emotion metrics despite manual speaker selection.
Abstract
from arXiv · showhide
Human speech goes beyond the mere transfer of information; it is a profound exchange of emotions and a connection between individuals. While Text-to-Speech (TTS) models have made huge progress, they still face challenges in controlling the emotional expression in the generated speech. In this work, we propose EmoVoice, a novel emotion-controllable TTS model that exploits large language models (LLMs) to enable fine-grained freestyle natural language emotion control, and a phoneme boost variant design that makes the model output phoneme tokens and audio tokens in parallel to enhance content consistency, inspired by chain-of-thought (CoT) and chain-of-modality (CoM) techniques. Besides, we introduce EmoVoice-DB, a high-quality 40-hour English emotion dataset featuring expressive speech and fine-grained emotion labels with natural language descriptions. EmoVoice achieves state-of-the-art performance on the English EmoVoice-DB test set using only synthetic training data, and on the Chinese Secap test set using our in-house data. We further investigate the reliability of existing emotion evaluation metrics and their alignment with human perceptual preferences, and explore using SOTA multimodal LLMs GPT-4o-audio and Gemini to assess emotional speech. Dataset, code, checkpoints, and demo samples are available at https://github.com/yanghaha0908/EmoVoice.
1 Introduction
Emotion-controllable TTS aims to produce speech with nuanced, user-directed emotional expression, but existing datasets, controls, and evaluation methods remain limited. EmoVoice addresses these gaps through freestyle natural-language prompting, new data, and metric analysis.
- Existing emotion-controllable TTS research relies mainly on coarse emotion categories, which cannot fully capture nuanced emotional states.
- Open-source emotion speech datasets total less than 300 hours across languages and often contain coarse labels, indistinct emotions, or inconsistent annotations.
- Current emotion metrics depend on classifiers or embedding similarity, but their reliability for fine-grained evaluation remains unproven.
- EmoVoice uses an LLM to interpret freestyle natural-language emotion descriptions directly, while EmoVoice-PP predicts phoneme and audio tokens in parallel for content consistency.
- The paper introduces EmoVoice-DB, reports state-of-the-art results on English and Chinese test sets, and evaluates metric alignment with human preferences.
2 Related work
Prior emotional TTS work spans coarse category control, natural-language style prompting, and LLM-based speech generation. EmoVoice is positioned against these approaches through fine-grained emotion prompting and its distinct modeling design.
- Emotional TTS: Emotional TTS systems commonly use coarse labels such as happy, sad, and angry, while some diffusion models also control emotion intensity.
- Prompt-based control: Natural-language prompting has been used for style transfer and attributes such as gender, pitch, duration, and energy, with some systems targeting speaking style or voice cloning.
- Instruction-based TTS: CosyVoice systems provide instruction-based control using large training collections, whereas EmoVoice differs through group token modeling and its fine-grained emotion-control focus.
- LLM-based TTS: LLM-based TTS research integrates language models with speech codecs or speech tokens, including parameter-efficient fine-tuning and next-token speech generation.
3 Methodology
EmoVoice formats emotion descriptions and text as natural-language input to a causal language model that generates speech tokens. Its phoneme-boost variant adds parallel phoneme prediction to guide audio generation and improve content consistency.
- EmoVoice: EmoVoice uses Qwen2.5-0.5B and prompts it with a fine-grained emotion description plus the text to synthesize.
- EmoVoice: The model autoregressively predicts 50Hz CosyVoice semantic tokens, which are converted into waveforms by flow matching and HiFi-GAN.
- EmoVoice: Semantic group modeling predicts multiple semantic tokens per step, compressing sequence length while accelerating training and reducing modeling difficulty.
- EmoVoice-PP: EmoVoice-PP simultaneously predicts semantic and phoneme tokens, using lower-rate phonemes as intermediate supervision during inference to improve content consistency.
- Training pipeline: Training first pre-trains the model on standard TTS data, then fine-tunes it on instruction data pairing emotion descriptions with expressive speech.
- Training pipeline: The architecture receives emotion description and text, then autoregressively generates audio token groups and optional phoneme tokens in parallel.
4 EmoVoice-DB
EmoVoice-DB is a synthetic, high-quality English emotional speech dataset designed around fine-grained natural-language emotion descriptions. Its construction combines generated text, constrained descriptions, and expressive speech synthesis.
- Dataset overview: EmoVoice-DB contains over 20,000 expressive speech samples totaling approximately 40 hours, with detailed natural-language emotional annotations.
- Dataset overview: The dataset covers seven balanced emotion categories, diverse text types, and five speaker timbres.
- Construction process: Its construction generates text and emotional descriptions with GPT-4o before synthesizing emotional speech using GPT-4o-audio.
- Construction process: Text spans novel prose, emotionally charged dialogue, and observational phrases, with up to two capitalized words marking prosodic stress.
- Construction process: Emotion descriptions use single-sentence present-participle phrases focused on vocal affect rather than contextual events.
5 Experimental Setup
The experiments use synthetic and internal speech data for pretraining and emotional fine-tuning, with EmoVoice-DB providing structured dataset statistics and examples.
- Data: The English model relies entirely on synthetic training data, while the Chinese model uses internal industrial training data and Secap for evaluation.Pretraining uses VoiceAssistant for English and Belle for Chinese; emotional fine-tuning combines synthetic GPT-4o-audio data, LAION’s Got Talent, and EmoVoice-DB for English.
- Data: EmoVoice-DB is summarized through dataset statistics and examples, while its remaining samples support training after held-out validation and test splits.The dataset construction and split details are reported alongside the experimental setup.
- Evaluation: The models are evaluated with WER, emotion similarity, emotion classification recall rate, UTMOS, and subjective MOS across English and Chinese test settings.Table 2 covers the English EmoVoice-DB test set, while Secap provides the Chinese evaluation setting.
6.1 Main Results
EmoVoice delivers strong objective and subjective emotional-speech results on English and Chinese evaluations, while the Chinese comparison also exposes limitations in GPT-series speech quality.
- English objective results: 0.9118 emotion similarity, 0.424 recall, 2.62 WER, and 4.345 UTMOS characterize EmoVoice(1.5B)’s balanced English performance.It also maintains competitive content consistency and emotional expressiveness relative to GPT-4o-mini-tts while exceeding it in speech quality.
- English subjective results: 3.507 MOS places EmoVoice(1.5B) near GPT-4o-mini-tts at 3.598 and above CosyVoice2 at 2.138 and PromptTTS at 1.415.The subjective evaluation measures emotional expressiveness and adherence to emotional instructions on a 1–5 scale.
- Chinese results: EmoVoice-PP achieves the best major-metric performance on Secap, with 7.6 WER, 0.7939 emotion similarity, and 0.434 recall.Its WER approaches the human-reference upper bound of 6.62, but limited emotion-specific fine-tuning data leaves an expressiveness gap.
- Chinese results: GPT-series models lag across Chinese emotion metrics despite manual speaker selection, with unnatural prosody and accented timbre observed during inference.The comparison therefore reflects both emotional performance and Chinese speech-quality limitations.
6.2 Abalation Study
The ablations compare output structures, LLM scale and initialization, and emotion-description augmentation. Parallel phoneme guidance, larger LLMs, LLM initialization, and prompt augmentation improve content consistency or emotion-related performance under the tested settings.
- Different output-side structures: EmoVoice-PP and EmoVoice-PT achieve the highest emotion similarity of 0.9115 and recall rate of 0.405 among output-structure variants.Emotion metrics are otherwise nearly comparable, while perceptual quality remains consistently high.
- Different output-side structures: WERs are 2.73 for EmoVoice and 3.06 for EmoVoice-PP, while EmoVoice-PP reaches 3.94 and 3.11 on the EmoVoice-DB and Seed-TTS pretraining tests.The parallel phoneme sequence improves content consistency in pretraining.
- Results on hard-case test set: On the hard-case test set, EmoVoice-PP outperforms EmoVoice and EmoVoice-PT in WER, emotion similarity, and recall rate.The test set contains difficult pronunciations and semantic inconsistencies between text and emotional instructions; text guidance also helps, but less than phoneme guidance.
- Scaling LLM size: Scaling from 0.5B to 1.5B reduces pretraining WER from 4.73% to 3.83%, a relative reduction of 19.03%.During emotion fine-tuning, recall rises from 0.395 to 0.424 and WER falls from 2.73% to 2.62%.
- LLM initialization: LLM initialization consistently improves performance across EmoVoice variants, lowering WER and enhancing emotion conveyance.The authors attribute the WER gains to better alignment between textual and acoustic representations.
- Emotion description augmentation: Emotion-description augmentation reduces WER from 3.83 to 2.73, a relative improvement of 28.72%.Rewritten descriptions expose the model to more diverse emotional prompts and improve emotional-semantic understanding.
7 Discussion on Emotion Evaluation Metrics
The paper evaluates emotion similarity and multimodal LLM ratings against human judgments. System-level correlations are generally high, but sentence-level alignment is weak, limiting fine-grained evaluation reliability.
- Existing metrics: Emotion similarity provides reference value for overall model quality but is insufficiently sensitive for fine-grained comparisons between closely matched models.Its scores generally align with MOS while obscuring subtle differences.
- Multimodal LLM evaluation: The evaluation samples 360 audios rated by 25 human raters, then uses a balanced 100-audio set spanning MOS levels from 1 to 5.GPT-4o and Gemini are evaluated alongside CosyVoice and PromptTTS systems.
- Evaluation results: System-level Spearman’s ρ values are generally high, but every metric has sentence-level Spearman’s ρ below 50%.This indicates generally weak alignment with human perception at the individual-sentence level.
- Evaluation results: LLMs still cannot effectively evaluate emotional speech, with Gemini ratings relatively stable and GPT-4o ratings more variable.Repeated-rating variability is assessed using standard deviation.
8 Conclusions
The paper presents EmoVoice for fine-grained emotion-controllable TTS, its phoneme-boost variant and 40-hour dataset, and evaluates emotion metrics and multimodal LLM judges. It reports strong benchmark performance but finds current evaluation methods insufficiently precise for fine-grained human-aligned assessment.
- Contributions: EmoVoice uses LLM-based freestyle text prompting, while EmoVoice-PP outputs phoneme and audio tokens in parallel to enhance content consistency.The paper also compares parallel, sequential, and interleaved output structures.
- Contributions: EmoVoice-DB is a 40-hour English emotion dataset with expressive speech and natural-language emotion-description labels.The dataset supports the paper’s fine-grained emotion-control setting.
- Results: EmoVoice achieves state-of-the-art performance on the English EmoVoice-DB test set using synthetic training data and on the Chinese Secap test set using in-house data.These are the paper’s reported benchmark outcomes.
- Evaluation: Existing emotion metrics and multimodal LLM judges lack the precision and sensitivity required for accurate emotion evaluation.The conclusion emphasizes alignment with human perception as an unmet need.