Source-linked AI summary
EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance
Yiwei Guo, Chenpeng Du, Xie Chen, Kai Yu
TL;DR
Emotional TTS models often lack intensity control, and existing approaches can rely on separately calculated intensity values. EmoDiff uses soft-label classifier guidance to control emotion intensity; the paper reports improved quality, controllability, and sample diversity over a baseline.
Problem
Emotional TTS models typically use categorical emotion labels without intensity control, while prior approaches may require separate intensity calculation and optimization.
Source: §1 · p. 1Method
EmoDiff guides reverse diffusion sampling with a classifier and a soft label that weights the target emotion against Neutral by intensity α.
Source: §1 · p. 1, §3.2 · p. 2Results
Subjective and objective evaluations show that EmoDiff outperforms the baseline in TTS quality, intensity controllability, and sample diversity.
Source: §5 · p. 4Takeaways & Limitations
Using the soft-label weight as classifier-guidance intensity provides arbitrary specified emotion intensities without extra optimization.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Although current neural text-to-speech (TTS) models are able to generate high-quality speech, intensity controllable emotional TTS is still a challenging task. Most existing methods need external optimizations for intensity calculation, leading to suboptimal results or degraded quality. In this paper, we propose EmoDiff, a diffusion-based TTS model where emotion intensity can be manipulated by a proposed soft-label guidance technique derived from classifier guidance. Specifically, instead of being guided with a one-hot vector for the specified emotion, EmoDiff is guided with a soft label where the value of the specified emotion and \textit{Neutral} is set to $\alpha$ and $1-\alpha$ respectively. The $\alpha$ here represents the emotion intensity and can be chosen from 0 to 1. Our experiments show that EmoDiff can precisely control the emotion intensity while maintaining high voice quality. Moreover, diverse speech with specified emotion intensity can be generated by sampling in the reverse denoising process.
1. INTRODUCTION
Emotional TTS models commonly use categorical labels without intensity control, while prior intensity methods can require extra optimization and risk quality degradation. EmoDiff uses soft-label classifier guidance to control intensity and reports good speech quality and diverse samples.
1. INTRODUCTION
- Current emotional TTS models generally accept an emotion label but lack intensity control, despite high-quality neural TTS being available.The paper contrasts emotional TTS with prosody modeling that does not require a specific label in advance.Source: §1 · p. 1
- Prior intensity methods use manually constructed relative-attribute rankings or manipulate emotion embeddings, with the latter depending on embedding-space structure and extra constraints.The ranking approach uses a max-margin optimization solved by support vector machines; the manually separated stage may introduce bias.Source: §1 · p. 1
- Some prior intensity-control methods can degrade speech quality, including MixedEmotion despite its quality-oriented pretraining.MixedEmotion is described as an autoregressive model using RAR-derived intensity values to weight emotion embeddings.Source: §1 · p. 1
- EmoDiff defines intensity through a soft label blending the target emotion and Neutral, then guides reverse denoising with a classifier.The target-emotion and Neutral weights are α and 1 − α, respectively; α ranges from 0 to 1.Source: §1 · p. 1
- The soft-label weight serves as the classifier-guidance intensity, enabling arbitrary specified intensities without extra optimization.The paper defines intensity in terms of classifier probability.Source: §1 · p. 1
- EmoDiff reports good-quality, natural speech and diverse samples, including varied samples within the same emotion.The proposed conditional sampling approach extends classifier guidance, which steers diffusion sampling using a classifier gradient and a one-hot class label.Source: §1 · p. 1
2. DIFFUSION MODELS WITH CLASSIFIER GUIDANCE
Diffusion models generate samples by reversing a noisy forward process with a learned score, while classifier guidance adds class information to conditional sampling. The section also notes that standard classifier guidance does not directly handle soft labels.
2.1. Denoising Diffusion Models and TTS Applications
- For TTS, the diffusion model conditions its terminal distribution on phoneme representations and is built on GradTTS.The input mel-spectrogram frame is modeled with a forward SDE whose conditional noisy-state distribution can be sampled directly.Source: §2.1 · p. 2
- The reverse-time SDE uses an estimated score to generate realistic samples from noise, following the forward process's trajectory distribution.The model estimates the score at noisy states along the SDE trajectory.Source: §2.1 · p. 2
2.2. Conditional Sampling Based on Classifier Guidance
- Classifier guidance adds the classifier gradient to an unconditional model's score to sample from a class-conditional distribution.The classifier is evaluated on noisy states at diffusion times along the SDE trajectory.Source: §2.2 · p. 2
- Scaling classifier gradients trades stronger class correlation at larger guidance levels for greater sample variability at smaller levels.The section states that ordinary classifier guidance cannot directly handle soft labels, motivating its extension for emotion intensity control.Source: §2.2 · p. 2
3. EMODIFF
EmoDiff trains an emotion-unconditional acoustic model and a noisy-speech emotion classifier, then uses soft-label guidance during reverse denoising. The guidance weights a target emotion against Neutral and extends to weighted combinations of basic emotions.
3.1. Unconditional Acoustic Model and Classifier Training
- The acoustic model is trained on emotional data without emotion conditions, using duration, diffusion, and prior losses.It follows GradTTS and uses explicit duration sequences from forced alignment.Source: §3.1 · p. 2
- A cross-entropy-trained classifier predicts emotion from noisy mel-spectrograms, diffusion time, and phoneme-dependent Gaussian means while the acoustic model stays frozen.The classifier is conditioned on text, though later notation omits that condition when unambiguous.Source: §3.1 · p. 2
3.2. Intensity Controllable Sampling with Soft-Label Guidance
- Soft-label guidance represents emotion intensity α by weighting a target emotion against Neutral, and combines their classifier gradients in the sampling direction.At α values of 1 or 0, the guidance reduces to standard classifier guidance.Source: §3.2 · p. 2
- During reverse-time SDE sampling, the classifier guides each update toward speech matching the specified emotion distribution and intensity.Sampling starts from noise centered on the phoneme-dependent mean; the guidance term is scaled by guidance level γ.Source: §3.2 · p. 3, Figure 1 · p. 3
- The guidance generalizes to mixtures of basic emotions by weighting their classifier gradients, with the weights treated as a target emotion distribution.The gradient decreases cross-entropy between that target distribution and the classifier output.Source: §3.2 · p. 3
4. EXPERIMENTS AND RESULTS
Experiments assess EmoDiff’s speech quality, intensity controllability, and sample diversity; reported results favor it over MixedEmotion across these dimensions.
4.2. Emotional TTS Quality
- EmoDiff outperforms MixedEmotion on MOS and MCD, while its MOS is close to hard-conditioned GradTTS and its MCD differs only slightly.The comparison controlled EmoDiff and MixedEmotion at α = 1.0.Source: Table 1 · p. 3, §4.2 · p. 4
4.1. Experimental Setup
- Experiments used the English ESD data across ten speakers, with a classifier trained on one female speaker and HifiGAN used as vocoder.The unconditional acoustic model was trained on all ten English speakers.Source: §4.1 · p. 3
4.3. Controllability of Emotion Intensity
- Across α = 0.0–1.0, EmoDiff covers a larger target-emotion probability range with lower error than the baseline.The evaluation uses average target-emotion classification probability; lower error ranges indicate more stable control.Source: Figure 2 · p. 4, §4.3 · p. 4
4.4. Diversity of Emotional Samples
- EmoDiff is strongly preferred for diversity over MixedEmotion for Angry, Happy, and Surprise, but only narrowly for Sad.The authors attribute the Sad result mainly to duration variation in the autoregressive MixedEmotion model.Source: Figure 3 · p. 4, §4.4 · p. 4
5. CONCLUSION
The conclusion presents soft-label guidance as a direct way to control emotional TTS intensity and reports gains over baselines in quality, controllability, and diversity.
5. CONCLUSION
- EmoDiff defines emotion intensity as a weighted sum of a specific emotion and Neutral, then uses soft-label guidance to control intensity directly.The method extends classifier guidance beyond one-hot emotion labels.Source: §5 · p. 4
- Subjective and objective evaluations show EmoDiff outperforming baselines in TTS quality, intensity controllability, and sample diversity.Source: §5 · p. 4
- Applying soft-label guidance to more complicated natural emotions is left for future work.Source: §5 · p. 4