EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis

1. Introduction

• Reference-based style transfer and emotion representations have been explored, but achieving intensity-controllable emotional TTS remains challenging.Prior approaches include style tokens, multiscale prosody embeddings, SER or ASR features, scaling, interpolation, quantization, and relative attributes.Source: §1 · p. 1
• Previous methods predominantly synthesize a limited number of emotion types, and few explore correlations or interactions between emotions.The introduction notes that humans can experience multiple emotional states simultaneously and describes primary emotions as sources of mixtures.Source: §1 · p. 1
• Existing mixed-emotion approaches face practical or modeling challenges, including costly data collection, difficulty obtaining suitable reference audio, and noisy-audio classifier guidance.The introduction also reports noticeable quality degradation for MixedEmotion’s manually defined emotion attribute mixtures.Source: §1 · p. 1
• EmoMix combines diffusion-predicted noises conditioned on different emotions in one runtime sampling process and mixes Neutral with a primary emotion to control intensity.The pre-trained SER model supplies emotion embeddings; Figure 1 marks the extended mixed-emotion sampling path in green.Source: §1 · p. 1, Figure 1 · p. 2
• EmoMix uses high-dimensional SER embeddings for seen and unseen primary-emotion conditioning and is reported to preserve naturalness and quality while mixing and controlling emotions.The authors state that generated samples exhibit high naturalness and quality.Source: §1 · p. 1

2.1. Preliminary on Score-based Diffusion Model

• The diffusion framework maps data toward standard normal noise and estimates scores with an emotion-conditioned noise predictor for the reverse process.The predictor is trained across diffusion timesteps; text and speaker conditions are also used, though the paper’s simplified notation omits them.Source: §2.1 · p. 2

2.2. Emotion Conditioning with SER

• A pretrained SER model extracts a continuous emotion embedding from reference audio, while wav2vec 2.0 supplies speaker conditioning to the emotion- and speaker-conditioned duration prediction and denoiser.The denoiser iteratively refines linguistic representations into mel-spectrograms.Source: §2.2 · p. 2
• An additional SER-based style reconstruction loss compares Gram-matrix features of reference and synthesized mel-spectrograms to encourage similar style.The Gram matrices come from a 3-D CNN feature map in the SER model.Source: §2.2 · p. 2

2.3. Run-time Emotion Mixing

• At runtime, EmoMix denoises first with a base emotion, then combines emotion-conditioned noises over a later interval to create weighted mixtures without directly modeling target-mixture conditions.The method averages embeddings from samples of the same primary emotion to reduce instability from a single reference.Source: §2.3 · p. 3
• Mixing noise from Neutral and a target primary emotion with varying weights interpolates between them to control primary-emotion intensity.The interpolation is applied at runtime during the reverse process.Source: §2.3 · p. 3

3.1. Experiment Setup

• The experiments use English ESD speech from 10 speakers with five emotions, leaving Angry unseen; the SER model is trained on IEMOCAP.The estimator uses GradTTS-style U-Net and linear attention modules, trains for 1M steps, and uses HiFi-GAN as vocoder.Source: §3.1 · p. 3

3.2. Evaluation of Emotional Speech

• EmoMix surpasses baselines in SMOS, has comparable MOS and the lowest MCD, and remains satisfactory on unseen Angry synthesis.The comparison includes MixedEmotion and EmoDiff; Table 1 covers single-primary-emotion synthesis.Source: §3.2 · p. 3, Table 1 · p. 3

3.3. Ablation Study

• Ablations indicate that continuous SER emotion representations matter, while removing style reconstruction loss reduces CSMOS.The study compares SER use and style reconstruction loss in Table 2.Source: §3.3 · pp. 3–4, Table 2 · p. 4

3.4. Evaluation of Mixed Emotion

• As mixed-in Surprise noise increases, Surprise probability rises while Happy or Sad probability falls; EmoMix causes only slight speech-quality degradation versus MixedEmotion's obvious degradation.The study also reports CMOS comparisons across 10 mixed-emotion types; mixed-in intensity settings are capped at γ=0.8 to preserve the base emotion.Source: §3.4 · p. 4, Figure 2 · p. 4, Table 3 · p. 4
• Human evaluation indicates that Neutral mixed with a target primary emotion yields samples that accurately reflect the desired intensity.The results also show diverse primary emotions from Neutral and primary-emotion mixing.Source: §3.4 · p. 4

• Evaluations demonstrate that EmoMix can generate speech with various mixed emotions.Source: §4 · p. 4
• EmoMix manually combines noise at runtime to produce different emotional mixtures without explicitly modeling mixed emotions.Source: §4 · p. 4
• EmoMix is a controllable emotional TTS framework for exploring mixed-emotion synthesis and intensity control.Source: §4 · p. 4