Source-linked AI summary
EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech
Daria Diatlova, Vitaly Shutov
TL;DR
Emotional TTS must convey emotion and other speech characteristics that are absent from text alone, including uneven emotional emphasis across a sentence. EmoSpeech extends FastSpeech2 with conditioning that varies emotional strength across tokens. The authors report improved quality over prior FastSpeech2 emotional-speech extensions and higher MOS and emotion-recognition accuracy than an existing implementation.
Problem
Emotional TTS must convey emotion and other speech characteristics absent from simple text input, including uneven emotion across a text sequence.
Source: §1 · p. 1Method
EmoSpeech extends FastSpeech2 with conditioning that lets emotion contribute to each token at varying intensity.
Source: §1 · p. 1Results
Takeaways & Limitations
Conditional cross-attention selects different phonemes to accent for different emotions, and the authors report more natural-sounding speech than the baseline.
Source: §3.5 · p. 3, §6 · p. 6The test set contains only 25 utterances, limiting confidence in MOS validity, and fixed utterances and annotators may risk biasing emotion evaluation.
Source: §5 · p. 6
Abstract
from arXiv · showhide
State-of-the-art speech synthesis models try to get as close as possible to the human voice. Hence, modelling emotions is an essential part of Text-To-Speech (TTS) research. In our work, we selected FastSpeech2 as the starting point and proposed a series of modifications for synthesizing emotional speech. According to automatic and human evaluation, our model, EmoSpeech, surpasses existing models regarding both MOS score and emotion recognition accuracy in generated speech. We provided a detailed ablation study for every extension to FastSpeech2 architecture that forms EmoSpeech. The uneven distribution of emotions in the text is crucial for better, synthesized speech and intonation perception. Our model includes a conditioning mechanism that effectively handles this issue by allowing emotions to contribute to each phone with varying intensity levels. The human assessment indicates that proposed modifications generate audio with higher MOS and emotional expressiveness.
1. Introduction
EmoSpeech extends FastSpeech2 for emotional speech synthesis while retaining its inference-speed advantage for high-load settings. Its per-token conditioning addresses uneven emotional strength across text, and the proposed model outperforms an existing FastSpeech2 extension in MOS and emotion-recognition accuracy.
1. Introduction
- FastSpeech2 is chosen because Transformer-based models can offer faster inference than some newer diffusion models, which matters in high-load environments.Source: §1 · p. 1
- The work addresses how uneven emotion distribution across a text sequence affects emotional speech synthesis.Source: §1 · p. 1
- EmoSpeech outperforms an existing FastSpeech2 extension [10] in MOS and emotion-recognition accuracy without added inference-speed latency.Source: §1 · p. 1
- Its conditioning mechanism lets emotion contribute to each token with varying strength, accounting for the relationship between intonation and emotion.Source: §1 · p. 1
2. Related Work
Prior emotional TTS approaches condition on categorical labels, reference speech, or textual descriptions; this work selects categorical conditioning for its fixed speaker and emotion setting. The related techniques discussed include conditional layer normalization and adversarial training with feature matching.
2. Related Work
- Earlier work includes style-control vectors [11], emotion embeddings in recurrent models [10], and Global Style Tokens for utterance-level conditioning.Source: §2 · p. 1
- Emotional TTS conditioning is grouped into categorical emotion labels, reference speech, and textual descriptions.Source: §2 · p. 1
- The paper selects categorical labels because its target setting uses a small fixed set of speakers and emotions in a high-load product environment.Source: §2 · p. 1
- Expressive FastSpeech2 adds an emotion embedding to the encoder output, making it a relevant speed-compatible comparison for this work.Source: §2 · p. 2
2.1. Technical details of adapted modifications
- AdaSpeech's Conditional Layer Norm conditions normalization scale and bias on speaker embeddings; AdaSpeech4 applies it in both encoder and decoder.Source: §2.1 · p. 2
- GANSpeech applies a JCU discriminator in a second adversarial-training phase to address quality degradation after extensive conditioning.Source: §2.1 · p. 2
- GANSpeech also uses feature matching between real and generated mel-spectrogram discriminator features, alongside reconstruction and adversarial losses.Source: §2.1 · p. 2
3. Model Description
EmoSpeech extends FastSpeech2 with speaker- and emotion-conditioning components, including attention that reweights tokens by emotion. The model also uses adversarial training to address quality degradation that may arise with multi-speaker conditioning.
3.1. FastSpeech2
- FastSpeech2 encodes input tokens, predicts pitch, energy, and durations, then decodes the length-regulated representations into a mel spectrogram.A vocoder later upsamples the mel spectrogram to a waveform.Source: §3.1 · p. 2
3.2. Conditioning Embedding
- EmoSpeech concatenates speaker and emotion embeddings into a conditioning vector and adds it to the encoder output before the variance adaptor.The embedding is expanded along the sequence-length dimension.Source: §3.2 · p. 2
3.3. eGeMAPS Predictor
- The eGeMAPS predictor adds utterance-level acoustic descriptors to the variance adaptor; the initial design uses the 80th and 50th percentiles of logarithmic F0.It predicts utterance-level values, unlike the token-level pitch and energy predictors.Source: §3.3 · p. 2
3.4. Conditional Layer Norm
- Conditional Layer Norm replaces traditional Layer Norm in EmoSpeech’s encoder and decoder, conditioned on concatenated speaker and emotion embeddings.The paper adopts this approach for emotional speech synthesis after its use in zero-shot synthesis.Source: §3.4 · p. 3
3.5. Conditional Cross Attention
- Conditional Cross-Attention reweights text tokens according to the target emotion, replacing uniform addition of the conditioning embedding.Its attention uses the speaker-emotion vector as conditioning and is implemented with multi-head logic in the encoder and decoder.Source: §3.5 · p. 3
3.6. Adversarial Training
- Adversarial training applies a conditional JCU discriminator alongside EmoSpeech in one training phase, using concatenated speaker and emotion embeddings for conditioning.The training objective also adds a feature-matching term for stabilization and an MSE loss for eGeMAPS prediction.Source: §3.6 · p. 3
4. Data Preprocessing
The preprocessing and feature-selection pipeline prepares the ESD English subset for multi-speaker, multi-emotion synthesis and evaluates reduced eGeMAPS feature sets. Annotators slightly favored the model trained to predict two features.
4. Data Preprocessing
- The English ESD subset provides multiple speakers and five emotion labels, with 11,015 words and 997 unique lexical words.The labels are Neutral, Angry, Happy, Sad, and Surprised.Source: §4 · p. 3
- For MOS estimation, the authors sampled a speaker- and emotion-balanced, non-parallel subset of 14 unique utterances from the test data.They selected one utterance for each of five speakers and five emotions, with some utterance IDs shared.Source: §4 · p. 4
- Preprocessing extracts phonemes, durations, pitch, energy, and utterance-level eGeMAPS features, then normalizes the acoustic features.The phonemes and durations come from MFA; pitch comes from pyworld, and eGeMAPS features from openSMILE.Source: §4 · p. 4
- To balance feature count and emotion-classification accuracy, the authors set a 60% accuracy threshold and retained seven RFE-selected eGeMAPS features.Classification accuracy increased with additional features, but its relative gain diminished.Source: §4 · p. 4
- The model trained to predict 2 eGeMAPS features was preferred more often by annotators, with a slight advantage.The preference survey compared the baseline with models predicting from 1 to 7 features.Source: §4 · p. 4, Figure 4 · p. 4
5. Experiments and Results
Experiments evaluate sequential FastSpeech2 modifications through automatic and human assessments. EmoSpeech improves reported synthesis quality, while emotion-accuracy, naturalness-preference, and attention analyses add qualifications and context.
5. Experiments and Results
- The added modifications increase model size by 15% over the configured Expressive FastSpeech2 baseline without a notable inference-speed decrease.Source: §5 · p. 5
- In MOS and NISQA evaluations, EmoSpeech outperforms the baseline and intermediate modifications, with quality close to reconstructed utterances.The test-set results report similar quality to reconstructed utterances.Source: Table 2 · p. 5, §5 · p. 5
- CCA attention weights vary by emotion and sentence position: surprise rises at sentence ends, sadness has mid-sentence spikes, and anger spikes early and mid-sentence.The authors report that these emotion-specific patterns generalize across the balanced test set.Source: §5 · p. 5
- Modification #3 has the highest accuracy across emotions, while baseline anger accuracy is highest amid artifact-driven anger judgments.The authors expect an accuracy drop after adversarial training, which was added to prevent quality degradation and slightly smooth sharp intonation peaks.Source: §5 · p. 5, Table 3 · p. 6
- EmoSpeech is preferred by 10% more annotators than the baseline in pairwise naturalness judgments given a target emotion.Figure 5 reports annotators’ percentage preferences when comparing generated utterances.Source: §5 · p. 5, Figure 5 · p. 6
- Human-evaluation validity is bounded by a 25-utterance test subset and fixed annotators and utterances, which may make perfect surprise accuracy reflect distinctiveness rather than best expression.Source: §5 · p. 6
6. Conclusion
EmoSpeech extends FastSpeech2 for emotional speech while retaining its speed. Experiments report improved model quality over prior FastSpeech2 extensions, with emotion-dependent phoneme accents contributing to more natural-sounding speech.
6. Conclusion
- EmoSpeech’s modifications enhance model quality and outperform the previous FastSpeech2 extension for emotional TTS, Expressive FastSpeech2.The paper also reports that EmoSpeech remains as fast as the original FastSpeech2.Source: §6 · p. 6
- Conditional cross-attention accounts for uneven emotion distribution across a sentence, and different emotions lead the model to accent different phonemes.The authors report that these emotion-specific accents make EmoSpeech sound more natural than the baseline.Source: §6 · p. 6
- EmoSpeech extends FastSpeech2 to synthesize emotional speech while conditioning on a given emotion and retaining the original model’s speed.Source: §6 · p. 6