Source-linked AI summary
Emotional End-to-End Neural Speech Synthesizer
Younggun Lee, Azam Rabiee, Soo-Young Lee
TL;DR
Tacotron's exposure bias and irregular attention alignment pose challenges for emotional speech synthesis. The paper modifies the model with attention-alignment changes, including context-vector input and a residual connection. The authors report sharper, clearer alignments and describe training and generating speech for emotion labels, while providing no quantitative results.
Problem
The original Tacotron can accumulate prediction errors at test time, disrupting attention alignment, and the authors observed irregular alignments in the middle of long inputs.
Source: §2.1 · p. 2, §4 · p. 3Method
The authors modify emotional Tacotron with monotonic attention, semi-teacher-forced training, prior context input to the attention RNN, and a residual connection in the CBHG text encoder.
Source: §2.1 · p. 3, §4.1 · p. 3, §4.2 · p. 4Results
The authors observed sharper and clearer attention alignments after applying the residual connection with the Section 4.1 method.
Source: §4.2 · p. 4Takeaways & Limitations
The authors report that their modified Tacotron can train and generate speech for given emotion labels.
Source: §3 · p. 3, §5 · p. 4The authors lacked a suitable Korean emotion classifier and MOS testers, so they provided generated speech instead of quantitative results.
Source: §3 · p. 3
Abstract
from arXiv · showhide
In this paper, we introduce an emotional speech synthesizer based on the recent end-to-end neural model, named Tacotron. Despite its benefits, we found that the original Tacotron suffers from the exposure bias problem and irregularity of the attention alignment. Later, we address the problem by utilization of context vector and residual connection at recurrent neural networks (RNNs). Our experiments showed that the model could successfully train and generate speech for given emotion labels.
1 Introduction
Tacotron is an end-to-end neural text-to-speech model trained from scratch on text–audio pairs. This paper modifies it to generate emotional speech and addresses its difficulty generating the middle part of speech.
1 Introduction
- Tacotron is an end-to-end sequence-to-sequence TTS model trained from scratch on text–audio pairs [11].The paper also describes it as using attention [2].Source: §1 · p. 1
- Tacotron's results are described as comparable to previous approaches such as concatenative models.The paper notes that Tacotron produced natural speech samples with competitive mean opinion score.Source: §1 · p. 1
- The paper modifies Tacotron to generate emotional speech and reports that the original model had difficulty generating speech's middle part.The modifications aim to facilitate information flow in Tacotron.Source: §1 · p. 1
2 Emotional speech synthesizer
The emotional Tacotron conditions speech generation on an emotion embedding. Its decoder uses attention, while monotonic attention and semi-teacher-forced training address alignment irregularity and exposure bias.
2 Emotional speech synthesizer
- Emotional Tacotron adds a learned emotion embedding to the encoder–decoder model, which generates Mel spectrogram frames before a post-processor synthesizes speech waveforms.The embedding provides variation for specifications such as emotion or personality.Source: §2 · p. 1
2.1 Attention-based emotional decoder
- The decoder uses a pre-net, an attention RNN, and two residual decoder RNN layers to predict r Mel-spectrogram frames per time step.It starts from an all-zero GO frame and feeds the previous time frame to the pre-net.Source: §2.1 · p. 2
- During training, Tacotron's ground-truth previous-frame input is unavailable at test time, where accumulated prediction noise can disrupt attention alignment, especially for long outputs.The authors illustrate attention loss around decoder time step 30 in the original model.Source: §2.1 · p. 2
- Monotonic attention constrains alignment to a monotonic pattern, and the authors report a cleaner alignment than the original Tacotron's.They also observed a correlation between output quality and alignment cleanness and sharpness.Source: §2.1 · p. 3
- Semi-teacher-forced training feeds the decoder the average of the ground-truth and predicted previous spectrogram frames, 0.5(yt−1 + ŷt−1).The authors use this mixture to add noise during training, reflecting the noisy generated input used at test time.Source: §2.1 · p. 3
3 Experiments
The model was trained on Korean speech recordings with emotion labels. The authors report that they lacked suitable Korean emotion-classifier and MOS testers, so they did not provide quantitative results.
3 Experiments
- The training data comprised Korean text–audio–emotion pairs read by a female actor in six emotions: neutral, angry, fear, happy, sad, and surprise.Scripts were the same across emotions except happy; the authors observed that the different script did not affect generated happy speech.Source: §3 · p. 3
- The processed dataset contained 21 hours of speech, using sequences under 200 characters and audio shorter than 8.7 seconds after silence trimming.The authors state that silence trimming was crucial for training.Source: §3 · p. 3
- Without a suitable Korean emotion classifier and MOS testers, the authors provided generated speech online instead of quantitative results.The speech was uploaded to GitHub.Source: §3 · p. 3
4 Discussion on prediction of attention alignment
The authors connect attention-alignment irregularities to information flow in the encoder and attention RNN. They add prior context to the attention RNN and a residual path to the text encoder; they report sharper, clearer alignments.
4.1 Utilization of Context Vector
- The attention RNN receives the previous context vector, a weighted sum of text encodings, alongside its usual input to provide information about previously attended text.The authors note that pronunciation of a character can span multiple spectrogram frames, so attention can stay on similar text across decoder steps.Source: §4.1 · p. 3
4 Discussion on prediction of attention alignment
- The authors observed irregular middle-sequence alignments on long inputs and conjectured that limited CBHG bi-RNN hidden-state capacity contributed to them.The hidden state must represent both current-step text and long-term dependencies.Source: §4.2 · p. 4
4.2 Residual Connections in CBHG
- The CBHG output adds the text input to the bi-directional RNN output, providing a residual connection that carries current-step information.The authors say this leaves the hidden state less congested and helps encode text information.Source: §4.2 · p. 4
- The combined residual-connection method and Section 4.1 method produced sharper, clearer attention alignments compared with the original Tacotron.Figure 3 contrasts original Tacotron alignment with the model using context vector and residual connection.Source: §4.2 · p. 4, Figure 3 · p. 4
5 Conclusion
The conclusion presents modified Tacotron as an end-to-end emotional speech synthesizer and links generated-speech quality to attention-alignment sharpness and cleanness. It also identifies phase handling and slow Griffin–Lim reconstruction as current concerns.
5 Conclusion
- The modified end-to-end Tacotron takes character sequences and desired emotions as input and generates corresponding speech waveforms.Source: §5 · p. 4
- Generated-speech quality is highly correlated with attention-alignment sharpness and cleanness, motivating tricks to improve alignment.Source: §5 · p. 4
- The model currently does not account for spectrogram phase, which may affect the intelligibility of emotional speech.Source: §5 · p. 4
- The authors identify slow Griffin–Lim reconstruction as a synthesis bottleneck and aim to produce dynamic TTS with different personalities.They are also investigating improvements to quality or generation speed.Source: §5 · p. 4