Source-linked AI summary
Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset
Kun Zhou, Berrak Sisman, Rui Liu, Haizhou Li
TL;DR
Emotional voice conversion needs to transfer prosody while preserving linguistic content and speaker identity, but fixed discrete styles limit existing encoder-decoder systems. The paper proposes a VAW-GAN framework conditioned on pretrained SER deep emotional features to transfer seen and unseen styles, and introduces the multilingual, multispeaker ESD dataset. DeepEST achieves competitive results for both seen and unseen emotions over the baseline.
Problem
Existing emotional voice-conversion systems rely on fixed conversion pairs or discrete emotion labels, while emotional prosody is difficult to capture with a simple labeling scheme.
Method
DeepEST uses a VAW-GAN encoder-decoder conditioned on deep emotional features extracted by a pretrained SER model to transfer emotional style during training and run-time inference.
Results
DeepEST achieves competitive results for both seen and unseen emotions over the baseline framework.
Takeaways & Limitations
The framework supports one-to-many emotional style transfer, while ESD provides a multilingual, multispeaker dataset for speech synthesis and voice conversion.
Abstract
from arXiv · showhide
Emotional voice conversion aims to transform emotional prosody in speech while preserving the linguistic content and speaker identity. Prior studies show that it is possible to disentangle emotional prosody using an encoder-decoder network conditioned on discrete representation, such as one-hot emotion labels. Such networks learn to remember a fixed set of emotional styles. In this paper, we propose a novel framework based on variational auto-encoding Wasserstein generative adversarial network (VAW-GAN), which makes use of a pre-trained speech emotion recognition (SER) model to transfer emotional style during training and at run-time inference. In this way, the network is able to transfer both seen and unseen emotional style to a new utterance. We show that the proposed framework achieves remarkable performance by consistently outperforming the baseline framework. This paper also marks the release of an emotional speech dataset (ESD) for voice conversion, which has multiple speakers and languages.
1. INTRODUCTION
Emotional voice conversion transfers emotional style while preserving speech content and speaker identity, but existing approaches face limits from parallel-data requirements, fixed conversion pairs, and discrete emotion labels. The paper proposes one-to-many emotional style transfer using continuous deep emotional features and releases the multilingual, multispeaker ESD corpus.
- Emotional voice conversion transfers an utterance’s emotional style while focusing on emotional-state transfer rather than speaker-identity conversion.Applications include expressive text-to-speech and conversational agents.
- Recent emotional voice-conversion systems using parallel data are constrained because such training data is not widely available in real-life applications.
- Nonparallel CycleGAN and autoencoder approaches typically support fixed conversion pairs rather than arbitrary target styles.
- The proposed emotional style transfer framework conditions the network on emotional style and supports one-to-many conversion, including unseen emotions at run time.
- The paper combines VAW-GAN, pretrained SER-based deep emotional features, emotional-element disentanglement and recomposition, and the multilingual, multispeaker ESD corpus.
2. ANALYSIS OF DEEP EMOTIONAL FEATURES
The paper motivates continuous deep emotional features because emotional prosody spans multiple attributes and is difficult to represent with a small set of discrete labels. A t-SNE visualization suggests that these features form clear emotion-related groups across speakers.
- Deep neural networks learn data-driven emotional features that are less dependent on human-crafted acoustic representations.
- Emotion can be represented categorically or continuously, with recent work exploring continuous spaces instead of finite emotion labels.
- Deep emotional features from four speakers formed clear emotion groups in a t-SNE visualization of utterances sharing the same content.The speakers included two male and two female speakers.
3. ONE-TO-MANY EMOTIONAL STYLE TRANSFER
DeepEST is a three-stage VAW-GAN framework that uses a pretrained SER model to describe emotional style continuously, trains an encoder-decoder to separate emotion from other information, and applies target-style features during conversion.
- Framework stages: DeepEST comprises emotion-descriptor training, VAW-GAN encoder-decoder training, and a run-time conversion interface.
- Stage I: Emotion Descriptor Training: The SER emotion descriptor extracts deep emotional features Φ from an input utterance using CNN, BLSTM, attention, and fully connected layers.The resulting utterance-level feature is used for emotion prediction and style description.
- Stage II: Encoder-Decoder Training with VAW-GAN: The encoder learns a latent representation z intended to retain phonetic and speaker information while being independent of emotion.
- Stage II: Encoder-Decoder Training with VAW-GAN: The decoder reconstructs spectral features from z while conditioning generation on deep emotional features and F0 values.The model uses adversarial training in which the discriminator distinguishes real from reconstructed features.
- Stage III: Run-Time Conversion Interface: At run time, DeepEST averages SER features from reference utterances and combines them with encoded source features and converted F0 to synthesize the target emotional style.The converted spectral features and F0 are synthesized with the WORLD vocoder.
4. EXPERIMENTS
The experiments evaluate DeepEST on seen and unseen emotion transfer using ESD, objective metrics, and listening tests. The dataset supports multilingual, multispeaker evaluation, while DeepEST is compared with a one-pair-at-a-time VAW-GAN-EVC baseline.
- 4.1. Emotional Speech Dataset (ESD): ESD contains 350 parallel utterances from 10 native English and 10 native Mandarin speakers, covering five emotions.Each language has five male and five female speakers; recordings are sampled at 16 kHz and stored in 16 bits.
- 4.2. Experimental Setup: Experiments use four English speakers and evaluate neutral-to-happy, neutral-to-sad, and unseen neutral-to-angry conversion.For each speaker, the ESD utterances are split into 300 training, 30 reference, and 20 test utterances.
- 4.2. Experimental Setup: DeepEST extracts WORLD-vocoder speech features and conditions its decoder on deep emotional features, a latent representation, and F0.The deep emotional feature is 256-dimensional, the latent representation is 128-dimensional, and F0 is used as a controllable attribute.
- 4.3. Objective Evaluation: DeepEST outperforms VAW-GAN-EVC for all seen emotions and remains comparable for unseen angry emotion in objective evaluation.The baseline requires three separately trained models for all conversion pairs, whereas DeepEST performs all emotion mappings within one model.
- 4.4. Subjective Evaluation: In subjective tests, DeepEST achieves better speech-quality results than the baseline for seen and unseen combinations, while emotion-similarity results vary by emotion.DeepEST outperforms the baseline for neutral-to-sad similarity, whereas the baseline performs better for neutral-to-happy; unseen angry remains comparable.
5. CONCLUSION
The paper proposes a one-to-many emotional style transfer framework based on VAW-GAN without parallel data and introduces the ESD dataset. DeepEST uses continuous SER-derived emotional features and achieves competitive seen and unseen emotion-transfer results over the baseline.
- 5. CONCLUSION: DeepEST performs one-to-many emotional style transfer without requiring parallel training data.The framework conditions generation on deep emotional features and F0 values.
- 5. CONCLUSION: DeepEST achieves competitive results for both seen and unseen emotions compared with the baseline framework.The conclusion presents these results as validating the proposed framework’s effectiveness.
- 5. CONCLUSION: The paper introduces ESD, a new emotional speech dataset for speech synthesis and voice conversion.