Source-linked AI summary
Emotional Voice Conversion: Theory, Databases and ESD
Kun Zhou, Berrak Sisman, Rui Liu, Haizhou Li
TL;DR
Emotional voice conversion requires data and methods that preserve linguistic content and speaker identity while changing emotion, but suitable large-scale, multi-speaker databases were lacking. This paper reviews the field, releases the ESD database, and evaluates state-of-the-art systems, finding that ESD supports emotional voice conversion studies under controlled recording conditions.
Problem
Emotional voice conversion needs suitable large-scale, multi-speaker emotional speech data, but existing databases were not designed for this purpose.
Method
The paper reviews emotional voice conversion research and databases, releases ESD, and evaluates state-of-the-art emotional voice conversion frameworks on it.
Results
Both objective and subjective evaluations confirm that ESD provides a shared emotional voice conversion task under a controlled environment.
Takeaways & Limitations
ESD is publicly available for multi-speaker and cross-lingual emotional voice conversion studies.
Takeaways & Limitations
Existing emotional speech databases vary in lexical content and recording environments, limiting comparability for controlled emotional voice conversion studies.
Abstract
from arXiv · showhide
In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 hours of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release.
1. Introduction
Emotional voice conversion changes an utterance’s emotional state while preserving linguistic information and speaker identity. The field needs suitable large, multi-speaker emotional speech data because existing voice-conversion databases do not meet that need.
- Motivation: Emotional voice conversion converts an utterance from one emotional state to another while preserving linguistic information and speaker identity.It targets applications such as acting or disguising emotions and addresses the limited emotional expressiveness of speech synthesis.
- Motivation: Unlike speaker voice conversion, emotional conversion must model both spectrum and prosody because emotion is supra-segmental and complex.Framewise spectral mapping alone is insufficient; segmental-level prosodic dynamics also require treatment.
- Prior work: Early emotional voice conversion used GMM, sparse representation, and HMM-GMM-F0 methods, while recent work includes DNN, highway neural network, and DBLSTM approaches.The overview traces a progression from statistical techniques to deep-learning-based systems.
- Research gap: Existing databases such as VCTK, CMU-Arctic, and VCC were designed for speaker voice conversion rather than emotional voice conversion.The paper reports no large-scale, multi-speaker, open-source emotional speech database at the time of writing.
- Contribution: The paper reviews emotional voice conversion research and databases, releases the ESD database, and evaluates state-of-the-art systems on it.ESD contains 29 hours from 10 native English and 10 native Chinese speakers across five emotion categories, with studio recordings.
2. Overview of emotional voice conversion
Emotional voice conversion models emotion through acoustic and linguistic representations, using parallel or non-parallel data and methods ranging from feature mapping to domain translation. The overview emphasizes the joint importance of spectrum and prosody, while ASR-derived PPGs can improve generalization across speakers and emotions.
- Motivation: Emotional voice conversion supports expressive text-to-speech, speech emotion recognition, and conversational agents, but data-driven systems depend heavily on training data.The paper frames database availability as central to progress in these applications.
- Emotion representation: Emotion is represented categorically or dimensionally, while learned deep features provide another route for representing emotional style.Categorical labels are straightforward but do not model subtle emotional differences; dimensional representations use properties such as arousal, valence, and dominance.
- Feature extraction and modeling: Parallel emotional voice conversion typically extracts features, aligns paired sequences, and maps source features to target features.Common spectral features include MCC, LPCC, and LSF, while prosodic features include pitch, energy, duration, and F0.
- Feature extraction and modeling: Frame-level systems use statistical or neural models to map spectrum and prosody, but frame-level mapping does not explicitly handle duration.F0 is treated as an essential prosodic component spanning linguistic and emotional intonation across multiple time scales.
- Emotional domain translation: Non-parallel methods address the practical difficulty and expense of collecting paired speech data.Domain-translation approaches include CycleGAN and StarGAN, with StarGAN extending translation to multiple emotion domains.
- Leveraging TTS or ASR systems: PPGs extracted by speaker-independent, emotion-independent ASR provide phonetic information that improves generalization with multi-speaker and multi-emotion data.The PPG-based framework uses a pre-trained SI-ASR system to derive the auxiliary representations.
3. Databases in emotional voice conversion research
Existing emotional speech databases provide useful resources but often remain limited in lexical, language, speaker, and recording diversity. These constraints affect their suitability for speaker-independent, cross-lingual, and synthesis-oriented emotional voice conversion.
- Overview: Existing databases are surveyed as resources for developing and evaluating emotional voice conversion systems.The overview is presented as broad but not exhaustive, covering widely used emotional speech databases.
- Lexical variability: Limited lexical variability is common, with some databases covering emotion categories using only a small set of sentences.RAVDESS, for example, uses two sentences, while other databases provide only limited sentence variety.
- Language variability: 19 listed databases show imbalanced language coverage, including nine databases containing only English speech.The imbalance makes learning cross-lingual emotion representations difficult.
- Speaker variability: Many databases lack sufficient speaker variability for speaker-independent emotional voice conversion.Examples include datasets with four speakers, five speakers, or only male speakers.
- Confounding factors: Some databases with many speakers include confounding factors such as accents, laughter, sighs, or uncontrolled recording conditions.These factors are described as unwanted for emotional voice conversion database design.
- Recording environment: Large and diverse databases such as IEMOCAP, MSP-IMPROV, and CHEAVD can contain overlapping speech or external noise that limits their suitability for synthesis.Their recording setups may affect synthesized-audio performance.
4. ESD database
ESD was created as an open emotional speech resource with broad speaker and lexical coverage. The paper presents its design, analyzes the database, and evaluates it using emotion recognition and learned emotion embeddings.
- ESD database: ESD aims to provide the research community with a large emotional speech database offering sufficient speaker and lexical coverage.The paper also reports database analysis and evaluation through speech emotion recognition and emotion-embedding visualization.
4.1. Addressing the research need
ESD is designed to address limitations in existing emotional speech databases while retaining parallel, acted utterances. Its design targets lexical, language, speaker, confounding-factor, and recording-environment gaps.
- Design rationale: Acted, parallel emotional speech is chosen so speakers express different emotions using the same linguistic content.This design follows common practice in emotional speech synthesis.
- Research need: ESD explicitly targets gaps in lexical variability, language variability, speaker variability, confounding factors, and recording environment.The database is presented as addressing five weaknesses identified in existing resources.
- Lexical variability: 350 parallel utterances are proposed to improve lexical coverage and enable comparisons of the same sentence across speakers and emotions.Parallel content supports controlled comparison while increasing sentence variety.
- Speaker variability: The proposed speaker set contains 10 Chinese and 10 English speakers with balanced gender representation.The authors expect this variety to facilitate speaker-independent emotional voice conversion studies.
4.2. Design
ESD contains parallel emotional utterances from native English and Chinese speakers, organized by emotion and data split. Its design also records lexical statistics for both languages.
- Speakers: 20 native speakers contribute to ESD: 10 English speakers and 10 Chinese speakers.Each speaker contributes 350 utterances.
- Emotion categories: ESD covers five emotion categories: Neutral, Happy, Angry, Sad, and Surprise.The database is made available for research purposes upon request.
- Data partition: Each speaker-emotion set is divided into evaluation, test, and training subsets.The first 20 utterances form evaluation, the next 30 form test, and the remainder form training.
- Directory organization: The directory organizes 20 speaker folders, five emotion folders per speaker, and three split folders within each emotion folder.Transcriptions are stored in each speaker’s folder.
4.3. Statistics
ESD provides parallel Chinese and English speech with broad lexical coverage and measurable duration and F0 patterns across emotions. Its statistics support multilingual analysis of emotional speech.
- Lexical variability: 20,025 Chinese characters and 939 unique characters are represented, averaging 11.5 characters per utterance.
- Lexical variability: 11,015 English words and 997 unique lexical words are represented, averaging about 6.31 words per utterance.The selected scripts aim to provide broad lexical coverage while reducing repetition.
- Duration: Sad utterances tend to have higher duration means and variances than happy utterances in both languages.The reported pattern motivates examining shared emotional-expression codes across languages.
- Duration: Average utterance and word durations are 2.76 s and 0.44 s for English, versus 3.22 s and 0.28 s for Chinese.Chinese utterances are longer on average, while Chinese character duration is shorter than English word duration.
- Fundamental frequency (F0): Female speakers typically have higher F0 means and standard variances than male speakers, while happy and surprise show the highest F0 patterns.Angry F0 is generally above neutral and sad but has lower variance than happy and surprise.
4.4. Emotion classification
The ESD emotion labels are evaluated with speaker-independent speech emotion recognition and deep-feature visualization. Recognition accuracy is high for both languages, supporting consistent emotional rendering.
- Emotion classification: The SER experiment trains separate Chinese and English systems using frame-level openSMILE features extracted from 25 ms windows with 10 ms shifts.The extracted vectors are 312-dimensional and include zero-crossing rate, voicing probability, MFCCs, and mel-spectrum features.
- Emotion classification: 92.0% Chinese and 89.0% English SER accuracy indicate high emotion consistency in ESD speech.The systems use five-class softmax classification after LSTM processing.
- Emotion classification: Each SER model uses an LSTM, a 256-node ReLU fully connected layer, dropout, and a five-output softmax classifier.
- Deep emotional features: Deep emotional features from 20 test utterances per emotion are visualized with t-SNE in two dimensions.The visualization is used to examine whether intermediate SER activations characterize emotion states.
5. Emotional voice conversion on ESD
The paper benchmarks CycleGAN-EVC and VAWGAN-EVC on ESD using objective and listening evaluations. CycleGAN-EVC generally performs better, especially for speech quality, but requires substantially more training time.
- Experimental setup: CycleGAN-EVC learns pair-wise emotion-domain translation, whereas VAWGAN-EVC uses emotion codes to control conversion.Both frameworks are implemented as reference systems on ESD.
- Experimental setup: Experiments use one male ESD speaker and convert neutral speech to angry, happy, sad, and surprise with 350 parallel utterances per emotion.
- Objective evaluation: Both systems achieve lower MCD than Zero effort for all evaluated conversion pairs, while CycleGAN-EVC consistently outperforms VAWGAN-EVC.MCD is computed from 24-dimensional mel-cepstral coefficients; lower MCD indicates better conversion.
- Subjective evaluation: CycleGAN-EVC outperforms VAWGAN-EVC in MOS speech quality for all emotion conversion pairs.MOS evaluates preservation of linguistic information and speaker identity together with converted-speech naturalness.
- Subjective evaluation: CycleGAN-EVC is preferred for emotion similarity in Neu-Sur, Neu-Ang, and Neu-Hap, while results are comparable in Neu-Sad.The XAB test compares each converted sample with the reference target emotion.
- Discussion: CycleGAN-EVC takes nearly 72 h to train versus about 7 h for VAWGAN-EVC, while VAWGAN-EVC supports controllable many-to-many emotion conversion.The comparison uses the same training-data amount and hardware setup.
- Limitations: Listener assessments may mismatch the speaker’s intended emotion, a discrepancy reserved for future study.
- Discussion: Objective and subjective evaluations support ESD as a shared emotional voice conversion task under controlled recording conditions.The speech samples are publicly available.
6. Other voice conversion and text-to-speech on ESD
ESD extends beyond emotional voice conversion to speaker voice conversion and text-to-speech research. Its multilingual, multi-speaker, publicly usable design addresses gaps left by emotion-free or private databases.
- Scope: ESD supports other voice conversion and text-to-speech studies in addition to emotional voice conversion.
- Voice conversion: Emotional voice conversion requires attention to prosody as well as spectrum because emotion involves their interplay.This differs from speaker voice conversion, which has mostly focused on spectrum while treating prosody as speaker-independent.
- Voice conversion: Speaker voice conversion databases are abundant, but their emotion-free speech is unsuitable for emotional voice conversion.ESD’s multi-speaker and cross-lingual nature enables studies beyond those existing databases.
- Text-to-speech: Emotional text-to-speech research often relies on private internal databases, limiting related research.
- Text-to-speech: ESD fills the emotional text-to-speech data gap, and successful frameworks have already been built and evaluated on it.Reported examples include joint emotional voice conversion and text-to-speech training and reinforcement-learning-based emotional TTS.
7. Conclusion
The paper surveys recent emotional voice conversion research and databases, releases the ESD database, and evaluates state-of-the-art systems on it as a reference benchmark.
- The paper provides a comprehensive overview of recent emotional voice conversion research and existing emotional speech databases.
- The authors publicly release the ESD database, described as one of the largest emotional speech databases in the literature.
- Experiments on ESD establish a reference benchmark for state-of-the-art emotional voice conversion studies.
CRediT authorship contribution statement
The authors contributed across study design, data acquisition, analysis, manuscript drafting, and critical revision.
- Kun Zhou contributed to study conception and design, data acquisition, data analysis or interpretation, manuscript drafting, and critical revision.
- Berrak Sisman contributed to study conception and design, data acquisition, and critical manuscript revision.
- Rui Liu contributed to manuscript drafting and critical revision.
- Haizhou Li contributed to critical manuscript revision.