Source-linked AI summary
Transfer Learning for Improving Speech Emotion Classification Accuracy
Siddique Latif, Rajib Rana, Shahzad Younis, Junaid Qadir, Julien Epps
TL;DR
Speech emotion recognition systems that work on a single corpus can lose performance across corpora and languages, amid variation and scarce resources. This paper evaluates DBN transfer learning across five speech databases. DBNs outperform sparse autoencoders, and experiments report better-than-baseline accuracy when training includes multiple languages or a fraction of target data.
Problem
Speech emotion recognition systems that perform promisingly on the same corpus have been shown to perform poorly across different languages and conditions, while corpus diversity and scarce language resources complicate generalization.
Source: §1 · p. 1Method
Results
DBNs outperformed sparse autoencoders, and training with many languages or a small fraction of target data improved baseline accuracy in the reported experiments.
Source: §5 · pp. 3–4, §6 · p. 4Takeaways & Limitations
The findings support building robust speech emotion recognition systems using data from multiple languages, including for languages with limited or no data.
Source: §6 · p. 4
Abstract
from arXiv · showhide
The majority of existing speech emotion recognition research focuses on automatic emotion detection using training and testing data from same corpus collected under the same conditions. The performance of such systems has been shown to drop significantly in cross-corpus and cross-language scenarios. To address the problem, this paper exploits a transfer learning technique to improve the performance of speech emotion recognition systems that is novel in cross-language and cross-corpus scenarios. Evaluations on five different corpora in three different languages show that Deep Belief Networks (DBNs) offer better accuracy than previous approaches on cross-corpus emotion recognition, relative to a Sparse Autoencoder and SVM baseline system. Results also suggest that using a large number of languages for training and using a small fraction of the target data in training can significantly boost accuracy compared with baseline also for the corpus with limited training examples.
1. Introduction
Speech emotion recognition performs well when training and testing use the same corpus, but performance drops across languages and corpora. The paper investigates DBN transfer learning across five emotional speech databases to address this challenge.
1. Introduction
- Speech emotion recognition systems perform promisingly on the same corpus but not as well when languages, age groups, and conditions differ.The paper describes this as a real-application challenge.Source: §1 · p. 1
- Corpus variation in language, emotion type, and labeling, combined with scarce speech resources for many languages, makes generalizing from one corpus difficult.The passage notes that 389 languages account for 94% of the world’s population, yet adequate speech resources remain scarce even for those languages.Source: §1 · p. 1
- The impact of DBNs for cross-corpus and cross-language transfer remains underexplored, as does how to maximize transfer performance.Prior speech transfer research had focused on basic DNNs; DBNs are highlighted for their generalization power.Source: §1 · p. 1
- DBN transfer learning is investigated across five widely used emotional speech databases to improve cross-corpus emotion recognition.The authors report accuracy gains comparable to baseline using transfer learning.Source: §1 · p. 1
2. Related Work
Prior cross-corpus and cross-language emotion-recognition work explored SVMs, DNNs, and other transfer architectures, while DBN-based transfer remained rare. This paper uses DBNs for speech-emotion transfer, motivated by their generalization and feature-learning properties.
2. Related Work
- Prior cross-corpus work includes SVM studies, DaLSR, and sparse autoencoders, with reported comparisons to independent learning or other models.The cited studies used different corpora and experimental setups.Source: §2 · pp. 1–2
- Cross-acoustic transfer and speaker or language recognition used DNNs trained on speech data to transfer learned features to target domains.Those studies reported enhanced sound-event classification or performance gains.Source: §2 · p. 2
- DBNs are used for speech-emotion transfer because their generalization power and long-range feature learning are considered useful for cross-corpus and cross-language recognition.The paper also notes that RBMs, DBNs’ building blocks, are universal approximators.Source: §2 · p. 2
- Progressive Neural Networks have been applied to emotion, speaker, and gender tasks, whereas this paper focuses on emotion recognition alone [21].The related work also describes DaLSR and sparse-autoencoder approaches to cross-corpus emotion recognition.Source: §2 · p. 2
3. Experimental Setup
The experiments use five diverse speech corpora, binary valence labels, eGeMAPS acoustic features, and DBNs built from stacked RBMs. The DBNs are pretrained layer by layer before discriminative fine-tuning.
3.1. Speech Databases
- Five diverse corpora with different annotation schemes are evaluated using binary positive/negative valence mapping to make transfer comparisons consistent.The mapping is adopted from prior work and provided with corpus information in Table 1.Source: §3.1 · p. 2, Table 1 · p. 3
3.2. Speech Features
- The speech input uses the 88-feature eGeMAPS set, combining frequency, energy, spectral, cepstral, and dynamic information.Its components include statistics over low-level descriptors, temporal and unvoiced-segment features, loudness and pitch functionals, and additional dynamic and cepstral components.Source: §3.2 · p. 2
3.3. Deep Belief Networks
- A DBN stacks Restricted Boltzmann Machines, which are pretrained greedily layer by layer and then discriminatively fine-tuned.The RBM is described as a stochastic network with visible and hidden units and no within-layer connections.Source: §3.3 · p. 2
- The RBM’s joint distribution is defined over visible and hidden units, with connection weights and conditional probabilities described in the setup.The conditional probabilities use a sigmoid function; Table 1 presents corpus information and binary valence-label mappings.Source: §3.3 · p. 2, Table 1 · p. 3
- The selected DBN has three RBM layers with 1,000, 1,000, and 2,000 hidden units, using a 10^-3 learning rate and 500 epochs.This configuration was selected using cross-validation on validation data.Source: §3.3 · p. 3
4. Results
The section tests DBN-based emotion recognition across within-corpus and cross-corpus/language scenarios, including target-data and multi-language training.
4.1. Within Corpus Scheme
- With 75% training data and 25% unseen test data, DBN outperforms sparse AE across all databases in the within-corpus baseline.Figure 1 compares baseline accuracy using DBN and sparse AE on different databases.Source: §4.1 · p. 3, Figure 1 · p. 3
4.2. Language Tests
- DBN outperforms sparse AE in every tested cross-corpus language scenario.The tests used FAU-AIBO (German) or IEMOCAP (English) for training; FAU-AIBO evaluation included pairwise testing on OHM and MONT.Source: §4.2 · p. 3
4.3. Percentage of Target Data
- Including 10% to 80% target-dataset data in training significantly improves recognition rates over baseline for EMOVO, EMO-DB, and SAVEE.IEMOCAP and FAU-AIBO were used separately for training.Source: §4.3 · p. 3
4.4. Multi-language Training
- Joint IEMOCAP/FAU-AIBO training was tested across remaining corpora and within each source; leave-one-data-out performed better than baseline and two-language training.For FAU-AIBO, evaluation used OHM and MONT independently and averaged results; IEMOCAP evaluation used two sessions with two-fold cross-validation.Source: §4.4 · p. 3
5. Discussion
Leave-one-out training across languages reaches the highest reported accuracy, including on an Italian corpus tested from German and English data. Adding a fraction of target data can also beat baseline, though the outcome varies with the training corpus.
5. Discussion
- Leave-one-out training achieves the highest accuracy, including on EMOVO, where Italian emotions are predicted from German and English data.The authors suggest that training across a wide range of languages helps learn features useful for an unknown language.Source: §5 · p. 3, Figure 4 · p. 4
- Figure 2 shows recognition below baseline when training uses either IEMOCAP or FAU-AIBO, including tests on targets with a similar language.The discussion attributes performance drops to differences in studio conditions, age, language, and emotional-corpus type; it points to multilingual training or target data as remedies.Source: Figure 2 · p. 4, §5 · p. 4
- Figure 3 depicts the impact of using a percentage of target date with training data, with 3a showing training with IEMOCAP and 3b training with FAU-AIBO.Source: Figure 3 · p. 4
- Including around 20% target data—about 90 EMO-DB utterances—can exceed baseline accuracy, but results are worse when training uses FAU-AIBO.IEMOCAP performs better on German-language EMO-DB than FAU-AIBO does; the authors note that FAU-AIBO contains children’s speech and EMO-DB adult speech.Source: §5 · p. 4, Figure 3 · p. 4
6. Conclusions
The paper evaluates DBNs for transfer learning in cross-corpus and cross-language speech emotion recognition. It reports that DBNs outperform sparse autoencoders and that multilingual training or adding a small fraction of target data can improve baseline accuracy.
6. Conclusions
- DBNs outperform sparse autoencoders in cross-corpus and cross-language speech emotion recognition experiments.The authors attribute this to DBNs’ increased feature-learning abilities.Source: §6 · p. 4
- Training with many languages can improve baseline accuracy, including for emotion recognition in languages with very limited or no datasets.The authors identify multilingual data as useful for building robust speech emotion recognition systems.Source: §6 · p. 4
- Including a small fraction of target data during single-corpus training can also improve baseline accuracy.Source: §6 · p. 4