Source-linked AI summary
Multimodal Speech Emotion Recognition Using Audio and Text
Seunghyun Yoon, Seokhyun Byun, Kyomin Jung
TL;DR
Speech emotion recognition is difficult because existing approaches rely heavily on audio features while facing limited training data and low-level emotional cues. The paper proposes a dual recurrent encoder that independently encodes audio and text, combines their representations, and achieves 68.8%–71.8% accuracy on IEMOCAP while reducing neutral-class misclassification bias.
Problem
Speech emotion recognition is challenging because training data are limited and emotional characteristics must be learned from low-level speech signals.
Method
The model independently encodes audio and text sequences with dual recurrent neural networks, then combines their representations to predict emotion classes.
Results
68.8% to 71.8% accuracies are obtained on IEMOCAP, and the model outperforms other state-of-the-art methods in classifying four emotion categories.
Takeaways & Limitations
Using multimodal audio and textual information supports emotion classification and reduces the neutral-class misclassification bias seen in previous audio-focused models.
Takeaways & Limitations
The approach assumes that speech transcripts can be extracted from audio signals with high accuracy using advancing ASR technology.
Abstract
from arXiv · showhide
Speech emotion recognition is a challenging task, and extensive reliance has been placed on models that use audio features in building well-performing classifiers. In this paper, we propose a novel deep dual recurrent encoder model that utilizes text data and audio signals simultaneously to obtain a better understanding of speech data. As emotional dialogue is composed of sound and spoken content, our model encodes the information from audio and text sequences using dual recurrent neural networks (RNNs) and then combines the information from these sources to predict the emotion class. This architecture analyzes speech data from the signal level to the language level, and it thus utilizes the information within the data more comprehensively than models that focus on audio features. Extensive experiments are conducted to investigate the efficacy and properties of the proposed model. Our proposed model outperforms previous state-of-the-art methods in assigning data to one of four emotion categories (i.e., angry, happy, sad and neutral) when the model is applied to the IEMOCAP dataset, as reflected by accuracies ranging from 68.8% to 71.8%.
1. INTRODUCTION
Speech emotion recognition remains challenging because it must learn emotional characteristics from low-level speech signals despite limited training data. The paper proposes combining high-level text transcriptions with low-level audio signals and reports improved performance on IEMOCAP.
- Speech emotion recognition aims to classify speech into emotional labels such as happy, sad, neutral, and angry.
- Limited training data and the need to learn emotions from low-level speech signals make speech emotion recognition challenging.
- The proposed model combines high-level text transcription with low-level audio signals to use low-resource datasets more effectively.
- The deep dual recurrent encoder simultaneously uses audio and text data to recognize emotions from speech.
- The model outperforms previous state-of-the-art methods on the IEMOCAP dataset, with reported performance ranging from 68.8% to 71.8%.
2. RELATED WORK
Earlier speech emotion recognition research used classical machine learning, neural architectures, auxiliary tasks, transfer learning, and combinations of acoustic and language information. The paper identifies a gap in end-to-end neural models that simultaneously use speech signals and text sequences.
- Classical approaches to speech emotion recognition included hidden Markov models, support vector machines, and decision-tree methods.
- Neural approaches evolved from deep networks extracting high-level features from raw audio toward more complex architectures.
- Researchers also used auxiliary tasks and transfer learning to help neural models learn from available or external data.
- Other studies combined acoustic features with language information or assessed emotional salience in phoneme and word sequences.
- The paper addresses the absence of an end-to-end neural model using speech signals and text sequences simultaneously.
3. MODEL
The model section develops separate recurrent encoders for audio and text, then combines their representations for multimodal emotion classification. An attention extension uses audio representations to weight emotionally informative transcript states.
- The methodology introduces recurrent encoders for audio and text before presenting a dual recurrent encoder for simultaneous multimodal processing.
- Audio Recurrent Encoder (ARE): The audio recurrent encoder processes sequential MFCC features with an RNN and concatenates the final hidden state with prosodic features.MFCC and prosodic features are extracted from the audio signal using openSMILE.
- Text Recurrent Encoder (TRE): The text recurrent encoder tokenizes transcripts, maps tokens to 300-dimensional embeddings, and predicts emotion from the text-RNN's final hidden state.
- Multimodal Dual Recurrent Encoder (MDRE): The MDRE independently encodes audio and transcript inputs, transforms both modalities, and predicts emotion from their concatenated encoding vectors.Audio encoding combines recurrent MFCC information with prosodic features, while textual encoding uses the transcript's final recurrent state.
- Multimodal Dual Recurrent Encoder with Attention (MDREA): The MDREA attention mechanism scores transcript states by their dot products with the final audio encoding, then uses weighted text states with audio features for classification.The weighted text representation is concatenated with the audio encoding before the softmax prediction.
4. EXPERIMENTAL SETUP AND DATASET
The study evaluates multimodal speech emotion recognition on IEMOCAP using four emotion categories, audio features, transcripts, and ASR-processed transcripts.
- The IEMOCAP dataset contains five sessions and is evaluated using majority-agreement categorical labels for happy, sad, angry, and neutral.
- Audio representations use MFCC and prosodic features, while text inputs use transcripts processed as token sequences.The MFCC feature set contains 39 features, including delta and acceleration coefficients.
- Table 1 reports model performance comparisons, with the top two models marked according to unweighted average recall.
- The experiments compare models trained with released transcripts against models trained with Google Cloud Speech API transcripts.The ASR-processed transcripts have a word error rate of 5.53%.
5. EMPIRICAL RESULTS
Experiments use five-fold cross-validation and repeated evaluation to compare audio, text, multimodal, attention, and ASR-based models. The multimodal MDRE model achieves the strongest reported performance, while error analysis shows improved class discrimination and reduced sad-to-happy confusion.
- Five-fold cross-validation uses 8:0.5:1.5 training, development, and testing splits, with ten training and evaluation runs per fold.Performance is summarized using mean scores and standard deviations.
- 0.718 WAP is achieved by MDRE, which outperforms the evaluated baseline and reaches state-of-the-art performance.The authors attribute the gain to combining multimodal information.
- 0.690 to 0.688 WAP is the comparison reported for MDREA against the best existing research result, although MDREA does not match MDRE.The authors suggest insufficient data for determining the more complex model parameters and leave audio-text alignment for future work.
- 0.691 to 0.688 WAP is achieved by MDRE-ASR over the previous best-performing model despite degraded ASR-based performance.The processed transcripts have a 5.53% word error rate.
- 43.51% of happy instances are classified as neutral by ARE, yielding 35.15% accuracy for the happy class.The analysis reports frequent confusion of emotion classes with neutral.
- 75.73% happy-class accuracy is obtained by TRE compared with 35.15% for ARE, while TRE predicts sad instances as happy 16.20% of the time.The authors connect the gain to differences in word distributions between happy and neutral expressions.
- 9.15% is the reduced sad-to-happy error rate for MDRE, down from 16.20% for TRE.MDRE also increases the correctly predicted accuracies for all classes along the confusion-matrix diagonal.
6. CONCLUSIONS
The paper concludes that a multimodal dual recurrent encoder combines audio and text sequences to classify four speech-emotion categories. It reports accuracies from 68.8% to 71.8% on IEMOCAP and identifies multimodal extensions as future work.
- MDRE encodes audio and text independently with dual RNNs, then combines their representations with a feed-forward neural model to predict emotion classes.
- 68.8% to 71.8% accuracies are obtained when the proposed model classifies angry, happy, sad, and neutral categories on IEMOCAP.
- The proposed approach addresses frequent neutral-class misclassification observed in previous audio-focused models.
- Future work extends the modalities to audio, text, and video and investigates attention mechanisms for multiple modalities.