Source-linked AI summary
Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
Samarth Tripathi, Sarthak Tripathi, Homayoon Beigi
TL;DR
Automatic emotion recognition on IEMOCAP has largely emphasized speech despite the dataset’s speech, text, and motion-capture modalities. This paper trains deep-learning models for each modality, fuses their final representations, and reports performance matching prior research while noting an unequal evaluation setup.
Problem
Prior IEMOCAP research concentrated largely on speech, while the dataset also provides text and motion-capture information relevant to multimodal emotion recognition.
Method
The paper trains modality-specific neural models for speech, text, and motion capture, then concatenates selected representations for final-layer multimodal fusion.
Results
The speech model matches prior research under multiple evaluation conditions, and the final multimodal model reports performance matching prior state of the art.
Takeaways & Limitations
Final-stage fusion makes the proposed multimodal system modular, allowing an individual modality model to be replaced without affecting the remaining modalities.
Abstract
from arXiv · showhide
Emotion recognition has become an important field of research in Human Computer Interactions as we improve upon the techniques for modelling the various aspects of behaviour. With the advancement of technology our understanding of emotions are advancing, there is a growing need for automatic emotion recognition systems. One of the directions the research is heading is the use of Neural Networks which are adept at estimating complex functions that depend on a large number and diverse source of input data. In this paper we attempt to exploit this effectiveness of Neural networks to enable us to perform multimodal Emotion recognition on IEMOCAP dataset using data from Speech, Text, and Motion capture data from face expressions, rotation and hand movements. Prior research has concentrated on Emotion detection from Speech on the IEMOCAP dataset, but our approach is the first that uses the multiple modes of data offered by IEMOCAP for a more robust and accurate emotion detection.
1. INTRODUCTION
The paper addresses the need for automatic emotion recognition by combining speech, text, and motion-capture data from IEMOCAP. It develops modality-specific deep-learning models and fuses them at the final stage for a stronger, more robust detector.
- IEMOCAP provides approximately 12 hours of audio-visual data containing facial recordings, speech, and text transcriptions.
- The study combines speech, text, and motion-capture modalities to build a stronger and more robust emotion detector.
- It first optimizes individual modality models, then combines their stronger variants in an ensemble with fusion only at the final stage.
- The modular design allows a missing modality to require retraining only of the prefinal layer rather than the other modalities.
- Using motion-capture data instead of video avoids the need for 3D convolutions and enables the use of 2D convolutions.
2. RELATED WORKS
Prior IEMOCAP research largely emphasized speech emotion detection, while later work expanded toward multimodal feature extraction and contextual fusion. Existing approaches used recurrent, convolutional, and connectionist-training methods across modalities.
- Early IEMOCAP studies focused on speech, using segment-level features with MLPs or recurrent networks such as LSTMs and BLSTMs.
- Speech studies extracted frame-level acoustic features including MFCCs and modeled utterance labels with recurrent architectures.
- A CTC-based speech approach addressed utterances in which emotionality occurs only in a few words or phonemes.
- A later multimodal system combined 3D-CNN visual features, text-CNN features, openSMILE audio features, and contextual LSTM fusion layers.
3. EXPERIMENTAL SETUP
The experimental setup uses four emotion classes from IEMOCAP and represents speech, text, and motion capture in modality-specific formats. Training and testing are split by sessions and actors to preserve speaker-agnostic evaluation.
- IEMOCAP contains 12 hours of dialogue recordings from 10 actors, divided into 3-to-15-second utterances labeled by 3-4 assessors.
- The experiments retain four labels: anger, excitement, neutral, and sadness.
- Motion-capture inputs record facial expressions, hand positions, and head rotations, supplying 165, 18, and 6 dimensions respectively.
- Speech is converted into 34-feature sequences using 0.2-second windows, 0.1-second steps, and a maximum of 100 frames per utterance.
- Text uses 300-dimensional pretrained GloVe embeddings with a maximum sequence length of 500, while MoCap features are sampled into 200 partitions and averaged.
- The final model trains on 3838 dialogues from the first four sessions and tests on 1098 dialogues from Session 5, separating two actors for speaker-agnostic prediction.
4. MODELS
The paper evaluates modality-specific neural architectures before selecting models for speech, text, and MoCap. Attention-based bidirectional LSTMs perform strongly for speech, while the selected unimodal models are later used for multimodal fusion.
- 4.1. Speech Based Emotion Detection: Speech Model4 uses bidirectional LSTMs with attention and performs best among the evaluated speech models.
- 4.1. Speech Based Emotion Detection: The selected speech model matches prior research under both improvisation-only and combined scripted-and-improvisation evaluation conditions.
- 4.2. Text based Emotion Recognition: Text experiments compare convolutional and stacked-LSTM models initialized with GloVe embeddings, with randomized 128-dimensional initialization also tested.
- 4.3. MoCap based Emotion Detection: MoCap experiments use LSTM and convolutional models for head rotation, hand movement, and facial-expression inputs.
5. RESULTS
The final model combines the strongest unimodal architectures through feature fusion, then optimizes key architectural and dropout hyperparameters before test evaluation.
- The final model selects specialized architectures for text, speech, and motion-capture modalities before removing their softmax layers.Text uses stacked LSTMs with GloVe embeddings, speech uses bidirectional LSTMs with attention, and motion capture uses stacked convolution layers.
- Their final fully connected representations are concatenated and passed through an additional 256-neuron fully connected layer.
- Random-proposer hyperparameter optimization tunes speech and text LSTM sizes, the combined model’s final-layer width, and dropout.The best configuration is selected on the validation set and evaluated on the test set using training and validation data.
- The reported performance matches the prior state of the art.
6. CONCLUSION
The paper performs multimodal emotion recognition on IEMOCAP by combining speech, text, and motion-capture data with final-layer fusion. This design emphasizes robustness, modularity, and faster processing than video-based alternatives.
- The system combines speech, text, and motion-capture data for multimodal emotion classification on IEMOCAP.
- Final-stage fusion means that a missing modality requires retraining only the final fully connected layer.
- Optimizing modalities independently makes the combined model modular, allowing one individual model to be replaced without affecting the others.
- Motion-capture data with 2D convolutions supports faster training and inference than video recordings with 3D convolutions.