Source-linked AI summary
Speech Emotion Recognition with Dual-Sequence LSTM Architecture
Jianyou Wang, Michael Xue, Ryan Culhane, Enmao Diao, Jie Ding, Vahid Tarokh
TL;DR
The paper addresses the potential for textual information to mislead speech emotion recognition when the same words express different emotions. It proposes a jointly trained dual-level audio model using MFCC features and two mel-spectrogram resolutions. The model outperforms state-of-the-art unimodal SER models and is comparable with a multimodal model on mean weighted accuracy.
Problem
Textual information may mislead speech emotion recognition because utterances with the same words can convey different emotions, while the potential of audio-only SER remains unexplored.
Source: §1 · p. 1Method
A jointly trained dual-level model uses an MFCC LSTM and a Dual-Sequence LSTM that processes two mel-spectrogram sequences simultaneously.
Source: §1 · p. 1, §2.3.4 · p. 3Results
The model exceeds state-of-the-art unimodal SER models by at least 6.8% in mean weighted accuracy and 5.9% in mean unweighted accuracy, and matches a multimodal model in mean weighted accuracy.
Source: §3.3 · p. 4Takeaways & Limitations
The results show that audio-only unimodal SER models have not reached their full potential on IEMOCAP.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Speech Emotion Recognition (SER) has emerged as a critical component of the next generation human-machine interfacing technologies. In this work, we propose a new dual-level model that predicts emotions based on both MFCC features and mel-spectrograms produced from raw audio signals. Each utterance is preprocessed into MFCC features and two mel-spectrograms at different time-frequency resolutions. A standard LSTM processes the MFCC features, while a novel LSTM architecture, denoted as Dual-Sequence LSTM (DS-LSTM), processes the two mel-spectrograms simultaneously. The outputs are later averaged to produce a final classification of the utterance. Our proposed model achieves, on average, a weighted accuracy of 72.7% and an unweighted accuracy of 73.3%---a 6% improvement over current state-of-the-art unimodal models---and is comparable with multimodal models that leverage textual information as well as audio signals.
1. INTRODUCTION
The introduction argues that audio-only speech emotion recognition remains worth improving because textual information can mislead when identical words convey different emotions. It presents a jointly trained dual-level model, a two-resolution Dual-Sequence LSTM, and nearest-neighbor interpolation for variable-length utterances.
1. INTRODUCTION
- The introduction frames detecting emotional content in speech as a capability with potential benefits for human-machine interaction.It describes emotion as integral to daily conversations and discusses machines understanding or producing emotion.Source: §1 · p. 1
- Multimodal SER models generally outperform unimodal models by 6–7%, but textual information may mislead when utterances with identical content express different emotions.The authors argue that audio signals’ full potential in unimodal SER has not been explored.Source: §1 · p. 1
- The proposed dual-level model jointly trains separate networks on MFCC features and mel-spectrograms, extending the unimodal SER framework.The authors report that the model provides a significant increase in accuracy.Source: §1 · p. 1
- The Dual-Sequence LSTM processes two one-dimensional mel-spectrogram sequences simultaneously to harness complementary time and frequency resolution information.One spectrogram has better time resolution and the other better frequency resolution.Source: §1 · p. 1
- Nearest-neighbor interpolation addresses variable utterance lengths and is reported to work better than truncation or padding, which can lose information or increase computational cost.The interpolation is applied along the time axis to each mel-spectrogram.Source: §1 · p. 2
2. RESEARCH METHODOLOGY
The methodology combines MFCC features and mel-spectrograms at two time-frequency resolutions, then processes them through separate recurrent paths whose classification outputs are averaged. Variable-length spectrograms are aligned before the dual-sequence model processes them.
2.1. Dataset Description
- The evaluation uses 5,531 audio utterances from IEMOCAP, covering happy, neutral, angry, and sad speech from scripted and improvised conversations.The four emotion classes account for 29.5%, 30.8%, 19.9%, and 19.5% of utterances, respectively.Source: §2.1 · p. 2
2.2. Preprocessing
- Each utterance yields 39-dimensional MFCC sequences and paired 256- and 512-point mel-spectrograms, while nearest-neighbor interpolation aligns spectrogram time axes to the median length before logarithmic transformation.The narrower-window spectrogram provides better time resolution; the wider-window spectrogram provides better frequency resolution.Source: §2.2 · p. 2
2.3.1. Dual-Level Architecture
- The dual-level architecture trains separate MFCC and mel-spectrogram models, averages their classification outputs for prediction, and averages their cross-entropy losses.The MFCC branch is MLSTM and the spectrogram branch is MDS-LSTM.Source: §2.3.1 · p. 2
2.3.2. LSTM for MFCC Features
- The MFCC sequence is processed by a standard two-layer, single-directional LSTM, whose outputs are mean-pooled before classification.Each input vector contains 39 features.Source: §2.3.2 · pp. 2–3
2.3.3. CNN for Mel-Spectrograms
- Separate two-layer CNN blocks process the two spectrograms; because their outputs have different time-step counts, adjacent steps in the longer sequence are averaged to align them.The longer sequence has approximately twice as many time steps as the shorter one before alignment.Source: §2.3.3 · p. 3
2.3.4. Dual-Sequence LSTM
- DS-LSTM processes the aligned sequences simultaneously, using separate intermediate memory cells and an extra input gate before combining their contributions into its cell state.Its hidden state is also batch-normalized as input to the next layer; recurrent batch normalization is used on the gates to speed training and provide a more robust regularization effect.Source: §2.3.4 · p. 3
3. EXPERIMENTAL SETUP AND RESULTS
The experiments compare the proposed audio-based dual-level model with component baselines and prior systems. They report gains over unimodal baselines and assess how the model’s dual-level and DS-LSTM components contribute.
3.2. Baseline Methods
- Six baselines cover MFCC-only, single- and dual-resolution spectrogram, and combined MFCC–spectrogram models; comparisons also include four state-of-the-art models.Base 4 averages outputs from separate LSTMs, unlike the proposed DS-LSTM, which processes both sequences within one cell.Source: §3.2 · pp. 3–4
3.1. Experimental Setup
- Evaluation used standard 5-fold crossvalidation; the LSTMs were two-layer, single-directional networks with 200-node gates.The recurrent batch-normalization parameters were initialized to 0.1 for weights and 0 for biases; Adam used a learning rate of 0.0001.Source: §3.1 · p. 3, §3.2 · p. 4
3.3. Results and Analysis
- At least 6.8% in mean weighted accuracy and 5.9% in mean unweighted accuracy over state-of-the-art unimodal SER models; the model is also comparable with in mean weighted accuracy.It exceeds all baseline models by at least 4.2% in mean weighted accuracy and 4.0% in mean unweighted accuracy. Table 1 reports mean WA and mean UA over 5-fold crossvalidation.Source: Table 1 · p. 4, §3.3 · p. 4
- Single-resolution mel-spectrogram models and their naive combination perform slightly worse than the MFCC-only Base 1, while dual-level Bases 5 and 6 outperform Bases 1–4.The authors attribute Base 4’s result to the two LSTMs not interacting, leaving complementary information between the spectrogram sequences underused.Source: §3.3 · p. 4
- Replacing Base 4’s two separate LSTMs with DS-LSTM raises weighted accuracy by 5% and reduces parameters by 25%; MDS-LSTM significantly outperforms Bases 1–4.The DS-LSTM cell uses six neural networks, versus eight across the two separate LSTMs in Base 4.Source: §3.3 · p. 4
4. CONCLUSION
The conclusion presents combined MFCC and mel-spectrogram audio features, processed with a dual-level model and DS-LSTM, as effective for emotion recognition on IEMOCAP. The model outperforms baseline and state-of-the-art unimodal models and is comparable with multimodal models.
4. CONCLUSION
- On IEMOCAP, the audio-only model significantly outperforms baseline and state-of-the-art unimodal models and is comparable with multimodal models.The authors conclude that unimodal models relying only on audio signals have not reached their full potential.Source: §4 · p. 4