Source-linked AI summary
Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech
Michael Neumann, Ngoc Thang Vu
TL;DR
Speech emotion recognition is challenging because emotional expressions are complex and large datasets are limited. The paper proposes an attentive convolutional neural network with multi-view learning and evaluates features, signal lengths, and speech types. Recognition performance strongly depends on whether speech is improvised or scripted, and the study reports state-of-the-art results on improvised IEMOCAP speech.
Problem
Speech emotion recognition is challenging because emotional expressions are affected by factors such as age and gender, and large datasets are lacking.
Method
The paper evaluates an attentive convolutional neural network with multi-view learning across acoustic features, input signal lengths, and improvised versus scripted speech.
Results
Recognition performance strongly depends on speech type, while logMel, MFCC, and eGeMAPS perform similarly and prosody performs notably worse; the study reports state-of-the-art results on improvised IEMOCAP speech.
Takeaways & Limitations
Short signal snippets can retain relatively high recognition performance, and speech data should be carefully selected for the target application.
Abstract
from arXiv · showhide
Speech emotion recognition is an important and challenging task in the realm of human-computer interaction. Prior work proposed a variety of models and feature sets for training a system. In this work, we conduct extensive experiments using an attentive convolutional neural network with multi-view learning objective function. We compare system performance using different lengths of the input signal, different types of acoustic features and different types of emotion speech (improvised/scripted). Our experimental results on the Interactive Emotional Motion Capture (IEMOCAP) database reveal that the recognition performance strongly depends on the type of speech data independent of the choice of input features. Furthermore, we achieved state-of-the-art results on the improvised speech data of IEMOCAP.
1. Introduction
This work addresses speech emotion recognition, a challenging task affected by complex emotional expressions and limited data. It proposes an attentive CNN and evaluates feature types, signal lengths, and improvised versus scripted speech.
- Speech emotion recognition is challenging because emotional expressions are influenced by factors such as age and gender, while large datasets are scarce.
- Attention mechanisms can help CNNs process long or complex input signals.
- The proposed attentive convolutional neural network combines CNNs with attention mechanisms for emotion recognition.
- The study compares acoustic features, signal lengths, and improvised versus scripted speech, and reports state-of-the-art results on improvised IEMOCAP speech.
2. Model
The model combines convolution, pooling, attention, and multi-view learning to predict categorical emotions alongside activation and valence categories. Convolution extracts salient features, while attention weights information across input segments.
- The ACNN uses a CNN, an attention layer, and a fully connected softmax layer to predict emotional categories.The CNN learns audio representations, and the attention layer computes a weighted sum from different input parts.
- The CNN receives overlapping audio segments as d-dimensional feature vectors and arranges each utterance as a d × s input matrix.Two-dimensional kernels span all d features during convolution.
- Max pooling selects salient convolutional features before concatenation and softmax classification.
- Attention computes weights for input vectors and produces their weighted sum using a trainable linear scoring function.
- The model combines max-pooled CNN features with the attention vector because emotional information is distributed across signal parts and the input is noisy.
- Multi-view learning adds activation and valence information to categorical emotion labels during training.
3. Input Features
The study evaluates logMel filter-banks, MFCCs, prosody, and eGeMAPS as alternative acoustic inputs. These feature sets differ in representation and extraction, while eGeMAPS is a hand-crafted affective-computing set.
- The evaluated feature sets are 26 logMel filter-banks, 13 MFCCs, prosody features, and eGeMAPS.
- LogMel, MFCC, and prosody features are extracted from 25ms frames with a 10ms shift using openSMILE.
- Prosody features include loudness, F0-related measures, voicing probability, jitter, and shimmer.
- eGeMAPS is a hand-crafted affective-computing feature set containing 25 low-level descriptors for frequency, energy, and spectral parameters.
4. Data
Experiments use IEMOCAP, a roughly 12-hour audiovisual corpus containing scripted and improvised speech with categorical and continuous emotion annotations. The study analyzes separate improvised, scripted, and combined subsets with fixed-length inputs.
- IEMOCAP contains about 12 hours of audiovisual data from scripted play and improvised speech scenarios.
- The annotations include categorical emotions and activation, valence, and dominance dimensions scored from 1 to 5.
- The experiments use 2,943 improvised turns, 2,588 scripted turns, and 5,531 turns from all sessions.
- Inputs are fixed at 7.5s by cutting longer turns and zero-padding shorter turns.The mean turn length is 4.46s, with observed durations from 0.6s to 34.1s.
- For multi-view learning, activation and valence labels are grouped into low, medium, and high categories.
5. Experimental Results
Experiments compare attentive and standard CNNs across speech subsets, acoustic features, learning objectives, and signal lengths. Performance varies strongly with speech type, while short inputs can retain useful recognition accuracy.
- Experimental setup: Speaker-independent evaluation uses leave-one-session-out cross-validation on improvised, scripted, and combined IEMOCAP subsets.Each model, dataset, and feature combination is run six times with different random seeds, reporting averaged results and accuracy ranges.
- Improvised speech: 62.11% mean accuracy is the best improvised-speech result, while 63.85% is the best single result and exceeds the 62.85% state-of-the-art baseline.The 62.11% result uses ACNN with multi-view learning and logMel features; the 63.85% result uses ACNN with single-view learning.
- Scripted speech: 53.19% is the best scripted-speech accuracy, achieved with ACNN using MFCC with single-view learning or eGeMAPS with multi-view learning.Scripted-speech results are generally notably lower than improvised-speech results, and MFCC and eGeMAPS outperform logMel in this subset.
- All data: 56.10% is the best mean accuracy on all data, achieved by ACNN with multi-view learning and logMel features.MFCC and logMel perform similarly, eGeMAPS is slightly lower, and prosody performs notably worse; this result outperforms related work on the same data.
- Feature fusion: Early fusion slightly improves scripted-speech accuracy to 53.69% but decreases results for improvised speech and the complete dataset.The authors suggest the CNN may not learn more discriminatory features because convolution kernels span all features.
- Signal length: Accuracy decreases with shorter input, but improvised speech loses 3.4% with logMel versus a 7.5% drop for scripted speech.The authors conclude that relatively short snippets can support emotion recognition with only a small accuracy loss, which is relevant to real-time prediction while speaking.
6. Error analysis
The ACNN’s errors differ substantially between improvised and scripted speech, with recurring emotion confusions linked to class distributions and affective dimensions.
- In improvised speech, happy is predicted for 49.12% of angry samples, while sad is classified best at 73.01%.The frequent angry–happy confusion is attributed to both emotions having high activation, making valence harder to predict.
- Neutral is frequently confused with other classes because it lies near the center of the activation–valence space.
- Scripted sessions show high angry accuracy but relatively low accuracy for sad and happy, with more errors across almost all classes.Different class distributions, including many angry samples, explain part but not all of the discrepancy.
- The analysis suggests improvised speech is more variable and therefore easier for discriminating affective states, though more data is needed for confirmation.
- On the complete dataset, error patterns fall between improvised and scripted speech, and the angry–happy confusion is less severe than in improvised data.
7. Conclusion
The study compares acoustic features and signal lengths with an attentive CNN and finds that speech type matters more than feature choice, while short inputs remain effective.
- LogMel, MFCC, and eGeMAPS produce similar results, whereas prosodic features perform notably worse.The authors suggest the smaller number of prosodic features may explain their lower performance.
- Recognition performance differs strongly between improvised and scripted speech, with better results on improvised speech.
- Performance decreases slightly as signal length decreases but remains relatively high for signals as short as two seconds.
- The similar feature results suggest that model architecture and the amount and kind of training data may matter more than the particular feature choice.
- Future work includes testing the ACNN on a different database.