Source-linked AI summary
Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention
Zixuan Peng, Yu Lu, Shengfeng Pan, Yunfeng Liu
TL;DR
Speech emotion recognition must account for acoustic and lexical information, which can interact in expressed emotion. The paper combines multi-scale CNNs, statistical pooling, and attention for audio and text features. On four-class IEMOCAP, MSCNN-SPU-ATT outperforms prior state-of-the-art methods in WA and UA.
Problem
Speech emotion can depend on interacting acoustic and lexical information, motivating recognition methods that consider both modalities.
Source: §1 · p. 1Method
The approach applies multi-scale CNNs and statistical pooling to audio and text features, with an attention module combining audio context and text representations.
Source: §3.1 · p. 2, §3.2 · p. 2, §3.3 · p. 2Results
5.0% WA and 5.2% UA improvement over prior state-of-the-art methods are reported for four-class IEMOCAP classification.
Source: §5 · p. 5Takeaways & Limitations
The authors report competitive performance on ASR-processed transcripts, supporting the model’s robustness when ground-truth transcripts are unavailable.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Emotion recognition from speech is a challenging task. Re-cent advances in deep learning have led bi-directional recur-rent neural network (Bi-RNN) and attention mechanism as astandard method for speech emotion recognition, extractingand attending multi-modal features - audio and text, and thenfusing them for downstream emotion classification tasks. Inthis paper, we propose a simple yet efficient neural networkarchitecture to exploit both acoustic and lexical informationfrom speech. The proposed framework using multi-scale con-volutional layers (MSCNN) to obtain both audio and text hid-den representations. Then, a statistical pooling unit (SPU)is used to further extract the features in each modality. Be-sides, an attention module can be built on top of the MSCNN-SPU (audio) and MSCNN (text) to further improve the perfor-mance. Extensive experiments show that the proposed modeloutperforms previous state-of-the-art methods on IEMOCAPdataset with four emotion categories (i.e., angry, happy, sadand neutral) in both weighted accuracy (WA) and unweightedaccuracy (UA), with an improvement of 5.0% and 5.2% respectively under the ASR setting.
1. INTRODUCTION
Speech emotion can depend on the interaction between acoustic and lexical information, motivating a multimodal approach. The paper proposes MSCNN-SPU and an attention-enhanced version, reporting improvements over prior state-of-the-art methods on IEMOCAP.
1. INTRODUCTION
- Speech emotion can depend on interacting acoustic and lexical cues, making both modalities relevant to recognition.The passage illustrates that an utterance can be ambiguous without prosody and that a flat voice can affect how lexical content is interpreted.Source: §1 · p. 1
- The paper proposes MSCNN-SPU to learn speech and text modalities together, with MSCNN-SPU-ATT adding attention to further improve performance.The authors report that MSCNN-SPU outperforms prior state-of-the-art approaches, and that attention further improves on MSCNN-SPU.Source: §1 · p. 1
- 5.0% WA and 5.2% UA relative improvement are reported for MSCNN-SPU-ATT over prior state-of-the-art methods on four-class IEMOCAP classification.The conclusion also reports competitive results on ASR-processed transcripts when ground-truth transcripts are unavailable.Source: §1 · p. 1
2. RELATED WORK
Prior speech emotion recognition work includes classical statistical approaches, deep models using audio or text alone, and multimodal systems. Existing multimodal methods use varied fusion and attention strategies, sometimes with increasingly complex architectures.
2. RELATED WORK
- Classical approaches modeled handcrafted temporal signal features with HMMs or used high-level statistical features with GMMs.Source: §2 · p. 2
- Deep-learning studies applied CNNs to audio or text and RNNs to sequence modeling, while often using only one modality.Source: §2 · p. 2
- Multimodal systems combine audio and text using methods including WordNet with audio features, BiLSTM and multi-resolution CNNs, or attention-based alignment and fusion.The cited work also explored increasingly complicated schemes such as residual learning and attention.Source: §2 · p. 2
3. MODEL
The model processes audio and text with parallel multi-scale CNN paths, pools their features, and uses attention to combine modality representations before emotion classification.
3.1. Multi-scale CNN
- Multi-scale CNNs process audio and text separately with parallel convolutions using different kernel sizes to form modality-specific feature maps.The audio sequence consists of MFCC vectors, while the text sequence consists of word embeddings.Source: §3.1 · p. 2
3.2. Statistical Pooling Unit (SPU)
- The statistical pooling unit applies parallel global max, average, and standard-deviation pooling along the sequence direction to MSCNN outputs.The paper motivates multiple pooling types as providing different abstractions that may help model different emotions.Source: §3.2 · p. 2
3.3. Attention
- Bi-modal attention uses audio MSCNN-SPU outputs as context to weight feature maps from the text MSCNN.The context vectors correspond to max-, average-, and standard-deviation-pooled audio features.Source: §3.3 · p. 2, Figure 1 · p. 3
3.4. Classification Layer
- The classifier concatenates audio and text SPU features, the attended vector, and SWEM features before dimensionality reduction and softmax classification.Categorical cross entropy is used as the loss function.Source: §3.4 · p. 3
4. EXPERIMENTS
Experiments evaluate the model on IEMOCAP using speech, transcripts, and ASR-processed transcripts. The proposed approach outperforms prior approaches on both WA and UA, including under the ASR setting.
4.1. Data
- IEMOCAP evaluation uses 5,531 utterances across four emotions and 10-fold cross-validation, with 8:1:1 train/dev/test splits and results averaged over three runs.The selected categories are angry, sad, neutral, and happy, with excited merged into happy.Source: §4.1 · p. 3
4.2. Feature Extraction and Implementation Details
- Audio inputs combine 96-dimensional MFCC features with X-vector embeddings, while text uses 300-dimensional GloVe embeddings for ground-truth or Google ASR transcripts.Google ASR transcripts have a reported 5.80% WER; training uses 128 filters per CNN layer, Adam at 0.0005, and dropout 0.3.Source: §4.2 · pp. 3–4
4.3. Performance Evaluation
- Table 2 presents an ablation study of the proposed model and indicates gains for each component.The supplied passage does not state the component-level values.Source: Table 2 · p. 4, §4.3 · p. 4
- 5.0% WA and 5.2% UA gains over previous SOTA are reported under the ASR setting; ground-truth transcripts yield 80.3% WA and 81.4% UA.ASR transcripts reduce performance by roughly 2% compared with ground-truth transcripts, while the model remains competitive.Source: §4.3 · p. 4, Table 1 · p. 4
5. CONCLUSIONS
The proposed MSCNN-SPU-ATT combines audio and text information for speech emotion recognition. On IEMOCAP four-class classification, it outperforms previous SOTA by 5.0% in WA and 5.2% in UA, including with ASR transcripts.
5. CONCLUSIONS
- The MSCNN-SPU-ATT combines audio content and text information for four-class emotion recognition on IEMOCAP.The model is also tested with ASR-processed transcripts and achieves competitive results when ground-truth transcripts are unavailable.Source: §5 · pp. 4–5
- 5.0% in WA and 5.2% in UA are the reported gains over previous SOTA on IEMOCAP four-class emotion classification.The conclusion describes the ASR-transcript results as competitive and as showing robustness in a real-world scenario without ground-truth transcripts.Source: §5 · pp. 4–5