Source-linked AI summary
Speech Emotion Recognition with Co-Attention based Multi-level Acoustic Information
Heqing Zou, Yuke Si, Chen Chen, Deepu Rajan, Eng Siong Chng
TL;DR
The paper addresses the challenge of extracting comprehensive acoustic information for audio-only speech emotion recognition. It combines MFCC, spectrogram, and wav2vec2 representations with co-attention, achieving competitive IEMOCAP performance across two speaker-independent validation strategies.
Problem
Audio-only SER still faces the challenge of extracting and utilizing comprehensive in-depth acoustic information.
Method
The system encodes MFCC, spectrogram, and raw-audio information with BiLSTM, CNN, and wav2vec2, then fuses them using co-attention.
Results
The model achieves 69.80% UA and 71.05% WA for leave-one-session-out validation, plus 72.70% UA for leave-one-speaker-out validation on IEMOCAP.
Takeaways & Limitations
Multi-level acoustic information and co-attention provide a competitive approach for speaker-independent SER on IEMOCAP.
Abstract
from arXiv · showhide
Speech Emotion Recognition (SER) aims to help the machine to understand human's subjective emotion from only audio information. However, extracting and utilizing comprehensive in-depth audio information is still a challenging task. In this paper, we propose an end-to-end speech emotion recognition system using multi-level acoustic information with a newly designed co-attention module. We firstly extract multi-level acoustic information, including MFCC, spectrogram, and the embedded high-level acoustic information with CNN, BiLSTM and wav2vec2, respectively. Then these extracted features are treated as multimodal inputs and fused by the proposed co-attention mechanism. Experiments are carried on the IEMOCAP dataset, and our model achieves competitive performance with two different speaker-independent cross-validation strategies. Our code is available on GitHub.
1. INTRODUCTION
The paper frames audio-only SER as a practical alternative when transcripts are unavailable or error-prone, and proposes multi-level acoustic fusion to capture complementary speech information. Its system combines specialized encoders with co-attention to improve emotion prediction.
- Motivation: Audio-only SER avoids dependence on unavailable or error-prone ASR transcriptions while remaining easier to implement than multimodal recognition.The paper motivates integrating multiple acoustic information sources to utilize more complete audio information.
- Related work: MFCC and spectrogram features capture frequency-domain information but may overlook speech signals in the time domain.MFCC is described as a low-level, human-knowledge-based feature, while spectrograms can provide deep high-level information.
- Related work: Existing systems use varied encoders and attention mechanisms to extract and fuse acoustic features from different signals.Examples include CNN, LSTM, gated multi-feature units, domain-adversarial autoencoders, self-attention, and cross-modal attention.
- Contribution: The proposed system uses CNN, BiLSTM, and wav2vec2 to encode spectrogram, MFCC, and raw-audio information, respectively.The three encoded features are fused with co-attention before final emotion prediction.
2. PROPOSED METHOD
The proposed method extracts three acoustic representations from each utterance segment and uses co-attention to weight wav2vec2 frames based on MFCC and spectrogram features. The weighted wav2vec2 representation is then concatenated with the other encoded features for emotion classification.
- Model overview: Each audio segment supplies MFCC, spectrogram, and wav2vec2 inputs to separate feature encoders before fusion.The model overview treats these representations as complementary acoustic information.
- Multi-level acoustic information: MFCC, spectrogram, and wav2vec2 encode low-level, high-level, and raw-audio acoustic information spanning frequency and time domains.MFCC is processed with BiLSTM, spectrogram features with a pretrained AlexNet pathway, and raw audio with wav2vec2.
- Co-attention-based fusion: Co-attention generates frame weights for wav2vec2 from MFCC and spectrogram features instead of relying only on the last or average wav2vec2 frame.This design addresses information loss across the wav2vec2 sequence dimension.
- Fusion and objective: The final MFCC, spectrogram, and weighted wav2vec2 features are concatenated for emotion prediction using cross-entropy loss.The fused representation is used for the final classification task.
3. EXPERIMENT
The experiments evaluate the proposed SER system on IEMOCAP using speaker-independent cross-validation, specified acoustic preprocessing, and standard training settings.
- 3.1. Datasets: IEMOCAP contains 5,531 utterances across angry, sad, happy, and neutral after merging happy with excited.The data come from ten actors and include audio, video, transcriptions, and motion-capture information.
- 3.1. Datasets: The model is evaluated with 5-fold leave-one-session-out and 10-fold leave-one-speaker-out cross-validation for speaker-independent results.Weighted accuracy (WA) and unweighted accuracy (UA) are used as evaluation metrics.
- 3.2. Experimental Setup: Audio is sampled at 16 kHz and split into 3-second segments, with zero padding applied to shorter segments.The utterance-level prediction is determined from all segments belonging to that utterance.
- 3.2. Experimental Setup: The system uses MFCC, spectrogram, and wav2vec2 embeddings to represent complementary speech information.MFCC is extracted from raw segments, spectrograms use 40-ms windows and 10-ms hops, and W2E comes from pretrained wav2vec2.
- 3.2. Experimental Setup: The implementation uses PyTorch, AdamW with a 1e-5 learning rate, batch size 64, and early stopping after 8 epochs.The paper states that the code will be available on GitHub.
4. RESULTS AND ANALYSIS
The results show competitive speaker-independent performance and indicate that combining multiple acoustic levels with co-attention improves the fused representation and classification outcomes.
- 4.1. Results and Comparison: The proposed method achieves 69.80% UA and 71.05% WA under leave-one-session-out validation.These are reported as the best results in the corresponding evaluation setting.
- 4.1. Results and Comparison: Under leave-one-speaker-out validation, the method reaches 72.70% UA and 71.64% WA.The paper reports the UA as highest and the WA as competitive on the unbalanced IEMOCAP dataset.
- 4.2. Ablation Study: The ablation study compares individual acoustic levels, feature combinations with W2E, and combinations using co-attention-weighted W2E.W2E performs better than MFCC or spectrogram alone, while the full combination and co-attention contribute to overall performance.
- 4.2. Ablation Study: Co-attention improves over direct concatenation by 4.42% in WA and 4.89% in UA.The comparison is reported between the last two rows of the ablation table.
- 4.2. Ablation Study: t-SNE visualizations show clearer classification boundaries after co-attention for both weighted W2E and final combined features.The normalized confusion matrix also shows better final classification results with co-attention than without it.
5. CONCLUSION
The paper concludes that co-attention-based fusion of multi-level acoustic information provides complementary features for speech emotion recognition and performs competitively across speaker-independent evaluations.
- 5. CONCLUSION: The proposed system combines feature-specific encoders for raw audio, MFCC, and spectrogram information in a co-attention-based SER pipeline.The fusion produces weighted wav2vec2 embeddings and combines the final features for emotion prediction.
- 5. CONCLUSION: Experiments on IEMOCAP report competitive performance across different speaker-independent cross-validation methods.The paper proposes future work combining knowledge from different languages or datasets.