Source-linked AI summary
Speech Emotion Recognition using Self-Supervised Features
Edmilson Morais, Ron Hoory, Weizhong Zhu, Itai Gat, Matheus Damasceno, Hagai Aronowitz
TL;DR
SER has relatively few studies using self-supervised features despite their success in NLP. The paper builds a modular Upstream + Downstream E2E system and evaluates fine-tuning, aggregation, and classification configurations on IEMOCAP, reporting SOTA speech-only performance and a result comparable to multimodal systems.
Problem
Self-supervised features are successful in NLP but remain insufficiently investigated for SER, especially when only speech is available.
Method
A modular Upstream + Downstream E2E SER system uses self-supervised feature models, fine-tuning, checkpoint averaging, and configurable downstream aggregation and classification.
Results
The system reaches SOTA results, including the best reported speech-only SER result to the authors’ knowledge under 5-fold CV on IEMOCAP.
Takeaways & Limitations
Carefully fine-tuned and averaged self-supervised acoustic features can substantially improve E2E SER performance and may match results from multimodal systems.
Takeaways & Limitations
The fifth baseline uses ground-truth transcriptions and context-dependent text embeddings with a window size of [-3,3].
Abstract
from arXiv · showhide
Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation. In this paper we introduce a modular End-to- End (E2E) SER system based on an Upstream + Downstream architecture paradigm, which allows easy use/integration of a large variety of self-supervised features. Several SER experiments for predicting categorical emotion classes from the IEMOCAP dataset are performed. These experiments investigate interactions among fine-tuning of self-supervised feature models, aggregation of frame-level features into utterance-level features and back-end classification networks. The proposed monomodal speechonly based system not only achieves SOTA results, but also brings light to the possibility of powerful and well finetuned self-supervised acoustic features that reach results similar to the results achieved by SOTA multimodal systems using both Speech and Text modalities.
1. INTRODUCTION
SER remains difficult because emotions are complex, ambiguous, personal, and expressed through multiple cues, while datasets often inadequately cover speaker variation. The paper therefore investigates self-supervised features through a modular speech-only E2E system and targets performance comparable to multimodal systems.
- SER is challenging because emotions are complex, ambiguous, highly personal, and expressed through simultaneous vocal, linguistic, facial, and bodily cues.
- Limited speaker coverage in public datasets makes it difficult to represent personal variation in emotional expression.
- Self-supervised features have shown strong results in NLP, but only few studies have investigated them for SER.
- The paper introduces a modular Upstream + Downstream E2E SER system for integrating diverse self-supervised features and evaluating configurations.
- The proposed speech-only system aims to reach SOTA results comparable to multimodal systems using Speech and Text modalities.
2. PROPOSED MODEL
The proposed SER model maps speech to categorical emotion labels using task-independent self-supervised feature extraction and task-dependent classification. It uses Upstream fine-tuning, checkpoint averaging, and configurable Downstream processing on IEMOCAP.
- The model maps continuous speech into discrete categorical emotion labels using an Upstream + Downstream paradigm.
- The Upstream extracts features with a task-independent self-supervised encoder, while the Downstream classifies those features into emotion categories.
- The system uses Wav2Vec 2.0 and huBERT Upstream models, with Mean Average Pooling or ECAPA-TDNN followed by a linear classifier.
- Upstream models are fine-tuned with emotion labels and jointly trained with Mean-Average Pooling and a linear classifier.
- Five validation-selected fine-tuned checkpoints are averaged separately for each fold to reduce Upstream output variance without observing test data.
3. EXPERIMENTAL SETUP
The evaluation varies fine-tuning, checkpoint averaging, feature fusion, and aggregation choices, then compares the resulting systems with acoustic and multimodal baselines. The experiments also support comparison with published SOTA results.
- Evaluation goals: The experiments examine fine-tuning, Upstream and Downstream checkpoint averaging, Wav2Vec 2.0–huBERT combinations, and Mean Pooling versus ECAPA-TDNN.
- Evaluation configurations: Experiments 1–4 use Wav2Vec 2.0 or huBERT with Mean Average aggregation, comparing systems without averaging against systems averaging both model types.
- Evaluation configurations: Experiments 5–6 use the same averaged Upstream and Downstream setup as experiments 3–4 but replace the aggregator with ECAPA-TDNN.
- Fusion comparisons: Experiment 7 performs early fusion of Wav2Vec 2.0 and huBERT features before ECAPA-TDNN, whereas experiment 8 performs later fusion of their utterance embeddings.
- Baselines: Baseline experiments use standard Filter-Bank acoustic features, BERT text embeddings, or their later fusion, with fine-tuning and averaging applied to BERT.
- External comparison: The study compares its SER results with SOTA results reported in the literature.
4. RESULTS
The experiments evaluate how input features, fusion, aggregation, averaging, and modality affect SER performance, with experiment 7 achieving the reported best speech-only result on IEMOCAP using 5-fold cross-validation.
- Experimental organization: Table 1 contains 11 experiments across speech-only and baseline configurations, reporting input modality, Upstream and Downstream settings, WACC, and UACC.The table separates Mean Pooling experiments, ECAPA-TDNN experiments, and baseline experiments.
- Model configuration effects: Averaging Upstream models improves WACC by 2.38%, while averaging Downstream models improves WACC by 2.21%.These comparisons are made between experiments 1–2 and 3–4.
- Model configuration effects: ECAPA-TDNN as a feature aggregator slightly outperforms mean average pooling.The comparison uses experiments 5–6 against experiments 3–4.
- Feature fusion: Early fusion of huBERT and Wav2Vec 2.0 features outperforms later fusion.The comparison is between experiments 7 and 8.
- Baseline comparisons: Self-supervised Wav2Vec 2.0 and huBERT features outperform standard Filter-Bank features by a very large margin and outperform the fine-tuned averaged BERT baseline by around 6%.The BERT baseline uses ground-truth transcriptions and context-dependent text embeddings with a window size of [-3,3].
- Overall result: Experiment 7 reaches SOTA performance and is reported as the best result so far for 5-fold CV on IEMOCAP with speech-only input.Table 2 compares the proposed method with audio-only and audio+text literature baselines.
5. CONCLUSIONS AND FUTURE WORKS
The paper concludes that its Upstream + Downstream E2E SER model supports pretrained, fine-tuned, and averaged Upstream models, while future work will extend it to multitask learning and multimodality.
- Conclusions: The E2E SER model enables use of pretrained, fine-tuned, and averaged Upstream models within an Upstream + Downstream paradigm.The conclusion presents this as the paper's central model contribution.
- Conclusions: Carefully fine-tuned and averaged Upstream and Downstream models can significantly improve E2E SER performance on the 5-fold CV IEMOCAP experiments.The conclusion attributes the improvement to designed combinations of these model components.
- Future work: Future work will extend the model to multitask learning and multimodal inputs, including speech + text and speech + text + visual modalities.These directions are stated as planned extensions.