Source-linked AI summary
Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings
Leonardo Pepino, Pablo Riera, Luciana Ferrer
TL;DR
Small emotion-recognition datasets make sophisticated deep learning difficult, motivating transfer learning with pretrained wav2vec 2.0 features. The paper learns weighted combinations of model layers with simple downstream networks and reports superior results on IEMOCAP and RAVDESS compared with recent literature.
Problem
Speech emotion recognition is constrained by relatively small datasets, which makes sophisticated deep learning approaches challenging.
Method
The paper extracts pretrained wav2vec 2.0 features, learns trainable weighted combinations of layers, and models them with simple neural networks.
Results
The proposed approaches achieve superior results on IEMOCAP and RAVDESS compared with recent literature, including average recall of 84.1 ± 1.2% on RAVDESS and 72.1 ± 0.9% on IEMOCAP in one comparison.
Takeaways & Limitations
Combining information from different wav2vec 2.0 layers improves results over using only encoder outputs, while adding prosodic features provides additional gains.
Takeaways & Limitations
The evaluation discards samples without agreement across annotators and samples from nontarget emotions, an acknowledged non-ideal practice.
Abstract
from arXiv · showhide
Emotion recognition datasets are relatively small, making the use of the more sophisticated deep learning approaches challenging. In this work, we propose a transfer learning method for speech emotion recognition where features extracted from pre-trained wav2vec 2.0 models are modeled using simple neural networks. We propose to combine the output of several layers from the pre-trained model using trainable weights which are learned jointly with the downstream model. Further, we compare performance using two different wav2vec 2.0 models, with and without finetuning for speech recognition. We evaluate our proposed approaches on two standard emotion databases IEMOCAP and RAVDESS, showing superior performance compared to results in the literature.
1. Introduction
Speech emotion recognition matters for socially capable voice assistants, but progress is constrained by small datasets and limited gains from deep models. The paper therefore investigates wav2vec 2.0 representations as a transfer-learning approach for this task.
- Voice assistants that ignore emotion can be perceived as cold, socially inept, untrustworthy, or incompetent.
- Standard emotion-recognition datasets are relatively small, usually containing only a few dozen speakers.
- Deep neural networks have delivered limited gains for emotion recognition compared with their gains in automatic speech recognition and speaker identification.
- Transfer learning uses models trained on data-rich auxiliary tasks as feature extractors or fine-tuned models for data-scarce target tasks.
- The paper applies wav2vec 2.0 representations to speech emotion recognition, combines information from multiple layers, and evaluates the approach on IEMOCAP and RAVDESS.
2. Methods
The study extracts representations from pretrained wav2vec 2.0 models and evaluates simple downstream networks on emotion datasets. Its central method learns a weighted combination of local and transformer-layer activations, alongside Dense, LSTM, and Fusion variants.
- Wav2vec 2.0 model: Wav2vec 2.0 learns self-supervised representations from raw audio through local encoding, contextualized transformer encoding, and quantization.
- Pretraining: Pretraining masks contiguous local-encoder time steps and trains the contextualized encoder to reproduce their quantized representations.
- Feature extraction: The study compares a base wav2vec 2.0 model pretrained on LibriSpeech without ASR fine-tuning against one fine-tuned with 960 hours of LibriSpeech ASR data.
- Features: Features include local-encoder outputs, contextualized outputs, weighted combinations of transformer-block outputs, spectrograms, and eGeMAPS descriptors.
- Downstream models: The Dense downstream model applies two 128-neuron ReLU layers with dropout, averages representations over time, and keeps wav2vec 2.0 weights frozen.
- Fusion model: The trainable weighted-average layer learns coefficients for local-encoder and transformer-layer activations before downstream modeling.
- Downstream models: The LSTM variant replaces the second dense layer, while the Fusion model adds an eGeMAPS branch whose outputs are concatenated before classification.
- Datasets: IEMOCAP uses four emotion classes with five-fold session-out cross-validation, while RAVDESS uses seven classes and actor-based train, validation, and test splits.
3. Results and discussion
Across IEMOCAP and RAVDESS, wav2vec 2.0 representations—especially pretrained features combined across layers—outperformed the evaluated baselines and literature comparisons. Performance depended on normalization and model configuration, with speaker normalization substantially better than global normalization.
- The evaluation reports average recall across emotion classes using five training runs with different random seeds.Table 1 covers the explored feature and model configurations using the Dense downstream model.
- wav2vec2-PT local encoder features outperformed both spectrogram and eGeMAPS baselines despite using only local-window information and simple global averaging.The pretrained local representations were comparable to more complex models in the cited comparison.
- wav2vec2-PT features outperformed wav2vec2-FT features in all evaluated cases, with contextualized wav2vec2-FT features performing worst in the table.The authors hypothesize that ASR finetuning can remove information relevant to emotion recognition, such as pitch.
- Weighted averaging of local encoder and transformer-block outputs performed better than using only local or contextual encoder outputs.The approach combines information from multiple representation layers through jointly trained weights.
- Middle layers received larger learned weights, while wav2vec2-PT assigned relatively more weight to later layers than wav2vec2-FT.Weights were similar across training seeds, indicating limited sensitivity to neural-network initialization.
- Global normalization significantly degraded results compared with speaker normalization, especially for RAVDESS.The authors attribute the effect to speaker normalization reducing speaker information and emphasizing emotion-related variation.
- Adding eGeMAPS features produced modest improvements, suggesting wav2vec 2.0 features may omit information present in eGeMAPS.An LSTM replacement did not improve over the simpler dense layer.
- 84.1 ± 1.2% average recall in RAVDESS and 72.1 ± 0.9% in IEMOCAP exceeded the compared model’s 64.3% and 71.6%, respectively.The comparison used the experimental setup reported for the reference model, including its dataset-specific protocols.
4. Conclusions
The study finds that pretrained wav2vec 2.0 features, modeled with trainable layer combinations and simple downstream networks, outperform recent literature on IEMOCAP and RAVDESS.
- The proposed wav2vec 2.0 approaches achieved superior results on both IEMOCAP and RAVDESS compared with recent literature.
- Trainable weighted combinations of wav2vec 2.0 layers improved results over using only encoder outputs.
- Combining wav2vec 2.0 features with prosodic features produced additional gains, indicating that the embeddings lack some prosodic information needed for emotion recognition.
- The self-supervised wav2vec 2.0 model outperformed the version finetuned for automatic speech recognition.