PaperlayerView exact revision on arXiv

Source-linked AI summary

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, Björn W. Schuller

arXiv:2203.07378v4eess.AScs.LGcs.SD

TL;DR

The paper addresses limited evidence about transformer model size and pre-training data in SER, alongside unresolved generalisation, robustness, fairness, and efficiency questions. It systematically evaluates pre-trained wav2vec 2.0 and HuBERT variants for dimensional SER across MSP-Podcast, IEMOCAP, and MOSI. Transformers achieve state-of-the-art valence performance without explicit linguistic information, while remaining individual-speaker fairness issues require further work.

  • Problem

    Existing SER studies had not evaluated model size and pre-training data comprehensively and had limited attention to generalisation, robustness, fairness, and efficiency.

  • Method

    The study fine-tunes several pre-trained wav2vec 2.0 and HuBERT variants on arousal, dominance, and valence, then evaluates them in-domain and cross-corpus.

  • Results

    Transformers obtain state-of-the-art valence recognition on MSP-Podcast with CCC .638 without explicit linguistic information and show strong cross-corpus generalisation and robustness.

  • Takeaways & Limitations

    Transformer architectures constitute the new state-of-the-art in SER, with valence success linked to implicit linguistic information learned during transformer-layer fine-tuning.

  • Takeaways & Limitations

    Individual fairness remains challenging, and IEMOCAP’s scripted conversations can leak text information that produces overoptimistic results when exploited by transformers.

Abstract

from arXiv · show

Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during fine-tuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.

1 Introduction

The paper addresses persistent challenges in speech emotion recognition, especially weak valence performance, limited generalisation and robustness, and fairness concerns. It evaluates wav2vec 2.0 and HuBERT transformers for dimensional SER and reports state-of-the-art valence results without explicit linguistic inputs.

  • SER supports better human-to-machine interactions and can model discrete emotions or emotional dimensions such as arousal, valence, and dominance.
  • Linguistic information is better suited to valence, whereas paralinguistic information works better for arousal and dominance and can generalise across languages.
  • Three major challenges are inferior valence performance, generalisation and robustness issues, and individual- and group-level fairness concerns.
  • The study evaluates pre-trained wav2vec 2.0 and HuBERT variants across architecture, pre-training data, generalisation, robustness, fairness, and efficiency.
  • The authors report state-of-the-art valence results and release their best-performing model to the community.

2 Related Work

Prior work established transformer-based SER improvements but left uncertainty about architecture size, pre-training data, dimensional performance, robustness, and fairness. This paper positions its systematic comparison against those unresolved questions and related categorical and dimensional SER studies.

  • Prior transformer SER studies on IEMOCAP commonly used wav2vec 2.0 or HuBERT for four-class emotion recognition, ranked by UAR and WAR.
  • Fine-tuning pre-trained weights yields a 10% boost, while additional ASR fine-tuning does not help SER and can reduce performance by 3.2%.
  • Large architectures are typically better than base architectures, although gains can be small, while HuBERT outperforms wav2vec 2.0 in the cited comparison.
  • Average pooling with a linear classifier appears sufficient when transformer layers are fine-tuned, whereas some more complex models omit downstream fine-tuning.
  • Dimensional SER work found pre-trained models particularly effective for valence, with a multimodal wav2vec 2.0/HuBERT-plus-BERT system reaching CCC .683 on MSP-Podcast.
  • The paper identifies unresolved effects of pre-training data quantity, domain, and language, and evaluates robustness against emotion-preserving augmentations and fairness across speakers and gender.

3 Experimental setup

The experiments compare wav2vec 2.0 and HuBERT architectures, sizes, and pre-training conditions using partial transformer fine-tuning for dimensional SER. Evaluation spans in-domain and cross-corpus data, with correctness, robustness, and fairness measures.

  • Models: The study evaluates wav2vec 2.0 and HuBERT, whose architectures process normalised raw waveforms through convolutional feature extraction and transformer encoders.
  • Models: wav2vec 2.0 and HuBERT differ primarily in self-supervised pre-training objectives applied to masked speech representations.
  • Models: The models include base architectures with 12 transformer layers and 95M parameters and large architectures with 24 layers and 317M parameters, across varied pre-training datasets.
  • Training: Downstream training averages the last transformer-layer hidden states, uses CCC loss and ADAM, and selects the best checkpoint over five epochs.
  • Training: Training freezes the CNN layers while fine-tuning the transformer layers, following prior evidence that partial fine-tuning performs better.
  • Datasets: MSP-Podcast supplies multitask arousal, dominance, and valence training data, while IEMOCAP and MOSI provide cross-corpus evaluation for dimensional labels and valence sentiment.
  • Evaluation: Robustness is tested with emotion-preserving signal augmentations, and fairness focuses on gender because other protected-attribute data are insufficient.

4 Evaluation

Transformer-based models are evaluated across emotional dimensions, datasets, and text-fusion settings. They especially improve valence, while explicit linguistic information mainly benefits valence and can harm arousal and dominance.

  • 4 Evaluation: Valence CCC improves by .26 over non-transformer architectures, closing the performance gap.The evaluation compares wav2vec 2.0 and HuBERT models with a CNN14 baseline across in-domain and cross-corpus settings.
  • 4 Evaluation: w2v2-L-robust reaches CCC scores of .745/.634 for arousal/dominance on MSP-Podcast and .663/.518 on IEMOCAP.These are the overall best reported transformer results for those dimensions in the cited evaluation.
  • 4 Evaluation: w2v2-L-robust achieves valence CCCs of .635 on MSP-Podcast and .448 on IEMOCAP, plus .539 for sentiment on MOSI.Valence performance varies substantially across models, from .359 for w2v2-L-xls-r to .636 for hubert-b on MSP-Podcast.
  • 4.2 Does explicit linguistic information further improve performance?: Text-audio fusion helps valence but not arousal or dominance, where performance deteriorates.The fusion adds pooled BERT embeddings from automatic transcriptions to pooled fine-tuned transformer states before training a regression head.

4.3 Do the models implicitly learn linguistic information?

The models implicitly learn linguistic information during fine-tuning, particularly for valence prediction. Evidence from synthetic speech and frozen-layer comparisons links valence performance to sentiment sensitivity and shows that transformer-layer fine-tuning is necessary.

  • 4.3 Do the models implicitly learn linguistic information?: Transformer models implicitly capture linguistic information from audio, with sentiment learning depending on pre-training data.Multilingual pre-training can make sentiment learning more difficult, and valence performance correlates with sentiment prediction ability.
  • 4.3 Do the models implicitly learn linguistic information?: Valence performance gaps between models correspond to their ability to predict sentiment on synthetic speech.Models that perform well on original files also perform well on synthetic versions generated from MSP-Podcast transcriptions.
  • 4.4 How important is a fine-tuning of the transformer layers?: Fine-tuning transformer layers is necessary for state-of-the-art performance, especially on valence.The largest gains occur for hubert-L and w2v2-L-robust, which benefit least from added text information.
  • 4.4 How important is a fine-tuning of the transformer layers?: Fine-tuning improves performance in all tested cases, with the largest gains for hubert-L and w2v2-L-robust.These models also benefit less from additional text information, indicating that transformer-layer fine-tuning enables linguistic-content capture for valence.

4.5 Do the models generalise better across different domains?

Transformer-based models generalise better across domains than the CNN14 baseline, especially for arousal and dominance, while valence remains inconclusive. Pre-training data strongly affects valence and sentiment performance.

  • 11% arousal, 21% dominance, and 30% valence CCC drops occur for w2v2-L-robust on IEMOCAP, with a 15% sentiment drop on MOSI.For CNN14, the corresponding drops are 34% for arousal and 52% for dominance.
  • Transformer-based models show smaller CCC drops than CNN14 for arousal and dominance, indicating better cross-corpus generalisation.The evidence does not support a final conclusion for valence, although MOSI sentiment shows a promising trend.
  • Arousal and dominance performance is similar across tested models, whereas pre-training data has a strong effect on valence and sentiment.Mixing domains improves w2v2-L-robust over clean-speech-only w2v2-L, but multilingual pre-training models perform worse on English speech.
  • The comparison varies pre-training size and domain across several wav2vec 2.0 models with the same large architecture.This design isolates how pre-training data influences downstream performance.

4.7 Does a larger architecture lead to better performance?

Larger architectures do not inherently improve performance: model size has limited influence, while robustness and pre-training characteristics matter more. The best model remains robust under many input augmentations, although severe changes can substantially reduce CCC.

  • Larger architectures do not lead to better performance per se, and larger models with different pre-training data can perform worse than smaller models.Architecture size alone is therefore not the decisive factor in the comparison.
  • Base and large wav2vec 2.0 models perform similarly, while small models are comparable to large models for arousal and dominance across domains.For valence, the supplied passage indicates that small models can outperform w2v2-L.
  • w2v2-L-robust has the highest robustness, while hubert-b has the lowest among the tested models.Across augmentations, w2v2-L-robust averages a CCC change of −0.068, compared with −0.108 for hubert-b.
  • CCC changes remain below .05 in absolute value for 20 dB background sounds or signal fades, but can reach −.278 for speed changes or more severe noise.These results concern w2v2-L-robust predictions relative to clean data.

4.9 Are the models fair regarding the gender of the speaker?

The models are generally fair across biological sex groups, with smaller differences for arousal and dominance than for valence. Individual-speaker performance is less consistent, with some speakers receiving low CCC values and model disagreements across valence.

  • Models are more fair for arousal and dominance than for valence, where most achieve higher CCC for female than male speakers.CNN14 is the main exception for valence-related comparisons.
  • The gender fairness score is CCCfemale − CCCmale, with 0 representing equal performance across female and male speakers.Positive values favor female speakers, while negative values favor male speakers.
  • Averaged across databases and dimensions, w2v2-L has the best gender fairness score at .007, while CNN14 has −.043.The other transformer models fall between these values in the reported ranking.
  • Best foundation models perform similarly for most MSP-Podcast speakers but deteriorate to low CCC for some speakers.All models show low CCC for at least one speaker on the three tasks, using speakers with at least 200 test samples.
  • Model agreement across speakers is strongest for arousal and dominance but weaker for valence, with Spearman correlations reaching .333 for valence.This indicates speaker-level behavior can differ despite similar overall test-set performance.

4.11 Why do foundation models generalise so well?

Transformer architectures provide speaker- and domain-abstracted latent spaces even without pre-training, helping explain their generalisation. Pre-training is nevertheless critical for learning useful valence representations and downstream performance.

  • Even without pre-training, wav2vec 2.0 produces a latent space that abstracts away from domain and speaker better than CNN14.The CNN14 space separates data sources, gender groups, and individual speakers more strongly.
  • Pre-training marginally improves arousal and dominance but is critical for valence.Without pre-training, wav2vec 2.0 performs worse than CNN14 in most comparisons, especially for valence and sentiment.
  • Only the pre-trained model shows a smooth transition from low to high valence scores in the latent space.Both pre-trained and untrained wav2vec 2.0 spaces lack clear domain, gender, or speaker clusters.

5 Efficiency

The paper evaluates efficiency through optimisation stability, computational complexity, and data efficiency. Pre-training accelerates convergence and stabilises training, while pruning and data reduction can preserve performance selectively across dimensions and corpora.

  • Optimisation stability: Pre-training reduces convergence time and performance volatility across training runs.Compared with CNN14, w2v2-b converges faster and permits reducing training to 5 epochs.
  • Computational complexity: Half of the transformer layers can be removed without performance loss, although reducing below 12 layers harms valence.The pruning experiment successively removes top layers from w2v2-L-robust before fine-tuning.
  • Data efficiency: Training-data reduction preserves arousal and dominance performance more readily than valence performance.For arousal and dominance, performance can saturate at 25% of MSP-Podcast data, while valence generally improves with more data.
  • Data efficiency: For arousal and dominance, 25% of MSP-Podcast data yields a loss below .02, while 12.5% on IEMOCAP stays within .05.These reductions apply to the reported dimension-specific comparisons and leave development and test sets untouched.
  • Data efficiency: Valence behaves differently: MOSI gains almost .1 in CCC with the remaining 25% of data, while IEMOCAP loses less than .05 at 7.5%.The paper links these patterns to linguistic diversity and vocabulary differences across corpora.

6 Summary

The unified analysis finds that pre-training is especially important for valence and stabilises training, while transformer models generalise well and improve group fairness. Individual-speaker fairness remains challenging, and fine-tuning self-attention enables integration of linguistic and paralinguistic information.

  • Effect of pre-training: Pre-training is essential for strong performance, especially valence, and acts as regularisation that stabilises training and reduces data and iteration requirements.Randomly initialised wav2vec 2.0 performs substantially worse on all three dimensions and fails to capture valence information.
  • Generalisation: Transformer models show strong cross-corpus generalisation and robustness, with these traits appearing primarily architectural rather than dependent on pre-training.The reported robustness and generalisation traits also appear in models initialised from random weights.
  • Fairness: Transformer architectures are fairer than CNN14 for the available gender group variable, but individual-speaker fairness remains difficult.The paper identifies individual fairness as an important and insufficiently investigated SER issue.
  • Integration of linguistic and paralinguistic streams: Fine-tuning self-attention layers enables transformers to integrate linguistic and paralinguistic information in the speech signal.Well-performing valence models retain effectiveness on synthesised speech without emotional intonation and do not benefit from explicit text fusion; frozen self-attention fails in that setting.

7 Conclusion

The paper establishes transformer-based foundation models as the new state-of-the-art direction for speech emotion recognition. It reports strong valence performance without explicit linguistic information, alongside gains in robustness, gender-group fairness, and cross-domain generalisation, while individual-level fairness remains unresolved.

  • Conclusion: CCC .638 on MSP-Podcast is reported as state-of-the-art valence recognition without explicit linguistic information.The paper attributes this result to implicit linguistic information learned through fine-tuning self-attention layers.
  • Conclusion: Transformer architectures are more robust to small perturbations, fair at the gender-group level, and generalise across domains.The conclusion qualifies fairness as group-level rather than individual-level.
  • Conclusion: The authors release w2v2-L-robust-12 to support reproducibility.This is identified as the best-performing model.
Loading 2203.07378v4…