Source-linked AI summary
Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognition
Li-Wei Chen, Alexander Rudnicky
TL;DR
SER lacks abundant annotated data and differs from the speech used for ASR pretraining. This paper evaluates V-FT and TAPT for wav2vec 2.0 and introduces P-TAPT, which learns emotion-specific contextualized representations. P-TAPT yields the strongest reported IEMOCAP result, with a 7.4% absolute UA improvement over prior state-of-the-art performance and greater gains in low-resource settings.
Problem
SER has limited annotated data, while mismatch between ASR pretraining speech and emotive target speech motivates adaptation methods.
Method
The paper compares V-FT and TAPT and introduces P-TAPT, which replaces audio reconstruction with masked-frame emotion-state prediction using pseudo-labels.
Results
P-TAPT achieves a 7.4% absolute improvement in UA over state-of-the-art IEMOCAP performance and is more data-efficient than TAPT under low-resource conditions.
Takeaways & Limitations
Fine-tuning strategies that address domain shift and learn contextualized emotion representations improve wav2vec 2.0 for SER.
Abstract
from arXiv · showhide
While Wav2Vec 2.0 has been proposed for speech recognition (ASR), it can also be used for speech emotion recognition (SER); its performance can be significantly improved using different fine-tuning strategies. Two baseline methods, vanilla fine-tuning (V-FT) and task adaptive pretraining (TAPT) are first presented. We show that V-FT is able to outperform state-of-the-art models on the IEMOCAP dataset. TAPT, an existing NLP fine-tuning strategy, further improves the performance on SER. We also introduce a novel fine-tuning method termed P-TAPT, which modifies the TAPT objective to learn contextualized emotion representations. Experiments show that P-TAPT performs better than TAPT, especially under low-resource settings. Compared to prior works in this literature, our top-line system achieved a 7.4\% absolute improvement in unweighted accuracy (UA) over the state-of-the-art performance on IEMOCAP. Our code is publicly available.
1. INTRODUCTION
Speech emotion recognition is constrained by limited annotated data and poor cross-domain generalization. This paper explores wav2vec 2.0 fine-tuning, including TAPT and a new P-TAPT strategy, for improved SER.
- SER datasets are small relative to ASR datasets, limiting deep-learning improvements and potentially weakening generalization to domains such as call centers.
- Wav2vec 2.0 uses a transformer-based masked-learning objective to predict missing frames from remaining context.
- TAPT continues pretraining on the target dataset to address mismatch between pretraining and target-domain data.
- V-FT outperforms state-of-the-art SER methods on IEMOCAP, while TAPT further improves wav2vec 2.0 fine-tuning.
- P-TAPT modifies the fine-tuning procedure to improve performance, especially under low-resource conditions, achieving a 7.4% absolute UA increase over IEMOCAP state-of-the-art performance.
2. METHOD
The paper compares vanilla fine-tuning and TAPT with P-TAPT, which adapts wav2vec 2.0 toward emotion-specific contextualized representations using pseudo-labels.
- The method review establishes wav2vec 2.0 as the backbone, followed by V-FT and TAPT baselines and the novel P-TAPT procedure.
- Wav2vec 2.0 combines a CNN feature encoder, transformer module, and quantization module to learn contextualized representations from raw audio.
- Vanilla fine-tuning: V-FT averages final-layer contextualized representations across time, then applies ReLU and a linear classifier for utterance-level emotion prediction.
- Task adaptive pretraining: TAPT continues wav2vec 2.0 pretraining on the target SER dataset during an intermediate task-adaptation stage.
- Pseudo-label task adaptive pretraining: P-TAPT predicts emotion-state cluster assignments for masked frames instead of reconstructing missing audio features, producing emotion-specific representations.
3. EXPERIMENTAL SETUP
The experiments evaluate speech-only SER systems on IEMOCAP and SAVEE under speaker-independent protocols, using unweighted accuracy as the metric.
- Evaluation uses the speech modality on IEMOCAP and SAVEE.
- Datasets: IEMOCAP contains about 7 hours across five sessions and uses four emotion categories after merging excited with happy.
- Datasets: SAVEE contains 480 utterances from four male speakers, covering seven emotion categories and 30 minutes of speech.
- Evaluation: Systems are evaluated with unweighted accuracy under speaker-independent testing, using LibriSpeech-pretrained wav2vec models.
- Evaluation: IEMOCAP uses leave-one-session-out testing, whereas SAVEE leaves one speaker out in each fold.
4. RESULTS AND DISCUSSION
P-TAPT consistently improves over TAPT on IEMOCAP, while TAPT and P-TAPT substantially outperform V-FT on SAVEE. P-TAPT is especially advantageous with limited IEMOCAP training data and achieves a 7.4% absolute UA improvement over prior state-of-the-art models.
- On IEMOCAP, TAPT improves over V-FT for all sessions except the first, while P-TAPT outperforms TAPT in every session.
- On SAVEE, both TAPT and P-TAPT outperform V-FT by a large margin, but P-TAPT performs very close to TAPT.
- Domain shift exists between the pretraining corpus and both SER datasets, providing room for TAPT to improve performance.The pretraining loss analysis uses LibriSpeech as the pretraining reference and compares it with SAVEE and IEMOCAP.
- P-TAPT is more data-efficient than TAPT when IEMOCAP training data are limited to one hour or less.The comparison uses the ratio of improvements over V-FT; lower ratios indicate a larger relative improvement from P-TAPT.
- 7.4% absolute UA improvement over state-of-the-art models is achieved by P-TAPT on IEMOCAP.
5. CONCLUSION
The paper presents fine-tuning strategies for wav2vec 2.0 on SER, including a method for learning contextualized emotion representations. These strategies achieve state-of-the-art performance on IEMOCAP and support further study across tasks and multimodal settings.
- The paper presents multiple wav2vec 2.0 fine-tuning strategies for speech emotion recognition.
- The methods address domain shift in SER and show that addressing it improves performance.
- The paper introduces contextualized emotion representation learning and demonstrates its advantage for fine-tuning wav2vec 2.0 on SER.
- The reported strategies produce state-of-the-art performance on the IEMOCAP corpus.