Source-linked AI summary
x-vectors meet emotions: A study on dependencies between emotion and speaker recognition
Raghavendra Pappagari, Tianzi Wang, Jesus Villalba, Nanxin Chen, Najim Dehak
TL;DR
The paper examines whether speaker-recognition models can be reused for emotion recognition and how emotion affects speaker verification. It transfers x-vector representations to SER, including fine-tuning, and reports improved emotion recognition alongside sensitivity of verification performance to test-speaker emotion.
Problem
The paper examines how speaker-recognition representations relate to emotion recognition and whether test-speaker emotion affects speaker verification.
Source: §1 · p. 1Method
The study transfers pre-trained x-vector models to emotion recognition, including fine-tuning, and evaluates speaker verification on emotion datasets.
Source: §1 · p. 1, §2.1 · p. 2Results
Fine-tuned x-vector models improve emotion recognition, while speaker verification is sensitive to test-speaker emotion; same-emotion trials perform better than cross-emotion trials, with angry utterances common among the worst-performing cross-emotion trials.
Source: §5 · p. 4Takeaways & Limitations
The authors identify emotion-invariant speaker verification models as a direction for future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In this work, we explore the dependencies between speaker recognition and emotion recognition. We first show that knowledge learned for speaker recognition can be reused for emotion recognition through transfer learning. Then, we show the effect of emotion on speaker recognition. For emotion recognition, we show that using a simple linear model is enough to obtain good performance on the features extracted from pre-trained models such as the x-vector model. Then, we improve emotion recognition performance by fine-tuning for emotion classification. We evaluated our experiments on three different types of datasets: IEMOCAP, MSP-Podcast, and Crema-D. By fine-tuning, we obtained 30.40%, 7.99%, and 8.61% absolute improvement on IEMOCAP, MSP-Podcast, and Crema-D respectively over baseline model with no pre-training. Finally, we present results on the effect of emotion on speaker verification. We observed that speaker verification performance is prone to changes in test speaker emotions. We found that trials with angry utterances performed worst in all three datasets. We hope our analysis will initiate a new line of research in the speaker recognition community.
1. INTRODUCTION
The paper investigates whether speaker-recognition representations can support emotion recognition and how emotion affects speaker verification. It evaluates transfer learning, fine-tuning, and emotion-data augmentation, while addressing speaker overlap in prior emotion-related work.
1. INTRODUCTION
- Shared acoustic cues, including pitch, fundamental frequencies, and acoustic energy, motivate testing whether speaker-trained models can be reused for emotion recognition.The authors hypothesize that these cues support both tasks.Source: §1 · p. 1
- The experiments extend earlier transfer-learning work and address an identified gap: prior study of speaker-based utterance representations did not examine emotion-related tasks.The paper situates its approach alongside work reusing ASR models for SER and studying downstream tasks with i-vectors and x-vectors [14, 15, 16].Source: §1 · p. 1
- Prior work found emotion information in speaker-based representations, but included overlapping speakers across training and test splits; this study avoids that issue in its SER experiments.The authors also distinguish their use of pre-trained and fine-tuned models from that prior study.Source: §1 · p. 1
- The study transfers speaker-discriminative models to emotion recognition and examines emotion’s effect on speaker verification.It also evaluates fine-tuning and emotion-data augmentation.Source: §1 · p. 1
2. OUR APPROACH
The approach reuses an x-vector model for emotion recognition, either by applying a linear classifier to extracted embeddings or by fine-tuning for emotion targets. Its multi-head attention pooling summarizes frame-level outputs into an utterance embedding for speaker classification.
2. OUR APPROACH
- The study uses x-vectors for downstream tasks because of their reported superiority over i-vectors and ease of adaptation.The method section presents the model and its transfer-learning approach for SER.Source: §2 · pp. 1–2
2.1. x-Vector Model
- The ResNet-34 frame network feeds multi-head attention pooling, whose head embeddings are concatenated and projected before two fully connected speaker-classification layers.The network is trained with angular softmax loss.Source: §2.1 · p. 2, Table 1 · p. 2
2.2. Emotion Recognition
- SER transfer either applies logistic regression to extracted x-vectors or replaces the speaker-discriminative output layer with an emotion-discriminative layer and fine-tunes.Both approaches are compared with OpenSMILE features and clean- and augmented-data x-vector models.Source: §2.2 · p. 2
3. EXPERIMENTAL SETUP
Experiments use three emotion datasets spanning acted and natural speech, with different content restrictions and dataset splits. Acoustic inputs are extracted as frame-based MFCC or GeMAPS features.
3.1. Datasets
- The study evaluates emotion recognition on IEMOCAP, MSP-Podcast, and Crema-D, which differ in whether speech is acted or natural and whether content is restricted.IEMOCAP is acted, MSP-Podcast is natural, and Crema-D is acted with a restricted sentence list.Source: §3.1 · p. 2
3.1.1. IEMOCAP
- IEMOCAP uses four emotion classes and five-fold cross-validation with speaker- and session-disjoint training and testing.Happy and excited are combined; weighted F-score is averaged across folds.Source: §3.1.1 · p. 2
3.1.2. MSP-Podcast Dataset
- MSP-Podcast experiments use five emotion classes and the standard Release 1.4 train, development, and test splits.The splits contain 610 training speakers, 30 development speakers, and 50 test speakers.Source: §3.1.2 · p. 2
3.1.3. Crema-D Dataset
- Crema-D experiments use four emotion classes and actor-disjoint training, development, and test sets.Disgust and fear are discarded; the dataset uses 12 sentences, with 51 actors for training, 8 for development, and 32 for testing.Source: §3.1.3 · pp. 2–3
3.2. Feature Extraction
- MFCC inputs use 23 dimensions with 10 ms frame shifts and 25 ms frames, while GeMAPS uses 88 dimensions at the same frame settings.A simple energy-based detector removes silence; the pre-trained models use MFCC.Source: §3.2 · p. 3
4. RESULTS
Fine-tuned x-vector models improve emotion recognition across three datasets, while speaker verification varies with test-speaker emotion. The results also show dataset- and training-condition differences in emotion recognition and verification.
4.1. Emotion recognition
- Pre-trained models significantly improved over random initialization, and fine-tuning improved results in all cases except a 3.03% drop on Crema-D with ResNet-clean.In clean-data training, ResNet-clean exceeded ResNet-aug by 2.94%, 4.17%, and 3.38% on IEMOCAP, MSP-Podcast, and Crema-D, respectively.Source: §4.1 · p. 3
- Emotion-data augmentation helped on IEMOCAP and Crema-D except with ResNet-clean, while MSP-Podcast improvements were unclear.The reported augmentation gains occurred with fixed pre-trained models but not when fine-tuning for emotion recognition.Source: §4.1 · p. 3
- 30.40%, 7.99%, and 8.61% absolute improvements over the randomly initialized ResNet (MFCC) baseline were obtained on IEMOCAP, MSP-Podcast, and Crema-D, respectively.Fine-tuned ResNet-aug performed best on IEMOCAP and Crema-D in Clean+aug training, at 70.30% and 81.54%; on MSP-Podcast it reached 58.46% with clean training data.Source: §4.1 · p. 3
4.2. Speaker Verification
- IEMOCAP verification EERs were very high, which the authors attribute to very short utterances and domain mismatch; most IEMOCAP utterances were under four seconds.MSP-Podcast utterances were mostly short but longer than IEMOCAP, while Crema-D had better EERs despite shorter utterances, possibly because phonetic content was limited to 12 sentences.Source: Figure 1 · p. 3, §4.2 · p. 4
- Speaker-verification EER was worse when test and enrollment emotions differed; angry emotion appeared in the worst cross-emotion pair on each dataset.Cross-emotion trials were worst overall except on MSP-Podcast; enrollment emotions are table rows and test emotions are columns.Source: §4.2 · p. 4
5. CONCLUSIONS AND FUTURE WORK
Fine-tuning x-vector models improved emotion recognition across three datasets, while speaker verification was sensitive to test-speaker emotion. Same-emotion trials performed better than cross-emotion trials, with angry utterances common among the worst-performing cross-emotion trials.
5. CONCLUSIONS AND FUTURE WORK
- 30.40%, 7.99%, and 8.61% absolute improvement on IEMOCAP, MSP-Podcast, and Crema-D, respectively, over the baseline model with no pre-training.These were the reported gains from fine-tuning pre-trained x-vector models.Source: §5 · p. 4
- Fine-tuned pre-trained x-vector models achieved the best emotion-recognition performance on all three datasets; unaugmented x-vector features outperformed augmented features.Data augmentation for emotion classification improved results on two of the three datasets, while the unaugmented x-vector model was not robust to noise.Source: §5 · p. 4
- Features from pre-trained models performed better than emotion-specific GeMAPS features for emotion recognition.Source: §5 · p. 4
- Speaker verification was sensitive to test-speaker emotion: same-emotion trials performed better than cross-emotion trials, and angry utterances were common among the worst-performing cross-emotion trials.The conclusion reports this pattern across all three datasets.Source: §5 · p. 4
- The authors identify emotion-invariant speaker verification models as future work.Source: §5 · p. 4