SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training

1. Introduction

• Speaker variability and scarce emotion labels complicate SER, particularly when models must generalize to unseen speakers.Emotion and speaker information are intertwined acoustically, and speaker-specific correlations learned on fixed speakers may not transfer.Source: §1 · p. 1
• Pretrained representations can reduce reliance on annotated data, but emotion-only fine-tuning does not guarantee suppression of speaker patterns.Residual speaker-specific patterns can limit generalization to unseen speakers.Source: §1 · p. 1
• SISER replaces a conventional encoder with wav2vec 2.0 and a shallow speaker classifier with ECAPA-TDNN under entropy-based adversarial training.The stronger discriminator is intended to detect speaker-relevant patterns that a linear classifier may miss.Source: §1 · p. 1
• The paper identifies speaker-classifier architecture as important to disentanglement and presents SISER as a framework combining wav2vec 2.0, ECAPA-TDNN, and entropy-based adversarial training.The contribution statement highlights stronger speaker discrimination as central to more thorough disentanglement.Source: §1 · p. 1

2.1. Self-supervised representations for speech emotion recognition

• wav2vec 2.0 learns contextualized frame-level speech features from raw waveforms and has been applied to SER under limited supervision.Prior work also reports complementary acoustic information across self-supervised model layers.Source: §2.1 · pp. 1–2

2.2. Adversarial training and speaker discriminator design

• Entropy maximization encourages a uniform posterior over speaker identities, unlike gradient reversal, which does not specify how speaker classification should fail.The cited discussion notes that a collapsed representation can also satisfy gradient reversal.Source: §2.2 · p. 2
• Prior SER adversarial frameworks mostly used shallow speaker classifiers, while SISER uses ECAPA-TDNN’s multi-scale context and channel-dependent attention as its discriminator.The authors describe this as the first use of ECAPA-TDNN in an entropy-based adversarial SER scheme.Source: §2.2 · p. 2

3. Methodology

• The methodology targets fixed-dimensional utterance embeddings that maximize emotion information while removing speaker-related information through adversarial training.The procedure places a feature encoder and strong speaker classifier in an adversarial relationship.Source: §3 · p. 2

3.1. Model Structure

• The model uses wav2vec 2.0 to encode raw waveforms into contextualized frame-level representations, which feed an emotion classifier and an ECAPA-TDNN speaker classifier.The emotion head predicts categories; the speaker head predicts training-fold identities using multi-scale temporal context.Source: §3.1 · p. 2

3.2. Adversarial Training

• Entropy maximization targets a uniform speaker posterior, whereas gradient reversal only penalizes speaker discrimination without specifying the encoder’s failure pattern.The approach also addresses the limited capacity of shallow speaker classifiers, which may miss speaker-relevant dimensions.Source: §3.2 · p. 2

3.2.1. Training of SC

• During the speaker-classifier step, SC minimizes cross-entropy for speaker prediction while ENC and EC remain frozen.This update improves speaker identification from the encoder output before the adversarial encoder update.Source: §3.2.1 · p. 2

3.2.2. Training of ENC and EC

• ENC and EC jointly preserve emotion-predictive content while maximizing SC’s speaker-output entropy; λ balances emotion classification against speaker suppression.Smaller λ strengthens suppression but risks discarding emotion-relevant information; ECAPA-TDNN supplies multi-scale, channel-attentive adversarial gradients.Source: §3.2.2 · pp. 2–3

4. Dataset

• IEMOCAP contributes 5,531 utterances from ten speakers across five sessions, labeled as angry, happy, neutral, or sad.Excitement and happiness are merged; the resulting counts are 1,103 angry, 1,636 happy, 1,708 neutral, and 1,084 sad utterances.Source: §4 · p. 3
• The reproduced baseline omits speed-perturbation augmentation so the experiments can isolate encoder and speaker-classifier design.The authors also describe this as reflecting a setting where augmentation may not always be applicable.Source: §4 · p. 3

5. Experiment Setup

• Main experiments use 10-fold leave-one-session cross-validation, with eight speakers for training, one for validation, and one unseen speaker for testing.All reported results reflect performance on unseen speakers.Source: §5 · p. 3
• For wav2vec 2.0 variants, the pretrained base model is fine-tuned on IEMOCAP with only its last two transformer layers unfrozen.The baseline follows the feature-extraction procedure of.Source: §5 · p. 3
• Training uses Adam at 1×10−4 for 300 epochs with batch size 64; the emotion head has three PReLU fully connected layers, and experiments run on one NVIDIA A100 GPU.The ECAPA-TDNN input dimension is adjusted to 768, and λ is set to 0.5 following.Source: §5 · p. 3

6.1. Evaluation on IEMOCAP

• The baseline achieves 51.15% UA and 50.14% WA; wav2vec 2.0 without adversarial training achieves 56.46% UA and 54.45% WA, while ECAPA-TDNN with GRL achieves 56.95% UA and 56.01% WA.The baseline result is without data augmentation; the GRL result is slightly below the wav2vec 2.0 vanilla system.Source: §6.1 · p. 3
• 60.63% UA and 58.53% WA; SISER improves 9.48% in UA over the baseline without data augmentation.It also achieves comparable performance to the augmented baseline (59.91% UA) without data augmentation.Source: §6.1 · p. 3

6.2. t-SNE Visualization

• SISER embeddings show clearer emotion-category separation than the baseline and vanilla wav2vec 2.0 embeddings in the two-speaker Session 1 visualization.Figure 2 colors embeddings by emotion and uses marker shapes to indicate speaker identity.Source: §6.2 · p. 4, Figure 2 · p. 4

6.3. Ablation Study

• ECAPA-TDNN adds 7.13% UA with CNN+GRU and 6.49% with wav2vec 2.0; wav2vec 2.0 adds 1.73% UA with FC and 1.09% with ECAPA-TDNN.The Fold 2 ablation compares two encoder types and two speaker-classifier configurations.Source: §6.3 · p. 4, Table 2 · p. 4

6.4. 10-fold Stability Analysis

• Across 10 folds, SISER has mean UA of 62.73% ± 1.72, versus 54.20% ± 3.55 for the baseline and 59.22% ± 2.68 for vanilla wav2vec 2.0.The lower variance is described as indicating more consistent representations across speaker pairings and improved robustness to training-set composition.Source: §6.4 · p. 4, Table 3 · p. 4

6.5. Conclusion

• The conclusion identifies speaker-classifier discriminative capacity as a key factor in adversarial disentanglement and anticipates gains from more sophisticated classification heads.The paper uses a simple emotion classification head to isolate speaker-invariant representation learning.Source: §6.5 · p. 4

• AI is used only for editing.Source: §8 · p. 5