Augmenting Generative Adversarial Networks for Speech Emotion Recognition

• Limited labelled SER data constrains performance and can cause dimensionality problems, while low-dimensional features do not always preserve the best recognition performance.Source: §1 · p. 1
• Mixup is combined with GANs to address convergence issues in synthetic emotional feature generation and learn compressed emotional representations.Source: §1 · p. 1
• The framework evaluates compressed representations, synthetic samples, and training-data augmentation using within-corpus and cross-corpus SER experiments.Source: §1 · p. 1
• Results show better performance than recent studies across real, synthetic, and augmented-data settings.Source: §1 · p. 1

• Prior GAN-based SER studies used CycleGANs or conditional GANs to generate synthetic emotional features, but vanilla GAN convergence and synthetic-feature performance remained limited.Source: §2 · p. 1
• Generative and GAN-based methods learned emotional representations from unlabelled data in semi-supervised settings, language-invariant features, or lower-dimensional encodings.Source: §2 · pp. 1–2
• Mixup has improved speaker verification and ASR systems, but had not previously been used with GANs to augment feature learning and generation.Source: §2 · p. 2

3. Proposed Framework

• The framework consists of mixup and GAN components that jointly support representation learning and synthetic feature generation.Source: §3 · p. 2

3.1. Mixup Augmentation

• Mixup creates virtual training examples by convexly interpolating randomly selected training samples and labels, extending the training distribution with linear combinations.Source: §3.1 · p. 2

3.2. Generative Adversarial Networks (GANs)

• A GAN trains a generator and discriminator adversarially so generated samples mimic real data, supporting feature learning and generation despite limited SER data.Source: §3.2 · p. 2

3.3. Augmenting GANs

• The modified GAN feeds mixup samples to an encoder, uses encoded features as generator inputs, and lets the generator also act as the autoencoder decoder.Source: §3.3 · p. 2
• The discriminator distinguishes synthetic samples from real samples with λ = 0, 1, while excluding mixed samples from discriminator updates to focus generation on real-sample structure.Source: §3.3 · p. 2, Figure 1 · p. 2
• Training iteratively updates the autoencoder, generator, and discriminator, with the final discriminator update restricted to real samples.Source: §3.3 · p. 3

4.1. Dataset

• IEMOCAP supplies 5,531 utterances across angry, happy, neutral, and sad classes, while MSP-IMPROV supplies 7,798 utterances for cross-corpus evaluation.Source: §4.1 · p. 3

4.2. Features

• The experiments use 1,582-dimensional IS10 features containing prosodic, spectral, and energy-based information, with mixup applied directly to training features and labels.Source: §4.2 · p. 3

4.3. Model Configuration and Training

• The encoder and decoder each comprise two hidden layers with 1,000 and 500 units, while the discriminator has two hidden layers with 1,000 neurons each; the autoencoder uses dropout regularisation of 0.5.Source: §4.3 · p. 3
• Evaluation uses speaker-independent leave-one-session-out cross-validation, UAR, five repetitions, and reported means with standard deviations.Source: §4.3 · p. 3
• Synthetic-feature experiments use min-max normalisation, whereas cross-corpus classification uses separate z-normalisation.Source: §4.3 · p. 3

5.1.1. Synthetic features

• Compared with, the framework achieves better classification results for real, synthetic, and real+synthetic feature settings.Compared with, it performs better for real and real+synthetic features but slightly worse for synthetic features.Source: §5.1.1 · p. 3, Table 1 · p. 3
• Real+synthetic features improve prediction performance compared with real features alone in the IEMOCAP confusion-matrix analysis.The synthetic-feature experiment compares real, synthetic, and combined inputs; the corresponding visual is Figure 2.Source: §5.1.1 · pp. 3–4, Figure 2 · p. 4
• Synthetic-feature predictions are comparable to real-feature predictions for angry and sad, but lower for happy and neutral.Compared with, the proposed model produces results closer to real samples across classes, whereas performs poorly on happy and neutral.Source: §5.1.1 · pp. 3–4, Figure 2 · p. 4

5.1.2. Encoded Features

• The framework performs better than other dimensionality-reduction techniques while encoding emotional information in lower-dimensional features.The comparison uses SVM classification on 25-dimensional representations, including mixup-enhanced baselines.Source: §5.1.2 · p. 4, Table 2 · p. 4

5.2. Cross-corpus evaluation

• The proposed model achieves improved cross-corpus SER results compared with and, using synthetic data and training data augmented with synthetic samples.Source: §5.2 · p. 4

6. Conclusions

• Mixup-augmented GANs improve compressed emotional representation learning and generate synthetic features that support SER performance improvement.The conclusion reports benefits for representation learning, synthetic feature generation, and training-data augmentation.Source: §6 · p. 4