Source-linked AI summary
Adversarial Auto-encoders for Speech Based Emotion Recognition
Saurabh Sahu, Rahul Gupta, Ganesh Sivaraman, Wael AbdAlmageed, Carol Espy-Wilson
TL;DR
Emotion recognition uses high-dimensional features that are difficult to analyze, and curated datasets are limited. This paper tests adversarial auto-encoders for compressing speech features and generating synthetic samples. The authors report that low-dimensional codes retain class discriminability and that synthetic samples show promise for classification.
Problem
Method
The authors use class-regularized adversarial auto-encoders to test classification from compressed code vectors and from synthetic samples used alone or alongside real data.
Source: §1 · pp. 1–2, §3.3.3 · p. 4Results
Low-dimensional code vectors retain class discriminability, while synthetic-only training exceeds chance and adding synthetic data yields a marginal, nonsignificant UAR increase.
Source: §4 · p. 4, §3.3.2 · p. 4, §3.3.3 · p. 4Takeaways & Limitations
The findings support low-dimensional code vectors for emotion-feature analysis and identify synthetic feature generation as a promising avenue for classification.
Source: §4 · p. 4, §3.3.2 · p. 4, §3.3.3 · p. 4The paper defers analyzing test-utterance membership in individual GMM components.
Source: §3.3.3 · p. 4
Abstract
from arXiv · showhide
Recently, generative adversarial networks and adversarial autoencoders have gained a lot of attention in machine learning community due to their exceptional performance in tasks such as digit classification and face recognition. They map the autoencoder's bottleneck layer output (termed as code vectors) to different noise Probability Distribution Functions (PDFs), that can be further regularized to cluster based on class information. In addition, they also allow a generation of synthetic samples by sampling the code vectors from the mapped PDFs. Inspired by these properties, we investigate the application of adversarial autoencoders to the domain of emotion recognition. Specifically, we conduct experiments on the following two aspects: (i) their ability to encode high dimensional feature vector representations for emotional utterances into a compressed space (with a minimal loss of emotion class discriminability in the compressed space), and (ii) their ability to regenerate synthetic samples in the original feature space, to be later used for purposes such as training emotion recognition classifiers. We demonstrate the promise of adversarial autoencoders with regards to these aspects on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpus and present our analysis.
1. Introduction
The introduction motivates adversarial auto-encoders as a way to address difficult high-dimensional feature analysis and limited training data in emotion recognition. It proposes evaluating both compressed code vectors and synthetic samples.
1. Introduction
- Emotion-recognition systems often use high-dimensional features that are difficult to analyze, while curated behavioral datasets can be expensive and time-consuming to obtain.The paper identifies both issues as motivation for its approach.Source: §1 · p. 1
- The study applies adversarial auto-encoders to emotion recognition, motivated by their prior use in digit and face recognition and their potential for compression and data generation.The authors describe this as the first application of adversarial auto-encoders to emotion recognition.Source: §1 · p. 1
- The experiments compare low-dimensional code-vector classification across adversarial auto-encoders, auto-encoders, PCA, and LDA, then test synthetic-only and augmented training data.The code vectors are the adversarial auto-encoder bottleneck outputs.Source: §1 · pp. 1–2
2. Background on adversarial auto-encoders
The background describes an adversarial auto-encoder as an auto-encoder paired with a discriminator that regularizes encoded data against samples from a chosen prior. Class information can be incorporated by matching mixture components to classes.
2. Background on adversarial auto-encoders
- The generator auto-encoder reconstructs input x through a bottleneck, while a discriminator distinguishes encoded code vectors from samples drawn from an arbitrary distribution.The generator learns to confuse the discriminator, which learns to distinguish the encoded and sampled inputs.Source: §2 · p. 2
- For class conditioning, the prior can be a mixture with one component per class, and one-hot class labels are supplied to the discriminator to associate components with classes.The paper attributes this setup to Makhzani et al. [7].Source: §2 · p. 2
3. Adversarial auto-encoders for emotion recognition
The study tests whether adversarial auto-encoders can compress speech-emotion features while retaining class information and generate synthetic features for emotion classification. On IEMOCAP, code vectors retained discriminative information, while synthetic-only training exceeded chance and adding synthetic data yielded a marginal, nonsignificant UAR increase.
3.1. Dataset
- The experiments used 4,490 IEMOCAP utterances across neutral, angry, sad, and happy labels, with majority annotator agreement.The dataset contains approximately 12 hours of speech from 10 subjects across five sessions.Source: §3.1 · p. 2
3.2. Features
- The input representation comprised 1,582 openSMILE spectral, prosody, and energy features.The paper notes that this high-dimensional feature space makes data-point analysis challenging.Source: §3.2 · p. 2
3.3. Experimental setup
- Five-fold cross-validation used one IEMOCAP session for testing, keeping training and test speakers independent.The experiments examined feature compression and synthetic-data creation.Source: §3.3 · p. 2
3.3.1. Training the adversarial auto-encoder
- The adversarial auto-encoder used label-informed regularization with a four-component GMM prior, chosen to encourage components to correspond to the four emotion labels.Training combined reconstruction loss with adversarial updates that trained the encoder to fool the discriminator.Source: §3.3.1 · p. 3
- Inner-fold validation selected a two-dimensional code; training and test loss trends converged, and test codes appeared fairly separable by emotion.Training-set instances were reported as perfectly encoded into specific GMM components, while test samples were fairly separable.Source: §3.3.1 · p. 3, Figure 2 · p. 3, Figure 3 · p. 4
3.3.2. Classification using the code vectors
- Code vectors performed fairly close to full openSMILE features (p=0.15) and better than other compression techniques, while vanilla auto-encoders performed less well.The authors report that the compressed vectors capture emotion-label differences in the original feature space to a fairly high degree.Source: §3.3.2 · pp. 3–4, Table 1 · p. 4
3.3.3. Classification using synthetically generated samples
- Synthetic-only training performed significantly above chance (p<0.05), while adding synthetic data marginally but not significantly increased overall UAR.The generated features did not follow the real samples’ class-marginal distribution because that distribution depended on the code-vector sampling strategy.Source: Table 2 · p. 4, §3.3.3 · p. 4
4. Conclusion
The adversarial auto-encoder produces low-dimensional code vectors without losing the class discriminability of higher-dimensional features, and synthetic samples show promise for improving real-world emotion classification. Future work will examine the code space, synthetic-vector classification, frame-level features, and joint analysis of engagement and emotion.
4. Conclusion
- The adversarial auto-encoder's low-dimensional code vectors retain the class discriminability of higher-dimensional features.The conclusion states that the code vectors can be obtained in a low-dimensional subspace without losing that discriminability.Source: §4 · p. 4
- Synthetic samples show promise as a method for improving classification of real-world data.Source: §4 · p. 4
- Future work will analyze emotional utterances in the low-dimensional code space and further improve classification schemes using synthetic vectors.Source: §4 · p. 4
- The authors plan to test frame-level rather than utterance-level features, based on their belief that temporal feature dynamics can improve classification.Source: §4 · p. 4
- The adversarial auto-encoder may also be used to analyze engagement jointly with emotional states.Source: §4 · p. 4