Source-linked AI summary
Domain Adversarial for Acoustic Emotion Recognition
Mohammed Abdelwahab, Carlos Busso
TL;DR
Speech emotion recognition loses performance when train and test distributions differ, and obtaining target-domain emotion labels is costly. The paper uses adversarial multitask training with unlabeled target data to learn domain-invariant representations, reporting consistent gains across emotional dimensions. It also analyzes representation convergence and architectural choices, while noting a limitation for multiple source domains.
Problem
Speech emotion recognition must handle mismatched train and test distributions without repeatedly collecting expensive target-domain emotion labels.
Method
A domain-adversarial network jointly predicts emotional attributes from labeled source data and trains a domain classifier on labeled source and unlabeled target data.
Results
Adversarial training improves CCC by 27.3% relative to source-only training on average, with gains across arousal, valence, and dominance.
Takeaways & Limitations
Unlabeled target-domain data can support domain-invariant representations that reduce train-test mismatch while retaining emotion-discriminative information.
Takeaways & Limitations
When multiple sources are combined, the approach treats them as one domain, motivating alternative asymmetric representations as a possible improvement.
Abstract
from arXiv · showhide
The performance of speech emotion recognition is affected by the differences in data distributions between train (source domain) and test (target domain) sets used to build and evaluate the models. This is a common problem, as multiple studies have shown that the performance of emotional classifiers drop when they are exposed to data that does not match the distribution used to build the emotion classifiers. The difference in data distributions becomes very clear when the training and testing data come from different domains, causing a large performance gap between validation and testing performance. Due to the high cost of annotating new data and the abundance of unlabeled data, it is crucial to extract as much useful information as possible from the available unlabeled data. This study looks into the use of adversarial multitask training to extract a common representation between train and test domains. The primary task is to predict emotional attribute-based descriptors for arousal, valence, or dominance. The secondary task is to learn a common representation where the train and test domains cannot be distinguished. By using a gradient reversal layer, the gradients coming from the domain classifier are used to bring the source and target domain representations closer. We show that exploiting unlabeled data consistently leads to better emotion recognition performance across all emotional dimensions. We visualize the effect of adversarial training on the feature representation across the proposed deep learning architecture. The analysis shows that the data representations for the train and test domains converge as the data is passed to deeper layers of the network. We also evaluate the difference in performance when we use a shallow neural network versus a \emph{deep neural network} (DNN) and the effect of the number of shared layers used by the task and domain classifiers.
I. INTRODUCTION
Speech emotion recognition suffers when training and testing data distributions differ, while annotating target-domain emotion labels is expensive. The paper proposes domain-adversarial training that uses unlabeled target data to learn representations supporting robust emotion recognition.
- Motivation: Different train-test data distributions degrade speech emotion recognition performance.Building target-specific labeled datasets can be expensive.
- Proposed solution: The proposed approach combines labeled data from similar source domains with unlabeled target-domain data to improve generalization.This avoids requiring additional emotional annotations for the new target domain.
- Proposed solution: The model learns a common representation in which source and target samples become indistinguishable while retaining information for predicting arousal, valence, and dominance.Adversarial multitask training aligns domain distributions through back-propagation.
- Findings: Adversarial training yields a 27.3% relative improvement in CCC on average over source-only training.The study also examines representation alignment and how the number of shared layers affects performance.
II. RELATED WORK
Prior work addresses train-test mismatch by normalizing, reweighting, reconstructing, or factorizing cross-domain features. This paper introduces domain-adversarial training for speech emotion recognition as an unsupervised alternative that learns domain-robust representations from unlabeled target data.
- Mismatch reduction: Cross-corpus evaluation commonly reveals reduced speech emotion recognition performance under changing conditions.This motivates methods that reduce variability between train and test domains.
- Mismatch reduction: Feature-space methods reduce mismatch through corpus-specific normalization or weighting training examples resembling the test distribution.Examples include separate feature normalization and kernel-based distribution matching.
- Shared representations: Auto-encoders and related models learn shared representations or transformations intended to make features more robust across domains.These approaches use reconstruction objectives or shared-feature constraints.
- Shared representations: NMF-based methods use data from both domains and MMD regularization to minimize feature-distribution differences.They can incorporate unlabeled data or target-domain labels as constraints.
- Domain adversarial training: This work applies DANN to speech emotion recognition, where it had not previously been used despite prior use in speech recognition.The intended contribution is a principled reduction of train-test mismatch.
III. PROPOSED APPROACH
The proposed unsupervised framework uses unlabeled target data to learn discriminative features shared across source and target domains. A domain-adversarial network aligns representations while preserving the primary emotion-recognition task.
- Motivation: The approach creates a discriminative feature representation by leveraging unlabeled target-domain data.It is designed to reduce mismatch between source and target domains without target labels.
- Domain-adversarial training: A gradient reversal layer makes source and target samples indistinguishable to the domain classifier while preserving task-relevant information.The reversed domain gradient updates shared features in the direction that aligns the domains.
- Network architecture: The network contains shared feature representations, a task classifier, and a domain classifier trained with labeled source and unlabeled target data.The task predicts emotional attributes, while the domain classifier uses domain membership.
- Scope: The unsupervised framework improves target-domain performance and can also operate when limited target labels are available.In that setting, the method functions as a semi-supervised approach.
B. Domain Adversarial Neural Network for Emotion Recognition
The domain-adversarial network jointly predicts emotional attributes and domain membership, using unlabeled target data to learn representations that preserve emotion information while reducing source–target differences.
- Architecture: The network trains a main emotional-attribute predictor and a domain classifier using labeled source data and unlabeled target data.The domain classifier distinguishes source from target examples without requiring emotional labels.
- Architecture: Shared layers learn a regularized feature representation, while separate parameters serve the main prediction and domain-classification tasks.The objective uses parameters θf, θy, and θd for shared features, the main task, and the domain classifier.
- Optimization: The prediction and domain-classification losses compete adversarially in a minimax objective controlled by the regularization multiplier λ.The two losses are optimized toward a saddle point rather than independently.
- Optimization: A gradient reversal layer reverses domain-classifier gradients so feature updates make source and target representations more similar.The source classification loss is minimized while the domain-classification loss is maximized.
- Outcome: The resulting common representation retains source-domain discriminative information while reducing domain distinguishability and improving target-domain performance without new emotional labels.The domain classifier can be driven toward random-chance performance as the distributions align.
IV. EXPERIMENTAL EVALUATION
The evaluation defines emotion recognition as dimensional regression and tests domain adaptation across three emotional speech databases, with two source corpora and one target corpus.
- Evaluation setting: The main task estimates arousal, valence, and dominance as continuous emotional attributes in a regression framework.The attributes correspond to calm–activated, negative–positive, and weak–strong dimensions.
- Evaluation setting: The multi-corpus evaluation uses USC-IEMOCAP and MSP-IMPROV as source-domain databases and MSP-Podcast as the target-domain database.The target-domain data are used to represent the testing condition.
1) The USC-IEMOCAP Corpus:
The evaluation draws on acted and improvised emotional-speech resources, with annotations covering arousal, valence, and dominance and a separate natural-speech target corpus.
- USC-IEMOCAP: USC-IEMOCAP is an audiovisual corpus recorded from ten actors during dyadic interactions, although this study uses only its audio.Its recordings total approximately 12 hours and use scripted and improvised elicitation frameworks.
- MSP-IMPROV: MSP-IMPROV records actors in dyadic sessions designed to elicit more natural emotional behavior while controlling lexical and emotional content.Its improvised scenarios target happy, angry, sad, and neutral reactions and include surrounding interaction turns.
- MSP-IMPROV: MSP-IMPROV contains 8,438 emotional turns from 12 actors, annotated by at least five evaluators on arousal, valence, and dominance.The dimensional annotations use integer values from one to five.
- MSP-Podcast: MSP-Podcast provides natural speech from many speakers and varied recording conditions, including spontaneous conversations conveying emotional behavior.The recordings come from Creative Commons audio-sharing sources.
- Annotations: The broader corpus annotation procedure uses at least five raters, averages their scores, and shifts consensus values to the range −3 to 3.The study uses 14,227 labeled sentences, including 4,283 test sentences from 50 speakers.
B. Acoustic Features
The evaluation uses high-dimensional ComParE acoustic features, domain-specific robust normalization, and configurable shallow or deep network components alongside matched baselines.
- Acoustic representation: The acoustic representation contains 6,373 sentence-level features derived from 65 frame-level low-level descriptors.The descriptors include MFCCs, fundamental frequency, and energy, with higher-level statistics computed from them.
- Preprocessing: Features are normalized separately by domain using trimmed quantiles, then values exceeding ten standard deviations are set to zero.Quantile-based means and variances reduce the influence of outliers during normalization.
- Network structure: The DANN architecture contains feature-representation, task-classifier, and domain-classifier layers, with the number of shared representation layers treated as a network parameter.The domain classifier uses two layers, while the task classifier generally uses two layers.
- Network structure: The shallow-network comparison uses one task-classifier layer and one feature-representation layer, while retaining two domain-classifier layers.This isolates the effect of network depth in the evaluation.
- Baselines: The baselines include source-only training and within-target-corpus training, using comparable architectures but omitting domain-classification layers.The first baseline represents train–test mismatch, whereas the second assumes labeled target-domain training data.
E. Implementation
The implementation uses regularized neural networks, scheduled adversarial loss weighting, and domain-balanced unlabeled target samples. Performance is evaluated with RMSE, Pearson’s correlation, and especially CCC, which combines MSE and correlation.
- The models use batch normalization, dropout, max-norm weight regularization, gradient clipping, and MSE for the main regression task.Dropout is p=0.2 at the input layer and p=0.5 elsewhere; max-norm is four and clip norm is ten.
- The DANN parameter λ controls the tradeoff between task and domain classification losses, increasing from zero during the first ten epochs to 1 by training completion.Models are trained for 100 epochs with batch size 256 and Adam learning rate 5e−4.
- Unlabeled target-domain samples are randomly selected in numbers matching the source-domain samples to balance domain-classification training.
- Performance is reported using RMSE, Pearson’s correlation coefficient, and concordance correlation coefficient between ground-truth and estimated values.
- CCC is the primary analysis metric because it combines mean square error and Pearson’s correlation coefficient.Its definition uses the correlation, standard deviations, and means of predicted scores and ground-truth labels.
A. Number of Layers For the Shared Feature Representation
The study varies the number of layers shared by the domain and task classifiers to assess its effect on target-domain regression. Performance generally favors fewer shared layers for arousal, while optimal depth varies across corpora for valence and dominance.
- A. Number of Layers For the Shared Feature Representation: The experiment varies one to four shared layers between the domain and task classifiers and evaluates their effect on target-domain regression performance.The domain and task classifier layers each contain two layers, and the evaluation uses the target-domain validation set.
- A. Number of Layers For the Shared Feature Representation: Arousal performs consistently better with one shared layer, while adding more shared layers degrades performance.The improvement is reflected by lower RMSE and higher PR and CCC; CCC differences are statistically significant except for one specified MSP-IMPROV two-layer case.
- A. Number of Layers For the Shared Feature Representation: For valence and dominance, the best number of shared layers varies by corpus, with two or three layers performing best in most cases.
- A. Number of Layers For the Shared Feature Representation: The selected feature-representation depths are one shared layer for arousal, two for valence, and three for dominance.These settings are chosen from the target-domain validation results.
B. Regression Performance of the DANN Model
Across matched conditions, DANN improves cross-corpus emotion recognition over source-trained baselines and narrows the gap to within-corpus performance. Improvements are significant across emotional dimensions and source-domain settings, while depth helps mainly in within-corpus models.
- DANN improves average concordance correlation coefficient over source-trained systems across 900 matched conditions, with a statistically significant 6.6% gain.The conditions span three emotional dimensions, twenty trials, three sources, and five structures.
- Within-corpus performance is close to double cross-corpus performance, demonstrating the importance of reducing train-test mismatch.
- DANN reduces the gap between within-corpus and cross-corpus results while using unlabeled target-domain data.
- Emotional dimensions: DANN yields relative improvements of 22.8% for arousal, 33.4% for valence, and 15.5% for dominance over source models.The differences are statistically significant across all emotional dimensions; CCC values are lower for valence.
- Source domains: Relative improvements over source-domain training are 7.7% for USC-IEMOCAP, 36.4% for MSP-IMPROV, and 25% when all corpora are combined.Combining sources also improves average emotion-recognition performance by adding training variability.
- Network structure: Deep structures significantly outperform shallow structures for within-corpus models, but differences are not significant for source or DANN models.
C. Data Representation
The paper visualizes feature distributions across network layers to examine how DANN forms a shared representation. In the four-layer example, source and target representations become progressively less distinguishable until the final shared layer.
- The analysis uses t-SNE 2D projections to visualize source and target feature distributions at different network layers.
- At the first shared layer, source and target features remain sufficiently dissimilar for the domain classifier to distinguish them.
- By the second layer, the domain difference decreases, although some regions remain dominated by samples from one domain.
- At the third layer, source and target features become similar enough to confuse the domain classifier.
- At the fourth shared layer, target features become indistinguishable from source features, and this common representation feeds the emotion regressor.
VI. CONCLUSIONS
The study finds that domain-adversarial training uses unlabeled target data to improve emotion recognition while aligning source and target representations. Training remains sensitive to adversarial-network convergence and the handling of multiple source domains.
- Adversarial training improved emotion recognition over models trained exclusively on source-domain data.The evaluation also found that source and target domains became indistinguishable in the learned representation.
- The framework reduced the feature-space gap between source and target domains while retaining information for the primary regression task.
- Adversarial-network training can fail to converge for certain parameter settings, making parameter tuning important.The paper notes that this difficulty is common in minimax problems.
- The approach addresses train-test domain mismatch, a central challenge in speech emotion recognition.
- When multiple source domains are combined as one, the approach learns a representation common across those sources.The authors propose asymmetric source-specific representations as a possible alternative.