PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)

John Gideon, Melvin G McInnis, Emily Mower Provost

arXiv:1903.12094v2cs.LGcs.SDeess.ASstat.ML

TL;DR

Cross-corpus speech emotion recognition remains challenging because models often fail on unseen datasets, while adversarial methods can be difficult to converge. ADDoG and MADDoG move dataset representations toward one another; the paper reports consistent convergence and improved performance without target labels, including in-the-wild settings.

  • Problem

    Speech emotion models trained on one dataset often fail on unseen datasets, and existing adversarial approaches can have convergence difficulties or uncertain emotion preservation.

    Source: §1 · p. 2, §2.4 · p. 5

  • Method

    ADDoG iteratively moves representations from different datasets closer together, while MADDoG extends the approach to multiple datasets simultaneously.

    Source: §1 · p. 3, §7 · p. 13

  • Results

    ADDoG consistently converges and significantly improves performance without labelled target data across the experiments, including laboratory and in-the-wild settings.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    The meet-in-the-middle approach provides a more straightforward-to-train route to generalized speech emotion representations across datasets.

    Source: §7 · p. 13

    Prior discriminative methods may not preserve emotion or converge reliably, and earlier demonstrations largely used laboratory-recorded datasets.

    Source: §2.4 · p. 5

Abstract

from arXiv · show

Automatic speech emotion recognition provides computers with critical context to enable user understanding. While methods trained and tested within the same dataset have been shown successful, they often fail when applied to unseen datasets. To address this, recent work has focused on adversarial methods to find more generalized representations of emotional speech. However, many of these methods have issues converging, and only involve datasets collected in laboratory conditions. In this paper, we introduce Adversarial Discriminative Domain Generalization (ADDoG), which follows an easier to train "meet in the middle" approach. The model iteratively moves representations learned for each dataset closer to one another, improving cross-dataset generalization. We also introduce Multiclass ADDoG, or MADDoG, which is able to extend the proposed method to more than two datasets, simultaneously. Our results show consistent convergence for the introduced methods, with significantly improved results when not using labels from the target dataset. We also show how, in most cases, ADDoG and MADDoG can be used to improve upon baseline state-of-the-art methods when target dataset labels are added and in-the-wild data are considered. Even though our experiments focus on cross-corpus speech emotion, these methods could be used to remove unwanted factors of variation in other settings.

1 INTRODUCTION

Speech emotion models often fail across datasets, and prior adversarial approaches face convergence or emotion-preservation concerns. ADDoG and MADDoG move dataset representations toward a shared representation, with reported improvements across the experiments.

1 INTRODUCTION

  • Cross-dataset transfer is difficult because speech emotion datasets are small and vary in noise, language, recording quality, elicitation, and participant demographics.Prior discriminative methods also raise concerns about preserving emotion or converging reliably, [20], [21].Source: §1 · p. 2
  • ADDoG uses a critic network to iteratively move dataset representations closer together, a “meet in the middle” approach reported to converge consistently.MADDoG extends the approach to multiple datasets simultaneously.Source: §1 · p. 3
  • The experiments cover cross-corpus testing with unlabelled test features, added labelled target data, and both directions between laboratory and in-the-wild data.They also investigate training with three datasets simultaneously using MADDoG.Source: §1 · p. 3
  • ADDoG significantly improves over baseline systems without added labelled target data in all experiments, with lower repeated-run variance than CNN.The improvement margin decreases as more labelled target data are added.Source: §1 · p. 3
  • MADDoG can improve further on ADDoG when multiple source datasets are available by explicitly modelling their differences.Source: §1 · p. 3

2 RELATED WORKS

Prior cross-corpus speech emotion work spans feature and model adaptation, adversarial transfer, and domain-generalized representations. The review highlights unresolved concerns about preserving emotion, convergence, and evaluation beyond laboratory-recorded datasets.

2.1 Cross-Corpus Speech Emotion Recognition

  • Cross-corpus methods progressed from normalization, sample selection, and fusion to transfer and feature adaptation, auxiliary-task deep models, and transductive use of known test data.Dataset differences include recording conditions and emotion-label distributions.Source: §2.1 · pp. 3–4

2.2 Adversarial Methods

  • Adversarial approaches include generative augmentation and transfer, but transfer quality remains problematic; discriminative ADDA aligns source and target representations while risking classifier-relevant structure.WGANs were introduced to improve GAN convergence [15], while CycleGAN speech transfer showed a quality gap between real and transferred samples [18].Source: §2.2 · pp. 4–5

2.3 Domain Generalization

  • Domain generalization seeks a middle-ground representation across domains, using autoencoders or discriminative DANNs to learn representations intended to work across datasets.DANNs combine a feature extractor and label classifier with a domain classifier whose gradient is reversed [20].Source: §2.3 · p. 5

2.4 Open Challenges

  • Open challenges include whether compressed or aligned representations preserve emotion, DANN convergence difficulties, and limited prior demonstrations on laboratory-recorded corpora.Emotion content can be lost under compression,, and the review calls for more simultaneous datasets and in-the-wild evaluation.Source: §2.4 · p. 5

3 DATASETS

The section describes IEMOCAP, MSP-Improv, and PRIORI Emotion, then explains the shared valence-labeling and audio-preprocessing choices used across them.

3 DATASETS

  • IEMOCAP, MSP-Improv, and PRIORI Emotion are the three datasets used in the study.IEMOCAP was recorded across five actor sessions; MSP-Improv used target sentences to control lexical content while retaining more natural emotional expression; PRIORI Emotion is an affect-annotated subset of a larger bipolar mood dataset.Source: §3 · p. 5, §3.1 · p. 6, §3.2 · p. 6

3.1 IEMOCAP

  • IEMOCAP contains 10,039 speaker-turn utterances annotated for valence and activation on a 1–5 scale.The recordings came from ten actors performing scripted and improvised scenarios, with two to four annotations per utterance.Source: §3.1 · p. 6

3.2 MSP-Improv

  • MSP-Improv contains 8,438 utterances over 8.9 hours, with crowd-sourced valence and activation ratings on a 1–5 scale.Its target sentences control lexical content across emotional scenarios; the corpus also includes improvised turns, natural interactions, and read sentences.Source: §3.2 · p. 6

3.3 PRIORI Emotion Dataset

  • PRIORI Emotion annotations were manually reviewed and rated by 11 annotators on a 9-point scale, with 2–6 ratings per segment.The study selects segments from 12 consenting subjects; downsampling retains 11,402 utterances over 21.7 hours from 13,611 segments over 25.2 hours.Source: §3.3 · p. 6

3.4 Emotion Labeling

  • The study uses valence ratings rather than discrete classes and excludes utterances without a clear majority bin when forming three-bin soft labels.Valence is the focus because preliminary experiments found no benefit to domain generalization for activation; the middle bin corresponds to rating 3 for IEMOCAP and MSP-Improv and 5 for PRIORI Emotion.Source: §3.4 · p. 6

3.5 Preprocessing and Feature Extraction

  • Audio is downsampled to 16 kHz for IEMOCAP–MSP-Improv experiments or 8 kHz when PRIORI is included, then converted to 40-dimensional log Mel filter banks.The CNN encoder uses convolutions and global pooling to produce a 128-dimensional utterance representation, which the classifier maps to three-bin valence probabilities.Source: §3.5 · p. 6, Figure 2 · p. 7

4 CLASSIFICATION MODELS

The section presents CNN, ADDoG, and MADDoG for valence classification from speech features. ADDoG aligns source and target representations while preserving emotion information; MADDoG extends this alignment across multiple datasets.

4 CLASSIFICATION MODELS

  • All three models use MFB inputs to predict three-bin valence; ADDoG and MADDoG use unlabeled target data to generalize representations, unlike the CNN baseline.The target data include test examples available during training without labels; the CNN uses labeled data only.Source: §4 · p. 7

4.1 CNN

  • The CNN encodes utterances with convolutions and max pooling, then classifies valence with fully connected layers and softmax.Its training uses weighted cross-entropy on labeled source and target data, without unlabeled data.Source: §4.1 · p. 7

4.2 ADDoG

  • ADDoG iteratively trains an emotion classifier and a critic that estimates the Wasserstein distance between source and target representations, moving them closer while retaining emotion information.The critic is trained separately from the encoder and classifier; the encoder and classifier then train with emotion loss and a term that reduces the distance between dataset representations.Source: §4.2 · pp. 7–8, Figure 3 · p. 7

4.3 MADDoG

  • MADDoG extends ADDoG to multiple datasets by giving the critic an output for each dataset, aligning dataset representations individually rather than grouping them as source and target.Its feature-encoder and classifier loss uses a weighting parameter λ=0.1 for the critic contribution.Source: §4.3 · pp. 8–9

5 EXPERIMENTAL DESIGN

The four experiments test cross-dataset emotion classification as target labels become available and as training and test data span laboratory and in-the-wild settings. They compare different source-target arrangements and methods under a shared evaluation protocol.

5 EXPERIMENTAL DESIGN

  • Four experiment sets examine target-label availability and cross-dataset settings, with the final two incorporating laboratory and in-the-wild datasets.Experiments use UAR, consistent folds across methods, and repeated runs; the first experiment is repeated 50 times and the others 10 times.Source: §5 · p. 9

5.1 Experiment 1: Cross-Dataset

  • With no labelled target data, Experiment 1 trains on IEMOCAP or MSP-Improv and tests on the other, using unlabelled test features during ADDoG training.Both datasets were recorded in a laboratory; source data are split randomly 80:20 for training and validation, and each experiment is repeated 50 times.Source: §5.1 · p. 9

5.2 Experiment 2: Increasing Target Labels

  • Experiment 2 varies labelled target utterances from 0 to 3,200 while keeping source and test utterance counts constant.Target data are split in half for testing and label sampling; the folds are swapped to obtain predictions for both halves, while unselected target data are discarded.Source: §5.2 · pp. 9–10, Figure 4 · p. 10

5.3 Experiment 3: To In-the-Wild Data

  • Experiment 3 trains on IEMOCAP, MSP-Improv, or both and tests on in-the-wild PRIORI Emotion, using CNN, SP, and ADDoG, plus MADDoG for combined-source training.The settings combine differences in recording environment, recording quality, and elicitation strategy.Source: §5.3 · p. 10

5.4 Experiment 4: From In-the-Wild Data

  • Experiment 4 reverses that direction, training on in-the-wild PRIORI Emotion and testing on laboratory-recorded IEMOCAP or MSP-Improv with CNN, SP, and ADDoG.MADDoG is not used because this experiment has only one training dataset.Source: §5.4 · p. 10

6 RESULTS

ADDoG significantly outperforms CNN in cross-laboratory testing without labelled target data, and its results show lower variance across repeats. With target labels or in-the-wild data, the relative advantage depends on label availability and whether multiple source datasets are used.

6 RESULTS

  • Reported UAR errors are standard deviations of mean subject UAR across repeats, and significance is assessed by analysis of variance over subject UARs.Significant results are marked by dots on plots or bolded and starred table values.Source: §6 · p. 10

6.1 Experiment 1: Cross-Dataset

  • ADDoG significantly outperforms CNN in both IEMOCAP-to-MSP-Improv directions without labelled target data, and has lower repeat-to-repeat standard deviation.Figure 5 shows lower output variance across epochs; the authors note this stability may lessen the problem of using a mismatched source validation set.Source: §6.1 · p. 10, Figure 5 · p. 11

6.2 Experiment 2: Increasing Target Labels

  • With labelled target data, ADDoG significantly improves on baseline methods across Experiment 2, although its margin of improvement decreases as target labels increase.The authors suggest representation generalization may have diminishing returns once sufficient target-domain labels are available.Source: §6.2 · p. 11, Figure 6 · p. 11
  • In the MSP-Improv-to-IEMOCAP case with 200 labelled IEMOCAP utterances, ADDoG performs better without labels than with them, while still significantly beating CNN and SP at 200.The authors suggest the validation set of 40 labelled utterances may be too small to estimate test performance reliably.Source: §6.2 · p. 11

6.3 Experiment 3: To In-the-Wild Data

  • For laboratory-to-PRIORI testing, ADDoG significantly improves performance with no or at most 800 labelled target utterances, but loses its advantage at larger label counts and is significantly worse in one case.The authors report lower performance than in prior experiments, attributing this to differences in recording conditions and elicitation strategy.Source: §6.3 · p. 12, Figure 7 · p. 12
  • When both laboratory datasets train the model for PRIORI testing, MADDoG outperforms all methods using labelled data, while ADDoG remains significantly better with no labelled target data.CNN does not benefit from adding IEMOCAP to MSP-Improv, whereas ADDoG performs at least as well as its MSP-Improv-only counterpart.Source: §6.3 · p. 12, Figure 7 · p. 12

6.4 Experiment 4: From In-the-Wild Data

  • For PRIORI-to-laboratory testing, ADDoG significantly improves in most cases with no or small amounts of labelled target data, while SP matches or exceeds it when substantial target labels are available.CNN performs worst; the authors suggest a method balancing generalization and specialization may be needed when many target labels are available.Source: §6.4 · pp. 12–13, Figure 8 · p. 13

7 DISCUSSION AND CONCLUSION

ADDoG consistently improves cross-corpus performance without target labels, with lower variance across repeats; results with target labels also show when generalization may give way to specialization. The authors discuss extending the approach to other speech factors and in-the-wild mood estimation.

7 DISCUSSION AND CONCLUSION

  • Environmental noise, recording devices, elicitation strategies, and subject demographics can distract from emotion and reduce performance, particularly with small datasets.Source: §7 · p. 13
  • ADDoG and MADDoG iteratively move dataset representations closer; Experiment 3 also uses MADDoG to train with three datasets simultaneously.Experiments 1–2 use laboratory datasets, while Experiments 3–4 examine in-the-wild PRIORI Emotion data.Source: §7 · p. 13
  • ADDoG consistently converges and significantly improves performance without target labels across four experiments, while reducing repeat-to-repeat variance.This stability matters because cross-corpus validation and test sets are mismatched.Source: §7 · p. 13
  • Combining labeled data works reasonably when IEMOCAP and MSP-Improv recording conditions closely match, but fails with substantially different PRIORI data; SP can match or beat ADDoG and MADDoG with enough target labels.Source: §7 · p. 13
  • The authors propose extending ADDoG and MADDoG beyond dataset shifts to factors such as gender, phoneme, subject, and recording device.They also aim to use predicted emotion as a mid-level feature for in-the-wild mood estimation by bringing together emotion and mood datasets [41].Source: §7 · pp. 13–14
  • When to generalize versus specialize remains an open trade-off; the authors note specialization may be better with enough target labels and propose a fusion method to estimate which approach fits current domains.Source: §7 · pp. 13–14
Loading 1903.12094v2…