PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Neural Networks for Transfer Learning in Emotion Recognition

John Gideon, Soheil Khorram, Zakaria Aldeneh, Dimitrios Dimitriadis, Emily Mower Provost

arXiv:1706.03256v1cs.LG

TL;DR

Small emotion datasets make it difficult to generalize across recording conditions and demographics, motivating transfer of related paralinguistic knowledge and knowledge across emotion datasets. The paper compares progressive neural networks with pre-training and fine-tuning for these transfers. Its results show benefits for emotion recognition, most clearly for speaker-to-emotion transfer, with further promise for gender and cross-dataset transfer.

  • Problem

    Relatively small emotion datasets make it difficult to generalize beyond a dataset’s recording conditions and subject demographics, motivating transfer of related knowledge.

    Source: §1 · p. 1

  • Method

    The paper compares progressive neural networks, which preserve source-task representations, with pre-training and fine-tuning for transfers among paralinguistic tasks and emotion datasets.

    Source: §1 · p. 1

  • Results

    ProgNets significantly outperform standard DNN and PT/FT for speaker-to-emotion transfer, and show promise for gender-to-emotion and cross-dataset transfer.

    Source: §1 · p. 2, §4.2 · p. 3, §6 · p. 4

  • Takeaways & Limitations

    Progressive networks can retain source-task representations while transferring knowledge, and the experiments demonstrate their usefulness for emotion recognition, particularly in speaker-to-emotion transfer.

    Source: §6 · p. 4

    The study examines transfer across tasks and datasets separately; transferring across both simultaneously, including between datasets with different characteristics, remains future work.

    Source: §6 · p. 4

Abstract

from arXiv · show

Many paralinguistic tasks are closely related and thus representations learned in one domain can be leveraged for another. In this paper, we investigate how knowledge can be transferred between three paralinguistic tasks: speaker, emotion, and gender recognition. Further, we extend this problem to cross-dataset tasks, asking how knowledge captured in one emotion dataset can be transferred to another. We focus on progressive neural networks and compare these networks to the conventional deep learning method of pre-training and fine-tuning. Progressive neural networks provide a way to transfer knowledge and avoid the forgetting effect present when pre-training neural networks on different tasks. Our experiments demonstrate that: (1) emotion recognition can benefit from using representations originally learned for different paralinguistic tasks and (2) transfer learning can effectively leverage additional datasets to improve the performance of emotion recognition systems.

1. Introduction

Because emotion datasets are relatively small, models may struggle to generalize across recording conditions and demographics. The paper investigates transfer between speaker, gender, and emotion recognition and between emotion datasets, using progressive networks alongside pre-training and fine-tuning.

1. Introduction

  • Small emotion datasets make it difficult to generalize beyond the recording conditions and subject demographics of a particular dataset.The paper proposes leveraging related paralinguistic knowledge and models trained on other emotion datasets as ways to augment emotion recognition.Source: §1 · p. 1
  • Prior work used joint gender-emotion modeling and speaker identity features, and showed that features such as pitch can support several paralinguistic predictions.Other studies applied multi-task learning across speech, song, and gender or treated dimensional and categorical labels as separate tasks.Source: §1 · p. 1
  • Pre-training and fine-tuning transfers a model from a source domain to a related target domain, and prior work applied it to speech recognition and emotion or facial-expression recognition.The paper uses PT/FT as its transfer-learning baseline.Source: §1 · p. 1
  • PT/FT can make initialization across a sequence of related tasks unclear, cause forgetting of the source task, and be difficult when network architectures differ [11].Progressive networks freeze earlier tasks and feed their intermediate representations into a new network, avoiding forgetting at the expense of added parameters [11] [13].Source: §1 · p. 1
  • The study examines transfer among emotion, speaker, and gender recognition, with emotion as the target, and transfer between IEMOCAP and MSP-IMPROV.It compares a DNN, DNN with PT/FT, and progressive networks, including when target-task data are limited.Source: §1 · pp. 1–2
  • Progressive networks significantly outperform standard DNN and PT/FT for transferring speaker-recognition knowledge to emotion recognition on IEMOCAP and MSP-IMPROV.The authors also report promise for gender-to-emotion and cross-dataset transfer, with results significantly better than a DNN without transfer learning.Source: §1 · p. 2

2. Datasets and Features

The study uses IEMOCAP and MSP-IMPROV speech utterances with majority-agreement labels for four emotion classes. It represents each utterance with summarized eGeMAPS features.

2.1. Datasets

  • The study uses utterances labeled happy, sad, angry, or neutral when there is majority agreement on the ground-truth label.Both datasets were collected to simulate natural dyadic interactions and use similar labeling schemes.Source: §2.1 · p. 2
  • IEMOCAP contains 5,531 utterances from ten speakers in five sessions, while MSP-IMPROV contains 7,798 utterances from 12 speakers in six sessions [14, 15].Each session includes one male and one female speaker; IEMOCAP combines excitement and happiness into the happy category.Source: §2.1 · p. 2

2.2. Features

  • Each utterance is represented by summarized eGeMAPS features, a standardized affective-computing set containing 88 frequency, energy, spectral, cepstral, and dynamic features.The summaries include distribution statistics and signal-slope measures, followed by dataset-specific global z-normalization.Source: §2.2 · p. 2
  • Figure 2 depicts the progressive-network architecture used for the experiments, with arrows indicating dense layer connections and black arrows indicating frozen transferred weights.The number of output units N varies by experiment.Source: Figure 2 · p. 2

3. Methods

The study compares target-only DNNs, pre-training and fine-tuning, and progressive networks for transfer learning. It evaluates performance using unweighted average recall and repeated speaker-stratified cross-validation.

3. Methods

  • The target-only DNN is the no-transfer baseline, while PT/FT pre-trains on a source task before fine-tuning on the target task.PT/FT assumes the target model can use knowledge from the source task [19].Source: §3 · p. 2
  • ProgNets freeze the source model, add a randomly initialized target model, connect the models, and train the new model with backpropagation [11].This preserves learned source-task information and avoids the forgetting effect associated with PT/FT [11].Source: §3 · p. 2
  • Because training data are limited, the ProgNets omit adaptation layers and pass representations between neighboring layers in a one-to-one fashion.The columns use equal numbers of layers, with layer k of the frozen column feeding layer k + 1 of the new column.Source: §3 · p. 2
  • Performance is compared using unweighted average recall (UAR), an unweighted accuracy measure that gives equal weight to each class and accounts for unbalanced datasets.Evaluation uses repeated ten-fold cross-validation stratified by speaker ID; each split reserves one fold for testing and one for early stopping.Source: §3 · p. 2
  • Figure 3 presents learning curves for speaker-to-emotion transfer, with surrounding regions showing standard deviations of UAR averaged across folds at each iteration.PT/FT reaches its best solution faster, but its final performance is lower than ProgNet’s; the figure compares these learning dynamics with the standard DNN.Source: Figure 3 · p. 3
  • The repeated ten-fold evaluation is run ten times, and the reported mean and standard deviation summarize UAR across folds and iterations.Significance is assessed with a repeated-cross-validation paired t-test, with p < 0.05 treated as significant.Source: §3 · p. 3

4. Paralinguistic Experiments

The experiments compare transfer from speaker and gender recognition to emotion recognition across IEMOCAP and MSP-IMPROV. ProgNets outperform competing systems for speaker transfer, while gender-transfer performance varies by corpus.

4.1. Experimental Setup

  • The experiments transfer speaker or gender knowledge to emotion recognition on IEMOCAP and MSP-IMPROV, comparing three systems and examining UAR and learning curves.The section reports system UAR on both datasets and analyzes convergence behavior.Source: §4.1 · p. 3, §4.2 · p. 3

4.2. Results

  • For speaker-to-emotion transfer, ProgNets significantly beat DNN and PT/FT on IEMOCAP and MSP-IMPROV.The reported p-values are 2.6E-3 and 2.0E-2 for IEMOCAP, and 8.8E-4 and 1.2E-2 for MSP-IMPROV, respectively.Source: Table 2 · p. 3, §4.2 · p. 3
  • Gender transfer was corpus-dependent: ProgNets beat DNN on MSP-IMPROV (p = 1.2E-2), not IEMOCAP, while PT/FT performed no better than DNN on either dataset.The authors hypothesize that the difference relates to gender-recognition UAR: 98.1% on MSP-IMPROV and 93.1% on IEMOCAP.Source: §4.2 · p. 3
  • PT/FT reached its best solution faster than ProgNets but had lower final performance; its learning-curve slope vanished after about 150 epochs, versus about 400 for ProgNets.Both transfer systems began with steeper slopes than the standard DNN.Source: §4.2 · p. 3

5. Cross-Dataset Experiments

Cross-dataset experiments transfer emotion models between IEMOCAP and MSP-IMPROV while varying the amount of target training data. ProgNets significantly outperform DNN in several conditions, with their relative benefit depending on target-set size.

5.1. Experimental Setup

  • The source model uses all eight folds, while transfer is tested in both dataset directions with target training sets of 8, 4, 2, or 1 folds.The standard DNN is trained only on the target dataset; PT/FT and ProgNet adapt a model trained on the source dataset.Source: §5.1 · p. 4

5.2. Results

  • ProgNets significantly outperformed DNN from MSP-IMPROV to IEMOCAP with 1, 2, or 4 training folds, and in the reverse direction with 1, 2, or 8 folds; PT/FT improved significantly only in the first direction at 1 or 2 folds.The comparisons use different target fold counts from the eight available training folds.Source: §5.2 · p. 4
  • ProgNets’ larger number of transferred weights may make them more beneficial with larger target datasets; the authors note PT/FT may be preferable in some small-data cases.They hypothesize this may explain PT/FT’s better performance at 1 or 2 folds on the smaller IEMOCAP target.Source: §5.2 · p. 4

6. Conclusion

The conclusion presents ProgNets as useful for transferring knowledge between paralinguistic tasks and emotion datasets. Combining task and dataset transfer across datasets with differing characteristics remains future work.

6. Conclusion

  • ProgNets significantly outperformed DNN and PT/FT for speaker-to-emotion transfer, and improved on systems without source information for gender-to-emotion and dataset transfer.The paper describes ProgNets as retaining source-task representations while avoiding the forgetting effect associated with pre-training and fine-tuning.Source: §6 · p. 4
  • The study transfers across tasks and datasets separately; transferring across both simultaneously, including differing recording conditions, is left for future work.The cited examples include telephone versus high-fidelity speech.Source: §6 · p. 4
Loading 1706.03256v1…