PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Speech Emotion Recognition with Ladder Networks

Srinivas Parthasarathy, Carlos Busso

arXiv:1905.02921v1eess.AS

TL;DR

Speech emotion recognition struggles to generalize across conditions, while collecting emotional labels is expensive. This paper uses ladder networks to jointly predict emotional attributes and reconstruct feature representations without labels for the auxiliary task. The approach outperforms STL and MTL baselines, with substantial gains in cross-corpus evaluation.

  • Problem

    SER models need to generalize across conditions, but emotional labels are expensive to collect and available labeled datasets are limited.

    Source: §I · p. 1

  • Method

    Ladder networks jointly predict emotional attributes and reconstruct input and intermediate feature representations through an unsupervised denoising-autoencoder task.

    Source: B. Ladder Network for Speech Emotion… · p. 4, §VI · p. 10

  • Results

    Cross-corpus ladder systems significantly improved over baselines; with unlabeled USC-IEMOCAP data, valence gains over STL reached 74.1%.

    Source: §I · p. 2, B. Cross Corpus Results · p. 8

  • Takeaways & Limitations

    The results indicate that ladder networks can use unlabeled target-domain data to improve SER generalization across corpora.

    Source: §I · p. 2, A. Motivation · p. 3

Abstract

from arXiv · show

Speech emotion recognition (SER) systems find applications in various fields such as healthcare, education, and security and defense. A major drawback of these systems is their lack of generalization across different conditions. This problem can be solved by training models on large amounts of labeled data from the target domain, which is expensive and time-consuming. Another approach is to increase the generalization of the models. An effective way to achieve this goal is by regularizing the models through multitask learning (MTL), where auxiliary tasks are learned along with the primary task. These methods often require the use of labeled data which is computationally expensive to collect for emotion recognition (gender, speaker identity, age or other emotional descriptors). This study proposes the use of ladder networks for emotion recognition, which utilizes an unsupervised auxiliary task. The primary task is a regression problem to predict emotional attributes. The auxiliary task is the reconstruction of intermediate feature representations using a denoising autoencoder. This auxiliary task does not require labels so it is possible to train the framework in a semi-supervised fashion with abundant unlabeled data from the target domain. This study shows that the proposed approach creates a powerful framework for SER, achieving superior performance than fully supervised single-task learning (STL) and MTL baselines. The approach is implemented with several acoustic features, showing that ladder networks generalize significantly better in cross-corpus settings. Compared to the STL baselines, the proposed approach achieves relative gains in concordance correlation coefficient (CCC) between 3.0% and 3.5% for within corpus evaluations, and between 16.1% and 74.1% for cross corpus evaluations, highlighting the power of the architecture.

I. INTRODUCTION

Speech emotion recognition needs to generalize across conditions, but emotional labels are costly to collect and benchmark datasets are limited. This study uses ladder networks to combine supervised emotional-attribute regression with unsupervised reconstruction, then evaluates the approach across corpora and acoustic features.

I. INTRODUCTION

  • Limited, costly emotional labels constrain SER datasets and make generalization across conditions a central challenge.Emotion annotations rely on cognitively intense perceptual evaluations, often from a limited number of evaluators.Source: §I · p. 1
  • Ladder networks pair emotional-attribute prediction with unsupervised reconstruction, enabling semi-supervised training on unlabeled target-domain recordings.The reconstruction task denoises intermediate feature representations and does not require emotional labels.Source: §I · pp. 1–2
  • The study evaluates arousal, valence, and dominance regression on MSP-Podcast, comparing STL and MTL baselines in supervised and semi-supervised settings.The semi-supervised setting uses unlabeled data around ten times the amount of labeled data in the corpus.Source: §I · p. 2
  • The proposed framework achieves state-of-the-art performance on MSP-Podcast and significant cross-corpus improvements, especially when adapting with unlabeled target-domain data.The study also tests frame-level LLD and MFB inputs, with significant improvements over baselines in most cases.Source: §I · p. 2

II. BACKGROUND

The background frames SER as a task with limited labeled data and reviews semi-supervised learning and auxiliary-task approaches. The paper distinguishes its joint unsupervised reconstruction and supervised regression approach from prior methods.

II. BACKGROUND

  • The study represents emotion through arousal, valence, and dominance, while noting that speech emotion databases often contain only a few hours of labeled data.Unlabeled data is presented as a way to improve generalization to a new domain.Source: §II · p. 2

A. Semi-Supervised Learning

  • Inductive semi-supervised learning iteratively adds confident classifier predictions to labeled training data, while co-training uses agreement across distinct feature views to augment it.Prior SER work has applied both strategies, including co-training combined with active learning.Source: A. Semi-Supervised Learning · p. 2
  • Prior deep-learning SER methods combined autoencoders or unsupervised feature learning with emotion recognition, whereas this study jointly trains reconstruction and regression.The authors describe joint training as helping the auxiliary reconstruction task support discriminative SER representations.Source: A. Semi-Supervised Learning · p. 2

B. Auxiliary Tasks and Multitask Learning

  • SER auxiliary-task studies have used emotional attributes, categories, gender, speaker identity, naturalness, age, and domain representations alongside emotion recognition.These examples include multitask learning and domain-adversarial approaches.Source: B. Auxiliary Tasks and Multitask… · p. 3

C. Ladder Networks

Ladder networks use lateral connections to support unsupervised reconstruction alongside supervised learning. This study motivates the architecture as a way to use unlabeled target-domain data and jointly optimize emotion regression with reconstruction.

C. Ladder Networks

  • Ladder networks extend denoising autoencoder reconstruction with supervised classification or regression; lateral connections and layerwise noise contributed to improved performance in prior work.The architecture originated with Valpola and was extended for supervised tasks by Rasmus and colleagues.Source: C. Ladder Networks · p. 3
  • Unlike a prior SER approach that learned features before a separate SVM, this formulation jointly optimizes reconstruction and regression to produce emotional-attribute predictions end to end.The paper states that its formulation requires no additional regressor.Source: C. Ladder Networks · p. 3

A. Motivation

  • Limited emotional labels and source–target distribution differences motivate regularization with unlabeled target-domain data to improve cross-corpus generalization.The proposed combination uses MTL to learn dependencies among emotional attributes and ladder networks to leverage unlabeled data.Source: A. Motivation · p. 3

B. Ladder Network for Speech Emotion Recognition

Ladder networks combine supervised emotional-attribute prediction with unsupervised reconstruction of input and intermediate representations. Noisy encoder representations are denoised through decoder connections, while the clean encoder representations are used at inference.

B. Ladder Network for Speech Emotion Recognition

  • Ladder networks jointly train the emotional-attribute regressor and an autoencoder that reconstructs input and intermediate feature representations.Unlike a separately trained autoencoder, this reconstruction is combined with the supervised task.Source: B. Ladder Network for Speech Emotion… · pp. 3–4
  • Noisy representations are created by adding Gaussian noise at each encoder layer; the final noisy representation trains the supervised regression task.The clean encoder path supplies reconstruction targets, and clean representations are used for inference.Source: B. Ladder Network for Speech Emotion… · p. 4
  • Batch normalization on intermediate layers prevents the trivial reconstruction solution in which representations collapse to a constant.The input layer is excluded from this normalization.Source: B. Ladder Network for Speech Emotion… · p. 4
  • Decoder denoising combines top-down decoder information with lateral connections from corresponding encoder layers, bypassing higher encoder layers.The lateral connections support reconstruction while allowing upper encoder layers to provide abstract discriminative information for regression.Source: B. Ladder Network for Speech Emotion… · p. 4
  • The total ladder loss adds layer-weighted reconstruction losses to the supervised emotional-attribute loss when labeled samples are available.Each reconstruction loss is weighted by its layer-specific hyperparameter.Source: B. Ladder Network for Speech Emotion… · p. 4

C. Multitask Learning for Emotion Recognition

Multitask learning jointly predicts arousal, valence, and dominance using shared and attribute-dependent layers. The ladder formulation can combine this supervised multitask loss with unsupervised reconstruction.

C. Multitask Learning for Emotion Recognition

  • The MTL architecture jointly predicts arousal, valence, and dominance through shared and attribute-dependent layers.Its overall loss combines the individual attribute losses using weights α and β, with the remaining weight assigned to dominance.Source: C. Multitask Learning for Emotion… · p. 4, Figure 2 · p. 5
  • α and β are constrained to [0, 1] with α + β ≤ 1; particular weight choices recover separate STL frameworks for each attribute.The arousal, valence, and dominance STL cases use (α = 1, β = 0), (α = 0, β = 1), and (α = 0, β = 0), respectively.Source: C. Multitask Learning for Emotion… · p. 5
  • Replacing the supervised loss in the ladder objective with the MTL loss combines supervised auxiliary prediction with unsupervised reconstruction.The study evaluates ladder implementations with both STL and MTL.Source: C. Multitask Learning for Emotion… · p. 5

IV. EXPERIMENTAL SETUP

Experiments use MSP-Podcast for within-corpus evaluation and USC-IEMOCAP and MSP-IMPROV for cross-corpus evaluation. Acoustic inputs are fixed-dimensional ComParE high-level descriptors computed from frame-level features.

A. Datasets

  • MSP-Podcast supplies 50 hours of labeled speech across 29,440 turns, with speaker-independent train, development, and test partitions.The respective labeled partitions contain 18,346, 3,753, and 7,341 turns; over 300 hours of unlabeled data are also used for semi-supervised training.Source: A. Datasets · p. 5
  • USC-IEMOCAP and MSP-IMPROV provide the two cross-corpus evaluation databases, with arousal, valence, and dominance annotations.USC-IEMOCAP contains 10,527 turns from 10 actors; MSP-IMPROV includes improvised scenarios and interactions during breaks.Source: A. Datasets · p. 5

B. Acoustic Features

  • ComParE features are extracted in two steps: frame-level LLDs are summarized into segment-level high-level descriptors (HLDs).The set includes 130 LLDs and 6,373 HLDs; HLDs provide fixed-dimensional inputs for variable-length speech segments.Source: B. Acoustic Features · pp. 5–6

C. System Description

The study compares STL and MTL baselines with supervised and semi-supervised ladder implementations. Systems are selected on development data and evaluated on test data using CCC.

C. System Description

  • Each condition’s best development-set system is evaluated on the test set, and performance is reported there.All regression systems are trained on the training set and optimized using the development set.Source: C. System Description · p. 6
  • The STL baseline trains separate attribute models, whereas the MTL baseline jointly predicts all three attributes using supervised auxiliary tasks.The MTL baseline follows Parthasarathy and Busso and omits the ladder network.Source: C. System Description · p. 6
  • Ladder variants use labeled-only (L) or labeled-plus-unlabeled (UL) training and either STL or MTL prediction.For UL training, unlabeled and labeled mini-batches are alternated.Source: C. System Description · p. 6

D. System Architecture

  • Models use two 256-node hidden layers with ReLU activations and 6,373-dimensional HLD inputs to predict an emotional attribute.The best-development-epoch model is evaluated on the test set.Source: D. System Architecture · p. 6
  • Ladder networks use input-to-first-layer dropout p = 0.1, Gaussian encoder noise with variance σ2 = 0.3, and reconstruction weight λl = 1.The reconstruction weight was not exhaustively optimized; the authors note that a broader search might yield better results.Source: D. System Architecture · p. 6
  • Models are trained and evaluated using CCC, and all neural networks use the NADAM optimizer.CCC maximizes Pearson correlation while minimizing the difference between true and predicted means.Source: D. System Architecture · p. 6

A. Within Corpus Results

Within-corpus MSP-Podcast evaluation found ladder networks improved CCC over STL for arousal and dominance, with the strongest test performance using unlabeled data. Valence favored the baselines in the test set.

A. Within Corpus Results

  • On the development set, the best ladder systems significantly outperformed STL for arousal and dominance, with MTL using labeled data alone achieving the best performance on those attributes.CCC was evaluated on the MSP-Podcast development and test sets.Source: A. Within Corpus Results · p. 7, Table I · p. 7
  • On the test set, ladder systems significantly beat both baselines for arousal, and ladder systems trained with labeled and unlabeled data did so for dominance; unlabeled-data systems performed best on both attributes.For valence, the baselines performed better in within-corpus evaluation.Source: A. Within Corpus Results · p. 7
  • 3.0% for arousal and 3.5% for dominance were the relative gains of the proposed architectures over STL across development and test evaluations.The reported averages visualize general trends across those evaluations.Source: A. Within Corpus Results · p. 7

B. Cross Corpus Results

Cross-corpus evaluation trained models on MSP-Podcast and tested them on USC-IEMOCAP or MSP-IMPROV, with target-corpus unlabeled data included for unsupervised reconstruction. Ladder networks achieved significant gains over baselines across emotional attributes, with reported relative gains varying by target corpus and attribute.

B. Cross Corpus Results

  • Models were trained on MSP-Podcast and tested on speaker-independent partitions of USC-IEMOCAP or MSP-IMPROV; target-corpus speech without emotional labels was used for unsupervised reconstruction in the unlabeled setting.The cross-corpus table reports average CCC across folds and standard deviations; emotional attribute scales were transformed to match the target corpora.Source: B. Cross Corpus Results · pp. 7–8, Table II · p. 8
  • On USC-IEMOCAP, fully labeled ladder systems significantly improved over STL for arousal and dominance, and over MTL for dominance; Ladder + L + STL significantly improved valence over STL.With target-corpus unlabeled data, significant gains over both baselines extended to all three attributes.Source: B. Cross Corpus Results · p. 8
  • 11.3% for arousal, 74.1% for valence, and 25.9% for dominance were the relative gains over STL on USC-IEMOCAP with target-corpus unlabeled data.Under this setting, the systems significantly outperformed the baselines for all three attributes.Source: Figure 4 · p. 8, B. Cross Corpus Results · p. 8
  • 16.1% for arousal, 40% for valence, and 5.5% for dominance were the relative gains over STL for Lad + UL + MTL on MSP-IMPROV.The systems significantly outperformed the baselines for all three attributes in this setting.Source: B. Cross Corpus Results · p. 8, Figure 5 · p. 9
  • On MSP-IMPROV, most ladder architectures improved CCC significantly over STL and MTL; valence gains held across proposed architectures, while arousal and dominance gains were significant with unlabeled data.With target-corpus unlabeled data, scores were closer to the within-corpus baseline.Source: B. Cross Corpus Results · p. 8, Figure 5 · p. 9

C. Results with Frame-Level Features

Frame-level CNN ladder networks improve selected within-corpus results, especially for valence, while feature choice affects performance across emotional attributes. The evaluation uses LLD and MFB inputs and constrains reconstruction to fully connected layers.

C. Results with Frame-Level Features

  • The CNN compares 65D ComParE LLDs and 40-band MFB sequences with HLD systems, using 10-second inputs that are padded or truncated.Frame-level inputs pass through four convolutional layers, each followed by max pooling, then two fully connected layers and a linear output layer.Source: C. Results with Frame-Level Features · p. 9, Figure 6 · p. 10
  • For LLD-based CNNs, ladder networks significantly beat STL on arousal and STL/MTL on valence, with valence gains up to 14.3% on test; dominance was statistically similar.Table III reports within-corpus MSP-Podcast CCC results for CNNs trained with LLDs or MFB features.Source: Table III · p. 10, C. Results with Frame-Level Features · p. 10
  • Because frame-level ladder training is computationally expensive, reconstruction costs cover only the two fully connected layers, with 29,440 unlabeled samples selected per epoch.The number of unlabeled samples per epoch matches the number of labeled samples; the analysis is within-corpus and CNN parameters are optimized on the MSP-Podcast development set.Source: C. Results with Frame-Level Features · pp. 9–10
  • The authors expect better results if reconstruction loss also includes the convolutional layers.The current reconstruction loss is limited to the fully connected layers after flattening.Source: C. Results with Frame-Level Features · p. 10
  • Arousal and dominance perform similarly with HLD and CNN-LLD features, but HLD performs better for valence; MFB results are consistently lower.MFB provides spectral information only, while LLDs and HLDs also provide prosodic and voice-quality cues important for SER.Source: C. Results with Frame-Level Features · p. 10

VI. CONCLUSIONS

The study concludes that ladder networks can improve speech emotion recognition, with especially strong generalization gains in cross-corpus evaluations. It also reports similar performance from a CNN implementation using frame-level features and identifies several directions for further work.

VI. CONCLUSIONS

  • Cross-corpus evaluations show significant gains for ladder networks, particularly when unlabeled data from the target domain is used.The models were trained on MSP-Podcast and evaluated on USC-IEMOCAP and MSP-IMPROV.Source: §VI · p. 10
  • The CNN implementation with frame-level features achieves performance similar to the sentence-level-feature implementation.The study evaluated different feature inputs and reports similar performance for the CNN-based implementation.Source: §VI · p. 10
  • Future work will explore learned representations as inputs, other modalities, improved valence prediction, and better frame-level performance.The authors identify acoustic valence prediction and frame-level valence performance as particular areas for improvement.Source: §VI · p. 11
Loading 1905.02921v1…