PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised ECG Representation Learning for Emotion Recognition

Pritam Sarkar, Ali Etemad

arXiv:2002.03898v2eess.SPcs.LGstat.ML

TL;DR

ECG emotion recognition is limited by the task specificity, computational cost, and annotation demands of fully supervised training. The paper learns ECG representations through multi-task signal-transformation recognition on unlabeled data, then trains emotion classifiers using frozen convolutional layers and labeled data. It reports improved performance over fully supervised training and new state-of-the-art results across four emotion-recognition tasks and four datasets.

  • Problem

    Fully supervised ECG emotion-recognition models require retraining for each task, often learn task-specific representations, and generally need large human-annotated datasets.

  • Method

    A two-stage multi-task self-supervised framework learns ECG representations from signal-transformation recognition and then trains dense emotion-classification layers with labeled ECG data.

  • Results

    The approach outperforms fully supervised training and sets new state-of-the-art results for arousal, valence, affective states, and stress across AMIGOS, DREAMER, WESAD, and SWELL.

  • Takeaways & Limitations

    Multi-task self-supervision can learn ECG representations that generalize across datasets and emotion categories, with transformation-task difficulty influencing representation quality.

  • Takeaways & Limitations

    The proposed solution performs poorly in subject-independent emotion recognition, with possible limitations in learning subject-invariant features from raw ECG using convolutions alone.

Abstract

from arXiv · show

We exploit a self-supervised deep multi-task learning framework for electrocardiogram (ECG) -based emotion recognition. The proposed solution consists of two stages of learning a) learning ECG representations and b) learning to classify emotions. ECG representations are learned by a signal transformation recognition network. The network learns high-level abstract representations from unlabeled ECG data. Six different signal transformations are applied to the ECG signals, and transformation recognition is performed as pretext tasks. Training the model on pretext tasks helps the network learn spatiotemporal representations that generalize well across different datasets and different emotion categories. We transfer the weights of the self-supervised network to an emotion recognition network, where the convolutional layers are kept frozen and the dense layers are trained with labelled ECG data. We show that the proposed solution considerably improves the performance compared to a network trained using fully-supervised learning. New state-of-the-art results are set in classification of arousal, valence, affective states, and stress for the four utilized datasets. Extensive experiments are performed, providing interesting insights into the impact of using a multi-task self-supervised structure instead of a single-task model, as well as the optimum level of difficulty required for the pretext self-supervised tasks.

1 INTRODUCTION

The paper addresses limitations of fully supervised ECG emotion recognition with a multi-task self-supervised representation-learning framework. It reports improved emotion-recognition performance and analyzes how transformation-task design affects learned representations.

  • Fully supervised ECG emotion-recognition models require task-specific retraining, substantial computation, and large human-annotated datasets.
  • The proposed framework learns ECG representations from automatically labeled signal-transformation tasks before supervised emotion classification.The framework uses self-supervised learning for representation extraction and human-annotated labels for the downstream task.
  • The self-supervised network outperforms the same network trained in a fully supervised fashion for emotion recognition.
  • Multi-task self-supervision produces better representations than single-task learning, with an intermediate transformation-task difficulty appearing optimal.The analysis reports that simpler tasks fail to properly learn representations, while extremely hard tasks prevent generalized representations.
  • New state-of-the-art results are reported for arousal, valence, affective states, and stress across AMIGOS, DREAMER, WESAD, and SWELL.The learned ECG representations are reported to generalize across all four datasets.
  • The paper extends prior work by adding datasets, transformation analyses, difficulty studies, architecture comparisons, parameter analyses, depth analysis, and multi-dataset training experiments.

2 BACKGROUND AND RELATED WORK

Prior ECG affective-computing studies use physiological features and classifiers across diverse datasets and emotion tasks. Self-supervised learning has shown generalized-representation benefits in other domains, motivating its application to ECG emotion recognition.

  • ECG contains cardiac waveforms whose information can support analysis of individuals’ cardiac states and affective signals.The described beat includes P, QRS, and T waves, with electrodes commonly configured as 12-, 5-, 3-lead, or single-channel sensors.
  • Prior work evaluated arousal, valence, stress, cognitive load, expertise, and affective states using datasets including AMIGOS, DREAMER, WESAD, and SWELL.
  • Earlier ECG emotion studies used HRV, time-domain, frequency-domain, wavelet, heart-rate, and inter-beat-interval features with conventional classifiers.
  • 2.3 Self-supervised Representation Learning: Self-supervised learning has produced generalized and robust representations in computer vision, speech, and natural language processing.These approaches use automatically generated labels rather than human-annotated labels.
  • 2.3 Self-supervised Representation Learning: The paper identifies ECG representation learning as an unaddressed application of self-supervised learning for emotion recognition.

3 SELF-SUPERVISED SOLUTION

The framework first learns ECG representations from unlabeled signals through multi-task recognition of six transformations, then transfers shared convolutional weights to a supervised emotion classifier. The transformation network uses shared convolutional layers with task-specific branches, while downstream dense layers are trained with emotion labels.

  • 3 SELF-SUPERVISED SOLUTION: The solution has two stages: learning ECG representations and using them to classify emotional classes.The first stage learns representations from signal transformations; the second uses original ECG data and human-annotated emotion labels.
  • 3.1 Self-supervised Learning of ECG Representations: A multi-task CNN learns ECG representations by recognizing six signal transformations with automatically generated labels.The transformed signals and labels are stacked into input-output matrices for transformation recognition.
  • 3.1 Self-supervised Learning of ECG Representations: The transformation network minimizes a weighted average of individual task losses to learn its trainable parameters.Each task has a predicted probability and loss coefficient contributing to the total loss.
  • 3.1 Self-supervised Learning of ECG Representations: The six transformations include noise addition, scaling, negation, temporal inversion, permutation, and time-warping.These transformations modify signal amplitude, temporal order, segment order, timing, or noise characteristics.
  • 3.1 Self-supervised Learning of ECG Representations: The pretext architecture contains shared convolutional blocks and task-specific dense branches, with feature embeddings extracted after the shared blocks.The architecture uses three shared convolutional blocks and seven task-specific branches; the extracted embeddings provide ECG representations.
  • 3.2 Emotion Recognition: Shared convolutional weights are transferred to the emotion network and frozen, while its fully connected layers are trained with labeled data.The downstream network outputs class probabilities and minimizes cross-entropy loss for emotion classification.

4 EXPERIMENTS

Experiments evaluate the framework across four public ECG datasets collected with different participants, scenarios, protocols, and hardware. Preprocessing reduces inter-dataset differences, and ten-fold evaluation examines transformation and emotion-network training behavior.

  • 4 EXPERIMENTS: Four publicly available datasets are used to evaluate the framework across varied subjects, circumstances, collection protocols, and hardware.The datasets are AMIGOS, DREAMER, WESAD, and SWELL.
  • 4.1.1 AMIGOS: AMIGOS measures self-assessed arousal and valence during multimedia viewing in individual and group contexts.Its ECG was recorded with Shimmer sensors at 256 Hz from 40 participants.
  • 4.1 Datasets: DREAMER elicits nine emotions from 23 participants using movie clips, while WESAD studies neutral, stressed, amused, and meditated states.DREAMER uses audio-video stimuli; WESAD records ECG from 17 participants at 700 Hz.
  • 4.1 Datasets: SWELL examines employee stress and emotional attributes in office scenarios involving normal work, time pressure, and interruptions.ECG was collected from 25 participants using a TMSI MOBI device at 2048 Hz.
  • 4.2 Preprocessing: Signals from SWELL and WESAD are downsampled to 256 Hz, and baseline wander is removed from all four datasets.These preprocessing steps address differences in sampling rate and other inter-dataset signal properties.
  • 4.3 Implementation and Training: Ten-fold cross-validation uses 90% of the shuffled preprocessed data for training and 10% for testing.Both networks use Adam with learning rate 0.001 and batch size 128.
  • 4.3 Implementation and Training: Transformation losses stabilize at different rates: temporal inversion, negation, permutation, and time-warping converge earlier than other transformations.WESAD and SWELL emotion losses stabilize before 100 epochs, whereas AMIGOS and DREAMER stabilize later.

5 RESULTS

The proposed self-supervised ECG network achieves strong transformation-recognition performance and state-of-the-art emotion-classification results across four datasets and multiple tasks.

  • Signal Transformation Recognition: 0.972 average F1 and 0.992 average accuracy are reported across all transformation-recognition tasks.The model also achieves low standard deviations across training folds.
  • Signal Transformation Recognition: 0.959–0.987 F1 scores and 0.989–0.996 accuracies are obtained across the four datasets for transformation recognition.The consistently high scores and low standard deviations indicate stable performance across datasets.
  • Emotion Recognition: The emotion-recognition evaluation covers arousal, valence, affective states, and stress using task-specific datasets and class counts.Arousal and valence use AMIGOS, DREAMER, and SWELL; affective states use WESAD; stress uses SWELL.
  • Comparison with Prior Work: 88.9% and 87.5% accuracies are achieved for arousal and valence classification, outperforming prior work and the fully-supervised baseline CNN.These results are reported for the specified classification setting and datasets.
  • Comparison with Prior Work: 96.9% accuracy is achieved on WESAD affective-state classification, while 93.3% accuracy is achieved on SWELL stress detection.The proposed model outperforms the cited prior methods and baseline comparisons in these evaluations.
  • Comparison with Prior Work: Prior-work comparisons exclude studies using different validation schemes, multiple modalities, or regression rather than classification.This filtering is described as a measure for fair comparison.

6 ANALYSIS AND DISCUSSION

The analysis examines how transformation difficulty affects downstream emotion recognition and identifies a suitable difficulty range for learning useful representations.

  • Transformation Difficulty: High-amplitude noise, many segments, extreme scaling, or large stretch factors reduce emotion-recognition accuracy when transformed signals are highly distorted.The reported examples include SNR < 10, more than 20 segments, scaling below 1.5 or above 0.6, and stretch factors above 1.4.
  • Transformation Difficulty: Transformation tasks that are too difficult produce poor transformation-recognition accuracy and poor emotion-classification performance.This pattern is reported for noise addition, scaling, and permutation.
  • Transformation Difficulty: A suitable range of transformation parameters places the pretext tasks at an effective difficulty level for optimum learning.The conclusion follows from the observed relationship between transformation difficulty and downstream emotion recognition.

D: SWELL

The discussion analyzes multi-task self-supervision, parameter choices, embedding depth, dataset aggregation, and limitations of the ECG representation framework.

  • Multi-task Self-supervision: (15, 0.9, 20, 9, 1.05) is identified as the optimal parameter vector for SNR, scaling, permutation segments, time-warping segments, and stretch factor.The analysis uses a six-dimensional representation of the jointly varied parameters.
  • Multi-task Self-supervision: Multi-task self-supervision consistently outperforms single-task setups, especially on AMIGOS and DREAMER.The authors associate different transformations with learning different aspects of ECG representations.
  • Multi-task Self-supervision: Scaling has greater impact on emotion recognition than noise addition and negation, while time-warping has greater impact than permutation and temporal inversion.These comparisons are reported among the six transformation-recognition tasks.
  • Multi-task Losses: The weighted average of individual losses accounts for different loss-convergence rates in the multi-task network.Temporal inversion and spatial negation losses saturate rapidly, reaching nearly perfect F1 scores after 5–7 epochs.
  • Network Embeddings: Conv-block 3 embeddings provide the best downstream emotion-recognition performance compared with embeddings from earlier blocks or their stacked combination.The authors interpret the last convolutional layer as more informative and generalizable for downstream tasks.
  • Multiple Datasets: Training the self-supervised network on all four datasets improves emotion classification compared with training on individual datasets, with significant gains for AMIGOS, DREAMER, and WESAD.Only marginal improvement is observed for SWELL.
  • Limitations and Future Work: The proposed solution performs poorly in subject-independent emotion recognition without user-specific calibration.The authors hypothesize that subject-invariant features may require handcrafted features or calibration unavailable for some datasets.
  • Limitations and Future Work: Future work includes multimodal emotion recognition, other ECG domains, and cross-subject and cross-corpus evaluation.EEG is identified as an additional modality available in some datasets.

7 CONCLUSION

The proposed self-supervised deep multi-task approach improves ECG emotion-recognition performance across four datasets and multiple emotion-related tasks. Experiments examine transformation-task difficulty, multi-task versus single-task learning, and multi-dataset training.

  • The approach significantly improves classification performance over a fully supervised solution across AMIGOS, DREAMER, WESAD, and SWELL.It reports new state-of-the-art results for arousal, valence, affective states, and stress classification.
  • The study evaluates self-supervised transformation-recognition tasks for learning ECG representations and examines their impact on emotion classification.
  • An optimum difficulty level in the pretext tasks produces better ECG representations for emotion classification.
  • A multi-task CNN outperforms a single-task network for learning ECG representations.
  • Training the self-supervised network with multiple datasets benefits the learned ECG representations.
Loading 2002.03898v2…