Source-linked AI summary
Self-supervised Learning for ECG-based Emotion Recognition
Pritam Sarkar, Ali Etemad
TL;DR
Deep supervised ECG emotion recognition often faces a shortage of labeled data. This paper uses self-supervised signal-transformation training and transfers the learned convolutional weights to an emotion classifier. The authors report state-of-the-art performance on SWELL and AMIGOS, with acceptable results using significantly less data.
Problem
Deep supervised models for ECG affective computing often lack sufficient labeled data.
Source: §1 · p. 1Method
The method trains a multi-task CNN to recognize six automatically labeled ECG transformations, then transfers its convolutional weights to an emotion-recognition network.
Source: §1 · p. 1Results
The authors report state-of-the-art performance on SWELL and AMIGOS and acceptable results using significantly less data.
Source: §5 · p. 4Takeaways & Limitations
The authors conclude that self-supervised ECG features can support emotion recognition with significantly less data while yielding acceptable results.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We present an electrocardiogram (ECG) -based emotion recognition system using self-supervised learning. Our proposed architecture consists of two main networks, a signal transformation recognition network and an emotion recognition network. First, unlabelled data are used to successfully train the former network to detect specific pre-determined signal transformations in the self-supervised learning step. Next, the weights of the convolutional layers of this network are transferred to the emotion recognition network, and two dense layers are trained in order to classify arousal and valence scores. We show that our self-supervised approach helps the model learn the ECG feature manifold required for emotion recognition, performing equal or better than the fully-supervised version of the model. Our proposed method outperforms the state-of-the-art in ECG-based emotion recognition with two publicly available datasets, SWELL and AMIGOS. Further analysis highlights the advantage of our self-supervised approach in requiring significantly less data to achieve acceptable results.
1. INTRODUCTION
ECG can support emotion recognition, but deep supervised models often lack enough labeled data. The paper proposes multi-task self-supervised learning for ECG emotion recognition and reports competitive or better performance than a fully supervised model, with state-of-the-art results on SWELL and AMIGOS.
1. INTRODUCTION
- Insufficient labeled data often limits training deep supervised models for ECG-based affective computing.The paper motivates self-supervised learning as a response to this constraint.Source: §1 · p. 1
- The method trains a CNN on six signal transformations with automatically generated labels, then transfers its convolutional weights for emotion classification on SWELL and AMIGOS.The authors report that the approach is competitive with or better than the same network trained fully supervised.Source: §1 · p. 1
- The authors report state-of-the-art arousal and valence detection on SWELL and AMIGOS.They also report acceptable results using significantly less data.Source: §1 · p. 1
2. RELATED WORK
Prior work applies self-supervised learning across vision, language, speech, and signal processing. The cited examples use automatically labeled transformations or other pretext tasks to learn representations for downstream tasks.
2. RELATED WORK
- Human activity recognition uses signal transformations as pretext tasks across six public datasets, with reported high-level feature learning in convolutional networks.Self-supervised methods are also described for 3D pose estimation [12] and video action recognition.Source: §2 · pp. 1–2
3. PROPOSED METHOD
The method learns ECG representations by predicting automatically generated signal-transformation labels, then uses the learned feature manifold for emotion classification. It applies six transformations and transfers shared convolutional layers into a downstream network whose dense layers are trained on emotion labels.
3.1. Self-supervised Learning
- The pretext network predicts transformation-task probabilities from ECG inputs, and its weighted task losses train a feature manifold for downstream classification.The downstream model uses that manifold to predict class probabilities by minimizing cross-entropy.Source: §3.1 · p. 2
3.2. Signal Transformation Tasks
- Six ECG transformations—noise addition, scaling, negation, horizontal flipping, permutation, and time-warping—supply automatically generated labels alongside the original signal.Transformation parameters are selected empirically to maximize final emotion-recognition performance.Source: §3.2 · p. 2
3.3. Network Architecture
- The transformation network shares three convolutional blocks across tasks and uses task-specific dense layers, while the emotion network transfers and freezes the convolutional weights.Only the emotion network's dense layers are trained with ECG signals and emotion labels.Source: §3.3 · pp. 2–3, Figure 1 · p. 2
4. EXPERIMENTS AND RESULTS
Experiments use SWELL and AMIGOS ECG data, with affect labels thresholded into binary classes and evaluated by 10-fold cross-validation. The self-supervised model performs competitively with or better than supervised alternatives, including when emotion training uses only 1% of labeled data.
4.1. Datasets
- SWELL included 25 participants and AMIGOS 40; both recorded self-assessed arousal and valence on a 1–9 scale.SWELL ECG was downsampled from 2048 Hz to 256 Hz to match AMIGOS, and both datasets received high-pass filtering.Source: §4.1 · p. 3
4.2. Model Training
- ECG was segmented into 10-second windows, and six transformed variations per segment were used to train the signal-transformation network.The networks used Adam with learning rate 0.001 and batch size 128; the transformation network trained for 30 epochs and emotion recognition for 100.Source: §4.2 · p. 3
4.3. Performance and Comparison
- The signal-transformation model reaches steady state at 13 epochs, with different steady-state losses indicating varied task difficulty.The reported loss curves average results and standard deviations across 10 training folds.Source: §4.3 · p. 3, Figure 2 · p. 4
- 96% arousal and 95.6% valence accuracy on SWELL; 85.1% arousal and 84% valence accuracy on AMIGOS, with self-supervised performance competitive with or better than fully supervised results.Tables 2 and 3 present the dataset results and comparisons with prior work and the emotion network without self-supervision.Source: §4.3 · p. 3, Table 2 · p. 4, Table 3 · p. 4
- On SWELL, the self-supervised model performs with higher accuracy than prior work, while arousal/valence regression results are not directly comparable to classification.Prior ECG work on SWELL mostly focused on stress detection; the authors also report stress-detection accuracies of 98.3% for self-supervised and 98.4% for fully supervised methods.Source: §4.3 · p. 4, Table 2 · p. 4
- The self-supervised model outperforms prior work on AMIGOS, although some comparisons use different validation schemes.Prior AMIGOS work reported F1 scores of 54.5% and 55.1% with Gaussian Naive Bayes, while a CNN reported arousal and valence accuracies of 81% and 71%.Source: §4.3 · p. 4, Table 3 · p. 4
- With only 1% of labeled data, the fully supervised network often performs considerably worse than the self-supervised model.The self-supervised signal-transformation network was trained on the entire unlabeled dataset before emotion training with 1% of labels.Source: §4.3 · p. 4, Figure 3 · p. 4
5. CONCLUSIONS AND FUTURE WORK
The conclusion reports state-of-the-art ECG emotion-recognition performance on SWELL and AMIGOS, with acceptable results using significantly less data. Future work will examine how individual transformation tasks and network layers contribute to downstream emotion classification.
5. CONCLUSIONS AND FUTURE WORK
- The approach achieved state-of-the-art performance on SWELL and AMIGOS and yielded acceptable results with significantly less data.The method transfers convolutional weights from a network trained to recognize signal transformations to an emotion-recognition network.Source: §5 · p. 4
- Future work will analyze each transformation-recognition task's contribution to emotion recognition and study individual network layers' downstream contributions..Source: §5 · pp. 4–5