Source-linked AI summary
GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition
Yang Li, Ji Chen, Fu Li, Boxun Fu, Hao Wu, Youshuo Ji, Yijin Zhou, Yi Niu, Guangming Shi, Wenming Zheng
TL;DR
EEG emotion recognition is challenged by weak generalization, limited use of EEG structure, and emotional noise labels. GMSS combines graph-based spatial and frequency jigsaw tasks with contrastive learning, and experiments show SOTA performance across three benchmark datasets.
Problem
EEG emotion recognition struggles to generalize across subjects and situations, capture discriminative EEG representations, and handle emotional noise labels.
Method
GMSS uses graph-based spatial and frequency jigsaw tasks plus contrastive learning to learn general EEG emotion representations.
Results
GMSS achieves SOTA performance on SEED, SEED-IV, and MPED under unsupervised and supervised comparisons.
Takeaways & Limitations
Combining multiple self-supervised tasks improves the discriminative and general features learned for EEG emotional signals.
Abstract
from arXiv · showhide
Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby decreasing the chance of overfitting on the original task, i.e., emotion recognition task. In particular, the spatial jigsaw puzzle task aims to capture the intrinsic spatial relationships of different brain regions. Considering the importance of frequency information in EEG emotional signals, the goal of the frequency jigsaw puzzle task is to explore the crucial frequency bands for EEG emotion recognition. To further regularize the learned features and encourage the network to learn inherent representations, contrastive learning task is adopted in this work by mapping the transformed data into a common feature space. The performance of the proposed GMSS is compared with several popular unsupervised and supervised methods. Experiments on SEED, SEED-IV, and MPED datasets show that the proposed model has remarkable advantages in learning more discriminative and general features for EEG emotional signals.
1 INTRODUCTION
EEG emotion recognition faces generalization, representation, and noisy-label challenges because EEG signals are heterogeneous and nonstationary. GMSS addresses these issues with graph-based multi-task self-supervised learning and reports SOTA performance on three benchmark datasets.
- Challenges: Traditional methods depend on handcrafted features and expert experience, motivating learned representations that better exploit EEG characteristics.
- Challenges: EEG emotion recognition must generalize across subjects and situations despite heterogeneous, nonstationary signals and distribution shifts.Existing domain-adaptation methods may use unlabeled testing data, limiting their suitability for some settings.
- GMSS: GMSS combines spatial and frequency graph-based jigsaw tasks with contrastive learning to learn general EEG emotion representations and reduce overfitting.The spatial task models relationships among brain regions, while the frequency task explores important frequency bands.
- GMSS: Multiple self-supervised pretext tasks are designed to address generalization, discriminative representation learning, and emotional noise labels.
- Contribution: The work presents multi-task learning as a first approach for improving generalization and avoiding overfitting in EEG emotion recognition.
- Results: GMSS achieves SOTA performance on three benchmark datasets under both unsupervised and supervised comparisons.The evaluated datasets are SEED, SEED-IV, and MPED.
2 RELATED WORKS
The related work reviews graph neural networks, multi-task learning, and self-supervised learning for EEG emotion recognition, highlighting limitations addressed by GMSS.
- Graph Neural Network: GNNs suit non-Euclidean EEG data, while CNNs are designed for Euclidean data.
- Graph Neural Network: Some EEG emotion-recognition GNNs restrict aggregation to second- or third-order neighbors, potentially losing information between distant nodes.
- Multi-Task Learning: Multi-task learning shares knowledge across related tasks and can encourage more general representations while reducing overfitting risk.
- Self-Supervised Learning: Self-supervised learning uses data attributes to generate pseudo-labels from unlabeled data rather than relying on human annotations.
- Self-Supervised Learning: The framework includes unsupervised upstream training, frozen-feature-extractor downstream training, and a supervised mode executing all parts simultaneously.
- Self-Supervised Learning: GMSS combines spatial and frequency jigsaw puzzles with contrastive learning in a multi-task framework focused on EEG emotion characteristics.
3 GRAPH-BASED MULTI-TASK SELF-SUPERVISED LEARNING FOR EEG EMOTION RECOGNITION
GMSS uses a shared graph-based feature extractor with spatial and frequency jigsaw tasks plus contrastive learning to learn general EEG emotion representations. It supports unsupervised transfer and supervised joint training while modeling electrode topology with graph convolutions.
- Model overview: GMSS combines spatial jigsaw, frequency jigsaw, and contrastive learning tasks that share a common feature extractor.The model uses spatial and frequency heads, a projection head, and an emotion-classification head.
- Spatial jigsaw puzzle: The spatial jigsaw partitions EEG electrodes into 10 brain-region blocks and learns to distinguish selected permutations using 128 pseudo-label classes.The operator selects 128 permutations with maximum Hamming distance from the original ordering, rather than using all 10! possible permutations.
- Frequency jigsaw puzzle: The frequency jigsaw transforms five emotion-related bands—δ, θ, α, β, and γ—and classifies the resulting data among 120 permutation classes.The task is designed to learn relationships among frequency bands and identify bands important for EEG emotion recognition.
- Contrastive learning: Contrastive learning maps spatially and frequency-transformed versions of the same EEG sample into a common feature space, bringing positive pairs closer and negative pairs farther apart.The method applies spatial and frequency transformations to generate multiple views before projection and similarity-based loss computation.
- Training and feature extraction: GMSS trains either without emotion labels for frozen-feature transfer or jointly with supervised emotion classification, using graph convolutions based on EEG topology.Chebyshev polynomials provide the graph convolution, with K = 2 selected to avoid over-smoothing.
4 EXPERIMENTS
Experiments evaluate GMSS on SEED, SEED-IV, and MPED EEG emotion datasets. The datasets use 62-channel EEG recordings collected while subjects watched emotional video clips.
- Datasets: GMSS is evaluated on the SEED, SEED-IV, and MPED EEG emotion datasets.All three datasets use EEG recordings collected with the ESI NeuroScan system and 62 electrodes positioned according to the 10-20 system.
- Data collection: The recordings were collected while subjects watched emotional video clips in quiet, comfortable, and non-interfering environments.
4.1 Experimental Dataset
The experiments use three benchmark datasets with different subject and session structures. SEED contains 15 subjects, three sessions per subject, and trials inducing happy, neutral, or sad emotions.
- SEED: SEED contains 15 subjects and three sessions per subject, with 15 film clips per session inducing happy, neutral, and sad emotions.Each emotion has five clips, yielding 15 trials per session and approximately 3400 samples per session.
- SEED: Each SEED trial contains approximately 185–238 samples.
4.2 Experimental Protocol
The evaluation includes subject-dependent and subject-independent protocols. Subject-dependent splits use different trials from the same subject, while subject-independent evaluation uses leave-one-subject-out testing, with results summarized across sessions or subjects.
- Experimental protocols: Subject-dependent experiments train and test on different EEG trials from the same subject.
- Subject-dependent protocol: For subject-dependent evaluation, SEED uses the first nine trials for training and the remaining six trials for testing in each session.
- Experimental protocols: Subject-independent experiments train on the remaining subjects and test on one held-out subject using leave-one-subject-out cross-validation.Every subject serves once as the test subject.
- Reported metrics: Tables 2 and 3 report mean/std classification accuracy for unsupervised and supervised modes across SEED, SEED-IV, and MPED.The reported averages are computed across sessions for subject-dependent experiments and across subjects for subject-independent experiments.
4.3 Experimental Details
GMSS uses dataset-specific EEG representations and a graph convolution that aggregates second-order neighbors. The model is trained with Adam using batches of 100 on an Nvidia 3080 GPU.
- SEED and SEED-IV use released differential entropy features, while MPED uses short-time Fourier transform inputs.
- The input X has size 62 × 5, each electrode output has dimension 32, and graph convolution uses K = 2.
- GMSS is implemented in PyTorch and trained on an Nvidia 3080 GPU with Adam and batch size 100.
4.4 Experimental Results
GMSS is evaluated in unsupervised and supervised settings across SEED, SEED-IV, and MPED, using self-supervised pretraining, linear evaluation, and comparisons with established methods. It achieves strong results while showing dataset- and emotion-specific recognition patterns.
- Unsupervised Mode: GMSS pretrains with two jigsaw tasks and contrastive learning, then evaluates a frozen feature extractor with a linear classifier.
- Unsupervised Mode: GMSS improves accuracy by 5.86%, 8.52%, 2.02%, 8.51%, 4.34%, and 2.66% over existing SOTA methods across subject-dependent and subject-independent tests on three datasets.
- Supervised Mode: In supervised subject-dependent experiments, GMSS attains the best performance across SEED, SEED-IV, and MPED compared with the listed methods.
- Supervised Mode: GMSS reaches 2.24% and 7% higher results than RGNN on SEED and SEED-IV, respectively, while achieving 40.16% versus BiHDM’s 40.34% on MPED.
- Supervised Mode: In supervised subject-dependent results, SEED emotion accuracies are approximately 90%, with happy above 95%, while all SEED-IV emotion accuracies exceed 80%.
- Supervised Mode: For subject-independent supervised recognition, funny, neutral, and fear achieve acceptable results on MPED, whereas joy, sad, disgust, and anger require attention.
4.5 Discussion
Discussion analyses learned representations, task ablations, and graph-filter order. The results support contributions from multiple pretext tasks and show that performance peaks at K = 2 before declining for larger values.
- Representation Visualization: GMSS learned representations form clearer same-emotion clusters than original EEG data in MPED t-SNE visualization.
- Ablation Study: Ablation experiments evaluate individual and combined spatial jigsaw, frequency jigsaw, and contrastive learning tasks in unsupervised and supervised modes.
- Ablation Study: The spatial jigsaw task achieves the best performance in four of six single-task results, while the frequency jigsaw task achieves the best performance in two.
- Ablation Study: Combining two tasks improves accuracy over single-task models, and GMSS using all pretext tasks achieves the best performance overall.
- Filter-Size Analysis: GMSS performs best at Chebyshev filter size K = 2, while performance shows a noticeable decline when K exceeds 2.
5 CONCLUSION
The paper proposes GMSS, a graph-based multi-task self-supervised model for EEG emotion recognition. It combines spatial and frequency jigsaw tasks with contrastive learning to improve representation learning and model generalization.
- GMSS combines spatial and frequency jigsaw tasks with contrastive learning to learn EEG emotion representations.
- The spatial jigsaw task studies EEG spatial patterns, while the frequency jigsaw task reveals intrinsic frequency bands for emotion recognition.
- The authors report that multiple self-supervised tasks improve model generalization, recognition ability, and resilience to emotion noise labels.