PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

Xinke Shen, Xianggen Liu, Xin Hu, Dan Zhang, Sen Song

arXiv:2109.09559v2cs.HCcs.LGeess.SPq-bio.NC

TL;DR

Cross-subject emotion recognition is limited by substantial individual differences in EEG signals and often requires subject-specific data. CLISA uses contrastive learning to align EEG representations across subjects receiving the same stimuli, achieving comparable or better performance on two datasets and generalizing to unseen stimuli. The learned representations also offer potential insight into emotion-related neural substrates.

  • Problem

    Substantial inter-subject EEG variability makes cross-subject emotion recognition difficult, while practical systems often require extensive data from new subjects.

  • Method

    CLISA uses contrastive learning with spatial-temporal EEG encoding to align representations from different subjects responding to the same or different emotional stimuli.

  • Results

    CLISA achieved comparable or better performance than state-of-the-art methods on THU-EP and SEED, including 71.9±8.8% binary accuracy on THU-EP and 86.4±6.4% three-class accuracy on SEED.

  • Takeaways & Limitations

    The learned representations generalized to unseen emotional stimuli and could provide insights into the neural substrates of emotion.

  • Takeaways & Limitations

    Validation used young adults, so further studies should include broader age ranges for a more generalized model.

Abstract

from arXiv · show

EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing.

1 INTRODUCTION

EEG offers a portable, objective measure for emotion recognition, but strong inter-subject variability makes cross-subject generalization difficult. CLISA addresses this challenge by learning subject-invariant EEG representations through contrastive inter-subject alignment.

  • Motivation: EEG provides a relatively direct, objective, portable, and cost-effective measure of emotional responses.Unlike behavioral measures, EEG responses are difficult to disguise or consciously restrain.
  • Existing EEG representations: Researchers commonly use differential entropy features and models of inter-electrode relationships to represent emotion-related EEG activity.Approaches include GNNs, LSTMs, network-based features, CNNs, and attention mechanisms.
  • Challenge: Subject-dependent methods often require roughly half an hour to more than one hour of data from each subject, limiting practical deployment.This motivates methods that generalize to new users with less subject-specific data.
  • Challenge: Cross-subject recognition remains difficult because emotion-related EEG activities vary substantially across individuals.On SEED, accuracy could be 58% cross-subject versus 96% intra-subject with the same classifier.
  • Proposed approach: CLISA learns a subject-invariant EEG space by aligning representations from subjects receiving the same emotional stimuli.The approach is inspired by inter-subject correlation findings that shared stimulus-specific responses can encode emotional states.
  • Proposed approach: CLISA is designed to generalize to new subjects without extensive target data and to different emotional stimuli used during testing.The authors describe the resulting representation as general for emotion processing rather than specific to training stimuli.

2 RELATED WORK

Prior work addresses cross-subject EEG recognition through domain adaptation, domain generalization, and contrastive learning. CLISA builds on inter-subject correlation and synchronized emotion-EEG datasets to align representations across subjects and stimuli.

  • Domain adaptation: Classical domain adaptation methods align source- and target-subject distributions, but require target-domain data during training.Examples include TCA, KPCA, TPT, and adaptive subspace feature matching.
  • Domain generalization: Domain generalization learns representations across multiple training subjects so models can be applied to new subjects without their data.Domain residual networks combine domain-shared and domain-specific weights with domain-adversarial training.
  • Contrastive learning: Contrastive learning distinguishes similar from dissimilar data pairs and can improve representations for downstream tasks.It includes context-instance and instance-instance contrastive paradigms.
  • Contrastive learning: Prior EEG contrastive-learning studies typically learn from augmented views of the same original sample before applying representations to downstream tasks.One example uses temporal masking and linear scaling for augmentation.
  • Inter-subject correlation: Inter-subject correlation studies show that consistent neural responses to shared stimuli can contain information about mental states.However, ISC analysis generally requires subjects to share a sensory environment and engage in similar tasks.
  • Dataset basis: Emotion EEG datasets commonly expose groups of subjects to the same tasks and stimuli, enabling contrasts between same- and different-stimulus responses.Such contrasts are intended to support representations that generalize to unseen subjects and emotional stimuli.
  • Dataset basis: Increasing the number of subjects can increase the number of contrasts quadratically, motivating larger multi-subject emotion EEG datasets.The paper notes that public datasets commonly contain 10 to 40 subjects and introduces THU-EP alongside SEED.

3 THE CONTRASTIVE LEARNING METHOD FOR INTER-SUBJECT ALIGNMENT (CLISA)

CLISA aligns EEG representations across subjects by contrasting signals from the same emotional-stimulus segment against signals from different segments. Its convolutional encoder and projector support contrastive training, after which aligned representations yield differential-entropy features for emotion recognition.

  • 3.1 The Contrastive Learning Procedure: CLISA uses a data sampler, base encoder, projector, and contrastive loss to learn inter-subject-aligned EEG representations.The base encoder applies spatial and temporal convolutions, while the projector maps representations into a latent space for similarity computation.
  • 3.1 The Contrastive Learning Procedure: The sampler pairs EEG segments from two subjects when they correspond to the same video time segment, treating other minibatch samples as negatives.Training enumerates subject pairs and constructs minibatches from corresponding trials and time segments.
  • 3.1 The Contrastive Learning Procedure: The base encoder first transforms EEG channels into latent signals with spatial convolution, then learns temporal patterns with temporal convolution.Spatial convolution models relationships among EEG channels, while temporal filters can extract representations in specific frequency bands.
  • 3.1 The Contrastive Learning Procedure: The projector uses nonlinear and depthwise spatiotemporal convolutions to produce vectors for similarity calculation while reducing parameter size.Depthwise convolutions support pattern extraction for each frequency band.
  • 3.1 The Contrastive Learning Procedure: The contrastive loss increases similarity for positive EEG pairs relative to negative pairs by minimizing normalized temperature-scaled cross-entropy loss.The framework optimizes the base encoder and projector jointly during contrastive learning.
  • 3.2 The Prediction Procedure: During prediction, the trained base encoder aligns representations, and trained differential-entropy features are extracted, concatenated across time, smoothed, and used for emotion recognition.Differential entropy measures time-series complexity and is equivalent to logarithmic energy spectrum in a specific frequency band.

4 EXPERIMENTS

Experiments evaluated CLISA on THU-EP and SEED, using dataset-specific preprocessing, model settings, and cross-subject evaluation protocols. A second generalizability test assessed transfer to both new subjects and unseen stimuli.

  • Datasets: CLISA was evaluated on THU-EP with 80 subjects and SEED, a widely used benchmark with 15 subjects.THU-EP was selected as a relatively large benchmark for cross-subject emotion recognition.
  • Datasets: THU-EP included 80 college students and 28 emotional video clips spanning negative, positive, and other emotional categories.The supplied passage identifies the participant count and the use of emotional video stimuli.
  • Preprocessing: EEG preprocessing included bandpass filtering, artifact removal or denoising, and common-average rereferencing for the two datasets.THU-EP additionally used ICA-based artifact removal, while SEED data had been downsampled and prefiltered by its provider.
  • Implementation Details: The model used dataset-specific convolutional settings, including spatial filters matched to EEG channel dimensions for THU-EP and SEED.Temporal filter and pooling lengths for SEED were scaled according to its sampling frequency.
  • Implementation Details: Contrastive-learning samples were 5 seconds long with a 2-second step, whereas prediction samples were 1 second long.The contrastive-learning duration balanced emotional information against the number of available training samples.
  • Performance Evaluation: Evaluation used 10-fold cross-validation for THU-EP and leave-one-subject-out cross-validation for SEED, plus a test on new subjects and unseen stimuli.The generalizability test trained on two-thirds of trials from training subjects and tested on the remaining trials from testing subjects.

5 RESULTS

CLISA improved cross-subject emotion recognition across THU-EP and SEED, generalized to unseen stimuli, and produced subject-aligned representations with separable emotions. Performance also increased as more subjects contributed to contrastive learning.

  • Emotion recognition performance: 71.9±8.8% binary accuracy on THU-EP, improving 6.7% over DE+MLP and outperforming SA, CorrCA, and SeqCLR.The improvement over DE+MLP was significant (corrected p<0.001).
  • Representation alignment: CLISA’s learned DE features merged across subjects while preserving separation between positive and negative emotions in t-SNE space.Original DE features remained scattered by subject, whereas trained features reduced subject discrepancy without loss of emotional separability.
  • Effect of training-subject number: Validation accuracy rose considerably as the number of training subjects used for contrastive learning increased from 8 to 72.The experiment varied contrastive-learning subjects while keeping all training subjects in the prediction procedure.
  • Generalizability: 63.4±17.1% accuracy on unseen THU-EP stimuli exceeded the highest baseline of 60.7±19.7%, but the improvement was not significant.The authors interpreted this as evidence that the model did not simply memorize stimuli seen during contrastive learning.
  • Nine-class classification: CLISA exceeded competing methods by 10.2% on nine-class THU-EP emotion classification, with corrected ps<0.001 for all comparisons.The consistent gains covered SA, CorrCA, and SeqCLR.
  • SEED and spatiotemporal analyses: 86.4±6.4% accuracy on SEED surpassed DE+MLP, SA, CorrCA, and SeqCLR, while 77.4±13.4% on unseen stimuli was highest but not significant.The learned representations also showed emotion-specific spatial and frequency patterns across SEED and THU-EP analyses.

6 DISCUSSION

CLISA aligns EEG representations across subjects and supports both practical cross-subject recognition and analysis of emotion-related neural patterns. Its learned patterns reveal shared and distinct spatial-temporal characteristics across emotion dimensions and categories, while several scope and interpretability limitations remain.

  • Methodological contributions: CLISA uses fine-grained temporal alignment, EEG time-series inputs, and CNN-based processing to learn common representations across subjects.Unlike methods based mainly on domain classifiers or extracted features, CLISA matches data corresponding to the same video segment and combines spatial dimensions from EEG time series.
  • Methodological contributions: Adaptive feature normalization reduces the need for extensive testing-subject data during adaptation, enhancing practical applicability.Most domain adaptation methods require substantial testing-subject data for adaptation, whereas CLISA performs subject alignment without that requirement.
  • Neural-pattern insights: Spatial activations in temporal regions with frequencies above 20 Hz were important for positive emotion on both THU-EP and SEED.Negative-emotion activations were more anterior on SEED but more posterior on THU-EP, potentially reflecting context-specific responses requiring further investigation.
  • Neural-pattern insights: Positive and negative emotions generally shared spatial activations, but their frequency responses differed across both datasets.The spatial overlap supports valence-general brain regions, while distinct frequency responses indicate differing dynamic processing mechanisms.
  • Neural-pattern insights: Nine-class emotion classification showed emotion-specific spatial and temporal patterns, with more distinct spatial representations than the valence dimension.Temporal patterns were distinguishable for both discrete emotion categories and valence.
  • Limitations: Validation used young adults, the projector had limited neurophysiological interpretability, and shared patterns across emotion categories may be insufficiently revealed.The datasets had mean ages of 20.16 and 23.27 years, motivating broader age coverage and further work on interpretable projectors and shared patterns.

7 CONCLUSIONS

The study proposed CLISA, a contrastive-learning method for inter-subject alignment, and evaluated it for cross-subject emotion recognition. It achieved comparable or better performance than state-of-the-art methods on two datasets, generalized to unseen stimuli, and offered interpretable neural-pattern analyses.

  • 7 CONCLUSIONS: CLISA achieved comparable or better performance than state-of-the-art methods on the THU-EP and SEED datasets.Reported accuracies were 71.9±8.8% for binary and 45.7±11.8% for nine-class THU-EP classification, and 86.4±6.4% for three-class SEED classification.
  • 7 CONCLUSIONS: CLISA generalized to unseen emotional stimuli better than other comparison methods.The conclusion specifically reports improved generalization to emotional stimuli not seen during testing-related evaluation.
  • 7 CONCLUSIONS: Visualized spatial and temporal filters demonstrated CLISA’s potential to provide insights into the neural substrates of emotion.The conclusion links the model’s important filters to analysis of emotion-related neural patterns.

SUPPLEMENTARY MATERIALS

The supplementary materials provide source-localization figures, spatiotemporal feature visualizations, subject-level accuracy tables, temporal-filter analyses, and statistical-testing notes for the reported experiments.

  • Figures: Figure S1 provides source localization for important-feature spatial activation patterns on the SEED dataset.Frequencies shown below each emotion category indicate the maximal frequency response of the corresponding temporal filter.
  • Figures: Figures S2–S5 show spatiotemporal characteristics and source localization for important features in THU-EP binary and nine-class classification.The figures cover two important features for each classification setting and identify maximal temporal-filter frequencies beneath emotion categories.
  • Ablation and implementation: Tables S1 and S2 report performance under different temporal convolution filter lengths in the THU-EP binary-classification setting.The tables separately vary filter lengths in the base encoder and projector.
  • Subject-level results: Tables S3–S5 report subject-level classification accuracies for THU-EP binary, THU-EP nine-class, and SEED classification.These tables organize performance at the individual-subject level for the corresponding datasets and tasks.
  • Statistical analysis: The supplementary statistical notes specify paired t-tests comparing CLISA with other models using t-values and Bonferroni-corrected p-values.Parenthesized p-values are identified as uncorrected in the accompanying note.
Loading 2109.09559v2…