PaperlayerView exact revision on arXiv

Source-linked AI summary

A Novel Bi-hemispheric Discrepancy Model for EEG Emotion Recognition

Yang Li, Wenming Zheng, Lei Wang, Yuan Zong, Lei Qi, Zhen Cui, Tong Zhang, Tengfei Song

arXiv:1906.01704v1q-bio.NCeess.SP

TL;DR

EEG emotion recognition can exploit the different emotional responses of the brain’s two hemispheres, but requires features that preserve electrode structure and capture their discrepancy. BiHDM uses spatial RNNs, pairwise hemispheric operations, and domain-adversarial learning to construct such representations. It achieves state-of-the-art performance across three public EEG emotional datasets, including reported accuracy gains in subject-independent experiments.

  • Problem

    Existing EEG emotion recognition research must extract discriminative emotion features and classify them, while hemispheric emotional asymmetry remains a promising property to utilize.

  • Method

    BiHDM uses directional hemispheric RNNs, pairwise operations on symmetric electrodes, higher-level feature processing, and a domain discriminator for domain-invariant representations.

  • Results

    BiHDM outperforms compared methods on three public EEG emotional datasets; subject-independent accuracy improvements are 2.2%, 3.5% and 2.4%, respectively.

  • Takeaways & Limitations

    The experiments support incorporating asymmetric differential information between the left and right hemispheres into EEG emotion recognition.

Abstract

from arXiv · show

The neuroscience study has revealed the discrepancy of emotion expression between left and right hemispheres of human brain. Inspired by this study, in this paper, we propose a novel bi-hemispheric discrepancy model (BiHDM) to learn the asymmetric differences between two hemispheres for electroencephalograph (EEG) emotion recognition. Concretely, we first employ four directed recurrent neural networks (RNNs) based on two spatial orientations to traverse electrode signals on two separate brain regions, which enables the model to obtain the deep representations of all the EEG electrodes' signals while keeping the intrinsic spatial dependence. Then we design a pairwise subnetwork to capture the discrepancy information between two hemispheres and extract higher-level features for final classification. Besides, in order to reduce the domain shift between training and testing data, we use a domain discriminator that adversarially induces the overall feature learning module to generate emotion-related but domain-invariant feature, which can further promote EEG emotion recognition. We conduct experiments on three public EEG emotional datasets, and the experiments show that the new state-of-the-art results can be achieved.

I. INTRODUCTION

BiHDM addresses EEG emotion recognition by modeling asymmetric information between the brain’s left and right hemispheres while preserving electrode spatial structure. It combines hemispheric feature extraction, pairwise discrepancy modeling, and domain-adversarial learning.

  • I. INTRODUCTION: EEG emotion recognition is motivated by evidence that the anatomically symmetric hemispheres respond differently to emotions.This hemispheric discrepancy is presented as a promising source of information for improving recognition.
  • I. INTRODUCTION: BiHDM learns discrepant features between left and right hemispheres for EEG emotion recognition.The model targets higher-level asymmetric features expected to contain discriminative information.
  • I. INTRODUCTION: Horizontal and vertical traversing RNNs extract electrode features while retaining EEG electrodes’ intrinsic spatial relationships.The predefined electrode coordinates allow spatial graph structure to be represented through directed traversals.
  • I. INTRODUCTION: Pairwise operations on symmetric electrodes model asymmetric discrepancy information between the two hemispheres.The operations are applied after hemispheric feature extraction to measure differences between paired electrode representations.
  • I. INTRODUCTION: A domain discriminator encourages emotion-related but domain-invariant representations under EEG distribution shifts between training and testing subjects.The shift is especially relevant to subject-independent recognition, where source and target data come from different subjects.
  • I. INTRODUCTION: Experiments on three public EEG emotional datasets verify the discrimination and effectiveness of differential information between the hemispheres.The paper presents this integration of hemispheric electrode discrepancy into deep learning as novel.

II. THE PROPOSED MODEL FOR EEG EMOTION

The proposed model uses separate spatial RNN processing for the left and right hemispheres, with horizontal and vertical traversals that preserve electrode location relationships. The resulting electrode representations are prepared for later interaction and classification.

  • 1) Obtaining the deep representation for each electrode:: BiHDM first obtains deep representations for all EEG electrodes in the left and right hemispheres.Two independent hemispheric RNN modules separately scan the electrode data.
  • 1) Obtaining the deep representation for each electrode:: Each hemispheric RNN traverses electrodes using predefined horizontal and vertical spatial stacks.The two directional traversals provide complementary ways to simplify spatial graph modeling.
  • 1) Obtaining the deep representation for each electrode:: The EEG electrodes are modeled as nodes in separate left- and right-hemisphere spatial graphs connected by neighboring-electrode relations.A predefined forward evolution sequence traverses these graphs for recurrent processing.
  • 1) Obtaining the deep representation for each electrode:: The framework combines the electrode representations into downstream features used by the classifier and discriminator.Figure 1 summarizes four RNN modules and the construction of a final vector from interacting electrode representations.
  • 1) Obtaining the deep representation for each electrode:: The resulting hidden states represent electrode data while retaining its location-based structural relation.The representations are produced for both horizontal and vertical directions.

2) Interaction between the paired electrodes on two hemispheres:

BiHDM compares representations from symmetric electrode locations across hemispheres and then summarizes the resulting discrepancies into higher-level features. Multiple pairwise operations capture complementary aspects of hemispheric difference.

  • 2) Interaction between the paired electrodes on two hemispheres:: BiHDM identifies paired electrodes at symmetric scalp locations and applies pairwise operations to their deep representations.These operations produce deep asymmetric differential features for subsequent processing.
  • 2) Interaction between the paired electrodes on two hemispheres:: Subtraction, division, and inner product model hemispheric discrepancy from different aspects.Division represents relative magnitude ratio, while the inner product represents similarity or dissimilarity.
  • 2) Interaction between the paired electrodes on two hemispheres:: A further RNN processes the differential asymmetric features to capture higher-level discrepancy discriminative features.This processing is performed for the horizontal and vertical streams.
  • 2) Interaction between the paired electrodes on two hemispheres:: Projection matrices detect salient emotion-related information among the paired differential features.Learnable transformations are applied to the higher-level discrepancy features.
  • 2) Interaction between the paired electrodes on two hemispheres:: Learnable mapping matrices summarize the directional features into the model’s output representation.The horizontal and vertical streams are integrated after higher-level discrepancy processing.

3) Discriminative prediction and domain adversarial strategy:

BiHDM applies supervised classification to its extracted EEG features and adversarial domain alignment to obtain emotion-distinguishable, domain-invariant representations.

  • Discriminative prediction: The output feature vector is linearly transformed into C emotion logits before softmax classification.The transformation uses learnable matrices P and bc, with C denoting the number of emotion types.
  • Discriminative prediction: The softmax layer converts BiHDM’s output into class probabilities for EEG emotion classification.The predicted label is the class with the highest probability.
  • Discriminative prediction: The classifier loss uses ground-truth labels to encourage discriminative features for emotion recognition.The feature extractor and classifier parameters are optimized using the supervised classification objective.
  • Domain adversarial strategy: A domain discriminator cooperates with the classifier to align source and target feature distributions.The strategy encourages representations that distinguish emotions while remaining invariant to domain differences.

B. The optimization of BiHDM

BiHDM jointly optimizes supervised classification and adversarial domain discrimination to produce emotion-related, domain-invariant representations, particularly under subject-independent distribution shifts.

  • The optimization of BiHDM: The overall objective minimizes classification loss while maximizing domain-discriminator loss with respect to feature extraction.The source and target datasets are combined in the overall optimization formulation.
  • The optimization of BiHDM: This max-min objective targets emotion-related but domain-invariant representations for EEG emotion recognition.The stated motivation is the substantial distribution shift between source and target subjects in subject-independent tasks.
  • The optimization of BiHDM: A gradient reversal layer converts the discriminator’s maximizing problem into a minimization problem suitable for stochastic gradient descent.The layer preserves forward activations and reverses gradient signs during backpropagation.

A. Setting up

The evaluation uses three public EEG emotion datasets, paired left-right electrode inputs, subject-dependent protocols, and comparisons with established methods.

  • A. Setting up: The experiments use 62-channel EEG recorded at 1000 Hz and pair 31 electrodes at symmetric left-right scalp locations.The paired operation follows electrode locations based on the international 10-20 system.
  • A. Setting up: MPED contains 30 subjects, seven refined emotion types, and 28 trials per subject session.The study uses released differential-entropy features for SEED and SEED-IV and STFT features for MPED.
  • A. Setting up: For subject-dependent evaluation, source and target trials are split according to dataset-specific protocols for SEED, SEED-IV, and MPED.SEED uses nine training and six testing trials per session; SEED-IV uses 16 and eight, while MPED uses 21 and seven.
  • A. Setting up: BiHDM is compared with twelve methods, including conventional, deep-learning, graph-based, and domain-adaptation approaches.The comparison includes SVM, RF, CCA, DBN, GCNN, DGCNN, DANN, BiDANN, and EmotionMeter among others.
  • A. Setting up: BiHDM outperforms all compared methods on the three datasets and improves over EmotionMeter by 4% on SEED-IV.The paired t-test analysis reports that BiHDM is significantly better than the baseline method at significance level 0.05.

2) The subject-independent experiment:

The subject-independent experiment evaluates cross-subject generalization with leave-one-subject-out validation and domain-adaptation baselines; BiHDM achieves the best reported performance across all three datasets.

  • 2) The subject-independent experiment:: Leave-one-subject-out validation uses one subject for testing and the remaining subjects for training, repeating until every subject is tested once.Mean accuracy and standard deviation are the evaluation metrics.
  • 2) The subject-independent experiment:: The comparison includes twelve methods spanning importance estimation, transfer learning, classifiers, domain adaptation, and EEG neural networks.The listed methods include KLIEP, ULSIF, STM, SVM, TCA, GFK, DANN, DGCNN, DAN, BiDANN, and A-LSTM.
  • 2) The subject-independent experiment:: BiHDM achieves the best performance on SEED, SEED-IV, and MPED in subject-independent EEG emotion recognition.The reported accuracy improvements over existing state-of-the-art methods are 2.2%, 3.5%, and 2.4%, respectively.
  • 2) The subject-independent experiment:: Paired t-tests at significance level 0.05 indicate that BiHDM is significantly better than the baseline method.The statistical comparison is reported alongside the subject-independent results.

C. Confusion matrix

BiHDM’s confusion matrices show dataset- and emotion-specific recognition differences between subject-dependent and subject-independent experiments. Pairwise subtraction performs best among the tested discrepancy operations, while electrode maps highlight frontal and temporal asymmetry.

  • Confusion matrices: Subject-independent recognition decreases neutral and sad rates by about 10% and 9%, respectively, on SEED, while happy decreases 3%.The same pattern appears on SEED-IV; happy shows more similar brain reflections across people than neutral and sad.
  • Confusion matrices: On MPED, funny, neutral, and sad are easier to recognize in subject-dependent experiments, whereas sad recognition decreases significantly across subjects.The paper suggests sad-emotion patterns vary considerably between subjects.
  • Different pairwise operations: Subtraction achieves the best performance among BiHDM-S, BiHDM-D, and BiHDM-I because it directly measures interhemispheric discrepancy.Division and inner product describe differences from other aspects and remain comparable with methods reported elsewhere.
  • Different pairwise operations: Replacing subtraction with concatenation lowers subject-dependent accuracy on SEED, SEED-IV, and MPED from 93.12%, 74.35%, and 40.34% to 90.52%, 72.68%, and 37.89%.These results support explicitly extracting discrepancy through pairwise operations.
  • Electrode activity maps: Frontal asymmetry contributes most across datasets, while temporal asymmetry contributes similarly on MPED and varies by emotion.Positive emotions show temporal activity comparable to or greater than frontal activity; neutral and sad emphasize frontal asymmetry.

B. Electrodes reduction

The electrode-reduction experiments evaluate paired electrodes from frontal and temporal regions because practical EEG systems favor fewer electrodes. Reduced-electrode configurations remain comparable to using all 31 paired electrodes, especially on SEED.

  • Electrode selection: The study selects paired electrodes from frontal and temporal regions in configurations named Frontal (6), Frontal (10), Temporal (6), and Temporal (9).The configurations are motivated by activity-map results identifying frontal and temporal asymmetry as important.
  • Performance comparison: Fewer-electrode configurations produce recognition results comparable to all 31 paired electrodes, especially on SEED.The authors therefore consider reduced-electrode EEG emotion-recognition systems possible.
  • Brain-region comparison: Temporal-lobe electrodes outperform frontal-lobe electrodes in the reduced-electrode comparisons.The authors interpret this as suggesting a stronger association between temporal-lobe activity and emotion expression in EEG recognition.

C. The performance based on single hemispheric EEG data

The paper tests whether one hemisphere alone is sufficient and evaluates the spatial RNN design. Left-hemisphere input outperforms right-hemisphere input, while bidirectional spatial traversal outperforms single-direction variants.

  • Single hemispheric EEG data: BiHDM fed with left-hemisphere data performs better than with right-hemisphere data, especially on SEED-IV and MPED.The comparison uses subject-dependent EEG emotion-recognition experiments.
  • Spatial feature extraction: The horizontal-and-vertical RNN strategy achieves much better performance than either single-directional RNN variant.The variants are BiHDM-h and BiHDM-v, corresponding to single directional spatial feature extraction.
  • Conclusion: The experiments support incorporating asymmetric differential information between the two cerebral hemispheres into EEG emotion recognition.The conclusion identifies this incorporation as effective while proposing further study of alternative hemispheric differential operations.
Loading 1906.01704v1…