Source-linked AI summary
MS-MDA: Multisource Marginal Distribution Adaptation for Cross-subject and Cross-session EEG Emotion Recognition
Hao Chen, Ming Jin, Zhunan Li, Cunhang Fan, Jinpeng Li, Huiguang He
TL;DR
EEG emotion recognition must handle variability across subjects and sessions, while merging multiple sources can disrupt their marginal distributions. MS-MDA uses shared low-level features and separate source-target adaptation branches, and it outperforms comparison methods across the evaluated transfer scenarios.
Problem
Subject and session variability challenges transferable EEG emotion-recognition models, and merging multiple sources can violate source-marginal-distribution assumptions or increase adaptation difficulty.
Method
MS-MDA extracts shared domain-invariant features and uses an independent one-to-one domain-adaptation branch for each source-target pair to learn domain-specific features.
Results
MS-MDA outperforms comparison methods and state-of-the-art models in the evaluated cross-session and cross-subject scenarios on SEED and SEED-IV.
Takeaways & Limitations
Separate multi-source adaptation avoids disrupting EEG source marginal distributions and shows particular improvement when many source domains are available.
Abstract
from arXiv · showhide
As an essential element for the diagnosis and rehabilitation of psychiatric disorders, the electroencephalogram (EEG) based emotion recognition has achieved significant progress due to its high precision and reliability. However, one obstacle to practicality lies in the variability between subjects and sessions. Although several studies have adopted domain adaptation (DA) approaches to tackle this problem, most of them treat multiple EEG data from different subjects and sessions together as a single source domain for transfer, which either fails to satisfy the assumption of domain adaptation that the source has a certain marginal distribution, or increases the difficulty of adaptation. We therefore propose the multi-source marginal distribution adaptation (MS-MDA) for EEG emotion recognition, which takes both domain-invariant and domain-specific features into consideration. First, we assume that different EEG data share the same low-level features, then we construct independent branches for multiple EEG data source domains to adopt one-to-one domain adaptation and extract domain-specific features. Finally, the inference is made by multiple branches. We evaluate our method on SEED and SEED-IV for recognizing three and four emotions, respectively. Experimental results show that the MS-MDA outperforms the comparison methods and state-of-the-art models in cross-session and cross-subject transfer scenarios in our settings. Codes at https://github.com/VoiceBeer/MS-MDA.
I. INTRODUCTION
EEG emotion recognition is challenged by subject and session variability, while merging multiple source domains can disrupt their marginal distributions. MS-MDA addresses this with shared low-level features and separate source-target adaptation branches.
- EEG signals vary across subjects and sessions, making models difficult to share across cross-subject and cross-session scenarios.
- Combining multiple EEG sources into one domain can ignore source-specific marginal distributions and introduce larger bias.
- MS-MDA extracts shared domain-invariant features, then pairs each source with the target for one-to-one adaptation and domain-specific feature extraction.
- The paper proposes MS-MDA to avoid disrupting EEG marginal distributions in multi-source adaptation.
- Experiments on SEED and SEED-IV report that MS-MDA outperforms comparison methods and generalizes well.
II. RELATED WORK
Prior EEG emotion-recognition studies apply domain adaptation and deep learning, but many merge multiple sources into one domain. MS-MDA instead transfers sources separately while retaining domain-invariant features.
- Earlier EEG affective-BCI studies used transfer component analysis, kernel methods, subspace matching, and robust PCA for domain adaptation.
- Deep adaptation networks reduce source-target distance with maximum mean discrepancy, while other methods align invariant subspaces or joint distributions.
- Many prior works combine multiple sources into one, ignoring differences between EEG-domain marginal distributions and limiting generalization.
- MS-MDA transfers multiple source domains separately and combines domain-invariant features with a computationally efficient MLP-based framework.
- A. Diagram: An EEG emotion-recognition pipeline evokes emotions, records and labels EEG, preprocesses signals, extracts features, and performs pattern recognition.
B. Scenarios
The paper evaluates cross-subject and cross-session transfer using established SEED-family EEG emotion datasets. Source and target domains are defined by subjects or sessions within the experimental collections.
- Scenarios: Cross-subject transfer uses one session’s data from other subjects as sources and new-subject data as the target.
- Scenarios: Cross-session transfer uses earlier sessions from one subject as sources and a later session as the target.
- Scenarios: The experiments use three sessions and fifteen subjects, with results averaged across subjects for cross-session and across sessions for cross-subject transfer.
- Datasets: SEED contains positive, neutral, and negative emotion EEG signals from fifteen subjects across three sessions, recorded with a 62-channel system.
- Datasets: SEED-IV contains three sessions, fifteen subjects, and twenty-four film clips, adding eye-movement features to EEG signals.
D. Pre-processing
The method preprocesses EEG signals into differential-entropy features, then applies MS-MDA through shared and domain-specific branches to classify target-domain emotions.
- Pre-processing: EEG signals are down-sampled to 200 Hz and band-pass filtered between 1 Hz and 75 Hz before feature extraction.
- Feature extraction: Differential Entropy features from five frequency bands are used as the model input.
- Feature representation: Each source session or subject is represented as trial × 310, combining 62 channels with five frequency bands.
- MS-MDA architecture: A common feature extractor maps all domains into a shared latent space to obtain domain-invariant features.
- MS-MDA architecture: N domain-specific branches pair each source with the target, compute MMD-based adaptation, and extract domain-specific features.
- MS-MDA architecture: Domain-specific classifiers produce source classification losses and multiple target predictions, whose discrepancy is minimized before averaging target outputs.
- Optimization: The total training objective combines classification, MMD, and discrepancy losses to improve classifiers, align domains, and converge predictions.
V. EXPERIMENTS
Experiments evaluate EEG emotion classification and normalization-related adaptation behavior on the SEED and SEED-IV datasets, alongside exploratory comparisons.
- Experimental scope: Experiments classify emotions on SEED and SEED-IV and include a normalization study for EEG domain adaptation.
- Experimental scope: Additional exploratory experiments complement evaluations of MS-MDA and the comparison methods.
A. Implementation Details
Implementation uses lightweight multilayer and linear modules suited to extracted EEG vectors, with training settings reported alongside comparison results.
- Feature extractor: The common feature extractor uses a 3-layer MLP because the model processes extracted DE vectors rather than raw EEG signals.
- Evaluation: The implementation reports comparison results on both SEED and SEED-IV under the designated experimental settings.
- Feature dimensions: The common feature extractor reduces 310-dimensional inputs to 64 dimensions, while domain-specific modules reduce 64 dimensions to 32 and then to the category count.
B. Results
MS-MDA substantially outperforms comparison methods across the reported SEED and SEED-IV transfer scenarios. The gains are especially pronounced for cross-subject adaptation, where multiple sources can be modeled separately.
- MS-MDA largely outperforms comparison methods in most transfer scenarios on SEED and SEED-IV.
- 7% and 3% are the minimum improvements for SEED cross-session and cross-subject transfer, respectively.
- 7% and 18% are the minimum improvements for SEED-IV cross-session and cross-subject transfer, respectively.
- Cross-subject gains may be stronger because it uses 14 sources, compared with 2 sources in cross-session transfer.The authors connect the larger source count with the benefit of treating individuals as separate source domains.
C. Ablation Study
The ablation study evaluates the contribution of MS-MDA components by removing them individually on SEED and SEED-IV. Removing the MMD loss causes a significant performance drop.
- The ablation study compares the full MS-MDA model with versions that remove individual components on SEED and SEED-IV.
- Removing the MMD loss leaves classification and discrepancy losses but causes a significant drop versus the full model.The authors interpret this drop as indicating the important effect of domain adaptation.
D. Normalization
The study examines three normalization methods and two processing orders for multi-source EEG inputs. Electrode-wise normalization outperforms the alternatives across all three evaluated sets.
- The input is reshaped from session × channel × trial × band into session × trial × (channel*band), yielding trial × 310 matrices per session.
- The experiments compare sample-wise, electrode-wise, and global-wise normalization methods.The three categories are illustrated as distinct normalization strategies for EEG feature matrices.
- The two processing orders either normalize before concatenating sources or concatenate before normalization.
- Electrode-wise normalization outperforms the other methods in all three evaluated sets.MS-MDA and DAN are both evaluated with the three normalization methods and two normalization orders on SEED and SEED-IV.
E. Additions
Additional experiments study discrepancy-loss settings, batch size, training epochs, and learned feature distributions. MMD is influential, discrepancy loss requires tuning, and MS-MDA converges faster with moderate accuracy.
- MMD loss is influential, while discrepancy-loss performance depends on its weight and when it is introduced during training.
- A discrepancy-loss weight of 0.01 applied throughout training is selected based on the reported experiments.
- Larger batch sizes reduce performance in both MS-MDA and DAN, especially at batch size 512 versus 256 on SEED-IV.
- Increasing training epochs produces no substantial improvement, while MS-MDA achieves moderate accuracy and converges faster.
- The cross-subject t-SNE visualization samples 100 EEG instances per subject, colors source domains separately, and marks target samples in black.
VI. DISCUSSION
MS-MDA improves adaptation performance, particularly when many source domains are involved, but its multi-branch design increases training cost as source domains grow.
- MS-MDA significantly improves results over selective methods and prior work, especially in cross-subject adaptation with many source domains.
- Ablation analysis: Removing MMD causes a significant performance drop on both datasets, while removing discrepancy loss has a smaller effect.
- Normalization analysis: Electrode-wise normalization produces the strongest performance improvement among the evaluated normalization methods.
- Loss-coefficient analysis: The discrepancy-loss coefficient achieves the best results when set to 0.01 times the MMD-loss coefficient in full-model training.
- Hyper-parameter analysis: Both MS-MDA and DAN decline as batch size increases, while performance increases slightly with more epochs.
- Limitations: Training time increases linearly with the number of source domains because each source receives a separate domain-adaptation branch.