Source-linked AI summary
Investigating EEG-Based Functional Connectivity Patterns for Multimodal Emotion Recognition
Xun Wu, Wei-Long Zheng, Bao-Liang Lu
TL;DR
The paper addresses limited investigation of emotion-related EEG functional connectivity and the dominance of single-channel features in affective BCI. It selects emotion-relevant subnetworks, evaluates connectivity features across three datasets, and combines EEG with complementary modalities, achieving strong multimodal recognition performance.
Problem
Emotion-related brain functional connectivity networks are not well investigated, while prior EEG emotion-recognition studies mainly use single-channel feature extraction.
Method
The paper selects emotion-relevant critical subnetworks, evaluates strength, clustering coefficient, and eigenvector centrality, and applies DCCA to combine EEG connectivity with other modalities.
Results
Across SEED, SEED-V, and DEAP, strength was the best connectivity feature, while DCCA multimodal recognition achieved 95.08 ± 6.42% on SEED, 84.51 ± 5.11% on SEED-V, and 85.34 ± 2.90%/86.61 ± 3.76% for DEAP arousal/valence.
Takeaways & Limitations
The findings identify emotion-associated connectivity patterns and support complementary EEG connectivity and eye-movement representations for multimodal emotion recognition.
Takeaways & Limitations
The affective models are subject-dependent, although subnetworks are selected using training samples from all subjects.
Abstract
from arXiv · showhide
Compared with the rich studies on the motor brain-computer interface (BCI), the recently emerging affective BCI presents distinct challenges since the brain functional connectivity networks involving emotion are not well investigated. Previous studies on emotion recognition based on electroencephalography (EEG) signals mainly rely on single-channel-based feature extraction methods. In this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public emotion EEG datasets: SEED, SEED-V, and DEAP. The strength feature achieves the best classification performance and outperforms the state-of-the-art differential entropy feature based on single-channel analysis. The experimental results reveal that distinct functional connectivity patterns are exhibited for the five emotions of disgust, fear, sadness, happiness, and neutrality. Furthermore, we construct a multimodal emotion recognition model by combining the functional connectivity features from EEG and the features from eye movements or physiological signals using deep canonical correlation analysis. The classification accuracies of multimodal emotion recognition are 95.08/6.42% on the SEED dataset, 84.51/5.11% on the SEED-V dataset, and 85.34/2.90% and 86.61/3.76% for arousal and valence on the DEAP dataset, respectively. The results demonstrate the complementary representation properties of the EEG connectivity features with eye movement data. In addition, we find that the brain networks constructed with 18 channels achieve comparable performance with that of the 62-channel network in multimodal emotion recognition and enable easier setups for BCI systems in real scenarios.
1 INTRODUCTION
Affective BCI emotion recognition remains underexplored at the level of brain functional connectivity, while most EEG approaches use single-channel features. This paper investigates emotion-relevant EEG connectivity patterns and their multimodal value with eye movement or physiological data.
- Affective BCI faces distinct challenges because emotion-related brain functional connectivity networks are not well investigated.
- Most EEG emotion-recognition studies independently extract features within single channels, despite emotion involving circuits across multiple cerebral regions.
- The study evaluates EEG connectivity features for emotion recognition on the SEED, SEED-V, and DEAP public datasets.
- The proposed work selects emotion-relevant critical subnetworks, evaluates strength, clustering coefficient, and eigenvector centrality, and combines EEG connectivity with eye movement or physiological features.
- The contributions include identifying emotion-associated connectivity patterns and examining whether fewer-electrode brain networks could support affective BCI systems in real scenarios.
2 RELATED WORK
Prior emotion-recognition research spans behavioral and physiological modalities, with EEG connectivity and multimodal EEG–eye-tracking methods emerging as complementary directions. Existing work motivates connectivity-based features and DCCA fusion for affective BCI.
- Emotion-recognition modalities include external behavioral signals such as facial expression and speech and internal physiological signals such as EEG, ECG, respiration, and galvanic skin response.
- EEG features have been developed in temporal, frequency, and time-frequency domains, but raw EEG is nonstationary and commonly contaminated by artifacts.
- Brain connectivity research distinguishes structural, functional, and effective connectivity, with functional connectivity describing statistical relations between brain regions.
- Only a few prior EEG emotion studies used multichannel connectivity measures such as phase locking value, correlation, coherence, or phase synchronization index.
- Eye movement data, particularly pupil responses, have been linked to emotional arousal and valence and can be collected with wearable, portable, noninvasive eye-tracking glasses.
- Prior multimodal studies used autoencoders, LSTM models, and DCCA, with reported evidence that DCCA outperformed BDAE and bimodal-LSTM models.
3 EMOTION EXPERIMENT DESIGN
The emotion experiments simultaneously recorded EEG and eye movements while participants viewed selected movie clips intended to elicit five prototypical emotions. The laboratory protocol used wearable sensing, screening, self-assessment, and relaxation periods.
- The experiment simultaneously recorded EEG and eye movement signals while subjects watched movie clips designed to elicit disgust, fear, sadness, happiness, and neutrality.
- Nine clips for each emotion were selected after 20 volunteers rated their elicitation success, with every chosen clip receiving a mean score of at least 3 out of 5.
- Sixteen subjects with normal hearing and normal or corrected-to-normal vision were recruited, with personality screening used during participant selection.
- EEG was collected with a 62-channel wet-electrode cap and eye movements with SMI eye-tracking glasses during emotional movie viewing.
- Each trial included a 15-second introduction followed by a movie clip and 15 or 30 seconds of self-assessment and relaxation to mitigate emotions.
- The SEED dataset contains 62-channel EEG and eye movement data for sad, happy, and neutral emotions from 9 subjects across 27 experiments.
- The experiments received ethics approval, and all participants provided informed consent before participation.
4.1 Preprocessing
The preprocessing pipeline removed artifacts from EEG and eye movement recordings before feature extraction. EEG was baseline-corrected, bandpass-filtered, and downsampled, while eye signals used EOG/FPZ correction and PCA for pupil-related luminance effects.
- EEG preprocessing used Curry 7 for baseline correction, a 1–50 Hz bandpass filter, and downsampling from 1000 Hz to 200 Hz.
- Eye movement artifacts were removed using EOG and FPZ recordings, while principal component analysis addressed luminance-related effects in pupil diameter.
4.2 Brain Functional Connectivity Network
The paper constructs EEG functional connectivity networks by measuring channel associations across frequency bands and organizing them as weighted connectivity matrices. Networks are built from 4-second EEG segments using either correlation or coherence, with corresponding 18-channel subnetworks derived from the 62-channel matrices.
- EEG connectivity networks represent electrodes as vertices and associations between pairs of EEG signals as edges.
- 4.2.2 Edge Measurement: Pearson’s correlation coefficient measures linear temporal relations between EEG signals, whereas spectral coherence measures frequency-domain connectivity.
- 4.2.3 Network Construction: Each sample is segmented into nonoverlapping 4-second EEG windows, with five corresponding segments produced by the five frequency bands.
- 4.2.3 Network Construction: For each frequency band, pairwise channel associations form a 62 × 62 symmetric connectivity matrix, whose diagonal self-correlations are set to zero.
- 4.2.3 Network Construction: Five brain connectivity networks are obtained per sample, while an 18 × 18 network is formed by selecting the corresponding elements of the 62 × 62 matrix.
4.3 Emotion-Relevant Critical Subnetwork Selection
The emotion-relevant critical subnetwork algorithm selects strong, emotion-associated edges from training-set networks and merges them into frequency-specific subnetworks. This reduces weak artifact-related associations while targeting patterns shared across subjects, although the resulting affective models remain subject-dependent.
- Weak residual artifacts can create weak associations that obscure brain-network topology, motivating threshold-based edge selection.
- The algorithm uses all subjects’ training samples to explore universal emotion-relevant connectivity patterns, while affective models remain subject-dependent.
- For each emotion and frequency band, networks are averaged, strongest associations are retained with a proportional threshold, and critical edges are merged.
- The proportional threshold is tuned as a model hyperparameter from 0.0 to 1.0 in increments of 0.01.
- The merged critical edges and original vertices define the emotion-relevant critical subnetwork G* = (V, E*).
4.4 Feature Extraction
The paper extracts network-topology features from emotion-relevant EEG connectivity matrices and combines them with eye-movement features for multimodal modeling. The three EEG features quantify connectivity strength, functional segregation, and vertex importance.
- Critical connectivity matrices are derived for each sample from five frequency-specific emotion-relevant subnetworks.
- The strength feature measures network topology using positive and negative associations connected to each vertex.
- The clustering coefficient measures functional segregation by quantifying clusters and the fraction of triangles around each vertex.
- Eigenvector centrality evaluates a vertex’s significance in interacting with other vertices and facilitating network integration.
- Across each frequency band, strength, clustering coefficient, and eigenvector centrality feature dimensions are 2N + 2, 2N + 2, and N, respectively.
- 4.4.2 Eye Movement Features: The eye-movement representation is a 33-dimensional feature derived from pupil, fixation, blink, saccade, and event statistics.
4.5 Classification
The classification experiments combine smoothed, selected multimodal features with deep canonical correlation analysis and evaluate them across SEED, SEED-V, and DEAP setups. The DCCA output is fused and classified with an SVM, while hyperparameters are tuned by cross-validation and grid search.
- Feature Processing: LDS smoothing and mRMR selection address smooth emotional variation and reduce unrelated or redundant high-dimensional EEG information.
- Deep Canonical Correlation Analysis Model: DCCA learns correlated representations from two modalities through nonlinear layers and CCA before feature fusion.The model architecture comprises stacked nonlinear layers, CCA calculation, and a feature fusion layer.
- Deep Canonical Correlation Analysis Model: The fused multimodal feature is defined as a weighted average of the transformed features and then fed into an SVM.
- Hyperparameter Tuning: The DCCA layer widths are searched under n1 ≥ n2 ≥ n3 with each width in {25, 26, 27, 28}, while the learning rate ranges from 10^-8 to 10^-4.
- Experiment Setups: The evaluation uses SEED, SEED-V, and DEAP, including three-class SEED and five-class SEED-V emotion classification tasks.SEED uses the first nine trials for training and the last six for testing; SEED-V uses three-fold cross-validation.
5 EXPERIMENTAL RESULTS AND DISCUSSION
Across SEED, SEED-V, and DEAP, EEG connectivity features showed strong emotion-discrimination ability, with strength generally performing best. Combining strength-based EEG connectivity with eye movements or peripheral physiological signals further improved multimodal recognition, while 18-channel networks often matched 62-channel performance.
- Strength and eigenvector centrality generally performed better with correlation, whereas clustering coefficient performed better with coherence.
- 18-channel brain networks achieved considerable performance compared with 62-channel networks, including 84% happiness accuracy for both approaches.
- 80.17 ± 7.12% was obtained on SEED, exceeding the 78.51 ± 14.32% differential entropy result; DEAP strength accuracies were 73.42 ± 4.67% for arousal and 76.10 ± 4.49% for valence.
- 84.51±5.11% was achieved by DCCA on SEED-V, outperforming strength alone at 74.05±7.09%, eye movement data at 65.21±7.60%, and feature-level fusion at 78.03±6.07%.
- EEG connectivity and eye movement data showed complementary emotion representations: EEG favored happiness, whereas eye movements excelled at fear detection.
- 95.08 ± 6.42% was achieved on SEED, while DEAP reached 85.34 ± 2.90% for arousal and 86.61 ± 3.76% for valence using multimodal fusion.
5.3 Critical Frequency Bands
On SEED-V, β and γ bands were most effective for five-emotion classification, and stable, emotion-specific connectivity patterns emerged across frequency bands. Eighteen-channel networks achieved performance comparable to 62-channel networks.
- β and γ frequency bands were superior to other bands for classifying the five emotions.
- 18-channel frequency-band networks achieved performance comparable to 62-channel networks, supporting the possibility of using 18 electrodes in real-world emotion recognition.
- The critical-subnetwork procedure produced 25 subnetworks after thresholding and five after merging, corresponding to emotion-frequency and frequency-level groupings.
- Stable connectivity patterns were observed when critical subnetworks were recalculated across three cross-validation folds.
- Negative affective states showed higher positive frontal connectivity in the δ band, while disgust and fear differed in γ-band positive and negative connectivity.
- Similar connectivity patterns for sadness and neutrality could account for their confusion, despite different δ-band regional connectivity profiles.
- Happiness showed widespread positive and negative δ-band connectivity, distinct θ-band frontal-parietal and frontal-temporal patterns, and γ-band similarity with fear and disgust.
6 CONCLUSION
The paper identifies emotion-associated EEG functional connectivity patterns and shows that connectivity-based features support emotion recognition across three public datasets. Multimodal models further combine EEG connectivity with eye-movement or physiological features, while 18-channel networks remain promising for practical affective BCI systems.
- The emotion-relevant critical subnetwork selection algorithm was evaluated with strength, clustering coefficient, and eigenvector centrality on SEED, SEED-V, and DEAP.
- The experiments revealed emotion-associated brain functional connectivity patterns across the evaluated datasets.
- The strength feature was the best EEG connectivity feature and outperformed the state-of-the-art differential entropy feature based on single-channel analysis.
- Multimodal classification achieved 95.08 ± 6.42% on SEED and 84.51 ± 5.11% on SEED-V.
- On DEAP, multimodal classification achieved 85.34 ± 2.90% and 86.61 ± 3.76%.
- EEG connectivity features had complementary representation properties with eye-movement data, and 18-channel networks were promising for multimodal emotion recognition in actual scenarios.