TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition

Yi Ding; Neethu Robinson; Su Zhang; Qiuhao Zeng; Cuntai Guan

Source-linked AI summary

TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition

Yi Ding, Neethu Robinson, Su Zhang, Qiuhao Zeng, Cuntai Guan

arXiv:2104.02935v4Apr 7, 2021cs.LG

TL;DR

Emotion recognition from EEG requires representations that reflect both rapidly changing brain activity and asymmetric spatial responses, while conventional approaches may depend on hand-engineered features or single-scale kernels. TSception uses multi-scale temporal kernels, hemisphere-aware spatial kernels, and high-level fusion, and it achieves the highest classification results among compared methods in most experiments on two public datasets. The method also uses relatively few trainable parameters, although cross-subject generalization and cropped-segment length remain future considerations.

Problem
EEG emotion recognition must capture dynamic temporal activity and hemispheric asymmetry, while many prior methods rely on manually extracted features or single-scale kernels.
Method
TSception is a multi-scale temporal-spatial convolutional network using sampling-rate-related temporal kernels, global and hemisphere spatial kernels, and high-level fusion.
Results
TSception achieves the highest classification results among compared methods in most experiments on DEAP and MAHNOB-HCI, with 1/4 or 1/10 of their trainable parameters.
Takeaways & Limitations
The results indicate promising and reasonably general performance for arousal-valence prediction, with efficiency that may benefit future online BCI usage.
Takeaways & Limitations
The study leaves TSception’s cross-subject generalization and the effect of cropped-experiment segment length for future investigation.

Abstract

from arXiv · show

The high temporal resolution and the asymmetric spatial activations are essential attributes of electroencephalogram (EEG) underlying emotional processes in the brain. To learn the temporal dynamics and spatial asymmetry of EEG towards accurate and generalized emotion recognition, we propose TSception, a multi-scale convolutional neural network that can classify emotions from EEG. TSception consists of dynamic temporal, asymmetric spatial, and high-level fusion layers, which learn discriminative representations in the time and channel dimensions simultaneously. The dynamic temporal layer consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of EEG, which learns the dynamic temporal and frequency representations of EEG. The asymmetric spatial layer takes advantage of the asymmetric EEG patterns for emotion, learning the discriminative global and hemisphere representations. The learned spatial representations will be fused by a high-level fusion layer. Using more generalized cross-validation settings, the proposed method is evaluated on two publicly available datasets DEAP and MAHNOB-HCI. The performance of the proposed network is compared with prior reported methods such as SVM, KNN, FBFgMDM, FBTSC, Unsupervised learning, DeepConvNet, ShallowConvNet, and EEGNet. TSception achieves higher classification accuracies and F1 scores than other methods in most of the experiments. The codes are available at https://github.com/yi-ding-cs/TSception

1 INTRODUCTION

EEG emotion recognition is motivated by EEG’s temporal resolution and emotion-related hemispheric asymmetry, while prior approaches often rely on manually extracted features or single-scale convolutions. TSception addresses these issues with multi-scale temporal-spatial convolution, asymmetric spatial kernels, and evaluation designed to assess generalization.

EEG can capture changing brain states at the sub-second level, supporting emotion recognition through brain-computer interfaces.
Because EEG contains activity across frequency bands and varies dynamically, single-sized temporal kernels may not capture emotion-related processing at different time scales.
Prior EEG emotion-recognition methods often rely on manually extracted features, requiring substantial domain knowledge.
TSception uses multi-scale temporal-spatial convolution to learn temporal dynamics and spatial asymmetry directly from EEG for emotional-state classification.Its design includes dynamic temporal kernels, asymmetric spatial processing, and high-level fusion.
The experiments use DEAP and MAHNOB-HCI with generalized cross-validation, alongside ablation and interpretability analyses.The analyses examine module contributions and informative EEG regions identified by saliency maps.

2 MULTI-SCALE CONVOLUTIONAL NEURAL NET-

TSception is a multi-scale convolutional network that learns EEG temporal dynamics, global spatial structure, and left-right hemispheric asymmetry for emotion recognition. Its representations are fused before classification.

Architecture: TSception processes EEG as channel-by-time data through dynamic temporal, asymmetric spatial, high-level fusion, and classifier components.The temporal layer operates channel by channel; spatial processing then learns global and hemispheric representations before classification.
Dynamic Temporal Layer: Temporal kernels use sampling-rate-related lengths to capture multi-scale temporal and frequency representations.The model uses multiple kernel scales, including ratios [0.5, 0.25, 0.125] when L = 3 and α = 0.5.
Dynamic Temporal Layer: Parallel temporal convolutions apply LeakyReLU and average pooling, with pooling reducing noise effects and feature dimensionality.Outputs from temporal levels are concatenated and batch-normalized into the dynamic temporal representation.
Asymmetric Spatial Layer: The asymmetric spatial layer combines global kernels of size (c, 1) with shared hemisphere kernels of size (0.5 · c, 1) to learn global and left-right patterns.The hemisphere kernels use paired channel arrangements and shared weights across the two hemispheres.
Asymmetric Spatial Layer: Central electrodes Fz, Cz, Pz, and Oz are removed so the model can learn left-right hemispheric asymmetry from paired channels.The remaining electrodes are organized by left and right hemisphere, with the spatial output concatenating global and hemisphere representations.
High-level Fusion Layer: A convolutional fusion layer combines global and hemisphere representations, followed by activation, average pooling, batch normalization, global average pooling, and softmax classification.Global average pooling is used to reduce over-fitting and model size before fully connected classification layers.

3 EXPERIMENTS

The experiments evaluate TSception using EEG data from two publicly available emotion-recognition benchmark datasets. The supplied experimental procedure also summarizes the model's sequential computation from input EEG through temporal, spatial, and fusion stages.

Datasets: TSception is evaluated on the publicly available DEAP and MAHNOB-HCI benchmark datasets.DEAP is described as a database for emotion analysis using physiological signals, while MAHNOB-HCI is a multimodal affect-recognition database.
Algorithm: The algorithm takes EEG data Xn ∈ R^c×l and a ground-truth label y as input, producing a TSception prediction.The procedure then computes temporal-kernel sizes and temporal features before generating ZT.
Algorithm: The algorithm obtains the asymmetric spatial representation ZS after applying spatial processing to the temporal output ZT.It separately invokes temporal-kernel and spatial-kernel stages before continuing to fusion.
Algorithm: The high-level fusion stage computes Zfusion from the spatial representation before the final prediction step.The supplied algorithm explicitly identifies fusion as the next stage after obtaining ZS.

17 end

The experiments use DEAP and MAHNOB-HCI EEG datasets, preprocess and binarize arousal and valence labels, and evaluate TSception with accuracy and F1 under two cross-validation settings. Implementation details specify sampling-dependent temporal kernels and shared hyperparameters across datasets.

Datasets: DEAP contains 32 subjects and 40 trials per subject, while MAHNOB-HCI contributes 27 usable subjects with EEG recorded from 32 electrodes.DEAP uses 512 Hz EEG; MAHNOB-HCI uses 256 Hz EEG.
Pre-processing: DEAP preprocessing removes the baseline, downsamples EEG to 128 Hz, removes EOG, applies a 4.0-45 Hz band-pass filter, and averages channels to a common reference.Labels are thresholded at 5 to form low and high classes for arousal and valence.
Pre-processing: MAHNOB-HCI preprocessing removes pre- and post-trial baselines and applies a 0.3-45 Hz band-pass filter, retaining the delta band.
Performance Evaluation Metrics: Accuracy measures correct predictions over all samples, while F1 combines precision and recall and is added for imbalanced labels.The paper reports both metrics across its experiments.
Experiment Settings: The study evaluates cropped trials with trial-wise 10-fold cross-validation and leave-one-trial-out cross-validation, keeping test data unseen during training and validation.Mean accuracy and F1 score across subjects are reported as final results.
Implementation Details: Temporal kernel lengths are 64, 32, and 16 for both datasets, while most hyperparameters are shared across datasets to test generalization.DEAP uses batch size 64, MAHNOB-HCI uses 32, and training runs for up to 500 epochs.

4 RESULTS AND ANALYSIS

TSception generally outperformed compared methods across DEAP and MAHNOB-HCI, while ablations showed that all three functional layers contribute to classification. Its asymmetric spatial layer had the largest overall impact, and saliency maps highlighted distributed, asymmetric regional contributions.

Statistical Analysis: On DEAP, TSception exceeded SVM by 1.2% accuracy and 5.91% F1 score for arousal, and by 3.95% accuracy and 4.46% F1 score for valence.The valence improvements were statistically significant at p < 0.05.
Statistical Analysis: On MAHNOB-HCI, TSception achieved the best valence accuracy and F1 score, while its arousal F1 score exceeded KNN by 4.41%.KNN achieved the best arousal accuracy, whereas SVM achieved the best arousal F1 score.
Statistical Analysis: TSception used 12,563 parameters, approximately one-quarter of ShallowConvNet and one-tenth of DeepConvNet, while achieving the highest results in most DEAP deep-learning comparisons.On DEAP, its accuracy was 61.57% for arousal and 59.14% for valence, with F1 scores of 63.24% and 62.33%, respectively.
Statistical Analysis: TSception achieved the highest classification results in most experiments across the two public datasets and showed a decent extent of generality.The evaluation reported accuracy and F1 score comparisons on DEAP and MAHNOB-HCI using generalized cross-validation settings.
Ablation Study: Removing any functional layer reduced accuracy and F1 score, with the asymmetric spatial layer producing the largest drops: 1.5%/1.84% accuracy and 1.84%/1.59% F1 for arousal/valence.The high-level fusion layer contributed more to arousal, whereas the asymmetric spatial layer contributed especially strongly to valence.
Ablation Study: Hemisphere kernels learned more discriminative representations than global kernels, although removing either spatial-kernel type degraded arousal and valence performance.Saliency maps emphasized frontal, temporal, parietal, and occipital regions, with regional patterns differing across arousal, valence, and subject performance.

5 DISCUSSION AND CONCLUSION

TSception combines multi-scale temporal-spatial modeling with generalized evaluation to support EEG emotion recognition. It achieves strong comparative performance and parameter efficiency, while interpretability analyses identify emotion-related spatial patterns and motivate future studies of cross-subject generalization and segment length.

Evaluation strategy: Trial-wise 10-fold cross-validation of cropped trials is used to reduce inflated results caused by correlated adjacent segments from the same trial appearing across training and testing.Randomly shuffling samples before splitting can produce very high results that drop under trial-wise separation.
Performance and efficiency: TSception achieves the highest classification results among compared methods in most experiments on two public datasets, while using 1/4 or 1/10 of their trainable parameters.The authors associate this efficiency and effectiveness with potential online BCI usage.
Interpretability and ablation: Ablation and interpretability experiments indicate that all TSception modules contribute positively, with the asymmetric spatial layer contributing most to classification results.Saliency maps were used to examine whether the network learned emotion-related information.
Interpretability and ablation: Saliency analyses show frontal, temporal, parietal, and occipital activation across subjects, whereas high-F1 subjects mainly show frontal, temporal, and parietal activation.The averaged topological map in Fig. 6 is produced by averaging saliency maps along the time dimension.
Interpretability and ablation: Right-hemisphere lateralization appears for high-F1 valence subjects but not high-F1 arousal subjects, possibly because frontal information is sufficient for arousal decisions.Occipital activity contributes for all subjects but is less useful than frontal and temporal activity for high-F1 subjects; music videos may explain stronger occipital activity in DEAP.
Conclusion and future work: The paper concludes that TSception shows promising and reasonably general performance for arousal-valence prediction, while leaving cross-subject generalization and cropped-segment length for future study.The conclusion specifically identifies generalization across subjects and the effect of segment length as open questions.