PaperlayerView exact revision on arXiv

Source-linked AI summary

TSception: A Deep Learning Framework for Emotion Detection Using EEG

Yi Ding, Neethu Robinson, Qiuhao Zeng, Duo Chen, Aung Aung Phyo Wai, Tih-Shih Lee, Cuntai Guan

arXiv:2004.02965v2eess.SPcs.LGstat.ML

TL;DR

EEG-based emotional-arousal detection remains challenging despite its relevance to psychological diagnosis and therapy. TSception learns temporal and spatial EEG representations, and in a VR study with 18 healthy subjects it achieved 86.03% classification accuracy, the highest among the compared methods.

  • Problem

    EEG-based emotional-arousal detection remains challenging despite its relevance to psychological diagnosis and therapy.

    Source: §I · p. 1

  • Method

    TSception uses multi-scale temporal and spatial convolutional learners to represent EEG in time and space, including frontal hemispheric information.

    Source: §I · p. 2, A. General Structure of Proposed… · p. 2, C. Spatial Learner · p. 4

  • Results

    86.03% accuracy: TSception achieved the highest classification accuracy among SVM (RP), SVM (DE), EEGNet, LSTM, Tception, and Sception.

    Source: §VI · p. 6, §V · p. 6

  • Takeaways & Limitations

    The authors conclude TSception's general structure means it can be applied to EEG signal classification generally.

    Source: §VI · p. 6

Abstract

from arXiv · show

In this paper, we propose a deep learning framework, TSception, for emotion detection from electroencephalogram (EEG). TSception consists of temporal and spatial convolutional layers, which learn discriminative representations in the time and channel domains simultaneously. The temporal learner consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of the EEG signal, which learns multiple temporal and frequency representations. The spatial learner takes advantage of the asymmetry property of emotion responses at the frontal brain area to learn the discriminative representations from the left and right hemispheres of the brain. In our study, a system is designed to study the emotional arousal in an immersive virtual reality (VR) environment. EEG data were collected from 18 healthy subjects using this system to evaluate the performance of the proposed deep learning network for the classification of low and high emotional arousal states. The proposed method is compared with SVM, EEGNet, and LSTM. TSception achieves a high classification accuracy of 86.03%, which outperforms the prior methods significantly (p<0.05). The code is available at https://github.com/deepBrains/TSception

I. INTRODUCTION

EEG-based emotional-arousal detection remains challenging despite its relevance to psychological diagnosis and therapy. The paper introduces TSception and a VR-BCI data-collection system to study low and high arousal states.

I. INTRODUCTION

  • Emotional arousal detection remains challenging, although arousal is relevant to psychological diagnosis, therapy, and emotion-focused therapy.The cited examples include anxiety disorders and autistic spectrum disorders.Source: §I · p. 1
  • The authors designed a VR-BCI system and collected EEG data from 18 healthy subjects to study emotional arousal in immersive VR.The system was designed to collect emotional-arousal EEG data in the VR environment.Source: §I · p. 2
  • TSception combines temporal and spatial learners to represent EEG in time and space, using multi-scale temporal kernels and frontal hemispheric asymmetry.The temporal kernels learn multiple temporal and frequency representations; hemisphere kernels target information from the left and right brain.Source: §I · p. 2
  • The proposed method is compared with SVM using relative power and differential entropy features, EEGNet, LSTM, Tception, and Sception.Tception and Sception are described as simplified self-comparison versions.Source: §I · p. 2

II. RELATED WORK

The related work motivates multi-scale convolution through Inception and reviews CNN approaches to EEG, where channels encode spatial information and time encodes temporal information.

A. GoogleNet and Inception Block

  • Inception blocks use split, transform, and merge operations with multi-scale convolutions to learn varied representations.The cited example uses 1x1, 3x3, and 5x5 kernels.Source: A. GoogleNet and Inception Block · p. 2

B. Convolutional Neural Network for EEG Data

  • EEG segments are treated as channel-by-time data, with sliding windows producing classifier inputs and electrodes encoding spatial information.The channel dimension corresponds to electrodes, while the time dimension carries temporal information.Source: B. Convolutional Neural Network for… · p. 2
  • Prior EEG CNNs include spatial-map convolutions, global spatial and temporal convolutions, and EEGNet's global spatial kernel with single-size temporal kernels.The reviewed work also includes deep CNNs for motor imagery and attentive mental-state detection.Source: B. Convolutional Neural Network for… · p. 2

III. METHODOLOGY

TSception processes raw EEG through temporal and then spatial learners before classifying learned features. Its temporal learner uses parallel multi-scale kernels tied to the signal sampling rate, followed by feature concatenation and batch normalization.

A. General Structure of Proposed Network - TSception

  • TSception is an end-to-end raw-EEG classifier that passes features through temporal and spatial learners before two fully connected layers map them to labels.The network has three parts: temporal learner, spatial learner, and classifier.Source: A. General Structure of Proposed… · p. 2, Figure 1 · p. 3

B. Temporal Learner

  • The temporal learner uses multiple 1D kernel sizes set as ratios of the EEG sampling rate, with one kernel type for each learner level.The levels are indexed from 1 to L.Source: B. Temporal Learner · p. 2
  • After each temporal convolution, ReLU activation and average pooling down-sample features; outputs from the levels are concatenated and batch-normalized.The passage states average pooling can reduce noise effects and feature dimension.Source: Figure 1 · p. 3, B. Temporal Learner · p. 3
  • Multi-scale temporal kernels are applied in parallel; longer kernels target longer-term and lower-frequency patterns, while shorter kernels target shorter-term and higher-frequency patterns.The described configuration uses three levels with ratio coefficients 0.5, 0.25, and 0.125.Source: B. Temporal Learner · p. 3

C. Spatial Learner

TSception’s spatial learner uses channel-location-aware kernels to capture global and hemispheric EEG relationships. The local-kernel option is left for future research.

C. Spatial Learner

  • A global kernel spanning all channels captures global spatial relation patterns.Its size is (C, 1), where C is the number of channels.Source: C. Spatial Learner · pp. 3–4
  • The spatial learner orders channels by hemisphere and uses global and hemisphere kernels to capture spatial relationships and left-right asymmetry.The input channels are arranged as [channel_left, channel_right]; the hemisphere kernel is shared across hemispheres without overlap.Source: C. Spatial Learner · pp. 3–4
  • The hemisphere kernel has size (0.5 · C, 1), steps by (0.5·C, 1), and extracts relationships between the left and right hemispheres.It shares weights across the two non-overlapping hemispheres to extract asymmetry patterns.Source: C. Spatial Learner · p. 4
  • Only global and hemisphere kernels are used; the proposed local kernel remains a possible alternative for future research.A local kernel would span the channels in a functionally defined brain subarea.Source: C. Spatial Learner · p. 4

D. Optimization of TSception

TSception is optimized with cross-entropy plus L1 regularization, and training uses validation-accuracy-based early stopping.

D. Optimization of TSception

  • The objective combines cross-entropy with an L1 penalty on model weights to keep weights small and help avoid over-fitting.The loss is expressed using ground-truth labels, predicted labels, the L1 coefficient λ, and model weights θ_i.Source: D. Optimization of TSception · p. 4
  • Training stops when validation accuracy no longer increases for a specified number of epochs.Early stopping is used to address over-fitting.Source: D. Optimization of TSception · p. 4

IV. EXPERIMENT

The experiment evaluates low- versus high-arousal EEG classification using a VR-BCI system and two contrasting VR stimuli. Participants completed repeated stimulus trials, with EEG collected under controlled conditions.

IV. EXPERIMENT

  • Eighteen healthy subjects (9 males and 9 females, ages 23–49) provided EEG through a four-channel MUSE headband at 256 Hz in a VR-BCI system.The channels were TP9, AF7, AF8, and TP10; sessions took place in an isolated, softly illuminated room, and the study was approved by NTU’s Institutional Review Board.Source: A. Data Acquisition · pp. 4–5
  • The training procedure takes raw EEG, labels, and a TSception model with temporal and spatial kernel counts, then uses Adam and saves the model when validation accuracy improves.The algorithm initializes the validation-accuracy maximum and applies an early-stopping patient criterion.Source: A. Data Acquisition · p. 4
  • The low-arousal stimulus showed a slowly flying white bird over a snowy frozen lake, while the high-arousal stimulus was a fast, random stone-avoidance game.The low-arousal scene used a first-person perspective and soft background music; the game used controller responses and adaptive difficulty to maintain arousal.Source: A. Data Acquisition · pp. 4–5, Figure 2 · p. 4
  • Each subject completed three sessions, with two trials per session; each stimulus lasted one minute, separated by a five-second relaxing break.The stimuli were presented as low- and high-arousal conditions.Source: A. Data Acquisition · p. 5, Figure 3 · p. 5

B. Signal Processing

The signal-processing pipeline filters EEG before model use and extracts frequency-band features for the SVM baseline.

B. Signal Processing

  • EEG is band-pass filtered from 0.3 Hz to 45 Hz, and electrooculography is removed with MNE before deep-learning methods use the processed data.Feature extraction is additionally required for SVM.Source: B. Signal Processing · p. 5
  • SVM inputs use relative power and differential entropy from nine frequency bands filtered in 4-Hz intervals from 4–8 Hz through 36–40 Hz.The passage identifies x_i as data in the i-th frequency band and n as the number of frequency bands; differential entropy uses the standard deviation of x_i.Source: B. Signal Processing · p. 5

C. Parameter Setting

TSception’s parameter configuration specifies its temporal and spatial kernels, optimizer settings, regularization, and implementation.

C. Parameter Setting

  • The proposed model was implemented with PyTorch, and its source code is available online.Source: C. Parameter Setting · p. 5
  • Three temporal-kernel levels use α = [0.5, 0.25, 0.125], with 9 kernels per level; each global and hemisphere spatial type uses 6 kernels.The first fully connected layer has 128 hidden nodes.Source: C. Parameter Setting · p. 5
  • Adam training uses a learning rate of 0.001, mini-batches of 128, dropout 0.3, early-stopping patience 4, and an L1 coefficient of 1e-06.The model was implemented in PyTorch, and its source code is available online.Source: C. Parameter Setting · p. 5

D. Experiment Setting

The experiments use subject-dependent, leave-one-session-out validation, with segmented EEG data and separate handling for deep models and SVMs.

D. Experiment Setting

  • Subject-dependent evaluation used leave-one-session-out cross-validation across each subject’s three sessions, training on two sessions and testing on one.Source: D. Experiment Setting · p. 5
  • EEG was split into 4-second windows with a 100 ms (25 data point) step, yielding 574 samples per stimulus.Source: D. Experiment Setting · p. 5
  • Each training fold reserved 20% of its training data for validation and early stopping; deep models received raw segments, while SVMs used manually extracted features.The passage reports training, validation, and testing set dimensions for each cross-validation step.Source: D. Experiment Setting · p. 5

V. RESULT AND ANALYSIS

TSception leads the reported method comparisons, and the self-comparison results favor combining temporal and spatial learners. The analysis also reports that temporal information contributes more than spatial information alone.

V. RESULT AND ANALYSIS

  • Tception removes TSception’s spatial learner, while Sception removes its temporal learner.The comparisons include these simplified versions alongside EEGNet, LSTM, and SVM baselines.Source: §V · p. 5
  • 86.03% accuracy made TSception highest among compared methods, ahead of Tception (83.9%) and SVM using DE (82.23%).The comparison reports results for leave-one-session-out cross-validation.Source: Figure 4 · p. 6, §V · p. 6
  • SVM using DE scored 82.23% versus 80.73% with RP; EEGNet scored 79.96% versus Sception’s 77.39%, indicating more useful temporal than spatial patterns alone.The authors attribute the lower EEGNet and Sception results to limited extraction of dynamic temporal information.Source: §V · p. 6
  • TSception improved accuracy by 2.4% over Tception and 8.34% over Sception, while having fewer trainable parameters than both.The authors report a shorter final feature vector for TSception because it extracts temporal and spatial patterns in sequence.Source: §V · p. 6
  • The authors offer cross-information between temporal and spatial patterns and reduced overfitting as possible explanations for the combined model’s results.Source: §V · p. 6

VI. CONCLUSION

TSception combines multi-scale temporal and spatial convolutional learners for EEG emotion classification. In an evaluation with 18 healthy subjects in immersive VR, it achieved the highest reported accuracy among the compared methods.

VI. CONCLUSION

  • TSception uses multi-scale temporal and spatial convolutional kernels to learn EEG representations in time and space simultaneously.The temporal learner extracts multi-frequency and multi-temporal patterns; the spatial learner uses hemisphere kernels to capture information from the left and right hemispheres.Source: §VI · p. 6
  • The evaluation collected EEG from 18 healthy subjects in a VR-BCI system to study emotional arousal in an immersive VR environment.Source: §VI · p. 6
  • TSception achieved the highest reported classification accuracy among the compared methods, at 86.03%.The authors state that the framework’s general structure means it can be applied to EEG signal classification generally.Source: §VI · p. 6

Related papers

Loading 2004.02965v2…