PaperlayerView exact revision on arXiv

Source-linked AI summary

4D Attention-based Neural Network for EEG Emotion Recognition

Guowen Xiao, Mengwen Ye, Bowen Xu, Zhendi Chen, Quansheng Ren

arXiv:2101.05484v1cs.LG

TL;DR

EEG emotion recognition remains challenging because integrating information across domains and capturing discriminative patterns are difficult. 4D-aNN combines 4D spatial-spectral-temporal representations with attention in a CNN and bidirectional LSTM. It achieves state-of-the-art performance on SEED under intra-subject splitting, and ablation results support the effectiveness of its attention mechanisms.

  • Problem

    Integrating information across EEG domains and adaptively capturing discriminative local patterns remain challenging for emotion recognition.

    Source: Introduction · pp. 2–3

  • Method

    4D-aNN transforms EEG into 4D spatial-spectral-temporal representations and applies spatial and spectral attention in a CNN and temporal attention in a bidirectional LSTM.

    Source: Introduction · p. 3, Conclusion · p. 11

  • Results

    4D-aNN achieves state-of-the-art performance on SEED under intra-subject splitting, with an average ACC of all subjects of 96.10%.

    Source: Introduction · p. 3, Results · p. 9

  • Takeaways & Limitations

    Without all attention mechanisms, 4D-aNN's classification accuracy decreases by 2.17%, the largest decrease among the compared ablations.

    Source: Discussion · p. 11

Abstract

from arXiv · show

Electroencephalograph (EEG) emotion recognition is a significant task in the brain-computer interface field. Although many deep learning methods are proposed recently, it is still challenging to make full use of the information contained in different domains of EEG signals. In this paper, we present a novel method, called four-dimensional attention-based neural network (4D-aNN) for EEG emotion recognition. First, raw EEG signals are transformed into 4D spatial-spectral-temporal representations. Then, the proposed 4D-aNN adopts spectral and spatial attention mechanisms to adaptively assign the weights of different brain regions and frequency bands, and a convolutional neural network (CNN) is utilized to deal with the spectral and spatial information of the 4D representations. Moreover, a temporal attention mechanism is integrated into a bidirectional Long Short-Term Memory (LSTM) to explore temporal dependencies of the 4D representations. Our model achieves state-of-the-art performance on the SEED dataset under intra-subject splitting. The experimental results have shown the effectiveness of the attention mechanisms in different domains for EEG emotion recognition.

Introduction

EEG emotion recognition can use spatial, spectral, and temporal information, but integrating these domains and adaptively capturing discriminative patterns remains challenging. The paper proposes 4D-aNN to address this challenge and reports state-of-the-art performance on SEED under intra-subject splitting.

Introduction

  • EEG is non-invasive, portable, and inexpensive, and its spatial, spectral, and temporal information is used to study emotion recognition.The passage describes applications including disease detection, human-computer interaction, and workload estimation.Source: Introduction · p. 2
  • Traditional EEG emotion recognition uses hand-crafted features and shallow classifiers, which require expert knowledge to design and select features.Deep learning methods have also been applied to EEG emotion recognition, and the cited approaches outperform shallow models.Source: Introduction · p. 2
  • The paper identifies a need to integrate information across domains while adaptively capturing important brain regions, frequency bands, and timestamps for EEG emotion recognition.Prior work leaves some differences among brain regions and frequency bands underused.Source: Introduction · pp. 2–3
  • 4D-aNN transforms raw EEG signals into 4D spatial-spectral-temporal representations and applies attention to spatial and spectral dimensions in a CNN and temporal slices in a bidirectional LSTM.The stated contribution is to fuse information across domains and capture discriminative patterns.Source: Introduction · p. 3
  • 4D-aNN achieves state-of-the-art performance on the SEED dataset under intra-subject splitting.This is the paper's reported experimental scope.Source: Introduction · p. 3

Method

4D-aNN processes 4D spatial-spectral-temporal EEG representations through an attention-based CNN, an attention-based bidirectional LSTM, and a classifier. The representations combine DE and PSD features across channels, frequency bands, and temporal slices while preserving electrode positions.

Method

  • 4D-aNN comprises a 4D spatial-spectral-temporal representation, an attention-based CNN, an attention-based bidirectional LSTM, and a classifier.These components form the overall model structure.Source: Method · p. 3

4D spatial-spectral-temporal representation

Attention-based CNN

The attention-based CNN extracts spatial-spectral representations from each temporal slice. Its convolutional attention modules apply spectral and spatial attention to help capture important frequency bands and brain regions.

Attention-based CNN

  • For each temporal slice, the CNN extracts spatial-spectral information and a convolutional attention module explores discriminative local patterns.The resulting representation is the spatial and spectral representation of that slice.Source: Attention-based CNN · p. 5
  • Spectral attention identifies valuable frequency bands, while spatial attention adaptively captures critical brain regions in each 2D feature map.The attention modules follow the convolutional layers.Source: Attention-based CNN · p. 5
  • The CNN contains four convolutional layers, four convolutional attention modules, one max-pooling layer, and one fully connected layer.The figure's upper block labels indicate layer types and the lower labels indicate output tensor shapes.Source: Attention-based CNN · p. 5, Figure 4 · p. 5

Convolutional attention module

The convolutional attention module combines spectral and spatial attention to adaptively identify valuable frequency bands and brain regions. It refines convolutional feature maps using these attention weights.

Convolutional attention module

  • The convolutional attention module has spectral and spatial submodules that adaptively capture important frequency bands and brain regions.The spectral submodule is applied before the spatial submodule in the described feature-refinement sequence.Source: Convolutional attention module · p. 6, Figure 5 · p. 7
  • Spectral attention uses spatial-wise average and maximum pooling to identify valuable frequency bands.The resulting spectral attention weights the feature maps element-wise.Source: Convolutional attention module · p. 6
  • Spatial attention pools across the spectral dimension and weights corresponding spatial regions in the refined feature tensor.Its attention map is generated from concatenated average and maximum representations, followed by convolution and sigmoid activation.Source: Convolutional attention module · p. 6

Attention-based bidirectional LSTM

The attention-based bidirectional LSTM processes the CNN output sequence in both time directions and uses temporal attention to weight information across slices. It aggregates the attended outputs into a high-level representation.

Attention-based bidirectional LSTM

  • The bidirectional LSTM processes the CNN sequence forward and in reverse, concatenating the two directional outputs at each time step.Each unidirectional LSTM has 36 memory cells, and the concatenated output has 72 dimensions.Source: Attention-based bidirectional LSTM · p. 7, Figure 6 · p. 8
  • Temporal attention considers outputs from all bidirectional LSTM nodes, rather than only the last node, to represent the importance of different temporal slices.The temporal attention is generated using two fully connected layers, a ReLU activation, and softmax.Source: Attention-based bidirectional LSTM · pp. 7–8
  • The high-level representation integrates spatial, spectral, and temporal information by weighting each output coordinate across temporal slices.The resulting representation has 72 elements.Source: Attention-based bidirectional LSTM · p. 8

Classifier

The classifier maps the high-level EEG representation to class probabilities and predicts the class with the largest probability.

Classifier

  • A fully connected layer followed by softmax produces class probabilities for each 4D sample.The layer uses learnable parameters.Source: Classifier · p. 8
  • The output vector gives the probability that the 4D sample belongs to each of the C classes.The probability vector is denoted Pre.Source: Classifier · p. 8
  • The predicted label is the class with the largest probability.Source: Classifier · p. 8

Experiment

The section previews the experiment by introducing a dataset, describing the settings, and reporting results on that dataset.

Experiment

SEED Dataset

The SEED dataset provides EEG recordings for positive, neutral, and negative emotions from 15 healthy subjects across three experiments per subject. The protocol uses emotion-eliciting film clips and manually removes severely contaminated signals before filtering.

SEED Dataset

  • SEED contains positive, neutral, and negative emotion data, with five approximately four-minute film clips selected for each category.The clips were selected to elicit their target emotion.Source: SEED Dataset · p. 8
  • Fifteen healthy subjects participated: seven males and eight females, with a reported mean age of 23.27 ± 2.37.Source: SEED Dataset · p. 8
  • Each subject completed three experiments of 15 clip-viewing processes; recordings used 62 channels, were down-sampled to 200 Hz, manually cleaned of severe EMG/EOG contamination, and filtered from 0.3 to 50 Hz.Each viewing process included a 5-second start hint, a 4-minute clip, 45-second self-assessment, and 15-second rest; same-emotion clips were not consecutive.Source: SEED Dataset · p. 8

Settings

The experiments use 4D segments with a 19 × 19 electrode map, five frequency bands, and segment length T = 3, evaluated by fivefold cross-validation. The comparisons include five EEG emotion-recognition baselines with differing spatial, spectral, and temporal designs.

Settings

  • Each experiment uses 19 × 19 electrode maps, all five frequency bands, and segment length T = 3, yielding about 1,128 samples.The method averages ACC and STD across fivefold cross-validation for each experiment, then across three experiments per subject and all subjects.Source: Settings · p. 8
  • Training uses Adam with learning rate 0.0003, batch size 12, and at most 150 epochs on an NVIDIA GTX 1080 GPU.Source: Settings · p. 8

Baseline Models

  • The five baselines range from spatial-only HCNN and hemisphere-aware BiHDM to topology-aware RGNN and models combining spatial, spectral, and temporal information, 4D-CRNN and SST-EmotionNet.SST-EmotionNet also uses attention mechanisms.Source: Baseline Models · p. 9

Results

On SEED under intra-subject splitting, 4D-aNN reports the highest compared-model performance, with average ACC of 96.10%. Ablations and emotion-specific heatmaps further characterize the attention mechanisms and regional patterns.

Results

  • Nine subjects performed above the 96.10% average ACC; Fig. 7 presents results for each of three experiments and the average classification accuracy per subject.Source: Results · p. 9, Figure 7 · p. 9
  • 96.10% average ACC is reported for 4D-aNN on SEED under intra-subject splitting, versus 96.02% classification accuracy for SST-EmotionNet.The comparison table reports average ACC and STD (%); the paper describes 4D-aNN as state of the art among the compared models.Source: Table 1 · p. 9, Results · p. 9
  • 2.17% is the accuracy decrease when all attention mechanisms are removed; individual removals decrease accuracy by 0.63% for spectral, 0.47% for spatial, and 1.19% for temporal attention.The all-attention ablation is the worst among the compared models.Source: Results · p. 10, Figure 8 · p. 10
  • For subject #15, heatmaps identify FC5, FC3, and C5 for positive emotion; CP5, CP3, and CP1 for neutral; and PO7, PO5, and P3 for negative emotion.The figure caption states that darker red regions denote greater contributions to recognizing the corresponding emotion.Source: Results · p. 10, Figure 9 · p. 10
  • The critical brain regions may vary with subjects, time, and emotions, motivating attention mechanisms that adaptively capture discriminative patterns.Source: Results · p. 10

Discussion

The discussion describes 4D-aNN’s attention across spatial, spectral, and temporal EEG information, reports better SEED performance than all baselines, and examines the attention modules through ablations.

Discussion

  • An attention-based CNN weights frequency-band features and brain-region maps, while a bidirectional LSTM with temporal attention weights all temporal slices rather than only its last output.Source: Discussion · p. 11
  • Removing spatial, spectral, and temporal attention decreases classification accuracy by 0.47%, 0.63%, and 1.19%, respectively; removing all three decreases it by 2.17%.Source: Discussion · p. 11

Conclusion

  • 4D-aNN takes spatial-spectral-temporal EEG representations as input, uses a CNN for spatial and spectral information, and applies a bidirectional LSTM to CNN outputs for temporal dependencies.Source: Conclusion · p. 11
  • On SEED, 4D-aNN performs better than all baselines.Source: Conclusion · p. 11
Loading 2101.05484v1…