PaperlayerView exact revision on arXiv

Source-linked AI summary

Subject Independent Emotion Recognition using EEG Signals Employing Attention Driven Neural Networks

Arjun, Aniket Singh Rajpoot, Mahesh Raveendranatha Panicker

arXiv:2106.03461v3cs.NEeess.SP

TL;DR

Subject variability limits the generalization of EEG emotion-recognition methods across people. This paper learns subject-invariant representations with an unsupervised LSTM channel-attention autoencoder, then classifies them using an attention-based CNN. The authors report competitive subject-independent emotion-recognition results and state-of-the-art seizure-detection results on CHB-MIT.

  • Problem

    Subject variability challenges generalization across people, while large labeled medical EEG datasets are difficult to obtain.

    Source: §1 · p. 2

  • Method

    An unsupervised LSTM channel-attention autoencoder learns subject-invariant EEG representations, which an attention-based CNN uses for classification.

    Source: §1 · p. 3

  • Results

    The framework achieved competitive subject-independent emotion-recognition results on SEED and DEAP and state-of-the-art epileptic seizure-detection results on CHB-MIT.

    Source: §5 · pp. 11–12

  • Takeaways & Limitations

    The authors present the framework as a promising approach for diagnosing and helping people with epilepsy and Alzheimer’s, and apply it to emotion and seizure tasks.

    Source: §5 · p. 12

    The framework remains more complex than classical machine-learning approaches, resulting in higher training time.

    Source: §4 · p. 11

Abstract

from arXiv · show

In the recent past, deep learning-based approaches have significantly improved the classification accuracy when compared to classical signal processing and machine learning based frameworks. But most of them were subject-dependent studies which were not able to generalize on the subject-independent tasks due to the inter-subject variability present in EEG data. In this work, a novel deep learning framework capable of doing subject-independent emotion recognition is presented, consisting of two parts. First, an unsupervised Long Short-Term Memory (LSTM) with channel-attention autoencoder is proposed for getting a subject-invariant latent vector subspace i.e., intrinsic variables present in the EEG data of each individual. Secondly, a convolutional neural network (CNN) with attention framework is presented for performing the task of subject-independent emotion recognition on the encoded lower dimensional latent space representations obtained from the proposed LSTM with channel-attention autoencoder. With the attention mechanism, the proposed approach could highlight the significant time-segments of the EEG signal, which contributes to the emotion under consideration as validated by the results. The proposed approach has been validated using publicly available datasets for EEG signals such as DEAP dataset, SEED dataset and CHB-MIT dataset. The proposed end-to-end deep learning framework removes the requirement of different hand engineered features and provides a single comprehensive task agnostic EEG analysis tool capable of performing various kinds of EEG analysis on subject independent data.

1. Introduction

EEG offers a real-time physiological signal for emotion recognition, but subject variability and reliance on engineered features complicate subject-independent analysis. The paper proposes an unsupervised latent encoder followed by an attention-based classifier, and extends the framework to seizure detection and time-segment analysis.

1. Introduction

  • EEG captures brain state in real time, while prior approaches face handcrafted-feature and cross-subject generalization challenges.The paper motivates subject-independent methods and unsupervised learning partly by the difficulty of obtaining large labeled medical datasets.Source: §1 · pp. 1–2
  • An unsupervised LSTM with channel-attention autoencoder is proposed to learn lower-dimensional, subject-invariant EEG representations across individuals.The approach builds on the hypothesis that neural-network bases formed during similar emotions remain similar across individuals.Source: §1 · pp. 2–3
  • A CNN with attention classifies the encoded representations and identifies key time frames in EEG signals while learning under supervision.The contribution description characterizes time-frame identification as unsupervised.Source: §1 · p. 3
  • The framework is also tested for epileptic seizure detection on CHB-MIT to examine its use beyond emotion recognition.The paper additionally investigates temporal localization of emotional outbursts and common latent subspaces across subjects.Source: §1 · p. 3

2.1 Dataset

The study validates its approach on DEAP, SEED, and CHB-MIT, which provide emotion-recognition and seizure-detection EEG data. The datasets differ in participants, recording protocols, and labels.

2.1 Dataset

  • The approach was validated using DEAP, SEED, and CHB-MIT, with dataset specifications summarized in Table 2.The datasets cover emotion recognition and epileptic seizure detection.Source: §2.1 · p. 3

2.1.1 Data set 1: DEAP Dataset

  • DEAP includes recordings from 32 participants watching 40 one-minute music videos; this study uses binary valence and arousal labels split at 5.The 32-channel recordings were downsampled to 128 Hz, band-pass filtered to 4–45 Hz, and segmented into 60-second samples after removing the initial 3-second baseline.Source: §2.1.1 · p. 3

2.1.2 Data set 2: SEED Dataset

  • SEED contains EEG recordings from 15 participants who watched 15 Chinese movie clips, with positive and negative labels used in this study.Participants completed the experiment three times at approximately weekly intervals; recordings were downsampled to 200 Hz and band-pass filtered to 0–75 Hz.Source: §2.1.2 · p. 3

2.1.3 Data set 3: CHB-MIT Dataset

  • CHB-MIT contains EEG from 23 children with epilepsy; this study classifies pre-ictal versus ictal, inter-ictal versus ictal, and pre-ictal versus inter-ictal signals.The dataset provides expert-determined seizure start and end times; recordings were collected for up to several days after anti-seizure treatment withdrawal.Source: §2.1.3 · p. 4

2.2 Methods

The methods combine an unsupervised LSTM channel-attention autoencoder for subject-invariant EEG encodings with an attention-based CNN classifier. Attention is used in both encoding and classification.

2.2 Methods

  • The methods comprise an encoder, an attention mechanism, and a classifier for EEG analysis.The autoencoder supplies the low-dimensional representation used by the classifier.Source: §2.2 · p. 4, §2.2.2 · p. 4

2.2.1 Attention Mechanism

  • Attention computes scalar weights from hidden states, applies SoftMax across the input sequence, and combines the weighted states into a context vector.The paper integrates attention into both the autoencoder and classifier.Source: §2.2.1 · p. 4

2.2.2 Latent Representation of EEG Channels

  • The autoencoder learns an EEG feature space by encoding inputs into a latent representation and decoding that representation back toward the input.The latent representation is intended to be lower-dimensional and similar across subjects.Source: (1) Autoencoder · p. 4, §2.2.2 · p. 4
  • Channel attention is added to the LSTM autoencoder encoder to bring similar channel encodings closer in latent space and support reconstruction.It scales hidden states while preserving the timestep dimension; training uses noisy inputs and clean outputs, with AWGN set to unity signal-to-noise ratio.Source: (2) LSTM with Channel Attention… · pp. 4–5, Figure 2 · p. 5

2.2.3 Classifier

  • The classifier uses two convolutional blocks for nearby and long-range relationships, then highway gating and attention to summarize temporal features before class prediction.Instance normalization and PReLU were found to facilitate training best.Source: §2.2.3 · pp. 5–6, Figure 3 · p. 5, Figure 4 · p. 6

3. Experiment and Results

The experiments evaluate the framework on DEAP, SEED, and CHB-MIT, compare its performance with alternative approaches, and analyze the learned temporal, cross-subject, and channel representations.

3.1 Experimental Setup

  • Subject-independent accuracy was estimated by leaving one subject out for evaluation, training on the others, and averaging results across subjects.The encoded representations used 8 latent dimensions for DEAP and CHB-MIT and 16 for SEED before CNN-attention classification.Source: §3.1 · pp. 6–7

3.2.1 Data set 1: DEAP Dataset

  • On DEAP, the proposed LSTM channel-attention autoencoder and CNN-attention combination outperformed the other tested encoder-classifier frameworks.The CWT- or Mel-spectrogram-image approach with pretrained VGG-16 did not yield good results.Source: Table 3 · p. 7, §3.2.1 · p. 7

3.2.2 Data set 2: SEED Dataset

  • For SEED, recordings were restricted to the central 150 seconds, and sessions were evaluated separately before averaging; the proposed framework was reported as superior.The dataset recordings lasted approximately 200–240 seconds, and results were averaged across three sessions.Source: §3.2.2 · p. 7, Table 4 · p. 7

3.2.3 Data set 3: CHB-MIT Dataset

  • For CHB-MIT, the analysis used 20-second segments from pre-ictal, interictal, and ictal states, selected patients, and the intersection of their channels.The dataset contains long recordings and irregular channel counts; validation results are identified in Table 6.Source: §3.2.3 · p. 8, Table 6 · p. 8

3.3 Comparison with Approaches in Literature

  • The framework performs at par with recent state-of-the-art studies and achieves state-of-the-art performance on CHB-MIT epileptic seizure state detection.The authors attribute its performance to subject-invariant latent representations and attention-based identification of important time segments.Source: §3.3 · p. 8

3.4 Latent Time Analysis

  • After attention, DEAP activations concentrated in short time bursts, while CNN activations began at annotated ictal-state onset in CHB-MIT.The authors report that the classifier identifies important time segments while learning the classification task under supervision.Source: §3.4 · p. 8, Figure 5 · p. 9, Figure 6 · p. 10

3.6 Localization in Space (EEG Channels) Analysis

  • Latent dimensions generally preserved the correspondence between highly or weakly activated dimensions and raw EEG channels, but this needs validation on larger datasets and with other methods.The analysis also notes that different brain regions can be activated across individuals experiencing happiness.Source: §3.6 · pp. 10–11, Figure 8 · p. 11

3.5 Latent Dimension Analysis

  • The 8-dimensional LSTM-autoencoder representations formed closer cross-subject latent subspaces than PCA representations in the DEAP visualization.The authors interpret this pattern as capturing intrinsic EEG variables and learning subject-invariant representations.Source: Figure 7 · p. 10, §3.5 · p. 10

4 Discussion

The discussion frames the framework as a response to inter-subject variability and task-sensitive hand-engineered features, while noting training-time and explainability limits. It presents broader generalizability and application agnosticism as potential advantages, not established superiority over hand-engineered tools.

4 Discussion

  • The LSTM channel-attention autoencoder targets subject-invariant EEG encodings, which the CNN-attention classifier uses for emotion recognition and seizure detection; the framework also identifies important time segments.The authors report state-of-the-art subject-independent accuracy on CHB-MIT.Source: §4 · p. 11
  • Hand-engineered features may not transfer across tasks, and deep models can still struggle with leave-one-subject-out evaluation when each person's EEG forms a separate cluster.The proposed autoencoder is intended to address this inter-subject variance by learning a subject-invariant latent subspace.Source: §4 · p. 11
  • The models remain more complex than classical machine-learning approaches, resulting in higher training time.The authors state that GPU parallelization can reduce inference time relative to extracting non-parallelizable hand-engineered features.Source: §4 · p. 11
  • The authors say the approach does not outperform hand-engineered-feature tools, though it may offer greater generalizability and application agnosticism; explainability was incomplete without a custom-collected dataset.They identify source imaging, time localization, and neuronal dynamics as areas needing further explainability work.Source: §4 · p. 11

5 Conclusions

The conclusions report competitive subject-independent emotion-recognition results on SEED and DEAP and state-of-the-art results for seizure detection on CHB-MIT. They also summarize the framework's latent-representation and time-localization aims and provide authorship, conflict, data, and code information.

5 Conclusions

  • The work investigates autoencoder-based subject-invariant latent representations of multichannel EEG and uses those vectors for classification, with attention intended to track long-term dependencies.The conclusions also describe identifying important EEG time segments in an unsupervised manner.Source: §5 · pp. 11–12
  • The authors report competitive subject-independent emotion-recognition results on SEED and DEAP and state-of-the-art results for seizure detection on CHB-MIT.They also report key-time-frame identification and localization of emotional bursts in time.Source: §5 · pp. 11–12

CRediT Authorship Contribution Statement

  • Arjun and Aniket Singh Rajpoot are credited with conceptualization, data curation, methodology, software, and original-draft writing; Mahesh Raveendranatha Panicker is credited with conceptualization, supervision, and review and editing.Source: CRediT Authorship Contribution Statement · p. 12

Declaration of Competing Interest

Data Availability

  • The authors identify DEAP, SEED, and CHB-MIT as the three datasets supporting the project results.Source: Data Availability · p. 12

Supplementary Materials

A. More Encoder & Classifier Architecture’s on DEAP & SEED Dataset

The supplied supplementary table captions indicate additional encoder-and-classifier architecture validation results for DEAP and SEED; they do not provide cell values or identify a best-performing configuration.

A. More Encoder & Classifier Architecture’s on DEAP & SEED Dataset

  • Table S1 presents additional validation results for encoder and classifier architectures on DEAP.Source: Table S1 · p. 15
  • Together, Tables S1 and S2 cover additional architecture validation on both DEAP and SEED.Source: Table S1 · p. 15, Table S2 · p. 15
  • Table S2 presents additional validation results for encoder and classifier architectures on SEED.Source: Table S2 · p. 15

B. Results of Different Latent Dimensions Encoding on DEAP & SEED Dataset

The supplementary tables cover latent-dimension results for DEAP and SEED. For SEED, the authors select 16 dimensions because performance differences from 32 were small and the lower-dimensional model had less architectural complexity.

B. Results of Different Latent Dimensions Encoding on DEAP & SEED Dataset

  • Table S3 presents results for DEAP using different latent dimensions with 32 input EEG channels.Source: Table S3 · p. 15
  • Table S4 presents results for SEED using different latent dimensions with 62 input EEG channels.Source: Table S4 · p. 15
  • For SEED, 16 latent dimensions were selected over 32 because performance differed little and the 16-dimensional model had lower parameter complexity.Source: B. Results of Different Latent… · p. 15

C. Architecture Design of the Proposed Architecture • DEAP DATASET

The architecture specifications give dataset-specific trainable-parameter counts for the autoencoders and classifiers, and list tensor sizes across latent, convolutional, attention, dense, and flattened stages.

DEAP DATASET

  • For DEAP, the LSTM channel-attention autoencoder has 14K trainable parameters, and the CNN attention classifier has approximately 3.36M.The counts are reported in the corresponding architecture table captions.Source: Table S5 · p. 16, Table S6 · p. 16
  • Listed tensor sizes include 8064x8 latent encodings, 8064x32 reconstructed EEG signals, and dense features of 2016x256.The feature listings also include a flattened representation of 516,096.Source: C. Architecture Design of the Proposed… · p. 16

SEED DATASET

  • For SEED, the LSTM channel-attention autoencoder has 54K trainable parameters, and the CNN attention classifier has approximately 6.2M.The counts are reported in the corresponding architecture table captions.Source: Table S7 · p. 17, Table S8 · p. 17

DEAP DATASET

Loading 2106.03461v3…