PaperlayerView exact revision on arXiv

Source-linked AI summary

EEG based Emotion Recognition: A Tutorial and Review

Xiang Li, Yazhou Zhang, Prayag Tiwari, Dawei Song, Bin Hu, Meihong Yang, Zhigang Zhao, Neeraj Kumar, Pekka Marttinen

arXiv:2203.11279v1eess.SPcs.AIcs.LG

TL;DR

Existing EEG-based emotion-recognition reviews need updating because they are fragmented, technique-specific, and omit newer methodologies and holistic technical routes. This paper surveys recent representative work, provides a beginner-oriented tutorial, explains psychological and physiological foundations, organizes methods into technical routes, and discusses challenges and future investigations. It identifies data-constrained learning and related issues such as domain shift and limited labeled data as important directions for the field.

  • Problem

    Prior EEG-based emotion-recognition reviews are outdated or fragmented, omit systematic coverage of newer deep-learning approaches, and neglect holistic technical routes.

  • Method

    The paper combines a tutorial on psychological, physiological, and technical foundations with a review organized by technical routes and research motivations.

  • Results

    The review identifies deep learning as a trend and surveys domain adaptation, data-constrained learning, evaluation strategies, and potential large-scale pretrained EEG models.

  • Takeaways & Limitations

    The paper provides a starting guide for researchers and highlights domain shift, few-shot learning, data limitations, and multi-source modeling as future research directions.

  • Takeaways & Limitations

    EEG feature extraction can impose high overhead and depend on subtle parameter settings, while multi-source data increases inter-subject and inter-session discrepancies.

Abstract

from arXiv · show

Emotion recognition technology through analyzing the EEG signal is currently an essential concept in Artificial Intelligence and holds great potential in emotional health care, human-computer interaction, multimedia content recommendation, etc. Though there have been several works devoted to reviewing EEG-based emotion recognition, the content of these reviews needs to be updated. In addition, those works are either fragmented in content or only focus on specific techniques adopted in this area but neglect the holistic perspective of the entire technical routes. Hence, in this paper, we review from the perspective of researchers who try to take the first step on this topic. We review the recent representative works in the EEG-based emotion recognition research and provide a tutorial to guide the researchers to start from the beginning. The scientific basis of EEG-based emotion recognition in the psychological and physiological levels is introduced. Further, we categorize these reviewed works into different technical routes and illustrate the theoretical basis and the research motivation, which will help the readers better understand why those techniques are studied and employed. At last, existing challenges and future investigations are also discussed in this paper, which guides the researchers to decide potential future research directions.

1 INTRODUCTION

EEG-based emotion recognition is framed as a multidisciplinary pattern-recognition problem with broad applications, requiring quantified emotion labels, suitable data, preprocessing, and learned representations. This review updates fragmented prior surveys by providing a tutorial, organizing technical routes, and discussing emerging challenges and directions.

  • Motivation: EEG emotion recognition is challenging because models must identify latent emotion-related components in weak and noisy physiological signals.EEG recordings contain redundant noise and implicit relationships with cognitive processes.
  • Motivation: Emotion recognition supports affective computing and applications including human-computer interaction, driving assistance, affective-disorder diagnosis, and therapy.The field draws on computer science, engineering, psychology, neuroscience, and medical science.
  • Technical foundation: A recognition pipeline defines and quantifies the target, acquires valid data, preprocesses signals, and extracts or learns representative features.These steps support construction of pattern-recognition models and decision boundaries.
  • Review scope: The review adds recent deep-learning methodologies, categorizes studies by holistic technical routes, explains their theoretical motivations, and discusses domain shift, few-shot learning, and pretrained EEG models.It is explicitly designed to guide researchers beginning work in EEG-based emotion recognition.
  • Emotion quantification: Emotion can be represented as discrete basic states or continuous dimensions, with continuous models addressing vague boundaries, evolving states, and cross-cultural label differences.The Valence axis measures happy-to-sad degrees, while Arousal measures activated-to-calm states.

3 EEG PREPROCESSING AND FEATURE ENGINEERING

EEG preprocessing and feature engineering address noise and extract emotion-relevant representations before recognition modeling. The reviewed features span time-domain statistics, event-related potentials, and frequency-domain power measures.

  • Preprocessing: High-sampled EEG can support precise real-time recognition, but higher sampling rates may add noise and increase processing, feature-engineering, and training costs.Sampling choices therefore involve a resolution, noise, and computation trade-off.
  • Preprocessing: Preprocessing and feature engineering are needed because EEG contains bodily and environmental noise and implicit correlations with cognitive processes.Collected signals are generally not used directly for recognition models.
  • Data quality: Incomplete or highly noisy recordings caused by technical or personal issues should be abandoned rather than further processed and analyzed.Researchers may also select a subset of EEG channels when their specificity is established.
  • Time-domain features: Time-domain features include statistics such as mean, variance, skewness, kurtosis, peak-to-peak interval, and higher-order crossing counts.Event-related potentials are also used to reflect underlying cognitive processes.
  • Frequency-domain features: Frequency-domain features such as power spectral density quantify signal power within bands including Delta, Theta, Alpha, Beta, and Gamma.PSD is commonly obtained with fast Fourier transform or Welch’s method.

3.4 Time-Frequency Domain Composition Features

Time-frequency features address EEG's non-stationarity by jointly representing changing frequency components and temporal information. The review describes wavelet-based and other decomposition methods for extracting features from nonlinear, unsteady EEG signals.

  • Time-frequency analysis is needed because EEG frequency components change with neural and cognitive processes, making static time- or frequency-domain views incomplete.
  • Discrete wavelet transform decomposes EEG into approximation coefficients for high-scale, low-frequency components and detail coefficients for low-scale, high-frequency components.Approximation coefficients at one scale can be further decomposed into finer coefficients.
  • At a 128Hz sampling rate, EEG can be represented through seven decomposition levels and their corresponding frequency bands.
  • Wavelet energy and wavelet entropy features can be computed within each frequency range from the decomposition coefficients.The decomposition scale is indexed by j, while k denotes the number of wavelet coefficients.
  • EMD, MEMD, and VMD decompose multichannel EEG into intrinsic mode functions for extracting more representative nonlinear, unsteady-signal features.
  • Nonlinear-dynamical analysis complements time-frequency features because EEG exhibits properties of a nonlinear dynamic system and has multifractal dimensions.Fractal dimension describes the complexity and self-similarity of a chaotic nonlinear system.

3.6 Asymmetry Features

Asymmetry features represent differences between corresponding activity measures across the brain's hemispheres. Brain-network features extend this view by modeling relationships among distributed regions and channels for emotion classification.

  • Emotion-related asymmetry features draw on lateralization of brain activity observed in emotion processing and mood disorders.
  • Spectral power asymmetry can be extracted from Theta, Alpha, Beta, and Gamma bands across fourteen left-right electrode pairs.
  • Asymmetry indices are generally computed as differences or ratios between corresponding measures from two signal sources.The compared measures can include power spectrum and fractal dimension.
  • Functional connectivity graphs and their structural characteristics model correlations among brain regions that may distinguish emotional states.
  • Brain networks are estimated from multichannel time correlations or spectral coherence using measures including Pearson correlation, mutual information, PLI, PLV, and Granger causality.PLV ranges from 0 to 1, representing the degree of phase coupling between two EEG channels.
  • Handcrafted EEG features require substantial domain knowledge, have uncertain relationships with emotional states, and can be disturbed by physiological artifacts such as cardiac activity and eye movement.

3.8 Feature Processing

Feature processing selects informative EEG representations while balancing computational cost, model dependence, and interpretability. The review distinguishes filter and wrapper selection methods and cautions that recognition-oriented importance does not automatically support cognitive interpretation.

  • Automatic feature selection ranks EEG features by target relationship or model-derived importance and retains the most vital features for model design.
  • Filter methods do not depend on a recognition model and usually cost less computationally than wrapper methods, making them suitable for real-time and big-data scenarios.Examples include chi-squared tests, mutual information, and ANOVA F-tests.
  • Chi-squared selection retains features closely related to emotion classes, while mutual-information methods evaluate feature-emotion interdependence.MRMR is identified as a representative mutual-information-based approach.
  • Wrapper methods operate with a specific machine-learning model; recursive feature elimination iteratively removes features with smaller weights.
  • Feature importance depends on the research objective: selected features may improve recognition but be unsuitable for interpreting cognitive phenomena.Correlated features can receive low weights despite preserving information relevant to emotional cognition.
  • EEG features vary over short periods while emotions may remain relatively stable, and adding features can improve accuracy while increasing noise and computational cost.

4 PATTERN RECOGNITION TECHNICAL ROUTES APPLIED IN THE FIELD

The review organizes EEG emotion-recognition research as pattern recognition and summarizes its principal technical routes in a flow chart. Deep-learning-based routes are presented as an emerging direction alongside established approaches.

  • EEG emotion recognition follows pattern-recognition steps that classify target samples using existing data and measurement criteria.
  • Figure 7 divides existing pattern-recognition approaches into distinct technical routes used in related EEG emotion-recognition studies.
  • Deep-learning-based technical routes have emerged with advances in graph and image processing and natural language processing.

4.1 Route: 0→1→2→4→11

The simplest route classifies an emotional state by thresholding a specific EEG feature. Because the threshold is fixed per subject and experience-dependent, this route lacks adaptability and is not mainstream.

  • A feature value exceeding a fixed threshold determines whether a sample belongs to a particular emotional state.
  • Subject-specific thresholds depend on experience, limiting the route’s adaptability.
  • The review treats threshold-based recognition as non-mainstream and does not describe it extensively.

4.2 Route: 0→1→2→5→11

This route applies traditional machine-learning models to EEG representations for emotion recognition. The review identifies supervised and unsupervised approaches, while highlighting gcForest’s robustness and adaptive complexity.

  • Traditional supervised models include discriminant analysis, SVM, KNN, Bayesian methods, random forests, and perceptrons.
  • Unsupervised approaches include manifold learning and clustering.
  • Figure 7 summarizes the pattern-recognition technical routes used in EEG-based emotion recognition.
  • gcForest uses fewer hyperparameters, remains robust to their settings, and adapts model complexity to dataset size.

4.3 Route: 0→1→2→3→11

This route uses handcrafted EEG features or learned representations with deep models, increasingly incorporating channel relationships, graph structure, spatial asymmetry, and adaptive connectivity.

  • Deep models can act as classifiers over handcrafted EEG features while learning nonlinear representations through representation learning and universal approximation.
  • CNN-based approaches address inter-channel information by converting EEG into two-dimensional representations suitable for convolution.
  • CapsNet is motivated by CNN limitations in recognizing differently oriented features and preserving precise spatial correlations.
  • Graph-based methods model relationships among EEG channels and functional brain networks beyond spatial proximity.
  • Dynamic GCNN training adaptively updates the adjacency matrix, and DE-feature graphs achieved the best reported performance among evaluated feature graphs.
  • VPR uses random walks and LSTM encoding to identify salient electrode pathways, while BiHDM uses four RNNs to model left-right hemispheric information.

4.4 Route: 0→1→3→11 and 0→3→11

This route emphasizes end-to-end learning from raw EEG and hybrid spatial-temporal modeling. The reviewed approaches also expose practical limits involving handcrafted preprocessing, computation, short windows, fluctuating labels, and two-stage pipelines.

  • End-to-end models learn representations directly from raw EEG, addressing information loss associated with handcrafted feature maps.
  • EEGNet uses depthwise and separable convolutions in an EEG-specific ConvNet architecture.
  • Short input segments limit long-trial dependency modeling, while trial-level labels may overlook continuously evolving emotional states.
  • RNNs model sequences, whereas CNN-RNN hybrids learn spatial and temporal representations either sequentially in cascade or simultaneously in parallel.
  • Three-dimensional models transform channel-time frames into channel-topology-time cubes and can produce time-varying emotion predictions.
  • The cascade and parallel hybrid models consistently outperform the state-of-the-art methods reported in the review.
  • Latent-source-factor plus LSTM systems may lose information through sequence sampling and remain inconvenient two-stage pipelines.
  • RNN approaches can be more computationally intensive than CNNs, particularly for long sequences, because backpropagation through time is memory-intensive.

4.5 Route: 3→7→11 and 5→7→11

User-independent EEG emotion recognition is challenged by domain shifts across users and time, motivating calibration, factor-specific modeling, transfer learning, and neural adaptation strategies. Reviewed approaches include source selection, fine-tuning, adversarial alignment, knowledge distillation, and meta learning.

  • Motivation: User-independent recognition is emphasized because subject-dependent models may not satisfy practical application requirements.The review identifies user-independent modeling as a practical focus despite degraded performance from distribution differences.
  • Motivation: Domain shift arises from inter-subject neurophysiological differences and within-source non-stationarity across epochs.Participant characteristics and changing mental or technical conditions can alter EEG distributions.
  • Calibration and factor-specific modeling: Calibration reduces participant differences by using baseline physiological characteristics, while aspect-oriented models specialize for factors such as gender or culture.Gender- and culture-specific models improved recognition accuracy relative to general models in the cited experiments.
  • Calibration and factor-specific modeling: Stratified normalization significantly outperforms batch-normalization approaches in cross-subject emotion recognition settings.The passage reports a comparative result without specifying the numerical performance values.
  • Transfer learning: Transfer learning addresses domain shift when source and target domains share feature spaces and task types, including through fine-tuning source models on target data.Deep-learning transfer approaches may fine-tune the whole network or only part of its structure.
  • Neural domain adaptation: Domain-adaptation networks align domains using adversarial feature extraction, Wasserstein objectives, or MK-MMD discrepancy measurements.DANNs seek domain-invariant features, while WGANDA transfers differential-entropy characteristics into a common space.
  • Neural domain adaptation: MCD addresses class boundaries but may eliminate useful features when source and target domains are substantially dissimilar.This limitation motivates methods intended to overcome weaknesses of both DANN and MCD.
  • Distillation and meta learning: Knowledge distillation transfers representations from a large cross-subject teacher to a lighter student using capsule-based architectures, while meta learning supports cross-subject adaptation.The reviewed meta-learning methods include interactions between base and meta learners and adaptation with limited target data.

4.6 Route: 3→8→11 and 5→8→11

Ensemble learning combines multiple models or weak classifiers to improve EEG emotion-recognition performance and generalization. AdaBoost exemplifies weighted combination of weak classifiers into a strong classifier.

  • Ensemble learning: Ensemble learning integrates multiple models through Bagging, Boosting, Stacking, and Voting strategies.The reviewed approaches exploit multiple decision boundaries for EEG-based emotion recognition.
  • AdaBoost: AdaBoost combines weak classifiers according to their weights to form one strong classifier.The stated goals are improved recognition performance, better generalization, and reduced over-fitting.

4.7 Route: 0→1→2→6→11

Fuzzy Logic provides an approximate-reasoning route for EEG emotion recognition when signals and labels are noisy, imprecise, or uncertain. Fuzzy clustering assigns samples graded membership across multiple categories.

  • Motivation: Noisy, imprecise, and uncertain EEG signals or labels create precise-modeling problems.Fuzzy Logic is introduced as a foundation for approximate reasoning based on fuzzy set theory.
  • Fuzzy clustering: Fuzzy clustering assigns samples to multiple categories with degrees of membership rather than a single categorical assignment.Fuzzy C-Means and Fuzzy k-Means are identified as representative methods.

4.8 Route: 3→9→11

Deep learning is a prevailing approach in EEG-based emotion recognition, but its parameter requirements make adequate, high-quality labeled data a central challenge. Data augmentation and few-shot learning are reviewed as routes for data-constrained learning.

  • Challenge: Deep learning is described as the trend in EEG-based emotion recognition.The review contrasts its growing use with the need for sufficient labeled data to optimize larger models.
  • Challenge: Deep neural networks require adequate, high-quality training data because they contain more parameters than shallow models.The paper identifies data-constrained learning as a promising route for addressing this limitation.
  • Data-constrained learning: Data augmentation generates new samples and more variable representations, supporting complex-model training and reducing over-fitting.The passage frames augmentation as a way to improve robustness to transformations likely in real applications.
  • Data-constrained learning: Few-shot learning builds models that generalize with limited samples and without retraining on data from new classes.It is also described as an N-shot-K-way learning problem.

4.9 Route: 3→10→11 and 5→10→11

This section presents AutoML as a route for automatically constructing EEG emotion-recognition models, including model engineering, selection, and hyperparameter optimization. It also situates few-shot learning as a paradigm that maps query embeddings using support-set samples.

  • Route: 3→10→11 and 5→10→11: AutoML aims to generate models with strong classification performance while minimizing generalization error for a specific EEG emotion-recognition problem.It addresses model engineering, model selection, model building, and hyperparameter optimization.
  • Route: 3→10→11 and 5→10→11: The general few-shot learning paradigm generates embeddings for support and query sets, then maps query embeddings using support samples.
  • Route: 3→10→11 and 5→10→11: Firefly integrated optimization simultaneously performs automatic parameter optimization, feature selection, and classifier selection.

5 PERFORMANCE EVALUATION

Performance evaluation in EEG emotion recognition depends on benchmark datasets, ground-truth ratings or labels, evaluation metrics, and data-splitting strategies aligned with the research objective. The reviewed results span subject-dependent and subject-independent settings across several datasets and emotion-class configurations.

  • 5 PERFORMANCE EVALUATION: Recognition methods are evaluated on benchmark datasets by comparing predicted emotion ratings or labels with ground truth.DEAP and SEED are among the benchmark datasets discussed, with DEAP widely used and cited.
  • 5 PERFORMANCE EVALUATION: Classification performance can be compared using Precision, Sensitivity, Specificity, and F-score derived from a confusion matrix.
  • 5 PERFORMANCE EVALUATION: Regression performance is commonly assessed with mean squared error, mean absolute error, and coefficient of determination R^2.
  • 5.3 Data Split and Validation Strategy: Data-splitting strategies should reflect whether the objective is subject-dependent or subject-independent modeling, because the two settings test different generalization conditions.The review recommends leave-K-trial-out and leave-K-subject-trials-out strategies without segmenting original trial data.
  • 5 PERFORMANCE EVALUATION: Representative results include 96.02% on SEED three-class recognition and 84.92% on SEED-IV four-class recognition for SST-EmotionNet.
  • 5 PERFORMANCE EVALUATION: The review reports subject-independent results including 86.7% on SEED three-class modeling, 88.33% for MCD_DA, and 84.23% in another three-class setting.
  • 6 CONCLUSION AND DISCUSSION: The review identifies effective EEG representation as an open challenge because feature extraction can be costly, parameter-sensitive, and redundant despite retaining only a few critical features.It also points toward large-scale pre-trained EEG models while noting differences across acquisition devices and sources.
Loading 2203.11279v1…