PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Augmentation for Enhancing EEG-based Emotion Recognition with Deep Generative Models

Yun Luo, Li-Zhen Zhu, Zi-Yu Wan, Bao-Liang Lu

arXiv:2006.05331v2eess.SPcs.LG

TL;DR

EEG emotion recognition lacks abundant labeled training data, making accurate affective-model development difficult. The paper proposes three VAE- and GAN-based augmentation methods with full or selective sample use, and reports improved recognition performance on SEED and DEAP compared with existing augmentation methods.

  • Problem

    Scarce EEG training data make it difficult to build accurate emotion-recognition models.

  • Method

    The paper generates PSD and DE EEG features with cWGAN, sVAE, and sWGAN, then augments training sets using all or selected generated samples.

  • Results

    Augmented training datasets improved EEG emotion-recognition performance on SEED and DEAP and outperformed conditional VAE, Gaussian noise, and rotational augmentation.

  • Takeaways & Limitations

    Deep generative augmentation can enhance EEG-based emotion-recognition models across datasets, features, and classifiers within the study’s evaluation.

Abstract

from arXiv · show

The data scarcity problem in emotion recognition from electroencephalography (EEG) leads to difficulty in building an affective model with high accuracy using machine learning algorithms or deep neural networks. Inspired by emerging deep generative models, we propose three methods for augmenting EEG training data to enhance the performance of emotion recognition models. Our proposed methods are based on two deep generative models, variational autoencoder (VAE) and generative adversarial network (GAN), and two data augmentation strategies. For the full usage strategy, all of the generated data are augmented to the training dataset without judging the quality of the generated data, while for partial usage, only high-quality data are selected and appended to the training dataset. These three methods are called conditional Wasserstein GAN (cWGAN), selective VAE (sVAE), and selective WGAN (sWGAN). To evaluate the effectiveness of these methods, we perform a systematic experimental study on two public EEG datasets for emotion recognition, namely, SEED and DEAP. We first generate realistic-like EEG training data in two forms: power spectral density and differential entropy. Then, we augment the original training datasets with a different number of generated realistic-like EEG data. Finally, we train support vector machines and deep neural networks with shortcut layers to build affective models using the original and augmented training datasets. The experimental results demonstrate that the augmented training datasets produced by our methods enhance the performance of EEG-based emotion recognition models and outperform the existing data augmentation methods such as conditional VAE, Gaussian noise, and rotational data augmentation.

I. INTRODUCTION

EEG emotion recognition is constrained by scarce, costly, noisy labeled data, motivating deep-generative augmentation. The paper proposes three methods that generate EEG features and evaluates them across datasets, classifiers, and augmentation strategies.

  • EEG data acquisition is costly, time-consuming, uncomfortable for subjects, and affected by noise and artifacts, limiting large-scale labeled datasets.
  • The paper introduces cWGAN, sVAE, and sWGAN, combining VAE or WGAN generators with full-use or selective-use augmentation strategies.cWGAN appends all generated samples, whereas sVAE and sWGAN select high-confidence samples using SVM classifiers.
  • The framework extracts DE and PSD features from SEED and DEAP, augments the original training data, and evaluates SVMs and shortcut-layer DNNs.
  • The study systematically compares features, generative methods, classifiers, and conventional augmentation methods using two EEG datasets.The comparison includes conditional VAE, Gaussian noise, and rotational data augmentation, with 5-fold cross-validation described in the experimental setup.

II. RELATED WORK

Related work covers EEG emotion-recognition datasets and deep generative models, emphasizing VAE and GAN as mechanisms for learning data distributions and generating new samples. It also notes GAN training challenges, especially instability and mode collapse.

  • EEG emotion-recognition research uses datasets such as DEAP and SEED, created from music videos and movie clips to elicit labeled emotional responses.
  • VAEs learn latent representations through encoder-decoder models and can generate realistic-like data from learned distributions.
  • GANs use a generator and discriminator in an adversarial process, while conditional variants add labels to control generated-data categories.
  • GANs have demonstrated strong generation ability but remain affected by nonconvergence, mode collapse, diminished gradients, and training instability.

C. Data Augmentation

Data augmentation expands datasets through label-preserving transformations, while this work applies generative models to EEG features and classifiers. The section also explains VAE training through reconstruction and latent-distribution regularization.

  • Data augmentation generates new labeled samples through transformations that preserve labels and can reduce overfitting while improving classification performance.
  • The paper uses GAN-based augmentation and shortcut-layer DNN classifiers alongside conventional approaches for EEG emotion recognition.
  • A VAE combines an encoder and decoder to map data into a latent representation and reconstruct data from that representation.
  • The VAE objective balances expected log-likelihood, which promotes reconstruction, against KL divergence, which regularizes latent representations and maintains their diversity.

B. WGAN

WGAN trains a generator against a discriminator to approximate real data distributions, while Wasserstein distance and gradient penalties address instability and gradient problems.

  • B. WGAN: A generator produces synthetic data from noise, while a discriminator distinguishes real distribution Xr from generated distribution Xg.
  • B. WGAN: The adversarial game reaches equilibrium when Xr = Xg, corresponding to matching real and generated distributions.
  • B. WGAN: WGAN replaces Jensen-Shannon divergence with Wasserstein distance, which provides useful and smooth gradients even when distributions do not overlap.
  • B. WGAN: The practical implementation replaces the 1-Lipschitz function with discriminator D and enforces ||D||L ≤ 1.
  • B. WGAN: Gradient penalty adds a loss term to enforce the Lipschitz constraint, whereas weight clipping can reduce model capacity and impair convergence.
  • B. WGAN: The interpolation points used for gradient penalty are defined as x̂ = αxr + (1 −α)xg, with α sampled uniformly from [0, 1].

C. cWGAN

cWGAN applies gradient-penalty WGAN to EEG emotion recognition and conditions generation on specified categories.

  • C. cWGAN: cWGAN generates EEG data with specified categories for emotion recognition using the gradient-penalty version of WGAN.
  • C. cWGAN: The gradient penalty stabilizes cWGAN training by enforcing the Lipschitz constraint and is reported to support high-quality generation and quick convergence.
  • C. cWGAN: The cWGAN objective uses the real-data category distribution Yr to define conditional discriminator and generator terms.

D. sVAE and sWGAN

sVAE and sWGAN enlarge EEG training datasets by retaining only generated samples judged high quality through classifier confidence.

  • D. sVAE and sWGAN: sVAE and sWGAN generate samples with VAE or WGAN, then select high-quality samples before adding them to the training dataset.
  • D. sVAE and sWGAN: A classifier trained on the original dataset evaluates generated samples, and only samples with confidence above a threshold are appended.
  • D. sVAE and sWGAN: The appended dataset is used to train a new classifier, and generation and selection are repeated until enough generated data are obtained.

E. cVAE

cVAE controls generated EEG categories by conditioning its encoder and decoder on labels, while conventional augmentation methods include Gaussian noise and electrode-coordinate rotations.

  • E. cVAE: cVAE controls the generated category by adding the corresponding label to both the encoder and decoder.
  • E. cVAE: The encoder receives a training point and its label, while the decoder generates data with a specified label from Gaussian noise.
  • E. cVAE: Gaussian-noise augmentation creates additional data by adding noise to original training samples, with µ = 0 and σ = 0.001 in the experiment.
  • E. cVAE: Rotational data augmentation addresses electrode-position shifts by rotating three coordinates to create data associated with new electrode positions.

H. Classifier

The study uses SVMs and deep neural networks with randomly inserted residual shortcut functions to evaluate augmented EEG emotion-recognition data.

  • SVMs and deep neural networks are the two classifiers used to build affective models.
  • The deep networks contain residual functions that use shortcuts to jump over layers and mitigate vanishing gradients.
  • Linear projections match input and output dimensions when residual blocks connect layers with different node counts.
  • The residual mapping is F(x) = W2σ(W1x), followed by an additional ReLU before the next layer.
  • The sVAE and sWGAN workflow repeatedly trains a classifier, filters generated samples by confidence threshold, and returns the selected set.

A. Dataset Description

Experiments use the SEED and DEAP EEG datasets, extract PSD and DE features across frequency bands, and evaluate recognition with five-fold cross-validation.

  • SEED contains EEG recordings from 15 participants viewing 15 clips designed to elicit positive, neutral, or negative emotions.
  • DEAP contains EEG and peripheral physiological recordings from 32 participants watching 40 one-minute music-video excerpts rated on multiple affective dimensions.
  • The study generates differential entropy and power spectral density features, using STFT to extract PSD from preprocessed EEG.
  • A linear dynamic system filters both PSD and DE features to account for the dynamic characteristics of EEG emotion-recognition tasks.
  • SEED features have 310 dimensions across five bands and 3,394 labeled samples, whereas DEAP features have 128 dimensions across four bands and 2,400 samples.
  • Each experiment uses five-fold cross-validation, averaging five model accuracies and then averaging across experiments for final accuracy.

D. Hyperparameter Details

The experiments tune classifier hyperparameters and compare augmentation methods across datasets, features, classifiers, and augmentation sizes, with selective WGAN generally achieving the strongest reported performance.

  • The SVM uses a linear kernel with c searched from 2^-10 to 2^10, while DNN learning rates, depth, batch size, and residual connections are randomly searched.
  • sWGAN achieves the best reported SEED PSD SVM accuracy of 67.7% with 20,000 appended samples, exceeding cVAE, Gaussian noise, and RDA.
  • For SEED DE features, sWGAN reaches 90.8% with SVM and 93.5% with the shortcut DNN, outperforming conventional augmentation methods.
  • On DEAP DE, sWGAN reaches 50.8% with SVM, while sVAE reaches 49.3% with DNN; both outperform the reported conventional-method results.
  • 10.2% mean-accuracy improvement is reported for SEED DE features using sWGAN with the DNN classifier, while DEAP’s highest improvement is 5.4% using sWGAN with SVM.
  • All augmentation methods peak and then decline as more samples are appended, although most proposed-method experiments remain above baseline below 20,000 generated samples.

D. Visualization of the Generated Data

Visualizations indicate that generated differential-entropy features resemble real EEG data, cluster by emotion, and supplement sparse class manifolds, while excessive generation can introduce misleading samples.

  • Generated data had similar scalp distributions to real data, including emotion-specific beta, gamma, and alpha response patterns.
  • t-SNE showed emotion-specific clusters, with generated points close to corresponding real data, indicating realistic information suitable for augmentation.
  • Generated samples supplemented sparse data manifolds and improved classifier margins by adding realistic yet diverse information.
  • Increasing the number of generated samples raised the likelihood of bad-quality examples that could mislead classifiers and reduce accuracy.

E. Discussions on Different Affective Models

Across datasets, features, and classifiers, the proposed augmentation methods improved emotion-recognition performance, with differences in stability, accuracy, and computational time among methods.

  • The proposed methods improved performance across different datasets, features, and classifiers, demonstrating generalization and effectiveness.
  • sVAE provided better stability than cWGAN and sWGAN, while VAE was described as more stable than adversarial training.
  • sWGAN generally achieved higher classification accuracy than sVAE and consistently outperformed cWGAN on both datasets.
  • cWGAN converged faster than the selective methods because it used all generated data without quality selection.
  • Accuracy improvements reached 10.2% on SEED and 5.4% on DEAP when models were trained with augmented datasets.
Loading 2006.05331v2…