PaperlayerView exact revision on arXiv

Source-linked AI summary

GANSER: A Self-supervised Data Augmentation Framework for EEG-based Emotion Recognition

Zhi Zhang, Sheng-hua Zhong, Yan Liu

arXiv:2109.03124v2cs.LGcs.AIcs.HC

TL;DR

Limited data makes it difficult to train accurate and stable deep models for EEG-based emotion recognition. GANSER combines adversarial training with self-supervised learning to generate EEG augmentations and guide classifier training. The authors report that it outperforms state-of-the-art methods on DEAP valence and arousal and improves classification performance on benchmark datasets.

  • Problem

    Limited training data makes it difficult to build effective deep-learning models for EEG-based emotion recognition.

    Source: §I · pp. 1–2

  • Method

    GANSER combines adversarial EEG generation with masking-based self-supervision, using simulated samples and masking probability to guide emotion-classifier learning.

    Source: §I · p. 2, §VI · p. 12

  • Results

    GANSER outperforms five state-of-the-art studies on DEAP for both valence and arousal.

    Source: B. Overall Performance on DEAP · p. 9

  • Takeaways & Limitations

    Experiments on DEAP and DREAMER show that the framework synthesizes high-quality, diverse EEG samples and improves classification performance.

    Source: §VI · p. 12

Abstract

from arXiv · show

The data scarcity problem in Electroencephalography (EEG) based affective computing results into difficulty in building an effective model with high accuracy and stability using machine learning algorithms especially deep learning models. Data augmentation has recently achieved considerable performance improvement for deep learning models: increased accuracy, stability, and reduced over-fitting. In this paper, we propose a novel data augmentation framework, namely Generative Adversarial Network-based Self-supervised Data Augmentation (GANSER). As the first to combine adversarial training with self-supervised learning for EEG-based emotion recognition, the proposed framework can generate high-quality and high-diversity simulated EEG samples. In particular, we utilize adversarial training to learn an EEG generator and force the generated EEG signals to approximate the distribution of real samples, ensuring the quality of augmented samples. A transformation function is employed to mask parts of EEG signals and force the generator to synthesize potential EEG signals based on the remaining parts, to produce a wide variety of samples. The masking possibility during transformation is introduced as prior knowledge to guide to extract distinguishable features for simulated EEG signals and generalize the classifier to the augmented sample space. Finally, extensive experiments demonstrate our proposed method can help emotion recognition for performance gain and achieve state-of-the-art results.

I. INTRODUCTION

EEG-based emotion recognition faces data scarcity alongside noisy, high-dimensional, non-stationary signals, making deep models vulnerable to overfitting and limited generalization. GANSER combines adversarial generation and self-supervised learning to create diverse EEG augmentations and use them to train emotion classifiers.

Motivation

  • EEG’s low signal-to-noise ratio, high feature dimension relative to trial counts, and non-stationarity complicate learning and cross-subject or cross-trial generalization.The paper notes that cognitive-neuroscience datasets often contain only hundreds to a few thousand trials despite many electrodes and seconds of sampling per trial.Source: §I · p. 1
  • With limited training data, complex deep models risk overfitting training-set quirks, which further limits their generalizability.The paper links this concern to EEG’s low initial ratio of samples to features.Source: §I · pp. 1–2

Contributions

  • GANSER combines adversarial training and self-supervision: masking prompts EEG synthesis from remaining signal parts, while masking probability guides feature extraction and classifier generalization.Adversarial training encourages generated signals to approximate the real-sample distribution; simulated signals are then used as augmented training samples.Source: §I · p. 2
  • The authors report that GANSER significantly outperforms existing state-of-the-art methods and assess generated-sample quality and diversity quantitatively and visually.The contribution summary describes the experiments as extensive but provides no metric values here.Source: §I · p. 2

II. RELATED WORK

Prior EEG emotion-recognition work uses deep networks to model signal patterns and connectivity, while augmentation studies include transformations, noise, and generative models. The paper identifies quality and diversity assessment of augmented EEG as an open challenge.

A. EEG-based Emotion Recognition

  • Deep EEG emotion-recognition studies use recurrent and convolutional networks for signal features, connectivity, or subject-related variation, including STRNN, CNN-RNN, 3D CNN, and GAN-based domain-shift work.Other cited approaches model brain connectivity or reduce subject-related information.Source: A. EEG-based Emotion Recognition · p. 2

B. Data Augmentation for EEG-based Emotion Recognition

  • EEG augmentation studies apply geometric distortions, Gaussian noise, and methods such as sliding windows, sampling, Fourier transforms, and segment recombination.The paper also reviews GAN-based approaches for EEG augmentation, including conditional WGAN and cBEGAN.Source: B. Data Augmentation for EEG-based… · pp. 2–3
  • Assessing augmented EEG quality and diversity remains an open challenge because, unlike images, signal samples are difficult to judge as still resembling their original class.Source: B. Data Augmentation for EEG-based… · p. 3

III. PROPOSED METHOD

GANSER uses an Adversarial Augmentation Network (AAN) to synthesize EEG samples and a Multi-factor Training Network (MTN) to guide classifier training on those samples. Its AAN combines masking, adversarial distribution matching, and architectures designed for EEG’s spatio-temporal structure.

A. Overall Framework

  • The overall framework feeds real EEG samples to AAN for augmentation, then trains an emotion classifier on the simulated samples under MTN guidance.Source: A. Overall Framework · p. 3

B. Adversarial Augmentation Network

  • Masking Transformation removes randomly selected EEG values so the GAN reconstructs missing parts from the remaining signal, enabling new samples.A larger τ masks more of the source signal and is used as an augmentation factor for sample diversity.Source: B. Adversarial Augmentation Network · pp. 3–4
  • The GAN learns to match generated EEG to the real-sample distribution, while masking supplies signal-level variation and adversarial training encourages realistic feature distributions.The generator and discriminator are optimized in turn; the discriminator distinguishes real from simulated distributions as the generator improves its EEG samples.Source: B. Adversarial Augmentation Network · p. 4
  • The generator uses an adapted UNet to synthesize missing values while preserving high-resolution EEG details through encoder-decoder skip connections.Channel Masking zeros locations without electrodes so the generator focuses on measured signal values rather than fitting nonexistent channels.Source: B. Adversarial Augmentation Network · p. 5
  • STNet serves as discriminator and combines convolutional, separable-convolution, and Inception components to model spatio-temporal and multi-scale EEG patterns.The separable layer captures spatial and temporal correlations, while the Inception block fuses features at different spatial scales.Source: B. Adversarial Augmentation Network · p. 5

C. Multi-factor Training Network

The Multi-factor Training Network uses augmentation-dependent confidence to supervise feature similarity between real and generated EEG, while retaining ordinary supervision on real samples. Its training pipeline regenerates diverse augmented samples rather than relying on a fixed saved set.

C. Multi-factor Training Network

  • Randomly synthesized EEG can approximate the real-sample distribution without a preset augmentation count, avoiding reliance on a fixed bank of samples.The generator is trained adversarially to produce varied samples that fit the real-sample feature distribution.Source: C. Multi-factor Training Network · p. 5
  • Because augmentation can move borderline valence or arousal samples across category thresholds, MTN supervises augmented samples using uncertainty labels.The paper notes that high and low valence or arousal are not cleanly separated by an artificial threshold.Source: C. Multi-factor Training Network · p. 5
  • MTN uses augmentation factor τ as surrogate confidence, weighting feature-distribution matching according to how much of the source EEG is preserved.Small τ preserves more source values and calls for high-confidence matching; large τ preserves less and calls for lower confidence.Source: C. Multi-factor Training Network · p. 6
  • The training loss combines cross-entropy on real EEG labels with confidence-weighted feature-distance constraints between real and generated samples.The classifier's feature extractor is used to measure the distance between each real EEG sample and its generated counterpart.Source: C. Multi-factor Training Network · p. 6
  • GANSER trains the classifier end to end with generated EEG paired with real samples, regenerating augmentations between epochs rather than saving a preset set.The paper states that this avoids saving and reloading samples and that regenerated samples vary between epochs.Source: C. Multi-factor Training Network · p. 6
  • The Multi-factor Self-supervised Learning loss adaptively guides distinguishable features for simulated EEG and generalizes the classifier to the augmented sample space.Its feature-matching weights vary with the surrogate confidence assigned to each augmentation.Source: C. Multi-factor Training Network · p. 6

IV. IMPLEMENTATION DETAILS

The implementation-details section covers the generator, discriminator, classifier architectures, and experimental hyperparameters.

IV. IMPLEMENTATION DETAILS

  • The section supplements GANSER's implementation details with network architectures for G, D, and C and the hyperparameters used in experiments.Source: §IV · p. 6

A. Network Architectures

GANSER uses an encoder-decoder generator and an STNet architecture for discrimination and emotion classification. The classifier adds classification-specific operations to the shared STNet design.

A. Network Architectures

  • The generator uses a contracting encoder path and a skip-connected expansive decoder path to synthesize EEG, then masks locations without electrodes to zero.The generator has seven convolutional layers.Source: A. Network Architectures · pp. 6–7, Figure 3 · p. 7
  • STNet serves as the discriminator, extracting EEG features with convolutional blocks, separable convolution, and an Inception block before predicting whether a sample is fake.Its Inception block combines 1 × 1, 3 × 3, and 5 × 5 convolutional layers.Source: Figure 4 · p. 7, A. Network Architectures · pp. 7–8
  • The classifier shares STNet's design but adds dropout with p = 0.5 and a two- or four-class softmax output.The output size corresponds to the emotion categories.Source: Figure 4 · p. 7, A. Network Architectures · p. 8

B. Other Configuration

The configuration specifies the implementation environment, optimizer settings, and two-stage training procedure.

B. Other Configuration

  • GANSER is implemented in PyTorch on eight NVIDIA Tesla V100 GPUs, with Adam, β1 = 0.9, β2 = 0.99, weight decay 0.0005, and batch size 64.The AAN is trained first to generate augmented EEG; the MTN then trains the emotion classifier.Source: B. Other Configuration · p. 8

V. EXPERIMENTS

The DEAP experiments use EEG recordings and participant ratings to create binary valence and arousal classification tasks. Preprocessing divides each trial into one-second samples and removes each epoch’s mean baseline.

A. Dataset and Preprocessing on DEAP

  • DEAP records 32-channel EEG and eight-channel peripheral signals from 32 subjects watching 40 one-minute music videos, with ratings for arousal, valence, liking, and dominance.Source: A. Dataset and Preprocessing on DEAP · p. 8
  • EEG is down-sampled to 128 Hz, band-pass filtered from 4 to 45 Hz, divided into non-overlapping one-second chunks, and mean-baseline corrected.The resulting 40 trials yield 2,400 EEG samples.Source: A. Dataset and Preprocessing on DEAP · p. 8
  • Median rating 5.0 divides valence and arousal into high (>5.0) and low (≤5.0) labels, making each emotion dimension a binary classification task.Source: A. Dataset and Preprocessing on DEAP · p. 8

B. Overall Performance on DEAP

On DEAP, GANSER outperforms the compared state-of-the-art methods on valence and arousal and exceeds GAN-based augmentation approaches on four-category classification. Ablations report gains from the GAN and masking transformation, with additional improvement from the multi-factor self-supervised loss.

B. Overall Performance on DEAP

  • Over 93% accuracy on both valence and arousal, with a margin near 1.0% over the second-best method, outperforms five state-of-the-art studies on DEAP.Table II reports average accuracy for valence, arousal, and four-category classification.Source: Table II · p. 9, B. Overall Performance on DEAP · p. 9
  • GANSER exceeds existing GAN-based augmentation frameworks by over 8.0% in four-category classification and correctly classifies nearly 90% of EEG signals across valence and arousal simultaneously.Source: B. Overall Performance on DEAP · p. 9

C. Ablation Study on DEAP

  • The classifier reaches 91.39% valence and 92.22% arousal accuracy without augmentation; adding GAN and masking transformation brings about a 1.0% gain, and MSL raises performance above 93.5%.The authors attribute the MSL improvement to accounting for augmented-label uncertainty and providing further training guidance.Source: C. Ablation Study on DEAP · p. 9, Table III · p. 9

D. Qualitative and Quantitative Experiments on DEAP

The paper assesses generated EEG quality with FSTD and visualizes real and synthesized signals. Its ablations indicate that each proposed generator component contributes to distributional similarity, while visualizations show similar spatio-temporal activation patterns with some variation.

D. Qualitative and Quantitative Experiments on DEAP

  • FSTD measures the Wasserstein distance between real and generated EEG feature distributions, and a smaller value indicates greater similarity.Features are extracted by STNet, and the distributions are summarized by their means and covariance matrices.Source: D. Qualitative and Quantitative… · p. 10
  • Removing any proposed generator component raises average FSTD, with STNet having the largest effect across valence and arousal.Channel masking particularly affects valence FSTD, while UNet helps preserve signal scale for arousal.Source: D. Qualitative and Quantitative… · p. 10
  • Topographic maps show real and generated signals with nearly consistent activation areas across space and time, alongside shifts in activation that indicate sample diversity.Maps show high and low energy in red and blue, respectively, at four time points.Source: D. Qualitative and Quantitative… · p. 10
  • Without the specially designed network, generated samples preserve limited spatio-temporal features and are difficult to distinguish across categories, consistent with possible mode collapse.Source: D. Qualitative and Quantitative… · p. 11, Figure 5 · p. 11

E. Dataset and Preprocessing on DREAMER

DREAMER contains EEG recordings and valence and arousal ratings from 23 subjects watching film clips. The experiments split signals into one-second samples, baseline-correct epochs, and classify each emotion dimension as high or low.

E. Dataset and Preprocessing on DREAMER

F. Overall Performance on DREAMER

On DREAMER, GANSER outperforms most compared methods on valence and arousal, and the conclusion reports improved classification performance on both DREAMER and DEAP. The framework combines adversarial EEG augmentation with masking-guided self-supervision; future work targets controllable semantic augmentation.

F. Overall Performance on DREAMER

VI. CONCLUSION AND FUTURE WORK

  • GANSER combines AAN, which synthesizes diverse EEG samples through masking and adversarial training, with MTN’s masking-prior-guided self-supervised feature learning.MTN uses the masking possibility as prior knowledge to guide feature extraction and generalize the classifier to the augmented sample space.Source: §VI · p. 12
  • Experiments on DEAP and DREAMER show that GANSER synthesizes high-quality, diverse EEG samples and improves classification performance.The conclusion attributes this synthesis to features of real EEG signals.Source: §VI · p. 12
  • Future work will explore semantic augmentation that can control environmental noise, artifacts, and simulated EEG features at valence or arousal dimensions.Source: §VI · p. 12

Related papers

Loading 2109.03124v2…