Source-linked AI summary
Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks
Pouya Bashivan, Irina Rish, Mohammed Yeasin, Noel Codella
TL;DR
EEG cognitive-event modeling needs representations that remain robust to inter- and intra-subject differences and measurement noise. The paper converts multichannel EEG into topology-preserving multispectral image sequences and applies recurrent-convolutional networks. On mental-load classification, the approach significantly improved classification accuracy over state-of-the-art results.
Problem
EEG cognitive-event modeling requires representations invariant to inter- and intra-subject differences and inherent measurement noise.
Method
The approach transforms multichannel EEG into topology-preserving multispectral image sequences, then uses ConvNets and LSTMs to learn spatial, spectral, and temporal representations.
Results
Significant improvements in classification accuracy over state-of-the-art results were demonstrated on the mental-load classification task.
Takeaways & Limitations
The image-sequence representation can be applied across EEG hardware setups and may support combining datasets after comparable preprocessing.
Takeaways & Limitations
The approach reduces required data by manually extracting power features rather than operating directly on raw EEG time series.
Abstract
from arXiv · showhide
One of the challenges in modeling cognitive events from electroencephalogram (EEG) data is finding representations that are invariant to inter- and intra-subject differences, as well as to inherent noise associated with such data. Herein, we propose a novel approach for learning such representations from multi-channel EEG time-series, and demonstrate its advantages in the context of mental load classification task. First, we transform EEG activities into a sequence of topology-preserving multi-spectral images, as opposed to standard EEG analysis techniques that ignore such spatial information. Next, we train a deep recurrent-convolutional network inspired by state-of-the-art video classification to learn robust representations from the sequence of images. The proposed approach is designed to preserve the spatial, spectral, and temporal structure of EEG which leads to finding features that are less sensitive to variations and distortions within each dimension. Empirical evaluation on the cognitive load classification task demonstrated significant improvements in classification accuracy over current state-of-the-art approaches in this field.
1 INTRODUCTION
Deep neural networks remain relatively underexplored for neuroimaging because datasets are often too small for large-scale models. EEG recognition also requires representations robust to subject-related and signal-acquisition variations, motivating a structure-preserving recurrent-convolutional approach.
- Research gap: Neuroimaging datasets often contain too few samples to adequately train deep networks with millions of parameters.The advantages of deep networks over traditional machine-learning methods are more apparent with very large datasets.
- Related work: Prior work used deep belief, convolutional, and recurrent networks to learn representations from fMRI and EEG, showing potential benefits despite moderate dataset sizes.Reported work includes supervised pretraining and representation extraction from EEG time series.
- Research gap: EEG mental-state recognition traditionally combines manually selected features from continuous time series with supervised learning.This approach seeks a discriminative manifold between mental states.
- Challenge: EEG recognition must handle translations and deformations across spatial, frequency, and temporal dimensions caused by subject differences and acquisition protocols.These variations can arise from individual differences in cortical mapping or functioning.
- Approach: The proposed approach converts low-level EEG features into topology-preserving multispectral image sequences and processes them with recurrent-convolutional networks.ConvNets extract spatial and spectral representations from frames, while recurrent processing captures temporal patterns.
2 OUR APPROACH
The approach preserves EEG’s spatial, spectral, and temporal structure by converting multi-channel recordings into topology-preserving multi-spectral image sequences and processing them with recurrent-convolutional networks.
- Making images from EEG time-series: Spectral power from EEG time series is mapped into 2-D topographical images, with multiple color channels representing frequency bands and consecutive images representing temporal evolution.The method preserves spatial, frequency, and time structure rather than aggregating measurements into a feature vector.
- Making images from EEG time-series: Azimuthal Equidistant Projection maps three-dimensional scalp electrode locations onto a 2-D surface while preserving distances from the projection center.The projection is applied because electrode locations are spatially distributed over the scalp and the transformation should preserve neighboring relationships.
- Multi-frame approach: The multi-frame approach divides each trial into 0.5-second windows, produces seven frames, and feeds their sequence into a recurrent-convolutional network.Temporal modeling is motivated by the dynamic nature of brain activity and implemented with recurrent or temporal-convolutional layers.
- Single-frame approach: The single-frame approach constructs one image from spectral measurements over the complete 3.5-second trial and evaluates ConvNet configurations with varying depths.The configurations use stacked convolutional and max-pooling layers, followed by a fully connected layer and softmax output.
- Architecture: The architecture applies a shared ConvNet to each frame, then aggregates frame representations using temporal max-pooling, temporal convolution, or LSTM layers.The evaluated alternatives are illustrated as multi-frame architectures combining ConvNets with temporal operators and classification layers.
- Multi-frame approach: LSTM models sequence inputs by iteratively computing hidden and output vectors, using memory cells and gated inputs and outputs to capture long-term dependencies.The best tested LSTM configuration used one layer with 128 memory cells, with only the prediction after the complete frame sequence passed to the fully connected layer.
3 BASELINE METHODS
The study compares EEG classification against SVM, random forest, sparse logistic regression, and a three-layer deep belief network baseline.
- Support-Vector Machine: SVM hyperparameters C and γ were selected by grid search with cross-validation on the training set.The search used C values from 0.01 to 100 and γ values from 0.1 to 10.
- Random Forest: Random forest was evaluated as an ensemble of independent random decision trees using varied estimator counts.The number of estimators ranged from 5 to 1000.
- Logistic Regression: Sparse logistic regression used l1-regularization, with its regularization parameter selected through training-set cross-validation.The searched logarithmic range for C was [10^-2, 10^3].
- Deep Belief Network: The deep belief network used one Gaussian-Binary restricted Boltzmann machine followed by two Binary restricted Boltzmann machines.Its final representation was connected to a two-way softmax prediction layer.
- Deep Belief Network: The deep belief network used layer sizes 512, 512, and 128, then fine-tuned with batch stochastic gradient descent and l1-regularization.The softmax layer had four units for the four-class prediction task.
4 EXPERIMENTS ON AN EEG DATASET
The experiments used a working-memory task in which participants retained character arrays and judged whether a later test character matched the memorized set. EEG trials were collected from participants and evaluated with leave-subject-out testing.
- Working-memory task: Working memory was studied as transient information retention that constrains performance across cognitive functions.The experiment used increasing cognitive load to probe working-memory capacity.
- Dataset: EEG was recorded from 15 participants using 64 scalp electrodes sampled at 500 Hz, with two subjects excluded for excessive noise.The electrodes followed standard 10-10 locations.
- Working-memory task: Participants viewed a character SET for 500 ms, maintained it for three seconds, and pressed a button when the TEST character matched a set item.The diagram describes the working-memory trial sequence.
- Evaluation: Performance evaluation used leave-subject-out cross-validation, with each fold testing all trials from one subject.Only correctly responded trials were retained, leaving 2670 trials from 13 subjects.
5 RESULTS
Results compare single-frame ConvNets with multi-frame recurrent-convolutional models, examining architecture, projection, temporal aggregation, and learned representations. Temporal convolution and LSTM improved the proposed model, while temporal max-pooling and unresolved spectral-temporal modeling remained limitations.
- 5.1 Single-frame classification: The single-frame analysis compared power-feature images and ConvNet configurations, including VGG-style architectures with different convolution and max-pooling depths.The analysis sought the best architecture for images generated from complete EEG trials.
- 5.1 Single-frame classification: Architecture D, containing seven convolution layers, achieved the best single-frame result and was marginally better than baseline methods.Differences among the four configurations were not statistically significant, while architecture D reduced error by up to 12% for hard-to-classify subjects.
- 5.1 Single-frame classification: Topology-preserving projection produced a slight classification-error improvement over non-topology-preserving projection, especially around peripheral image regions.The comparison used an orthographic projection onto the z=0 plane.
- 5.2 Multi-frame classification: Temporal convolution and LSTM significantly improved multi-frame classification, with 32 temporal kernels outperforming 16 kernels at 11.32% versus 12.86% error.The combined temporal-convolution and LSTM structure produced the dataset’s best result.
- 5.2 Multi-frame classification: Baseline classifiers had slightly lower test errors in the single-frame setting, whereas temporal dynamics increasingly improved the proposed model.The authors attribute the baseline difference to the increased feature count in multi-frame inputs and identify complex spectral-temporal relationships as an open question.
- 5.2 Multi-frame classification: Max-pooling over time underperformed the single-frame ConvNet because selecting peak activations can ignore periods of cortical-region inactivation.This provides a rationale for the lower errors observed when temporal convolution and LSTM were added.
- 5.3 Visualizing the learned representations: Deeper feature maps had sparser activation regions and showed strong frequency selectivity, including features linked to frontal theta, beta, and parietal alpha activity.The authors visualized these representations by back-projecting feature maps into input space.
6 CONCLUSIONS
The paper represents EEG as topology-preserving multi-spectral image sequences and models them with recurrent-convolutional networks for mental-load classification. This approach improves classification accuracy and supports comparison across EEG hardware and datasets.
- EEG activities are transformed into topology-preserving multi-spectral images, preserving spatial information that standard EEG analyses may ignore.
- A deep recurrent-convolutional network learns robust representations from the resulting sequence of EEG images.
- The approach demonstrates significant improvements in classification accuracy over state-of-the-art results for mental-load classification.
- The preprocessing transform can produce comparable EEG frames from different hardware configurations, including setups with different electrode counts.The transform requires the spatial coordinates of electrodes for each setup.
- Comparable EEG frames could enable merging datasets acquired from varied EEG sources, with unsupervised pretraining proposed as a future direction.
APPENDIX
The appendix illustrates substantial variability in EEG responses across subjects and across repeated runs by the same subject under identical task conditions.
- The examples are intended to demonstrate potentially high inter-subject and intra-subject variability in observed task responses.
- Average EEG frames under the same cognitive-load condition are compared across two subjects and across multiple runs for one subject.