Source-linked AI summary
Deep learning with convolutional neural networks for EEG decoding and visualization
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, Tonio Ball
TL;DR
Understanding what convolutional networks learn from EEG remains important for interpreting brain-signal features. This paper evaluates end-to-end ConvNets for motor decoding, examines design choices, and introduces visualization methods; ConvNets achieve accurate decoding comparable to FBCSP while revealing frequency-band features relevant to motor decoding.
Problem
Understanding which brain-signal features convolutional networks use remains an important open problem for neuroscientists interpreting EEG decoding.
Method
The study systematically evaluates end-to-end ConvNets for motor-related EEG decoding and applies novel visualizations to inspect learned features.
Results
ConvNets achieve motor-decoding accuracies at least in the same range as FBCSP, with batch normalization, exponential linear units, and cropped training improving performance.
Takeaways & Limitations
ConvNets are a promising EEG-decoding contender and, combined with visualization techniques, may support EEG-based brain mapping.
Takeaways & Limitations
No final verdict is possible about an optimal EEG-decoding method, if a single best method exists across the variety of EEG-decoding problems.
Abstract
from arXiv · showhide
PLEASE READ AND CITE THE REVISED VERSION at Human Brain Mapping: http://onlinelibrary.wiley.com/doi/10.1002/hbm.23730/full Code available here: https://github.com/robintibor/braindecode
1 Introduction
The study investigates end-to-end convolutional neural networks (ConvNets) for EEG decoding, addressing how architectures, design choices, and training strategies affect accuracy and how learned features can be visualized. It evaluates multiple ConvNet architectures and introduces findings on decoding performance, training crops, and motor-relevant EEG band-power representations.
- Motivation: ConvNets are investigated for end-to-end EEG decoding because they can learn from raw data without a priori feature selection.This approach is motivated by the possibility that relevant brain-signal features are not all known in advance.
- Motivation: EEG differs from image inputs because it is a dynamic scalp-based time series with comparatively low signal-to-noise ratio.These properties make EEG decoding a distinct application context for ConvNets.
- Study design: The study compares shallow, deep, hybrid, and residual ConvNets spanning 2, 5, and 31 convolutional layers.It also evaluates design choices including dropout, batch normalization, and exponential linear units, plus training-data sampling strategies.
- Main contributions: End-to-end deep ConvNets reach accuracies at least in the same range as FBCSP for decoding movement-related information from EEG.This is presented as the first demonstration of this result in the study.
- Main contributions: Batch normalization and exponential linear units are crucial for high decoding accuracies, while cropped training can increase deep-ConvNet accuracy.The study also presents efficient training on more input crops per EEG trial and visualizations suggesting use of motor-relevant alpha, beta, and gamma band power with meaningful spatial distributions.
2 Methods
The study formulates EEG decoding as supervised classification over labeled electrode-by-time trials, then evaluates FBCSP and several ConvNet designs with minimal preprocessing. The methods also introduce visualization approaches for identifying spectral features used by ConvNets and perturbation effects on predictions.
- Supervised EEG decoding: Each subject’s EEG dataset comprises labeled trials represented as E-electrode × T-time-step input matrices, with movement-related class labels.Labels can include imagined or executed left-hand, right-hand, feet, and rest movements.
- Decoding methods: FBCSP serves as the established benchmark, while ConvNets jointly learn feature extraction and classification rather than separating these stages.FBCSP was selected because it performed strongly on the BCI Competition IV dataset 2a and related EEG decoding competitions.
- ConvNet architectures: The deep ConvNet was designed as a generic architecture capable of extracting a wide range of EEG features without restriction to specific feature types.Its design was inspired by successful computer-vision architectures and motivated by interest in generic feature learning.
- ConvNet architectures: The shallow ConvNet was tailored to band-power decoding, using temporal and spatial convolutions analogous to FBCSP while embedding all computations in one jointly optimized network.Multiple pooling regions also allow it to learn temporal structure in band-power changes within a trial.
- ConvNet architectures: A hybrid ConvNet combined the shallow network’s specific feature extraction with the deep network’s generic feature extraction by fusing both networks after their final layers.The hybrid design replaced the individual four-filter softmax classification layers with a fused architecture.
- Visualization methods: The methods proposed correlation maps and input-perturbation analysis to visualize spectral features used by ConvNets and assess how changing input amplitudes affects predictions.Correlation maps use mean envelope values from frequency bands such as alpha, beta, and gamma; perturbation analysis complements correlation because correlation does not establish causation.
3 Results
The results show that appropriately designed ConvNets matched or slightly exceeded FBCSP decoding, while residual and hybrid variants performed worse or no better. Visualization methods revealed physiologically plausible spatial-frequency features and layer-wise or perturbation-based relationships, with interpretive caution.
- Decoding performance: FBCSP reached the previously reported baseline accuracy of 67.6% using the 0.5–2.5 s window on BCI Competition Dataset 2a.The implementation was validated against published accuracies before evaluating ConvNets.
- Decoding performance: Both deep and shallow ConvNets achieved accuracies similar to FBCSP, with statistically significant advantages in some settings.For subject means, shallow ConvNet 0–fend Hz and deep ConvNet 4–fend Hz did not significantly differ from FBCSP.
- Decoding performance: Confusion matrices were very similar across FBCSP and ConvNets, with most errors involving Hand (L) / Hand (R) and Feet / Rest.Seven significant deep-versus-shallow differences favored the deep ConvNet.
- Architecture and training effects: Hybrid ConvNets were slightly but insignificantly worse than deep ConvNets: 83.8% vs 84.0% on 0–fend Hz and 82.1% vs 83.1% on 4–fend Hz.Hybrid-versus-deep comparisons were reported with p>0.5 and p>0.9, respectively.
- Visualization: Band-power topographies and correlation maps revealed plausible motor-task features across alpha, beta, and gamma bands, while perturbation maps showed spatially localized effects on predictions.The visualization methods mapped learned features across layers and supported single-trial decoding interpretation, but correlations do not by themselves imply corresponding data amplitude changes.
4 Discussion
ConvNets achieved accurate end-to-end EEG motor decoding with minimal preprocessing, reaching or slightly exceeding FBCSP when appropriate deep-learning techniques and cropped training were used. The study also introduced visualization methods, while noting that ConvNets remain contenders rather than definitively optimal decoders and that their visualizations primarily capture known band-power features.
- Decoding performance: ConvNets enabled accurate end-to-end motor decoding from EEG with minimal preprocessing, and recent deep-learning techniques further improved performance.The evaluated techniques included batch normalization, dropout, and exponential linear units.
- Decoding performance: Deep and shallow ConvNets reached at least FBCSP accuracies, with both slightly exceeding FBCSP in the combined-dataset comparison.FBCSP was treated as the validated de facto standard for motor decoding from EEG recordings.
- Architecture: Shallow ConvNets performed as well as deep ConvNets, whereas hybrid and residual architectures did not show the same result.The authors relate this observation to which discriminative EEG features different architectures can best use.
- Limitations: 3.5 percent was the maximum significant improvement over FBCSP on the combined dataset, indicating that ConvNets did not improve accuracy by a large margin.The combined dataset contained substantial variation per subject.
- Conclusion: ConvNets are promising EEG-decoding contenders, but no final verdict identifies an optimal method for the variety of EEG-decoding problems.The authors conclude that ConvNets combined with innovative visualization techniques may open new windows for EEG-based brain mapping.
- Training strategy: Cropped training is particularly applicable to online decoding because crop size calibrates the trade-off between decoding delay and accuracy.Smaller crops can reduce delay, whereas larger crops can contain more trial information.
- Visualization: The study developed two complementary visualization methods, but these visualizations were designed only to show how ConvNets use known band-power features.The authors identify investigation of novel or unknown features and newer visualization methods as promising next steps.
A Supplementary Materials · A.1 Related Work
The supplementary related-work table catalogs convolutional-neural-network studies for EEG decoding, including their input representations, architectures, and visualization findings. The cited studies span motor imagery, cognitive performance, epileptic and wearable seizure detection, and RSVP classification.
- A.1 Related Work: Visualization studies linked learned weights or activations to spatial regions, frequency bands, and electrophysiological markers.Reported findings include largest weights over prefrontal and temporal cortex, clusters corresponding to delta, theta, and alpha bands, and links to cognitive electrophysiological markers.
- A.1 Related Work: Architectural modifications in the surveyed models included stacked autoencoders, restricted Boltzmann machines, LSTM postprocessing, and transfer learning.The table fragments also mention varying convolutional-layer counts and pretraining first.
- A.1 Related Work: Related applications included driver cognitive-performance prediction and epileptic intracranial-EEG analysis.The passages identify deep convolutional networks for driver-performance prediction and deep learning for epileptic intracranial EEG.
- A.1 Related Work: Some prior analyses found increasingly synchronized neighboring activations as training iterations increased.Other visualization results allowed identification of areas considered critical for decoding.
- A.1 Related Work: Other cited CNN applications covered single-trial EEG RSVP classification and wearable seizure detection.The related-work entries specifically mention convolutional neural networks for RSVP classification and wearable seizure detection with transfer learning.
- A.1 Related Work: Table S1 organizes previous EEG-decoding publications by frequency-domain input, layer counts, and model-specific architectural features.Frequency-domain inputs contain amplitudes or amplitude transformations, never phase information; dense-layer counts include parametrized classification layers.
A.2 FBCSP implementation
The FBCSP implementation used shrinkage-regularized linear discriminant analysis with one-vs-one weighted voting for multiclass decoding and feature selection to limit overfitting. Its performance closely matched a published implementation on the same dataset, averaging 67.59% versus 67.01% across subjects.
- A.2 FBCSP implementation: The implementation used shrinkage-regularized linear discriminant analysis and one-vs-one majority weighted voting for multiclass decoding.Classifier outputs were scaled to a common range, summed across class pairs, and the highest-scoring class was selected.
- A.2 FBCSP implementation: FBCSP feature selection preselected spatial filters using the ratio of corresponding power features for both classes.The method reflects the observation that few spatial filters from few frequency bands often suffice, whereas many filters can overfit.
- A.2 FBCSP implementation: 67.59% for the implementation versus 67.01% for the published implementation on average across subjects, with p>0.7 in a Wilcoxon signed-rank test.The comparison served as a sanity check on the FBCSP implementation, including its feature selection and filter-bank design.
A.3 Residual network architecture
The residual network contains 31 convolutional layers and uses zero-padded shortcut connections when channel dimensionality increases. Its architecture table specifies kernel and output-size conventions and ELU nonlinearities for later convolutions and residual blocks.
- A.3 Residual network architecture: The ResNet comprises 31 convolutional layers, reaching a depth associated with convergence problems in non-residual ConvNets.This depth matches the point at which the original ResNet paper reported convergence difficulties without residual blocks.
- A.3 Residual network architecture: When channels increase, zero-padding matches the incoming feature map to the shortcut’s new channel dimensionality, following option A of the original ResNet.The padding is applied to the incoming feature map in layers that increase channel count.
- A.3 Residual network architecture: The architecture table reports output sizes as time x height x channels, with EEG channels represented in the height dimension rather than the layer-channel dimension.The table distinguishes network-layer input channels from EEG channels.
- A.3 Residual network architecture: The second convolution and all residual blocks use ELU nonlinearities.This nonlinear activation choice is specified in the residual network hyperparameter table.
A.4 Optimization and early stopping
The study optimizes ConvNets with Adam, a stochastic-gradient-descent variant suited to high-dimensional parameter spaces. Training uses validation-based early stopping, halting the first phase when validation accuracy fails to improve for a predefined number of epochs.
- Optimization: Adam optimizes the ConvNet’s large number of high-dimensional parameters.Adam is described as a variant of stochastic gradient descent designed for high-dimensional parameters.
- Early stopping: Training splits the dataset into training and validation folds for early stopping.The strategy was developed in computer vision and is used throughout the study.
- Early stopping: Early stopping halts the first training phase when validation accuracy does not improve for a predefined number of epochs.The stopping criterion is based on the absence of validation-accuracy improvement over the specified patience period.
A.5 Visualization methods · A.5.1 Input-feature unit-output correlation maps
Input-feature unit-output correlation maps relate convolutional-filter outputs to frequency-resolved squared-envelope power within receptive fields. The method compares trained and untrained networks, summarizes electrode-wise correlations as scalp plots, and examines within- versus between-trial effects and class-label encoding.
- A.5.1 Input-feature unit-output correlation maps: Input-feature unit-output correlation maps visualize frequency-resolved correlations between convolutional-filter outputs and receptive-field sample power.The maps are illustrated in Figure S1.
- A.5.1 Input-feature unit-output correlation maps: For each frequency band, the signal is bandpass-filtered, its envelope computed, and moving-window averages of squared envelopes formed using receptive-field-sized windows.These squared mean-envelope values serve as the input features whose effects on unit outputs are evaluated.
- A.5.1 Input-feature unit-output correlation maps: Unit outputs are computed on the original signal, then linearly correlated with squared mean-envelope values across frequency bands for each convolutional filter.The correlations are intended to indicate whether a filter approximates receptive-field squared mean envelope power.
- A.5.1 Input-feature unit-output correlation maps: Correlations concatenate samples from all trials, so they reflect both within-trial and between-trial effects, and are computed for a 0–119 Hz filter bank.The method could be extended to disentangle the two effects; an example uses one electrode and one subject.
- A.5.1 Input-feature unit-output correlation maps: Scalp plots average absolute correlations across units for each filter and electrode, then subtract the untrained-model plot from the trained-model plot.This subtraction is intended to separate effects of network training from effects of architecture.
- A.5.1 Input-feature unit-output correlation maps: The resulting maps are also compared with scalp maps based on linear correlations between mean squared envelopes and one-hot-encoded class features.For each trial, sensor, and frequency band, moving-window squared-envelope vectors are constructed using a last-layer receptive-field-sized window.
- A.5.1 Input-feature unit-output correlation maps: To test whether correlations arise from final class-label encoding rather than the analyzed features, unit outputs are correlated directly with class labels.The supplied passage indicates that this procedure follows the preceding class-feature-correlation method.
- A.5.1 Input-feature unit-output correlation maps: Class-label encoding is summarized per layer using mean absolute correlations across classes and filters, with a second maximum-based coefficient also computed.The supplied passage states that these summaries assess average and maximal class-label encoding strength.
A.5.2 Input-perturbation network-prediction correlation map
The analysis estimated how amplitude perturbations causally affect deep ConvNet predictions by adding Gaussian noise to Fourier-domain amplitudes while preserving phases, then correlating perturbations with prediction changes. It also checked that perturbed inputs did not cause trial misclassification.
- Perturbation procedure: Training trials were Fourier-transformed, and Gaussian noise with mean 0 and variance 1 was added to amplitudes while phases remained unperturbed.The perturbations were applied to the original trial amplitudes.
- Correlation analysis: The analysis correlated changes in input amplitudes with changes in ConvNet predictions to assess their relationship.Input-amplitude changes corresponded to the added perturbation noise.
- Validity check: Perturbed inputs were checked to ensure they did not cause the ConvNet to misclassify trials.This control addressed the possibility that even small perturbations can induce misclassification.
A.5.3 EEG spectral power topographies
The study visualized class-specific EEG spectral power modulations in the High-Gamma Dataset by computing band-specific envelope-class correlations and group-averaged topographies. These topographies were designed for comparison with input-feature unit-output network correlation maps.
- A.5.3 EEG spectral power topographies: Band-specific envelope-class correlations were computed in the alpha, beta, and gamma bands for all High-Gamma Dataset classes.The analysis visualized class-specific EEG spectral power modulations.
- A.5.3 EEG spectral power topographies: Group-averaged topographies of the correlation maps were used to visualize the resulting spectral power modulations.
- A.5.3 EEG spectral power topographies: These topographies could be readily compared with input-feature unit-output network correlation maps.The comparison followed the power-class correlation map approach described in Section A.5.1.
A.6 Dataset details … A.9 Software implementation and hardware
The study used two EEG motor-imagery datasets with matched sampling and preprocessing choices, while limiting artifact removal and applying Wilcoxon tests for accuracy differences. ConvNet and FBCSP analyses used GPU- and CPU-based implementations, respectively.
- A.6 Dataset details: The BCI Competition Dataset contained 22 electrodes, 9 subjects, 2 sessions, and 288 four-second trials per session, with session-specific training and test sets.The imagined movements were left hand, right hand, feet, and tongue.
- A.6 Dataset details: The High-Gamma Dataset recorded roughly 1000 four-second executed-movement trials from 20 subjects using 128 electrodes, later reduced to 44 motor-cortex sensors.The subjects included 9 female and 4 left-handed participants, with age 27.5±3.2 years (mean±std).
- A.6 Dataset details: Participants performed repetitive toe clenching, sequential left- or right-hand finger tapping, or relaxation in response to gray-arrow visual cues.The movements were selected to require little proximal muscular activity while remaining sufficiently complex to engage subjects.
- A.7 EEG preprocessing: The High-Gamma Dataset was resampled to 250 Hz, and signals were filtered to align the ConvNets’ frequency range with that available to CSPs.The BCI Competition Dataset was bandpass-filtered to 4-38 Hz; the supplied passage truncates the corresponding High-Gamma filtering details.
- A.7 EEG preprocessing: Exponential moving standardization used initial means and variances computed from the first 1000 samples, which were entirely within the training set and therefore excluded future test data.Standardization was not used for FBCSP because preliminary experiments showed worsened accuracies, and it was also omitted for the supplied truncated method.
- A.7 EEG preprocessing: Cleaning removed only trials with at least one channel outside ±800 µV, retaining lower-amplitude artifacts as potentially informative brain-signal trials.ConvNet feature visualization was used to check that classification relied on brain signals rather than artifacts.
- A.8 Statistics: Accuracy differences between decoding methods were tested with Wilcoxon signed-rank tests, using average ranks for ties and half-rank allocation for zero differences.Non-integer test statistics caused by ties or zeros were rounded upward.
- A.9 Software implementation and hardware: ConvNet experiments ran on Geforce GTX Titan Black GPUs with 6 GB memory, while FBCSP used Intel Xeon E5-2650 v2 CPUs and separate RAM configurations.ConvNets used Lasagne, while preprocessing and FBCSP used Wyrm; most ConvNet computation occurred on GPUs.