Deep learning with convolutional neural networks for EEG decoding and visualization

• End-to-end ConvNets are investigated for EEG decoding because they can learn features directly from raw data, although EEG differs fundamentally from image inputs.EEG is a dynamic time series measured across the scalp and has comparatively low signal-to-noise ratio, motivating adapted architectures and rigorous comparison with traditional methods.Source: §1 · p. 4
• It systematically tests ConvNet design choices, including dropout, batch normalization, and exponential linear units, alongside architecture-level effects on EEG decoding.The study addresses a gap in comparable evaluations of deep ConvNet design choices for EEG decoding.Source: §1 · pp. 4–5
• The study evaluates whole-trial versus cropped training because shifted trial windows can increase training examples when EEG datasets are relatively small.Cropped training is tested for both deep and shallow ConvNets, with a computationally efficient strategy for using more input crops per trial.Source: §1 · pp. 4–5
• The study compares shallow, deep, hybrid, and 31-layer residual ConvNets with FBCSP on two motor-decoding EEG datasets using minimal preprocessing.The ConvNets were adapted for multichannel EEG, and FBCSP provided a validated standard decoding baseline.Source: §1 · p. 4
• Two visualization methods examine whether ConvNets retain or causally use motor-relevant EEG band-power features across layers and outputs.The methods produce scalp topographies of information retention or causal effects for frequency bands including alpha, beta, and gamma, enabling comparison with FBCSP.Source: §1 · p. 5
• The study’s central contribution is showing that end-to-end deep ConvNets can achieve accuracies in the same range as FBCSP for movement-related EEG decoding.It also identifies batch normalization and exponential linear units as crucial for high accuracy and reports visual evidence that learned features have meaningful motor-relevant spatial distributions.Source: §1 · p. 5

2.1 Definitions and notation

• EEG decoding is defined per subject as assigning movement-related class labels to unseen trials represented by E-electrode-by-T-timestep input matrices.Labels can include Hand (Left), Hand (Right), Feet, and Rest.Source: §2.1.1 · p. 7, §2.1.2 · p. 7

2.1.2 Decoder; 2.5.1 Definition

• ConvNets jointly optimize feature extraction and classification with mini-batch stochastic gradient descent and backpropagation, unlike FBCSP’s separated stages.The final softmax layer produces subject-specific conditional probabilities over classes, while earlier layers learn the feature representation.Source: §2.1.2 · p. 7, §2.5.1 · p. 14

2.2 Filter bank common spatial patterns (FBCSP)

• FBCSP provides the benchmark pipeline by filtering frequency bands, learning CSP spatial filters, constructing log-variance features, and classifying trials separately from feature extraction.FBCSP was the best-performing method on the BCI Competition IV dataset 2a and won similar EEG decoding competitions.Source: §2.2 · pp. 7–8

2.4.1 Input representation

• The ConvNets represent EEG as a two-dimensional electrode-by-time array, enabling spatially global input filters and temporal hierarchies while reducing dimensionality versus scalp-image inputs.This representation reflects globally mixed spatial voltage patterns and multiscale temporal organization in EEG.Source: §2.4.1 · p. 9

2.4.2 Deep ConvNet for raw EEG signals

• The generic deep ConvNet uses an EEG-specific temporal-then-spatial first block, three further convolution-max-pooling blocks, and a dense softmax layer to learn broadly applicable features from raw signals.The first block is split to handle one input channel per electrode, with spatial filters spanning all electrodes.Source: §2.4.2 · p. 9

2.4.3 Shallow ConvNet for raw EEG signals

• The shallow ConvNet embeds an FBCSP-like temporal and spatial filtering, squaring, mean pooling, and logarithmic activation in one jointly optimized network that can model within-trial band-power dynamics.Its temporal kernel is 25 samples rather than 10, and multiple pooling regions permit learning temporal structure within trials.Source: §2.4.3 · pp. 11–12

2.4.4–2.4.6 ConvNet architectures and design choices

• The study evaluates architectural design choices, regularization, a hybrid network fusing deep and shallow features, and a 14-block residual ConvNet trained with ELU activations.The hybrid concatenates 60 deep and 40 shallow feature maps and retrains the combined network from scratch.Source: §2.4.4 · p. 12, §2.4.5 · p. 13, §2.4.6 · p. 13

2.5.2–2.5.4 Training strategies

• Training compares whole-trial inputs with cropped sliding windows, using 4.5-second trials for trial-wise training and about 2-second crops whose 625 predictions are averaged at test time.Cropped training increases the training set size by a factor of 625, uses highly correlated examples, and reduces the input from about 1000 to about 500 samples at 250 Hz.Source: §2.5.2 · p. 15, §2.5.3 · p. 15, §2.5.4 · p. 15

2.5.4 Cropped training

• Processing about 500 neighboring crops in one pass reduced cropped-training cost without substantial accuracy decreases, but increased memory use and was unsuitable for individual-sample predictions with padded convolutions.The method remained applicable when only average trial predictions were required.Source: §2.5.4 · p. 17
• The tied sample loss regularized cropped training by penalizing discrepancies between predictions from neighboring crops, encouraging features stable across adjacent inputs.It added the cross-entropy between neighboring predictions to the usual negative log-likelihood loss.Source: §2.5.4 · p. 17

2.5.5 Optimization and early stopping

• Adam optimization with a specific early-stopping strategy consistently yielded good accuracies in the experiments.The receptive-field schema shows that deeper units integrate progressively larger portions of the input.Source: §2.5.5 · p. 17, Figure 5 · p. 18

2.6.1 Correlating Input Features and Unit Outputs: Network Correlation Maps

• Network correlation maps related spectral-amplitude features within units’ receptive fields to unit outputs, using mean envelopes of alpha, beta, and gamma bands to visualize learned representations.Systematic correlations differing from an untrained network indicate increased representation of those features after training.Source: §2.6.1 · pp. 18–19
• Input-perturbation network-prediction correlation maps complemented correlation maps by perturbing existing feature amplitudes to assess whether amplitude changes caused prediction changes.The method addressed the first approach’s inability to distinguish correlation from causation.Source: §2.6.1 · p. 19

2.7 Data sets and preprocessing

• The study evaluated a 288-trial public BCI Competition IV dataset and a larger laboratory dataset with approximately 880 trials per subject, using minimal preprocessing and an additional causal third-order Butterworth high-pass condition above 4 Hz.The high-pass analysis was intended to reduce reliance on low-frequency eye-movement artifacts, and Table 2 covers FBCSP, deep-ConvNet, and shallow-ConvNet decoding accuracies across frequency ranges.Source: §2.7 · p. 19, §2.7.1 · p. 19, Table 2 · p. 20

3.1 Validation of FBCSP baseline

• FBCSP validation reproduced the published BCI Competition Dataset accuracy: 67.6% versus 67.0%, with no significant difference, while the 0.5–4-second window yielded 67.8%.The 0.5–4-second window was used for subsequent ConvNet comparisons.Source: §3.1 · p. 20

3.2 Architectures and design choices

• ConvNets matched FBCSP accuracies, with deep 0–fend-Hz and shallow 4–fend-Hz models significantly higher by 1.9% and 3.3%, respectively.For the other frequency-range comparisons, shallow 0–fend-Hz and deep 4–fend-Hz accuracies were not significantly different from FBCSP.Source: §3.2 · p. 20, Figure 6 · p. 21
• Confusion patterns were broadly similar across methods, with most High-Gamma mistakes involving Hand(L)/Hand(R) and Feet/Rest, together exceeding 50% of errors.The deep ConvNet outperformed the shallow ConvNet in seven significant confusion-matrix entries, while differences from FBCSP were limited.Source: §3.2 · p. 22, Table 3 · p. 22, Figure 7 · p. 23
• Batch normalization, dropout, and ELUs substantially improved deep ConvNet accuracy, enabling competitiveness with FBCSP, whereas residual networks performed consistently worse.Removing recent advances made the deep ConvNet significantly worse than FBCSP in both frequency ranges; residual-network differences were significant on the combined dataset.Source: §3.2 · pp. 22, 25, Table 4 · p. 25

3.3 Training Strategy

• Cropped training significantly improved deep-ConvNet accuracy on 4–fend-Hz data, but not in other settings, and FBCSP trained 5–45 times faster than cropped ConvNets.The reported speed factors were 27–45 on the BCI Competition Dataset and 5–9 on the High-Gamma Dataset; trained ConvNets nevertheless had fast prediction suitable for real-time BCI applications.Source: §3.3 · pp. 25, 27, Table 5 · p. 27

3.4 Visualization

• Band-power topographies showed expected motor-task desynchronization in alpha and beta bands and synchronization in gamma, including contralateral hand and vertex foot patterns.Alpha power decreased around contralateral pericentral regions for hand movements and around the vertex for feet, whereas gamma showed opposite changes.Source: §3.4 · pp. 27–28, Figure 11 · p. 28
• Input-feature unit-output correlations increasingly resembled power-class correlations with network depth, while absolute unit-output correlations increased almost linearly.These maps showed a progression from learned representations toward class-related EEG power patterns.Source: §3.4 · pp. 28–29, Figure 12 · p. 29, Figure 13 · p. 30
• Input-perturbation maps revealed alpha, beta, and gamma frequency peaks and global somatotopic causal contributions, including motor-cortical gamma effects for hand and foot decoding.The maps reflect prediction changes caused by independently perturbing localized features and therefore require caution when interpreting them as data amplitudes.Source: §3.4 · pp. 29–30, Figure 15 · p. 31, Figure 16 · p. 31

4.1.1 ConvNets vs. FBCSP

• Appropriately designed deep and shallow ConvNets reached or slightly exceeded FBCSP accuracies for combined-dataset motor decoding, supporting ConvNets as suitable EEG-decoding methods.The comparison does not establish large gains over FBCSP, and regularized FBCSP variants remain to be evaluated.Source: §4.1.1 · p. 32, §4 · p. 32

4.1.2 Role of recent deep learning advances

• Batch normalization, dropout, ELUs, and especially cropped training improved ConvNet performance, with the largest gains appearing for deep networks on 4–fendHz data.These results suggest overfitting and interactions among architectural and hyperparameter choices, while the ELU–ReLU accuracy difference remains unexplained.Source: §4.1.2 · p. 32, §4.2.1 · p. 34

4.1.3 ConvNet architectures and interactions with discriminative features

• Shallow ConvNets performed as well as deep ConvNets, while squaring was important for shallow but not deep networks because it directly supports band-power extraction.The architectural differences may reflect whether discriminative EEG information is primarily power-, phase-, or hierarchically structured.Source: §4.1.3 · p. 33

4.1.4 Possibilities for substantial decoding accuracy improvements?

• ConvNets did not clearly outperform FBCSP by a large margin, indicating substantial room for improvement through larger datasets, transfer learning, recurrent models, and future deep-learning methods.The authors conclude that no single optimal EEG-decoding method can yet be identified, although ConvNet-based decoding is a contender.Source: §4.1.4 · pp. 33–34

4.1.5 Further potential advantages of ConvNets for brain-signal decoding

• Deep ConvNets support transfer learning, supervised online adaptation, broader signal-decoding tasks, other recording modalities, and multi-network systems robust to distribution changes.These capabilities follow from iterative end-to-end training and joint optimization.Source: §4.1.5 · p. 34

4.2.1 Cropped training effect on accuracies

• Cropped training was necessary for competitive deep-ConvNet accuracy without very low frequencies, plausibly because shifted windows increase examples and reduce overfitting to within-trial phase.This is particularly consistent with learning band-power features in higher-frequency data.Source: §4.2.1 · p. 34

4.2.2 Suitability for online decoding

• Cropped training can trade decoding delay against accuracy online: smaller crops may reduce delay, whereas larger within-trial crops may provide more information and improve accuracy.The proposed trade-off requires confirmation in online experiments.Source: §4.2.2 · pp. 34–35

4.3.1 Insights from current visualizations

• Correlative and causal visualization methods mapped plausible alpha-, beta-, and gamma-band spatial features and showed that deep ConvNets learned band-power computation directly from raw EEG.The methods support subject-level and group-level spatial mapping of features used for single-trial decoding, but future visualizations should investigate unknown features and decision-relevant trial regions.Source: §4.3.1 · p. 35, §4.3.2 · pp. 35–36, §4.4 · p. 36

A.1 Related Work

• The surveyed literature spans motor-imagery, brain-computer-interface, cognitive-performance, epileptic, seizure-detection, and rapid-serial-visual-presentation EEG applications.The supplied fragments identify work on single-trial motor imagery, EEG-based brain-computer interfaces, driver cognition, epileptic intracranial EEG, automatic and wearable seizure detection, and RSVP classification.Source: §A.1 · pp. 46–47
• Largest EEG-decoding weights were found over prefrontal and temporal regions, alongside typical frequency subdivisions including delta, theta, and alpha, with plausible times of 300–500 ms.The cited frequency ranges include 4–7 Hz and 8–13 Hz.Source: §A.1 · pp. 46–48
• Reported architectural variations include stacked convolutional layers, autoencoders on ConvNet features, restricted Boltzmann machines, LSTMs, transfer learning, and differing kernel sizes or constraints.The fragments mention six-layer stacked autoencoders, replacement of convolutional layers by restricted Boltzmann machines, LSTM postprocessing, cross-subject supervision, and architectural constraints producing similar spatial topographies.Source: §A.1 · p. 47

A Supplementary Materials

• The supplementary materials identify visualization-based analyses as a way to locate critical areas and relate learned features to electrophysiological markers.The supplied fragments describe synchronized neighboring activations, spatially localized activations, and links between learned features and known electrophysiological markers of cognitive processing.Source: §A.1 · pp. 47–48

A.1 Related Work

• Table S1 surveys previous publications using convolutional neural networks for EEG decoding, covering input domains, layer configurations, baselines, and visualization findings.The table distinguishes frequency-domain inputs containing amplitude-derived features but no phase information, and clarifies that layer counts refer to EEG-decoding models when multiple modalities are used.Source: Table S1 · p. 49
• Table S1 states that no previous time-domain study showed with a suitable visualization technique that ConvNets learned band-power features, unlike the present study.The table also notes that some earlier studies used artificial neural networks without convolutions.Source: Table S1 · p. 49

A.2 FBCSP implementation

• FBCSP classified features with shrinkage-regularized linear discriminant analysis and one-vs-one majority weighted voting for multiple classes.Pairwise classifier outputs were scaled to a common range, summed across classes, and used to select the class with the highest sum.Source: §A.2 · p. 50
• Feature selection addressed overfitting by preselecting spatial filters and selecting final filter-bank features through inner cross-validation, using overlapping bands adapted to each dataset.Bands were 6 Hz wide with 3 Hz overlap up to 13 Hz and 8 Hz wide with 4 Hz overlap above 10 Hz; frequency limits were 38 Hz for the BCI Competition Dataset and 122 Hz for the High-Gamma Dataset.Source: §A.2 · p. 50
• 67.59% versus 67.01% average accuracy across subjects (p>0.7) showed that the FBCSP implementation closely matched published performance and provided a suitable ConvNet baseline.The comparison used the same BCI Competition Dataset and a Wilcoxon signed-rank test.Source: §A.2 · p. 50

• The ResNet contains 31 convolutional layers, matching a depth at which non-residual ConvNets showed convergence problems in the original ResNet study (He et al., 2015).Source: §A.3 · p. 50
• When channels increase, shortcut inputs are zero-padded to match the new dimensionality, following option A of the original ResNet paper (He et al., 2015).Source: §A.3 · p. 50
• The architecture uses ELU nonlinearities in the second convolution and all residual blocks, with a final 7x1x4 output representing seven time-distributed predictions for four classes.With cropped training, the network produces 424 predictions and averages them to predict the trial.Source: Table S2 · p. 51

Optimization

• Adam was used to optimize the ConvNet’s large number of high-dimensional parameters.Source: §A.4 · p. 51

Early stopping

• Training stopped when validation accuracy failed to improve for a predefined number of epochs, then resumed on combined training and validation data from the best validation parameters.The final phase ended when validation loss matched the training loss measured at the end of the first phase.Source: §A.4 · p. 51

Evaluation

• All reported accuracies were determined on an independent test set.This strategy used all training data while avoiding manual selection of the number of training epochs.Source: §A.4 · p. 51

A.5.1 Input-feature unit-output correlation maps

• Input-feature correlation maps visualize frequency-resolved correlations between convolutional-filter outputs and receptive-field sample power.The method is illustrated in Figure S1.Source: §A.5.1 · p. 52, Figure S1 · p. 53
• For each frequency band, signals were bandpass-filtered, envelopes computed, and moving-window squared mean envelopes matched to each layer’s receptive-field size.These envelope values served as input features for evaluating effects on unit outputs.Source: §A.5.1 · p. 52
• Linear correlations between receptive-field squared-envelope values and original-signal unit outputs estimate whether filters approximate frequency-specific local power.Unit outputs were computed for the given layer before correlating them with envelope features.Source: §A.5.1 · p. 52
• Correlations concatenated across all trials capture both within-trial and between-trial effects, with analyses spanning 0–119 Hz.A single-electrode, single-subject example is shown in Figure S2, including alpha/beta and gamma peaks.Source: §A.5.1 · p. 52, Figure S2 · p. 54
• Scalp plots average absolute filter correlations by electrode and frequency, then subtract untrained-model maps from trained-model maps to isolate training effects.Group maps average these differential subject-level scalp plots.Source: §A.5.1 · p. 52
• Layerwise unit-output/class-label correlations quantify average class encoding across filters and encoding by the most class-informative filters.Trained-minus-untrained differences show how class correlations arise through the ConvNet.Source: §A.5.1 · pp. 52, 54

A.5.2 Input-perturbation network-prediction correlation map

• Training trials were Fourier-transformed, their amplitudes received mean-zero, variance-one Gaussian noise, phases remained unchanged, and signals were inverse-transformed before prediction.Predictions were computed before and after perturbation, following the procedure summarized in Figure S3.Source: §A.5.2 · p. 54, Figure S3 · p. 55
• Above 99.5%: accuracies on perturbed trials remained above 99.5% of the corresponding unperturbed-data accuracies for all subjects’ training sets and perturbations.This check ensured the perturbations did not generally cause ConvNet misclassification.Source: §A.5.2 · p. 56
• The analysis correlated amplitude perturbations with changes in ConvNet outputs across 400 Gaussian-noise perturbations per trial.Predictions were measured directly before the softmax activation.Source: §A.5.2 · pp. 54, 56

EEG spectral power topographies

• Alpha-, beta-, and gamma-band envelope–class correlations produced group-averaged scalp topographies separately for all four High-Gamma Dataset classes.The maps correlated moving averages of squared envelopes with actual class labels, using the final layer’s receptive-field size as the averaging window.Source: §A.5.3 · p. 56
• The ConvNet-independent power maps were directly comparable to input-feature unit–output network correlation maps.Unlike the network visualization in Section A.5.1, the analysis did not subtract values from an untrained ConvNet.Source: §A.5.3 · p. 56
• These computations generated only the power topographies in Figure 11 and did not contribute to the preceding decoding analyses.Source: §A.5.3 · p. 56

A.6 Dataset details

• The BCI Competition Dataset contains 22-electrode EEG from 9 subjects across two sessions, with 288 four-second trials per session covering four imagined movements.The first session formed training data and the second formed test data.Source: §A.6 · p. 56
• The High-Gamma Dataset contains roughly 1000 four-second executed-movement trials per subject from 20 subjects, with training and test data separated by runs.Although recorded with 128 electrodes, analyses used 44 motor-cortex sensors; classes were left hand, right hand, both feet, and rest.Source: §A.6 · p. 56
• The movement-execution experiment used visual arrow cues, pseudo-randomized four-second trials, and instructed subjects to minimize non-task movement in a shielded EEG environment.Subjects performed toe clenching, sequential left- or right-hand finger tapping, or relaxation, with inter-trial intervals of 3–4 seconds.Source: §A.6 · p. 57

A.7 EEG preprocessing

• Both datasets were resampled to 250 Hz and frequency-filtered to align ConvNet inputs with the frequency range available to CSP methods.The BCI Competition Dataset used a 4–38 Hz bandpass, while the High-Gamma Dataset received the specified high-pass filtering and both datasets underwent electrode-wise exponential moving standardization.Source: §A.7 · p. 57
• Preprocessing removed only trials exceeding ±800 µV in any channel and retained lower-amplitude artifacts, while High-Gamma analyses used central motor-cortex electrodes because all-electrode inputs worsened both methods’ accuracies.Exponential standardization used only training-set initialization, and minimal preprocessing avoided manual feature extraction while remaining compatible with online BCI use.Source: §A.7 · pp. 57–58

A.8 Statistics

• Method accuracy differences were assessed with Wilcoxon signed-rank tests, using conservative handling of ties, zero differences, and non-integer test statistics.Tied observations received average ranks, zero differences split rank sums between positive and negative differences, and non-integer statistics were rounded upward.Source: §A.8 · p. 58

A.9 Software implementation and hardware

• ConvNet experiments ran on GTX Titan Black GPUs, while FBCSP used Xeon CPUs; ConvNets used Lasagne and preprocessing and FBCSP used Wyrm.The authors stated that code to reproduce the results would be made publicly available.Source: §A.9 · p. 58