PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer-based Spatial-Temporal Feature Learning for EEG Decoding

Yonghao Song, Xueyu Jia, Lie Yang, Longhan Xie

arXiv:2106.11170v1eess.SPcs.AIcs.LG

TL;DR

CNN-based EEG decoders can struggle with global dependencies, motivating S3T, a compact transformer that applies attention to spatial channels and temporal slices. On public EEG datasets, the method reaches state-of-the-art-level performance with fewer parameters, while its temporal and spatial modules contribute to accuracy. The main scope limitation is that cross-subject generalization remains insufficiently explored.

  • Problem

    CNN-based EEG decoding methods have limited ability to perceive global dependencies in paradigms where a trial has strong overall temporal relationships.

  • Method

    S3T applies attention to spatially filtered EEG feature channels and temporal slices to learn spatial and temporal features and form a distinguishable representation.

  • Results

    S3T reaches state-of-the-art-level EEG decoding performance with fewer parameters, while removing temporal and spatial transforming reduces mean accuracy by 35.93% and 3.33%, respectively.

  • Takeaways & Limitations

    Attention-based spatial and temporal transforming is reported as a compact approach for multi-category EEG decoding and a potential alternative backbone to CNNs.

  • Takeaways & Limitations

    Only subject-specific experiments were conducted, so cross-subject ability was not well explored; model hyperparameters were selected through preliminary experiments.

Abstract

from arXiv · show

At present, people usually use some methods based on convolutional neural networks (CNNs) for Electroencephalograph (EEG) decoding. However, CNNs have limitations in perceiving global dependencies, which is not adequate for common EEG paradigms with a strong overall relationship. Regarding this issue, we propose a novel EEG decoding method that mainly relies on the attention mechanism. The EEG data is firstly preprocessed and spatially filtered. And then, we apply attention transforming on the feature-channel dimension so that the model can enhance more relevant spatial features. The most crucial step is to slice the data in the time dimension for attention transforming, and finally obtain a highly distinguishable representation. At this time, global averaging pooling and a simple fully-connected layer are used to classify different categories of EEG data. Experiments on two public datasets indicate that the strategy of attention transforming effectively utilizes spatial and temporal features. And we have reached the level of the state-of-the-art in multi-classification of EEG, with fewer parameters. As far as we know, it is the first time that a detailed and complete method based on the transformer idea has been proposed in this field. It has good potential to promote the practicality of brain-computer interface (BCI). The source code can be found at: \textit{https://github.com/anranknight/EEG-Transformer}.

I. INTRODUCTION

EEG decoding supports BCI applications but must handle noisy signals, temporal dependencies, and relationships spanning complete trials. S3T addresses these issues with attention-based spatial and temporal transformations, achieving competitive performance with fewer parameters.

  • Motivation: BCI decodes EEG-based intentions into commands for assistive devices, while MI rehabilitation applications require accurate and robust brain-signal decoding.MI categories include imagined left-hand, right-hand, feet, and tongue movements.
  • Limitations of Existing Methods: CNNs achieve strong EEG-decoding performance but depend on kernel size, can lose time-series information, and struggle to perceive wide-ranging signal relationships.Small kernels limit receptive fields, whereas large kernels hinder deep-feature exploration and may require deeper, more computationally expensive networks.
  • Limitations of Existing Methods: Feature extraction is difficult because EEG contains substantial noise, and multi-class methods may simply stack spatially filtered feature channels without modeling their differing importance.FFT, CWT, and CSP address frequency, time-frequency, and spatial characteristics, respectively.
  • Proposed Method: S3T applies attention to feature channels and temporal slices, transforming spatially filtered EEG into a more distinguishable representation for classification.The framework is designed to focus selectively on relevant channels and capture dependencies across time.
  • Reported Outcome: Detailed experiments on public datasets report state-of-the-art-level competitiveness with fewer parameters.The contribution is presented as a tiny attention-based framework for learning spatial and temporal EEG features.

II. RELATED WORKS

EEG decoding has progressed from classical classifiers to CNNs and RNNs, but attention-based models offer another way to capture broader dependencies. S3T uses attention as its main feature-learning mechanism while retaining spatial and temporal processing components.

  • Classical and Deep Learning Methods: Classical EEG decoding used LDA, SVM, and MLP to model feature distributions or input-output relationships.These approaches separate categories through projections, hyperplanes, or hidden-layer mappings.
  • Classical and Deep Learning Methods: CNNs learn local feature dependencies effectively but their receptive field and computational cost depend strongly on convolution-kernel design.Small kernels restrict context, while large kernels can hinder deeper feature exploration.
  • Attention-Based Methods: Attention mechanisms were developed for sequence and image modeling by relating different positions or patches, motivating their use for EEG dependencies.Prior EEG studies also combined CNNs or LSTMs with attention to synthesize sliced signals or graph representations.
  • S3T: S3T is presented as a small attention-based backbone that transforms EEG using spatial and temporal information.Its stated design emphasizes global dependencies, feature-channel importance, and low computational cost.
  • Preprocessing: Raw EEG preprocessing includes segmentation, band-pass filtering, and standardization, with some calibration and artifact-removal procedures omitted.The described implementation filters signals to 4–40 Hz and uses training-data statistics for z-score standardization.

B. Spatial Filter

The spatial-filter stage adapts CSP to multi-class EEG by constructing one-versus-rest sub-filters and stacking them. This increases category-specific spatial differences while retaining temporal information and reducing computation through dimensional selection.

  • One-Versus-Rest CSP: CSP is extended to multi-classification with one-versus-rest decomposition into N binary tasks.Each task contrasts one category against all remaining categories.
  • Covariance Construction: For each one-versus-rest task, trial covariance matrices are averaged separately for the target category and the remaining categories.The resulting covariance matrices characterize joint variability between EEG channels.
  • Spatial Filter Derivation: Eigendecomposition and whitening jointly diagonalize the covariance structure to derive a spatial filter maximizing the difference between the target category and the rest.The spatial filter is based on eigenvectors and eigenvalues obtained from the transformed covariance matrices.
  • Filter Selection: The first S rows from each one-versus-rest output are retained as sub-filters to reduce computational complexity.The selected sub-filters are stacked into the final spatial filter W, which transforms preprocessed data X into filtered data Z.

C. Spatial Transforming

The spatial-transforming block uses attention to learn feature-channel importance rather than treating channels identically. Scaled dot-product attention produces weighted representations and uses a residual connection to support information flow.

  • Feature-Channel Attention: Feature-channel attention estimates relationships among channels and assigns importance scores so relevant channels receive greater emphasis.The mechanism is motivated by the inefficiency and mutual interference that can result when channel importance is ignored.
  • Attention Operation: Scaled dot-product attention computes a weighted representation from query, key, and value matrices.Softmax-generated weights are applied to the values through dot products.
  • Attention Operation: A residual connection combines the input and output to help gradients flow through the framework.The query, key, and value matrices are packed from vectors for simultaneous calculation.

D. Temporal Transforming

The temporal transforming stage uses attention to perceive global dependencies across EEG trial segments. After spatial feature-channel weighting, it reduces complexity, encodes temporal position, and applies feed-forward processing.

  • D. Temporal Transforming: The model applies attention to capture global temporal dependencies between different parts of an EEG trial.This reflects the assumption that brain-driven behavior unfolds as a complete process with relationships across the trial.
  • D. Temporal Transforming: After feature-channel weighting, the data is compressed to one dimension of length T to reduce computational complexity.
  • D. Temporal Transforming: Temporal segments are used because a segment better reflects EEG trends than a single sample point.
  • D. Temporal Transforming: The temporal attention block uses multi-head transformations with concatenated heads and a final linear output transformation.The surrounding architecture also includes a feed-forward block with two fully connected layers and GeLU activation.

E. Classifier

The classifier averages the temporally transformed slices, normalizes the pooled representation, and maps it to category probabilities using a fully connected layer and Softmax.

  • E. Classifier: Global pooling averages all temporal slices before layer normalization and a fully connected classification layer.
  • E. Classifier: The fully connected layer has one output neuron for each category, and Softmax converts its outputs into predicted probabilities.
  • E. Classifier: The classifier is trained with cross-entropy loss over M trials and N categories.The real label and predicted probability are defined for each trial-category pair.

IV. EXPERIMENTS AND RESULTS

The study evaluates S3T on datasets 2a and 2b of BCI Competition IV, using motor-imagery EEG recordings from public benchmark datasets with different task and electrode configurations.

  • IV. EXPERIMENTS AND RESULTS: S3T was evaluated on datasets 2a and 2b of the public BCI Competition IV benchmark.
  • IV. EXPERIMENTS AND RESULTS: Dataset 2a contains nine subjects, 22 electrodes, four motor-imagery tasks, and 288 trials per session.The tasks involve imagined movement of the left hand, right hand, both feet, and tongue.
  • IV. EXPERIMENTS AND RESULTS: Dataset 2b contains nine subjects recorded with three bipolar electrodes for left- and right-hand motor imagery.
  • IV. EXPERIMENTS AND RESULTS: For dataset 2b, each subject contributed two sessions without visual feedback and three with visual feedback, using 120 trials per session.Trials retained the 3–7 second segment.

B. Experiment details

Experiments assess S3T with confusion matrices, category-level metrics, and comparisons against representative classical and deep-learning baselines. The reported results indicate reliable classification, strong comparative performance, robustness, and lower computational cost from fewer parameters.

  • B. Experiment details: The experiments use ten-fold cross-validation with subject-specific models after removing electrooculogram channels and applying no artifact removal beyond band-pass filtering.
  • B. Experiment details: Confusion matrices show predicted labels horizontally and true labels vertically, while Table I reports accuracy, precision, recall, specificity, and F-score for each category.
  • B. Experiment details: The confusion-matrix results indicate good classification ability across categories without substantial bias.
  • B. Experiment details: S3T is compared with FBCSP, ConvNet, EEGNet, C2CM, CNN+LSTM, and DFL.
  • B. Experiment details: S3T reportedly achieves superior accuracy on both datasets with limited parameters and low standard deviation, indicating effective and robust EEG decoding.
  • B. Experiment details: In dataset 2a, S3T significantly improves over CNN+LSTM and DFL except for the weak results on Subject 05.The reported p-values are 0.0210 for CNN+LSTM and 0.0104 for DFL.
  • B. Experiment details: Fewer parameters reduce computational complexity and cost compared with parameter-heavy CNN-based methods.The paper characterizes the resulting performance as more cost-effective.

E. Ablating Study

The ablation study shows that temporal transforming is the backbone of S3T, while spatial transforming, position encoding, and feed-forward blocks also improve performance.

  • Temporal transforming is the backbone of S3T, with its removal reducing mean accuracy by 35.93%.
  • Removing spatial transforming decreases mean accuracy by 3.33%, with distinct improvements for poorly discriminated subjects 5 and 6.
  • Position encoding improves mean accuracy by 11.11%, compensating for using only the attention mechanism and highlighting the value of position information.
  • Adding feed-forward blocks increases mean accuracy by 2.22% and gives the model more robust learning capability.

F. Parameter Sensitivity

Parameter sensitivity tests vary slice size and position-encoding kernel size on the first subjects of datasets 2a and 2b. Performance is sensitive to small values but fluctuates only slightly at larger values, indicating stability.

  • Small slice sizes cause a sharp performance drop, while smaller slices are also more susceptible to noise or outliers.Slicing reduces computation and enables attention over many short sequences that better reflect signal state.
  • The sensitivity tests independently vary the position-encoding kernel and slice size on the first subject of datasets 2a and 2b.
  • Smaller position-encoding kernels expressly reduce performance because they capture valuable sequence-position information over too narrow a range.
  • At larger position kernels and slice sizes, results fluctuate slightly, confirming that the method remains stable as these parameters change.

V. DISCUSSION

S3T combines attention-based spatial and temporal transforming with limited convolution to capture EEG dependencies while controlling model size. The discussion reports competitive decoding, explains design choices, and identifies subject-specific evaluation and preliminary hyperparameter selection as limitations.

  • Method: Attention mechanisms address CNNs’ limited perception of global dependencies in EEG, while feature-channel attention weights spatially filtered channels.Temporal attention operates on sliced signal segments, producing a more distinguishable representation for classification.
  • Method: Temporal transforming is the method’s backbone: slicing continuous sample points into segments improves effectiveness and lowers computation cost relative to directly processing sample points.The slices also reduce the influence of individual outliers by emphasizing short continuous regions.
  • Method: Convolution remains necessary because slice-based attention partly ignores sequence order, whereas nearby-area convolution supplies positional information.A convolutional position-encoding component complements attention’s global dependency modeling.
  • Efficiency: Feature-channel compression and averaging all temporal slices reduce parameters, enabling classification with a small fully connected layer.The authors link these operations to maintaining a relatively low model scale while preserving good accuracy.
  • Limitations: The evaluation is limited to subject-specific experiments, while cross-subject ability remains insufficiently explored; dropout and batch-size choices came from preliminary experiments.The authors propose cross-subject tests and data augmentation as future work.
  • Results: S3T reaches state-of-the-art-level multi-category EEG decoding with fewer parameters and is presented as a potential complement to commonly used deep-learning methods.The paper also reports module effectiveness and visualization evidence supporting spatial- and temporal-feature transformation.
Loading 2106.11170v1…