PaperlayerView exact revision on arXiv

Source-linked AI summary

EEG-Deformer: A Dense Convolutional Transformer for Brain-computer Interfaces

Yi Ding, Yong Li, Hao Sun, Rui Liu, Chengxuan Tong, Chenyu Liu, Xinliang Zhou, Cuntai Guan

arXiv:2405.00719v2eess.SPcs.LGq-bio.NC

TL;DR

EEG decoding requires perceiving subtle temporal dynamics, but existing convolutional Transformers do not explicitly capture coarse- and fine-grained patterns together. EEG-Deformer combines a coarse-to-fine Transformer with dense purification of multi-level temporal information. Across three cognitive tasks, the authors report that it outperforms current state-of-the-art methods.

  • Problem

    EEG signals vary across subjects and are susceptible to interference, while existing convolutional Transformers may not capture coarse- and fine-grained temporal patterns together or exploit intermediate-layer information.

    Source: §I · pp. 1–2

  • Method

    EEG-Deformer combines a CNN-based shallow encoder, HCT blocks with a fine-grained temporal-learning branch, and dense information purification of multi-level temporal representations.

    Source: §I · p. 2

  • Results

    Across attention, fatigue, and cognitive workload classification on three public datasets, EEG-Deformer demonstrates superiority over current state-of-the-art methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    The evaluation supports EEG-Deformer for cross-subject EEG classification on three benchmark datasets.

    Source: §VI · p. 9

    Evaluation does not cover all commonly used EEG-classification tasks, and performance standard deviations are higher than certain baselines on some datasets.

    Source: H. Limitations and future work · p. 9

Abstract

from arXiv · show

Effectively learning the temporal dynamics in electroencephalogram (EEG) signals is challenging yet essential for decoding brain activities using brain-computer interfaces (BCIs). Although Transformers are popular for their long-term sequential learning ability in the BCI field, most methods combining Transformers with convolutional neural networks (CNNs) fail to capture the coarse-to-fine temporal dynamics of EEG signals. To overcome this limitation, we introduce EEG-Deformer, which incorporates two main novel components into a CNN-Transformer: (1) a Hierarchical Coarse-to-Fine Transformer (HCT) block that integrates a Fine-grained Temporal Learning (FTL) branch into Transformers, effectively discerning coarse-to-fine temporal patterns; and (2) a Dense Information Purification (DIP) module, which utilizes multi-level, purified temporal information to enhance decoding accuracy. Comprehensive experiments on three representative cognitive tasks-cognitive attention, driving fatigue, and mental workload detection-consistently confirm the generalizability of our proposed EEG-Deformer, demonstrating that it either outperforms or performs comparably to existing state-of-the-art methods. Visualization results show that EEG-Deformer learns from neurophysiologically meaningful brain regions for the corresponding cognitive tasks. The source code can be found at https://github.com/yi-ding-cs/EEG-Deformer.

I. INTRODUCTION

EEG decoding depends on subtle temporal dynamics, but CNN-Transformer methods do not explicitly capture coarse- and fine-grained patterns together and may overlook intermediate-layer information. EEG-Deformer combines a hierarchical coarse-to-fine Transformer with dense information purification, and experiments across three cognitive tasks report superiority over current state-of-the-art methods.

I. INTRODUCTION

  • EEG signals vary across subjects and face external interference, while existing methods may miss fine-grained or long-term dynamics and intermediate-layer temporal information.The paper identifies movement artifacts, eye blinks, and environmental factors as sources of interference.Source: §I · pp. 1–2
  • EEG-Deformer’s HCT integrates a fine-grained temporal-learning branch with Transformer representations to capture coarse- and fine-grained temporal dynamics.A 1-D CNN in the FTL branch captures short-period dynamics, whose representations are adaptively fused with coarse-grained Transformer representations.Source: §I · p. 2
  • DIP densely transmits intermediate HCT representations to the final representation through IP-Units that apply log power to fine-grained features.The paper states that log power represents activity in filtered signals, reduces dimensions and learnable parameters, and retains critical frequency characteristics related to brain activity.Source: §I · p. 2
  • Across three public datasets covering attention, fatigue, and cognitive workload classification, EEG-Deformer’s results demonstrate superiority over current state-of-the-art methods.Source: §I · p. 2

II. RELATED WORK

CNN methods learn EEG features directly, but short temporal kernels limit their ability to capture long-term patterns. CNN-Transformer methods capture global dependencies, yet often do not capture both coarse- and fine-grained temporal patterns or use information across layers; EEG-Deformer combines a shallow feature encoder, HCT blocks, and dense information purification.

A. CNNs for EEG Decoding

  • Short temporal kernels in 1-D CNN methods limit their ability to capture long-term EEG patterns.The section describes DeepConvNet, EEGNet, and TSception as CNN approaches to EEG feature extraction and temporal decoding.Source: A. CNNs for EEG Decoding · p. 2

B. Transformers for BCI

  • CNN-Transformer methods capture global dependencies, but often miss either coarse- or fine-grained temporal patterns and multi-level information across layers.Source: B. Transformers for BCI · p. 2

III. METHOD

  • EEG-Deformer uses a shallow feature encoder, HCT blocks, and DIP to extract temporal dynamics at different timescales and fuse representations from each HCT block.The shallow encoder uses CNN layers and a tokenizer; its representation is rearranged and position-encoded before Transformer processing.Source: §III · pp. 2–3, Figure 2 · p. 3, Figure 3 · p. 3

A. Shallow Feature Encoders

The shallow encoder converts EEG into temporally positioned tokens, while HCT blocks combine coarse temporal learning with a fine-grained CNN branch.

A. Shallow Feature Encoders

  • A two-layer CNN extracts shallow temporal and spatial EEG features, then rearranges them into tokens and adds learnable positional encoding.Temporal kernels are sized for approximately 100 ms; spatial kernels span all EEG channels.Source: A. Shallow Feature Encoders · p. 3

B. Hierarchical Coarse-to-fine Transformer

  • HCT combines Transformer self-attention over tokens for coarse temporal information with a CNN branch for fine-grained EEG features.The branches are summed to form the HCT output; the FTL branch uses a short 1-D CNN kernel moving along time.Source: B. Hierarchical Coarse-to-fine… · pp. 3–4

C. Dense Information Purification

Dense Information Purification extracts frequency-informed features from multiple HCT layers and combines them for classification.

C. Dense Information Purification

  • Densely connected IP-Units progressively extract multi-level temporal information and reduce the size of bypassed representations.They use a power layer to purify information and encode EEG frequency information in learned 1-D representations.Source: C. Dense Information Purification · p. 4
  • The purified information from all HCT layers is concatenated with the flattened final HCT output before a linear classifier projects it onto class labels.The concatenated embedding is the classifier's input.Source: C. Dense Information Purification · p. 4

A. Datasets and Pre-processing

The evaluation uses three EEG datasets covering cognitive attention, driving fatigue, and mental workload.

A. Datasets and Pre-processing

  • The three benchmark datasets represent cognitive attention, driving fatigue, and cognitive workload classification tasks.Dataset I uses a discrimination/selection response task; Dataset II records EEG during virtual-reality driving; Dataset III includes serial subtraction and baseline EEG.Source: A. Datasets and Pre-processing · pp. 4–5
  • Dataset I uses 4-second attention segments with 50% overlap after downsampling EEG to 200 Hz.Only the first 20 seconds of each attention trial are used, alongside rest periods.Source: A. Datasets and Pre-processing · p. 5
  • Driving-fatigue trials use the 3 seconds before lane-departure events, with reaction time used to measure fatigue levels.Local reaction time is trial-specific; global reaction time averages local reaction times over the preceding 90-second window.Source: A. Datasets and Pre-processing · p. 5
  • Mental-workload trials last 60 seconds, and the final 60 seconds of rest EEG provide the low-workload data.Trials are segmented with a 4-second sliding window and a 2-second step.Source: A. Datasets and Pre-processing · p. 5

B. Baselines

The baselines span graph networks, CNNs, and Transformers, providing comparisons across different approaches to EEG decoding.

B. Baselines

1) DGCNN:

  • DGCNN learns an adjacency matrix to update EEG-channel relationships during training.Source: 1) DGCNN · p. 6

2) LGGNet:

  • LGGNet combines multi-scale temporal convolutions and attentive fusion with graph filtering of local and global EEG relationships.Source: 2) LGGNet · p. 6

3) EEGNet:

  • EEGNet is a compact CNN that uses depth-wise and separable convolutions for EEG feature extraction.Source: 3) EEGNet · p. 6

4) TSception:

The section describes additional CNN and Transformer baselines, then specifies the subject-independent evaluation protocol and metrics.

4) TSception:

  • TSception combines multi-scale temporal convolutions, asymmetric spatial layers, and feature fusion to extract EEG patterns.Source: 4) TSception · p. 6

5) EEG-ViT:

  • EEG-ViT tokenizes non-overlapping temporal EEG segments with a linear projection for Transformer processing.Source: 5) EEG-ViT · p. 6

6) SSVEPformer:

  • SSVEPformer uses channel combination, an encoder, and an MLP head to extract spectrum information from complex-spectrum EEG.Source: 6) SSVEPformer · p. 6

7) EEG-Transformer:

8) EEG-Conformer:

  • EEG-Conformer uses 1-D CNNs for local features and self-attention for global correlations in temporal features.Source: 8) EEG-Conformer · p. 6

C. Experiment Settings

  • All datasets use leave-one-subject-out evaluation, with one subject held out for testing and the remaining data split 80%/20% for training and validation; ACC and Macro-F1 are reported.Source: C. Experiment Settings · p. 6

D. Implementation Details

EEG-Deformer leads the reported attention, fatigue, and workload results; the section also describes implementation settings, IP-Unit effects, and task-specific saliency regions.

D. Implementation Details

A. Classification Results

  • 82.72% accuracy and 82.36% macro-F1 lead the attention results; versus EEG-Conformer, gains are 2.92% accuracy and 3.35% macro-F1.Source: A. Classification Results · p. 7
  • 79.32% accuracy and 75.83% macro-F1 lead fatigue classification, exceeding EEG-Conformer by 4.96% accuracy and 4.18% macro-F1.Source: A. Classification Results · p. 7
  • 73.18% accuracy and 69.99% macro-F1 lead workload classification; TSception ranks second, followed closely by EEG-Conformer.Source: A. Classification Results · p. 7
  • Removing the first IP-Unit most affects datasets I and II, while the third is most important for dataset III; deeper IP-Units appear more important with longer EEG segments.Source: Figure 8 · p. 8
  • The authors report that convolutional Transformers excel over traditional Transformers and attribute EEG-Deformer's advantage over EEG-Conformer to coarse-to-fine temporal capture and multi-layer information use.Source: A. Classification Results · p. 8

B. Ablation

Ablations indicate contributions from the FTL branch, dense connections, and IP-Units, with IP-Unit importance varying across datasets.

B. Ablation

  • Removing FTL causes the largest overall performance decrease; removing IP-Units causes the second-largest decreases on datasets I and II, while removing dense connections reduces ACC and macro-F1 across all datasets.Source: B. Ablation · p. 8

C. Ablation of IP-Units

  • The first IP-Unit matters most on datasets I and II, while the third matters most on dataset III; deeper IP-Units appear more important for longer input segments.Source: C. Ablation of IP-Units · p. 8
  • The model comparison plots ACC against MACs, with bubble size representing model size; EEG-Deformer is described as balancing accuracy and computational complexity.Source: Figure 9 · p. 8

D. Effect of Different Types of IP-Unit

EEG-Deformer outperforms the evaluated baselines across three benchmark datasets, while ablations favor power-based purification and applying IP-Unit to the FTL output. Comparisons show a promising accuracy-efficiency balance, and saliency maps highlight task-related brain regions; evaluation remains limited to selected tasks.

D. Effect of Different Types of IP-Unit

  • Power-based information purification yields the highest accuracy and macro-F1 across all three datasets among the tested methods.The alternatives are mean and standard deviation (std) purification.Source: D. Effect of Different Types of IP-Unit · p. 9

E. Effect of Different Locations of The IP-Unit

  • Applying IP-Unit to the FTL output produces the highest decoding performance across all three datasets among the tested placements.The compared locations are the fused output, coarse-grained representation, and fine-grained representation.Source: E. Effect of Different Locations of… · p. 9

F. The Parameters Comparison of EEG-Deformer and Other Baseline Methods.

  • EEG-Deformer achieves a promising accuracy-efficiency balance across three datasets, with its relative model size and MACs varying by dataset.It uses fewer MACs than similarly sized models on dataset I; on dataset II it has the highest performance with relatively smaller size and fewer MACs; on dataset III it uses more MACs but is smaller than models requiring more MACs.Source: F. The Parameters Comparison of… · p. 9

G. Visualization

  • Saliency maps show task-related informative regions: frontal and parietal for attention, frontal, temporal, and parietal for fatigue, and frontal and parietal for workload.The maps cover five randomly selected subjects and an across-subject average; the reported regional patterns align with prior findings.Source: G. Visualization · p. 9

H. Limitations and future work

  • The evaluation covers selected EEG classification tasks rather than all commonly used tasks, and performance variability is higher than some baselines on some datasets.The authors identify broader task coverage and improved robustness as future-work priorities.Source: H. Limitations and future work · p. 9

VI. CONCLUSION

  • EEG-Deformer consistently outperforms baselines across three benchmark datasets for cross-subject EEG classification.The conclusion describes the model as a convolutional Transformer using HCT blocks to capture coarse- and fine-grained temporal dynamics and dense purification of multi-level information.Source: §VI · p. 9

Related papers

Loading 2405.00719v2…