Source-linked AI summary
BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data
Demetres Kostas, Stephane Aroca-Ouellette, Frank Rudzicz
TL;DR
BCI EEG DNNs must generalize across variable subjects, hardware, sessions, and tasks despite limited labelled data. The paper adapts self-supervised sequence-learning methods from speech and language modelling to pre-train BENDR representations and a transformer architecture on EEG. A single pre-trained model transfers to novel EEG sequences and supports fine-tuning across downstream BCI and EEG classification tasks, while remaining subject to architectural and dataset-design limitations identified by the authors.
Problem
BCI EEG DNNs have limited labelled data and struggle to learn features that generalize across subjects, sessions, hardware, and task-specific contexts.
Method
The paper adapts wav2vec 2.0-style self-supervised sequence learning, using contrastive prediction, convolutional downsampling, BENDR representations, and a transformer encoder for EEG.
Results
A single pre-trained model transfers to novel EEG sequences across hardware, subjects, and tasks, and its representations and full architecture can be fine-tuned for varied downstream BCI and EEG classification tasks.
Takeaways & Limitations
Self-supervised EEG pre-training provides a broader alternative to task-specific transfer approaches by leveraging unlabelled data across people, sessions, and tasks.
Takeaways & Limitations
The authors note unresolved architectural choices, including temporal resolution and replacing additive convolution with explicit positional encodings, and the difficulty of constructing a coherent, balanced cross-paradigm dataset.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) used for brain-computer-interface (BCI) classification are commonly expected to learn general features when trained across a variety of contexts, such that these features could be fine-tuned to specific contexts. While some success is found in such an approach, we suggest that this interpretation is limited and an alternative would better leverage the newly (publicly) available massive EEG datasets. We consider how to adapt techniques and architectures used for language modelling (LM), that appear capable of ingesting awesome amounts of data, towards the development of encephalography modelling (EM) with DNNs in the same vein. We specifically adapt an approach effectively used for automatic speech recognition, which similarly (to LMs) uses a self-supervised training objective to learn compressed representations of raw data signals. After adaptation to EEG, we find that a single pre-trained model is capable of modelling completely novel raw EEG sequences recorded with differing hardware, and different subjects performing different tasks. Furthermore, both the internal representations of this model and the entire architecture can be fine-tuned to a variety of downstream BCI and EEG classification tasks, outperforming prior work in more task-specific (sleep stage classification) self-supervision.
1 Introduction
Raw-EEG DNNs face limited relevant labels, substantial subject and session variability, and task-specific features, weakening transferability. The paper proposes self-supervised sequence learning inspired by language and speech modelling to learn more broadly transferable EEG representations from unlabelled data.
- 1 Introduction: Raw-EEG DNNs must jointly learn feature extraction and classification from limited, high-dimensional labelled data.This creates a central challenge for BCI applications, where useful features can be difficult to learn directly from raw sequences.
- 1 Introduction: Subject and session variability causes many model types’ classification performance to vary, revealing reliance on context-specific characteristics.Different BCI paradigms also favor different discriminative features, such as P300 versus motor imagery.
- 1 Introduction: Existing EEG DNNs lack known transferable properties or operations that extend easily across subjects, sessions, or tasks.Unlike computer vision, raw-EEG modelling lacks an established account of transferable low-level features.
- 1 Introduction: Shallower BCI networks constrain the range of learnable features and have not uniformly outperformed feature-engineering approaches.Their limited use of nonlinear depth may leave learned features insufficient or undesirable for some DNN applications.
- 1 Introduction: Self-supervised sequence learning can use unlabelled data from many people, sessions, and tasks to model EEG with less variability across confounding factors.The proposed direction is motivated by limited BCI labels and successful self-supervised approaches in language, speech, and image recognition.
- 1 Introduction: The paper adapts wav2vec 2.0 to EEG, encoding arbitrary EEG segments as learned feature-vector sequences called BENDR for downstream classification.The approach combines contrastive sequence learning with representations and a transformer-based sequence model that can support supervised fine-tuning.
2 Materials and methods
The study pre-trains BENDR on diverse EEG data using a two-stage convolutional-transformer architecture and a masked, contrastive self-supervised objective, then evaluates transfer across varied downstream datasets.
- Data: The pre-training corpus is designed to represent EEG broadly across subjects, sessions, tasks, timescales, hardware, and configurations.The Temple University Hospital EEG Corpus was identified as the closest publicly accessible dataset to this ideal.
- Data: Downstream evaluation combines publicly accessible BCI classification datasets with a sleep-stage classification task for comparison with prior self-supervised work.The battery supports comparisons with task-specific DNNs and provides insight into generality beyond event- or trial-based BCI data.
- Preprocessing: Preprocessing standardizes channel amplitudes, sampling frequencies, and electrode sets so downstream sequences match the pre-training configuration.A reduced Deep1010 mapping consistently indexes 19 UI 10/20 EEG channels, while resampling targets 256 Hz.
- Preprocessing: Pre-training extracts non-overlapping 60-second sequences containing 15,360 samples, while the architecture remains ostensibly agnostic to downstream sequence length.Early testing indicated better performance with larger sequences, although downstream datasets used shorter sequences.
- Model architecture: BENDR uses convolutional downsampling to encode raw EEG, followed by a transformer encoder that maps the encoded sequence into task-relevant representations.The first stage uses short-receptive-field 1D convolutions; the second uses layered multi-head self-attention.
- Model architecture: For sequence classification, a fixed prepended token enables the transformer to represent the entire sequence for downstream prediction.The token is a vector filled with −5, distinct from the input signal range.
- Model architecture: The architecture adapts wav2vec 2.0 to EEG by retaining multichannel input and omitting BENDR quantization for pre-training targets.The EEG encoder produces approximately 2.67 Hz representations from 256 Hz input, with each vector length 512.
- Pre-training: Pre-training masks BENDR positions and optimizes cosine-similarity discrimination between the transformer output, the correct BENDR vector, and 20 same-sequence distractors.The temperature factor κ is 0.1, and an additional mean squared BENDR activation term has weight 1.
3 Results
Pre-training generalized across datasets and subjects, while longer evaluation contexts improved contrastive accuracy. Fine-tuning transferred broadly to downstream EEG tasks, although performance was not uniformly state-of-the-art.
- Pre-training generalization: Contrastive accuracy varied little across the first four datasets and within each dataset, indicating limited subject-related variability during pre-training evaluation.P300 underperformed partly because it used the shortest evaluation sequence.
- Downstream fine-tuning: Fine-tuned linear classification bypassing the transformer after pre-training achieved the highest performance on four of five datasets.Using the transformer for classification performed similarly and was best on P300; confidence intervals always overlapped.
- Downstream fine-tuning: Fine-tuned BENDR variants outperformed prior task-specific contrastive self-supervision on the SSC dataset.The comparison concerns Banville et al.'s temporal-shuffling and relative-positioning tasks, which underperformed full supervision on the full SSC dataset.
- Pre-training generalization: Longer evaluation sequences made the contrastive task easier, with all datasets showing consistently similar performance at 20 seconds.Figure 2 plots contrastive accuracy against logarithmic evaluation length and shows accuracy rising for longer sequences.
- Downstream fine-tuning: Fine-tuned approaches were reasonably competitive with prior work on MMI despite using 19 rather than 64 channels, while pre-training showed broad transferability.Most results were not necessarily state-of-the-art, and randomly initialized models were more prone to overfitting.
4 Discussion
BENDR’s downstream performance was strongest with fine-tuned configurations, while the authors identify architectural, temporal-modelling, and dataset-construction limitations for future work.
- Model applicability: Transformer-based EEG models had not previously been assessed in this setting, and the randomly initialized full architecture was ineffective.The authors suggest effective use may require pre-training or enough data.
- Downstream performance: Fine-tuned linear classification was highest performing on four of five datasets, while transformer-based classification performed similarly within overlapping confidence intervals.Transformer-based classification was highest performing on the P300 dataset.
- Limitations and future work: Future improvements should integrate spatial information more effectively because the present degree to which BENDR uses spatial information is unclear.The authors specifically propose better isolating temporal and spatial operations and evaluating larger, many-channel datasets such as MASS.
- Limitations and future work: The position encoder’s receptive field covered a little over 9 seconds, exceeding the input width of all but the sleep-classification task.The authors plan to balance larger downsampling and longer sequences against the harder problem of compressing more data.
- Limitations and future work: An aggregate multi-task EEG dataset could support more ImageNet-like pre-training, but coherent labels and long-tailed class distributions would require substantial adjustment.The authors note unresolved labeling questions across BCI paradigms and sleep data.
5 Conclusion
The paper proposes MLM-like self-supervised pre-training for BCI and EEG DNNs as an alternative to ImageNet-style transfer learning. It argues that this approach can leverage massive unlabelled EEG datasets and produce representations suited to different users and sessions.
- The paper proposes MLM-like self-supervised pre-training as a transfer-learning approach for BCI and EEG DNNs.
- The approach is intended to leverage potentially immense unlabelled data rather than relying on limited applicable labelled data for ImageNet-style pre-training.
- The resulting representations appear suited to different users and sessions, supporting broader deployment of computation and massive data scales in EEG and BCI.
A Downstream hyperparameters
Table 3 lists hyperparameters that varied between datasets while remaining fixed across the different model configurations.
- Hyperparameters varied between datasets but were not changed between model configurations.