PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI

Wei-Bang Jiang, Li-Ming Zhao, Bao-Liang Lu

arXiv:2405.18765v1cs.LG

TL;DR

Existing EEG models are typically task- and dataset-specific, while EEG data remain limited, heterogeneous, and noisy. LaBraM addresses this with channel-patch processing, neural tokenization, and masked pre-training on over 2,500 hours from about 20 datasets. It outperformed compared state-of-the-art methods across four downstream BCI tasks.

  • Problem

    EEG research lacks sufficiently large, consistently formatted data and a general representation-learning paradigm for models that must span diverse channels, lengths, and tasks.

  • Method

    LaBraM segments EEG into channel patches, learns neural codes with vector-quantized neural spectrum prediction, and pre-trains a neural Transformer by predicting masked-patch codes.

  • Results

    LaBraM outperformed all compared state-of-the-art methods on abnormal detection, event type classification, emotion recognition, and gait prediction.

  • Takeaways & Limitations

    Unsupervised pre-training on tremendous, diverse EEG data can produce a model validated across multiple downstream BCI tasks.

  • Takeaways & Limitations

    LaBraM still has a large scale gap from contemporary large vision and language models, requires costly full fine-tuning, and is trained only on unimodal EEG data.

Abstract

from arXiv · show

The current electroencephalogram (EEG) based deep learning models are typically designed for specific datasets and applications in brain-computer interaction (BCI), limiting the scale of the models and thus diminishing their perceptual capabilities and generalizability. Recently, Large Language Models (LLMs) have achieved unprecedented success in text processing, prompting us to explore the capabilities of Large EEG Models (LEMs). We hope that LEMs can break through the limitations of different task types of EEG datasets, and obtain universal perceptual capabilities of EEG signals through unsupervised pre-training. Then the models can be fine-tuned for different downstream tasks. However, compared to text data, the volume of EEG datasets is generally small and the format varies widely. For example, there can be mismatched numbers of electrodes, unequal length data samples, varied task designs, and low signal-to-noise ratio. To overcome these challenges, we propose a unified foundation model for EEG called Large Brain Model (LaBraM). LaBraM enables cross-dataset learning by segmenting the EEG signals into EEG channel patches. Vector-quantized neural spectrum prediction is used to train a semantically rich neural tokenizer that encodes continuous raw EEG channel patches into compact neural codes. We then pre-train neural Transformers by predicting the original neural codes for the masked EEG channel patches. The LaBraMs were pre-trained on about 2,500 hours of various types of EEG signals from around 20 datasets and validated on multiple different types of downstream tasks. Experiments on abnormal detection, event type classification, emotion recognition, and gait prediction show that our LaBraM outperforms all compared SOTA methods in their respective fields. Our code is available at https://github.com/935963004/LaBraM.

1 INTRODUCTION

EEG deep-learning models are constrained by small, heterogeneous datasets and task-specific designs, motivating LaBraM’s unified, unsupervised pre-training approach for generic EEG representations. LaBraM uses channel patches, neural tokenization, and Transformer modeling across diverse configurations and downstream BCI tasks.

  • Challenges: EEG models often target specific datasets and tasks because acquisition systems create mismatched channels and variable signal lengths.Expensive collection and limited task-specific data further constrain model scale and representation learning.
  • Challenges: Low signal-to-noise ratios and competing temporal and spatial characteristics make effective raw-EEG representation learning difficult.These challenges contribute to continued reliance on manually designed EEG features.
  • Approach: LaBraM was pre-trained on more than 2,500 hours of diverse EEG data collected across about 20 datasets.The authors describe this as the first EEG pre-training effort using such extensive and varied datasets.
  • Approach: Channel patches and flexible spatial and temporal embeddings let one LaBraM handle EEG inputs with varying channel counts and time lengths.The neural Transformer captures temporal and spatial features through patch-based processing.
  • Approach: A vector-quantized neural spectrum predictor creates compact neural vocabulary, which masked EEG modeling uses to pre-train the Transformer.Masked patches are predicted from visible patches through their neural tokens.
  • Evaluation: LaBraM surpassed all compared state-of-the-art methods across abnormal detection, event type classification, emotion recognition, and gait prediction.The reported evaluations cover four representative downstream BCI tasks.

2 METHOD

LaBraM handles variable EEG configurations by converting channel signals into patches, enriching them with temporal and spatial embeddings, and processing them with a neural Transformer. A vector-quantized neural tokenizer supplies discrete targets for masked EEG pre-training, with symmetric masking reusing token representations for efficiency.

  • Neural Transformer: EEG signals are segmented into non-overlapping channel patches, allowing inputs with varying channel counts and time lengths.Each sample contains one w-length patch per channel and temporal position.
  • Neural Transformer: A temporal convolution encoder converts each EEG patch into a patch embedding before Transformer-based patch interaction.The encoder uses 1-D convolution, group normalization, and GELU activation.
  • Neural Transformer: Learnable temporal and spatial embeddings provide position information for each patch, while the Transformer encoder models the resulting embedding sequence.Temporal embeddings index time patches and spatial embeddings correspond to EEG channels.
  • Neural Tokenizer: The neural tokenizer quantizes patch representations using nearest codebook embeddings and trains them to reconstruct Fourier-spectrum amplitude and phase.The decoder predicts spectrum amplitude and phase from normalized discrete embeddings using mean squared error.
  • Pre-training LaBraM: Masked EEG modeling replaces a proportion of patches with a learnable mask token and predicts their neural tokens from Transformer hidden vectors.Symmetric masking performs the complementary masking pattern while reusing the same discrete representations, reducing tokenizer overhead and increasing masking perspectives.

3 EXPERIMENTS

LaBraM is evaluated on multiple EEG tasks and datasets using several model scales, pre-training settings, and data sizes. Results indicate strong downstream performance, limited sensitivity to including downstream recordings during pre-training, and benefits from scaling.

  • Evaluation setup: LaBraM is systematically evaluated on downstream EEG datasets including TUAB abnormal detection and TUEV event type classification.Additional BCI-task results are reported in Appendix F.
  • State-of-the-art comparison: LaBraM-Base outperformed all compared baselines on TUAB and TUEV evaluation metrics, with particularly strong improvement on TUEV’s multi-class classification task.
  • Model scaling: Larger LaBraM variants performed better in the reported ordering: LaBraM-Huge, followed by LaBraM-Large and LaBraM-Base.The variants contain 5.8M, 46M, and 369M parameters, respectively.
  • Pre-training data overlap: Downstream performance was not significantly affected by whether TUAB and TUEV recordings were included in pre-training.The downstream recordings were disjoint from the pre-training datasets.
  • Data scaling: With 500 hours of training, LaBraM-Base exceeded the 2,500-hour model on TUAB and approached over 90% of its TUEV performance.Large-model gains slowed after 1,000 hours, whereas the Huge model continued an upward performance trend as data increased.

4 CONCLUSION

The paper concludes that LaBraM learns universal EEG embeddings through unsupervised pre-training on diverse data and supports downstream analysis across varied EEG tasks. It positions the model as a foundation for improving EEG representation learning and generalizability.

  • LaBraM learns universal EEG embeddings through unsupervised pre-training on over 2,500 hours of diverse EEG data.
  • Channel-patch segmentation and vector-quantized neural spectrum prediction allow LaBraM to handle EEG datasets with diverse configurations.
  • Neural Transformers represent temporal and spatial EEG features for downstream tasks including abnormal detection, event classification, emotion recognition, and gait prediction.
  • LaBraM outperformed the compared state-of-the-art methods across the reported downstream task domains.

B LABRAM PRE-TRAINING ANALYSIS

LaBraM pre-training is framed as a variational autoencoder-like procedure that reconstructs Fourier-spectrum information from masked EEG through discrete neural tokens. Training separates tokenizer learning from masked-token prediction.

  • The framework models recovery of an EEG sample’s Fourier spectrum from a masked EEG input using discrete neural tokens.
  • The neural tokenizer qϕ encodes EEG samples into discrete tokens, the decoder pψ predicts Fourier spectra, and pθ reconstructs tokens from corrupted input.
  • Training uses two stages: first optimize the vector-quantized neural spectrum tokenizer, then fix it while learning the masked-EEG prior.

C HYPERPARAMETER SETTINGS

The appendix lists separate hyperparameter tables for vector-quantized neural spectrum prediction, masked EEG pre-training, and downstream fine-tuning.

  • Table 3 lists hyperparameters for vector-quantized neural spectrum prediction training.
  • Table 4 lists hyperparameters for masked EEG pre-training.
  • Table 5 lists hyperparameters for downstream fine-tuning.

D PRE-TRAINING DATASET DESCRIPTION

The training collection combines EEG datasets spanning multiple tasks, electrode configurations, sampling rates, subjects, and recording durations. It includes 2534.78 total hours for tokenizer training and LaBraM pretraining, while Fourier-spectrum reconstruction shows stronger amplitude than phase recovery.

  • 2534.78 hours of EEG data were used for both vector-quantized neural spectrum prediction training and LaBraM pretraining.
  • The collection spans emotional, clinical, resting-state, motor-imagery, motor-movement, categorization, grasp-and-lift, and P300 spelling datasets.Examples include SEED Series, Siena Scalp EEG, BCI Competition IV-1, EEG Motor Movement/Imagery, and Inria BCI Challenge.
  • The training data include recordings from different participant populations and durations, ranging from 0.83-hour datasets to datasets totaling 166.75 hours.
  • The datasets vary substantially in channel count and sampling rate, including 31–64 channels and sampling rates from 160 to 2048 Hz.
  • Amplitude reconstruction follows the overall Fourier-spectrum trend better than phase reconstruction, although reconstruction loss decreases stably during training.The authors interpret the stable loss decrease as evidence that the discrete codebook learns high-level Fourier-domain information.

F MORE EXPERIMENTS ON OTHER BCI TASKS

Additional experiments evaluate LaBraM on emotion recognition and gait-related regression. LaBraM outperforms all baselines on SEED-V, while its larger models achieve the best performance among methods on MoBI.

  • SEED-V contains five emotion categories recorded from 20 subjects using 62-channel, 1000 Hz EEG.
  • For MoBI, training uses the first 10 minutes of each treadmill session, validation uses the subsequent 5 minutes, and testing follows a 5-minute session.
  • On SEED-V emotion recognition, LaBraMs outperform all baseline methods on all metrics.
  • On MoBI gait-related regression, the Base model achieves competitive results, while the Large and Huge models obtain better performance among all methods.
  • The vector-quantized neural spectrum prediction ablation compares discrete-token prediction, tokenizer-embedding prediction, raw-patch reconstruction, and Fourier-spectrum reconstruction.

H ABLATION ON MASK RATIO

Mask-ratio ablations on TUAB and TUEV identify settings near one-half masking as robust choices, with dataset-specific best-performing ratios.

  • 0.4 (0.6) is best for TUAB, whereas 0.5 (0.5) is best for TUEV under the reported symmetric-masking convention.
  • 0.5 (0.5) achieves smaller standard deviations on both TUAB and TUEV than the other evaluated mask ratios.
  • The authors conclude that 0.5 (0.5) is a relatively good mask ratio for LaBraM pre-training.

I ABLATION ON SYMMETRIC MASKING

The symmetric-masking ablation evaluates TUAB and TUEV and finds that removing symmetric masking generally reduces performance, especially on TUEV.

  • Performance decreases by a remarkable margin on most metrics for both TUAB and TUEV without symmetric masking.
  • The performance reduction without symmetric masking is especially pronounced on TUEV.
  • The base model improves slightly on TUAB without symmetric masking, unlike most other evaluated scenarios.

J LABRAM WITHOUT PRE-TRAINING

LaBraM’s masked EEG modeling pre-training is evaluated against training from scratch and through different fine-tuning depths. Pre-training is useful, while partial fine-tuning produces dataset-dependent results.

  • J LABRAM WITHOUT PRE-TRAINING: A steep performance drop occurs when LaBraM is trained downstream from scratch without pre-training.This comparison is presented as evidence for the usefulness of masked EEG modeling pre-training.
  • J LABRAM WITHOUT PRE-TRAINING: On TUAB, full fine-tuning and fine-tuning 12 or 8 Transformer blocks achieve quite similar results.Fine-tuning only 4 blocks or using linear probing causes slight degradation on TUAB.
  • J LABRAM WITHOUT PRE-TRAINING: On TUEV, fine-tuning 8 Transformer blocks achieves the best performance on all three metrics.Linear probing performs much worse than the other reported settings on TUEV.

L ABLATION ON SPATIAL EMBEDDINGS

Spatial embeddings address heterogeneous electrode configurations and are tested through ablation. The paper also identifies broader model-size, fine-tuning-cost, and modality boundaries.

  • L ABLATION ON SPATIAL EMBEDDINGS: Spatial embeddings help address heterogeneity in electrode configurations.During pre-training, the loss does not converge without spatial embeddings because the model needs them to identify masked patches for reconstruction.
  • L ABLATION ON SPATIAL EMBEDDINGS: Discarding spatial embeddings during downstream fine-tuning causes a significant performance drop.This ablation tests whether spatial embeddings remain useful after pre-training.
  • M DISCUSSION: LaBraM remains smaller than current large vision and language models despite using over 2,500 hours of EEG and 369M parameters.The authors characterize the work as a first step toward large EEG models for generic representations.
  • M DISCUSSION: LaBraM requires full fine-tuning for downstream adaptation, which can be costly in computation and memory.The authors suggest parameter-efficient methods such as adapters, prompt tuning, and LoRA as future directions.
  • M DISCUSSION: LaBraM is trained with unimodal EEG data, leaving multimodal large EEG models as a future research direction.The outlook proposes incorporating other modalities.
Loading 2405.18765v1…