PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals

Wei-Bang Jiang, Yansen Wang, Bao-Liang Lu, Dongsheng Li

arXiv:2409.00101v3eess.SPcs.HCcs.LG

TL;DR

Pre-trained EEG models often need task-specific fine-tuning, while EEG signals and tasks pose challenges for integration with LLMs. NeuroLM uses text-aligned neural tokens, multi-channel autoregressive pre-training, and multi-task instruction tuning; experiments across six EEG tasks report superior multi-task learning and inference performance.

  • Problem

    Pre-trained EEG models often require resource-intensive, task-specific fine-tuning, and integrating EEG with LLMs is challenged by unavailable EEG-text pairs and diverse task requirements.

    Source: §1 · p. 2

  • Method

    NeuroLM encodes EEG as text-aligned discrete neural tokens, pre-trains an LLM autoregressively across channels, and applies multi-task instruction tuning.

    Source: §1 · pp. 2–3

  • Results

    Experiments across six diverse EEG tasks report NeuroLM’s superior performance in multi-task learning and inference.

    Source: §1 · p. 2, §4 · p. 10

  • Takeaways & Limitations

    Joint instruction tuning enables multiple EEG tasks within one model and reduces the need for individual task fine-tuning.

    Source: §1 · p. 3

    On TUSL, performance appears unstable because the dataset has extremely limited samples.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts NeuroLM to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroLM showcases the huge potential of this multi-task learning paradigm.

1 INTRODUCTION

EEG models face noisy, nonstationary signals and diverse collection configurations, while existing pre-trained models still require task-specific fine-tuning. NeuroLM combines text-aligned neural tokens, multi-channel autoregressive pre-training, and multi-task instruction tuning in one model.

1 INTRODUCTION

  • Existing EEG models require individual fine-tuning for downstream datasets, despite challenges from low signal-to-noise ratios, nonstationarity, and varied collection configurations.Task-specific fine-tuning also demands substantial computational and storage resources.Source: §1 · p. 2
  • Integrating EEG with LLMs remains difficult because EEG-text pairs are unavailable, generic LLM representations are unresolved, and EEG tasks vary in specificity.A unified model must handle diverse tasks without compromising individual-task performance.Source: §1 · p. 2
  • NeuroLM trains a text-aligned neural tokenizer with vector-quantized temporal-frequency prediction to encode continuous EEG as discrete codes.Adversarial training aligns EEG and text embedding spaces.Source: §1 · p. 2, Figure 2 · p. 3
  • NeuroLM uses multi-channel autoregressive pre-training on 25,000 hours of EEG data to learn causal representations across channels.The approach is intended to capture diverse neural patterns for generalization across EEG tasks.Source: §1 · p. 3
  • Multi-task instruction tuning lets NeuroLM perform multiple downstream tasks in one model, reducing the need for task-specific fine-tuning.The paper evaluates tasks including abnormal detection, event classification, emotion recognition, sleep staging, workload prediction, and slowing-type classification.Source: §1 · pp. 2–3

2 METHOD

NeuroLM turns EEG into text-aligned discrete tokens, trains an LLM to predict EEG tokens across channels, and then uses instruction tuning to handle downstream BCI tasks.

2.1 TEXT-ALIGNED NEURAL TOKENIZER TRAINING

  • The VQ encoder maps EEG patch representations to nearest codebook embeddings, producing a sequence of discrete EEG tokens.Source: §2.1 · p. 3
  • The tokenizer predicts both EEG signals and frequency magnitudes from quantized representations, using separate temporal and frequency decoders.Source: §2.1 · p. 4
  • A domain classifier and gradient-reversal layer align EEG embeddings with text space, addressing challenges posed by limited labeled EEG-text pairs and EEG’s complex information.Source: §2.1 · p. 4

2.2 MULTI-CHANNEL AUTOREGRESSIVE PRE-TRAINING

  • Multi-channel autoregression trains the model to predict each channel’s next EEG token, using attention masks that expose tokens from all channels at the current and previous time steps.Source: §2.2 · p. 5

2.3 MULTI-TASK INSTRUCTION TUNING

  • Instruction tuning combines downstream datasets in one model, concatenates EEG and text instructions with [SEP], and calculates loss only on the answer text.Source: §2.3 · p. 6

3 EXPERIMENTS

Across six EEG datasets, NeuroLM was comparable to most single-task baselines but remained behind LaBraM; ablations examined robustness to instruction format and data size, as well as multi-channel autoregressive pre-training.

3.1 DOWNSREAM DATASETS

  • The six evaluations cover abnormality, event, emotion, sleep-stage, workload, and slowing classification across datasets of varied sizes.The datasets are TUAB, TUEV, SEED, HMC, Workload, and TUSL.Source: §3.1 · pp. 6–7

3.2 EXPERIMENTAL SETUP

  • Dataset-specific train, validation, and test splits were used, and NeuroLM variants used GPT-2 with 254M, 500M, and 1,696M parameters.EEG preprocessing included filtering, resampling to 200 Hz, and normalization; about 25,000 hours of EEG data were collected for training.Source: §3.1 · p. 7, §3.2 · p. 7

3.3 EXPERIMENTAL RESULTS

  • NeuroLM was comparable to most single-task baselines but remained behind LaBraM, with larger variants improving on most downstream datasets and greater overfitting risk.The authors note that direct comparisons with single-task baselines are not entirely fair because those models are trained and tested on individual datasets.Source: §3.3 · p. 7, Table 2 · p. 8, Table 3 · p. 8

3.4 ABLATION ON ROBUSTNESS

  • Option shuffling produced comparable performance on TUEV and HMC but significantly degraded TUSL performance, which the authors attribute to its smaller sample count.They expect similar TUSL results with more data.Source: §3.4 · p. 8, Figure 5 · p. 9

3.5 ABLATION ON INSTRUCTION DATA SIZE

  • NeuroLM maintained performance across instruction-data sizes; on TUEV and HMC, only NeuroLM and LaBraM were relatively unaffected by data-size changes.TUAB performance was stable for NeuroLM, LaBraM, and CNN-Transformer.Source: §3.5 · p. 9, Figure 6 · p. 9

3.6 VISUALIZATION CURVES OF MULTI-CHANNEL AUTOREGRESSION

  • Pre-training loss converged and validation perplexity decreased, while NeuroLM-L and NeuroLM-XL had similar validation perplexity, indicating the data did not yet meet billion-parameter training needs.The curves visualize pre-training loss, accuracy, and validation perplexity.Source: §3.6 · pp. 9–10, Figure 7 · p. 10

3.7 ABLATION ON MULTI-CHANNEL AUTOREGRESSIVE PRE-TRAINING

  • Ablation results showed significant downstream performance improvement when NeuroLM used multi-channel autoregressive pre-training.The approach predicts the next EEG tokens for each channel.Source: §3.7 · p. 10, Figure 8 · p. 10

4 CONCLUSION

NeuroLM integrates EEG with an LLM framework and shows superior multi-task learning and inference performance across six diverse EEG datasets. The related work contrasts this approach with prior EEG pre-training methods requiring task-specific fine-tuning and multimodal models that connect modality-specific encoders to LLMs.

4 CONCLUSION

  • Experiments across six diverse EEG datasets demonstrate NeuroLM’s superior performance in multi-task learning and inference.The conclusion describes NeuroLM as a step forward for brain-computer interfaces and healthcare.Source: §4 · p. 10

A RELATED WORK

  • Prior EEG pre-training methods require downstream task-specific fine-tuning and cannot perform multi-task learning and inference.Related work also surveys multimodal LLMs that connect modality-specific encoders with LLMs.Source: A RELATED WORK · p. 15

B INSTRUCTION DESIGN

  • Table 5 presents instruction designs for downstream datasets.Source: Table 5 · p. 15

C PRE-TRAINING DATASET DESCRIPTION

The pre-training data comprise multiple EEG datasets with varied configurations, totaling close to 25,000 hours after cleaning.

C PRE-TRAINING DATASET DESCRIPTION

D DETAILED EXPERIMENTAL SETTINGS

The appendix tables report hyperparameters separately for neural-tokenizer training, autoregressive pre-training, and instruction tuning.

D.1 HYPERPARAMETER SETTINGS

  • Table 7 reports hyperparameters for the neural tokenizer.Source: Table 7 · p. 17
  • Table 8 reports hyperparameters for autoregressive pre-training.Source: Table 8 · p. 17
  • Table 9 reports hyperparameters for instruction tuning.Source: Table 9 · p. 18

D.2 METRICS

The evaluation uses class-imbalance-aware metrics and monitors AUROC for binary classification and Cohen’s Kappa for multi-class classification.

D.2 METRICS

  • Balanced Accuracy averages recall across classes and is used for evaluating imbalanced classification datasets.Source: §D.2 · p. 18
  • AUC-PR measures the area under precision-versus-recall curves across threshold values for binary classification.Source: §D.2 · p. 18
  • Weighted F1 is the support-weighted harmonic mean of precision and recall, giving greater weight to classes with more instances.Source: §D.2 · p. 18
  • AUROC and Cohen’s Kappa serve as monitor scores for binary and multi-class classification, respectively.Source: §D.2 · p. 18

D.3 BASELINES

The experiments compare NeuroLM against individually fine-tuned EEG baselines and inspect how it attends across layers and channels. Attention shifts from text in shallow layers to EEG in deeper layers, with patterns varying by task.

D.3 BASELINES

  • BIOT, LaBraM, and five supervised methods serve as baselines, each fine-tuned separately because no multi-task EEG method was available.LaBraM was pre-trained on 2,500 hours of data using masked EEG modeling.Source: §D.3 · p. 18

E ATTENTION VISUALIZATION

  • Text attention concentrates in shallow layers, while EEG attention increases in deeper layers across datasets.For multiple-choice questions, attention to answer options is prominent between layers 6 and 9.Source: Figure 9 · p. 19, E ATTENTION VISUALIZATION · p. 20
  • NeuroLM appears to aggregate information at Cz for most datasets; for HMC, F4 and C3 are important while O2 is less effective for sleep-stage classification.Source: E ATTENTION VISUALIZATION · p. 20

F.1 ABLATION ON TEMPORAL-FREQUENCY PREDICTION

The ablations compare temporal- and frequency-domain reconstruction targets and examine EEG-text embedding alignment. The reported patterns vary by downstream task, while alignment is associated with normal instruction-tuning predictions rather than random outputs.

F.1 ABLATION ON TEMPORAL-FREQUENCY PREDICTION

  • Temporal reconstruction is more important on TUAB, Workload, and TUSL, whereas frequency reconstruction performs better on TUEV, SEED, and HMC.The comparison tests temporal-only, frequency-only, and combined reconstruction targets.Source: §F.1 · p. 20, Figure 10 · p. 20
  • Combining temporal and frequency reconstruction yields similar or higher performance on most tasks.The authors describe this as evidence that the tokenizer extracts compact EEG representations for language models.Source: §F.1 · p. 20, Figure 10 · p. 20

F.2 VISUALIZATION OF EEG AND TEXT EMBEDDINGS

  • Without alignment, EEG embeddings expand outside the text space and instruction tuning produces random words and near-zero scores across most downstream metrics; with alignment, predictions are normal.The authors associate the unaligned case with disordered attention and describe the aligned EEG space as mostly aligned with text space.Source: §F.2 · pp. 20–21, Figure 11 · p. 21

G ABLATION ON DIFFERENT PRE-TRAINING EPOCHS

The epoch ablation reports that 20 pre-training epochs produce the most best and second-best results overall, though some tasks peak earlier. The discussion identifies performance, tuning, and alignment limitations and suggests several possible improvements.

G ABLATION ON DIFFERENT PRE-TRAINING EPOCHS

H DISCUSSION

  • NeuroLM still trails end-to-end state-of-the-art methods on some downstream datasets, is sensitive to hyperparameters, and uses coarse space-wise alignment because high-quality EEG-text pairs are limited.The authors note that coarse alignment can challenge LLMs in extracting useful information from EEG tokens.Source: H DISCUSSION · p. 22
  • The authors propose newer LLMs, modality-specific experts, and finer-grained EEG-text alignment as possible improvements.The paper uses GPT-2 and suggests that predefined EEG descriptions and contrastive learning could support finer-grained alignment.Source: H DISCUSSION · p. 22
Loading 2409.00101v3…