Source-linked AI summary
WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities
Ziyi Zeng, Zhenyang Cai, Yixi Cai, Xidong Wang, Junying Chen, Rongsheng Wang, Yipeng Liu, Siqi Cai, Benyou Wang, Zhiguo Zhang, Haizhou Li
TL;DR
EEG interpretation models lack the combination of conversational proficiency, cross-task support, and open instruction-tuning resources. WaveMind aligns EEG with textual and visual modalities and introduces an open-source instruction-following dataset; the paper presents it as bridging EEG foundation and conversational models while enhancing downstream tasks.
Problem
Existing EEG foundation models lack conversational proficiency, dedicated EEG conversational models are limited to single tasks, and the research community lacks open-source EEG instruction-tuning datasets.
Source: §1 · pp. 1–2Method
WaveMind aligns EEG with textual and visual modalities in a unified representation space and introduces an open-source EEG instruction-following dataset for conversational interpretation.
Source: §1 · p. 2Results
WaveMind bridges cross-task EEG foundation models and EEG conversational models, enhancing downstream tasks through conversational ability, EEG signal awareness, and use of upstream training data.
Source: §8 · p. 11Takeaways & Limitations
The reported complementarity between cognitive and Brain State activities provides neural-level insights into integrated brain function.
Source: §8 · p. 12
Abstract
from arXiv · showhide
Electroencephalography (EEG) interpretation using multimodal large language models (MLLMs) offers a novel approach for analyzing brain signals. However, the complex nature of brain activity introduces critical challenges: EEG signals simultaneously encode both cognitive processes and intrinsic neural states, creating a mismatch in EEG paired-data modality that hinders effective cross-modal representation learning. Through a pivot investigation, we uncover complementary relationships between these modalities. Leveraging this insight, we propose mapping EEG signals and their corresponding modalities into a unified semantic space to achieve generalized interpretation. To fully enable conversational capabilities, we further introduce WaveMind-Instruct-338k, the first cross-task EEG dataset for instruction tuning. The resulting model demonstrates robust classification accuracy while supporting flexible, open-ended conversations across four downstream tasks, thereby offering valuable insights for both neuroscience research and the development of general-purpose EEG models.
1 Introduction
Existing EEG foundation models support diverse tasks but lack conversational proficiency, while dedicated conversational models remain limited to single tasks. WaveMind aligns textual and visual modalities and pairs an open-source instruction-tuning dataset with a cross-task conversational EEG model.
1 Introduction
- Existing EEG foundation models support diverse downstream tasks but lack conversational proficiency, while dedicated EEG conversational models are restricted to single-task execution.The introduction also notes that their interactive potential is not fully explored.Source: §1 · p. 1
- Prior EEG alignment approaches pair signals with visual stimuli or textual annotations such as emotional valence and clinical descriptors.The introduction argues that using only one paired modality constrains the utility of training data and model generalizability.Source: §1 · pp. 1–2
- A pivot investigation finds that paired modalities from distinct sources are complementary, motivating a cross-task EEG foundation model with conversational capabilities.The work scales and preprocesses data from five diverse datasets to construct WaveMind-Instruct.Source: §1 · p. 2
- The proposed framework aligns EEG with textual and visual modalities, broadening upstream data sources and supporting downstream interpretation without architectural modifications.The illustration describes shared-space compatibility with more upstream data and EEG interpretation across downstream tasks.Source: Figure 1 · p. 1, §1 · p. 2
- WaveMind-Instruct is an open-source instruction-following dataset with three instruction types and two dialogue scenarios, supporting WaveMind’s unified interpretation across diverse brain activities.The paper presents WaveMind as the first conversational EEG foundation model.Source: §1 · p. 2
2 Related Work
Related work covers EEG representation alignment, two approaches to EEG-to-text decoding, and EEG foundation models. The paper distinguishes semantic interpretation from fine-grained translation and identifies gaps in existing foundation models.
2 Related Work
- Representation alignment maps neural signals to brain-activity-related modalities; NICE aligned EEG representations with image features, and later work enabled language decoding from EEG.The passage describes alignment as an important method for EEG-to-text decoding.Source: §2 · p. 2
- EEG-to-text translation reconstructs thought word by word but requires millisecond-level annotations and performs poorly across languages and individuals, whereas interpretation targets semantic accuracy using coarse-grained paired data.The paper identifies WaveMind with the interpretation approach.Source: §2 · p. 2
- EEG foundation models are large-scale generic models trained on diverse datasets for cross-task brain-signal analysis, but current models focus narrowly on classification and lack conversational capabilities.The passage also says they fail to leverage Brain Cognition data, which captures higher-order processes such as language comprehension.Source: §2 · pp. 2–3
3 Pilot Study on Fusion of Multiple Paired Modalities
The pilot tests whether EEG alignment with heterogeneous image and text sources can outperform single-source alignment. Joint training on Brain Cognition and Brain State performed better than either alone, consistent with complementary expressions of brain activity.
Multiple Paired Modalities
- Image pairs represent visual stimuli that activate cognitive-related activity, while text pairs describe internal brain states in resting-state recordings.Source: §3 · p. 3
Motivation
- Prior EEG interpretation aligned signals with images or text alone, and no prior work had aligned heterogeneous sources from distinct origins.Source: §3 · p. 3
Pilot Study
- The pilot combined THING-EEG image pairs and TUEV text pairs after testing each paired modality separately.Source: §3 · p. 3
Analysis
- Joint training on Brain Cognition and Brain State outperformed either single-modality model, indicating cross-source generalization and complementary expressions of brain activity.Source: Table 2 · p. 3, §3 · p. 4
4 Data Engineering
Data Engineering scales the pilot's multimodal alignment to five EEG datasets and builds training inputs for alignment and instruction tuning. The pipeline standardizes EEG, constructs fine-grained text targets, and creates diverse instruction data.
4.1 Data Sources
- The study expands to five datasets spanning Brain Cognition image pairs and Brain State text pairs.THING-EEG and ImageNet-EEG provide visual-stimulus pairs; TUEV, TUAB, and SEED provide text annotations for events, abnormalities, and emotion.Source: §4 · p. 4, §4.1 · p. 4
4.2 EEG Preprocessing
- EEG recordings are mapped to 32 standard 10-20 channels, resampled to 512 Hz, and segmented into fixed one-second tensors.Short segments are replicated and longer ones partitioned; the resulting tensors have shape R32×512.Source: §4.2 · p. 4
4.4 Training Data Statistics
- Representation alignment uses 1.3M samples, while instruction tuning yields 338K samples.The training-data statistics are presented in Table 3; repeated sampling balances datasets with limited data.Source: Table 3 · p. 5, §4.4 · p. 5
4.4.1 Training Data for Encoder Representation Alignment
- Image-EEG alignment uses image features directly, while Text-EEG alignment uses manually constructed fine-grained descriptions because phase-level labels are too coarse.Source: §4.4.1 · p. 5
4.4.2 Training Data for Instruction Tuning
- Instruction tuning combines description, QA, and MCQ formats with consultation and non-medical analysis scenarios.Image-EEG descriptions and QA are derived from image captions; Text-EEG QA is seeded from collected annotation facts. Reviewers' average acceptance scores were 0.945 for captions and 0.966 for definitions, and deduplication retained approximately 70% of synthesized data.Source: §4.4.2 · pp. 5–6
5 WaveMind
WaveMind maps EEG and paired modalities into a unified semantic space for language decoding. Its architecture combines EEG, instruction, and retrieved category information, and its training proceeds through alignment, adapter cold-start, and instruction tuning.
5.2 Training Paradigm
- Stage I aligns image and text supervision in CLIP space with separate InfoNCE losses and trains ATMM on 1.2M EEG pairs.Adding the category-based auxiliary loss Lcls showed no performance improvement.Source: §5.2 · p. 6
- Stage II cold-starts the modality adapter on image-domain LLaVA-Pretrain-558K data, using shared CLIP space to initialize EEG-domain tuning.Source: §5.2 · pp. 6–7
5.1 Architecture
- ATMM encodes raw EEG into normalized features that an adapter maps to an EEG token, combined with instruction and retrieved category tokens for LLM decoding.The inference architecture integrates retrieval-augmented generation with the unified semantic representation.Source: §5.1 · p. 6, Figure 4 · p. 7
5.2 Training Paradigm
- Stage III tunes on WaveMind-Instruct-338K with LoRA and the modality adapter unfrozen while ATMM remains frozen.Source: §5.2 · p. 7
6 Experiments
Experiments assess WaveMind from MCQ classification through open-ended conversation, treating accurate category identification as a basis for reliable EEG-MLLM interpretation.
6.1.1 Training Setting
- Stage I took about 13 hours per encoder on one A40 GPU; Stages II and III used four A800 GPUs for 15 hours.The LLM backbone is Vicuna-1.5-7B with a two-layer MLP modality adapter, and the EEG encoder is ATMM.Source: §6.1.1 · p. 7
6.1.2 WaveMind-Bench
- WaveMind-Bench-12k samples test-set items into MCQs with 2, 4, or k options, where k is the dataset’s class count; THING-EEG uses 40 options for its 200-category zero-shot setting to maintain practicality.The authors plan to open-source the construction code because of the strong privacy and limitations of EEG datasets.Source: §6.1.2 · p. 7
6.1.3 Evaluation Metrics
- Weighted K-way accuracy evaluates classification; generation uses BLEU-1/2, METEOR, ROUGE-1/2, MiniLM-L12-v2 semantic similarity, and GPT-4o matching.Weighted K-way questions contain one correct and K−1 randomly selected incorrect categories.Source: §6.1.3 · p. 8
6.2.1 Classification Assessment
- Real EEG beats random baselines across datasets; RAG raises 40-class THING-EEG accuracy from 0.122 to 0.250 and ImageNet-EEG accuracy from 0.574 to 0.603.The reported RAG gains are particularly notable on cognitive tasks and MCQs with many options.Source: §6.2.1 · p. 8, Table 5 · p. 8
6.2.2 Conversational Assessment
- Without cues, EEG-image responses outperform the random baseline, and performance rises with cue granularity; Figure 5 compares caption-based evaluation and object-cue settings.The object cue provides k possible options, including the correct answer.Source: §6.2.2 · pp. 8–9, Figure 5 · p. 9
- WaveMind’s brain-state responses convey phase-based annotation categories in natural language, and RAG improves generation quality.GPT-4o judges whether responses imply or contain the paired category annotations.Source: §6.2.2 · p. 9, Figure 5 · p. 9
7 In-Depth Analysis
The in-depth analysis compares encoder perception, staged training choices, and real dialogue examples to examine WaveMind’s components and interaction capabilities.
7.1 Encoder Perception
- ATMM performs significantly better than baselines on THING-EEG feature retrieval and consistently ranks in the top three on SEED and TUEV brain-state datasets.Table 7 reports feature-retrieval accuracy, with separate closed-set and zero-shot, subject-dependent and subject-independent evaluations.Source: §7.1 · p. 9, Table 7 · p. 10
7.2.1 Encoder Alignment Matters (Stage I)
- With a randomly initialized encoder, SFT classification is not significantly above random guess, supporting alignment as a necessary foundation for effective unified representations.The comparison is reported in Table 8.Source: §7.2.1 · p. 10
7.2.2 Cold-Start Matters (Stage II)
- Cold-start tuning helps the model grasp recognition in CLIP space before EEG-domain fine-tuning, alleviating early optimization bias and promoting faster convergence.The reported benefit holds under different encoder settings and training paradigms.Source: §7.2.2 · p. 10
7.2.3 Encoder Should Freeze (Stage III)
- Frozen and unfrozen encoders show consistent results, but the frozen encoder is preferred given its advantages when paired with RAG.The comparison appears in Table 8.Source: §7.2.3 · p. 10, Table 8 · p. 10
7.2.4 Scaling Helps Diversity and Ability (Stage III)
- Full training data outperforms partial data in linguistic diversity and classification ability; deduplication and quality control improve the richness of generated content.Table 9 reports 2-way accuracy for classification ability across training-data scales.Source: §7.2.4 · pp. 10–11, Table 9 · p. 11
7.3 Case Study
- A real human-WaveMind dialogue shows the model handling common situations and inferring user intent from open-ended prompts without specific instructions.The authors use these cases to illustrate task identification and categorization in interaction.Source: §7.3 · p. 11, Figure 6 · p. 11
8 Conclusion
WaveMind uses a unified representation space to connect EEG foundation-model generalization with conversational interpretation. The authors report cross-task EEG perception and complementary patterns between cognitive and brain-state activities.
8 Conclusion
- WaveMind’s unified representation framework brings cross-task generalization and conversational EEG interpretation together, supporting conversational ability, EEG signal awareness, and use of upstream training data.The paper contrasts this integration with capabilities it says are lacking in NeuroLM [31] and Thought2EEG.Source: §8 · p. 11
- Training in the shared latent space supports cross-activity and cross-task perception and cold-start tuning with out-of-domain data before EEG-domain tuning.The authors report improved adaptability from this cold-start tuning and cite Tables 7 and 8.Source: §8 · p. 11
- The analysis reports complementary neural patterns between cognitive and brain-state activities and presents them as insights into integrated brain function.The authors point to manifestations of this complementarity in EEG.Source: §8 · p. 12
A.1 Ablation on Selected LLM Backbones and Encoders
WaveMind’s backbone variants showed no significant performance differences, while ATMM outperformed other encoders, especially on ImageNet and THING-EEG. The appendices also report complementary supervision gains and describe the model’s architecture, data retrieval, preprocessing, and limitations.
A.1 Ablation on Selected LLM Backbones and Encoders
- ATMM outperformed other encoders, especially on ImageNet and THING-EEG, while different LLM backbones showed no significant performance differences.The authors identify ATMM as their preferred encoder; they suggest data engineering or data quality, rather than the LLM backbone, may constrain performance.Source: §A.1 · p. 15
A.3 Supervision Modalities are Complementary
- Co-training cognitive and Brain State tasks improved performance across all downstream tasks, with the authors attributing this to richer EEG representations and complementary knowledge transfer.Table 12 reports the cross-data-source ablation on WaveMind-Bench.Source: §A.3 · p. 15, Table 12 · p. 16
B.1 Positive Impact
- The authors identify WaveMind as a chat-style EEG MLLM with cross-task and cross-scenario interpretation capabilities, alongside a benchmark for cognitive-category recognition.They describe the work as methodological support for human-machine neural interaction research.Source: §B.1 · p. 16
B.2 Negative Impact
- The authors warn that coarse annotations limit causal reasoning, attribution analyses may conflict with neuroscience principles, and generalization beyond THING-EEG is uncertain; benchmarks remain coarse and MCQ-only.They also flag potential bias from the encoder, backbones, and training data, and note that expanded data or model scale could increase energy concerns.Source: §B.2 · pp. 16–17
D Baseline EEG Encoder
- Baseline encoders designed for specific tasks were adapted by changing their input layers and replacing task-specific heads to enable cross-model, multi-task comparisons.The listed baselines include ChannelNet, NICE, EEG-ITNet, MLP, ShallowFBCSPNet, and ATM-S.Source: D Baseline EEG Encoder · p. 21
E.1 ATMM Encoder
- ATMM adapts ATM-S by removing its subject token, adding multi-head attention layers, and setting output features to 768 dimensions for CLIP-space alignment.Its processing pipeline combines channel attention, temporal-spatial feature extraction, and MLP abstraction.Source: §E.1 · p. 21
E.2.1 RAG Detail
- RAG retrieves EEG-matched supervision features and category names by cosine similarity, inserts names into the prompt, and ensures at least one name per task.The feature database contains 1,824 categories and retrieval uses k=420; the pseudocode likewise selects at least one feature per dataset before filling the remaining slots.Source: §E.2.1 · p. 21, §E.2.3 · p. 22
F Data Preprocessing Pipeline Detail
- Preprocessing standardizes EEG to 32 channels, resamples signals to 512 Hz, and forms one-second segments by repetition or segmentation as needed.Missing standard channels are interpolated from neighboring electrodes, and channels are sorted consistently.Source: F Data Preprocessing Pipeline Detail · p. 23
G Dataset Detail
This section describes EEG dataset and instruction-data construction, including quality controls, alignment resources, training settings, augmentations, dual-modality representation alignment, and retry confusion patterns across datasets.
G Dataset Detail
- The dataset suite spans THINGS-EEG, ImageNet-EEG, SEED, TUAB, and TUEV, covering visual, emotion, and clinical EEG data.The described configurations include 10-subject, 64-channel THINGS-EEG; six-subject, 128-channel ImageNet-EEG; 15-subject SEED; and TUAB abnormality and TUEV event annotations.Source: G Dataset Detail · p. 23
H.1 Detail Data Statistic
- 338k instruction-response pairs from five sources form WaveMind-Instruct, with 2k Reject Empty-EEG samples and consultation scenarios limited to TUEV and TUAB.Consultation data address EEG-based event and abnormality detection; the empty-input samples are intended to counter hallucinations without real EEG input.Source: §H.1 · p. 24
H.2 Quality Control Detail
- 0.945±0.023 for image-caption fidelity and 0.966±0.013 for clinical-annotation fidelity were achieved through human validation within a three-stage quality-control pipeline.Image captions and clinical annotations were checked separately; synthesis used captions and factual information as guidance, barred EEG-specific details and neural-mechanism speculation, then deduplicated at a 0.75 similarity threshold, retaining 70% of outputs.Source: §H.2 · pp. 24–25, Table 17 · p. 25
H.3.1 Description Instruction
- For image-supervision data, image-related vocabulary in generated captions is replaced with “EEG,” then a question is paired with the processed description for feature alignment.The captions are generated with Qwen2.5-VL.Source: §H.3.1 · p. 25
H.3.2 QA Instruction
- For cognitive QA, two paired-image captions are rewritten into EEG-grounded conversations, while brain-state QA rewrites manually refined factual annotations.Cognitive prompts prohibit naming objects in questions and require simple, caption-grounded answers that mention EEG; analysis and consultation use distinct conversational scenarios.Source: §H.3.2 · pp. 26–28
I.1 Encoder Representation Alignment; I.2.1 Testing Data Statistic
- Detailed hand-written descriptions support fine-grained text representations for contrastive alignment; the WaveMind test set comprises 36k multiple-choice questions.The passage points to Table 18 for annotation-description correspondences and Table 19 for test-set statistics.Source: §I.1 · p. 29, Table 18 · p. 29, §I.2.1 · p. 29
J.1 Training Setting; J.2 Data Augmentation
- Stage I uses batch size 1024, while Stages II–III use Zero-2; augmentations include global or channel-wise normalization/scaling and random amplitude scaling that preserves temporal structure.The amplitude factor is sampled per sample from U(1 − a, 1 + a), for example a = 0.1 for ±10% variation.Source: §J.1 · p. 30, §J.2 · p. 31
J.2 Data Augmentation
- Independent sample scaling augments EEG while preserving relative channel relationships and temporal dynamics.The independent scaling also simulates natural amplitude variations and adds batch-level diversity.Source: §J.2 · p. 31
J.3.1 Pseudo code for Representation Alignment
- The alignment pseudocode normalizes EEG, image, and text features, computes grouped InfoNCE losses, and combines visual and textual losses using λ.Samples are split into image- and text-labeled groups, with the final loss weighted as λLvis + (1 − λ)Ltxt.Source: §J.3.1 · p. 32
J.3.2 Confusion Matrix Analysis
- ImageNet EEG shows clearer main-diagonal confusion, while SEED neutral samples are often misclassified as positive or negative.The SEED pattern suggests ambiguity in neutral-emotion feature expression; the ImageNet pattern is consistent with Table 1 and may reflect standardized experiments and data-quality control.Source: §J.3.2 · p. 32
J.3.3 Representation Analysis
The analysis finds semantic clustering alongside cross-dataset mixing and strong diagonal associations in EEG representations. The asset notes describe privacy-constrained EEG data sharing and distinguish available resources from those still being prepared.
J.3.3 Representation Analysis
- Two text-supervision clusters separate emotion-related SEED from characteristic-related TUAB and TUEV, while image-supervision features form five semantic clusters.K-means clustering reveals these patterns in the two RSMs.Source: §J.3.3 · p. 32, Figure 10 · p. 34
- Different-source EEG features mix across datasets, and high-intensity RSM diagonals indicate stable semantic mappings; the authors characterize the results as semantic sensitivity and cross-dataset robustness.The authors say this provides theoretical support for a universal large-scale EEG model framework.Source: §J.3.3 · pp. 32–34
K Open-Source Asset
- Source code, synthesis instruction-tuning data, and the benchmark are uploaded, but privacy restrictions prevent sharing processed EEG; processing code is open-sourced for reproducibility.Source: K Open-Source Asset · p. 34
- WaveMind-Instruct-338K and the model checkpoint were still being prepared for release; WaveMind-Bench-12K and serving resources were listed.Source: K Open-Source Asset · p. 34