Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding

• EEG variability across subjects, protocols, and acquisition systems makes representations shared across datasets and tasks difficult to learn.Conventional deep-learning approaches are typically trained for a single dataset or task.Source: §1 · p. 1
• EEG foundation models require large-scale EEG-specific pretraining and separate downstream adaptation, leaving unified multi-task decoding in one model underexplored.Source: §1 · p. 2
• Whether a general-domain VLM can process EEG through a familiar image representation for unified decoding remains unclear.Existing EEG-language models rely on EEG-specific pretraining to align neural signals with language.Source: §1 · p. 2
• BraVista arranges channel-wise STFT spectrograms into a structured image paired with a natural-language task instruction.The study reports strong decoding across four BCI tasks, better performance with STFT images than raw-signal plots or topographies, and gradual degradation under controlled corruption.Source: §1 · p. 2, Figure 1 · p. 2

EEG Time–Frequency Representation

• Channel-wise STFT spectrograms are arranged in a grid to form a structured visual representation of multichannel EEG.The time–frequency representation preserves discriminative structure and gives vision models a two-dimensional input for capturing temporal–spectral correlations.Source: §2 · pp. 2–3

EEG Instruction Construction

• Each EEG sample is paired with a natural-language instruction specifying the decoding task, and its response is the ground-truth label.This casts decoding as instruction following and uses the VLM’s pretrained language understanding for instruction-conditioned prediction.Source: §2 · p. 3

Training Framework of BraVista

• BraVista adapts a general-purpose Qwen3-VL-2B by fully fine-tuning its vision encoder and applying LoRA to the language decoder.LoRA constrains task-specific decoder updates to a low-dimensional subspace while limiting drift from the pretrained language model.Source: §2 · p. 3

Optimization for Multi-task EEG Decoding

• Multi-task decoding is formulated as autoregressive generation conditioned on an EEG image, a textual instruction, and prior output tokens.The model predicts an output sequence from the joint multimodal input.Source: §2 · p. 3
• Supervised fine-tuning trains task-specific outputs from the joint EEG visual input and textual prompt.Gradients update the vision encoder and language-model LoRA parameters for coordinated adaptation.Source: §2 · p. 3

3.1 Evaluation Datasets and Baselines

• Evaluation covers HMC sleep staging, SEED emotion recognition, EEGMAT cognitive-workload classification, and TUAB abnormality detection, with task-specific and unified baselines.Results are reported as mean ± standard deviation over three random-seed runs; SEED-exposed foundation models are excluded to prevent data leakage.Source: §3.1 · pp. 3–4

3.2 BraVista Enables Multi-Task EEG Decoding Across Diverse BCI Tasks

• BraVista consistently performs strongly as one model across all four datasets and substantially surpasses the zero-shot Qwen3-VL-2B backbone on each.Figure 2a compares balanced accuracy against supervised models, EEG foundation models, NeuroLM, and the zero-shot VLM.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 Time–Frequency Images Are the Effective Interface

• STFT inputs achieve the highest balanced accuracy and weighted F1 across all four datasets, outperforming raw time-series plots and scalp topographies.The authors attribute the difference to STFT jointly representing temporal and spectral dynamics.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4 BraVista Relies on EEG Signal Structure Under Noise Corruption

• As noise increases from 20 dB to 0 dB, balanced accuracy and weighted F1 decline gradually across datasets but remain above chance under severe noise.The authors interpret the gradual decline as evidence that predictions depend on EEG signal quality rather than superficial visual patterns.Source: §3.4 · p. 5, Figure 4 · p. 5

4 Discussion

• BraVista enables unified multi-task EEG decoding with a general-domain vision–language model, without task-specific architectures or separate large-scale EEG-specific pretraining.The discussion frames visual representations as an interface for reusing vision–language models in neural signal decoding.Source: §4 · p. 6
• STFT spectrograms were the most effective evaluated EEG-to-image interface, while gradual noise-related degradation and other analyses supported reliance on EEG-relevant information.The other cited analyses include physiologically plausible class-level errors and learned-embedding structure.Source: §4 · p. 6
• Current BraVista outputs class labels without explicitly using VLM reasoning; the discussion proposes reasoning and contextual information as future directions.Examples of context include subject-specific metadata and clinical history.Source: §4 · p. 6

A Time–Frequency Representation Parameters

• The time–frequency representation uses 400-sample (2-second) windows with 300-sample (1.5-second) overlap and retains magnitude spectrograms, truncated at 100 Hz.The magnitude spectrogram captures localized spectral energy over time; the frequency bound is intended to focus on physiologically relevant bands and suppress high-frequency noise.Source: A Time–Frequency Representation… · p. 11

B Baselines

• Task-specific baselines also include spectrogram-based ResNet-style CNNs and CNN–Transformer or CNN–LSTM architectures.These models combine local feature extraction with temporal or global dependency modeling.Source: B Baselines · p. 11
• The comparison baselines include EEG foundation models using generic or cross-subject representations and task-specific models such as CNN- and transformer-based decoders.The described foundation models include CBraMod, CSBrain, NeuroLM, and LaBraM; task-specific examples include SPaRCNet and ST-Transformer.Source: B Baselines · pp. 11–12

C.1 Implementation Details

• BraVista was trained for 1 epoch with effective batch size 128 on 4 NVIDIA RTX A6000 GPUs for 12 hours.Training used local batch size 8, gradient accumulation 4, 3% linear warm-up, then cosine learning-rate decay; the decoder and vision encoder learning rates were 1 × 10−4 and 2 × 10−6, respectively.Source: §C.1 · p. 12

C.2 Hyperparameter Settings

• Table 1 reports hyperparameter settings for the jointly trained BraVista model.The table is identified as covering post-training hyperparameters.Source: §C.2 · p. 12, Table 1 · p. 12

C.3 Metrics

• Balanced accuracy averages recall across classes, giving each class equal weight in binary and multiclass EEG classification.It is used to limit majority-class performance from dominating scores on imbalanced datasets.Source: §C.3 · pp. 12–13
• Weighted F1 averages per-class F1 scores using class sample counts as weights, emphasizing performance on more prevalent classes.It complements balanced accuracy by reflecting the overall data distribution while capturing precision and recall.Source: §C.3 · p. 13

D Dataset Details

• Dataset details cover data, preprocessing, splits, and generated images and instructions, while Table 2 spans heterogeneous acquisition settings and tasks.Source: D Dataset Details · p. 13, Table 2 · p. 13

D.1 Sleep Stage Classification

• HMC provides overnight EEG recordings from 151 subjects, labeled as Wake, NREM1, NREM2, NREM3, or REM across four channels.Source: §D.1 · p. 13
• HMC signals are filtered, resampled to 200 Hz, and amplitude-normalized; subject splits use 100 for training, 25 for validation, and 26 for testing.Source: §D.1 · p. 13
• Each HMC segment becomes an STFT image with four channel panels arranged in a 1 × 4 grid and a prompt requiring one sleep-stage label.The prompt describes time and frequency axes and spectral-power intensity.Source: §D.1 · pp. 13–14

D.2 Emotion Recognition

• SEED recordings cover 15 subjects and three affective states—positive, neutral, and negative—across three sessions, with 15 trials per session.Source: §D.2 · p. 14
• Signals are filtered, resampled to 200 Hz, and amplitude-normalized; trials are split 9:3:3 for training, validation, and testing, with sessions merged.Source: §D.2 · p. 14
• Each preprocessed segment becomes an STFT image with 62 channel panels arranged in an 8 × 8 grid and paired with an emotion-classification prompt.The final image resolution is 896 × 896.Source: §D.2 · p. 14
• The prompt describes time–frequency panels whose color intensity reflects relative spectral power and requests one label: Positive, Neutral, or Negative.Source: §D.2 · pp. 14–15

D.3 Mental Arithmetic Task

• EEGMAT includes 36 subjects recorded across 19 channels during serial subtraction, representing high workload, and rest, representing low workload.Source: §D.3 · p. 15
• Non-overlapping 4-second windows are filtered, resampled to 200 Hz, and amplitude-normalized; subjects 0–25, 26–30, and 31–35 form training, validation, and test sets.Source: §D.3 · p. 15
• Each segment becomes an STFT image with 19 channel panels arranged in a 5 × 4 grid and paired with a mental-arithmetic detection prompt.The final image resolution is 560 × 896.Source: §D.3 · p. 15
• The prompt describes time–frequency panels whose color intensity reflects relative spectral power and requests one workload label: Low or High.Source: §D.3 · p. 15

D.4 Abnormality Detection

• TUAB is a clinical EEG corpus with 23 channels and expert-assessed normal or abnormal recording labels.Source: §D.4 · p. 16
• Recordings are divided into non-overlapping 10-second clips; signals are filtered, resampled to 200 Hz, and amplitude-normalized.Source: §D.4 · p. 16
• The official patient-level training and test split is retained, with the training set further divided 80:20 into training and validation subsets.Source: §D.4 · p. 16
• Each segment becomes an STFT image with 23 channel panels arranged in a 6 × 4 grid and paired with an abnormality-detection prompt.The final image resolution is 672 × 896.Source: §D.4 · p. 16
• The prompt describes time–frequency panels whose color intensity reflects relative spectral power and requests one label: Normal or Abnormal.Source: §D.4 · p. 16