PaperlayerView exact revision on arXiv

Source-linked AI summary

A Language-Guided Multimodal Foundation Model for Zero-Shot and Multi-Task Brain Signal Analysis

Mingzhi Chen, Yiyu Gui, Guibo Luo, Yuchao Yang

arXiv:2609.15740v1cs.LGcs.AI

TL;DR

Brain-signal analysis needs models that generalize beyond task-specific training while capturing the semantic structure of heterogeneous signals. METIS uses unified language-signal alignment and large-scale pretraining to support zero-shot and multi-task analysis, outperforming existing approaches across evaluation settings. The authors position it as a general-purpose assistant that can reduce dependence on labeled data, while noting the need for broader prospective validation.

  • Problem

    Heterogeneous brain signals limit existing models’ cross-task and cross-dataset generalization, while general-purpose multimodal models lack the domain knowledge needed to interpret them.

  • Method

    METIS aligns heterogeneous brain signals with natural-language instructions through a unified language-signal framework and large-scale pretraining.

  • Results

    METIS surpasses existing end-to-end and pretrained models in zero-shot and multi-task settings with minimal or no additional training, including 19.8% and 15.9% average cross-dataset advantages.

  • Takeaways & Limitations

    METIS supports general-purpose brain-signal analysis with natural-language instructions and may reduce reliance on labeled data and deployment time.

  • Takeaways & Limitations

    The training data are primarily from existing studies and require broader prospective, multi-center evaluation across populations, rare diseases, and acquisition hardware.

Abstract

from arXiv · show

Brain signal analysis is essential for both neuroscience research and clinical diagnostics, yet current approaches face critical limitations. End-to-end models require task-specific retraining and exhibit limited generalization, while pre-trained models lack semantic depth and still depend on extensive fine-tuning. Meanwhile, general-purpose multimodal foundation models, though powerful in other domains, struggle to interpret brain signals due to representational misalignment and lack of domain knowledge. This study introduces a multimodal foundation model for zero-shot and multi-task brain signal analysis (METIS) through a unified language-signal alignment framework. METIS is pretrained on the largest and most diverse brain-signal corpus to date, comprising over 70,000 h of recordings from more than 11,000 subjects across 20 datasets. In a comprehensive zero-shot evaluation across 12 datasets, METIS outperformed the leading generalist model by over 20.9% in average accuracy. Remarkably, without any fine-tuning, METIS's performance matches or exceeds that of supervised, task-specific models. Furthermore, METIS demonstrates exceptional data efficiency and strong generalization, achieving an average AUROC advantage of over 16.0% in few-shot settings and 15.9% in cross-dataset transfer. This work establishes a new paradigm for general-purpose brain signal analysis, paving the way for next-generation neurotechnology.

1 | Introduction

Brain-signal models struggle with heterogeneous data, task-specific retraining, limited semantic depth, and poor transfer from general-purpose multimodal models. METIS addresses these challenges with unified language-signal alignment and broad evaluation across diverse datasets and settings.

  • 1 | Introduction: Heterogeneous electrode setups, hardware, and participant cohorts make existing brain-signal models difficult to generalize across tasks and datasets.Task-specific end-to-end and pretrained models increase development costs and limit zero-shot and multi-task adaptability.
  • 1 | Introduction: General-purpose multimodal foundation models lack sufficient domain knowledge to interpret complex brain-signal dynamics, causing performance degradation when applied directly.Their capabilities in instruction following and reasoning do not transfer reliably to non-visual brain signals.
  • 1 | Introduction: METIS uses a universal signal encoder, multimodal attention, and mixture-of-experts module within a unified language-signal alignment framework.It formulates neural-state assessment and disease identification as question-answering tasks using brain signals and natural-language prompts.
  • 1 | Introduction: Over 70 000 h of recordings from more than 11 000 subjects across 20 datasets support METIS pretraining, followed by evaluation across 17 datasets and multiple task settings.The experiments compare METIS with specialized and general-purpose multimodal foundation models in zero-shot, few-shot, and cross-dataset transfer scenarios.
  • 1 | Introduction: METIS surpassed the best end-to-end and pretrained models in cross-dataset transfer by averages of 19.8% and 15.9%, respectively.These results support its reported generalizability across diverse brain-signal analysis settings.

2 | Results

METIS uses language-signal alignment and large-scale heterogeneous pretraining to support zero-shot, few-shot, and cross-domain brain-signal analysis. Across clinical tasks, it consistently outperforms specialized and general-purpose baselines while maintaining broad, stable performance with limited supervision.

  • Cross Dataset Transfer: 63.2% zero-shot AUROC on RatEpilepsy exceeded untrained baselines by more than 12% despite no prior rat-data exposure.With 10% target-domain fine-tuning, performance reached 83.6% AUROC.
  • Zero-Shot Generalization Across Diverse Clinical Tasks: 75.5% average zero-shot AUROC exceeded the 1%-shot supervised average of 73.2% and approached the 10%-shot level of 81.6%.With minimal fine-tuning, average AUROC increased to 82.0% with 1% data and 87.6% with 10% data.
  • Zero-Shot Generalization Across Diverse Clinical Tasks: 78.3% average accuracy across four datasets, outperforming ChatGPT by 28.6% in multiple-choice zero-shot evaluation.METIS also achieved a 70.7% average BERTScore, 17.0% above Grok, in open-ended evaluation.
  • Few-Shot Classification: 95.2% and 95.7% AUROC on Dreams under 2-shot and 8-shot settings exceeded BIOT by 39.8% and 34.0%, respectively.On Mayo, METIS achieved 93.6% and 93.7% AUROC under the same settings.
  • Cross Dataset Transfer: METIS’s balanced radar profile indicates task-invariant representations, unlike baselines whose uneven performance depends more heavily on task-specific properties.The model also maintained strong transfer on the unseen anesthesia-depth task, reaching 78.0% and 83.8% AUROC at 2 and 8 shots.

3 | Discussion

METIS reframes brain signal analysis as a general-purpose, language-guided foundation-model task, addressing limitations of task-specific models through unified signal-language representations and modular routing. Its reported advantages span zero-shot performance, cross-task and cross-dataset generalization, data efficiency, and potential clinical scalability, while broader prospective validation remains necessary.

  • 70.4% zero-shot average accuracy exceeded general-purpose multimodal foundation models’ 49.5% across 17 downstream datasets.METIS also surpassed existing end-to-end and pretrained models in zero-shot and multi-task settings with minimal or no additional training.
  • METIS aligns heterogeneous brain signals with natural-language instructions in a unified space and uses mixture-of-experts routing for adaptive computation.The framework maps signals from varied sources into shared representations while routing computation across specialized and broadly active experts.
  • METIS’s cross-dataset robustness is supported by topologically consistent representations and complementary specialized and generalist experts.The discussion links language-signal alignment, pretrained representations, and modular expert routing to robustness under distribution shifts.
  • Zero-shot performance surpassed supervised models trained with 1% labeled data and approached those trained with 10%, indicating potential to reduce annotation costs.The paper presents this comparison as evidence for practical data efficiency and reduced reliance on labeled data.
  • Broader applicability is bounded by training data drawn primarily from existing studies, with incomplete coverage of global populations, rare diseases, and acquisition hardware.The authors call for broader multi-center prospective cohorts to validate robustness and generalizability across clinical settings.
  • Future work targets scaling laws, million-hour pretraining data, and larger mixture-of-experts architectures to advance brain-signal foundation models.These directions are presented as three trajectories for extending the framework, with the supplied passage specifically supporting scaling-related expansion.

4 | Experimental Section

METIS combines a universal brain-signal encoder with a decoder-only multimodal Transformer, MoE computation, and hybrid signal-language attention. Instruction-driven autoregressive generation aligns neural dynamics with clinical semantics for unified analysis.

  • Instruction-Driven Pretraining: Overview and Objective: METIS learns to map temporally dynamic neural representations to high-level clinical semantics through signal-conditioned autoregressive generation.The objective supports zero-shot reasoning across heterogeneous brain-signal tasks without task-specific supervision.
  • Instruction-Driven Pretraining: Overview and Objective: METIS reformulates EEG and iEEG analysis as instruction-driven generation over concatenated signal, prompt, and answer sequences.The model uses a universal signal encoder, multimodal Transformer backbone, and modality-aware mechanisms for signal-language integration.
  • Universal Signal Encoder: Signal inputs are normalized, converted into log-scaled spectrograms with STFT, projected into channel-wise tokens, then fused after cross-channel attention and pooling.This produces a compact signal-token sequence that can be concatenated with embedded instruction tokens.
  • Transformer Backbone With Group Query Attention and Mixture-of-Experts: The decoder uses GQA and token-level sparse MoE layers, activating only top-k experts while a shared expert captures task-agnostic patterns.An auxiliary load-balancing loss encourages balanced expert utilization.
  • Modality-Aware Attention: A hybrid attention mask makes signal-token interactions bidirectional while enforcing causal text decoding and full text access to the signal sequence.This preserves within-signal integration and autoregressive answer generation across modalities.

4.3 | Pretraining Corpus Curation and Instruction-Answer Construction

METIS training uses a heterogeneous corpus of EEG and iEEG datasets converted into standardized instruction-answer pairs. The corpus spans diverse clinical and signal-analysis tasks while preserving dataset-specific acquisition and labeling protocols.

  • Pretraining Corpus Curation and Instruction-Answer Construction: The pretraining corpus combines 20 EEG and iEEG datasets covering sleep, epilepsy, neurological and psychiatric disorders, anomalies, and motor imagery.The datasets vary in channel configurations, sampling frequencies, subject counts, and annotation conventions.
  • Pretraining Corpus Curation and Instruction-Answer Construction: Each dataset receives standardized preprocessing that respects its native acquisition and labeling protocols before instruction-answer pairing.Signals were minimally cleaned according to source recommendations, and each instance was paired with a standardized natural-language instruction and canonical answer string.
  • Pretraining Corpus Curation and Instruction-Answer Construction: METIS uses open-ended and multiple-choice question-answering formats with fixed dataset-specific label sets and one instruction sentence per dataset.Multiple-choice prompts embed candidate answers as lettered options, while open-ended prompts require a canonical answer string.

4.4 | Evaluation Protocols

The evaluation tests METIS through uniform zero-shot classification, low-label linear probing, and source-to-target cross-dataset transfer. Experiments span heterogeneous datasets and clinical task types to assess generalization, label efficiency, and robustness.

  • Zero-Shot Classification: Zero-shot classification extracts logits for candidate answer tokens and selects the highest-probability candidate without decoding the full output sequence.This targeted decoding is applied uniformly across downstream classification tasks.
  • Few-Shot Evaluation: Few-shot evaluation tests 1-, 2-, 4-, and 8-shot settings using frozen pretrained backbones with only a task-specific linear head trained.Evaluation uses repeated random sampling or five-fold subject-wise cross-validation depending on dataset splits.
  • Cross-Dataset Transfer: Cross-dataset transfer trains on all available source data and tests on a distinct target dataset, using frozen-backbone linear probing for pretrained models.This protocol isolates learned representation quality under domain shift.
  • Evaluation Datasets: The benchmark comprises 17 datasets spanning sleep, epilepsy, neuropsychiatric and neurodegenerative disorders, anomalies, and anesthesia-related tasks.The datasets vary in channel configuration, sampling frequency, cohort size, and segmentation protocol.
  • Evaluation Datasets: The evaluation includes diverse EEG and iEEG settings, including sleep-stage, seizure, interictal-discharge, psychiatric, neurodegenerative, and cross-species tasks.Examples include ISRUC and Dreams for sleep staging, Mayo and IEDS for iEEG, and RatEpilepsy for cross-species assessment.

4.6 | Evaluation Metrics

Evaluation uses AUROC for classification and two question-answering protocols: multiple-choice accuracy and open-ended BERTScore.

  • AUROC is the primary classification metric because it is threshold-free and less sensitive to class imbalance than accuracy.
  • For binary tasks, AUROC is computed from positive-class scores as the probability that a randomly chosen positive ranks above a negative.
  • For multiclass tasks, the reported metric is macro-averaged one-versus-one AUROC across all unordered class pairs.
  • Multiple-choice evaluation selects the highest-probability candidate, whereas open-ended answers are scored with the F1 form of BERTScore against canonical references.

Supporting Information

The supporting information documents label inventories, instruction-answer templates, dataset compositions, and additional zero-shot evaluation examples and comparisons.

  • Supporting tables provide instruction-answer templates and canonical label inventories for TDBrain and TUSZ.
  • Supplementary figures show class composition through overall counts and per-fold proportions across multiple datasets.
  • Additional figures cover class composition for further datasets, including ADHD, schizophrenia, mood, epilepsy, and signal-anomaly benchmarks.
  • Supplementary examples illustrate zero-shot multiple-choice and open-ended outputs across sleep, psychiatric, seizure, and anomaly-detection tasks.
  • Figure S20 compares METIS with generalist multimodal models using multiple-choice accuracy and open-ended BERTScore across 12 datasets.

Model Configuration and Architectural Details

METIS uses a compact Transformer configuration with normalized attention and distinct feed-forward designs for standard and mixture-of-experts layers.

  • METIS has 12 Transformer layers with hidden dimension 512 and query-key normalization for training stability.
  • Standard feed-forward layers use intermediate dimension 2048, while routed and shared MoE experts use dimensions 512 and 1024, respectively.
  • The architecture combines standard Transformer components with mixture-of-experts feed-forward layers.

MoE Contribution Under Matched Model Scale

A parameter-matched comparison indicates that METIS’s sparse mixture-of-experts design improves overall zero-shot performance beyond total model scale alone.

  • 0.7550 mean zero-shot AUROC for METIS-MoE versus 0.6801 for METIS-Dense yielded a 7.49-percentage-point average gain across 12 datasets.
  • METIS-Dense matched model scale and evaluation conditions while replacing sparse expert feed-forward layers with dense feed-forward layers.
  • METIS-MoE outperformed METIS-Dense on 9 of 12 datasets, with especially large gains on several psychiatric and clinical signal benchmarks.
  • METIS-Dense performed slightly better on ISRUC, Dreams, and NMT, showing that dense capacity remained competitive for some structured tasks.

Dataset Usage and Overlap Control

Table S6 distinguishes pretraining from downstream evaluation datasets, while the study reports disjoint corpora and partitions without subject or recording overlap.

  • Table S6 maps each dataset to its pretraining or downstream-evaluation role.
  • Pretraining and downstream evaluation used disjoint datasets, with no subject-level overlap or inadvertent recording reuse.
  • Downstream training, validation, and test partitions shared no subjects or recordings, and official dataset splits were followed.

Effect of Pretraining Instruction Quality

Task-specific natural-language instructions materially improve zero-shot performance, especially by conditioning METIS on the appropriate semantic decision space across heterogeneous tasks.

  • 0.7550 mean AUROC with full instructions versus 0.6600 with weak instructions yielded a 9.50-percentage-point average gain across 12 datasets.The full-instruction model outperformed the weak-instruction model on 11 of 12 datasets.
  • Weak instructions caused smaller gaps on tasks related to high-frequency pretraining types but larger degradation on more heterogeneous clinical tasks.Strong signal-label associations and label options could partly compensate for weaker instructions in some related tasks.
  • Task-specific instructions help condition the model on the appropriate semantic decision space rather than dominant task distributions or label-option associations.
  • The findings indicate that language instructions contribute to task-aware signal-language alignment and improve zero-shot generalization across diverse clinical brain-signal tasks.

Prompt and Label Phrasing Robustness

METIS was tested under semantically equivalent prompt and label changes rather than exact template matching. Mean zero-shot performance declined modestly overall, although sensitivity varied substantially by dataset.

  • Prompt paraphrases and alternate labels were constructed as semantically equivalent variants grounded in original annotation protocols and standard clinical terminology.
  • 0.7301 mean AUROC under paraphrased prompts and 0.7235 under alternate labels decreased from the 0.7550 canonical setting by 2.49 and 3.15 percentage points.
  • METIS maintained overall stable zero-shot performance under controlled prompt and label surface-form changes.The robustness evaluation changed only instruction or answer-label surface forms while keeping model parameters, datasets, protocol, answer space, and annotations fixed.
  • Sensitivity was task-dependent: several datasets remained highly stable, whereas Mayo, IEDS, ADHD-121, ADFSU, and NMT showed larger drops.Tables S9 and S10 report per-variant results, with mean and maximum-drop summaries defined relative to canonical phrasing.
Loading 2609.15740v1…