NeuralBench: A Unifying Framework to Benchmark NeuroAI Models

1 Introduction

• Existing benchmarks cover restricted downstream evaluations, and no comprehensive framework standardizes brain foundation-model evaluation across neuroimaging modalities.Source: §1 · pp. 2–3
• NeuralBench is designed to maximize task and device coverage, validate datasets, task framing and preprocessing, and provide a unified entry point for multimodal pipelines.Source: §1 · p. 3
• NeuralBench is configurable across data sources, preprocessing, architectures, optimization and metrics through a compact interface for running and evaluating brain models.Source: §1 · p. 3
• NeuralBench-EEG v1.0 supports 36 tasks across 94 EEG datasets, adds cognitive decoding tasks often omitted from prior benchmarks, and includes illustrative MEG and fMRI tasks and models.Source: §1 · p. 3
• Current EEG foundation models only marginally outperform task-specific baselines, while cognitive decoding and clinical prediction remain challenging even for foundation models.Source: §1 · p. 3
• The authors release the codebase as a starting point for community collaboration across tasks and modalities.Source: §1 · p. 3

2.1 Benchmark architecture

• NeuralBench combines NeuralFetch for curated public datasets, NeuralSet for PyTorch-ready data and preprocessing, and NeuralTrain for modular model training.Task configurations specify data sources, splits, preprocessing, target processing, trainer settings and metrics.Source: §2.1 · p. 4

2.2 Task and dataset selection

• The EEG release selects 36 representative tasks across eight categories, prioritizing open datasets and challenging cognitive decoding and multilabel event-classification tasks.Cognitive tasks recover dense stimulus representations; event and artifact classification are reframed as multilabel tasks to accommodate overlapping events.Source: §2.2 · p. 5
• The benchmark prioritizes breadth with a core dataset per task and offers a Full variant with up to 24 datasets per task; overall it covers 94 datasets, 9,478 subjects and 13,603 hours.Some foundation models encountered portions of downstream datasets during pretraining; these overlaps are flagged rather than discarded, as the authors observed no clear trend of improved performance on the same data.Source: §2.2 · p. 5

2.3 Model architecture selection

• The benchmark includes eight task-specific architectures trained from scratch and six pretrained EEG foundation models typically finetuned on downstream tasks.Source: §2.3 · pp. 5, 7
• Table 1 catalogs each task’s core dataset, objective, channel and subject counts, target dimensionality, and any additional datasets in the Full variant.Source: Table 1 · p. 6
• Handcrafted-feature pipelines provide a non-deep-learning comparison, alongside chance-level and dummy baselines that establish performance floors.Source: §2.3 · p. 7

2.4 Downstream strategy protocol

• Foundation models share an end-to-end finetuning recipe with average-pooled outputs, a linear projection head, and common optimization settings, except for BENDR’s stability adjustments.Task-specific architectures are trained from scratch; model-specific adaptation techniques are intentionally deferred.Source: §2.4 · p. 7

2.5 Preprocessing

• Task-specific architectures use a common preprocessing configuration, while foundation models follow their original pipelines to match pretrained-checkpoint input statistics.The task-specific configuration resamples to 120 Hz, filters 0.1–75 Hz, applies notch filters and robust scaling, then clamps at 20.Source: §2.5 · p. 7

2.6 Data Splitting

• Dataset splits vary by task, using predefined, leave-concept-out, cross-subject, or random strategies, and each task is run with three random seeds.Clinical cross-subject splits may use stratification; all three runs share one train/validation/test partition.Source: §2.6 · p. 7

2.7 Training and evaluation metrics

• Training objectives and evaluation metrics vary by task, with retrieval using a brain-to-target-only CLIP loss at fixed temperature τ = 1.Classification uses cross-entropy, multilabel classification uses binary cross-entropy, and regression uses MSE; retrieval supports comparisons without assuming categorical targets.Source: §2.7 · p. 8

2.8 Model comparison

• Task comparisons report one representative metric per task type and normalize scores so dummy performance is 0 and theoretical perfect performance is 1.Global rankings average each model’s per-task rank; NeuralBench-EEG-Full first averages ranks across datasets of the same task.Source: §2.8 · p. 8

2.9 Computational considerations

• NeuralBench-EEG-Full requires 4,947 experiments and about 11 TB of disk space, with an estimated serial runtime of about 73 days.Median experiment time is 2.7 minutes, mean is 21.7 minutes, and the longest run takes 18.2 hours.Source: §2.9 · p. 8
• Figure 3 orders models by mean normalized rank and shows recent foundation models outranking most smaller task-specific models across downstream tasks.The figure also reports rank distributions, trainable parameters excluding the final projection, and publication year.Source: Figure 3 · p. 9

3.1 NeuralBench-EEG v1.0: Evaluating EEG foundation models on 36 diverse downstream tasks

• Foundation models have the highest average task ranks, while task-specific architectures also perform well despite much smaller parameter counts.CTNet has 150K parameters versus 69.2M for REVE.Source: §3.1 · p. 9

3.2 Future-proofing evaluation through task diversity and difficulty

• Common EEG tasks approach saturation, but cognitive decoding, rarer paradigms, and cross-subject motor imagery, P300, and N2pc remain challenging.The difficult tasks remain far from the performance ceiling, making them useful benchmarks for future EEG models.Source: §3.2 · pp. 9–10, Figure 5 · p. 11
• Foundation models often outperform task-specific models, but not on every task; REVE leads sentence, speech, and video decoding.Foundation models do not uniformly lead on evoked-response and clinical tasks.Source: §3.2 · p. 10, Figure 5 · p. 11

3.3 From NeuralBench-Core to NeuralBench-Full: Leveraging multi-study tasks for evaluating cross-dataset variability

• Core and Full model rankings are strongly concordant (Kendall’s τ = 0.926, p < 0.001), though broader dataset coverage can change positions, including CTNet overtaking LUNA.The Figure 6A comparison suggests Core rankings are a good proxy for Full rankings.Source: §3.3 · p. 11, Figure 6 · p. 12
• Rankings vary across datasets, especially for motor imagery and SSVEP; REVE and LaBraM are the most stable foundation models, while SSVEP winners differ by dataset.Task-specific models lead on one SSVEP dataset, and handcrafted features lead on several others.Source: §3.3 · p. 11, Figure 6 · p. 12
• Multiple datasets per task enable testing generalization and invariance across recording hardware, labs, and subject populations, as well as finetuning on some datasets and evaluating on held-out ones.NeuralBench-EEG-Full adds datasets beyond each task’s representative Core dataset.Source: §3.3 · pp. 11, 13

3.4 Extending NeuralBench to more modalities: MEG and fMRI

• The EEG-focused framework demonstrates support for two MEG tasks and one fMRI task; on MEG, EEG-pretrained REVE outperforms other models on typing decoding.The reported fMRI results use two task-specific architectures.Source: §3.4 · p. 13, Figure 7 · p. 13

EEG foundation vs. task-specific models

• Current EEG foundation models only marginally outperform specialized models, and downstream gains from self-supervised pretraining remain limited.The authors suggest finetuning in data-rich settings may let specialized models close the gap, and recommend evaluating few-shot and low-data regimes such as 5% of training data.Source: §4 · p. 13

EEG task expansions

• Performance remains modest across most expanded cognitive, phenotyping, clinical, and BCI tasks, which should remain diverse and representative of neuroscience and clinical applications.The authors recommend including both easier and more difficult tasks.Source: §4 · p. 14

Unique optimization strategies

• A standardized training recipe evaluates each architecture’s out-of-the-box utility rather than using model-specific optimization.The authors frame adaptability and stability under standard configurations as requirements for practical usefulness to the broader research community.Source: §4 · p. 14

Unique splitting strategy

• The fixed splitting strategies reduce computational burden and broaden task coverage, but limit assessment of performance variability compared with cross-validation.The authors invite future releases to incorporate stochastic partitioning for deeper statistical validation.Source: §4 · p. 14

Unique finetuning strategy

• End-to-end finetuning may not capture foundation models’ full potential; linear probing, parameter-efficient finetuning, and token aggregation are proposed for future study.The passage cites Hu et al. (2022) and Lester et al. (2021) for parameter-efficient finetuning.Source: §4 · p. 14

Beyond EEG

• The first release focuses on EEG but includes preliminary MEG and fMRI tasks and models, with promising transfer from EEG to MEG.Extending to new modalities requires substantial data curation and presents device-specific challenges, including fMRI autocorrelation that complicates train/test splits.Source: §4 · p. 14

Beyond decoding

• The current benchmark evaluates decoding models, while encoding, forecasting, and denoising tasks remain outside its present scope.The authors note these tasks can be accommodated, though integrating them presents challenges such as standardizing stimulus representations and evaluation metrics.Source: §4 · p. 14

Open call for community contribution

• The authors open-source NeuralBench and invite community contributions of datasets, tasks, models, and modalities to support a broader benchmarking effort.They describe unified, rigorous, reproducible, and comprehensive evaluation as important to integrated multimodal brain-data processing.Source: §4 · p. 14

Dataset references and core datasets

• Dataset references are provided in Table S1, while Table 1 gives detailed information about each task’s core datasets.References are not repeated when multiple datasets come from the same paper.Source: A Additional details on downstream… · p. 26

Stimulus use

• Stimuli are used only for cognitive decoding tasks that require alignment of brain data to dense stimulus representations; other tasks retain targets and discard stimulus information.Examples of retained targets include class labels for classification tasks.Source: A Additional details on downstream… · p. 26

Task documentation

• The benchmark documentation provides detailed, up-to-date descriptions of downstream tasks.The passage directs readers to the NeuralBench repository documentation.Source: A Additional details on downstream… · p. 26

Model categories

• Task-specific deep-learning architectures are among the model categories used on EEG downstream tasks.The passage lists them alongside handcrafted-feature sklearn baselines and foundation models.Source: B Task-specific architectures and… · p. 26
• Handcrafted-feature sklearn baselines are included among the model categories used on EEG downstream tasks.The passage also lists task-specific deep-learning architectures and foundation models.Source: B Task-specific architectures and… · p. 26
• Foundation models are included among the model categories used on EEG downstream tasks.The passage also lists task-specific deep-learning architectures and handcrafted-feature sklearn baselines.Source: B Task-specific architectures and… · p. 26

Provided splits

• Each dataset is split into fixed training, validation, and test partitions; provided development and evaluation splits are reused when available.The authors identify this approach for TUEV, TUAB, THINGS-EEG2, and BEETL AI Challenge datasets.Source: C Detailed splitting strategy · p. 26

Cognitive decoding splits

• Cognitive decoding generally uses leave-concept-out evaluation, with subjects appearing in both training and test sets while sessions or concepts are held out.The passage describes task-specific test-set construction for speech, sentence, word, video, and typing decoding.Source: C Detailed splitting strategy · p. 26

Cross-subject splits

• Most other tasks use cross-subject splits, typically holding out 20% of subjects; HBN-EEG predictions instead test on release 5, about 329 subjects.Clinical diagnosis splits are stratified by diagnosis label, and the authors note cross-subject generalization can be challenging for some tasks.Source: C Detailed splitting strategy · p. 26

Small-dataset splits

• Some datasets with fewer than 10 subjects use within-subject splits, while three low-sample datasets use fully random example-level splits.Within-subject splits hold out later sessions or runs; the named low-sample tasks include audiovisual stimulus and motor imagery classification.Source: C Detailed splitting strategy · pp. 26–27

Validation and repeated training

• Validation data is sampled from remaining data using the test split strategy or a similar one, targeting 20% of the overall dataset size; each model is trained three times with different initialization seeds.The validation sets are used for early stopping.Source: C Detailed splitting strategy · p. 27

• Binary and multiclass classification use balanced accuracy, with class count and true-positive and false-negative counts defined for its calculation.Source: D Performance metrics · p. 27
• Multilabel classification uses macro F1, with precision and recall defined for each class.Source: D Performance metrics · p. 27
• Univariate regression uses Pearson correlation between true and predicted values, their means, and the sample count.Source: D Performance metrics · p. 27
• Retrieval uses top-5 accuracy, with five retrieved candidates assessed per query against the true label or relevant item.Source: D Performance metrics · p. 27
• NeuralBench computes additional metrics, including RMSE for regression and median ranks for retrieval, but omits them from this report for brevity.Source: D Performance metrics · p. 27

• Figure S1 shows max-normalized scores over core datasets.Source: E Additional results on… · p. 27

• Figure S1 normalizes scores between dummy performance and the per-task best model to highlight variability relative to the highest-performing model.Source: Figure S1 · p. 28
• The release tables catalog task datasets and references, task-specific and foundation models, and handcrafted-feature baselines.The feature-based pipelines use the same preprocessing and splits as the deep learning models.Source: Table S1 · p. 29, Table S2 · p. 30, Table S3 · p. 30
• The model comparison includes chance, dummy, handcrafted, and pretraining-seen baselines, alongside task-specific architectures and foundation models.Source: F Additional results on… · p. 31
• Figure S2 compares task-specific and foundation-model performance on multi-dataset NeuralBench-EEG-Full v1.0 tasks.Source: F Additional results on… · pp. 27, 31