Controlled Evaluation of EEG Foundation Models: Random Initialisation, Band-Power and Validation-Selected Probes

• The evaluation addresses three commonly missing controls: a matched random-initialisation encoder, same-split band power, and validation-based layer selection.The introduction notes that reported protocols differ, making published numbers incomparable across papers.Source: §1 · p. 1
• 0.507 AUROC for the in-house model’s output-layer probe on Lee2019 was below the five-band log-power baseline’s 0.741 AUROC.An early front-end stage reached 0.708, while later stages lost that signal again.Source: §1 · p. 2
• The study evaluates six models under three controls across 30 model-dataset pairs, with five-seed frozen probing and fine-tuning plus four added task families.The added families are ERP, sleep, emotion, and within-subject MI; LuMamba is absent from within-subject MI.Source: §1 · p. 2
• Most fine-tuning gains sit within seed noise, while test-selected layers inflate peaks and band-power baselines are rarely beaten.The evaluation also finds random initialisation often matches pretraining under a frozen probe.Source: §1 · p. 2
• The paper distinguishes frozen-probe representation quality from trainability, defined as fine-tuning gain over random initialisation.Per-model tuning performance is a separate notion that the shared protocol bounds from below rather than measures.Source: §1 · p. 2

2 Related work

• The evaluated foundation models use varied objectives and input handling, but prior comparisons rely on published results from different protocols.The paper instead runs every model under one protocol with all three controls.Source: §2 · p. 2
• The study applies fixed-protocol controlled re-evaluation to EEG foundation models and adds validation-based layer selection because layer-wise probing yields many candidate values.Prior work includes reproducibility checklists, re-labelled benchmarks, and fixed-protocol comparisons (Musgrave et al., 2020).Source: §2 · p. 2
• Earlier EEG benchmarks report small gains over compact architectures, standardise cross-paradigm comparisons, or find specialist models trained from scratch competitive.The passage cites Lee et al. (2025), Xiong et al. (2026), and Liu et al. (2026), respectively.Source: §2 · p. 2
• The comparison includes canonical-band power, EEGNet, and MI filter-bank spatial filters on the same splits as the foundation models.These baselines represent classical EEG decoding approaches (Babiloni et al., 2009; Pfurtscheller & Lopes da Silva, 1999; Lawhern et al., 2018; Ang et al., 2008).Source: §2 · p. 2

3.1 Models

• The panel pairs the in-house model with five public EEG foundation models loaded from repository-pinned checkpoints; input handling differences are confounded with corpus and implementation maturity.Source: §3.1 · pp. 2–3

3.2 The in-house model

• The in-house JEPA-style model encodes unmasked windows before replacing masked outputs, predicts masked tokens against stop-gradient encoder projections, and regularizes those projections with SIGReg.Its released sinc filters act as low-pass filters with upper −3 dB edges of 0.24–0.73 Hz; Lee2019 output retains 0.58% of 8–30 Hz power versus 14% at input.Source: Figure 1 · p. 3, §3.2 · p. 4

3.3 Benchmarks and seed semantics

• The five original benchmarks use five-second, 200-Hz windows; four added task families share five-second reductions, while some checkpoint inputs only partly match pretraining.Seeds redraw subject-disjoint splits for sleep, emotion, TUSL, and TUAB; other listed benchmarks keep one split.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4 Controlled protocols

• Each frozen probe uses a validation-selected stage and is compared with matched random-init and five-band log-power references; fine-tuning gain is pretrained minus random-init at the best-validation epoch.A gain is called clear when it exceeds twice its seed standard deviation; this is a descriptive heuristic, not a corrected test.Source: §3.4 · p. 5

3.5 Diagnostic protocol

• The diagnostic protocol labels frozen-probe and fine-tuning gain signs to distinguish representation from trainability defects, tracks probes across pretraining checkpoints, and evaluates public models only post hoc.Source: §3.5 · pp. 5–6

4.1 Developing the in-house model

• The in-house sinc front end raised the Lee2019 probe to 0.715 versus 0.691 for band power, although its filters are low-pass rather than band-pass.In the final run, the pretrained-minus-random gain peaked at different checkpoints across benchmarks and had fallen on three of four benchmarks by 40,000 steps.Source: §4.1 · pp. 6–7

4.2 Choosing the layer on test inflates layer-wise results

• Validation selection lowers layer-wise peaks by +0.034 on average; 9 of 36 peak-versus-output gaps reverse, while 11 intermediate-layer gaps survive.The surviving largest gap is +0.202 for the in-house model on Lee2019, where the random-init curve follows the pretrained curve.Source: §4.2 · p. 7

4.3 The band-power anchor is rarely beaten

• Band power beats all six models on emotion and all five tested models on within-subject MI; sleep is closer, with five models clearing band power.For sleep, three of the five models that clear band power score higher still at random initialisation.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 A model that sees mostly slow components

• The in-house model ranks first among evaluated models on four motor-imagery and event-related tasks, but last on APAVA, sleep staging, and emotion.Its low-pass front end preserves slow and event-locked potentials better than band power above a few hertz; full fine-tuning reverses its frozen-probe ordering with CBraMod at 0.848 versus 0.515 on Lee2019.Source: §4.4 · p. 8

4.6 Fine-tuning gains mostly sit within seed noise

• Only 5 of 30 pairs show positive full fine-tuning gains beyond twice their seed standard deviation, while 1 negative pair clears that threshold.LUNA on APAVA remains a clear loss at −0.290±0.035 under three shared learning rates; per-model learning-rate tuning would add +0.031 on average across 15 pairs at one seed.Source: Table 3 · p. 9, §4.6 · pp. 9–10

4.5 Random initialisation often matches pretraining

• On added tasks, random initialisation equals or beats pretraining for 3 of 6 models on sleep, 3 of 6 on emotion, 2 of 5 on within-subject MI, and 1 of 6 on ERP.Checks find an average 0.030 difference between random-init draws and no low-label advantage at 10% of training data.Source: §4.5 · p. 9

4.7 Nonlinear readouts recover little

• Attention pooling does not recover the output layer, and a one-hidden-layer MLP lifts the output-stage probe by more than three points in at most 1 of 18 cells.The attention-pooling comparison covers two models on three leakage-free benchmarks; its score is lower than mean pooling in all six cells.Source: §4.7 · p. 10

5.1 Reporting practices

• Report matched random-init encoders and same-split band power, validation-selected layers with full curves, seed variation, and shared-protocol results as comparison floors rather than architecture rankings.The discussion also recommends reporting tuning penalties where measurable.Source: §5.1 · p. 10

5.2 Scope of the conclusions

• The six-model panel cannot establish causal rankings of pretraining objectives or input-handling designs because corpus, tuning, and implementation are confounded.Added tasks use reduced single-window forms; clinical event detection and sleep sequence models are outside the protocol, and the diagnostic protocol was used during development on only one model.Source: §5.2 · p. 10

5.3 Limitations

• The evaluation is limited to five public checkpoints and one in-house model, a single fine-tuning recipe, and five-second single windows; five models and the in-house corpus contain TUH-derived recordings.TUSL and TUAB may therefore have been seen in pretraining; two released analyses use test-selected stages as upper bounds, and the in-house front-end mostly passes sub-hertz content. Development involved held-out pretraining windows and Lee2019, making the study exploratory rather than pre-registered.Source: §5.3 · pp. 10–11

6 Conclusion

• With a matched random twin, band-power baseline, and validation-selected layer, most reported gains shrink or vanish in this panel; fine-tuning gains mostly sit within seed noise.The one clearly negative pair persists under matched learning rates, and per-cell results are released for checking; the authors identify controls during pretraining as a next step.Source: §6 · p. 11

A Models and checkpoints

• Table 4 records each evaluated model’s checkpoint by repository, file, and pinned revision..Source: Table 4 · p. 13, A Models and checkpoints · p. 14

B Protocol constants

• Table 6 lists preprocessing, probe, and fine-tuning constants taken from the evaluation code at build time.Source: B Protocol constants · p. 14, Table 6 · p. 14

C The in-house front-end as implemented

• Table 7 reports spectral power passed by the released in-house front-end on Lee2019 and pure-sine gains for four filters, alongside corrected-filter comparisons.The corrected comparison uses the time index in samples rather than n/fs in Eq. 1 and does not retrain the model.Source: Table 7 · p. 15, C The in-house front-end as implemented · p. 15

• Figure 6 plots probed stage from input to output by dataset and model, distinguishing pretrained, random-init, band-power, test-argmax, and validation-selected results.Source: Figure 6 · p. 16
• Table 8 reports validation-selected peak minus output stage and the test-argmax selection penalty across 36 frozen-probe cells on three leakage-free benchmarks.It distinguishes AUROC for APAVA and Lee2019 from accuracy for BCI IV 2a; in-house Lee2019 cells carry developer contact.Source: Table 8 · p. 16
• For four added task families, Table 9 reports frozen linear-probe test metrics at the validation-selected stage as three-seed means, plus pretrained-minus-random-init differences computed before rounding and a five-band log-power baseline on the same split.Within-subject splits are deterministic, so pretrained values are point estimates and only random-init carries a seed band; seed standard deviations are at most 0.024.Source: Table 9 · p. 17

E Added task families

• Tables 9 and 10 give the numbers behind Figure 4 for added task families, with Table 10 reporting standard deviations over three seeds.A 0.000 standard deviation indicates a deterministic within-subject pretrained probe.Source: E Added task families · p. 17, Table 10 · p. 17

F Random-initialisation draws

• The 18 random-init cells are two independent draws, whose mean absolute difference of 0.030 estimates the resolution of a single frozen random-init value.One draw uses a fixed seed in the validation-selected run; the label-budget re-extraction uses no fixed seed, with the same probe seed.Source: F Random-initialisation draws · p. 17
• Table 13 compares the canonical full-fine-tuning learning rate with each model’s validation-selected rate from 2e-5, 5e-5, and 1e-4, using one seed.Its penalty is the amount the canonical protocol gives away; positive values indicate tuning would have helped.Source: Table 13 · p. 18

G Readout robustness

• Attention-pooling readouts are compared with output-stage mean-pool linear probes, and one-hidden-layer MLP probes with output-stage linear probes.The reported attention-pooling comparison uses three seeds; the MLP comparison is for pretrained encoders and reports mean±std over three seeds.Source: Table 11 · p. 18, Table 12 · p. 18, G Readout robustness · p. 18

H Tuned anchor for fine-tuning

• Tuning lowered the test score in 2 pairs, while the canonical learning rate was retained in 4 pairs.Table 13 gives the per-model learning-rate grid.Source: H Tuned anchor for fine-tuning · p. 19

I Matched learning rates

• LUNA’s APAVA fine-tuning gain stays negative at every shared learning rate.The matched-rate runs use AUROC at the best-validation epoch over five seeds; the asymmetric schedule is a fresh run that differs from Table 3 within seed noise.Source: I Matched learning rates · p. 19, Table 14 · p. 19

J Label budget

• 3 of 18 full-budget pretrained-minus-random gaps disagree in sign with Table 18, and each larger gap is at most 0.085.Table 15 comes from a separate feature-extraction run; the passages do not attribute the sign flips to its different seed set or validation-selected stage versus Table 18’s test-argmax stage.Source: J Label budget · pp. 19–20, Table 15 · p. 19

K Within-subject against subject-independent motor imagery

• Table 16 contrasts subject-independent and within-subject BCI IV 2a protocols for the same models.The table caption specifies pretrained frozen-probe accuracy for four classes, with chance at 0.25, using the same seed and hyperparameters.Source: K Within-subject against… · p. 20, Table 16 · p. 20

L Four-mode labels

• At the default threshold of 0.01, 3 model-dataset pairs are labelled frozen-only.The four-mode labels cover thirty pairs at three thresholds; gains within |∆| ≤ T count as mixed.Source: L Four-mode labels · p. 20, Table 17 · p. 20

M Frozen probe on the original benchmarks as released

• Table 18 gives the released frozen half of the matrix on five original benchmarks using test-argmax peak stages.For the three leakage-free benchmarks, Section 4.2 replaces test-argmax selection with validation selection; the table also includes a five-band log-power baseline.Source: Table 18 · p. 20, M Frozen probe on the original… · p. 20

N Development record of the in-house model

• Table 19 records in-house model development values copied from project documents, without surviving raw output.The build re-checks each value against its cited project-document line and fails if the value is no longer there.Source: Table 19 · p. 21, N Development record of the in-house… · p. 21

O Input handling as a hypothesis

• Full-resolution tokenisers have 5, 5, and 4 positive datasets of five; channel-compressed LUNA and LuMamba have 3 each, and fixed-vocabulary EEGPT has 3.Corpus, tuning maturity, and implementation maturity vary together with input handling, so this ordering is a hypothesis for a study varying input handling alone (Döner et al., 2025; Jaegle et al., 2021).Source: O Input handling as a hypothesis · p. 21