PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkNet: Evolutionary Deep Ensembles for Efficient Motor-Imagery EEG Classification

Abdul Basit, Saim Rehman, Muhammad Shafique

arXiv:2609.33967v2cs.HCcs.AIcs.LGeess.SP

TL;DR

Subject-independent MI-EEG decoding must select models for unseen users, yet reported performance depends on validation and evaluation choices. ThinkNet uses isolated LOSO validation to guide model and inference-policy selection; source validation was informative against uniform choice but did not outperform fixed-family EEGNet and remained 6.08 points below the test-best family.

  • Problem

    Reported MI-EEG accuracy depends on preprocessing, subject and session splits, model capacity, and whether validation choices are separated from held-out testing, complicating compact-decoder selection for unseen users.

    Source: §I · p. 1

  • Method

    ThinkNet uses train-only normalization, strict LOSO splitting, evolutionary search, selected retraining, and validation-gated inference without held-out-subject selection feedback.

    Source: A. Overview · p. 2, D. Evolutionary Configuration Search · p. 3

  • Results

    Source-validation family selection reached 40.01% versus 38.29% for uniform choice and 40.29% for fixed-family EEGNet, while the test-best family reached 46.09%.

    Source: §I · p. 1, A. Selection Reliability and Search… · p. 5

  • Takeaways & Limitations

    ThinkNet treats compact MI-EEG model and inference-policy selection as a validation-controlled evaluation problem rather than a single-architecture benchmark.

    Source: §I · p. 1

    Validation-gated policies show positive mean gains for some candidate sets, but benefits vary by subject and incur measured inference cost.

    Source: D. Discussion · p. 7

Abstract

from arXiv · show

Practical assistive and rehabilitative brain--computer interfaces require subject-independent motor-imagery EEG (MI-EEG) decoders that generalize to new users under limited target-user data and constrained compute. However, held-out-subject performance can be overstated when test-subject information influences preprocessing, model selection, or ensemble selection. We present \textit{ThinkNet}, a validation-controlled framework that combines train-only normalization, validation-guided evolutionary search, and validation-gated inference to identify compact decoders and inference policies for held-out subjects. We evaluate four-class BCI Competition IV-2a (session T) decoding with nine Leave-One-Subject-Out (LOSO) folds, three seeds, seven fixed decoder entries, and a broader search over ten representative decoder families; the held-out subject is never used for normalization, hyperparameter, architecture, or ensemble-policy selection. In the fixed benchmark, the validation-selected compact decoder achieved 44.35$\pm$15.41\% accuracy with 4.9K parameters, 19 KB FP32 weights, and 0.99 ms batch-1 Orin CUDA inference. Across the broader search, compact models ($\leq$25K parameters) achieved higher mean held-out accuracy than mid-size and large alternatives after selected retraining (40.10\% vs. 35.09\% and 34.78\%). Validation-gated ensembling improved over validation-selected single-model inference, reaching 43.98$\pm$16.25\% in the fixed benchmark and 43.31$\pm$15.88\% for the compact six-family ensemble. A non-deployable oracle analysis revealed a 6.1-point family-selection gap and near-zero validation--test correlation, showing that validation reliability remains a key bottleneck under subject shift. Thus, ThinkNet is a validation-controlled framework for compact MI-EEG model and inference-policy selection, rather than a single-architecture benchmark.

I. INTRODUCTION

MI-EEG decoder selection for unseen users is difficult because performance depends on subject and session conditions, preprocessing, and whether validation decisions are isolated from held-out testing. ThinkNet frames this as validation-controlled selection and evaluation, connecting prior decoder and adaptation work to an explicit analysis of source-validation reliability.

I. INTRODUCTION

  • 40.01% source-validation family selection exceeded uniform choice but did not outperform fixed-family EEGNet at 40.29%, leaving 6.08 points of regret to the test-best family.Within-fold rank transfer was weak (ρ = 0.256).Source: §I · p. 1
  • ThinkNet combines strict LOSO evaluation, train-only normalization, evolutionary search, and validation-gated inference to select models and policies without held-out-subject feedback.The paper presents this integration as a validation-controlled protocol, not a new convolutional block.Source: §I · p. 1

II. RELATED WORK

  • Prior work spans hand-designed classical pipelines, compact and larger deep decoders, and subject-adaptation methods, while reported performance depends on preprocessing, splits, and validation separation.ThinkNet addresses the source-only setting in which the held-out subject is unavailable for normalization, architecture, hyperparameter, or policy selection.Source: §II · p. 2
  • ThinkNet distinguishes its focus by making the gap between source-validation choices and held-out-subject outcomes an explicit analysis target.The related-work comparison separates compactness, subject-independent evaluation, automated selection, and validation control.Source: §II · p. 2

A. Overview

ThinkNet is a zero-target framework for subject-independent MI-EEG decoder and inference-policy selection, with preprocessing, LOSO splitting, evolutionary search, retraining, and validation-gated inference. Source-subject validation guides selection while the held-out subject is excluded from selection feedback.

A. Overview

  • ThinkNet combines preprocessing, LOSO splitting, evolutionary search, selected retraining, and validation-gated inference without held-out-subject selection feedback.It is a zero-target setting: target adaptation using unlabeled or labeled target epochs is not performed.Source: A. Overview · p. 2

B. Problem Definition

  • In each LOSO fold, one subject is held out for testing while the decoder is trained on source subjects and selected using source-validation data.The held-out subject is excluded from source-validation selection.Source: B. Problem Definition · p. 2

A. Overview

  • Fold-specific normalization, validation, and selection use source data without held-out-subject feedback, while fixed unsupervised signal cleaning uses no labels or model feedback.Source: A. Overview · p. 2, Figure 1 · p. 3

B. Problem Definition

  • Validation-controlled selection uses an accuracy–size fitness, with trade-off weights fixed before held-out testing and no held-out data in fitness calculations.The compact search uses weights (0.7, 0.3), and the broader family search uses (0.75, 0.25).Source: B. Problem Definition · p. 3

C. Preprocessing

ThinkNet applies common, source-controlled preprocessing before evolutionary search. Search spans a compact decoder configuration space and ten representative model families, with validation guiding selection and retraining.

C. Preprocessing

  • All models share preprocessing that includes EOG-guided ICA, 4–40 Hz band-pass filtering, a 50 Hz notch, average referencing, and cue-locked epoching.Channel-wise z-score statistics are estimated only from source-training trials and applied unchanged; ICA component selection uses EOG correlation, not labels or model feedback.Source: C. Preprocessing · p. 3
  • Ocular cleaning is independently estimated within each recording and performed before LOSO training without labels, predictions, validation scores, or selection feedback.The figure illustrates fixed, unsupervised ICA-based ocular artifact attenuation on BCI IV-2a subject A01T.Source: C. Preprocessing · p. 3, Figure 2 · p. 3

D. Evolutionary Configuration Search

  • The compact evolutionary search selects depthwise-separable decoder configurations using source-subject validation and an accuracy–efficiency trade-off.The search varies filters, depth, kernel width, pooling, dropout, optimizer, learning rate, weight decay, batch size, and training budget; it yields 1,647 candidates overall.Source: D. Evolutionary Configuration Search · p. 3
  • The broader evolutionary search covers ten representative families, including compact EEG CNNs, temporal convolution, filter banks, attention/conformers, and larger convolutional baselines.Across nine folds and ten families, it runs 15,390 trainings; selected configurations are retrained with seeds 42–44, alongside a budget-matched random-search control.Source: D. Evolutionary Configuration Search · p. 3

E. Selector Diagnostics

Selector diagnostics compare validation-based family choice with a non-deployable test-best reference and examine how validation coverage affects selection and rank diagnostics.

E. Selector Diagnostics

  • Selector regret is defined as test-best minus validation-selected accuracy, with the test-best family serving only as a non-deployable reference.The diagnostics also compare validation selection with a fixed-family EEGNet control and uniform choice.Source: E. Selector Diagnostics · pp. 3–4
  • Subject-bootstrap confidence intervals are descriptive because the LOSO source-training sets overlap.Source: E. Selector Diagnostics · p. 4
  • Validation-coverage sensitivity compares one-source validation with a pooled three-source validation set, applying the same selector and rank diagnostics to both.The one-source arm uses the next cyclic source subject; the three-source arm pools trials from the next three cyclic source subjects.Source: E. Selector Diagnostics · p. 4

F. Fixed Baselines and Validation-Gated Ensembles

The fixed benchmark freezes configurations before nine-fold LOSO retraining, while the compact six-family ensemble is chosen by parameter screen alone. A validation gate selects whether to use an ensemble or the best single model before held-out logits are applied.

F. Fixed Baselines and Validation-Gated Ensembles

  • The compact six-family set is fixed by a median selected-size threshold of ≤25K parameters, without validation or test accuracy determining membership.The benchmark is a frozen post-discovery comparison; central target-isolated claims instead rely on fold-specific search and gating.Source: F. Fixed Baselines and… · p. 4
  • The gate compares the validation-best ensemble among three policies with the validation-best individual model, using a pre-specified margin of m = 0.The ensemble choices are raw-logit averaging, softmax averaging, and majority voting; validation ties follow a pre-specified candidate-policy order.Source: F. Fixed Baselines and… · p. 4
  • Only after the gate fixes the policy are held-out logits used to produce predictions and calculate held-out accuracy.The algorithm scores policies on validation logits and applies the chosen policy to held-out logits.Source: F. Fixed Baselines and… · p. 4, Figure 3 · p. 4

IV. EXPERIMENTAL SETUP

The experiments use fixed single-session LOSO protocols on BCI Competition IV-2a and an independent Lee2019 check. A frozen seven-entry decoder benchmark is trained without target adaptation or test-time augmentation.

A. Dataset and LOSO Protocol

  • BCI Competition IV-2a session T uses nine subjects, 22 EEG channels, four classes, and 288 trials per subject, with one subject held out per fold.EOG channels are used only for artifact scoring; cross-session generalization is outside scope.Source: A. Dataset and LOSO Protocol · p. 4

B. Models and Training

  • The fixed comparison uses seven decoder entries across 189 runs, with validation-accuracy early stopping and no target adaptation or test-time augmentation.The entries include the selected compact decoder, CNN, and Transformer, plus EEGNet, EEG-TCNet, Shallow ConvNet, and EEG-Inception.Source: B. Models and Training · p. 4

C. Search and Ensemble Evaluation

The evaluation combines compact and broader decoder searches, selected retraining, and validation-only ensemble-policy selection. Accuracy, model size, and inference latency are reported with reproducibility records linking aggregates to runs.

C. Search and Ensemble Evaluation

  • The broader search evaluates 15,390 trainings across 90 fold-family tasks, then retrains 270 selected configurations; matched random search uses the same spaces and budget.Compact search includes 1,647 candidates across 27 fold-seed runs; validation-coverage sensitivity retrains all ten selected families.Source: C. Search and Ensemble Evaluation · p. 5
  • Validation logits choose the gated policy before held-out logits are scored, and the test-best reference is used only to quantify selector regret.The validation-coverage sensitivity changes training-subject counts from seven to five, so paired results are descriptive rather than causal.Source: C. Search and Ensemble Evaluation · p. 5

D. Metrics and Reproducibility

  • Held-out-subject accuracy is the primary metric, with parameters, FP32 weight memory, and batch-1 Orin latency also reported.Latency uses a Jetson AGX Orin 64 GB in MAXN mode; ensemble timing includes serial member forwards and aggregation, but excludes acquisition and preprocessing.Source: D. Metrics and Reproducibility · p. 5
  • Seed-averaged subject scores are summarized with bootstrap intervals over nine BCI or 54 Lee2019 subjects, which are descriptive because LOSO training sets overlap.Stored splits, configurations, checkpoints, and CSVs trace aggregates to fold, seed, family, and checkpoint.Source: D. Metrics and Reproducibility · p. 5

A. Selection Reliability and Search Controls

Source-validation selection exceeds uniform choice but does not outperform fixed-family EEGNet, and an oracle test-best family remains substantially better. Compact models occupy the high-accuracy region, while search-control gains concern model size rather than held-out accuracy.

A. Selection Reliability and Search Controls

  • 40.01% source-validation selection exceeded uniform choice by 1.72 points but did not outperform fixed-family EEGNet at 40.29%; test-best reached 46.09%, leaving 6.08 points of selector regret.Within-fold rank transfer was weak (ρ = 0.256); the pooled correlation was ρ = 0.021.Source: A. Selection Reliability and Search… · p. 5
  • 40.10±14.06% for compact models exceeded the mid-size mean of 35.09±11.52% and large-model mean of 34.78±11.32% after retraining.The pre-specified ≤25K compact threshold changed selected accuracy by only −0.05 points (39.96% vs. 40.01%; [−0.54, 0.39]); compact entries averaged 2.83 points above noncompact entries ([1.24, 4.63]).Source: A. Selection Reliability and Search… · p. 5
  • 52.71% versus 51.46% source-validation accuracy favored evolution, with a 22.8% lower geometric-mean selected parameter count, but no detected held-out gain (−0.81 points).The parameter-count ratio was 0.77 ([0.60, 0.99]); the held-out difference interval was [−1.87, 0.15].Source: A. Selection Reliability and Search… · p. 5
  • Fig. 4 shows large fold-seed variation, compact models in the high-accuracy region, and several difficult held-out subjects across decoder families.Fig. 5 presents the candidate landscape, selected-family distributions, and seed-averaged subject comparisons between validation-selected and test-best family accuracy.Source: Figure 4 · p. 5, Figure 5 · p. 6
  • The accuracy sensitivity trend was descriptive: accuracy increased by +0.37 points (interval [−4.50, 4.54]), while training subjects simultaneously fell from seven to five.Source: A. Selection Reliability and Search… · p. 6

B. Frozen Post-Discovery Benchmark

The frozen benchmark’s selected compact decoder reached 44.35±15.70% accuracy with 4.9K parameters and 0.99 ms Orin latency. Validation-gated ensembling produced candidate-set-dependent mean gains, with additional inference cost.

C. Validation-Gated Ensemble

  • Validation-gated ensembling increased the observed mean by 3.81 points in the fixed benchmark and from 40.84% to 43.31% for compact-six.For compact-six, the subject-level seed-averaged gain was 2.47 points ([0.39, 4.89]); 5/9 subjects improved, two tied, and two worsened.Source: C. Validation-Gated Ensemble · p. 6

B. Frozen Post-Discovery Benchmark

  • 44.35±15.70% accuracy, 4.9K parameters, 19 KiB FP32 weights, and 0.99 ms Orin CUDA latency were achieved by the selected compact decoder.Its paired difference from reference EEGNet was +0.30 points (95% bootstrap CI [−1.07, 1.84]), so no accuracy advantage is claimed.Source: Table III · p. 6, B. Frozen Post-Discovery Benchmark · p. 6

C. Validation-Gated Ensemble

  • Compact-six gating selected an ensemble in 13/27 groups and a single model in 14/27, with measured ensemble branches requiring 9.41–9.49 ms.The full-ten gate selected a single in 13/27 groups and an ensemble in 14/27, yielding a 12.15-ms realized mean.Source: C. Validation-Gated Ensemble · p. 6, Table IV · p. 6

D. Discussion

The discussion finds that isolating held-out subjects does not make family selection reliable: validation beats uniform choice but not fixed-family EEGNet and leaves substantial regret. The conclusion presents validation-gated policies as candidate-set-dependent, higher-cost options and bounds the evidence to offline, single-session decoding.

D. Discussion

  • Target isolation does not ensure reliable family selection: validation beat uniform choice, not fixed-family EEGNet, and left substantial regret.The conclusion reports weak within-fold rank transfer and a 6.08-point gap from the test-best family.Source: D. Discussion · p. 6, §VI · p. 7
  • Restricting eligibility to compact models produced little observed mean change, while matched random search limits evolution’s supported role to compact mixed-space optimization.A fixed compact default may suffice relative to fixed-family EEGNet; the framework’s value here is auditing selection reliability and optimizing mixed accuracy–size spaces.Source: D. Discussion · p. 7
  • Validation-gated policies showed positive mean gains for some candidate sets, but gains varied by subject and incurred measured inference cost.The conclusion reports candidate-set-dependent gains at higher measured cost.Source: D. Discussion · p. 7, Figure 6 · p. 7, §VI · p. 7

VI. CONCLUSION

  • The evidence is offline and single-session and does not establish universal state-of-the-art, cross-session, online, or clinical effectiveness.The conclusion identifies subject-shift-aware validation and target adaptation as future priorities.Source: §VI · p. 7
Loading 2609.33967v2…