PaperlayerView exact revision on arXiv

Source-linked AI summary

MOABB: Trustworthy algorithm benchmarking for BCIs

Vinay Jayaram, Alexandre Barachant

arXiv:1805.06427v1cs.HC

TL;DR

BCI algorithm development lacks large, reproducible evidence because datasets, recording conditions, and implementations vary. MOABB standardizes public EEG data and pipeline interfaces, then benchmarks algorithms across datasets with automated statistical analysis. The results show strong dataset-dependent performance variation and that several established methods do not reliably outperform simpler alternatives across datasets.

  • Problem

    BCI validation is limited by small, heterogeneous datasets and scarce reproducible algorithm implementations, making broad comparison difficult.

  • Method

    MOABB aggregates public EEG datasets into a common format, standardizes preprocessing and pipeline interfaces, and automates cross-dataset benchmarking and statistical analysis.

  • Results

    Different datasets produce very different scores for identical pipelines, while established methods do not consistently outperform simpler alternatives across datasets.

  • Takeaways & Limitations

    Reliable BCI algorithm assessment requires evaluating pipelines across multiple datasets because recording and human-paradigm characteristics strongly shape classification outcomes.

Abstract

from arXiv · show

BCI algorithm development has long been hampered by two major issues: small sample sets and a lack of reproducibility. We offer a solution to both of these problems via a software suite that streamlines both the issues of finding and preprocessing data in a reliable manner, as well as that of using a consistent interface for machine learning methods. By building on recent advances in software for signal analysis implemented in the MNE toolkit, and the unified framework for machine learning offered by the scikit-learn project, we offer a system that can improve BCI algorithm development. This system is fully open-source under the BSD licence and available at https://github.com/NeuroTechX/moabb. To validate our efforts, we analyze a set of state-of-the-art decoding algorithms across 12 open access datasets, with over 250 subjects. Our analysis confirms that different datasets can result in very different results for identical processing pipelines, highlighting the need for trustworthy algorithm benchmarking in the field of BCIs, and further that many previously validated methods do not hold up when applied across different datasets, which has wide-reaching implications for practical BCIs.

I. INTRODUCTION

BCI validation is difficult because datasets, recording conditions, subjects, and implementations vary, while reproducible code and broad benchmarks remain scarce. MOABB addresses these problems by aggregating standardized public EEG datasets and state-of-the-art algorithms for automated, reproducible benchmarking.

  • BCI validation is difficult because hardware, paradigms, subjects, channels, tasks, physical setups, file formats, and documentation vary across datasets.
  • Private newly recorded datasets obscure whether offline results reflect coding issues or recording artifacts and limit generalization across hardware, paradigms, and subjects.
  • Heavy reliance on a small number of public datasets provides an attractive but narrow basis for validating BCI algorithms.
  • Limited algorithm code forces laboratories to reimplement competing methods, so comparisons often use same-lab methods or older standards such as CSP.
  • MOABB aggregates public EEG datasets into a common format, bundles state-of-the-art algorithms, and automates benchmarking and statistical analysis through an open-source Python package.
  • The initial validation uses binary imagined motor imagery across many datasets, while the framework also supports other channel types, multiclass paradigms, and transfer learning.

II. METHODS

MOABB unifies access to public BCI datasets and provides an extensible route for incorporating additional open-source data. Its software relies on MNE preprocessing and interface-conformance tests for candidate code.

  • MOABB unifies access to many public BCI datasets spanning varied user paradigms and recording conditions.
  • New open-source datasets can be added through the MNE toolkit when they are compatible with its framework.
  • Test functions check whether candidate code conforms to the project's software interface.

B. Context

MOABB defines context as the preprocessing and validation characteristics that transform continuous EEG into trials and evaluated pipeline performance. It separates trial construction from train-test splitting and performance reporting.

  • Context specifies preprocessing and validation procedures from continuous EEG time-series to pipeline performance for each subject or session.
  • 1) Paradigm: The paradigm converts continuous data into trials while controlling imagery or ERP type, preprocessing, trial length, and overlap.
  • 1) Paradigm: Paradigm settings include ICA cleaning, bandpass filtering, and decisions about the time window around event markers.

2) Evaluation:

MOABB fixes evaluation choices for training, testing, and performance reporting, while pipelines transform trial data into labels under a common scikit-learn interface. It then generates per-subject or per-session scores for visualization and statistical testing.

  • 2) Evaluation: Evaluation fixes train-test splits identically within an analysis and selects performance metrics appropriate to the prediction task.
  • A pipeline maps raw trial-wise data to labels, including spatial filtering and classifier fitting, using the scikit-learn interface.
  • MOABB trains each pipeline independently for every subject or recording session and produces scores such as accuracy or ROC-AUC for visualization and statistical testing.

III. STATISTICAL ANALYSIS

MOABB aggregates pipeline scores across datasets while accounting for cross-dataset differences and subject-level heterogeneity. It combines within-dataset paired tests to estimate whether one pipeline outperforms another.

  • Pipeline scores are not naively comparable across datasets because trial count, sampling rate, location, and hardware differ.
  • The procedure produces scores for every subject, dataset, and pipeline before synthesizing them into estimates of relative pipeline performance.
  • BCI-illiterate subjects can prevent any pipeline from reliably outperforming another, producing potentially bimodal score-difference distributions.
  • For each pipeline pair, MOABB uses a one-tailed paired permutation t-test below 20 subjects or a Wilcoxon signed-rank test within each dataset.
  • The resulting dataset-level p-values are combined using Stouffer’s method to support cross-dataset pipeline comparisons.

IV. EXPERIMENT

The experiment demonstrates MOABB through a large, side-by-side analysis of established motor-imagery BCI pipelines. It uses binary motor-imagery classification and within-session cross-validation as the evaluation setting.

  • IV. EXPERIMENT: The study conducts a first big-data, side-by-side analysis of state-of-the-art motor-imagery BCI pipelines drawn from many papers.
  • IV. EXPERIMENT: The initial validation focuses on binary imagined motor-imagery classification, the most widely used motor-imagery paradigm.
  • IV. EXPERIMENT: Within-session cross-validation is selected because it represents a best-case pipeline evaluation with minimal non-stationarity.

1) Paradigm:

The paradigm evaluation compares frequency-processing choices and scores binary classification with ROC-AUC under within-session 5-fold cross-validation. Session scores are averaged when datasets contain multiple sessions.

  • 1) Paradigm:: Two preprocessing pipelines are tested: one 8–35Hz bandpass and another using 8–35Hz bands in 4Hz increments.
  • 1) Paradigm:: All data are subsampled to 128Hz because otherwise memory requirements become prohibitive.
  • 1) Paradigm:: ROC-AUC scores 5-fold within-session cross-validation, with identical subject-level splits across pipelines.
  • 1) Paradigm:: ROC-AUC is used instead of classification accuracy because it is less sensitive to imbalanced classes across datasets.
  • 1) Paradigm:: Scores from multiple sessions are averaged to produce one score per subject.

B. Pipelines

MOABB implements literature-based BCI pipelines alongside two established reference pipelines, with hyperparameters selected by cross-validation.

  • B. Pipelines: The implemented pipelines include methods from the BCI literature, CSP + LDA, and channel-level variances + SVM.All hyperparameters are set via cross-validation.

V. RESULTS

Across datasets, BCI pipeline rankings are inconsistent: dataset effects are large, and several methods reverse their apparent advantage depending on the dataset.

  • Different datasets produce very different average scores independent of pipeline, with subject preparation contributing to the contrast.Pre-trained subjects in Zhou et al. differ substantially from the naive Physionet sample.
  • CSP generally outperforms channel log-variance features, but datasets and show the opposite trend.
  • Tangent space projection usually outperforms CSP, yet this relationship reverses in half of the sampled datasets.
  • Small-subject studies can yield untrustworthy confidence intervals even for strong standardized effects.The paper specifically identifies dataset as an example with very few subjects.
  • CSP variant performance depends heavily on dataset, with no clear overall trend.
  • Filter-bank CSP outperforms regular CSP on BNCI 2014 datasets, whereas regular CSP performs better on Physionet; covariance regularization does not improve CSP.The analysis also fails to validate a performance increase from Tikhonov weighting.
  • Across meta-effects, all other algorithms outperform log-variance features on average, while tangent space projection is best among CSP and its variants.These averages still show substantial variance across datasets.

VI. DISCUSSION

MOABB enables reproducible, cross-dataset BCI pipeline comparisons and shows that algorithm performance varies substantially across datasets. The analysis also finds that several established methods do not consistently outperform simpler alternatives.

  • MOABB system: The system compares BCI pipelines across datasets using automated statistical procedures and a common submission interface.It is designed to be extensible with new datasets and pipelines.
  • Algorithm comparisons: Channel log-variance features almost never outperform CSP or Riemannian methods.The meta-analysis reports substantial variance across datasets despite this overall pattern.
  • Algorithm comparisons: Tangent space classification is the best tested model for single-session classification.The paper presents this as one of its clearest findings.
  • Scope and caveats: The single-session cross-validation setting limits non-stationarity and makes regularization least useful in this analysis.The authors caution that regularization should not therefore be dismissed for cross-session classification.
  • Algorithm comparisons: CSP variants do not show a clear overall advantage over CSP because their relative performance depends heavily on the dataset.This dataset dependence is reported for DLCSPauto, TRCSP, and filter-bank CSP.

VII. CONCLUSION

The paper uses MOABB to aggregate datasets and test BCI algorithms across varied studies. Its analysis shows that the available benchmark sample remains only a fraction of the field’s accumulated data, motivating broader aggregation.

  • VII. CONCLUSION: MOABB gathers datasets and provides a system for testing algorithms across studies.The authors intend the platform to help address the BCI sample-size problem.
  • VII. CONCLUSION: The analysis included over 250 subjects but still represented only a fraction of the subjects recorded in BCI publications.The authors state that more varied setups could increase the system’s power.
Loading 1805.06427v1…