PaperlayerView exact revision on arXiv

Source-linked AI summary

RobustSeiz: An Open-Source Framework for Benchmarking the Robustness of EEG Seizure Detection Models

Mohammad Mohammadi, Alireza Zarei

arXiv:2609.04007v1cs.LGeess.SP

TL;DR

Clean-data accuracy alone does not capture EEG detectors’ stability under real-world acquisition variability, artifacts, and adversarial inputs. RobustSeiz standardizes public EEG corpora and applies controlled shifts with complementary detection, timing, and predictive-agreement measures. Its AWGN analysis shows that perturbation severity changes detection quality and uncertainty signals.

  • Problem

    Clean held-out EEG performance does not establish detector stability under clinical acquisition variability, artifacts, and adversarial inputs.

    Source: Discussion · p. 12

  • Method

    RobustSeiz evaluates subject-independent detectors on held-out standardized EEG data under controlled environment, noise, and adversarial shifts, profiling detection, onset timing, and Monte Carlo dropout agreement.

    Source: Datasets · p. 5, Evaluation setting · p. 5, Controlled perturbations · p. 7

  • Results

    As noise weakened, sensitivity and F1 recovered to 0.57 and approximately 0.62–0.63, while precision decreased to 0.69 and false positives increased to 16.7 per 24 h.

    Source: Representative robustness profile · p. 11

  • Takeaways & Limitations

    RobustSeiz provides a reproducible protocol for comparing detector robustness under realistic clinical stressors beyond clean-data accuracy.

    Source: Conclusion · pp. 12–13

    Large-scale multi-model, multi-site leaderboards remain future work, and Monte Carlo dropout profiling requires adapters that expose stochastic forward passes.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Despite strong performance on held-out electroencephalography (EEG) data, seizure detectors may fail under real-world acquisition variability, artifacts, and adversarial inputs. We introduce RobustSeiz, an open-source, model-agnostic framework that provides a standardized, reproducible protocol for stress-testing and comparing seizure detectors under controlled, clinically motivated distribution shifts before deployment. We standardize four public scalp-EEG corpora (CHB-MIT, TUSZ, Siena, and SeizeIT1) into BIDS-EEG trees and evaluate subject-independent detectors on held-out splits. Environment, noise, and adversarial transforms are swept over predefined hyperparameter grids. Each run reports sample- and event-level sensitivity, precision, F1, false positives per 24 h, Lead and Lag onset timing, and Monte Carlo dropout predictive agreement. RobustSeiz includes a Dockerized GPU pipeline, experiment registry, and full-evaluation and research-subset modes. We demonstrate the framework with a contemporary seizure detector on TUSZ across the complete implemented shift grid; an AWGN analysis illustrates how perturbation severity changes detection quality, onset timing, and predictive agreement. RobustSeiz provides a shared benchmarking standard for evaluating seizure-detector robustness under realistic clinical stressors, extending pre-deployment assessment beyond clean-data accuracy.

Lay Summary

Seizure detectors can face noisy, incomplete, or equipment-affected EEG recordings outside clean research data. RobustSeiz recreates these problems so researchers and clinicians can assess detector trustworthiness before patient care.

  • Seizure-detection software is increasingly used to monitor EEG recordings in hospitals and at home.Source: Lay Summary · p. 1
  • Real recordings may be noisy, incomplete, or affected by equipment problems, even when a model looks accurate on clean research data.Source: Lay Summary · p. 1
  • RobustSeiz deliberately recreates real-world problems to help assess whether a detector remains trustworthy before patient care.Source: Lay Summary · p. 1

Background and Significance

Clean held-out accuracy does not capture the acquisition variability and artifacts of clinical EEG, while adversarial perturbations pose an additional stress. RobustSeiz addresses limitations in prior evaluation approaches with end-to-end, event-level testing across deployment stressors.

  • Clean held-out evaluation does not establish reliability across clinical EEG’s varying noise, acquisition protocols, sites, and populations.Source: Background and Significance · p. 3
  • Clinical EEG variability and adversarial input changes can expose missed seizures, false alarms, or prediction errors that clean-data performance does not reveal.Source: Background and Significance · p. 3
  • Accuracy-only protocols such as SzCORE standardize clean-data comparisons but do not define shared deployment stressors or how to apply them with fixed annotations.Source: Background and Significance · p. 3
  • Prior latent-space diagnostics do not provide end-to-end event-level assessment across environment, noise, and adversarial stressors with onset timing and stratified uncertainty.Source: Background and Significance · p. 3
  • RobustSeiz applies domain-guided perturbations outside detectors and reports sample- and event-level accuracy, Lead and Lag timing, and Monte Carlo dropout agreement.Source: Background and Significance · p. 3
  • The demonstration presents a representative profile on TUSZ, with full numerical results across the perturbation grid in the Supplementary Appendix.Source: Background and Significance · p. 3

Objective

RobustSeiz is designed to provide a reproducible framework for assessing EEG seizure-detector stability under realistic acquisition, artifact, and adversarial conditions. It combines standardized datasets, controlled perturbations, complementary analyses, and reusable infrastructure.

  • The framework standardizes four public scalp-EEG corpora into BIDS-EEG and defines subject-independent evaluation on held-out splits.Source: Objective · p. 3
  • Its domain-guided taxonomy models real-world acquisition and operational variability with environment, noise, and adversarial perturbations.Source: Objective · pp. 3–4
  • Each perturbation’s control hyperparameters receive full sensitivity analysis rather than a single fixed-severity test.Source: Objective · p. 4
  • Detector profiles combine controlled-shift accuracy, Lead and Lag onset timing, and Monte Carlo dropout agreement in full-evaluation and research-subset modes.Source: Objective · p. 4
  • The release includes a reproducible Dockerized pipeline and experiment registry for stress-testing pluggable detectors under identical conditions.Source: Objective · p. 4

Materials and Methods

RobustSeiz standardizes public scalp-EEG data and applies controlled shifts to read-only evaluation recordings before running interchangeable subject-independent detectors. It reports detection quality, onset timing, and dropout-based agreement, with full and research-subset evaluation modes and a central experiment registry.

Study overview

  • RobustSeiz applies environment, noise, and adversarial shifts in memory to read-only EEG data, then runs pluggable detectors and logs matched-scope reports.The reports include accuracy, Lead and Lag onset timing, and Monte Carlo dropout agreement index ϕ.Source: Figure 1 · p. 4, Study overview · p. 4

Datasets

  • Four public scalp-EEG epilepsy corpora—CHB-MIT, TUSZ, Siena, and SeizeIT—are converted with community epilepsy2bids tooling into BIDS-EEG trees, with recordings stored as EDF files and events in tab-separated tables.The default layout uses 19 standard international 10-20 electrodes in a unipolar common-average montage; CHB-MIT remains bipolar, and missing electrodes in TUSZ-style recordings are zero-filled.Source: Datasets · pp. 4–5

Evaluation setting

  • Robustness is measured on held-out, subject-disjoint evaluation data as performance change from the unperturbed baseline under controlled shifts.The assumed detectors are not retrained, and reference annotations remain unchanged.Source: Evaluation setting · p. 6

Controlled perturbations

  • Domain-guided environment, noise, and adversarial transforms are swept across documented hyperparameter grids to produce robustness curves rather than a single robust/not-robust label.The shifts cover instrumentation and montage effects, acquisition noise and artifacts, and FGSM/PGD decision-boundary probes.Source: Controlled perturbations · pp. 6–7

Model interface

  • Interchangeable detector adapters expose sample-wise detection and agreement functions, while shifts remain outside detector packages.This keeps model swapping separate from the shared perturbation library.Source: Model interface · p. 8

Evaluation

  • Each evaluation profiles detection metrics, event-level Lead and Lag timing, and Monte Carlo dropout agreement, with full and research-subset modes sharing scoring rules.Detection metrics include sample- and event-based sensitivity, precision, F1, and false positives per 24 h; the research subset is a deterministic slice, while full mode covers the complete split.Source: Evaluation · pp. 8–9

Software

  • A Dockerized GPU service runs evaluation against read-only data and writes predictions and scores to an output tree, with experiment commands and documentation maintained in the repository.The framework also records runs in an experiment registry.Source: Software · p. 10, Figure 5 · p. 10

Results

RobustSeiz combines controlled-shift scoring, onset timing, and predictive agreement in a reproducible evaluation stack. A TUSZ demonstration shows that AWGN severity changes detection quality, timing, and agreement.

Results

  • The release provides clinically motivated environment, noise, and adversarial shifts, a model-agnostic inference contract, a Dockerized runner, and analysis stages for detection quality, Lead and Lag, and MCD agreement.A machine-readable registry stores parameters alongside metrics for auditability and robustness-curve reconstruction.Source: Results · p. 11
  • Full mode evaluates the complete split, whereas research-subset mode uses the first 10 lexicographically ordered subjects by default; both share shifts and scoring rules.Research-subset defaults include all sessions, T=5, and τ=0.8 for MCD.Source: Results · p. 11
  • The framework was demonstrated on TUSZ using one contemporary detector, with the full implemented shift grid evaluated in full mode and all shifts plus MCD in research-subset mode.Complete numerical outcomes across the perturbation grid are in Supplementary Appendix B.Source: Results · p. 11

Representative robustness profile

  • As noise weakened, sensitivity and F1 recovered to 0.57 and approximately 0.62–0.63, while precision decreased to 0.69 and false positives increased to 16.7 per 24 h.Event-level true positives increased from 121 to 195 as noise weakened.Source: Representative robustness profile · p. 11
  • Mean Lead decreased from approximately 157 s to 109 s, while mean Lag increased from approximately 27 s to 30 s as AWGN weakened.The figure reports timing among event-level true positives and the onset true-positive count.Source: Representative robustness profile · p. 11, Figure 6 · p. 11
  • Mean agreement index ϕ rose from 0.0063 at 0 dB to 0.0356 at 50 dB and remained 0.0330 at 100 dB as noise weakened.False-positive samples stayed above 0.5 (ϕ = 0.668–0.855), while false-negative samples remained near 0 (ϕ = 0.034–0.048).Source: Representative robustness profile · pp. 11–12, Figure 6 · p. 11

Discussion

The discussion frames robustness profiling as a pre-deployment complement to clean-data accuracy, using controlled stressors and multiple outcome measures. The authors also identify limits in the current framework’s scope and supported models.

Discussion

  • Clean evaluation scores do not certify stability under clinical acquisition and artifact conditions, where missed seizures and excess alarms are consequential.The framework treats robustness profiling as a pre-deployment requirement and distinguishes environment and noise shifts from adversarial probes of missed- and over-detection regimes.Source: Discussion · p. 12
  • Sweeping domain-guided shifts across control settings produces robustness curves that show where performance remains stable or collapses as stress intensifies.The design keeps shifts outside detector packages, and Lead/Lag and MCD add onset-timing and predictive-agreement readings to point predictions.Source: Discussion · p. 12
  • The authors recommend reporting stressor-specific changes in event F1, FP/24h, missed events, and alarms alongside clean accuracy for deployment decisions.Supplementary materials provide a Model Card, Reproducibility Checklist, and experiment-submission instructions for cross-group profile comparison.Source: Discussion · p. 12

Limitations

  • Large-scale multi-model, multi-site leaderboards remain future work; adversarial probes are adapter-white-box and use normalized-window ϵ rather than a calibrated microvolt scale.MCD requires stochastic-forward-pass adapters, some channel-dropout shifts stress montage requirements, and training orchestration is out of scope.Source: Limitations · p. 12

Conclusion

RobustSeiz is an open-source framework for benchmarking EEG seizure-detector robustness under realistic clinical conditions. It standardizes public corpora, applies controlled shifts, and profiles detection, timing, and predictive agreement to support reproducible comparisons.

Conclusion

  • RobustSeiz is an open-source framework for benchmarking the robustness of EEG seizure-detection models.Source: Conclusion · p. 12
  • The framework standardizes public corpora and applies clinically motivated environment, noise, and adversarial shifts with sensitivity analysis across perturbation hyperparameters.Source: Conclusion · pp. 12–13
  • Detector profiles combine accuracy metrics, Lead and Lag onset timing, and Monte Carlo dropout agreement, with software and a registry protocol released for reproducible, comparable studies.Source: Conclusion · p. 13

Funding

The research received no funding or sponsorship.

Funding

  • No funding or sponsorship of any kind was used in this research.Source: Funding · p. 13

A Data format

RobustSeiz standardizes EEG files, annotations, and inference outputs around shared BIDS-EEG paths, then demonstrates the pipeline on TUSZ with a frozen detector under matched shift protocols. The appendix is an end-to-end usage check, not a multi-model leaderboard.

A Data format

  • Shared BIDS-EEG paths and SCORE/HED-SCORE annotations let reference labels and detector hypotheses use the same subject–session–run layout.This keeps scoring independent of model-specific export formats.Source: A Data format · p. 17

A.1 BIDS-EEG compliant dataset

  • The evaluation tree pairs each recording with sidecar metadata and reference events; detector hypotheses mirror those paths, while inference leaves the source tree read-only.Perturbations are applied in memory.Source: §A.1 · p. 17, Figure 7 · p. 17

A.2 Annotation format

  • Annotation tables use one event per row, with onset and duration in seconds and event codes shared by references and detector outputs.Detector outputs may also include confidence, and channel fields identify affected electrodes or all channels.Source: §A.2 · p. 17, Figure 8 · p. 18
  • Seizure annotations use hierarchical ILAE 2017 codes; prolonged events retain the same onset–duration schema, and shifts never modify reference annotations.The top-level code is sz, with optional onset tags such as sz-foc and sz-gen.Source: §A.2 · p. 18

A.3 Recording content and montage

  • Signals use 256 Hz EDF and a fixed-order unipolar common-average montage of 19 international 10-20 electrodes, except CHB-MIT’s original bipolar layout.Missing electrodes in TUSZ-style recordings are zero-filled; auxiliary channels do not enter the common average.Source: §A.3 · p. 18

A.4 Hypothesis outputs

  • Hypothesis event files mirror reference paths, and optional per-sample Monte Carlo dropout agreement arrays are saved alongside them.Scoring compares detector events with untouched reference annotations using the same sample- and event-level rules in both modes.Source: §A.4 · p. 19

B RobustSeiz Benchmark

  • The appendix demonstrates pipeline use on TUSZ, the release’s largest and most widely used public scalp-EEG corpus, rather than claiming a multi-model leaderboard.Its numerical outcomes illustrate robustness-profile structure, not state-of-the-art robustness.Source: B RobustSeiz Benchmark · p. 19

B.1 Algorithm

  • SeizureTransformer’s frozen pretrained weights are stress-tested on TUSZ using full-mode and research-subset protocols across the same shift grid.Full mode covers the complete evaluation split; the research subset uses the first ten lexicographic subjects and adds Monte Carlo dropout.Source: §B.1 · pp. 19–20

B.2 Results

The TUSZ AWGN sweep shows that stronger noise suppresses seizure detections, while cleaner signals recover detection alongside more false alarms; onset timing and dropout agreement also vary. RobustSeiz pairs these profiles with standardized reporting and reproducibility guidance.

B.2 Results

  • At 0 dB, event sensitivity is 0.36, event F1 is 0.52, and false positives are 1.5 per 24 h; event F1 reaches 0.63 between 5 and 20 dB as false positives rise to 14.7 per 24 h.The strongest-noise setting detects 121 event-level true positives; at 50–100 dB, sensitivity is 0.57, event F1 is 0.62, and false positives are 16.7 per 24 h.Source: Figure 10 · p. 21, §B.2 · p. 21
  • At 0 dB, mean Lead is approximately 157.0 s and mean Lag is 27.3 s; at 50–100 dB, Lead is 108.6 s and Lag is approximately 30.4 s.The number of onset true positives expands from 121 to 195, so the Lead shift reflects a changing detected-event population, not necessarily later detection of the same seizures.Source: Figure 10 · p. 21, §B.2 · p. 21
  • Mean agreement ϕ rises from 0.0063 at 0 dB to 0.0356 at 50 dB, while false-positive samples have ϕ values of 0.668–0.855 and false-negative samples remain at 0.0335–0.0475.The false-positive trajectory is non-monotonic; false alarms generally receive seizure-dominant votes, while missed seizures receive non-seizure-dominant votes.Source: §B.2 · pp. 21–22

C.1 Model Card

  • The Model Card records detector and evaluation details, clean held-out baselines, and category-level average ranges for robustness metrics across shift sweeps.It covers event F1, false positives per 24 h, accuracy, sensitivity, Lead, Lag, onset counts, and mean agreement ϕ.Source: C Reports · p. 22, §C.1 · p. 22

C.2 RobustSeiz Reproducibility Checklist

  • The reproducibility checklist calls for documenting models, datasets, code, full shift grids, matched scoring scope, Lead and Lag, dropout stratification, and the experiment registry.It is a reporting aid, not a substitute for releasing code and data.Source: §C.2 · p. 22

C.3 Robustness reports

RobustSeiz uses experiments_results.csv as the machine-readable comparative record for reported robustness experiments, with a defined set of run, score, timing, and optional dropout fields.

C.3 Robustness reports

  • Each experiment appends one row to experiments_results.csv, the comparative artifact to submit with the paper or as supplementary material.Per-recording hypothesis trees support audit, but the registry is the source of demonstration tables and should not be hand-edited.Source: §C.3 · p. 23
  • The Model Card presents corpus baselines and category-level average ranges, while the reproducibility checklist covers required reporting items for models, datasets, code, and results.The checklist emphasizes controlled-shift details such as matched scoring scope, Lead and Lag, Monte Carlo dropout, and registry submission.Source: Figure 11 · p. 26, Figure 12 · p. 27
  • Registry rows identify runs and shifts, store hyperparameters, report sample- and event-based scores, summarize onset timing and counts, and include dropout fields only when enabled.A complete registry covers every experiment mentioned in the paper.Source: §C.3 · p. 23, Figure 13 · p. 28
Loading 2609.04007v1…