OOD Detection for EEG-based Machine Learning in High-Risk Environments

Philipp Bomatter; Henry Gouk

Source-linked AI summary

OOD Detection for EEG-based Machine Learning in High-Risk Environments

Philipp Bomatter, Henry Gouk

arXiv:2608.17620v1Aug 18, 2026cs.LG

TL;DR

EEG OOD detection is under-explored despite the risk of overconfident failures under distribution shifts and the limited downstream evaluation in prior work. The paper benchmarks controlled perturbations and broad discriminative and generative methods, linking detection to clinical prediction tasks. Generative methods detect unfamiliar EEG more effectively, while discriminative methods capture model uncertainty; combining both provides a more robust safety net.

Problem
EEG OOD detection is under-explored, and prior evaluations often omit its practical impact on downstream applications despite distribution-shift risks in high-risk settings.
Method
The study benchmarks discriminative and generative OOD methods on controlled, paired EEG perturbations and evaluates their effects in two clinical downstream prediction tasks.
Results
Generative methods substantially outperform discriminative methods for EEG OOD detection, while both paradigms predict downstream performance for different reasons.
Takeaways & Limitations
Combining generative OOD detection with discriminative uncertainty estimation can protect against unfamiliar data and difficult in-distribution cases.
Takeaways & Limitations
The perturbation benchmark is not exhaustive, motivating additional curated datasets covering a wider variety of distribution shifts.

Abstract

from arXiv · show

Machine learning models for electroencephalography (EEG) analysis show great promise across a wide range of applications, but their deployment in high-risk domains is hindered by their vulnerability to distribution shifts. Encountering out-of-distribution (OOD) data can lead to catastrophic, overconfident predictive failures. While OOD detection methods can mitigate these risks, they remain heavily under-explored for EEG. Moreover, evaluations in the broader literature typically evaluate OOD detection performance in isolation, ignoring their practical impact on downstream applications. To bridge this gap, we introduce a benchmark for EEG OOD detection, evaluate a broad range of methods, and furthermore evaluate their value in two clinical downstream prediction task. Our results disentangle OOD detection and model uncertainty estimation capabilities, which are frequently conflated in the literature, provide actionable insights about the current state of the art for EEG OOD detection and model uncertainty estimation, and demonstrate how complementary methods for both aspects can be combined to form a robust safety net for the deployment of EEG-based machine learning models in real-world applications.

1 Introduction

EEG machine-learning deployment in high-risk settings is limited by catastrophic, overconfident failures under distribution shifts, while EEG OOD evaluation remains under-explored. This work benchmarks discriminative and generative approaches, links OOD scores to clinical prediction, and separates OOD detection from model uncertainty.

Motivation: Distribution shifts from hardware, preprocessing, or patient-population differences can cause catastrophic, high-confidence failures on EEG data.Unknown shifts may remain even when specific quality controls exist.
Motivation: OOD detection can reduce risk by enabling abstention or deferral to a human expert when data is likely out of distribution.Methods broadly follow discriminative or generative paradigms.
Contribution: The study introduces a systematic EEG OOD evaluation linking detection performance to downstream clinical prediction tasks.It benchmarks controlled perturbations and methods based on both discriminative and generative models.
Findings: Discriminative methods were largely ineffective for OOD detection, whereas generative methods performed better as perturbation severity increased.In downstream evaluations, both paradigms predicted model performance for different reasons.
Implications: The benchmark distinguishes true OOD detection from model uncertainty and shows that their complementary signals can support safer deployment.Generative methods recognise unfamiliar data, while discriminative methods can proxy model uncertainty.

2 Related Work

EEG OOD research is limited relative to computer vision, and existing EEG and time-series benchmarks primarily test discriminative methods and semantic shifts. The paper addresses this narrower evaluation landscape with a broader clinical EEG setting and controlled perturbations.

EEG OOD Detection: Prior EEG OOD studies have mainly evaluated discriminative methods in motor-imagery or brain-computer-interface settings.This study considers a more general clinical context than the cited BCI work.
OOD Benchmarks: Computer-vision OOD benchmarking is more advanced, with OpenOOD providing notable evaluations of detection methods.The paper contrasts this mature literature with EEG and time-series research.
Positioning: The proposed evaluation extends prior EEG work by combining generative and discriminative methods with controlled perturbations and downstream clinical assessment.Its evaluation concept trains both model types and measures detection alongside prediction impact.
OOD Benchmarks: Time-series benchmarks relevant to EEG remain limited to discriminative approaches and report relatively poor overall performance.The cited benchmarks also rely heavily on semantic shifts such as held-out classes or disjoint datasets.

3 Methods

The benchmark creates paired EEG OOD data through controlled perturbations, evaluates discriminative and generative detection methods, and measures both ID–OOD discrimination and downstream prediction effects. Perturbations model plausible shifts with controllable severity, while methods use TCN or flow-matching UNet models.

Evaluation Framework: The evaluation trains an unsupervised generative model and a supervised discriminative model from training data, then perturbs test samples to create paired OOD examples.The resulting data supports evaluation against unperturbed in-distribution samples.
Evaluation Framework: OOD methods are assessed both by separating paired ID and OOD samples and by relating their scores to downstream discriminative-model accuracy.This connects detection performance with clinical prediction impact.
OOD Perturbation Suite: The perturbation suite simulates plausible EEG shifts while providing reproducible transformations, severity control, and exact pairing with ID data.Re-referencing is the exception because its alternatives lack a clear severity ordering.
OOD Perturbation Suite: Perturbations include sampling-rate changes that smooth signals and channel-order permutations that create implausible spatial topographies.These shifts reflect practical errors involving sampling settings, electrode connections, or channel mappings.
OOD Perturbation Suite: Additional perturbations alter reference schemes or filter cutoffs to remove progressively more low- or high-frequency activity.High-pass cutoffs increase from 2 Hz to 8 Hz, while low-pass cutoffs decrease from 45 Hz to 15 Hz.
Detection Methods: Discriminative methods use a TCN and include MSP, Energy, ODIN, and ASH, which derive scores from outputs, logits, inputs, or shaped activations.MSP uses maximum softmax probability, while ODIN adds temperature scaling and gradient-based input perturbations.
Detection Methods: Generative methods use a flow-matching UNet with Euler integration, including likelihood-based scoring and typicality.Typicality measures the absolute difference between negative log-likelihood and an entropy estimate from validation data.

4 Results

Generative methods substantially outperformed discriminative methods for EEG OOD detection, while downstream evaluations showed that OOD detection and model uncertainty capture different aspects of predictive reliability. Combining SITN-based OOD detection with MSP-based uncertainty estimation produced a more robust selective classification system.

4.1 OOD Detection Performance: Discriminative methods such as MSP, Energy, ODIN, and ASH performed largely at chance for OOD detection, whereas generative methods achieved considerably better performance.Generative methods also improved consistently as perturbation severity increased.
4.1 OOD Detection Performance: SITN consistently achieved the strongest OOD detection, including at low perturbation severity and conditions where other methods were near chance.On CAUEEG, DoSE narrowly exceeded SITN for re-referencing and the highest-severity low-pass-filter condition.
4.2 Downstream Performance Impact: In downstream TUAB evaluation, SITN separated ID and OOD samples effectively, while MSP predicted performance among ID samples but failed at the highest confidence levels.For ID-only data, SITN provided little performance information across most samples, with a drop for the most extreme 1% of SITN scores.
4.2 Downstream Performance Impact: The results distinguish OOD detection from model uncertainty: SITN identifies samples outside the training distribution, whereas MSP reflects uncertainty among in-distribution samples.These signals therefore address different sources of unreliable predictions.
4.3 Combination of OOD Detection and Model Uncertainty: At 100% coverage, accuracy fell from 80.5% on ID data to 67.5% after including resampling-perturbed OOD samples, while combining SITN and MSP achieved the best accuracy across both conditions.SITN protected against the OOD-related performance drop, whereas MSP mainly improved ID-only performance.

5 Discussion and Conclusions

Generative methods substantially outperform standard discriminative methods for EEG OOD detection, while downstream evaluations reveal complementary roles for detection and uncertainty estimation. Combining both signals can protect against distinct failure modes, although the benchmark does not cover the full range of real-world EEG shifts.

Generative approaches substantially outperform standard discriminative methods for realistic EEG covariate shifts, with SITN establishing a new state of the art.
Performance drops substantially on OOD samples in two clinical prediction tasks, underscoring the practical value of identifying distribution shifts.
Generative methods flag OOD samples, whereas discriminative methods identify high-uncertainty samples even when they remain in distribution.
Combining discriminative and generative signals can protect against both OOD-driven failure and high model uncertainty.
The perturbation-based benchmark provides controlled ground truth but does not represent the wider variety of distribution shifts found in EEG, especially patient-population differences.
The authors argue that applied OOD benchmarks should prioritize practical utility and unanticipated shifts rather than relying mainly on semantic shifts.

Appendix A OOD Detection Results on CAUEEG

On CAUEEG, standard discriminative scores operate largely at chance, while generative methods achieve better AUROC as perturbation severity increases. Mahalanobis distance improves over other discriminative methods for several perturbations but fails severely for high-pass filtering.

Generative-model-based methods achieve considerably better AUROC than MSP, Energy, ODIN, and ASH, with performance increasing at higher severity levels.
The Mahalanobis approach models density in the latent embedding space rather than deriving scores from network predictions.
Its implementation estimates class-conditional feature means and a shared covariance matrix from penultimate-layer representations, with diagonal regularization of ϵ = 10−5.
Mahalanobis distance yields superior OOD detection under Resampling, Channel Shuffle, and Re-referencing perturbations relative to other discriminative methods.
Mahalanobis distance performs worse than random chance under the High-pass Filter perturbation.

Appendix C Sample Size Dependence

OOD detection generally improves with more training data, and expanding participant diversity is more beneficial than adding the same number of segments from fewer participants. Generative methods, particularly SITN, remain strong even with limited data.

The scaling analysis starts from 25 participants and 5 segments per participant, while validation and test data remain fixed.
OOD detection performance generally increases as the amount of training data grows.
Increasing the number of participants is more impactful than adding an equal number of segments from a smaller participant cohort.
Generative methods, particularly SITN, perform well with very limited data and dominate discriminative methods across evaluated sample sizes.
Log-likelihood performance can decrease with more training data, a failure mode discussed in Appendix D.

Appendix D Limitations of Likelihoods for OOD Detection

Likelihoods are unreliable OOD signals for EEG because perturbed samples can receive higher likelihoods than in-distribution samples. High likelihood is associated with flat, simple EEG segments, while high-amplitude artefacts receive low likelihoods.

The observed EEG behaviour replicates likelihood limitations previously reported for computer-vision datasets.
OOD EEG samples consistently receive higher log-likelihoods than in-distribution samples across resampling perturbation severities.
The highest likelihoods are assigned to completely flat EEG segments, whereas high-amplitude artefacts receive particularly low likelihoods.
Figure D2 compares ID samples at 100 Hz with resampling-generated OOD samples at 125, 150, and 200 Hz, including the segments with lowest and highest likelihoods.

Appendix E Training Details

The generative and discriminative models used distinct architectures and objectives, with training stabilized through shared optimization settings and validation-checkpoint restoration.

Model training: The generative model used a UNet architecture with a flow-matching objective interpolating Gaussian noise and EEG data.Training minimized mean-squared error between the network output and the target vector field X − Z.
Model training: The discriminative model used a Temporal Convolutional Network trained with supervised cross-entropy loss for the prediction task.
Optimization: Both models used AdamW, gradient-norm clipping, batch size 128, and checkpoints selected by lowest validation loss.The generative and discriminative models were trained for 1,000,000 and 50,000 gradient steps, respectively.

Appendix F Exploratory Cross-dataset Experiments

Cross-dataset analyses were exploratory because the OOD status of one dataset relative to a model trained on the other is uncertain. Scores showed moderate or inconsistent dataset and subgroup discrimination, with no clear age trends and only a modest SITN pattern for abnormal EEGs.

Scope: Cross-dataset OOD interpretation was exploratory because it is unclear whether samples from one dataset should be considered OOD for models trained on the other.Potential hardware differences and overlapping participant characteristics further complicate this interpretation.
Dataset discrimination: DoSE and SITN showed AUROC below 0.55 for TUAB-trained models but 0.703 and 0.740, respectively, for CAUEEG-trained models.These values indicate asymmetric discrimination between the two datasets across training directions.
Score processing: Participant-level OOD scores were computed by averaging sample scores and minmax-scaled to [0, 1], with inversion so higher values always meant more OOD.
Subgroup patterns: Diagnosis-group scores on CAUEEG were strongly overlapping and inconsistent across methods, while age analyses showed no clear trends on either dataset.SITN assigned somewhat higher OOD scores to abnormal EEG recordings on TUAB, despite substantial overlap.

Appendix G Cross-participant BCI Experiment

In the cross-participant motor-imagery experiment, SITN clearly separated the training participant from unseen participants, while performance often declined substantially for unseen participants and some methods failed to identify a severe failure case.

Experimental setup: The experiment used nine BNCI 2014-001 participants, with left- and right-hand trials from two sessions and six runs per session.The setup retained 144 trials per session and participant.
OOD separation: SITN was the only method to clearly separate the training participant as in-distribution from all unseen participants as out-of-distribution.OOD scores and prediction accuracy were aggregated at the participant level by averaging.
Downstream performance: Prediction accuracy dropped considerably below within-participant test performance for six of eight unseen participants.
Failure case: MSP gave the third-lowest OOD score to a participant whose performance fell below chance, despite all generative methods, Energy, and ASH flagging that participant as highly OOD.

Appendix H Full Downstream Performance Results

The appendix reports full downstream evaluations for normality prediction and dementia diagnosis across perturbations and OOD methods. Results are organized by prediction accuracy within OOD-score percentile bins, with figures covering resampling, channel shuffle, re-referencing, and filtering perturbations.

Evaluation scope: Downstream evaluations covered TUAB normality prediction and CAUEEG dementia diagnosis across all perturbations and OOD detection methods.Normality prediction was binary with a 50% chance level, while dementia diagnosis was three-way with an approximately 33% chance level.
Evaluation format: Performance was reported as accuracy across percentile bins of OOD scores, with each x-axis tick marking the corresponding bin’s upper edge.
Normality prediction: Figures H8–H12 report TUAB normality-prediction results for resampling, channel shuffle, re-referencing, high-pass filtering, and low-pass filtering perturbations.The re-referencing figure notes irregular upper-percentile bin sizes caused by metric saturation and tied maximum scores.
Dementia diagnosis: Figures H13–H17 report CAUEEG dementia-diagnosis results for the same five perturbation types.The re-referencing figure again attributes irregular upper-percentile bin sizes to metric saturation and tied maximum scores.