Source-linked AI summary
NeuroDETR: Multiscale Discrete Spatiotemporal Tokenization for Asynchronous EEG Event Detection
Yaozhi Wen, Kun Zhao, Wei Wei, Haoyang Chen, Kangning Wang, Yi Sun, Shaowei Gu, Dingkun Liu, Shuang Qiu, Huiguang He
TL;DR
Asynchronous EEG detection must identify event classes and temporal boundaries in continuous recordings, where event-related patterns are embedded in background activity. NeuroDETR combines event-aware refinement with multiscale temporal and hierarchical spatial tokenization; across eight EEG datasets, it achieves the highest event-level F1 among compared methods, improving the average over DANCE by 4.29 percentage points.
Problem
Continuous EEG event detection must identify event classes and localize onset and offset without predefined boundaries, while preserving event cues embedded in background activity.
Source: §1 · pp. 1–2Method
NeuroDETR uses FOCAL to guide temporal refinement and spatial pooling, then combines multiscale temporal and hierarchical spatial vector-quantized representations for event-set prediction.
Source: §1 · p. 2Results
Across eight EEG datasets, NeuroDETR achieves the highest event-level F1 among compared methods, improving the average over DANCE by 4.29 percentage points.
Source: §4.4 · p. 7, §5 · p. 9Takeaways & Limitations
The results demonstrate the value of organizing event-relevant information across temporal scales and spatial regions for joint event classification and localization in continuous EEG.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Asynchronous EEG event detection supports practical brain-computer interfaces by identifying event classes and temporal boundaries directly from continuous recordings without predefined event boundaries at inference time. The primary challenge lies in extracting event-related patterns embedded in ongoing background activity. To address this, we propose \textbf{NeuroDETR}, a discrete spatiotemporal multiscale framework that combines structured tokenization with event-aware feature refinement. Multiscale temporal codebooks capture brief responses and sustained dynamics, while Spatial-Pyramid VQ hierarchically encodes spatial structure across EEG channels. These complementary representations form a compact event memory for jointly classifying events and estimating their onsets and offsets. Before tokenization, we apply FOCAL, an event-aware sparse attention module that uses predicted event presence to guide temporal feature refinement and spatial aggregation, emphasizing likely event periods while preserving context across the recording. Experiments on eight EEG datasets that span diverse paradigms demonstrate state-of-the-art performance. These findings provide a basis for the development of BCI systems that recognize meaningful neural activity during continuous interaction and monitoring.
1 INTRODUCTION
Asynchronous EEG decoding identifies event classes and their onset and offset in continuous recordings. NeuroDETR combines multiscale temporal and hierarchical spatial tokenization with event-aware refinement, and achieves the highest event-level F1 on eight datasets.
1 INTRODUCTION
- Sliding-window classifiers can fragment detections or merge adjacent events because fixed-length windows mismatch variable event durations and boundaries.Performance can also depend on window settings and post-processing rules.Source: §1 · p. 1
- EEG events vary in duration, frequency, and spatial distribution, and their responses are embedded in background activity, so feature aggregation must preserve weak or transient cues.Existing cross-attention methods aggregate long feature sequences into fixed continuous latent tokens without explicitly organizing cues across temporal scales and spatial regions.Source: §1 · p. 2
- NeuroDETR combines multiscale temporal codebooks for brief and sustained dynamics with hierarchical spatial codebooks to form a compact event memory for joint event classification and localization.The framework uses vector quantization to organize continuous EEG features into reusable codewords.Source: §1 · p. 2
- FOCAL uses predicted event presence to refine temporal features and spatial aggregation before quantization while retaining cross-window context.The resulting event-aware multiscale information is supplied to the DETR decoder to predict event classes and boundaries.Source: §1 · p. 2
- Across eight EEG datasets spanning P300, motor imagery, motor execution, speech, and seizure detection, NeuroDETR achieves the highest event-level F1 among evaluated methods.The authors state that this supports development of practical BCI applications.Source: §1 · p. 2
2 RELATED WORK
Prior asynchronous EEG detection ranges from sliding-window classification to query-based event localization. Related discrete EEG work develops tokenizers and spatial or temporal representations, while NeuroDETR combines these ideas for event-set prediction.
2.1 ASYNCHRONOUS EEG DECODING
- Earlier asynchronous EEG systems classify sliding windows or estimate control states, while event-instance methods include direct localization, center–duration regression, query-based set prediction, and dense segmentation.DANCE combines Perceiver-style resampling with query decoding to predict event classes and temporal boundaries.Source: §2.1 · p. 3
2.2 DISCRETE SPATIOTEMPORAL EEG REPRESENTATIONS
- Discrete EEG representations have supported translation, pretraining, and multimodal decoding, alongside tokenizers for time–frequency motifs and temporal–spectral vocabularies.Other related work models EEG with electrode graphs, multiscale temporal filters, regional and global spatial representations, hierarchical spatial tokens, or residual vector quantization.Source: §2.2 · p. 3
- NeuroDETR routes local temporal attention and weights regional-feature pooling with shared event-presence estimates before multiscale temporal and hierarchical spatial quantization.The resulting memory supports event-set prediction.Source: §2.2 · p. 3
3 METHOD
NeuroDETR maps continuous EEG to an unordered set of events, each with a class and normalized temporal boundaries. Its pipeline refines dense features, tokenizes temporal and spatial structure, fuses the memories, and predicts event sets with detection and auxiliary supervision.
3.1 PROBLEM FORMULATION AND OVERVIEW
- The method predicts events in parallel from continuous EEG using learnable event queries, with class 0 denoting no event and boundaries normalized by window duration.The target is an unordered event set, and the number of events can vary across windows.Source: §3.1 · p. 3
3.2 DENSE EEG ENCODING
- A convolutional backbone extracts dense features, which FOCAL refines through event-guided local attention and cross-window interaction.The backbone preserves temporal resolution for FOCAL and auxiliary boundary supervision.Source: §3.1 · p. 3, §3.2 · p. 3
3.3 FOCAL: EVENT-AWARE SPARSE ATTENTION
- FOCAL predicts event presence per window, selects likely-event windows for local attention, and uses axial attention to connect corresponding positions across windows.The density map and routing mask are shared across both FOCAL blocks, while the density head remains supervised under hard routing.Source: §3.3 · p. 4
3.4 SPATIOTEMPORAL VQ TOKENIZATION
- Independent scale-specific codebooks encode temporal dynamics and channel-group structure in the temporal and spatial branches.Quantizers assign features to nearest codewords; codebooks are EMA-updated, underused codes are reinitialized, and commitment losses use valid spatial tokens.Source: §3.4 · p. 5
3.4.1 MULTISCALE TEMPORAL VQ
- Multiscale temporal VQ pools features at multiple scales, interpolates codes to the finest pooled grid, averages them, and adds sinusoidal positional embeddings.A two-layer MLP reconstructs dense features from averaged codes resampled to length T; reconstruction uses codes before positional embedding and memory fusion.Source: §3.4.1 · p. 5
3.4.2 SPATIAL-PYRAMID VQ
- Spatial-Pyramid VQ groups raw channels at whole-head, five coarse-region, and twelve fine-cluster levels before backbone mixing.Density-guided pooling weights group features within temporal bins; the detached weights emphasize likely event periods without updating the density estimator. Finer levels quantize residuals relative to parent codes.Source: §3.4.2 · p. 5
3.5 EVENT MEMORY FUSION AND SET PREDICTION
- Learnable queries undergo temporal fusion with self-attention, sliding-window cross-attention, and feed-forward refinement, then gated spatial integration forms the fused memory.A DETR decoder applies event queries to that memory to predict class probabilities and start and end boundaries; background-class queries are discarded.Source: §3.5 · p. 6
3.6 TRAINING OBJECTIVES
- Training combines Hungarian-matched class and boundary detection losses with auxiliary boundary heatmap supervision, dense classification, and prediction consistency.The consistency term encourages agreement at covered positions; at uncovered positions, clamped zero vectors induce an entropy-regularization effect on dense predictions.Source: §3.6 · p. 6
4 EXPERIMENTS
Experiments evaluate NeuroDETR across eight datasets, comparing it with dense and query-based detectors using sample-wise and event-level F1. NeuroDETR leads on event-level F1 across all datasets, while ablations test its quantization, spatial hierarchy, and event guidance.
4.1 DATASETS
- The eight datasets cover P300, speech, motor imagery, motor execution, and seizure events with varied frequency, duration, and spatial distributions.P300 events may overlap; seizures range from several seconds to several minutes.Source: §4.1 · p. 7
4.2 BASELINES
- Comparisons include four dense detectors, DANCE, and a dataset-aware Random lower bound, using common data splits and event-level evaluation.Dense detectors use a 0.05 confidence threshold and NMS at temporal IoU 0.5.Source: §4.2 · p. 7
4.3 EVALUATION METRICS
- F1sample measures macro-averaged sample-wise classification, whereas F1event jointly evaluates event detection, classification, and localization under one-to-one matching.The main evaluation uses an IoU threshold of 0.5.Source: §4.3 · p. 7
4.4 MAIN RESULTS
- 57.50 average F1sample places NeuroDETR 3.32 percentage points above DANCE, the strongest baseline on average.It ranks first on most datasets in F1sample.Source: §4.4 · p. 7, Table 1 · p. 8
- 46.70 average F1event, 4.29 percentage points above DANCE, with NeuroDETR best on all eight datasets.The largest improvement over the strongest baseline is 8.99 points on BNCI2015.Source: §4.4 · p. 7, Table 2 · p. 8
4.5 ABLATION STUDIES
- The full model exceeds all five core ablations in event F1 on each dataset; removing FOCAL-associated processing and supervision drops scores most on BNCI2020, HDOP, and TUSZ.Those drops are 13.18, 6.14, and 17.08 points, respectively; M4 also reduces HDOP and TUSZ event F1 by 4.95 and 8.76 points.Source: Table 2 · p. 8, §4.5 · p. 8
- The joint quantization-and-guidance configuration has the highest event F1 on all eight datasets, with a 2.06-point average interaction contrast across seven cross-validation datasets.Contrasts vary by dataset, including −0.23 on BI2014b and −0.01 on Broderick.Source: Table 3 · p. 9, §4.5 · p. 9
4.6 VISUALIZATION OF DECODED EVENTS
- Example timelines from BNCI2020 and TUSZ show predicted event classes and boundaries against ground truth, with segment-level eIoU and F1event metrics.F1event uses IoU 0.5; eIoU averages temporal IoU over overlapping, same-class one-to-one matches.Source: Figure 3 · p. 9
5 CONCLUSION
NeuroDETR combines multiscale temporal and hierarchical spatial vector quantization with event-aware refinement to predict event classes and boundaries from continuous EEG. It achieves the highest event-level F1 among compared methods across eight datasets and at every evaluated temporal IoU threshold.
5 CONCLUSION
- Multiscale temporal VQ and Spatial-Pyramid VQ organize temporal dynamics and hierarchical spatial information into a compact event memory.FOCAL uses predicted event presence to guide temporal feature refinement and spatial pooling while preserving cross-window context.Source: §5 · p. 9
- The framework jointly predicts event classes, onsets, and offsets from continuous recordings.The predictions do not require predefined event boundaries at inference time.Source: §5 · p. 9
- Across eight datasets, NeuroDETR achieves the highest event-level F1 among compared methods, improving the average over DANCE by 4.29 percentage points.Ablations support complementary contributions from discrete spatiotemporal representations, multiscale structure, and event-aware refinement.Source: §5 · p. 10
A.1 EVENT-LEVEL F1 AT DIFFERENT TEMPORAL IOU THRESHOLDS
- NeuroDETR achieves the highest event-level F1 at every evaluated IoU threshold, including under stringent boundary-matching requirements.The evaluated thresholds are τIoU ∈ {0.1, 0.3, 0.5, 0.7, 0.9}; higher thresholds require closer alignment with annotated intervals.Source: §A.1 · p. 13, Figure 4 · p. 13
A.2 DETECTION ERROR ANALYSIS
The error analysis separates incorrect detections from missed events using event-level FDR and FNR. NeuroDETR has the highest event F1 on all eight datasets, while single-error-rate advantages for competing methods coincide with a worse complementary error rate.
A.2 DETECTION ERROR ANALYSIS
- FDR measures incorrect detections among predictions, while FNR measures missed events among annotations.A true positive requires a correct class and temporal IoU > 0.5 after one-to-one matching in descending confidence order.Source: §A.2 · p. 13
- NeuroDETR attains the highest F1event on all eight datasets, while achieving the lowest FDR and FNR on six datasets each.Where a competing method has a lower individual error rate, its complementary error rate is higher, so that single-metric advantage does not yield better overall detection.Source: §A.2 · p. 13
A.3 SENSITIVITY TO THE NUMBER OF EVENT QUERIES
Query-count sensitivity varies by dataset, and larger query budgets do not consistently improve event detection. The supplied sections also describe the spatial hierarchy, set-prediction losses, and auxiliary supervision used in NeuroDETR.
A.3 SENSITIVITY TO THE NUMBER OF EVENT QUERIES
- Larger query budgets do not consistently improve event detection, and query-count sensitivity varies by dataset.On HDOP, event F1 falls from 52.91 at 16 queries to 41.17 at 24, then recovers to 51.94 at 32; validation performance motivates dataset-specific query selection.Source: §A.3 · p. 14
B HIERARCHICAL BRAIN-REGION GROUPING
- The spatial hierarchy comprises one global electrode group, five coarse regions, and twelve fine clusters, with dataset-specific mapped channels.Each fine cluster belongs to a coarse region, all coarse regions share the global group as parent, and the two temporal clusters merge at the coarse level while the occipital group remains undivided.Source: B HIERARCHICAL BRAIN-REGION GROUPING · p. 15
C.1 MATCHING AND SET-PREDICTION LOSS
- Set prediction matches ground-truth events to queries, then supervises matched event classes and interval boundaries while assigning unmatched queries to background.Endpoint regression penalizes onset and offset errors; temporal generalized IoU adds interval-level supervision, including when predicted and target intervals do not overlap.Source: §C.1 · pp. 15–16
C.2 AUXILIARY SUPERVISION
- Auxiliary supervision combines boundary prediction, dense classification, and consistency between dense and query-derived predictions.At uncovered positions, the entropy term favors higher-entropy dense predictions, while dense classification uses annotation labels; the set-derived vector is constant there, and discrete interval masks block gradients to predicted boundaries through the projection.Source: §C.2 · pp. 16–17
D DATASETS
The evaluation uses eight EEG datasets spanning five task domains, with varied event types, durations, and recording conditions. The appendix summarizes dataset statistics and describes the split and input-window protocols.
D DATASETS
- Table 6 summarizes dataset domains, subject and channel counts, recording time, mean event duration, event coverage, data splits, and input-window lengths.Event time is the union of configured annotations; overlapping intervals count once when computing event time and coverage.Source: D DATASETS · pp. 17–18, Table 6 · p. 17
- The eight datasets span visual P300, motor imagery, motor execution, naturalistic speech, and clinical seizure detection.They include densely occurring oddball events, 0.2-second HDOP events, word-level speech annotations, and seizure annotations mapped to four configured categories.Source: D DATASETS · p. 18
- The seven non-clinical datasets use subject-grouped five-fold cross-validation, while TUSZ uses its official train/dev/eval split of 579, 53, and 43 subjects.Non-clinical input windows range from 8 to 32 seconds; TUSZ windows span 256 seconds.Source: D DATASETS · p. 18
E.1 BASELINE IMPLEMENTATIONS
NeuroDETR is compared with six event-level baselines under shared data and evaluation conditions; sliding-window classifiers are excluded because they repeatedly process overlapping portions of long recordings. All methods are evaluated using event-level F1 at temporal IoU 0.5.
Comparison setup
- The comparison includes a dataset-aware random lower bound and excludes sliding-window classifiers because overlapping windows repeatedly process portions of long recordings.Dense detectors use confidence threshold 0.05, class-wise NMS at temporal IoU 0.5, and retain at most Ne predictions per window.Source: §E.1 · p. 18
EventNet
- EventNet detects events from center-confidence peaks and duration estimates, extended here with class-specific heads that allow intervals from different classes to overlap.Decoded intervals use the shared post-processing.Source: §E.1 · pp. 18–19
Temporal detection baselines
- ActionFormer uses an EEG encoder with an adjusted multiscale feature pyramid and boundary regression, while DyFADet retains dynamic aggregation and duration-adaptive detection components.ActionFormer’s temporal strides and regression ranges are adjusted for EEG resolution and event durations; the DyFADet adaptation rescales supported regression ranges to dataset event durations.Source: §E.1 · p. 19
DOSED
- DOSED assigns default temporal events to positions and durations, jointly predicting class scores and localization offsets after overlap-based matching during training.Its default-event durations are estimated from training annotations in each fold.Source: §E.1 · p. 19
DANCE
- DANCE uses learnable queries to predict event categories and extents, with one-to-one bipartite matching during training and unmatched predictions assigned to no-event.The query-sensitivity sweep varies decoder-query count while holding ground-truth event capacity and annotations fixed.Source: §E.1 · p. 19
Unified evaluation
- Event-level F1 at temporal IoU 0.5 is computed by confidence-sorting predictions, greedily matching intervals by temporal IoU, and requiring both class correctness and threshold satisfaction for a true positive.Methods share dataset-specific channels, preprocessing, annotations, class mappings, temporal units, subject-level splits, and validation-only model selection.Source: §E.1 · pp. 18–19
F BACKBONE ARCHITECTURE
The SimpleConv backbone combines coordinate-informed channel mixing with a dilated temporal stack to extract spatial and temporal EEG features while preserving sample alignment for event localization and classification.
Spatial feature extraction
- Spatial channel mixing uses Fourier embeddings of electrode coordinates to determine learnable attention weights, mapping recording channels to fixed virtual channels before pointwise projection.The projection maps the mixed signals into the feature space used by the temporal stack.Source: F BACKBONE ARCHITECTURE · pp. 19–20
Temporal feature extraction
- Ten convolutional stages with increasing dilation expand temporal receptive fields while preserving sequence length for event localization and classification.Stages 1–9 use 512 hidden channels with batch normalization, GELU, and dropout; stage 10 outputs 128-dimensional features.Source: F BACKBONE ARCHITECTURE · p. 20
Architecture configuration
- Table 7 summarizes backbone module configurations and input–output dimensions, and all temporal convolutions use stride 1 with symmetric padding 4dℓ.Stages 1–9 also apply pointwise convolution followed by ReLU, while stage 10 contains only the temporal convolution.Source: Table 7 · p. 20, F BACKBONE ARCHITECTURE · p. 20
G IMPLEMENTATION DETAILS
Implementation uses robust-scaled EEG resampled to 128 Hz, subject-wise evaluation splits, AdamW training, and validation F1 for checkpoint selection and early stopping.
Data and evaluation
- Signals are resampled to 128 Hz and robust-scaled; seven datasets use five-fold subject-wise cross-validation with 10% of remaining subjects held out for validation, while TUSZ uses predefined splits.The random seed is 42, and checkpoint selection and early stopping use validation F1event at temporal IoU 0.5.Source: G IMPLEMENTATION DETAILS · p. 20
Training configuration
- Training uses AdamW with a OneCycle schedule, peak learning rate 10^-4, weight decay 10^-3, gradient norm capped at 1, and up to 100 epochs with patience 20.The learning rate increases during the first 10% of training steps; experiments use FP32 precision.Source: G IMPLEMENTATION DETAILS · p. 20
Compute setup
- The seven cross-validation datasets are trained on one GPU, while TUSZ uses eight-GPU distributed data parallel training with four windows per GPU and effective batch size 32.Source: G IMPLEMENTATION DETAILS · p. 20