PaperlayerView exact revision on arXiv

Source-linked AI summary

EEG-VID: Task-Guided Latent Predictive Pretraining for EEG Decoding and Assistive Target Selection

Guanzhong Sun, Junyi Ma, Yuxuan Wu, Yanzi Miao

arXiv:2609.00566v2cs.LGcs.AI

TL;DR

EEG decoding remains difficult under session and subject shifts because EEG is noisy and variable. EEG-VID addresses this with task-guided latent predictive pretraining followed by supervised fine-tuning, improving matched comparisons across datasets, backbones, and protocols. The results support transferable pretraining and offline scene-constrained assistive target selection, while remaining bounded by the study’s limited participants, calibration, and sensing setup.

  • Problem

    EEG decoding is challenged by noise, non-stationarity, and session and subject variability, while supervised decoders do not explicitly model cross-window predictive structure.

  • Method

    EEG-VID predicts future latent EEG states from historical windows using weak task guidance, then transfers the learned representation to supervised decoding.

  • Results

    41 of 42 matched backbone–dataset–protocol comparisons improve with Stage 1, including all 12 leave-one-subject-out settings.

  • Takeaways & Limitations

    The results support task-guided latent prediction as a transferable EEG pretraining strategy and an offline scene-informed target-selection proof of concept.

  • Takeaways & Limitations

    VIG-48 contains one participant, the robot-scene study uses subject-specific calibration, and the four-channel unconstrained-gaze setup cannot separate cortical from ocular contributions.

Abstract

from arXiv · show

We propose EEG-VID, a task-guided latent predictive pretraining framework for EEG decoding under session and subject shifts. EEG-VID predicts future latent EEG states from recent history using an exponential-moving-average target encoder and weak task guidance, followed by supervised fine-tuning. Across VIG-48 and BCI Competition IV-2a/IV-2b, Stage 1 improves mean accuracy in 41 of 42 matched backbone-dataset-protocol comparisons, including all 12 leave-one-subject-out settings, with a maximum gain of 16.22 percentage points. On the 48-region cross-day VIG-48 task, EEG-VID achieves 6.52% Top-1 and 30.50% Top-5 accuracy. In a separate six-participant offline robot-scene study, candidate-constrained target selection reaches 40.24% versus a 25% chance level after subject-specific calibration. These results support task-guided latent prediction as a transferable pretraining strategy for EEG decoding and scene-constrained assistive target selection.

I. INTRODUCTION

EEG-VID addresses noisy, non-stationary EEG decoding under session and subject shifts by combining task-guided latent prediction with supervised decoding. It is evaluated across datasets, backbones, protocols, and an assistive target-selection study.

  • Motivation: EEG signals vary across sessions and subjects, while supervised decoders do not explicitly model cross-window predictive structure.Relevant variability includes electrode placement, contact impedance, fatigue, and other non-stationary factors.
  • Approach: EEG-VID predicts future latent EEG states while adding weak task guidance to emphasize predictable components relevant to decoding.The framework uses task-guided latent predictive pretraining before supervised EEG decoding.
  • Results: 41 of 42 matched backbone–dataset–protocol comparisons improve with Stage 1, including all 12 leave-one-subject-out settings.Alternative recurrent and Transformer predictors outperform direct supervision in 48 of 54 comparisons.
  • Evaluation: Stage 1 is designed as a backbone-agnostic objective and tested across five EEG backbones, predictor architectures, and session- and subject-shift protocols.The evaluation includes pooled cross-session, within-subject, and leave-one-subject-out settings.
  • Assistive evaluation: The study additionally evaluates scene-constrained target selection using the learned spatial posterior in a six-participant offline robot-scene experiment.The broader context includes assistive robotics and EEG-based intention decoding.

III. METHODS

EEG-VID uses a two-stage architecture that learns temporal transitions between latent EEG states and transfers the resulting representation to supervised visual-intention region decoding.

  • Two-stage framework: EEG-VID first learns latent transitions from historical to future EEG representations, then transfers the learned dynamic representation to supervised region decoding.The temporal consistency constraint is combined with weak task guidance during pretraining.
  • Problem formulation: The task predicts the intended region label in an N × M visual grid containing NM candidate regions.Each trial presents a visual scene partitioned into grid regions.
  • Latent prediction: During training, future latent EEG representations are predicted from historical windows, while inference requires only the historical input sequence.This formulation models temporal transitions rather than decoding each EEG window independently.
  • Encoder: The online encoder combines a multi-scale temporal branch with a statistical branch to represent temporal dynamics and signal statistics.The temporal branch uses parallel one-dimensional convolutions, residual processing, positional embeddings, and Transformer-based dependency modeling.

2) Statistical branch:

EEG-VID combines temporal and statistical EEG features into latent states, then uses historical tokens, intention context, and future-state prediction with weak task guidance for pretraining.

  • Intention modeling: Historical latent tokens feed an intention branch that produces an intention representation while remaining masked from the future EEG target.The intention branch operates only on historical latent tokens and uses masked self-attention.
  • Latent prediction: The latent predictor conditions future-state prediction on EEG history and intention context, using a predictor architecture with attention and MLP blocks.The predicted future latent state represents EEG-state dynamics.
  • Statistical branch: The encoder fuses multi-scale temporal patterns with stable statistical descriptors through interaction features before projecting them into a latent EEG representation.The statistical branch uses channel-wise temporal descriptors, pairwise correlations, and binned log-power spectral features.
  • Target encoder: An EMA target encoder with stop-gradient stabilizes the prediction target and limits trivial co-adaptation between online and target representations.The EMA decay is fixed to λema = 0.995.
  • Weak task guidance: Weak auxiliary intention classification guides predictable representations toward decoding-relevant components, while the small classification weight does not replace latent prediction.The default task-guidance weight is λS1_cls = 0.1, but the tested sweep does not establish global optimality.
  • Training supervision: Stage 1 uses training-split labels through the intention loss, making the pretraining procedure weakly supervised rather than self-supervised.This qualification follows directly from the use of task labels in Lintent.

D. Supervised Visual Intention Region Decoding (Stage 2)

Stage 2 transfers the pretrained encoder, intention branch, and latent predictor to supervised region decoding, combining current evidence with temporal context and predictive dynamics.

  • Stage 2 transfer: Stage 2 transfers the online encoder, intention branch, and latent predictor into supervised EEG decoding.The decoder uses more than the current EEG window alone.
  • Decoder inputs: The decoder fuses current evidence, temporally aggregated intention context, and predicted dynamics for region classification.This design incorporates learned temporal structure into supervised decoding.
  • Spatial consistency: A coordinate consistency term exploits the 2-D grid geometry of region labels by comparing expected and ground-truth row and column coordinates.Class indices are mapped to grid coordinates before the consistency loss is computed.
  • Predictive retention: Fine-tuning retains a weak predictive-consistency term to preserve the transition structure learned during pretraining.The retention loss uses the same predictive weights as Stage 1.
  • Inference constraints: The future EEG window is used only to compute retention loss during training and is not required at inference.For datasets without 2-D label geometry, the coordinate-loss weight is set to zero.

E. Robot Target Selection

The robot-scene procedure constrains EEG-based spatial predictions to four scene-derived object candidates, enabling offline target selection without speech, gestures, or manual commands.

  • Scene candidates: SAM segments the four objects in each scene and maps their masks to labels on the same 6 × 8 spatial grid used by EEG-VID.The resulting candidate set contains four scene-dependent spatial labels.
  • Candidate-constrained selection: EEG-VID produces 48-way spatial logits, and the selected target is the candidate object whose SAM-derived label receives the largest logit.SAM supplies candidate labels but does not infer the user’s intended target.
  • Chance baseline: Random selection among the four candidate objects gives a 25% chance level.Every evaluated scene contains four candidates.
  • Evaluation setup: The offline experiment evaluates six tabletop configurations with four objects at varying spatial positions, using object masks to assign occupied grid cells.Each RGB observation is processed by SAM before EEG-based candidate selection.
  • Scope: The reported endpoint is target selection before motion planning; grasp execution is not evaluated.Selected masks may be fused with depth for downstream grasp planning, but execution remains outside the evaluation.

F. Data Acquisition, Ethics, and Preprocessing

The study combines in-house and benchmark EEG evaluations with documented acquisition, preprocessing, cross-day testing, and participant-specific calibration procedures.

  • Participants and ethics: Seven healthy adults participated: one contributed the VIG-48 cross-day corpus and six contributed the robot-scene dataset.All participants provided written informed consent.
  • Acquisition: In-house EEG was recorded at 256 Hz with four Muse S Athena electrodes while participants viewed targets on a 6 × 8 grid.Targets were highlighted for 3–6 seconds without flicker, eye tracking, or gaze-contingent hardware.
  • Preprocessing: A fixed preprocessing pipeline handled non-finite values, outliers, 0.5–45 Hz band-pass filtering, and temporal smoothing without ICA or dedicated ocular-artifact rejection.With four channels and no reference EOG, ocular components could not be identified reliably.
  • Cross-day evaluation: VIG-48 uses approximately 15,000 windows, with held-out test days disjoint from training days to measure cross-day distribution shift.Training and validation are separated by complete event identifiers before Stage 1 context construction.
  • Robot-scene calibration: Robot-scene data come from six additional participants across six configurations, with 48 subject-specific calibration trials before held-out testing.Each calibration trial lasts 10 seconds, providing 480 seconds of calibration EEG per participant.
  • Participant adaptation: Each participant’s model is adapted using only that participant’s calibration data, and evaluation reports window-level candidate-constrained target-selection accuracy.Stage 2 supervised fine-tuning is used for adaptation, without rerunning Stage 1 on robot calibration data.

3) Public benchmarks:

The public-benchmark evaluation tests Stage 1 transfer on standard motor-imagery EEG datasets under pooled, within-subject, and leave-one-subject-out protocols. These controlled comparisons use common preprocessing and event-level splitting, while the short trials yield partly padded predictive histories.

  • IV-2a is a four-class, 22-channel motor-imagery benchmark, whereas IV-2b is a two-class, three-channel benchmark.
  • Three protocols vary the availability of subject-specific training data: pooled cross-session, within-subject, and leave-one-subject-out evaluation.The leave-one-subject-out protocol excludes calibration data from the held-out subject.
  • Training and validation remain separated by complete event identifiers, with normalization estimated only from training windows before Stage 1 context construction.
  • All benchmark results use 1-s-window accuracies rather than complete-trial decisions, so absolute values are not directly comparable with published competition results.Within-table comparisons remain controlled because methods share preprocessing and decision protocols.
  • IV-2a and IV-2b provide only three Stage 1 samples per trial, with the first two partly left-padded and therefore shorter than the nominal 4-s history.This asymmetry may explain part of the variation in Stage 1 gains across datasets.

D. Baselines, Variants and Metrics

The study separates the Stage 1 training strategy from decoder architecture by comparing matched backbones, predictor replacements, and common evaluation procedures. Metrics cover Top-1 decoding, VIG-48 Top-5 accuracy, and candidate-constrained robot-scene selection.

  • Baselines and metrics: EEG-VID is compared with seven locally retrained methods and a no-Stage 1 ablation using mean ± standard deviation over five fixed seeds.
  • Baselines and metrics: Robot-scene performance is SAM-constrained target-selection accuracy, with the highest-logit candidate object compared against a 25% random-chance level.
  • Transfer variants: External-backbone variants replace each classifier with a latent adaptation layer, EMA future target, predictor, and lightweight auxiliary classifier using the shared Stage 1 losses.
  • Transfer variants: TLP variants initialize the original supervised decoder after Stage 1 and are compared with their corresponding backbone without Stage 1.
  • Predictor alternatives: LSTM, GRU, and standard Transformer predictors use the same dimensions, objective, task-guidance weight, splits, and training policy without predictor-specific tuning.
  • Statistical analysis: Accuracy-change counts are descriptive, while paired differences and repeated-run variability are summarized without treating reused comparisons as independent significance tests.

A. Pooled Cross-Session Comparison

The pooled cross-session results show that Stage 1 improves matched decoders across datasets and architectures, while the magnitude varies by backbone and task. The broader comparison supports transferability rather than a fixed architecture-specific advantage.

  • Pooled comparison: 6.52 ± 0.95%/30.50 ± 1.79% Top-1/Top-5 on VIG-48, 51.30 ± 0.42% on IV-2a, and 75.90 ± 1.23% on IV-2b are achieved by EEG-VID.Relative to the identical architecture without Stage 1, Top-1 gains are 1.47, 11.30, and 8.23 points, respectively.
  • Pooled comparison: EEG2Rep and BENDR remain below EEG-VID on the three Top-1 tasks and VIG-48 Top-5 under the shared local pipeline.The comparison is limited to the present low-channel, small-data regime.
  • Backbone transfer: All 15 external-backbone Top-1 mean changes are non-negative, with DeepConvNet gaining 19.29 points on IV-2a and TSConv changing by only 0.05±1.16 points on IV-2b.The TSConv result is interpreted as no measurable effect.
  • Backbone transfer: Some Stage 1-enhanced external backbones exceed EEG-VID in absolute accuracy, including TLP-Neuro3D on VIG-48 and TLP-DeepConvNet on IV-2a/IV-2b.This supports interpreting Stage 1 as a transferable training strategy rather than an architecture-specific advantage.
  • Subject-wise transfer: Under LOSO evaluation, the largest mean gain is +16.22 points for DeepConvNet, while EEG-VID improves from 33.55% to 40.80% on IV-2a.Stage 1 improves the mean for all six models on both public datasets.
  • Subject-wise transfer: Stage 1 produces a positive mean change in 41 of 42 matched decoder–dataset–protocol comparisons across pooled, within-subject, and LOSO evaluations.These counts are descriptive rather than population-level significance tests.
  • Objective ablation: Weak task guidance improves over latent-only pretraining in all 18 model–dataset comparisons, while latent-only EEG-VID still beats the no-Stage 1 model on all three datasets.
  • Objective ablation: Weights 0.2 and 0.3 underperform the default in 34 of 36 comparisons, although the tested sweep does not establish global optimality.

E. Predictor Architecture Robustness

Predictor-architecture ablations show that Stage 1 provides a transferable benefit across generic temporal predictors, while the proposed predictor adds a smaller architecture-dependent advantage. The section also describes scene-constrained selection, where EEG scores choose among SAM-derived candidates.

  • Predictor robustness: 48 of 54 pooled model–dataset comparisons favored recurrent or Transformer replacements over direct supervision.This includes all 18 IV-2a comparisons.
  • Scene-constrained selection: SAM supplies eligible object locations, while EEG-VID selects the candidate with the largest EEG score.The scene pipeline segments four objects, maps them to candidate grid labels, and restricts the EEG posterior accordingly.
  • Scene-constrained selection: 40.24% window-level candidate-constrained selection exceeded the 25% random chance level by 15.24 percentage points.The result concerns four scene-derived candidates and does not directly compare with unconstrained 48-way decoding.
  • Scope: The robot-scene experiment evaluates target selection before motion planning and does not measure grasp success.It therefore remains an offline selection test rather than a closed-loop manipulation evaluation.

VI. DISCUSSION

Discussion results identify task-guided latent prediction as the main transferable mechanism across subject and session shifts, with predictor design providing an additional gain. The study remains limited by single-participant visual data, subject-specific calibration, unconstrained gaze, protocol differences, and added computation.

  • Main findings: Stage 1 improves matched decoders across datasets and protocols, with the clearest gains under subject shift.Subject-wise results indicate that improvements are broadly distributed rather than driven by a small participant subset.
  • Mechanism: Latent prediction improves direct supervision, weak task guidance strengthens the representation, and generic temporal predictors retain the benefit.The ablations separate the training-objective contribution from predictor architecture.
  • Interpretation: Task-guided latent prediction is the primary transferable mechanism tested, while predictor design contributes an additional architecture-dependent gain.The selected task-guidance weight and predictor configuration are not necessarily optimal for other datasets or models.
  • Limitations: Unconstrained gaze and the four-channel montage prevent separating cortical, visually evoked, and ocular contributions without EOG or eye tracking.The visual decoding results are therefore interpreted at the wearable-device level rather than as source-specific cortical decoding.
  • Limitations: VIG-48 contains one participant, and the robot-scene study uses subject-specific calibration.Future work is needed to test multi-participant visual data and cross-subject transfer with less calibration.
  • Scope: The current results support transferable pretraining and an offline scene-informed selection proof of concept, not a deployable BCI or source-specific neural mechanism.Public-benchmark 1-s decisions, added Stage 1 computation, and the lack of closed-loop evaluation further bound the conclusions.
Loading 2609.00566v2…