PaperlayerView exact revision on arXiv

Source-linked AI summary

ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding

Kanglei Zhou, Chunyan Lan, Dongyang Li, Jun Zhu, Liyuan Wang

arXiv:2609.05094v1cs.AI

TL;DR

Robust EEG visual decoding is challenged by fixed semantic anchors that can become misaligned with changing, subject-dependent EEG representations. ProCA progressively adapts semantic supervision and constrains feature mixing using channel-wise and temporal importance, yielding gains across four decoding settings.

  • Problem

    Fixed visual or textual anchors may become misaligned with EEG representations that vary across trials, subjects, and learning stages, limiting robust decoding.

  • Method

    ProCA progressively refines class-level supervision from frozen vision-language priors to EEG-aware relations and applies structure-consistent interpolation based on channel-wise and temporal importance.

  • Results

    ProCA improves average relative Top-1/Top-5 performance across subject-dependent, subject-independent, strict cross-subject, and continual-adaptation settings.

  • Takeaways & Limitations

    Adaptive supervision and structure-aware interpolation jointly support stable and discriminative EEG–semantic alignment under distribution shift.

Abstract

from arXiv · show

Electroencephalogram (EEG) visual decoding aims to recover visual semantics from non-invasive neural time-series signals, for which robust alignment between noisy neural responses and stable semantic representations is key to achieving high-performance decoding. Despite recent advances in contrastive learning, robust EEG decoding remains challenging because existing methods rely on fixed visual or textual anchors whose semantic relations may become misaligned with EEG representations that vary across trials, subjects, and learning stages. Our empirical evidence shows that this instability appears across both standard EEG decoding protocols and more challenging robustness settings, including strict cross-subject transfer and realistic personalized continual adaptation. We provide a formal analysis showing that fixed semantic supervision can bias optimization when EEG-specific relations evolve, and that structure-agnostic perturbations may distort semantically important EEG components. To address these issues, we propose Progressive Contrastive Alignment (ProCA), a unified and model-agnostic framework for adaptive neural-semantic alignment. ProCA progressively refines class-level contrastive supervision from frozen vision-language priors to EEG-aware semantic relations, and introduces structure-consistent interpolation to constrain feature mixing according to channel-wise and temporal importance. Across subject-dependent, subject-independent, strict cross-subject transfer, and continual adaptation settings, ProCA achieves average relative Top-1/Top-5 gains of 7.4%/3.9%, 10.0%/4.6%, 28.1%/17.8%, and 16.8%/11.6%, respectively.

I. INTRODUCTION

EEG visual decoding requires robust alignment between noisy, subject-dependent neural responses and stable visual or linguistic semantics. ProCA addresses instability in fixed alignment by adapting supervision and respecting EEG structure, improving decoding across diverse settings.

  • EEG visual decoding recovers visual information from non-invasive neural time-series signals for brain-computer interfaces and visual-perception studies.
  • Low signal-to-noise ratios, physiological artifacts, ambiguous neural-semantic mappings, and subject variability make robust alignment difficult.
  • Existing contrastive methods align EEG with pretrained visual or textual embeddings but commonly assume static semantic relations.
  • ProCA progressively incorporates EEG-aware class relations into frozen vision-language priors and applies structure-consistent channel–time regularization.
  • 7.4%/3.9%, 10.0%/4.6%, 28.1%/17.8%, and 16.8%/11.6% average relative Top-1/Top-5 gains occur in subject-dependent, subject-independent, strict cross-subject, and continual-adaptation settings, respectively.

B. Evaluation Settings and Empirical Observations

The evaluation spans personalization, subject shifts, and evolving distributions, exposing instability in fixed contrastive supervision. Formal analysis motivates adaptive relations and structure-aware perturbations because EEG-specific relations and important channel–time components change.

  • Evaluation Settings: Four evaluation settings assess within-subject personalization, unseen-subject generalization, strict cross-subject transfer, and continual adaptation.
  • Empirical Observations: In 200-way strict cross-subject decoding, baseline Top-1/Top-5 accuracy drops from 28.5%/59.9% in the standard setting to 4.1%/14.3%.
  • Theoretical Analysis: Fixed supervision can bias optimization when the current EEG-specific relation structure differs from relations defined by visual or textual anchors.
  • Theoretical Analysis: Structure-agnostic perturbations may distort semantically important channel–time components because EEG units contribute unequally to semantic alignment.

D. Design Implications

ProCA addresses neural–semantic alignment instability by combining adaptive contrastive supervision with structure-aware regularization. Its objective aligns EEG representations with frozen semantic embeddings while progressively updating supervision using EEG-aware relations.

  • The framework combines adaptive class-level supervision with channel–time structure-consistent regularization for robust EEG visual decoding.
  • ProCA targets fixed contrastive supervision and structure-agnostic regularization as two sources of EEG–semantic alignment instability.
  • ProCA aligns EEG representations with frozen image and text embeddings while conditioning contrastive learning on a stage-dependent confusion structure.
  • The confusion structure begins from frozen image and text semantic spaces and is progressively updated through EEG–semantic interactions during training.

B. Progressive Confusion-guided Alignment (PCA)

Progressive Confusion-guided Alignment replaces fixed negative relations with a stage-dependent, class-level confusion structure. It begins with frozen vision–language priors and gradually incorporates EEG-aware relations as representations become informative.

  • PCA makes contrastive supervision consistent with the evolving structure of current EEG representations rather than relying only on fixed semantic relations.
  • Unlike hard-negative mining, PCA models class-level, stage-dependent confusion by progressively refining stable semantic priors with EEG–semantic interactions.
  • PCA initializes class confusion from frozen vision–language embeddings because early EEG representations are unreliable for estimating semantic relations.
  • EEG-aware class relations are periodically estimated from current neural representations and used to refine the confusion structure during training.
  • The stage-dependent update increases EEG-aware influence gradually, avoiding noisy early estimates while enabling later EEG-specific guidance.
  • Larger off-diagonal confusion values assign greater negative emphasis to easily confused class pairs in continuous class-level supervision.

C. Structure-Consistent Interpolation (SCI)

Structure-Consistent Interpolation regularizes EEG representations while respecting their channel-wise and temporal organization. It preserves important components more strongly and mixes less critical regions more freely.

  • SCI addresses the limitation of standard mixup, whose global coefficient treats EEG representations as structure-agnostic vectors.
  • SCI assigns channel–time-dependent interpolation strengths according to structural importance, preserving important EEG components and mixing less critical regions more strongly.
  • A parameter-free attribution operator estimates channel-wise and temporal-wise importance from intermediate EEG representations.
  • Importance-biased coefficients preserve high-importance activations near their anchors, while lower-importance units allow stronger mixing; an additional constraint prevents degenerate interpolation.
  • SCI forms a structured interpolation matrix Λ ∈ R^C×T and mixes two intermediate feature maps element-wise using these channel–time coefficients.
  • Unlike standard mixup, SCI preserves important channel–time components while encouraging smoothness and reducing semantic distortion from perturbing them.

V. EXPERIMENTS

ProCA is evaluated across complementary EEG visual decoding datasets, retrieval metrics, model baselines, and robustness settings. Results show consistent gains, including under leave-one-subject-out generalization, strict cross-subject transfer, and continual adaptation.

  • Datasets: ProCA is evaluated on THINGS-EEG2 and Alljoined-1.6M, covering controlled high-fidelity decoding and large-scale accessible EEG acquisition scenarios.THINGS-EEG2 includes 10 subjects and 82,160 EEG trials; Alljoined-1.6M contains approximately 1.6M EEG trials and more subjects.
  • Metrics: Top-1 and Top-5 accuracy measure whether the ground-truth stimulus ranks first or within the five retrieved candidates.Higher values indicate better EEG–semantic alignment.
  • Subject-Dependent Performance: ProCA improves subject-dependent decoding by +3.8% Top-1 and +2.5% Top-5 on THINGS-EEG2, with larger gains on Alljoined-1.6M.The reported averages cover 200-, 150-, and 100-way settings across backbones; Alljoined-1.6M reaches +15.7% Top-1 and +7.3% Top-5.
  • Strict Cross-Subject Transfer: In strict cross-subject transfer, ProCA improves NeuroBridge from 2.69% to 3.68% average Top-1 and from 10.08% to 12.03% average Top-5.The gains are broadly distributed across source subjects rather than attributable to favorable random seeds.
  • Continual Adaptation Performance: ProCA achieves average relative gains of 16.8% Top-1 and 11.6% Top-5 across continual adaptation settings.The comparison uses the highest-performing eligible non-ProCA baseline within each setting and an unweighted macro-average.

C. Ablation Study

The ablations show that ProCA’s adaptive supervision and structure-aware interpolation provide complementary gains, while its performance remains stable across hyperparameters and robust across alignment and optimization analyses.

  • Ablation Study: Removing PCA reduces average Top-1/Top-5 accuracy from 30.6%/62.4% to 29.2%/61.2%, confirming the value of adaptive class-level supervision.
  • Ablation Study: Removing SCI reduces average Top-1/Top-5 accuracy to 29.8%/61.8%, showing that structure-aware regularization contributes independently to robust alignment.
  • Ablation Study: Naive hard-negative mining and unconstrained mixup perform worse than progressive updating and structure-preserving interpolation, respectively.The comparison attributes the gains to adapting confusion structure and protecting important channel–time components.
  • Ablation Study: ProCA remains stable across αmax ∈[0.2, 1.0], typically performs best near αmax = 0.8, and generally reduces Hessian sharpness measures.The loss-landscape analysis uses the maximum Hessian eigenvalue and Hessian trace, where lower values indicate flatter minima.
  • Ablation Study: ProCA achieves the best average Top-1 and Top-5 accuracy across 200-, 150-, and 100-way subject-independent retrieval settings, beyond Euclidean Alignment.

APPENDIX A ADDITIONAL THEORETICAL ANALYSIS

The analysis formalizes two instability sources: fixed semantic supervision can become biased as EEG-specific relations evolve, and structure-agnostic perturbations can distort important EEG components.

  • APPENDIX A ADDITIONAL THEORETICAL ANALYSIS: The theoretical analysis does not establish a direct accuracy guarantee but formalizes optimization bias and semantic distortion as two instability sources.
  • APPENDIX A ADDITIONAL THEORETICAL ANALYSIS: The contrastive objective is conditioned on a semantic relation structure C through the EEG encoder fθ.
  • APPENDIX A ADDITIONAL THEORETICAL ANALYSIS: Fixed semantic supervision becomes increasingly mismatched when the ideal EEG-specific relation structure drifts from the static semantic prior.
  • APPENDIX A ADDITIONAL THEORETICAL ANALYSIS: The resulting relation mismatch can increase gradient deviation, supporting adaptive contrastive supervision instead of relying solely on static semantic anchors.

B. Proof of the Structure-Weighted Perturbation Bound

The structure-weighted perturbation analysis assigns greater influence to important channel–time units, bounding loss changes under assumptions of local smoothness.

  • B. Proof of the Structure-Weighted Perturbation Bound: The importance matrix A assigns a non-negative weight Ac,t to each EEG channel and temporal position, with η ensuring a strictly positive-definite weighted norm.
  • B. Proof of the Structure-Weighted Perturbation Bound: Under local smoothness in the dual weighted norm, Proposition 4 bounds the sample-level contrastive-loss change caused by perturbation δ.
  • B. Proof of the Structure-Weighted Perturbation Bound: Perturbations on high-importance units contribute more strongly to the structure-weighted bound, so robustness perturbations should preserve important channel–time regions.

C. Connection to Structure-Aware Interpolation

The interpolation analysis connects channel–time-dependent mixing to distortion control: preserving important units more strongly yields no larger importance-weighted perturbation than uniform interpolation.

  • C. Connection to Structure-Aware Interpolation: Channel–time-dependent interpolation uses Λ ∈[0, 1]C×T to control preservation strength for each unit from ha.
  • C. Connection to Structure-Aware Interpolation: Standard interpolation uses a single global coefficient λ, whereas structure-aware interpolation varies preservation across channel–time units.
  • C. Connection to Structure-Aware Interpolation: If Λc,t ≥ λ on structurally important units, their importance-weighted perturbation is no larger than under standard interpolation.
  • C. Connection to Structure-Aware Interpolation: Preserving important channel–time units with larger coefficients reduces their contribution to the structure-weighted perturbation norm and the associated semantic-distortion bound.

APPENDIX B PSEUDOCODE OF TRAINING PROCEDURE

ProCA trains with a single contrastive objective while progressively adapting semantic supervision and structure-aware regularization to evolving EEG representations.

  • Training procedure: ProCA reformulates EEG visual decoding as progressive contrastive alignment, adapting supervision as EEG representations evolve through successive optimization stages.The contrastive objective remains fixed while the confusion structure changes during training.
  • Training procedure: Training begins with a confusion structure derived from frozen image and text embeddings, providing a stable multimodal prior for early learning.The EEG encoder is optimized using the current confusion structure without additional auxiliary losses or task-specific objectives.
  • Training procedure: ProCA optimizes one contrastive learning objective throughout training, with progression controlled by confusion-structure evolution and structure-aware regularization.This keeps the training procedure conceptually simple while adapting to changing EEG representations.
  • Training procedure: Loss-landscape analysis compares local loss variations around baseline and ProCA parameters for EEG–image retrieval under identical inputs.The analysis uses the similarity objective for EEG–image retrieval, with ℓ2-normalized EEG and image features.
  • Training procedure: The evaluation fixes a randomly sampled mini-batch of 256 EEG–image pairs and uses a random-plane method to construct a two-dimensional parameter subspace.Keeping inputs unchanged isolates loss variation caused by parameter perturbations.
  • Training procedure: Image reconstruction uses retrieval and generation stages, combining a real-stimulus gallery with CLIP features and SDXL-based synthesis guided by EEG semantics.The supplied passages identify the retrieval gallery construction and the SDXL generation backbone.

8 Structure-consistent interpolation (SCI):

SCI estimates channel-wise and temporal importance, interpolates feature maps accordingly, and feeds the resulting representations into ProCA’s contrastive alignment and update procedure.

  • Structure-consistent interpolation (SCI): SCI estimates channel-wise and temporal importance from intermediate EEG feature maps before generating interpolated representations.The procedure explicitly extracts feature maps and computes importance values before interpolation.
  • Structure-consistent interpolation (SCI): The interpolated feature maps are propagated through the remaining encoder layers before contrastive alignment under the current confusion structure.The resulting representations are used to compute the ProCA loss.
  • Structure-consistent interpolation (SCI): ProCA periodically computes class-wise EEG centroids and EEG-interacted image and text confusion matrices, then fuses the branches to update the confusion structure.These updates provide the progressive supervision used by subsequent alignment steps.
  • Structure-consistent interpolation (SCI): During inference, the trained EEG encoder produces a normalized feature that is compared with gallery image features using cosine similarity to select the best retrieved image.The selected image is subsequently used as a visual prompt for IP-Adapter and SDXL generation.
  • Structure-consistent interpolation (SCI): Qualitative reconstruction comparisons use identical retrieval and generation settings for baseline and ProCA features, while ground-truth images are used only for visualization.The comparison assesses semantic decoding accuracy and visual consistency without using ground-truth images in retrieval or generation.
  • Structure-consistent interpolation (SCI): Strict cross-subject transfer tests whether representations remain transferable under physiological, signal-to-noise, channel-response, and activation-timing differences across unseen subjects.Single-subject training avoids distribution alignment from joint multi-subject training and provides a more stringent generalization test.
  • Structure-consistent interpolation (SCI): Continual adaptation updates a personalized decoder as visual concepts arrive sequentially and evaluates after each stage over all categories observed so far.Replay-based baselines use a fixed buffer capacity of 5,000 samples unless otherwise specified.

APPENDIX C ADDITIONAL EXPERIMENTS AND ANALYSIS

Additional analyses show that ProCA improves representation separation and reconstruction semantics, remains moderately robust to hyperparameter choices, adds modest training overhead, and retains intrinsic EEG decoding limitations.

  • Feature-space visualization: ProCA produces more compact and better-separated EEG feature clusters than the baseline under challenging cross-subject shifts and semantic ambiguity.The t-SNE comparison indicates improved representation discriminability and robustness.
  • Qualitative reconstruction: ProCA reconstructions are generally more semantically consistent than baseline outputs, although visually complex or ambiguous stimuli still produce failures.The failures reflect remaining challenges in real-world EEG reconstruction and deployment.
  • Failure cases: High intra-class variability, ambiguous semantic boundaries, visually similar categories, noise, and subject-specific variability remain important sources of reconstruction error.These conditions can cause drift toward correlated but incorrect categories or reduce semantic fidelity.
  • Failure cases: Even in failure cases, ProCA tends to preserve higher-level semantic attributes better than the baseline, but weak or noisy neural signals remain unresolved.The paper identifies richer neural signals or additional multimodal constraints as possible directions for addressing these limitations.
  • Hyperparameter sensitivity: Across 100-, 150-, and 200-way settings, the best performance occurs at αmax = 0.8 with a transition epoch around 25, while overly early transitions are more harmful than late ones.This pattern supports the stability of the progressive design rather than brittle hyperparameter tuning.
  • Hyperparameter sensitivity: 40.98%/74.88% Top-1/Top-5 is achieved in the 100-way setting at αmax = 0.8, compared with 40.50%/74.18% at αmax = 1.0.The modest variation indicates that overly aggressive EEG dominance is detrimental but does not cause large overall swings.
  • Hyperparameter sensitivity: 33.78%/67.70% Top-1/Top-5 is achieved in the 150-way setting at αmax = 0.8, while both under-weighting and over-weighting EEG reduce performance.The total variation across αmax remains small in this setting.
  • Computational analysis: 3–4 minutes of additional average training time are required for PCA and SCI, while SCI itself incurs negligible computational cost.The added operations are not a significant computational bottleneck relative to encoder and cross-modal propagation.
Loading 2609.05094v1…