ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding

• EEG visual decoding recovers visual information from non-invasive neural time-series signals for brain-computer interfaces and visual-perception studies.Source: §I · p. 1, A. EEG Visual Decoding Formulation · p. 3
• Low signal-to-noise ratios, physiological artifacts, ambiguous neural-semantic mappings, and subject variability make robust alignment difficult.Source: §I · p. 1
• Existing contrastive methods align EEG with pretrained visual or textual embeddings but commonly assume static semantic relations.Source: §I · p. 1, §II · p. 3
• ProCA progressively incorporates EEG-aware class relations into frozen vision-language priors and applies structure-consistent channel–time regularization.Source: §I · p. 2
• 7.4%/3.9%, 10.0%/4.6%, 28.1%/17.8%, and 16.8%/11.6% average relative Top-1/Top-5 gains occur in subject-dependent, subject-independent, strict cross-subject, and continual-adaptation settings, respectively.Source: §I · p. 2

Evaluation Settings

• Four evaluation settings assess within-subject personalization, unseen-subject generalization, strict cross-subject transfer, and continual adaptation.Source: B. Evaluation Settings and Empirical… · pp. 3–4

Empirical Observations

• In 200-way strict cross-subject decoding, baseline Top-1/Top-5 accuracy drops from 28.5%/59.9% in the standard setting to 4.1%/14.3%.Source: B. Evaluation Settings and Empirical… · p. 4

Theoretical Analysis

• Fixed supervision can bias optimization when the current EEG-specific relation structure differs from relations defined by visual or textual anchors.Source: C. Theoretical Analysis · p. 4
• Structure-agnostic perturbations may distort semantically important channel–time components because EEG units contribute unequally to semantic alignment.Source: C. Theoretical Analysis · p. 4

• The framework combines adaptive class-level supervision with channel–time structure-consistent regularization for robust EEG visual decoding.Source: D. Design Implications · p. 5
• ProCA targets fixed contrastive supervision and structure-agnostic regularization as two sources of EEG–semantic alignment instability.Source: §IV · p. 5
• ProCA aligns EEG representations with frozen image and text embeddings while conditioning contrastive learning on a stage-dependent confusion structure.Source: A. Framework Overview · p. 5
• The confusion structure begins from frozen image and text semantic spaces and is progressively updated through EEG–semantic interactions during training.Source: A. Framework Overview · p. 5

• PCA makes contrastive supervision consistent with the evolving structure of current EEG representations rather than relying only on fixed semantic relations.Source: B. Progressive Confusion-guided… · p. 5
• Unlike hard-negative mining, PCA models class-level, stage-dependent confusion by progressively refining stable semantic priors with EEG–semantic interactions.Source: B. Progressive Confusion-guided… · pp. 5–6
• PCA initializes class confusion from frozen vision–language embeddings because early EEG representations are unreliable for estimating semantic relations.Source: B. Progressive Confusion-guided… · p. 5
• EEG-aware class relations are periodically estimated from current neural representations and used to refine the confusion structure during training.Source: B. Progressive Confusion-guided… · pp. 5–6
• The stage-dependent update increases EEG-aware influence gradually, avoiding noisy early estimates while enabling later EEG-specific guidance.Source: B. Progressive Confusion-guided… · p. 6
• Larger off-diagonal confusion values assign greater negative emphasis to easily confused class pairs in continuous class-level supervision.Source: B. Progressive Confusion-guided… · p. 6

• SCI addresses the limitation of standard mixup, whose global coefficient treats EEG representations as structure-agnostic vectors.Source: C. Structure-Consistent Interpolation… · p. 6
• SCI assigns channel–time-dependent interpolation strengths according to structural importance, preserving important EEG components and mixing less critical regions more strongly.Source: C. Structure-Consistent Interpolation… · pp. 6–7
• A parameter-free attribution operator estimates channel-wise and temporal-wise importance from intermediate EEG representations.Source: C. Structure-Consistent Interpolation… · p. 6
• Importance-biased coefficients preserve high-importance activations near their anchors, while lower-importance units allow stronger mixing; an additional constraint prevents degenerate interpolation.Source: C. Structure-Consistent Interpolation… · p. 7
• SCI forms a structured interpolation matrix Λ ∈ R^C×T and mixes two intermediate feature maps element-wise using these channel–time coefficients.Source: C. Structure-Consistent Interpolation… · p. 7
• Unlike standard mixup, SCI preserves important channel–time components while encouraging smoothness and reducing semantic distortion from perturbing them.Source: C. Structure-Consistent Interpolation… · p. 7

Datasets

• ProCA is evaluated on THINGS-EEG2 and Alljoined-1.6M, covering controlled high-fidelity decoding and large-scale accessible EEG acquisition scenarios.THINGS-EEG2 includes 10 subjects and 82,160 EEG trials; Alljoined-1.6M contains approximately 1.6M EEG trials and more subjects.Source: A. Experimental Setting · p. 7

Metrics

• Top-1 and Top-5 accuracy measure whether the ground-truth stimulus ranks first or within the five retrieved candidates.Higher values indicate better EEG–semantic alignment.Source: A. Experimental Setting · p. 7

Subject-Dependent Performance

• ProCA improves subject-dependent decoding by +3.8% Top-1 and +2.5% Top-5 on THINGS-EEG2, with larger gains on Alljoined-1.6M.The reported averages cover 200-, 150-, and 100-way settings across backbones; Alljoined-1.6M reaches +15.7% Top-1 and +7.3% Top-5.Source: B. Comparison with the State-of-the-Art · p. 8

Strict Cross-Subject Transfer

• In strict cross-subject transfer, ProCA improves NeuroBridge from 2.69% to 3.68% average Top-1 and from 10.08% to 12.03% average Top-5.The gains are broadly distributed across source subjects rather than attributable to favorable random seeds.Source: B. Comparison with the State-of-the-Art · pp. 9–10

Continual Adaptation Performance

• ProCA achieves average relative gains of 16.8% Top-1 and 11.6% Top-5 across continual adaptation settings.The comparison uses the highest-performing eligible non-ProCA baseline within each setting and an unweighted macro-average.Source: B. Comparison with the State-of-the-Art · p. 10

Ablation Study

• Removing PCA reduces average Top-1/Top-5 accuracy from 30.6%/62.4% to 29.2%/61.2%, confirming the value of adaptive class-level supervision.Source: C. Ablation Study · p. 11
• Removing SCI reduces average Top-1/Top-5 accuracy to 29.8%/61.8%, showing that structure-aware regularization contributes independently to robust alignment.Source: C. Ablation Study · p. 11
• Naive hard-negative mining and unconstrained mixup perform worse than progressive updating and structure-preserving interpolation, respectively.The comparison attributes the gains to adapting confusion structure and protecting important channel–time components.Source: C. Ablation Study · p. 11
• ProCA remains stable across αmax ∈[0.2, 1.0], typically performs best near αmax = 0.8, and generally reduces Hessian sharpness measures.The loss-landscape analysis uses the maximum Hessian eigenvalue and Hessian trace, where lower values indicate flatter minima.Source: C. Ablation Study · p. 12, Figure 4 · p. 12, Figure 6 · p. 13
• ProCA achieves the best average Top-1 and Top-5 accuracy across 200-, 150-, and 100-way subject-independent retrieval settings, beyond Euclidean Alignment.Source: Figure 5 · p. 12, C. Ablation Study · p. 12

APPENDIX A ADDITIONAL THEORETICAL ANALYSIS

• The theoretical analysis does not establish a direct accuracy guarantee but formalizes optimization bias and semantic distortion as two instability sources.Source: Appendix A · p. 15
• The contrastive objective is conditioned on a semantic relation structure C through the EEG encoder fθ.Source: A. Proof of Gradient Bias under Fixed… · p. 15
• Fixed semantic supervision becomes increasingly mismatched when the ideal EEG-specific relation structure drifts from the static semantic prior.Source: A. Proof of Gradient Bias under Fixed… · p. 15
• The resulting relation mismatch can increase gradient deviation, supporting adaptive contrastive supervision instead of relying solely on static semantic anchors.Source: A. Proof of Gradient Bias under Fixed… · p. 15

B. Proof of the Structure-Weighted Perturbation Bound

• The importance matrix A assigns a non-negative weight Ac,t to each EEG channel and temporal position, with η ensuring a strictly positive-definite weighted norm.Source: B. Proof of the Structure-Weighted… · p. 16
• Under local smoothness in the dual weighted norm, Proposition 4 bounds the sample-level contrastive-loss change caused by perturbation δ.Source: B. Proof of the Structure-Weighted… · p. 16
• Perturbations on high-importance units contribute more strongly to the structure-weighted bound, so robustness perturbations should preserve important channel–time regions.Source: B. Proof of the Structure-Weighted… · p. 17

C. Connection to Structure-Aware Interpolation

• Channel–time-dependent interpolation uses Λ ∈[0, 1]C×T to control preservation strength for each unit from ha.Source: C. Connection to Structure-Aware… · p. 17
• Standard interpolation uses a single global coefficient λ, whereas structure-aware interpolation varies preservation across channel–time units.Source: C. Connection to Structure-Aware… · p. 17
• If Λc,t ≥ λ on structurally important units, their importance-weighted perturbation is no larger than under standard interpolation.Source: C. Connection to Structure-Aware… · p. 17
• Preserving important channel–time units with larger coefficients reduces their contribution to the structure-weighted perturbation norm and the associated semantic-distortion bound.Source: C. Connection to Structure-Aware… · p. 17

Training procedure

• ProCA reformulates EEG visual decoding as progressive contrastive alignment, adapting supervision as EEG representations evolve through successive optimization stages.The contrastive objective remains fixed while the confusion structure changes during training.Source: Appendix B · p. 18
• Training begins with a confusion structure derived from frozen image and text embeddings, providing a stable multimodal prior for early learning.The EEG encoder is optimized using the current confusion structure without additional auxiliary losses or task-specific objectives.Source: Appendix B · p. 18
• ProCA optimizes one contrastive learning objective throughout training, with progression controlled by confusion-structure evolution and structure-aware regularization.This keeps the training procedure conceptually simple while adapting to changing EEG representations.Source: Appendix B · p. 18
• Loss-landscape analysis compares local loss variations around baseline and ProCA parameters for EEG–image retrieval under identical inputs.The analysis uses the similarity objective for EEG–image retrieval, with ℓ2-normalized EEG and image features.Source: A. Additional Implementation Details · p. 18
• The evaluation fixes a randomly sampled mini-batch of 256 EEG–image pairs and uses a random-plane method to construct a two-dimensional parameter subspace.Keeping inputs unchanged isolates loss variation caused by parameter perturbations.Source: A. Additional Implementation Details · p. 18
• Image reconstruction uses retrieval and generation stages, combining a real-stimulus gallery with CLIP features and SDXL-based synthesis guided by EEG semantics.The supplied passages identify the retrieval gallery construction and the SDXL generation backbone.Source: A. Additional Implementation Details · p. 18

Structure-consistent interpolation (SCI)

• SCI estimates channel-wise and temporal importance from intermediate EEG feature maps before generating interpolated representations.The procedure explicitly extracts feature maps and computes importance values before interpolation.Source: §8 · p. 19
• The interpolated feature maps are propagated through the remaining encoder layers before contrastive alignment under the current confusion structure.The resulting representations are used to compute the ProCA loss.Source: §8 · p. 19
• ProCA periodically computes class-wise EEG centroids and EEG-interacted image and text confusion matrices, then fuses the branches to update the confusion structure.These updates provide the progressive supervision used by subsequent alignment steps.Source: §8 · p. 19
• During inference, the trained EEG encoder produces a normalized feature that is compared with gallery image features using cosine similarity to select the best retrieved image.The selected image is subsequently used as a visual prompt for IP-Adapter and SDXL generation.Source: §8 · p. 19
• Qualitative reconstruction comparisons use identical retrieval and generation settings for baseline and ProCA features, while ground-truth images are used only for visualization.The comparison assesses semantic decoding accuracy and visual consistency without using ground-truth images in retrieval or generation.Source: §8 · pp. 19–20
• Strict cross-subject transfer tests whether representations remain transferable under physiological, signal-to-noise, channel-response, and activation-timing differences across unseen subjects.Single-subject training avoids distribution alignment from joint multi-subject training and provides a more stringent generalization test.Source: §8 · p. 20
• Continual adaptation updates a personalized decoder as visual concepts arrive sequentially and evaluates after each stage over all categories observed so far.Replay-based baselines use a fixed buffer capacity of 5,000 samples unless otherwise specified.Source: §8 · p. 20

Feature-space visualization

• ProCA produces more compact and better-separated EEG feature clusters than the baseline under challenging cross-subject shifts and semantic ambiguity.The t-SNE comparison indicates improved representation discriminability and robustness.Source: Appendix C · p. 20, Figure A7 · p. 21

Qualitative reconstruction

• ProCA reconstructions are generally more semantically consistent than baseline outputs, although visually complex or ambiguous stimuli still produce failures.The failures reflect remaining challenges in real-world EEG reconstruction and deployment.Source: Appendix C · p. 20

Failure cases

• High intra-class variability, ambiguous semantic boundaries, visually similar categories, noise, and subject-specific variability remain important sources of reconstruction error.These conditions can cause drift toward correlated but incorrect categories or reduce semantic fidelity.Source: Appendix C · pp. 20, 22
• Even in failure cases, ProCA tends to preserve higher-level semantic attributes better than the baseline, but weak or noisy neural signals remain unresolved.The paper identifies richer neural signals or additional multimodal constraints as possible directions for addressing these limitations.Source: Appendix C · p. 22

Hyperparameter sensitivity

• Across 100-, 150-, and 200-way settings, the best performance occurs at αmax = 0.8 with a transition epoch around 25, while overly early transitions are more harmful than late ones.This pattern supports the stability of the progressive design rather than brittle hyperparameter tuning.Source: Appendix C · pp. 22–23
• 40.98%/74.88% Top-1/Top-5 is achieved in the 100-way setting at αmax = 0.8, compared with 40.50%/74.18% at αmax = 1.0.The modest variation indicates that overly aggressive EEG dominance is detrimental but does not cause large overall swings.Source: Appendix C · p. 22
• 33.78%/67.70% Top-1/Top-5 is achieved in the 150-way setting at αmax = 0.8, while both under-weighting and over-weighting EEG reduce performance.The total variation across αmax remains small in this setting.Source: Appendix C · p. 22

Computational analysis

• 3–4 minutes of additional average training time are required for PCA and SCI, while SCI itself incurs negligible computational cost.The added operations are not a significant computational bottleneck relative to encoder and cross-modal propagation.Source: A. Computational Overhead Discussion · pp. 23–24