PaperlayerView exact revision on arXiv

Source-linked AI summary

EEG-based Visual Retrieval and Reconstruction: From Neurally Visible Optimal Layer to Hierarchical Diffusion Generation

Minyi Wang, Zhenqin Wu, Rihui Li

arXiv:2609.02582v1cs.HC

TL;DR

EEG visual decoding is challenged by noisy recordings and fixed final-layer alignment that overlooks varying neural visibility across visual representations. The paper selects subject-specific intermediate CLIP layers, fuses them for retrieval, and uses them as a bridge for diffusion-based reconstruction. On THINGS-EEG, CSLS raises retrieval to 86.4% Top-1 accuracy, while two-stage reconstruction improves both low- and high-level metrics.

  • Problem

    Most EEG visual decoding methods align signals to a fixed final semantic layer despite hierarchical visual processing and subject-specific differences in neural visibility.

  • Method

    The framework selects subject-specific intermediate CLIP layers, fuses multi-NVOL features for contrastive retrieval with CSLS, and reconstructs NVOL features before mapping them to CLIP space for generation.

  • Results

    86.4% Top-1 accuracy was achieved with CSLS, while two-stage NVOL-to-semantic reconstruction improved PixCorr by 10.0% and SSIM by 5.4%.

  • Takeaways & Limitations

    Intermediate visual representations provide a shared alignment anchor for EEG retrieval and reconstruction, supporting more effective and interpretable neural decoding.

  • Takeaways & Limitations

    Experiments used THINGS-EEG, so performance on other datasets and acquisition setups remains to be tested.

Abstract

from arXiv · show

Decoding visual perception from electroencephalography (EEG) is important for non-invasive brain-computer interfaces (BCIs). However, most existing visual decoding pipelines directly align EEG features with semantic features from pretrained vision models. Those EEG signals carry information at more than one level and this practice disregards the varying neural visibility of different visual components in EEG signals, leading to cross modal mismatches and incomplete information use. In this work, we address this limitation through layer-wise contrastive learning. For each subject, the intermediate CLIP layer that maximizes retrieval performance is selected as the Neural Visibility Optimal Layer (NVOL). Built on NVOL, a hierarchical framework couples retrieval and generation through a shared intermediate representation. The retrieval branch fuses multi-NVOL features, aligns them to image embeddings via contrastive learning, and applies cross-domain similarity local scaling (CSLS) at test time to mitigate hubness. The generation branch reconstructs subject-specific NVOL features from EEG using a conditional diffusion prior, maps them to CLIP space through a lightweight adapter, and drives a pretrained Stable Diffusion XL model. Experimental validation on THINGS-EEG showed that, NVOL-based retrieval achieves 78.1\% mean Top-1 accuracy in 200-way retrieval, rising to 86.4\% with CSLS. Two-stage NVOL-to-semantic reconstruction also outperforms single-stage final-layer diffusion on semantic and structural metrics. By aligning EEG with layer-wise neural visibility rather than fixed high-level semantics, the proposed framework improves both retrieval accuracy and image reconstruction in EEG-based visual decoding.

I. INTRODUCTION

EEG-based visual decoding can support brain–computer interfaces, but noisy, coarse recordings and fixed final-layer targets limit how visual information is aligned and reconstructed. The paper addresses this with subject-specific intermediate-layer alignment and a unified retrieval–generation pipeline.

  • EEG offers portable, low-cost, high-temporal-resolution recordings, but noisy and spatially coarse signals make stable visual representation extraction difficult.These properties motivate EEG-based decoding while defining its methodological challenge.
  • Most methods align EEG with final-layer semantic embeddings, despite evidence that visual features have different neural visibility across the hierarchy.Prior work indicates global structure is relatively stable in scalp recordings, whereas high-level semantics have lower neural visibility.
  • High-dimensional EEG–image embeddings can exhibit hubness, while unstable retrieval features may also impair reconstruction quality.These issues motivate CSLS calibration and a shared representation for retrieval and generation.
  • The framework selects subject-specific intermediate CLIP layers with the strongest alignment and fuses multiple neurally visible layers instead of forcing one target.The selected alignment depth varies across participants but remains in intermediate layers.
  • Retrieval uses contrastive EEG–image alignment with CSLS, while generation reconstructs neurally visible features before mapping them to the final CLIP layer for SDXL.Both branches share a hierarchical EEG-to-NVOL-to-final-layer pipeline.

B. Visual foundation models and brain-feature alignment

The paper replaces fixed final-layer brain alignment with intermediate CLIP representations selected for neural visibility, then fuses them into a shared target for retrieval and reconstruction. The framework combines contrastive alignment, hierarchical feature fusion, and EEG encoding through ATM.

  • Visual foundation models and brain-feature alignment: Final-layer semantic alignment can under-use intermediate features that are more neurally observable, while fixed target layers may not suit all subjects.This motivates layer-wise, participant-sensitive alignment.
  • Visual foundation models and brain-feature alignment: The pipeline aligns EEG with hierarchical CLIP features for retrieval and uses those intermediate representations as a bridge to image reconstruction.Retrieval and generation therefore operate through a shared intermediate feature space.
  • Visual foundation models and brain-feature alignment: The ATM encoder combines temporal–spatial convolution with Transformer-based global integration and outputs a normalized 1024-dimensional EEG embedding.This embedding is used by the visual decoding framework.
  • NVOL Selection and Multi-layer Fusion: Intermediate CLIP layers 10, 12, and 14 were selected as NVOLs after screening layers 4–32, because peak retrieval alignment occurred in intermediate layers.The CLIP ViT-H/14 encoder has 32 Transformer blocks and hidden dimension d = 1280.
  • NVOL Selection and Multi-layer Fusion: Token-wise features from each selected layer are mean-pooled, projected, normalized, and fused with learnable softmax weights into a shared visual embedding.The fused embedding serves as the contrastive retrieval target and the NVOL supervision target for generation.
  • Visual foundation models and brain-feature alignment: The framework trains matched EEG and image embeddings through contrastive learning, then applies CSLS during inference to reduce cross-modal hubness.CSLS is used as post-hoc calibration for zero-shot retrieval.

3) Contrastive Training and CSLS Retrieval:

The retrieval branch evaluates EEG queries against image categories with zero-shot similarity, then applies CSLS to recalibrate cross-domain scores and reduce hubness.

  • EEG queries are evaluated in an N-way zero-shot classification task using ranked similarity scores over held-out candidate classes.
  • Hubness is quantified with the Gini coefficient from k-NN retrieval statistics, where values near 0 indicate uniform retrieval and values near 1 indicate strong skew.
  • CSLS calibrates EEG-to-image retrieval by subtracting row-wise and column-wise neighborhood averages from twice the raw similarity.
  • The hubness null model samples 200 queries over 200 categories and generates 20,000 single-subject and 100,000 group-mean null Gini coefficients.
  • Hubness significance is assessed with one-sided Monte Carlo p-values, with 30 tests corrected using the Benjamini–Hochberg false discovery rate procedure.

C. Generation Branch

The generation branch uses a two-stage pipeline that first reconstructs intermediate NVOL features from EEG and then maps them into final-layer CLIP semantic space for SDXL conditioning.

  • C. Generation Branch: The pipeline first applies an EEG-to-NVOL diffusion prior to reconstruct intermediate CLIP features.This intermediate reconstruction is the first stage of the generative process.
  • C. Generation Branch: An NVOL-to-semantic adapter then maps reconstructed NVOL features to final-layer CLIP embeddings.The adapter supplies the semantic representation used for downstream conditioning.
  • C. Generation Branch: The resulting final-layer CLIP embeddings condition a pretrained Stable Diffusion XL model.The two-stage design avoids directly mapping noisy EEG into final CLIP semantic space.

1) EEG-to-NVOL Diffusion Prior:

The EEG-to-NVOL diffusion prior models intermediate CLIP features conditionally from EEG and supplements noise prediction with latent, angular, relational, and anti-collapse supervision.

  • 1) EEG-to-NVOL Diffusion Prior:: A conditional DDPM models intermediate CLIP features v from EEG embedding e.At diffusion step t, the denoiser Gψ predicts the noise added to the noisy latent vt.
  • 1) EEG-to-NVOL Diffusion Prior:: Noise prediction is supplemented by Euclidean and angular supervision on the reconstructed clean latent.The clean latent is recovered from the predicted noise using the DDPM identity, while cosine similarity is computed after L2 normalization.
  • 1) EEG-to-NVOL Diffusion Prior:: Pairwise geometry loss preserves similarity patterns among related and unrelated samples within each mini-batch.It matches Gram matrices of predicted and target row-normalized NVOL embeddings, retaining target-space relational structure.
  • 1) EEG-to-NVOL Diffusion Prior:: Variance and covariance regularizers reduce dimensional collapse in predicted NVOL embeddings.The regularizers are adopted from VICReg and use a variance floor γ.
  • 1) EEG-to-NVOL Diffusion Prior:: The total prior objective combines noise prediction, point-wise, angular, pairwise geometry, and anti-collapse terms.These components jointly constrain both individual reconstructed features and their batchwise structure.
  • 1) EEG-to-NVOL Diffusion Prior:: The adapter maps layer-12 NVOL features to final-layer CLIP embeddings and uses a domain-consistency regularizer.Training uses real intermediate and final-layer features, whereas inference uses NVOL features reconstructed by the EEG-conditional diffusion prior.

2) NVOL-to-Semantic Adapter:

The adapter aligns real and diffusion-generated NVOL features to final-layer CLIP embeddings, combining reconstruction, smoothness, cosine, contrastive, generation, and domain-consistency objectives.

  • The adapter aligns predicted features with ground-truth final-layer CLIP embeddings using a composite alignment objective.The objective includes reconstruction, smoothness, cosine, and contrastive terms.
  • Real layer-12 features and diffusion-generated NVOL features are aligned against the same final-layer targets.The generated features enter through the generation path before adapter alignment.
  • Domain consistency regularization encourages agreement between real and generated representations during adapter training.The domain-consistency term follows view-consistency regularization.
  • Training uses λ_mse = 1.0, λ_smooth = 0.5, λ_cos = 0.5, λ_ctr = 0.2, λ_gen = 1.0, and λ_domain = 0.1.Only adapter parameters are updated with AdamW and cosine annealing.

D. Inference pipeline

The inference pipeline converts temporal–channel EEG embeddings into intermediate CLIP features, maps them to final-layer embeddings, and uses them to condition pretrained SDXL.

  • Temporal–channel EEG embeddings pass through a diffusion prior to reconstruct intermediate CLIP features.
  • A semantic adapter maps reconstructed intermediate features to final-layer CLIP embeddings for image synthesis.
  • SDXL receives the decoded embeddings through IP-Adapter, which injects image-prompt information via cross-attention without fine-tuning the backbone.

E. Time, Spatial and Frequency analysis

The evaluation examines EEG retrieval across temporal, spatial, and spectral dimensions on THINGS-EEG, while hubness analysis compares nearest-neighbor imbalance across representation layers.

  • Time, Spatial and Frequency analysis: Temporal analysis averages training trials within each subject and computes topographic maps at 100-ms intervals.Within-trial averaging is used to reduce masking effects associated with rapid serial visual presentation.
  • Datasets and evaluation: THINGS-EEG includes 10 participants, 1,654 training categories, and a held-out test set of 200 categories.Each training category has 10 images with four presentations per image; the test set uses one image per category across 80 trials per participant.
  • Hubness analysis: Figure 3 plots hubness Gini coefficients against representation layer for k = 5 and k = 10, including separate curves for subjects sub-01 to sub-10.Higher Gini values indicate more severe nearest-neighbor imbalance.

B. Implementation Details

The implementation uses CLIP ViT-H/14 with ATM EEG encoding and evaluates retrieval on THINGS-EEG, where intermediate-layer alignment, hubness analysis, and CSLS-based fusion support strong within-subject performance.

  • Implementation Details: The visual encoder uses pretrained CLIP ViT-H/14, while ATM serves as the EEG encoder under intra-subject and inter-subject settings.Experiments use PyTorch and a single NVIDIA H800 GPU.
  • Layer-wise retrieval: Peak retrieval accuracy occurs at intermediate CLIP layers such as L10, L12, and L14 for every subject.Both shallow layers emphasizing edges and textures and deepest layers emphasizing abstract semantics yield lower accuracy.
  • Hubness analysis: Hubness Gini index increases with CLIP layer depth, and every layer exceeds random-retrieval null thresholds with q < 1 × 10^-5 after correction.Intermediate layers have lower hubness and more uniform neighbor distributions.
  • Similarity analysis: Figure 4 compares category-organized EEG representational similarity matrices with ground-truth labels and top-5 predictions.
  • Retrieval Accuracy: 86.4% within-subject Top-1 and 98.5% Top-5 accuracy are achieved with multilayer fusion and CSLS post-processing.Between-subject retrieval reaches 19.9% Top-1 and 44.1% Top-5 accuracy under the same configuration.

D. Semantic Analysis

Semantic analyses show that EEG retrieval depends on temporally localized, posterior and frequency-specific signals, with intermediate CLIP layers providing the strongest alignment.

  • Semantic structure: Top-5 retrievals preserved category structure, with basil, bun, and antelope returning mainly plants, food, and animals, respectively.The corresponding representational similarity matrix showed category-level block structure.
  • Spatial analysis: Parietal and occipital masking caused substantial Top-1 declines, while Oz and surrounding occipital channels showed the greatest channel importance.Temporal and central contributions were smaller than posterior contributions.
  • Frequency analysis: Theta removal produced the largest retrieval decrease, peaking at intermediate CLIP layers L8–L14; gamma removal had little effect.Broadband removal also reduced accuracy across most layers.
  • Channel reference: Table IV provides the electrode names corresponding to the channel indices used in the single-channel analysis.This mapping supports interpretation of the channel-level results.

F. Reconstruction Performance

The hierarchical reconstruction framework uses intermediate EEG-visible representations as a bridge to semantic image generation and outperforms single-stage diffusion across low- and high-level metrics, while retaining dataset and modality limitations.

  • Reconstruction performance: TS-Diff achieved the best low- and high-level reconstruction metrics, reaching PixCorr 0.176 and SSIM 0.366.These values were 10.0% above ATM for PixCorr and 5.4% above CogCap for SSIM.
  • Hierarchical reconstruction: The pipeline first reconstructs an intermediate CLIP representation from EEG, then adapts it into the final semantic embedding for image synthesis.This two-stage design is contrasted with directly mapping EEG to final-layer features.
  • Shared representation: The framework aligns EEG with visual representations through contrastive learning and uses intermediate layers as a shared retrieval–generation feature space.Layer-wise results identified intermediate layers as the strongest alignment depth.
  • Retrieval outcome: Combining multi-layer interaction with CSLS raised Top-1 retrieval accuracy to 86.4%.The reported result was significantly higher than existing methods in the cited discussion.
  • Neural evidence: Intermediate-layer alignment was associated with posterior EEG contributions and a dominant [0.1, 0.6] s post-stimulus interval.The first 0.1 s contributed disproportionately to the decodable signal.
  • Limitations: The study was limited to THINGS-EEG, did not model cross-subject alignment variability, and found that reconstructions still lacked fine texture and instance-level detail.Transfer to other datasets, acquisition setups, and modalities remains to be tested.
  • Conclusion: The authors conclude that hierarchically organized visual representations may support more effective and interpretable neural decoding.This conclusion is framed within the studied EEG retrieval and reconstruction setting.
Loading 2609.02582v1…