Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Dongyang Li; Chen Wei; Shiying Li; Jiachen Zou; Haoyang Qin; Quanying Liu

Source-linked AI summary

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, Haoyang Qin, Quanying Liu

arXiv:2403.07721v7Mar 12, 2024cs.HCeess.SPq-bio.NC

TL;DR

The paper addresses the challenge of decoding and reconstructing visual stimuli from EEG, where performance has lagged fMRI despite EEG’s BCI advantages. It introduces ATM and a two-stage EEG-to-image framework, and reports competitive visual decoding and reconstruction performance with MEG and fMRI.

Problem
EEG visual decoding has been limited by low signal-to-noise ratio, low spatial resolution, inter-subject variability, and a performance gap relative to fMRI.
Method
The framework aligns EEG embeddings with CLIP image embeddings using ATM and generates images through a two-stage diffusion pipeline with high- and low-level visual pathways.
Results
The framework achieves state-of-the-art performance in classification, retrieval, and reconstruction, with performance close to fMRI and compatibility with MEG data.
Takeaways & Limitations
The results support EEG-based zero-shot visual decoding and reconstruction as a feasible approach for non-invasive BCI applications.
Takeaways & Limitations
Performance differs between cross-subject and within-subject settings because of inter-subject EEG differences, and unifying electrode montages across datasets remains challenging.

Abstract

from arXiv · show

How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of visual decoding and reconstruction based on functional Magnetic Resonance Imaging (fMRI). However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for visual decoding based on electroencephalography (EEG). In this study, we present an end-to-end EEG-based visual reconstruction zero-shot framework, consisting of a tailored brain encoder, called the Adaptive Thinking Mapper (ATM), which projects neural signals from different sources into the shared subspace as the clip embedding, and a two-stage multi-pipe EEG-to-image generation strategy. In stage one, EEG is embedded to align the high-level clip embedding, and then the prior diffusion model refines EEG embedding into image priors. A blurry image also decoded from EEG for maintaining the low-level feature. In stage two, we input both the high-level clip embedding, the blurry image and caption from EEG latent to a pre-trained diffusion model. Furthermore, we analyzed the impacts of different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. The experimental results indicate that our EEG-based visual zero-shot framework achieves SOTA performance in classification, retrieval and reconstruction, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. Our code is available at https://github.com/ncclab-sustech/EEG_Image_decode.

1 Introduction

The paper addresses EEG visual decoding and reconstruction as a lower-cost, higher-temporal-resolution alternative to fMRI-based approaches. It introduces an EEG/MEG framework combining a tailored encoder with two-stage image generation to support zero-shot visual tasks.

Motivation: EEG visual decoding is motivated by the need for non-invasive BCI methods that capture dynamic visual processing without fMRI’s cost and limited portability.Neural signals can support visual decoding and reconstruction while providing insight into visual perception.
Research gap: Existing EEG/MEG approaches enable visual decoding but remain substantially behind fMRI, partly because EEG has low signal-to-noise ratio, low spatial resolution, and high inter-subject variability.The paper positions this performance gap as the problem its framework addresses.
Contributions: The proposed framework combines a novel EEG encoder with a two-stage image-generation strategy for zero-shot classification, retrieval, and reconstruction.The contributions include applying the framework across EEG encoder architectures and extracting high- and low-level visual features for reconstruction.
Contributions: The framework reports state-of-the-art performance across three downstream visual decoding tasks and reconstructs images using less than 500ms of EEG.The paper also reports competitive performance with MEG and fMRI and compatibility with MEG data.

2 Method

The method aligns EEG embeddings with CLIP image representations using the ATM encoder, then generates images through a two-stage diffusion pipeline. Its architecture incorporates channel attention and temporal-spatial modeling, while the framework supports zero-shot decoding tasks and EEG/MEG evaluation.

EEG-image alignment: EEG embeddings are trained with paired EEG-image data to align with CLIP image embeddings, enabling zero-shot classification, retrieval, and reconstruction at inference.The encoder maps E ∈ R^{N × C × T} to EEG embeddings Z_E ∈ R^{N × F}, aligned with image embeddings Z_I ∈ R^{N × F}.
ATM encoder: ATM embeds EEG sequences as tokens, applies channel-wise attention, aggregates temporal-spatial information, and projects the result through an MLP.The design preserves channel relationships and uses temporal-spatial convolution to reduce overfitting; its convolution component is plug-and-play across EEG/MEG datasets.
Two-stage generation: The two-stage generator first predicts CLIP image embeddings from EEG with a prior diffusion model, then uses those embeddings in a pretrained SDXL and IP-Adapter generator to sample images.The joint formulation separates p(z_I|z_E), the prior stage, from p(I|z_I), the image-generation stage.
Training objective: The overall loss combines CLIP alignment and MSE consistency, with λ controlling their relative contributions for generation tasks.Classification and retrieval use CLIP loss alone, whereas generation adds MSE loss.
Evaluation: Across THINGS-EEG and THINGS-MEG, the framework compares nine encoders and reports the highest visual decoding performance among competing encoders.The comparison includes within-subject and cross-subject EEG settings and corresponding MEG evaluations.

3 Experiments

Experiments evaluate EEG embeddings for classification, retrieval, and image generation, alongside ATM ablations, reconstruction metrics, and temporal analyses.

3.2 EEG Decoding Performance: Our method outperforms other methods in average in-subject EEG image classification accuracy.
3.2 EEG Decoding Performance: Top-1 retrieval accuracy is significantly improved over previous models, while retrieved top-5 images retain high similarity to the original images.
ATM Ablation Study: ATM is evaluated through ablations of its MLP projector, Temporal-Spatial convolution, and channel-wise attention components.
3.3 Image Generation Performance: Generated images show high semantic similarity to seen images and diverse low-level visual features that can be manipulated through guidance scale.
3.4 Temporal Analysis: Within 500 ms after stimulus onset, classification accuracy reaches an upper limit of about 30%, while generated-image similarity increases with longer windows and becomes reliable after 500 ms.
Brain-Region Analysis: Using all brain regions is optimal for both retrieval and generation tasks.

4 Related Works

Related work frames EEG as a portable, high-temporal-resolution alternative to fMRI, while prior EEG studies still face decoding-performance and generalization limitations.

Visual Decoding using Neural Signals: EEG offers portability and high temporal resolution for practical visual decoding, whereas fMRI is less suitable for high-speed BCI applications.
Neural Decoding using EEG/MEG Data: Existing neural encoders include Temporal-Spatial modules, lightweight convolutional networks such as EEGNet and ShallowNet, and contrastive-learning projection methods.
Neural Decoding using EEG/MEG Data: Prior MEG and EEG studies made visual decoding possible through multimodal alignment, but their performance remained inferior to fMRI.
Limitations of Previous Studies: Previous EEG studies focused primarily on understanding visual perception rather than maximizing decoding performance, often using linear models for restricted prediction or classification settings.

5 Discussion and Conclusion

The paper presents EEG-based zero-shot visual decoding and reconstruction, reports performance close to fMRI with EEG, and analyzes temporal, spatial, and cross-subject effects.

Discussion and Conclusion: The framework is presented as the first EEG-based approach enabling zero-shot image classification, retrieval, and reconstruction, with ATM achieving state-of-the-art performance across these tasks.
Discussion and Conclusion: The two-stage EEG-guided reconstruction framework achieves performance close to fMRI using EEG and is compatible with MEG.
Neuroscience Insights: EEG visual information is predominantly contained within 200–400 ms, whereas MEG visual information lasts up to 800 ms; occipital and parietal areas mainly encode it.
Future Directions: Cross-subject and within-subject performance differs substantially, reflecting inter-subject EEG variability and motivating more flexible encoders, transfer learning, and meta-learning.

Supplementary Material:

The supplementary material includes the paper title.

Supplementary Material:: The paper is titled “Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion.”

A.1 EEG dataset

Experiments used THINGS-EEG for zero-shot visual decoding and evaluated classification, retrieval, and generation, with MEG data used to test ATM’s modality versatility.

THINGS-EEG contains recordings from 10 subjects across repeated visual experiments, with 16,540 training image conditions and 200 testing images.
MEG experiments use 271-channel recordings from four subjects and construct a zero-shot task by excluding 200 testing concepts from training.
Zero-shot classification uses CLIP text embeddings and evaluates Top-1 and Top-5 accuracy, while retrieval ranks images associated with EEG signals.
Generation evaluates semantic alignment by generating 10 images per test condition and measuring Top-1 and Top-5 retrieval accuracy against original images.

B.4 Model configuration

The model combines ATM design choices with conditional diffusion and a two-stage generation pipeline, using EEG to guide high-level image priors and final image synthesis.

ATM configuration: Preserving token dimensions for Temporal-Spatial convolution produced the strongest retrieval performance among tested ATM token strategies.
ATM configuration: Linear layers outperformed 1 × 1 convolutions, temporal FFNs were superior, and placing convolutions after the Transformer encoder performed best.
Training details: The training setup averages 80 repeated test instances to improve EEG signal-to-noise ratio while retaining all training repetitions.
Stage I: The EEG-conditioned diffusion stage learns the CLIP-embedding distribution p(z_I|z_E) and uses classifier-free guidance to balance conditioning fidelity and output diversity.
Stage II: The two-stage strategy uses prior-diffusion image embeddings for synthesis and reconstructs semantic and low-level visual features more accurately than direct one-stage EEG generation.
Stage II: The second stage synthesizes images from EEG-derived CLIP embeddings using pretrained SDXL and IP-Adapter models.

D Performance comparison

Performance is assessed with low- and high-level reconstruction metrics and cross-modality comparisons, with ATM-based EEG reconstruction reported as competitive with MEG and fMRI.

ATM-based EEG reconstruction on the THINGS dataset achieved performance over MEG reconstruction, according to the study’s comparison results.
The comparison uses PixCorr, SSIM, SwAV, and two-way identification with AlexNet, Inception, and CLIP features.
Across visual stimulus tasks, the reported results reached the same or better performance than fMRI and MEG.
Representational analysis clustered image embeddings within categories and examined their similarity to EEG-derived representations for Subject 8.

F Concept analysis

Concept analysis maps EEG-derived representations into a 42-dimensional concept space and compares them with image-derived concept representations.

The concept encoder converts CLIP image embeddings into 42-dimensional vectors, with each dimension representing a distinct concept.
ATM-projected EEG from 200 test categories is passed through the frozen concept encoder to obtain EEG-derived concept embeddings.
The analysis compares EEG, image, and text representations using similarity matrices, t-SNE latent-space visualizations, and concept-embedding matrices.

G Additional images results

Additional examples show that EEG-guided reconstructions can preserve original image semantics and, in stronger cases, low-level visual features. Results are organized by alignment quality and include subject-specific batches and retrieval outputs.

Additional image examples: Best reconstructions preserve both the original image semantics and low-level visual features.Medium reconstructions retain semantics with altered visual details, whereas worst reconstructions alter both semantic and low-level features.
Additional image examples: Figure 14 groups 12 generated images each into best, median, and worst alignment categories against the viewed images.The ranking uses cosine similarity between generated-image and original-image CLIP embeddings.

H Additional evaluation results

Additional evaluations examine subject-specific reconstructions, temporal windows, and modality performance. The reported analyses identify a 200–250 ms EEG endpoint peak and show that ATM outperforms other EEG embeddings in retrieval.

Accuracy for time windows: A clear accuracy peak occurs for EEG windows ending around 200–250 ms after image onset across embeddings.The study reports no second peak on THINGS-EEG, unlike a cited prior result, and notes that later high-level visual processing may still contribute.
Accuracy for time windows: Reconstructions are semantically incorrect during the first 50–200 ms, become basically correct after 250 ms, and stabilize after 500 ms.The authors relate this timing pattern to the absence of additional visual response while cautioning that later EEG may still support high-level feature decoding.
MEG evaluation: MEG accuracy is evaluated with growing windows from 50 ms to 1000 ms and sliding 100 ms windows using Top-1 and Top-5 accuracy.These figures extend the temporal-window analysis to the MEG modality.
Additional evaluation results: ATM outperformed all other EEG embedding methods in zero-shot retrieval on the THINGS-EEG dataset.Table 8 reports in-subject and cross-subject performance across 2-way, 4-way, 10-way, and 200-way Top-1 and Top-5 accuracy.