Source-linked AI summary
Decoding Natural Images from EEG for Object Recognition
Yonghao Song, Bingchuan Liu, Xiang Li, Nanlin Shi, Yijun Wang, Xiaorong Gao
TL;DR
Natural-image object recognition from EEG is attractive because EEG is convenient and portable, but its low signal-to-noise ratio and limited prior evidence make reliable decoding difficult. The paper introduces a self-supervised framework that aligns image and EEG representations through contrastive learning, achieving above-chance zero-shot recognition while examining biological plausibility across multiple dimensions. The findings support the feasibility of EEG-based image decoding and offer insights for neural decoding and brain-computer interfaces.
Problem
EEG is convenient and portable for natural-image decoding, but low signal-to-noise ratio and limited prior work leave reliable object recognition and biological plausibility insufficiently established.
Method
The framework separately encodes paired images and EEG signals, aligns their representations with contrastive learning, and uses attention modules to capture spatial correlations among EEG channels.
Results
The framework achieves zero-shot object-recognition performance above chance and provides biological evidence across temporal, spatial, spectral, and semantic aspects.
Takeaways & Limitations
The results demonstrate the feasibility of investigating natural-image information from EEG and provide insights relevant to neural decoding and brain-computer interfaces.
Takeaways & Limitations
Stable responses required multiple repetitions, possibly because the 100 ms stimulus duration made responses susceptible to being missed or disrupted by preceding stimuli.
Abstract
from arXiv · showhide
Electroencephalography (EEG) signals, known for convenient non-invasive acquisition but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Contrastive learning aligns these two modalities by constraining their similarity. With the framework, we attain significantly above-chance results on a comprehensive EEG-image dataset, achieving a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in challenging 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. The code will be released on https://github.com/eeyhsong/NICE-EEG.
1 INTRODUCTION
The paper addresses limitations of EEG-based natural-image decoding by using self-supervised cross-modal learning for object recognition. It evaluates zero-shot performance while examining the temporal, spatial, spectral, and semantic plausibility of the decoded brain activity.
- Motivation: EEG offers high temporal resolution, low cost, and portability, but low signal-to-noise ratio and flawed block designs limit stimulus-specific decoding.Existing studies also often used few categories or insufficient data, leaving multi-class recognition and biological plausibility unresolved.
- Approach: Self-supervised learning and contrastive alignment provide a route to learn intrinsic EEG representations without relying solely on predefined labels.The paper connects image stimuli and EEG responses by constraining their cross-modal similarity.
- Approach: The framework uses separate image and EEG encoders with contrastive learning to support zero-shot recognition of previously unseen image categories.Its contributions include applying this framework to EEG-based object recognition and reporting strong zero-shot performance on rich datasets.
- Biological analysis: Extensive experiments examine object recognition across temporal, spatial, spectral, and semantic aspects to assess biological plausibility.The authors state that these results are consistent with established neuroscientific knowledge.
- Spatial modeling: Self-attention and graph-attention modules capture spatial correlations among EEG channels and emphasize brain regions relevant to object recognition.The modules are presented as plug-and-play additions to the EEG encoder.
2 RELATED WORKS
Natural-image decoding seeks semantic information from brain activity, but accurate and rapid decoding remains difficult. The paper’s framework combines image and EEG encoders, contrastive matching, and attention-based EEG spatial modeling for object recognition.
- Related challenges: Accurate and rapid decoding of semantic information in natural images remains challenging despite progress on steady-state visual stimuli.The related work contrasts this challenge with fMRI-based estimation of semantic and shape information.
- Framework: The framework trains image and EEG encoders by increasing similarity for matched image-EEG pairs and decreasing similarity for unmatched pairs.At test time, EEG signals are matched against templates formed from a few unseen images for each target concept.
- EEG encoder: The EEG encoder uses temporal-spatial convolution with self-attention and graph-attention modules to reveal spatial features of brain activity.A linear layer projects the resulting feature dimension.
3 METHODS
NICE is a self-supervised framework that aligns image and EEG representations through contrastive learning for zero-shot natural-image decoding. Its EEG encoder combines temporal-spatial convolution with attention modules that model electrode correlations.
- Framework: NICE feeds paired images and EEG responses into separate image and EEG encoders, then aligns their representations with contrastive learning.Matched pairs are encouraged to have higher similarity than unmatched pairs, enabling shared cross-modal representations.
- EEG Encoder: The EEG encoder uses temporal-spatial convolution to extract temporal and spatial features from trials arranged as electrode channels by time samples.Temporal convolution precedes pooling, followed by convolution across electrodes; a linear projector matches the image-encoder feature size.
- Cross-Modal Features: Pre-trained image encoders provide semantically discriminative image features, while the EEG encoder learns representations aligned to them.The framework can use image feature extractors such as CLIP, ViT, or ResNet, reducing reliance on collecting large numbers of brain responses.
- Contrastive Objective: Normalized image and EEG features are compared with scaled pairwise cosine similarity and optimized using a symmetric InfoNCE loss.The loss averages image-to-EEG and EEG-to-image cross-entropy objectives over matched batch indices.
- Spatial Modules: Self-attention and graph attention act as spatial filters that model correlations among EEG electrode channels.Self-attention evaluates channel relationships, while graph attention updates each electrode using information from connected electrodes and attention coefficients.
4 EXPERIMENTS AND RESULTS
Experiments evaluate NICE on a 200-way zero-shot EEG-image task, compare encoder choices and attention variants, and analyze the temporal, spatial, spectral, and semantic information supporting decoding. Results show above-chance classification and biologically plausible EEG patterns, with performance depending on data size and repetition.
- Experiment setup: The dataset uses ten participants, 1654 training concepts with repeated images, and 200 test concepts presented across 80 repetitions.EEG trials were epoched after stimulus onset, baseline-corrected, down-sampled, noise-normalized, and repetition-averaged to improve signal-to-noise ratio.
- Overall performance: 13.8% top-1 and 39.5% top-5 accuracy were achieved by NICE in subject-dependent 200-way zero-shot classification, exceeding BraVL by 8.0% and 22.0%, respectively.The chance level was 0.5%; self-attention and graph attention improved top-1 accuracy by 0.9% and 1.8%, respectively.
- Encoder comparison: TSConv outperformed ShallowNet, DeepNet, and Conformer by 5.9%, 3.9%, and 2.7% average top-1 accuracy, respectively.Its improvement over EEGNet was 0.7% and was not statistically significant.
- Encoder comparison: CLIP produced the highest image-encoder results, while ViT was 1.7% lower and ResNet was 6.9% lower in average top-1 accuracy.Pre-trained ViT exceeded its non-pre-trained counterpart by 4.2%, whereas pre-trained ResNet decreased by 1.9%.
- Temporal dynamics: The informative EEG response window was 100-600 ms, with temporal-cortex activity and limited information in the initial 100 ms.Activity after 600 ms negatively affected performance, possibly because of noise from other stimuli.
- Spatial and spectral dynamics: Removing occipital electrodes reduced average accuracy by 3.8%, while removing temporal and parietal electrodes reduced it by 1.9% and 1.6%.The main frequency components were below 30 Hz, and theta and beta rhythms showed effective performance.
5 DISCUSSION AND CONCLUSION
The paper concludes that self-supervised EEG decoding can support natural-image object recognition in zero-shot tasks, with biological evidence across multiple aspects of brain activity. It also identifies repeated presentations and gamma-band information as unresolved limitations.
- Limitations: Stable responses require multiple repetitions, partly because the brief 100 ms stimulus may be missed or disrupted by neighboring stimuli.The authors plan to identify a more optimal stimulus-presentation window.
- Limitations: The study has not yet captured useful information from the gamma band, which remains an area for further investigation.The discussion notes that gamma responses can be modulated by cognitive processing such as attention and memory.
- Conclusion: The framework achieves zero-shot natural-image object recognition from EEG while providing rich biological evidence.The authors present this as a source of inspiration for practical brain-computer interfaces.
A.1 SANITY CHECK
The 200-way zero-shot evaluation establishes an experimental chance level close to the ideal 0.5%, while NICE performs significantly better.
- A.1 SANITY CHECK: 0.5% is the ideal chance level for the 200-way zero-shot task, and the experimentally measured chance is close to it.
- A.1 SANITY CHECK: NICE significantly outperforms the experimental chance level (p < 0.001).
A.2 BETTER ENCODERS
Replacing the image encoder with EVA-CLIP pretrained on LAION-2B improves both top-1 and top-5 accuracy, supporting the use of stronger image representations.
- A.2 BETTER ENCODERS: EVA-CLIP pretrained on LAION-2B improves overall top-1 accuracy (p < 0.001) and top-5 accuracy (p < 0.01).
- A.2 BETTER ENCODERS: The results indicate that NICE can use other well-designed EEG or image encoders.
A.3 COMPARISON OF PRE-TRAINING
The study compares pretrained and jointly trained image encoders, finding that pretrained encoders support efficient EEG-encoder training whereas non-pretrained training generalizes less effectively.
- A.3 COMPARISON OF PRE-TRAINING: Pretrained CLIP and ViT image encoders provide feature extraction from large-scale training and reduce computational demands.
- A.3 COMPARISON OF PRE-TRAINING: One 8GB GPU can train an EEG encoder for one subject in 5 minutes with the pretrained setup.
- A.3 COMPARISON OF PRE-TRAINING: Non-pretrained training jointly optimizes randomly initialized image and EEG encoders using ViT-B/16 and ResNet-50 examples.
- A.3 COMPARISON OF PRE-TRAINING: Non-pretrained training shows rapidly decreasing training loss but weaker validation-loss decline, indicating limited generalization.
A.4 SPATIAL MODULE INTERPRETATION
Attention visualizations indicate that the spatial modules reduce prefrontal influence and focus on temporal and occipital regions associated with visual processing and object recognition.
- A.4 SPATIAL MODULE INTERPRETATION: Both self-attention and graph-attention modules alleviate the influence of prefrontal regions across subjects.
- A.4 SPATIAL MODULE INTERPRETATION: Graph attention focuses on occipital regions, while self-attention captures temporal-region activity across multiple subjects.
- A.4 SPATIAL MODULE INTERPRETATION: The visualizations use Grad-CAM and attention weights for individual subjects and subject averages.
- A.4 SPATIAL MODULE INTERPRETATION: Occipital and temporal EEG activity is associated with visual processing and object recognition, providing implicit evidence for EEG-based object recognition.
A.5 PARAMETER SENSITIVITY
Parameter and auxiliary analyses show that contrastive-learning temperature affects results, residual connections particularly benefit the GA spatial module, and encoder design choices materially influence accuracy. MEG analyses also reproduce above-chance zero-shot decoding, while feature visualizations support separation across semantic categories.
- Parameter sensitivity: Temperature adjustments significantly influence contrastive-learning results.The temperature parameter is optimized as a log-parameterized multiplicative scalar, and comparisons are reported in Table 7.
- MEG experiments: 10.1% top-1 and 28.4% top-5 accuracy were achieved in 200-way zero-shot MEG classification.The MEG experiments also include a retrieval task using stimulus images as matching templates.
- MEG experiments: Temporal, spatial, and spectral MEG analyses produced results similar to EEG analyses, although temporal responses changed with longer stimuli.The MEG dataset used longer 500 ms stimuli, followed by a blank screen of 1000±200 ms.
- Spatial modules: Residual connections significantly improved GA top-1 accuracy, whereas their effect on SA was not significant.The residual connections were introduced to stabilize training for the spatial modules.
- EEG encoder comparison: Kernel size, pooling order, and activation function each significantly affected top-1 accuracy relative to TSConv.TSConv uses a concise architecture of temporal and spatial convolutional layers, and the comparison is summarized in Table 10.
- Feature distribution: t-SNE visualizations of animal, food, vehicle, and tool categories illustrate training–test feature distributions associated with zero-shot performance.The visualization compares visual and EEG features separately across training and test sets.