PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features

Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, Joseph Schmidt, Mubarak Shah

arXiv:1810.10974v2cs.CVcs.LGq-bio.NC

TL;DR

The paper addresses how human visual representations can be decoded from neural activity and related to computational visual features. It trains EEG and image encoders in a siamese joint embedding, then uses the learned space for classification and saliency detection. The approach improves EEG classification and saliency performance while supporting brain-guided analysis of visual features.

  • Problem

    Existing brain-decoding methods cannot monitor every neuron simultaneously, while prior multimodal approaches can tie learned representations to proxy classification tasks.

  • Method

    The method trains siamese EEG and image encoders with a triplet loss that brings matching representations together and separates mismatched pairs.

  • Results

    60.4% EEG classification accuracy is achieved after adding visual features, compared with 48.1% using the EEG encoder alone; the saliency detector also outperforms baseline detectors.

  • Takeaways & Limitations

    The learned joint representations can drive image classification and saliency detection from neural activity while mapping EEG signals to salient visual features.

  • Takeaways & Limitations

    The approach addresses the limitation of representations tied to a classification task, and one cited prior result was inflated by incorrect frequency filtering that retained EEG drift.

Abstract

from arXiv · show

This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that maximizes a compatibility measure between visual features and brain representations. We then carry out image classification and saliency detection on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn, can be used to effectively supervise the training of deep learning models, as demonstrated by the high performance of image classification and saliency detection on out-of-training classes. The obtained results show that the learned brain-visual features lead to improved performance and simultaneously bring deep models more in line with cognitive neuroscience work related to visual perception and attention.

1 INTRODUCTION

The paper connects brain activity with computational visual representations to improve machine models and support analysis of human visual processing. It introduces neural decoding, multimodal embedding, and applications spanning artificial intelligence and cognitive neuroscience.

  • Deep models achieve strong performance partly because their learned representations correlate with hierarchical brain representations in the visual pathway.
  • The paper first learns neural representations by classifying brain responses to natural images, then trains EEG and image encoders jointly in a siamese configuration.
  • Artificial Intelligence: The method supports visual-saliency extraction over time and localization of the cortical regions producing relevant information.
  • Artificial Intelligence: The approach decodes EEG signals related to visual tasks and identifies computational features that consistently match human neural activity.
  • Cognitive Neuroscience: For cognitive neuroscience, the approach offers AI-based analysis of neural responses in space and time without requiring experiments with multiple subjects and trials.
  • The paper evaluates image classification and saliency detection after introducing the joint brain-visual embedding framework.

2 RELATED WORK

Related work spans EEG classification, brain-representation decoding, brain-guided machine learning, and multimodal learning. The paper distinguishes its approach by learning a task-independent joint representation from noisy EEG and visual data.

  • EEG data classification: EEG classification has been applied mainly to brain-computer interfaces, clinical tasks, and monitoring cognitive functions.
  • EEG data classification: Many EEG models use controlled stimuli and single BCI paradigms, creating concerns that performance may fail to generalize when stimuli or tasks change.
  • Computational neuroscience: Neuroscience has mapped visual processing from oriented edges in V1 through increasingly complex representations in V4 and inferior temporal cortex.
  • Computational neuroscience: EEG, MEG, and fMRI provide information for reconstructing visual experiences, but none can monitor every neuron simultaneously.
  • Multimodal learning: Multimodal learning seeks joint embeddings that encode stimulus features across modalities while preserving within-modality and cross-modality similarities.
  • Multimodal learning: Unlike prior approaches tied to a proxy classification task, this method learns a reliable EEG-image representation using siamese encoders and a loss sensitive to intra-class differences.

3 MULTIMODAL LEARNING OF VISUAL-BRAIN FEA-

The paper learns a joint EEG-image embedding by training deep encoders to maximize compatibility between matched neural and visual representations while separating mismatched pairs. EEG-ChannelNet hierarchically extracts temporal and spatial features before projecting them into the shared space.

  • Motivation: Existing approaches learn brain representations through proxy tasks such as classification, which can make the resulting features task-dependent and class-discriminative.The paper identifies this as a limitation when the goal is to model relations between EEG and visual patterns.
  • Joint embedding: The proposed siamese network jointly maps EEG signals and images into a common space by maximizing similarity between corresponding representations.The encoders are trained on matched EEG/image pairs and contrasted with mismatched pairs.
  • Joint embedding: The training objective maximizes compatibility for matching EEG/image pairs without implicitly performing classification.This embedding formulation avoids tying the learned representation to class labels or a specific classification task.
  • Encoders’ architectures: EEG-ChannelNet processes neural signals hierarchically using temporal, spatial, and residual blocks before projecting features into the joint embedding space.Temporal convolutions capture within-channel patterns, while spatial convolutions model correlations across channels.
  • Encoders’ architectures: The EEG encoder is first tested for visual classification and is then trained within the siamese architecture for multimodal representation learning.This sequence assesses whether the encoder can decode visual information before joint embedding training.
  • Encoders’ architectures: The visual encoder uses a pretrained convolutional network followed by a linear projection, with end-to-end compatibility learning and image-encoder fine-tuning.Fine-tuning is intended to identify low- and middle-level visual-brain representations.

4 IMAGE CLASSIFICATION AND SALIENCY DETEC-

The learned brain-visual manifold supports image classification and saliency detection by using EEG-image compatibility as a signal for visual information. Saliency is estimated by measuring how compatibility changes when image patches are suppressed at multiple scales.

  • Applications: The learned EEG and visual embeddings can be used as feature extractors for image classification and as inputs to saliency detection.The approach leverages brain-visual compatibility learned from paired neural and image data.
  • Saliency detection: Saliency detection identifies image regions whose suppression changes the compatibility between an EEG signal and its corresponding image.The method analyzes compatibility fluctuations as image patches are removed at multiple scales.
  • Saliency detection: S(x, y, σ, e, v) is computed as the original compatibility score minus the score after masking the σ×σ region around pixel (x, y).The masked image is formed through element-wise multiplication with a binary suppression mask.
  • Saliency detection: Saliency values across multiple scales are combined by a normalized sum and then normalized separately for each image for visualization.The paper uses image-by-image normalization to produce the final visual representation.

5 VISUAL-RELATED BRAIN PROCESSES

The learned joint representation supports visual saliency analysis by measuring compatibility changes after targeted image or EEG alterations. These analyses identify image regions and EEG channels carrying visual information, with random channel replacements improving stability across samples.

  • Image saliency: Image saliency is estimated by masking patches at multiple scales and measuring the resulting change in EEG/image compatibility.Applying this analysis pixel by pixel produces a whole-image saliency map.
  • Brain-channel importance: EEG channel importance is measured as the compatibility difference between an original EEG/image pair and a pair with one channel suppressed.A larger compatibility drop indicates that the channel contains information matching visual data.
  • Brain-channel importance: Random Gaussian replacements make channel-importance estimates stable by preserving each original channel’s estimated mean and variance after low-pass filtering.The importance score averages compatibility differences over many replacements rather than relying on one arbitrary substitution.
  • Brain-channel importance: Channel importance is extended across dataset samples, including subsets such as object classes, to support broader conclusions than single EEG/image pairs.The expectation is computed over all dataset samples or a selected subset.

6 DECODING BRAIN REPRESENTATIONS

Prior differential analyses examined either image features or brain activity separately, leaving their correspondence unresolved. The proposed method links visual encoder features to EEG channels by comparing channel importance before and after feature-map removal.

  • Motivation: Earlier analyses identified influential visual features or relevant brain patterns, but did not determine which visual features correspond to which neural responses.The limitation concerns analyses performed in only one modality at a time.
  • Feature-channel association: The method removes image-encoder feature maps and maps the resulting compatibility changes to EEG channels that become least active compatibility-wise.This connects features at different visual scales with the brain regions represented by scalp channels.
  • Feature-channel association: A feature and channel are associated when removing the feature leaves the channel’s compatibility variation unchanged after accounting for the channel’s intrinsic importance.This criterion indicates that no encoded visual component remains unmatched for that channel.
  • Feature-channel association: Association between a channel and an image-encoder layer is estimated by averaging the feature-level association scores across that layer and, subsequently, across the dataset.The layer-level score indicates how much computational features resemble features processed by the brain at specific scalp locations.

7 EXPERIMENTS AND APPLICATIONS

The experiments test whether EEG encodes visual information and whether a learned brain-visual manifold supports classification, saliency detection, neural localization, and feature-brain correlation. The dataset uses filtered 128-channel EEG responses to 2,000 images from 40 object classes across six subjects.

  • Experimental objectives: The experiments first test EEG visual-information encoding and then assess the quality and correspondence of the learned joint brain-visual representation.Evaluation includes brain-signal and image classification, visual saliency detection, neural localization, and correlations between learned features and brain activity.
  • Brain-visual dataset: EEG data were collected from 6 subjects during block-based presentation, with each image shown for 0.5 seconds and 10-second black screens between class blocks.Images from each class were presented consecutively in a single sequence.
  • Brain-visual dataset: The dataset contains 2,000 images spanning 40 object classes, with classes selected to be recognizable, conceptually distinct, and visually uncluttered.Each class contains 50 ImageNet images.
  • Brain-visual dataset: 11,964 EEG segments remained after excluding 36 low-quality or unattended responses; each segment has 128 channels sampled at 1 kHz for 0.5 seconds.Segments are represented as 128×L matrices with approximately 500 samples per channel.
  • Preprocessing: Signals were band-pass filtered from 5 Hz to 95 Hz, notch-filtered at 50 Hz, and z-scored separately for each channel.The 95 Hz cutoff reflects the limited penetration of frequencies above approximately 100 Hz through the skull.
  • Neural visualization: Channel effects are visualized by plotting normalized channel-importance scores at electrode positions on a 2D scalp map and smoothing them with a Gaussian filter.The visualization uses a 400×400 map and a kernel standard deviation of 13 pixels.

7.2 Model implementation

The EEG encoder uses parallel temporal and spatial convolutional blocks whose outputs are concatenated to model temporal structure and relationships across EEG channels.

  • EEG encoder: The temporal block uses five parallel convolutional layers with dilation values 1, 2, 4, 8, and 16.The spatial block uses four parallel layers with kernel sizes from 128 channels down to 16.
  • EEG encoder: The implementation represents EEG as a 1 × 128 × 440 bidimensional input and applies 2D convolutions with one kernel dimension fixed at 1.Temporal layers use 1 × K kernels, while spatial layers use K × 1 kernels.

7.3 EEG classification

The EEG-ChannelNet experiment tests whether EEG contains visual-class information and how frequency bands and temporal windows affect decoding. Using high-frequency gamma signals across the full 20–460 ms interval produced the strongest reported comparison with prior methods.

  • Experiment design: EEG-ChannelNet was trained with a softmax layer to classify visual classes from EEG, validating the encoder and testing whether neural signals carry visual information.This classification experiment was not used to pre-train the full joint EEG-image embedding.
  • Experiment design: The experiment compared classification across EEG frequency sub-bands and temporal portions of the recorded signal.The study specifically considered the five conventional EEG bands and temporal subsequences of the original signals.
  • Temporal analysis: Shorter EEG segments performed worse than the full 20–460 ms interval, while excluding the first 110 ms reduced performance.The results indicate that both the complete time course and stimulus-onset period contributed to classification.
  • Temporal analysis: Adding the final 110 ms increased accuracy by 5 percentage points from 20–350 ms to 20–460 ms.The reported comparison associates the later interval with refinement of the learned features.
  • Classification results: 48.1% average test accuracy was achieved with the proposed approach using 55–95 Hz data, compared with EEGNet’s maximum accuracy of 31.9%.The comparison used high-frequency gamma-band data and the test split.

7.4 Siamese network training for classification

The siamese network jointly learns EEG and image encoders in a shared embedding, then evaluates the resulting representations through EEG and image classification. Joint learning improved EEG classification substantially and produced performance at least on par with supervised EEG-only training across tested configurations.

  • Training procedure: The siamese network samples one EEG signal with a positive and negative image and trains the encoders using a contrastive loss.The positive pair is (e_i, v_i), while (e_i, v_j) forms the negative pair.
  • Training procedure: After training, the EEG and image encoders serve as feature extractors in the joint embedding for both EEG and image classification.Validation accuracy is used to assess the multimodal representation and select encoder layouts.
  • Classification results: All tested joint-embedding configurations achieved classification accuracy on par with or better than EEG-only supervised training.Table 3 reports EEG and image accuracy for different image-encoder layouts.
  • Classification results: Adding visual features increased EEG classification by about 12 percentage points, from 48.1% to 60.4%.The reported EEG result uses the image encoder trained with Inception-v3.
  • Cross-modal effects: EEG classification benefited more from both modalities than image classification, because visual features guided learning toward more discriminative neural representations.The paper attributes the asymmetric benefit to the relative ease of image classification and the noisier, less-explored nature of neural activity data.
  • Cross-modal effects: The EEG classifier used EEG-encoder features guided by the joint visual representation rather than directly classifying visual features.The visual information itself was not used during EEG classification.

7.5 Saliency detection

The learned brain-visual embedding is evaluated for saliency detection by measuring compatibility changes when image regions are removed. Its saliency maps outperform tested baselines, and EEG time windows reveal a shift from feature-driven to context- and object-oriented attention.

  • Saliency method: Saliency was estimated from compatibility changes between the trained EEG and image encoders when image patches at multiple scales were removed.The analysis tested σ values of 3, 5, 9, 17, 33, and 65 pixels.
  • Evaluation setup: The evaluation used eye-tracking data from the same six subjects viewing 2,000 images, with SALICON and SalNet as baseline detectors.The eye movements were recorded with a 60-Hz Tobii T60 eye tracker.
  • Saliency results: The joint neural-visual saliency method outperformed the baseline saliency detectors and visual features alone on s-AUC, NSS, and CC measures.The paper reports the largest improvement for NSS, which is related to gaze-fixation scan paths and temporal saliency.
  • Temporal saliency: Across EEG time windows, early attention emphasized color contrast and edges, whereas later attention shifted toward context and object category.The tested ranges were 20–240 ms, 130–350 ms, and 240–460 ms.

7.6 Decoding Brain Representations

The analysis links EEG activity to visual representations across scalp locations and time, examining both class-level activation and correspondence with deep-network features. Results indicate temporally evolving visual processing and hierarchical alignment between neural activity and DCNN representations, while precise neurological interpretation remains outside the study’s scope.

  • Spatial and temporal neural representations: The analysis estimates cortex-level representation distributions by relating neural activity to visual content and deep-learned features, identifying informative scalp regions despite EEG’s limited cortical localization.The procedure combines coarse neural-image interaction analysis with fine neural-feature analysis.
  • Brain-guided saliency: The qualitative saliency comparison reports that the visual/EEG-driven detector more closely resembles human gaze maps and produces more detailed, less noisy reconstructions than comparison methods.The comparison includes SALICON, SalNet, and a visual-classifier-driven detector.
  • Temporal evolution of saliency: Saliency evolves from local and global visual features at 20–240 ms toward context and objects of interest at later windows, with the full time course integrating these contributions.The tested windows were 20–240 ms, 130–350 ms, 240–460 ms, and 20–460 ms.
  • Spatial and temporal neural representations: All visual classes rely heavily on early visual areas including V1 across tested time windows, while average activation progresses toward frontal and temporal regions over time.The reported progression links early visual processing with later higher cognitive and categorization-related regions.
  • Spatial and temporal neural representations: Visual content changes the activated scalp distribution; piano and electric-guitar classes engage regions nearer auditory cortex, consistent with sight-associated sound representations.This class-specific pattern is reported as an example of content-dependent activation.
  • Brain–DCNN correspondence: The learned compatibility measure relates DCNN-layer features to EEG scalp regions and timing, suggesting that hierarchical DCNN representations correlate with hierarchical neural processing.Association scores are averaged over channels to connect feature layers with spatial and temporal EEG activity.

8 CONCLUSION

The paper learns a joint image–EEG feature space with siamese encoders to use neural activity in image classification and saliency detection. The authors report reliable brain-guided visual decoding while limiting broader cognitive-neuroscience conclusions because the experiment is small.

  • Approach: The method learns a joint feature space for images and EEG recorded during image viewing by training two siamese encoders to maximize compatibility between corresponding inputs.The embeddings support computer-vision tasks supervised by brain activity.
  • Findings: Neural activity can be reliably used to drive image classification and saliency detection, while the learned model extracts neural representations and maps them to salient visual features.The conclusion frames these as the principal applications of the learned embeddings.
  • Scope: The study does not aim to draw general cognitive-neuroscience conclusions, particularly given the small scale of the cognitive experiment.The authors present the method as an AI-based strategy for reliable approximations of brain representations and scalp activity.
Loading 1810.10974v2…