PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain decoding: toward real-time reconstruction of visual perception

Yohann Benchetrit, Hubert Banville, Jean-Rémi King

arXiv:2310.19812v3eess.IVcs.AIcs.LGq-bio.NC

TL;DR

fMRI-based visual decoding faces temporal-resolution limits that challenge real-time use, leaving open whether MEG can generate natural images in real time. The paper aligns MEG signals with pretrained visual embeddings and uses them for retrieval and image generation, reporting a 7X improvement over linear baselines and continuous generation from temporally resolved signals.

  • Problem

    fMRI typically yields one brain-activity snapshot every two seconds, challenging real-time decoding, and it is unknown whether MEG can generate natural images in real time.

    Source: §1 · p. 2

  • Method

    The paper trains a three-module pipeline to align MEG activity with pretrained visual embeddings and generate images from MEG signals.

    Source: §1 · p. 2

  • Results

    The MEG decoder yields a 7X performance increase over linear baselines, helps reveal when semantic features are processed, and enables continuous image generation from temporally resolved signals.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results provide a step toward decoding visual processes continuously unfolding in the human brain.

    Source: §4 · p. 10

    The approach depends on pretrained models and aligns MEG signals to their embeddings; whether fine-tuning the image and generation modules or combining visual features improves decoding remains untested.

    Source: §4 · p. 10

Abstract

from arXiv · show

In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution ($\approx$0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution ($\approx$5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T fMRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding -- in real-time -- of the visual processes continuously unfolding within the human brain.

1 Introduction

The paper asks whether MEG’s high temporal resolution can support natural-image decoding where fMRI’s slower measurements challenge real-time use. It combines pretrained visual representations with MEG decoding and image generation, reporting improved retrieval and continuous image generation from temporally resolved signals.

Generating images from brain activity

  • Earlier fMRI generation pipelines map brain activity to pretrained image embeddings, then use those embeddings to condition pretrained image-generation models.The resulting images may be partly hallucinated, but the paper notes they can be easier to interpret than latent features.Source: §1 · pp. 1–2

The challenge of real-time decoding

  • fMRI’s blood-flow timescale typically provides one brain-activity snapshot every two seconds, challenging clinical use and real-time decoding.MEG measures activity at approximately 5,000 Hz, but has approximately 300 sensors compared with fMRI’s approximately 100,000 voxels.Source: §1 · p. 2
  • Whether MEG can generate natural images in real time remains unknown.Source: §1 · p. 2

Our approach

  • The approach aligns MEG activity with pretrained visual embeddings and generates images from a stream of MEG signals.It combines previous work on speech retrieval from MEG and image generation from fMRI.Source: §1 · p. 2
  • The MEG decoder yields a 7X performance increase over linear baselines, helps reveal when semantic features are processed, and enables continuous image generation from temporally resolved signals.The authors describe these contributions as a step toward understanding visual processes unfolding in the brain.Source: §1 · p. 2

2 Methods

The methods map MEG time windows to pretrained image-model representations for retrieval or generation. The pipeline combines contrastive and regression objectives, a temporal MEG encoder, pretrained image and generation models, and evaluations on modified THINGS-MEG splits.

2.1 Problem statement

  • MEG time windows are mapped to pretrained image representations so the model can retrieve or predict the presented image.Source: §2.1 · p. 3

2.2 Training objectives

  • Retrieval uses bidirectional CLIP/InfoNCE, generation uses mean squared error on unnormalized latents, and a tuned convex combination can train models with both objectives.The contrastive objective uses cosine similarity between predicted and target representations.Source: §2.2 · p. 3

2.3 Brain module

  • The brain module adapts a dilated residual ConvNet to predict one image latent per MEG window, using temporal aggregation and separate MLP heads for the two loss terms.The tested aggregation options are global average pooling, learned affine projection, and attention.Source: §2.3 · p. 3

2.4 Image modules

  • Image embeddings span supervised VGG-19, image-text-aligned CLIP, variational autoencoders, and human-engineered features.For vision transformers, the study also evaluates mean-token and CLS embeddings.Source: §2.4 · p. 4

2.5 Generation module

  • Image generation uses a latent diffusion model conditioned on CLIP-Vision, CLIP-Text, and AutoKL embeddings.The CLIP-Text embedding comes from the stimulus image’s THINGS object category.Source: §2.5 · p. 4

2.6 Training and computational considerations

  • Cross-participant models use Adam, early stopping on a validation set, and three random-seed runs per model.Training uses approximately 63,000 examples, a batch size of 128, and a single Volta GPU with 32 GB of memory.Source: §2.6 · p. 4

2.7 Evaluation

  • Retrieval is assessed with relative median rank or top-5 accuracy; generation is assessed with image-fidelity and feature-similarity metrics.Generation metrics include PixCorr, SSIM, SwAV, AlexNet, Inception, and CLIP scores; generations are sampled across best, middle, and worst metric-ranked blocks.Source: §2.7 · pp. 4–5

2.8 Dataset

  • The THINGS-MEG dataset includes 22,448 unique presented images, and the study modifies its splits to test unseen categories and retrieval among same-category images.The adapted training set removes images whose categories appear in the original test set; the large test set is built from removed images.Source: §2.8 · pp. 5–6

3 Results

A deep ConvNet improves image retrieval over linear baselines, while time-resolved retrieval and generation indicate that MEG captures category-level information but relatively little low-level detail.

ML as an effective tool to learn brain responses

  • 7X improvement over linear baselines; VGG-19, CLIP-Vision, and DINOv2 reach top-5 accuracies of 70.33±2.80%, 68.66±2.84%, and 68.00±2.86%, respectively.Figure 2 compares retrieval on small and large test sets; best latent representations yield around 70% and 13% accuracy, respectively.Source: §3 · p. 6, Figure 2 · p. 6

Temporally-resolved image retrieval

  • Time-resolved retrievals tend to come from the correct category, but do not show obvious low-level similarities to the images participants saw.Category-consistent examples appear mostly during the first sub-windows (t ≤1 s).Source: §3 · p. 7
  • Category-level information dominates visual representations as early as 250 ms; retrieval peaks after both image onset and offset, with DINOv2 particularly strong after offset.The reported peaks occur for windows ending around 200–275 ms after onset and 150–200 ms after offset.Source: §3 · pp. 6–7

Generating images from MEG

  • MEG generations often preserve the true image’s high-level category but little low-level detail; early onset and offset responses relate to low-level metrics, while 200–500 ms activity relates more to high-level features.The low-level detail examples include object position and color.Source: §3 · p. 7
  • A similar pipeline on 7T fMRI also recovers low-level features, suggesting MEG signals are comparatively harder to decode for those features.Source: §3 · p. 7

4 Discussion

The discussion situates MEG image decoding against earlier constrained decoders and presents open-set image generation as a novel step. It highlights interpretability and temporal-resolution applications while noting reconstruction limits and ethical considerations.

4 Discussion

  • Table 1 reports that NSD fMRI betas are significantly easier to decode than THINGS-MEG signals on reconstruction metrics.The comparison uses cross-validated Ridge regression and reports PixCorr, SSIM, AlexNet(2), AlexNet(5), Inception, SwAV, and CLIP.Source: Table 1 · p. 9
  • To the authors’ knowledge, no prior MEG study reliably generates an open set of images end-to-end, unlike earlier work limited to small-class or pairwise discrimination.Earlier EEG image work also focused on categorization and risked relying on autocorrelated noise from blockwise presentation.Source: §4 · p. 9
  • Generative decoding can make abstract neural representations more interpretable through concrete image predictions, including potentially unanticipated textures or object parts.The authors suggest this approach could help clarify when subjective visual representations arise and distinguish them from representations involved in copying sensory inputs.Source: §4 · p. 9
  • Real-time decoding is required for brain-computer interfaces, but clinical use may require extending this work to EEG, which is more common in clinical settings.The passage identifies EEG as having temporal resolution similar to MEG.Source: §4 · p. 9
  • MEG generations appear worse at preserving low-level features than a similar 7T fMRI pipeline, consistent with MEG’s lower spatial resolution and brief low-level feature windows.The authors suggest transient low-level features may have less impact on generation than more persistent high-level features.Source: §4 · p. 9
  • The approach depends on pretrained models and aligns MEG signals to their embeddings; whether fine-tuning image and generation modules or combining visual features improves decoding remains untested.Source: §4 · p. 10
  • For non-invasive recordings, decoding is reported as high for perception but diminished for imagination and disruptive tasks, making subject consent a technical as well as legal requirement.Source: §4 · p. 10

A Additional details on the brain module architecture

The brain module transforms MEG windows spatially, processes them over time with residual convolutional blocks, and aggregates temporal features before projecting them into target latents.

A Additional details on the brain module architecture

  • The module applies two spatial linear transformations, first conditioned on MEG sensor positions and then learned separately for each subject.Each subject receives an individual spatial projection matrix.Source: A Additional details on the brain… · p. 14
  • Its temporal backbone uses dilated 1D convolutional blocks with residual skip connections, then projects the output to 2048 features.Each block has three convolutional layers; the first two use GELU activations and the last uses GLU.Source: A Additional details on the brain… · p. 14
  • A learned affine projection aggregates the temporal dimension, and separate CLIP and MSE projection heads map the resulting features to target-latent dimensions.The compared aggregation options were global average pooling, learned affine projection, and Bahdanau attention; the affine projection was selected.Source: A Additional details on the brain… · p. 14

B Hyperparameter search

The hyperparameter search evaluates MEG-to-image retrieval configurations across data splits and model settings, then selects one configuration for the experiments.

B Hyperparameter search

  • The search randomly divides 79,392 MEG-image pairs into 60%-20%-20% train, validation, and test splits, keeping all presentations of each image together.Validation supports early stopping, while test data evaluate each configuration.Source: B Hyperparameter search · p. 14
  • The selected configuration uses a [-0.5, 1.0] s MEG window, learning rate 3 × 10^-4, batch size 128, two convolutional blocks, and affine temporal aggregation.It also uses spatial attention and subject-specific layers, one CLIP projection-head block, and an adapted CLIP loss with image-axis normalization, only the brain-to-image term, and fixed temperature τ = 1.Source: B Hyperparameter search · p. 14
  • Figure S1 searches retrieval performance across optimizer settings, brain-module architecture, MEG windows, temporal aggregation and projection-head depth, and CLIP-loss configuration.The figure identifies 0.05% as chance-level top-5 accuracy.Source: Figure S1 · p. 15

C Image embeddings

The retrieval evaluation compares image embeddings from several learning paradigms, variational autoencoders, and engineered visual features.

C Image embeddings

  • The retrieval evaluation groups candidate image embeddings by their associated paradigm.Source: C Image embeddings · p. 15
  • The supervised-learning candidate is the 1,000-dimensional last layer of VGG-19.Source: C Image embeddings · p. 15
  • Text/image-alignment candidates include CLIP-Vision and CLIP-Text hidden layers, with CLS and MEAN pooling variants.Source: C Image embeddings · p. 15
  • Self-supervised candidates use DINOv1 and DINOv2 output layers, including CLS and MEAN pooling variants.Source: C Image embeddings · p. 15
  • The candidate set also includes VDVAE layer activations, an AutoKL bottleneck latent, and engineered color-histogram, LBP, and HOG features.The VDVAE uses activations from its first 31 layers; HOG is configured with 8 orientations, 8 pixels per cell, and 2 cells per block.Source: C Image embeddings · pp. 15–16

D 7T fMRI dataset

The 7T fMRI analysis uses Natural Scenes Dataset data and evaluates reconstructions for participant 1. Its shared evaluation set and cross-validated Ridge regressions provide the basis for the reported examples.

D 7T fMRI dataset

  • The Natural Scenes Dataset contains fMRI data from 8 participants viewing 73,000 RGB images, with each participant seeing 10,000 unique images repeated three times.Only participants 1, 2, 5, and 7 completed all sessions.Source: D 7T fMRI dataset · p. 16
  • For each of four participants, Ridge regressions train on 9,000 participant-specific images and evaluate on a shared set of 1,000 images, with results reported for participant 1.The training and evaluation images each include three repetitions; the regression regularization coefficient is cross-validated with five folds.Source: D 7T fMRI dataset · p. 16
  • Figure S2 presents generated images conditioned on fMRI-based latent predictions, grouped as best, average, and worst retrievals by the sum of minus SwAV and SSIM.Source: D 7T fMRI dataset · p. 16

E Linear Ridge regression scores on pretrained image representations

The appendix compares the MEG brain module with a linear Ridge baseline and examines retrieval using intermediate VGG-19 layers. The brain module improves on the baseline, while some intermediate layers slightly outperform the final layer.

E Linear Ridge regression scores on pretrained image representations

F Impact of choice of layer in supervised models

G MEG-based image retrieval examples

The appendix provides examples of MEG retrievals and generations, evaluates retrieval with growing time windows, and reports participant-level generation metrics. Growing-window retrieval improves after image offset, while generated examples include several failure types.

G MEG-based image retrieval examples

H MEG-based image generation examples

  • Figure S5 shows diffusion-generated examples, while Figure S6 illustrates category errors, unrealistic chimeras, and generations that preserve few visual features.The authors speculate that better generation modules may help with chimeras and optimization on low- and high-level features may help with category errors.Source: H MEG-based image generation examples · p. 18

I Performance of temporally-resolved image retrieval with growing windows

  • All growing-window models considerably improve retrieval performance after image offset, suggesting that activity then represents both low- and high-level features.The analysis grows windows from -100 to 0 ms around image onset in 25 ms increments through 1,500 ms, training a separate model for each configuration.Source: I Performance of temporally-resolved… · p. 18

J Per-participant image generation performance

K Analysis of temporal aggregation layer weights

The temporal aggregation layer's learned weights are inspected to understand which time points contribute to decoding. The analysis finds different temporal patterns for low- and high-level features.

K Analysis of temporal aggregation layer weights

  • Mean absolute weights across seeds are plotted after initializing the temporal aggregation weights to zero; the learned weights indicate the time steps on which models focus.Source: K Analysis of temporal aggregation… · p. 19
  • Low-level features have close-to-zero temporal weights from 0.2–0.5 s, unlike high-level features such as VGG-19 and CLIP-Vision.The weights deviate from near-zero baseline after stimulus onset, during maintenance, and after offset.Source: K Analysis of temporal aggregation… · p. 19
  • The models rely on early low-level responses at 0–0.1 s, as well as activity during maintenance and after stimulus offset, to maximize decoding performance.Source: K Analysis of temporal aggregation… · p. 19

L Temporally-resolved image generation metrics

Temporally resolved generation metrics show different timing patterns for low- and high-level visual features: low-level metrics peak early, while high-level metrics plateau during image presentation.

L Temporally-resolved image generation metrics

  • Low-level metrics peak in the first 200 ms, while high-level metrics plateau throughout image presentation; low-level metrics also show a sharp peak after image offset.The analysis evaluates generation quality using 100-ms sliding windows.Source: L Temporally-resolved image generation… · p. 19
  • Retrieval accuracy plateaus a few hundred milliseconds after image onset and offset, complementing the temporally resolved generation-metric analysis.Figure S7 evaluates models trained on growing windows across different image embeddings.Source: Figure S7 · p. 24
  • Figure S9 compares normalized scores across post-stimulus windows and unnormalized scores at stimulus onset versus each metric’s post-stimulus maximum.Dashed lines mark chance performance; error bars show standard error for PixCorr, SSIM, and SwAV.Source: L Temporally-resolved image generation… · p. 19, Figure S9 · p. 25
Loading 2310.19812v3…