Mind Reader: Reconstructing complex images from brain activities

Sikun Lin; Thomas Sprague; Ambuj K Singh

Source-linked AI summary

Mind Reader: Reconstructing complex images from brain activities

Sikun Lin, Thomas Sprague, Ambuj K Singh

arXiv:2210.01769v1Sep 30, 2022q-bio.NCcs.CVcs.HCcs.LGeess.IV

TL;DR

The paper asks how complex, semantically rich images can be reconstructed from fMRI despite limited brain-imaging data. It maps fMRI into a pretrained vision-language space and conditions a generative model on those representations, producing photo-realistic reconstructions that usually reflect image content faithfully.

Problem
The paper addresses reconstructing complex, semantically rich images from fMRI rather than simpler stimuli, under scarce brain-imaging data.
Method
The method maps fMRI signals into a pretrained CLIP vision-language space and uses image-text embeddings to condition a generative reconstruction model.
Results
The reconstructed images are photo-realistic and, most of the time, faithfully reflect the observed image content.
Takeaways & Limitations
The pipeline supports relatively faithful reconstruction of complex images while incorporating semantic and low-level visual information.
Takeaways & Limitations
The study trains on one subject, and reconstructions favor semantic similarity over pixel-level fidelity, performing worse on fine-detail, less-semantic images such as single faces.

Abstract

from arXiv · show

Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and can reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applying state-of-the-art deep learning models to this problem. We find that incorporating an additional text modality is beneficial for the reconstruction problem compared to directly translating brain signals to images. Therefore, the modalities involved in our method are: (i) voxel-level fMRI signals, (ii) observed images that trigger the brain signals, and (iii) textual description of the images. To further address data scarcity, we leverage an aligned vision-language latent space pre-trained on massive datasets. Instead of training models from scratch to find a latent space shared by the three modalities, we encode fMRI signals into this pre-aligned latent space. Then, conditioned on embeddings in this space, we reconstruct images with a generative model. The reconstructed images from our pipeline balance both naturalness and fidelity: they are photo-realistic and capture the ground truth image contents well.

1 Introduction

The paper targets reconstruction of semantically rich, complex images from fMRI, extending beyond simpler stimuli while addressing scarce brain-imaging data. It adds text and maps fMRI into a pretrained vision-language space before generative reconstruction.

1 Introduction: The study reconstructs complex, multi-object images from human brain signals rather than only simple shapes or single centered objects.NSD provides natural images with richer contextual information and annotations, supporting this more natural reconstruction setting.
1 Introduction: Adding text provides contextual information suited to images containing multiple objects and intricate relationships.The paper motivates text as an additional modality because the brain is multimodal and complex scenes require semantic context.
1 Introduction: The method maps fMRI into a pretrained vision-language space shared by images and text, then conditions a generative model on the mapped representations.This separates fMRI mapping from large-scale vision-language representation learning and generative-model training.
1 Introduction: The contribution addresses data scarcity by incorporating text and leveraging pretrained latent representations and models.The approach focuses on semantic image representations while retaining low-level visual features.
1 Introduction: The authors report relatively faithful decoding of complex images and study brain-region properties through microstimulation.The paper describes the reconstruction setting as more natural than object-centered image reconstruction.

2 Method

The method screens informative captions, maps voxel-level fMRI signals to CLIP image and text embeddings, and uses those embeddings to condition a StyleGAN-based generator. Two-stage training and contrastive objectives align generated images with semantic and low-level visual features.

2.1 Caption Screening: Caption screening uses CLIP image-text alignment to retain captions that are more descriptive and specific to each image.For the illustrated sample, captions (2) and (3) are retained using a threshold equal to 0.5 × 0.519.
2.2 Mapping fMRI signals to CLIP space: The pipeline extracts task-related cortical voxels, flattens and standardizes them, then maps fMRI vectors to 512-dimensional CLIP image and text embeddings.The selected ROI contains 12682 to 17907 voxels across different NSD brains, and the mapping models use convolutional, residual, and linear layers.
2.2 Mapping fMRI signals to CLIP space: The mapping models are trained with MSE, cosine-similarity, and contrastive losses to encode fMRI signals into the CLIP space.The two models separately target image and caption embeddings, whose systematic modality differences motivate using both conditions.
2.3 Image reconstruction with CLIP embedding conditioning: The generator transforms mapped image and caption embeddings into condition codes, combines them with StyleGAN latent features, and generates the reconstructed image.The condition codes are injected into StyleSpace after projection through fully connected layers.
2.3 Image reconstruction with CLIP embedding conditioning: Contrastive losses align generated images with CLIP conditions and ResNet features, while the discriminator uses real/fake and semantic projection heads.These objectives target both semantic fidelity and lower-level features such as edges, corners, and shapes, while helping stabilize training.
2.3 Image reconstruction with CLIP embedding conditioning: Training proceeds in two stages: first fMRI-to-CLIP mapping models are trained, then the conditional generator and discriminator are finetuned with the mapping models frozen.The generator is built on Lafite, which adapts StyleGAN2 to conditional image generation from CLIP text embeddings.

3 Results

The pipeline reconstructs complex natural scenes from a single subject’s fMRI signals using mapped CLIP representations and conditional generation. Results show semantic alignment, photo-realistic reconstructions, and preservation of complex object relationships, while evaluation remains subject-specific.

3.1 Data and experimental setup: The study trains and evaluates on one subject because individual brain coordinates and encoding differences make shared-space mapping potentially information-losing.The NSD data used for this subject contains 27,750 fMRI-image pairs across 9,841 images, split image-wise into training and validation sets.
3.2 Mapping models from fMRI to CLIP embeddings: The mapping models produce semantically close retrieval mismatches, supporting their use as contexts for conditional generation.Incorrect matches are selected when mapped embeddings for other images are closer to the ground-truth CLIP embedding than the correct mapped embedding.
3.3 Conditional image generation: Generated images capture semantics and visual features relatively well, with fMRI-mapped embeddings driving more variation than the noise vector.Repeated scans of the same stimulus produce generations emphasizing different scene elements, such as fences, people, or flags.
3.3 Conditional image generation: Photo-realistic reconstructions reflect object shapes, backgrounds, and relationships in complex scenes while using 15,724 voxels.The authors compare the pipeline with prior methods using similar NSD images, while noting that direct one-to-one comparison is difficult because dataset complexity differs.
3.4 CLIP space as the intermediary: CLIP embeddings contain the most object-level information, while fMRI remains highly predictive and mapping into CLIP retains most fMRI information.The performance gap between direct fMRI prediction and mapped fMRI representations is described as minimal; both CLIP and fMRI perform worse on stuff than things categories.

4 Further discussions

The paper discusses using CLIP’s semantic space to address data scarcity, while acknowledging a trade-off between semantic similarity and pixel-level fidelity. It also identifies subject generalization, alternative latent spaces, generative models, and societal risks as open considerations.

4 Further discussions: CLIP’s semantic space addresses limited fMRI-image data by enabling separately pretrained generative components and semantically informative representations.The approach separates fMRI mapping from image-text representation learning and generator pretraining, allowing larger datasets to support the latter stages.
4 Further discussions: The pipeline favors semantically similar scenes over exact pixel reconstruction, which weakens performance on fine-detail, low-semantic images such as single faces.The authors identify better alignment of low-level visual features as a direction for future work.
4 Further discussions: Future work includes testing across subjects, examining personalized perceptions, evaluating other multimodal latent spaces and diffusion models, and studying imagery without ground-truth images.The paper specifically mentions subject differences, behavioral measures, TSM, and DALL-E 2 as possible directions.
4 Further discussions: Current recording constraints limit immediate social impact, but more capable neural devices could raise privacy, manipulation, and false-reconstruction risks.The paper suggests regulation, input discrimination, and pixel-level reconstruction checks as possible safeguards.

5 Conclusion

The paper presents a brain-signal-to-image pipeline for reconstructing complex scenes by incorporating text and a pretrained vision-language latent space. Its outputs are photo-realistic and usually reflect the observed image content while combining semantic and low-level visual information.

5 Conclusion: The pipeline maps fMRI signals into a pretrained vision-language space and conditions a generative model to reconstruct complex images.An additional contrastive loss incorporates low-level visual features into the semantic-based pipeline.
5 Conclusion: The reconstructed images are photo-realistic and, most of the time, faithfully reflect the image content.The paper frames the approach as opening opportunities to study brain functions and potentially support brain-computer interfaces.

Checklist

The checklist reports that the paper describes its contributions, limitations, societal impacts, experimental reproducibility materials, and training details. It also records several asset, consent, privacy, and participant-risk items as not applicable or delegated to cited dataset documentation.

Checklist: The paper reports that it describes its contributions and scope, limitations, potential negative societal impacts, and compliance with ethics guidelines.The checklist points to section 4 for limitations and societal impacts.
Checklist: The checklist marks participant instructions, risks, compensation, and several data-documentation items as not applicable or covered by the cited dataset paper.It also states that the code and data are proprietary and that asset licenses were not mentioned.

A.1 Data

The data discussion clarifies that NSD fMRI activities are inferred beta responses and can vary across repeated presentations of the same image. Training uses multiple image augmentations, which improve fMRI-to-CLIP mapping performance.

A.1 Data: fMRI activities differ when the same subject views the same image at different times, and the paper uses NSD-inferred betas rather than direct BOLD measurements.The betas are inferred from BOLD signals using general linear models.
A.1 Data: Figure 7 visualizes responses to two images repeated three times using only the first 200 voxels.The restricted voxel display is for visualization purposes.
A.1 Data: Image augmentation improves fMRI-to-CLIP image-embedding mapping performance during training.The tested augmentations include crops, flips, color jitter, grayscale conversion, blur, and random masking.

A.2 Experiment hyperparameters

The experiments use specified contrastive-loss, mapper, batch-size, learning-rate, and conditional-StyleGAN2 settings, alongside CLIP embedding thresholding and caption screening.

A.2 Experiment hyperparameters: The experiments set τ = 0.5 for all contrastive losses and use staged α weights for fMRI-CLIP mappers.The mappers begin with α1 = 0.4, α2 = 0.6, α3 = 0, then finetune with α1 = 0.2, α2 = 0.3, α3 = 0.5.
A.2 Experiment hyperparameters: Mapper training uses batch size 32 without contrastive loss and 128 with contrastive or VICReg loss, with learning rate 0.0004.
A.2 Experiment hyperparameters: Conditional StyleGAN2 uses λ1 = 5, λ2 = 10, and λ3 = 10 for its losses.
A.2 Experiment hyperparameters: CLIP embeddings are thresholded at ±1.5 to remove outliers, while caption screening makes retained caption embeddings more aligned.

A.3 Results for the fMRI-CLIP mapping models fm

The fMRI-CLIP mapping models are evaluated through forward and backward retrieval, with the best mapping configuration visualized and combined mappers improving retrieval, especially backward retrieval.

A.3 Results for the fMRI-CLIP mapping models fm: Forward retrieval asks whether the nearest mapped embedding has the correct ground-truth CLIP embedding, whereas backward retrieval reverses this matching direction.
A.3 Results for the fMRI-CLIP mapping models fm: Figure 9 visualizes the best mapping setting after thresholding, image augmentation, random-caption sampling, and staged loss finetuning.
A.3 Results for the fMRI-CLIP mapping models fm: Combining mapped embeddings from multiple mappers boosts retrieval performance, especially backward retrieval, beyond the individual ResNet and CLIP mappings.The combined similarity matrix is formed by a weighted sum of per-model batch similarity matrices.
A.3 Results for the fMRI-CLIP mapping models fm: The ResNet mapper fmr achieves 6 correct forward retrievals and 50 correct backward retrievals in a batch of 300.

A.4 Additional quantitative results (generator)

Additional quantitative evaluation measures image identification under different settings and candidate-set sizes, while retaining overlapping semantic categories in validation images.

A.4 Additional quantitative results (generator): The evaluation reports 2-way identification under different pipeline settings and n-way identification for n = 2, 5, 10, and 50.
A.4 Additional quantitative results (generator): For n-way identification, each reconstructed validation image is compared against n randomly selected images, including the ground-truth image.
A.4 Additional quantitative results (generator): The validation set contains 1477 image-fMRI pairs with overlapping categories because complex scenes contain multiple objects that are not straightforward to separate.

A.5 Visual results from microstimulation experiments

The visual analyses probe microstimulation, voxel masking, generator transfer, modality conditioning, failure modes, encoding, and downstream extensions of the reconstruction pipeline.

A.5 Visual results from microstimulation experiments: Masking visual-cortex voxels changes mapped CLIP embeddings, while microstimulation experiments generate images after increasing or silencing selected voxel activities.
A.5 Visual results from microstimulation experiments: CLIP is a compressed representation: fMRI-based object classification remains robust until masking exceeds 0.9, whereas mapped-CLIP prediction declines at an almost constant rate.
A.5 Visual results from microstimulation experiments: Using a Lafite generator pretrained on CC3M without extra finetuning preserves semantic alignment, but photo-realism decreases for out-of-distribution images and outputs contain source watermarks.
A.5 Visual results from microstimulation experiments: Image-conditioned embeddings emphasize visual attributes, text-conditioned embeddings emphasize semantics, and using both conditions exposes these complementary effects.
A.5 Visual results from microstimulation experiments: Reconstructions tend to fail for close-up details and semantically novel scenes, while also reflecting dataset biases, altered backgrounds, and mixed image characteristics.
A.5 Visual results from microstimulation experiments: The encoding-decoding cycle preserves image semantic information, and interpolated fMRI scans produce smooth transitions that may support movie reconstruction.