Source-linked AI summary
Natural scene reconstruction from fMRI signals using generative latent diffusion
Furkan Ozcelik, Rufin VanRullen
TL;DR
Brain-Diffuser targets the challenge of reconstructing complex natural scenes from fMRI while preserving both low-level structure and high-level semantics. It combines a VDVAE initial reconstruction with Versatile Diffusion guided by predicted visual and textual features, outperforming previous models on the Natural Scenes Dataset and producing ROI-optimal scenes consistent with neuroscientific knowledge.
Problem
Previous methods generally reconstructed either low-level visual properties or high-level scene semantics, leaving their joint reconstruction in complex natural scenes unresolved.
Method
Brain-Diffuser first predicts VDVAE latents for a low-level initial reconstruction, then uses image-to-image Versatile Diffusion conditioned jointly on predicted CLIP-Vision and CLIP-Text features.
Results
Brain-Diffuser outperforms previous models qualitatively and quantitatively on the Natural Scenes Dataset across high-level and low-level metrics.
Takeaways & Limitations
The method also generates ROI-optimal scenes consistent with neuroscientific knowledge, supporting applications in brain-computer interfaces and fundamental neuroscience.
Takeaways & Limitations
Future improvements may require alternatives to ridge regression if generative models reach a ceiling in linearly explaining brain activity, potentially requiring larger training datasets.
Abstract
from arXiv · showhide
In neural decoding research, one of the most intriguing topics is the reconstruction of perceived natural images based on fMRI signals. Previous studies have succeeded in re-creating different aspects of the visuals, such as low-level properties (shape, texture, layout) or high-level features (category of objects, descriptive semantics of scenes) but have typically failed to reconstruct these properties together for complex scene images. Generative AI has recently made a leap forward with latent diffusion models capable of generating high-complexity images. Here, we investigate how to take advantage of this innovative technology for brain decoding. We present a two-stage scene reconstruction framework called ``Brain-Diffuser''. In the first stage, starting from fMRI signals, we reconstruct images that capture low-level properties and overall layout using a VDVAE (Very Deep Variational Autoencoder) model. In the second stage, we use the image-to-image framework of a latent diffusion model (Versatile Diffusion) conditioned on predicted multimodal (text and visual) features, to generate final reconstructed images. On the publicly available Natural Scenes Dataset benchmark, our method outperforms previous models both qualitatively and quantitatively. When applied to synthetic fMRI patterns generated from individual ROI (region-of-interest) masks, our trained model creates compelling ``ROI-optimal'' scenes consistent with neuroscientific knowledge. Thus, the proposed methodology can have an impact on both applied (e.g. brain-computer interface) and fundamental neuroscience.
1 Introduction
Brain-Diffuser addresses the difficulty of reconstructing complex natural scenes with both low-level visual properties and high-level semantics. It combines VDVAE-based layout reconstruction with multimodal latent-diffusion guidance and reports stronger qualitative and quantitative performance than earlier models.
- Benchmark: The Natural Scenes Dataset is a demanding benchmark because it contains diverse, complex COCO images with multiple objects.It has become a de facto benchmark for fMRI-based natural scene reconstruction.
- Research gap: Earlier reconstruction studies typically captured either low-level properties or high-level semantic features, but not both together for complex scenes.The challenge is especially pronounced for multi-object COCO scenes in the Natural Scenes Dataset.
- Proposed framework: Brain-Diffuser uses a two-stage framework that first reconstructs low-level structure with VDVAE and then refines scenes with Versatile Diffusion conditioned on vision and language representations.The second stage is designed to combine layout-related information with semantic guidance.
- Results: Brain-Diffuser produces naturalistic reconstructions that preserve overall layout and semantic information while varying only modestly in finer details.The reported evaluation considers both high-level and low-level image properties.
- Results: Brain-Diffuser outperforms earlier models qualitatively and quantitatively on both high-level and low-level metrics.The authors describe this as establishing state-of-the-art performance.
2 Materials and Methods
The method uses pretrained generative models and learns regressions from fMRI patterns to their latent or feature representations. VDVAE supplies an initial low-level reconstruction, while Versatile Diffusion uses predicted visual and textual CLIP features to generate the final image.
- Dataset: The study uses the Natural Scenes Dataset, collected from 8 subjects viewing COCO images, and analyzes the 4 subjects who completed all trials.The resulting training set contained 8859 images.
- VDVAE first stage: In stage 1, ridge regression maps fMRI patterns to VDVAE latent variables, which are decoded into a low-level reconstruction used as the diffusion model’s initial guess.The VDVAE encoder and decoder are pretrained, while only the brain-to-latent regression is trained.
- VDVAE first stage: VDVAE is designed to model complex image distributions with hierarchical latent variables.The implementation uses latent variables from the first 31 of 75 layers; additional layers are sampled from the prior at test time.
- Versatile Diffusion second stage: Versatile Diffusion applies image-to-image latent diffusion, adding noise to the VDVAE reconstruction before jointly guided reverse diffusion.The model and CLIP encoders are pretrained and frozen, with only the brain-to-latent regression layers trained.
- Versatile Diffusion second stage: In stage 2, separate regressions predict CLIP-Vision and CLIP-Text features from fMRI patterns.The predicted features provide visual and language conditions for the final reconstruction.
- Reproducibility: The project provides code, pretrained weights, and scripts for test-image and ROI-based synthetic-pattern reconstructions.The materials are publicly available through the project repository.
3 Results and Analyses
Brain-Diffuser reconstructs naturalistic scenes that preserve ground-truth layout and semantics while differing in fine pixel-level details. Across quantitative comparisons, ablations, and ROI analyses, the results show complementary contributions from low-level and semantic representations and produce interpretable ROI-optimal images.
- Image reconstruction examples: Reconstructions preserve most ground-truth layout and semantics across subjects, despite differences in pixel-level details.Examples include consistent plane pose, groups of people, wheelchair presence, and highway structure.
- Image reconstruction examples: The model produces naturalistic alternate renditions rather than picture-perfect copies, with especially visible differences in color and contrast.
- Comparison with state of the art: Brain-Diffuser outperforms previous models by a decent margin across the reported image-quality metrics, spanning low-level and high-level properties.Comparisons with Takagi et al. and Gu et al. use the common 982-image test set; the Lin et al. comparison uses a matched split.
- Ablation Studies: The full model balances complementary strengths: Only-VDVAE excels at low-level measures, whereas removing VDVAE favors high-level measures but loses layout information.The authors conclude that VDVAE is necessary but insufficient, while the complete model achieves near-optimal high-level features with strong low-level improvements.
- Ablation Studies: Qualitatively, removing VDVAE loses object positioning, removing CLIP-Text can alter object count or placement, and removing CLIP-Vision yields blurry reconstructions.The street-and-buildings example highlights VDVAE’s contribution to layout, while ski and plane examples illustrate CLIP-Text effects.
- ROI-optimal stimuli: ROI-optimal images align with known visual-region properties, including detailed peripheral textures for V1, face imagery for Face-ROI, and character-like content for Word-ROI.The method is presented as a way to visualize functional properties in vivid, high-resolution images.
4 Discussion
Brain-Diffuser combines low-level image reconstruction with multimodal latent-diffusion refinement, producing natural-scene reconstructions that preserve layout and semantic information. On NSD, it outperforms prior models across low- and high-level metrics, while remaining dependent on future improvements in generative models and brain-to-latent mappings.
- Discussion: Brain-Diffuser first generates low-level “initial guess” reconstructions with VDVAE, then refines them through Versatile Diffusion using predicted visual and textual features.The framework uses an image-to-image diffusion pipeline for final reconstruction.
- Discussion: Brain-Diffuser reconstructions preserve most scene layout and semantic information despite not being perfectly identical to the ground-truth images.The reconstructed scenes also appear more naturalistic than reconstructions from earlier studies.
- Discussion: Brain-Diffuser outperforms previous models on both high-level and low-level metrics, advancing fMRI-based image reconstruction.The authors attribute the stronger comparison results against Takagi et al. partly to VDVAE low-level details and dual CLIP-Vision/CLIP-Text conditioning.
- Future work: Future progress may require better-suited generative models and alternatives to ridge regression if latent spaces reach a ceiling in explaining brain activity.Nonlinear regressions or deep hierarchical networks may require larger training datasets than ridge regression.