Source-linked AI summary
MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Paul S. Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A. Norman, Tanishq Mathew Abraham
TL;DR
MindEye2 addresses the need for high-quality visual reconstructions without the dozens of hours of subject-specific fMRI data traditionally required. It pretrains across seven subjects, functionally aligns a new subject with minimal data, and reconstructs images through CLIP-space prediction and SDXL-based generation. The approach achieves state-of-the-art retrieval and reconstruction performance with full data and competitive decoding using one hour of scanning.
Problem
Existing high-quality fMRI-to-image models are trained independently per subject and require dozens of hours of expensive fMRI data, limiting practical adoption.
Method
MindEye2 pretrains across subjects, linearly maps new-subject fMRI activity into a shared latent space, predicts CLIP image embeddings, and reconstructs pixels with fine-tuned SDXL.
Results
MindEye2 achieves state-of-the-art image retrieval and reconstruction performance with full data and competitive decoding using only 2.5% of a subject’s dataset, or 1 hour of scanning.
Takeaways & Limitations
The approach supports applying models pretrained on large-scale neuroimaging data to new subjects using minimal target-subject data.
Takeaways & Limitations
MindEye2 has only been shown on natural scenes, and decoding is sensitive to subject movement and task compliance.
Abstract
from arXiv · showhide
Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on GitHub.
1 Introduction
MindEye2 addresses the limited practical adoption of fMRI-to-image reconstruction by replacing independently trained, data-hungry subject models with shared-subject pretraining and minimal target-subject fine-tuning. Its functional alignment and integrated reconstruction pipeline support state-of-the-art performance with full data and competitive decoding from one hour of scanning.
- Single-subject approaches require dozens of hours of expensive fMRI data and have not generalized well across people, limiting practical adoption.
- The model is pretrained on other subjects, fine-tuned on a held-out subject, and maps brain activity into a shared CLIP-based image space for reconstruction.
- MindEye2 combines richer OpenCLIP embeddings, a fine-tuned Stable Diffusion XL unCLIP model, unified high- and low-level submodules, and predicted captions for refinement.
- 1 hour of scanning, equivalent to 2.5% of a subject’s full dataset, enables competitive decoding performance through MindEye2’s multi-subject alignment procedure.
- MindEye2 achieves state-of-the-art performance across image retrieval and reconstruction metrics when trained on the full Natural Scenes Dataset.
2 MindEye2
MindEye2 pretrains a shared reconstruction model across subjects, functionally aligns new brains with a simple linear mapping, and translates predicted image embeddings into pixels through SDXL-based generation. Its pipeline integrates diffusion, retrieval, low-level, and captioning components to preserve semantic and structural information during reconstruction.
- 2.1 Shared-Subject Functional Alignment: MindEye2 pretrains on 7 subjects and fine-tunes one held-out subject, enabling a shared pipeline rather than independently trained subject-specific models.The model uses native brain spaces with an initial functional alignment step, while later components are shared across subjects.
- 2.1 Shared-Subject Functional Alignment: Functional alignment is designed for low-sample, high-noise data and does not require subjects to view the same images, accommodating the Natural Scenes Dataset’s largely subject-unique stimuli.The authors describe simplicity and flexibility as strengths of the alignment procedure.
- 2.2 Backbone, Diffusion Prior, & Submodules: A linear mapping sends flattened fMRI activity into a 4096-dimensional shared-subject space before an MLP maps it to OpenCLIP ViT-bigG/14 image-token embeddings.The embeddings feed a diffusion prior and retrieval and low-level projectors.
- 2.2 Backbone, Diffusion Prior, & Submodules: The end-to-end objective sums diffusion-prior, retrieval, and low-level losses, allowing one pipeline to support reconstruction, retrieval, and low-level image information.The loss weights are α1 = .033 and α2 = .016.
- 2.4 Fine-tuning Stable Diffusion XL for unCLIP: A fine-tuned SDXL unCLIP model converts predicted OpenCLIP image embeddings to pixels, while base SDXL uses predicted captions to refine realism and semantic content.The unrefined reconstruction is passed through image-to-image refinement with text conditioning.
3 Results
MindEye2 achieves state-of-the-art fMRI-to-image performance with full NSD data and retains strong reconstruction, retrieval, and captioning performance after only 1 hour of target-subject fine-tuning.
- 3.1 fMRI-to-Image Reconstruction: MindEye2 achieved state-of-the-art performance across nearly all full-dataset reconstruction and retrieval metrics.The comparison averages results across NSD subjects 1, 2, 5, and 7.
- 3.1 fMRI-to-Image Reconstruction: 71.94% of human raters preferred refined over unrefined reconstructions, while 97.82% correctly identified MindEye2 reconstructions against random alternatives.Both behavioral results were statistically significant at p < 0.001.
- 3.1 fMRI-to-Image Reconstruction: With only 1 hour of data, MindEye2 improved qualitative reconstructions over other 1-hour approaches and was preferred to Brain Diffuser by human raters.These evaluations used the model fine-tuned with 1 hour of subject data.
- 3.1.1 VARYING AMOUNTS OF TRAINING DATA: Pretraining improved low-data performance, while pretrained and non-pretrained models became increasingly similar as training data increased.Non-pretrained models using 10 or 30 minutes showed significant instability, including potentially mode-collapsed outputs.
- 3.2 Image Captioning: MindEye2 captioning outperformed previous models across all reported metrics except one, indicating high-quality captions from brain activity.The evaluated metrics included ROUGE, METEOR, CLIP, and Sentence-Transformer scores.
- 3.3 Image/Brain Retrieval: Retrieval performance reached near-ceiling levels on established benchmarks and remained competitive when MindEye2 was trained with only 1 hour of data.Retrieval tests whether fMRI embeddings identify the exact matching image among semantically similar candidates.
- 3.5 Ablations: Ablations found that replacing the initial MLP with a linear layer and mapping to OpenCLIP bigG improved performance across all metrics.These changes were evaluated against the corresponding MindEye1 design choices.
- 3.5 Ablations: Removing diffusion-prior and low-level components harmed retrieval, while removing retrieval and low-level components harmed reconstruction.The ablations indicate that multiple training objectives provide mutually beneficial results.
4 Related Work
Related work develops methods for aligning brain data across subjects, but anatomical and some functional approaches face limitations in generalization or data requirements.
- Anatomical alignment is difficult because structural and functional brain topography differs substantially across people.
- Some functional alignment methods learn generalizable embedding spaces without requiring new subjects to view the same images.
- Prior across-subject reconstruction used subject-specific ridge-regression mappings into a shared space, but remained capped by the initial single-subject model and limited to linear fine-tuning.
5 Conclusion
MindEye2 reconstructs seen images from fMRI with scarce held-out-subject data and reaches state-of-the-art reconstruction and retrieval performance with full training data.
- MindEye2 achieves state-of-the-art reconstruction and retrieval metrics with full training data while matching prior reconstruction quality using only a fraction of training data.
- The method pretrains on multiple subjects, fine-tunes on scarce data from a held-out subject, maps fMRI activity to CLIP space, and reconstructs images with an SDXL-derived unCLIP model.
- MindEye2 has been demonstrated only on natural scenes, so other image distributions may require additional data or specialized generative models.
- The authors state that a single MRI session can provide enough data for high-quality visual-perception reconstructions, while noting systematic distortions and data-protection concerns.
6 Author Contributions
The author-contributions appendix directs readers to a separate detailed listing of individual contributions.
- Detailed author contributions are provided in Appendix A.1.
A.1 Author Contributions
The listed contributions cover project leadership, ablations, model comparisons, training infrastructure, and exploration of approaches not used in the final model.
- PSS led the project, drafted the initial manuscript, and contributed to all parts of MindEye2 development.
- MT contributed MindEye2 ablations, SDXL unCLIP comparisons, distributed-training improvements, and experiments with unused conditioning and retrieval approaches.
- The explored approaches included custom ControlNet and T2I adapters, COCO CLIP-caption retrieval, and diffusion-prior alignment to text embeddings.
A.2 Additional Dataset Information
The study uses normalized, flattened fMRI voxel responses from the NSD nsdgeneral region, with training-set statistics applied to both training and test data.
- A.2 Additional Dataset Information: Training and test fMRI data are voxel-wise Z-scored using means and standard deviations computed only from the training set.
A.3 MindEye2 (not pretrained) vs. MindEye1
MindEye2 outperforms MindEye1 even when both models are trained from scratch on one subject’s full 40-session dataset, indicating gains beyond cross-subject pretraining.
- A.3 MindEye2 (not pretrained) vs. MindEye1: MindEye2 outperforms MindEye1 across the reported evaluation metrics when both are trained from scratch on subject 1’s full 40-session dataset.The comparison indicates that the improvement is not explained solely by pretraining on other subjects.
A.4 Reconstruction Evaluations Across Varying Amounts of Training Data
The analysis examines how reconstruction, perceptual, and brain-correlation performance changes as subject 1 receives increasing numbers of fMRI training sessions.
- A.4 Reconstruction Evaluations Across Varying Amounts of Training Data: Low-level metric performance is plotted against training sessions, with a bold line showing the average across normalized metrics.
- A.4 Reconstruction Evaluations Across Varying Amounts of Training Data: High-level metric performance is plotted against training sessions, with SwAV and EffNet-B scores inverted so higher values consistently indicate better performance.
- A.4 Reconstruction Evaluations Across Varying Amounts of Training Data: Brain-correlation scores are compared across visual cortex, early visual areas, and higher visual areas as training data increase.The visual-cortex curve uses the nsdgeneral mask, while early visual cortex combines V1–V4.
A.5 Single-Subject Evaluations
The single-subject analyses compare 40-session and one-hour training, image-generation backends, retrieval constraints, latent-space organization, ROI behavior, and human judgments of reconstruction quality.
- A.5 Single-Subject Evaluations: MindEye2’s unCLIP model fine-tuned from Stable Diffusion XL outperforms Versatile Diffusion for reconstructing images from their respective ground-truth CLIP embeddings.The advantage is reported quantitatively and is visually apparent in the reconstruction comparison.
- A.5 Single-Subject Evaluations: OpenCLIP ViT-bigG image latents preserve low-level structure and high-level semantic information well enough for the SDXL unCLIP model to nearly recover the original image.
- A.5 Single-Subject Evaluations: Retrieval with OpenCLIP bigG embeddings could not scale to LAION-5B and showed dissimilar nearest neighbors in a 73,000-image MS-COCO index when the original image was absent.This limits direct reuse of MindEye1’s billion-image retrieval setup for MindEye2.
- A.5 Single-Subject Evaluations: Human raters identified the correct MindEye2 reconstruction 97.82% of the time, exceeding the reported 84.29% and 95.62% results of prior methods.
- A.5 Single-Subject Evaluations: Refined reconstructions were judged more similar to ground-truth images than unrefined reconstructions 71.94% of the time.This human preference supports the refinement step despite some quantitative metrics favoring unrefined outputs.
- A.5 Single-Subject Evaluations: MindEye2 reconstructions were preferred over Brain Diffuser reconstructions 53.01% of the time when both used one hour of training data.The result is reported as a statistically significant improvement over the previous one-hour reconstruction state of the art.