PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer

Roman Beliy, Amit Zalcher, Jonathan Kogman, Navve Wasserman, Michal Irani

arXiv:2510.25976v2cs.CVcs.AIq-bio.NC

TL;DR

Diffusion-based fMRI reconstructions can look realistic yet remain unfaithful to the seen image’s structure or semantics. Brain-IT maps shared functional voxel clusters to localized semantic and structural features for reconstruction, outperforming prior methods and matching full-data methods with one hour of new-subject data.

  • Problem

    Diffusion-based fMRI reconstructions can be visually pleasing yet miss or distort structural and semantic content of the seen image.

    Source: §1 · p. 2

  • Method

    Brain-IT maps shared functional brain-voxel clusters to localized semantic and structural image features used by complementary diffusion-guidance and coarse-layout branches.

    Source: §1 · pp. 2–3

  • Results

    Brain-IT outperforms previous methods on standard metrics, and 1 hour of new-subject data yields performance comparable to methods trained on 40 hours.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Shared functional clusters and common model components support integration across subjects and efficient transfer to new subjects with little subject-specific data.

    Source: §1 · p. 3

    Reconstructions remain imperfect, with semantic and fine-grained details sometimes unresolved.

    Source: §7 · p. 10

Abstract

from arXiv · show

Reconstructing images seen by people from their fMRI brain recordings provides a non-invasive window into the human brain. Despite recent progress enabled by diffusion models, current methods often lack faithfulness to the actual seen images. We present "Brain-IT", a brain-inspired approach that addresses this challenge through a Brain Interaction Transformer (BIT), allowing effective interactions between clusters of functionally-similar brain-voxels. These functional-clusters are shared by all subjects, serving as building blocks for integrating information both within and across brains. All model components are shared by all clusters & subjects, allowing efficient training with a limited amount of data. To guide the image reconstruction, BIT predicts two complementary localized patch-level image features: (i)high-level semantic features which steer the diffusion model toward the correct semantic content of the image; and (ii)low-level structural features which help to initialize the diffusion process with the correct coarse layout of the image. BIT's design enables direct flow of information from brain-voxel clusters to localized image features. Through these principles, our method achieves image reconstructions from fMRI that faithfully reconstruct the seen images, and surpass current SotA approaches both visually and by standard objective metrics. Moreover, with only 1-hour of fMRI data from a new subject, we achieve results comparable to current methods trained on full 40-hour recordings.

1 INTRODUCTION

Brain-IT addresses the gap between realistic fMRI reconstructions and faithful recovery of the images people saw. It maps shared functional brain-voxel clusters to localized image features and combines structural and semantic reconstruction branches.

Transfer learning

  • Figure 1 reports meaningful transfer-learning reconstructions from only 15 minutes of fMRI recordings for a new subject.Its full-data examples use 40 hours per subject.Source: Figure 1 · p. 2

Prior work

  • The introduction traces reconstruction methods from handcrafted and deep-learning features to diffusion models, while noting that individual variability remains a challenge.It also describes multi-subject integration as a response to scarce fMRI data.Source: §1 · p. 2

Motivation

  • Current methods can produce visually pleasing images that remain unfaithful in structure, such as position and color, and may miss or distort semantic content.The paper attributes this gap to fMRI representation extraction, mapping into image features, and their incorporation into generative models.Source: §1 · p. 2

Brain Interaction Transformer

  • BIT summarizes functionally similar brain-voxels in shared clusters and maps their interacting representations directly to localized image features.The shared clusters support information integration within and across subjects.Source: §1 · pp. 2–3

Results

  • Brain-IT outperforms prior methods on standard metrics, and with 1 hour of new-subject data achieves performance comparable to methods trained on 40 hours.The paper also reports meaningful reconstructions with 15 minutes of recordings.Source: Figure 1 · p. 2, §1 · p. 3

Complementary reconstruction branches

  • The pipeline uses semantic features to guide diffusion and inverts VGG-predicted structural features through Deep Image Prior to recover a coarse layout.The coarse image provides global structure, while diffusion refines it under semantic conditioning.Source: §1 · p. 3

2 RELATED WORK

Related work spans fMRI-to-image feature prediction, cross-subject integration, and low-level image reconstruction. Brain-IT’s stated distinctions are functional voxel clustering mapped to localized image features and a VGG-plus-DIP branch for coarse structure.

Field overview

  • The related-work overview traces a progression from handcrafted and CNN features through end-to-end and latent-code methods to diffusion-based image generation.It also notes growing interest in using multiple subjects to improve generalization when each subject has limited data.Source: §2 · p. 3

Image Features Prediction

  • Feature-prediction methods often compress voxels into a global embedding, while shared anatomical voxel patches still yield global image representations rather than localized information.Brain-IT instead maps functionally shared voxel clusters directly to localized image features.Source: §2 · p. 3

Pipeline overview

  • Figure 2 depicts Voxel-to-Cluster mapping into shared functional clusters, followed by VGG/DIP coarse reconstruction that initializes semantic diffusion guidance.Its overview also shows BIT predicting both semantic and VGG features from fMRI.Source: Figure 2 · p. 4

Cross-Subject Integration

  • Prior multi-subject methods commonly share whole-scan embeddings, whereas Brain-IT shares network weights across voxels within and across subjects.The related-work section says voxel-level similarities support cross-subject integration and adaptation to new subjects with limited data.Source: §2 · pp. 3–4

Low-Level Image Reconstruction

  • Unlike direct diffusion-latent regression or NeuroPictor’s U-Net feature manipulation, Brain-IT predicts VGG-based representations and inverts them through Deep Image Prior.The stated goal is more faithful low-level prediction of global structure and visual features.Source: §2 · p. 4

3 BRAIN-IT PIPELINE

Brain-IT maps fMRI into image features through shared functional voxel clusters, then combines coarse structural reconstruction with semantically guided diffusion. This two-branch pipeline uses the coarse image to initialize diffusion refinement.

3 BRAIN-IT PIPELINE

  • The pipeline predicts adapted CLIP and VGG image features from fMRI, using VGG features for a coarse reconstruction and semantic features to condition diffusion.At inference, the DIP-based coarse image initializes diffusion, which is refined under semantic guidance.Source: §3 · p. 4

3.1 IMAGE FEATURE PREDICTION

  • V2C groups functionally similar voxels from all subjects into 128 shared clusters, reducing 40K voxels and enabling information integration within and across brains.A Gaussian Mixture Model clusters voxel embeddings from the Brain Encoder of Beliy et al. (2024).Source: §3.1 · p. 5
  • BIT transforms fMRI signals and the V2C mapping into image features, including adapted CLIP embeddings for semantics and VGG features for low-level structure.Brain Tokens summarize voxels within clusters; attention layers refine them and extract image-feature information.Source: §3.1 · p. 5

3.2 IMAGE RECONSTRUCTION

  • The semantic branch conditions diffusion on adapted CLIP embeddings, while the low-level branch supplies coarse structure for diffusion refinement.The method jointly trains BIT and diffusion after an initial feature-alignment stage; the low-level image initializes diffusion in a noisy form.Source: §3.2 · p. 6
  • The low-level branch predicts multi-layer VGG features and inverts them through DIP to produce a coarse image for initializing semantic-guided diffusion.DIP optimizes an image whose frozen-VGG activations match BIT-predicted features under an L2 loss.Source: §3.2 · p. 6

4 BRAIN INTERACTION TRANSFORMER (BIT)

The Brain Tokenizer aggregates voxel activations into cluster-level Brain Tokens, which the Cross-Transformer refines and uses to predict localized image features.

Brain Tokenizer

  • The Brain Tokenizer converts fMRI activations into one Brain Token per functional voxel cluster by aggregating voxel information.Voxel embeddings modulate activations, and shared cluster embeddings guide the aggregation as an information-selection bottleneck.Source: §4 · p. 7

Cross-Transformer

  • Figure 5 compares one-hour reconstructions from Brain-IT, MindEye2, and MindTuner for Subject 1, with Brain-IT showing greater fidelity to the seen image.The caption points to Appendix Figure S3 for additional examples.Source: Figure 5 · p. 8
  • The Cross-Transformer refines Brain Tokens through self-attention and uses cross-attention to pass information to query tokens that predict localized image features.Each query token predicts one output image feature; the module uses five blocks with eight-head self- and cross-attention layers.Source: §4 · pp. 7–8

5 EXPERIMENTAL RESULTS

Across the reported evaluation, Brain-IT performs strongly on structural and semantic metrics, and its one-hour transfer results are comparable to prior methods trained on 40 hours. The experiments use NSD data and assess both image-level metrics and reconstruction examples.

5 EXPERIMENTAL RESULTS

  • The section also reports qualitative evaluation on NSD Synthetic as an out-of-distribution dataset.The section overview points to Figure S7 for this evaluation.Source: §5 · p. 8

5.1 EXPERIMENTAL SETTING

  • The evaluation uses NSD fMRI from eight subjects, with shared images as the test set and approximately 40K selected voxels.The reported standard table results average four subjects: Subjects 1, 2, 5, and 7.Source: §5.1 · p. 8

5.2 RECONSTRUCTION RESULTS

  • Figure 6 and Table 2 compare the semantic branch, low-level branch, and their combined reconstructions.The figure presents sample reconstructions; the table reports quantitative branch contributions.Source: Figure 6 · p. 9, Table 2 · p. 9
  • Brain-IT surpasses prior approaches by a large margin on all four low-level metrics, and achieves the best result on three high-level metrics, ranking second on one.The authors attribute the low-level advantage to BIT's VGG-feature predictions and their DIP inversion pipeline.Source: §5.2 · p. 9

5.3 TRANSFER LEARNING

  • With one hour of new-subject data, Brain-IT performs significantly better than MindEye2 and MindTuner on all metrics and is comparable to prior methods trained on 40 hours.The authors also report strong performance with 30 or 15 minutes of data.Source: §5.3 · p. 9

6 ABLATIONS AND ANALYSIS

The two reconstruction branches contribute complementary information: low-level features preserve structure, while semantic features capture image content. BIT cross-attention analysis also finds spatial and semantic selectivity among brain tokens.

Two-Branch Contribution

  • Combining the branches yields reconstructions that are more semantically accurate and structurally faithful than either branch alone.The low-level branch scores higher on pixel correlation and SSIM, while the semantic branch excels on perceptual metrics.Source: §6 · pp. 9–10

Brain-token selectivity

  • Different brain tokens consistently contribute to specific image-feature locations, with contralateral organization and selective attention to semantic regions such as faces, limbs, and text.The analysis averages cross-attention across layers and heads.Source: §6 · p. 10

7 DISCUSSION

Brain-IT reconstructs images from fMRI using shared functional clusters and supports transfer to new subjects with little subject-specific data. The discussion and appendix report ethical and reproducibility commitments and analyze training data, clustering, and cluster count.

7 DISCUSSION

  • Brain-IT reports improved structural and semantic similarity, while reconstructions can still miss semantic and fine-grained details.The authors suggest future work could explore more expressive feature spaces and note possible applications of BIT to other neuroscience questions.Source: §7 · p. 10

I. ETHICS STATEMENT

  • The work uses publicly available datasets and states that necessary measures to ensure ethical conduct were made.Source: §I · p. 11

II. REPRODUCIBILITY STATEMENT

  • The authors plan to release code, saved checkpoints, and reconstructed images upon publication to support reproducibility and comparisons under new metrics.Source: §II · p. 11

Appendix

  • The appendix provides ablations and analysis, further reconstruction evaluations and visuals, and additional technical details.Source: Appendix · p. 15

A ABLATIONS AND ANALYSIS

A.1 INFLUENCE OF UNLABELED IMAGES

A.2 FUNCTIONAL VS. ANATOMICAL CLUSTERING

  • Functional clustering provides superior reconstruction performance to anatomical alternatives, while the framework is compatible with coordinate- and atlas-based clustering.The tested Schaefer-1000 and Schaefer-400 atlases yielded 198 and 84 clusters, respectively.Source: §A.2 · p. 15, Table T2 · p. 15

A.3 EFFECT OF CLUSTERS NUMBER

  • Performance remains strong across a wide range of cluster counts, provided the count is not too small.The number of clusters is a method hyperparameter.Source: §A.3 · p. 16, Table T3 · p. 16

A.4 REFINEMENT STAGE VISUAL RESULTS

The refinement stage visually improves realism, while the two reconstruction branches contribute complementary strengths. The combined pipeline balances semantic and structural performance, and BIT outperforms an MLP baseline on low-level feature reconstruction.

A.4 REFINEMENT STAGE VISUAL RESULTS

  • A pretrained SDXL refinement stage slightly improves perceptual quality, enhancing realism, smoothing textures, and correcting details such as cat eyes.Source: §A.4 · p. 16, Figure S1 · p. 16

A.5 TWO-BRANCH CONTRIBUTION

  • Combining the semantic and low-level branches yields reconstructions that are both semantically accurate and structurally faithful.The low-level branch performs better on pixel correlation and SSIM, while the semantic branch leads on perceptual measures; the full pipeline balances both metric types.Source: §A.5 · p. 17, Table T4 · p. 17
  • BIT substantially outperforms an MLP with a 1K hidden dimension in predicting low-level visual features for Subject 1.Source: Table T5 · p. 17

B ADDITIONAL QUANTITATIVE RESULTS

The appendix evaluates additional metrics and transfer learning with very limited subject-specific data. It cautions that common metrics can obscure remaining performance gaps and reports strong transfer results with as little as 15 minutes of data.

B ADDITIONAL QUANTITATIVE RESULTS

  • Commonly reported saturated metrics, including 2-way retrieval, can misleadingly suggest little room for improvement; grayscale SSIM also misses chromatic information.The authors evaluate 1000-way CLIP retrieval, AlexNet-feature LPIPS, and RGB Color-SSIM to assess additional aspects of reconstruction quality.Source: B ADDITIONAL QUANTITATIVE RESULTS · p. 18, Table T6 · p. 18
  • With 15 minutes of subject-specific data (450 fMRI samples), Brain-IT achieves strong transfer performance and results comparable to prior prominent methods trained with 40 hours.The authors describe this as the first qualitative and quantitative comparison of this kind using less than one hour of subject-specific data.Source: B ADDITIONAL QUANTITATIVE RESULTS · p. 18, Table T7 · p. 18

C.1 RECONSTRUCTIONS WITH 40 HOURS OF DATA

Figure S2 presents qualitative reconstructions for subject 1, comparing Brain-IT with prominent previous methods across many images. The caption describes the results as visually faithful and semantically accurate.

C.1 RECONSTRUCTIONS WITH 40 HOURS OF DATA

  • Figure S2 compares Brain-IT reconstructions for subject 1 with prominent previous methods across a large number of images.The supplied caption identifies the comparison as demonstrating visual faithfulness alongside semantic accuracy.Source: Figure S2 · p. 19
  • Figure S2 Part B continues the qualitative reconstruction results for subject 1.Source: Figure S2 · p. 20
  • Figures S2 Parts C and D also present qualitative reconstruction results for subject 1.Source: Figure S2 · pp. 21–22

C.2 RECONSTRUCTIONS WITH TRANSFER-LEARNING ON LIMITED DATA

The supplementary results cover transfer learning with limited data, reconstructions across subjects and out-of-distribution stimuli, and analyses of brain-token selectivity and branch contributions. They also show cases where semantic content or image layout is not accurately predicted.

C.2 RECONSTRUCTIONS WITH TRANSFER-LEARNING ON LIMITED DATA

  • With 1 hour and even 30 minutes of transfer-learning data, Brain-IT reconstructions are described as comparable to prior methods using 40 hours.Source: Figure S3 · p. 23

C.3 IMAGE RECONSTRUCTIONS FOR ADDITIONAL SUBJECTS

  • Brain-IT produces effective reconstructions for subjects 1, 2, 5, and 7, with shared functional clusters and network modules leveraging cross-subject information.Source: Figure S4 · p. 27

C.4 BRAIN TOKENS SPATIAL AND SEMANTIC SELECTIVITY

  • Brain-token attention maps show contralateral spatial organization and tokens selectively contributing to semantic features such as hands, legs, words, and faces.The spatial and semantic analyses link attended image regions to the corresponding voxel clusters.Source: §C.4 · p. 29, Figure S5 · p. 30, Figure S6 · p. 31

C.5 RECONSTRUCTIONS ON NSD-SYNTHETIC

  • On NSD-Synthetic, an out-of-distribution extension of NSD, Brain-IT is applied without adaptation and the figure describes robust generalization.Source: Figure S7 · p. 32

C.6 TWO-BRANCH CONTRIBUTION IMAGE RECONSTRUCTIONS

  • The low-level and semantic branches contribute complementary information: low-level output often modifies semantic content accurately, while the semantic branch often carries important visual information.Source: Figure S8 · p. 33

C.7 FAILURE CASES

  • Some fMRI samples yield relatively accurate semantic content or visual layout, but not both, resulting in incorrect reconstructions.Source: Figure S9 · p. 33

D ADDITIONAL TECHNICAL DETAILS

D.1 TRAINING DETAILS

The technical details describe voxel sampling and unlabeled-image augmentation during training, the low-level reconstruction and diffusion pipeline, transfer learning, and the comparison protocol used for other methods.

D.1 TRAINING DETAILS

  • Training samples 15K voxels with replacement from about 40K per subject and augments labeled NSD data with predictions for about 120K unlabeled COCO images.For each unlabeled image, predicted fMRI for one randomly sampled subject is used as input.Source: §D.1 · p. 34

D.2 LOW-LEVEL BRANCH

  • The low-level branch uses predicted VGG features with DIP inversion to produce an image that initializes diffusion with structural information.The DIP output is upsampled from 112 × 112 to 256 × 256 before diffusion.Source: §D.2 · p. 34

D.3 INFERENCE TIME GENERATION

  • Diffusion starts from a partially noised low-level image, retaining structure while allowing semantic refinement over 38 denoising steps.The process begins at the noise level equivalent to step 14.Source: §D.3 · p. 35

D.4 TRANSFER LEARNING TRAINING DETAILS

  • Transfer learning adapts the Brain Encoder, then updates the new subject’s voxel embeddings while freezing other components and using limited subject-specific samples.The second step is supplemented with predicted fMRI responses from the adapted encoder on unlabeled data.Source: §D.4 · p. 35

D.5 EVALUATION AGAINST OTHER METHODS

  • Quantitative comparisons use previously published results and shared NSD test images, while Brain-IT is evaluated on all 40 sessions and the complete 1,000-image test set.Several listed baseline results use 982 rather than 1,000 test images because some methods were trained before the full NSD release.Source: §D.5 · p. 35

Related papers

Loading 2510.25976v2…